How can I get needs html from site?

Question

Thank you for your attention,and sorry for my poor english. I have been trying to get html from https://www.skiddle.com/festivals/dates.html without any success. I understand, that some parts download by js script, but I don't know how to get it. I've also tried to use 'session' but stay with the same results. Pls, advice me about what I need to use in code or what I need to explore.

thanks in advance!!!

There is my code

import requests
from bs4 import BeautifulSoup
import lxml
from selenium import webdriver
import time
import undetected_chromedriver
import json


headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36 Edg/121.0.0.0'
}
proxies = {
    'https': 'http://146.247.105.71:4827'
}


def get_location(url):
    response = requests.get(url, headers=headers, proxies=proxies)
    soup = BeautifulSoup(response.text, 'lxml')
    print(soup, '\n\n\nlox\n\n\n')

    # options = undetected_chromedriver.ChromeOptions()

    # options.add_argument('--proxy-server=146.247.105.71:4827')

    # driver = undetected_chromedriver.Chrome(
    #     options=options
    # )
    # driver.get(url)
    # time.sleep(5)
    # response = driver.page_source
    # driver.close()
    # driver.quit()
    # print(response)


def main():
    get_location(url='https://www.skiddle.com/festivals/dates.html')


if __name__ == '__main__':
    main()

I need links on each feastival's page.

Answer 1

Here is an example how you can print festival name + URL:

import requests
from bs4 import BeautifulSoup

url = "https://www.skiddle.com/festivals/dates.html"

soup = BeautifulSoup(requests.get(url).content, "html.parser")
for a in soup.select("li.margin-bottom-10 a"):
    print(f'{a.text:<50} {a["href"]}')

Prints:

...

Levitation '24 at Bedford Esquires                 /whats-on/Bedford/Bedford-Esquires/Levitation-24/37157298/
Day at Historic Centreville Park                   /whats-on/united-states/Historic-Centreville-Park/Day/36718089/
When We Were Young at Las Vegas USA                https://www.skiddle.com/festivals/when-we-were-young/
When We Were Young at Las Vegas USA                https://www.skiddle.com/festivals/when-we-were-young/
Damnation Festival 2024 at BEC Arena               https://www.skiddle.com/festivals/damnation/
Hard Rock Hell at Vauxhall Holiday Park            https://www.skiddle.com/festivals/hard-rock-hell/

...