I want to scrape some dynamic website by Google Colab.
Some website has drop-down list using <select> and <option> tag.
I want to select each option and get rendered HTML, but HTML doesn't change.
How I modify code below?
I tried this code:
from requests_html import AsyncHTMLSession
url = "https://hoge.com"
session = AsyncHTMLSession()
r = await session.get(url)
await r.html.arender(timeout=60, keep_page=True)
select = r.html.find("select#list_select_horse")
if len(select) > 0:
print(select[0]) // <Element 'select' ... >
option = r.html.find("option")
print(option) // [<Element 'option' value='1' selected='selected'>, <Element 'option' value='1'>, <Element 'option' value='2'>, ... ]
f = await r.html.page.select('select#list_select_horse', "3")
print(f) // ["3"]
elements = await r.html.page.querySelectorAll('option')
print(elements)
option = r.html.find("option")
print(option) // [<Element 'option' value='1' selected='selected'>, <Element 'option' value='1'>, <Element 'option' value='2'>, ... ]
I expect last row displays [<Element 'option' value='3' selected='selected'>, <Element 'option' value='1'>, <Element 'option' value='2'>, ... ]