Attempting to select a <textarea> on a website and submit it? (Using Python Mechanize)

Question

I am trying to submit the textarea on https://keywordsheeter.com/ (name="findersearchQueryInput") then I want to be able to print the results of the submitted textarea to my console.

I tried form.set_value, form.find_control, and form["something"] = "something".

I realized this isn't working because the initial select_form() isn't finding the name (findersearchQueryInput) which is the name of the textarea on keywordsheeter.com

I was expecting to be able to find the form and go from there.

This is the code I have so far:

import mechanize

br = mechanize.Browser()
br.set_handle_robots(False)
br.addheaders = [('User-agent',
                  'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')]
br.open("https://keywordsheeter.com/")

print(br.title())

print(br.forms())

br.select_form("findersearchQueryInput")  # < not working...

br["keto diet "] = ["findersearchQueryInput"]

br.submit()

print(br.form)

Thanks so much in advance!

Answer 1

You should use selenium because Mechanize does not support JavaScript

first install it pip install selenium then follow the instruction get the appropriate webDriver https://sites.google.com/a/chromium.org/chromedriver/downloads

Then you can modify your code like that:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

webdriver_path = "path/to/chromedriver"

browser = webdriver.Chrome(executable_path=webdriver_path)
browser.get("https://keywordsheeter.com/")

# Find the textarea and enter your text
textarea = browser.find_element_by_name("findersearchQueryInput")
textarea.send_keys("keto diet")

# Submit the form by pressing Enter
textarea.send_keys(Keys.RETURN)

# Wait for the results to load
wait = WebDriverWait(browser, 10)
results = wait.until(EC.presence_of_element_located((By.CLASS_NAME, "result-list")))

# Print the results
print(results.text)

# Close the browser
browser.quit()