Why can't I grab an element on a webpage with selenium in python? Tried by id, xpath, full xpath

Question

I'm trying to write a script that will auto-fill an online form with python and selenium, but I'm having trouble grabbing an element from the html code.

This is my starting code. Everything works until I try to identify the element. The element's id changes for every new instance of the website, so I'm trying a workaround which I believe grabs the correct id for each instance:

from selenium import webdriver
from selenium.webdriver.common.by import By
import requests
web = webdriver.Chrome()
url = 'https://lincdoc.ou.edu/lincdoc/doc/run/ouathletics/OU_AdvisingForm2#ldTimeoutUri'
web.get(url)
html_text = requests.get(url).text
instance_id_index = html_text.index("<div id=")
instance_id = html_text[instance_id_index+9] + html_text[instance_id_index+10] + html_text[instance_id_index+11] + html_text[instance_id_index+12] 
student_id = "ID NUMBER"
student_id_box_id = instance_id + "q4"

I've tried the following, which all come back with a no such element exception:

(1) tried and failed to grab student id textbox element by id

web.find_element(By.ID, student_id_box_id).send_keys(student_id)

(2) tried and failed to grab student id textbox element by xpath

student_id_box_xpath = '//*[@id="' + instance_id + 'q4"]'
web.find_element(By.XPATH, student_id_box_xpath).send_keys(student_id)

(3) tried and failed to grab student id textbox element by full xpath

student_id_box_xpath = '//*[@id="' + instance_id + 'q4"]'
student_id_box_full_xpath = student_id_box_xpath + '/html/body/div/div[2]/div/div[2]/div[2]/div[3]/div/div[3]/table/tbody[2]/tr[1]/td[3]/div/input'
web.find_element(By.XPATH, student_id_box_full_xpath).send_keys(student_id)

Can anyone shed some light on this? I'm very very new to programming and python and it took awhile for me to get even this far. Thanks in advance!

Answer 1

You can easily use an attribute = value css selector with $ ends with operator to specify the input element with id ending with q4

from selenium.webdriver.common.by import By
from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait 

web = webdriver.Chrome()
web.get('https://lincdoc.ou.edu/lincdoc/doc/run/ouathletics/OU_AdvisingForm2#ldTimeoutUri')
student_id_field = WebDriverWait(web, 5).until(EC.presence_of_element_located((By.CSS_SELECTOR, "input[id$=q4]")))
student_id_field.send_keys("some id goes here")

Answer 2

the id of this page is dynamic, it is better to use table, row as anchor element, I wrote one sample to solve this issue, you can view the full code from here

sample as the following:

from clicknium import clicknium as cc, locator, ui

tab = cc.chrome.open("https://lincdoc.ou.edu/lincdoc/doc/run/ouathletics/OU_AdvisingForm2#ldTimeoutUri")
tab.find_element(locator.chrome.lincdoc.text_advising_unit).set_text("Athletics-OMS 256/(405) 325-8373")
tab.find_element(locator.chrome.lincdoc.text_advising_name).set_text("AJ [email protected]")

tab.find_element(locator.chrome.lincdoc.text_student_id).set_text("student1")
tab.find_element(locator.chrome.lincdoc.text_first_name).set_text("tom")
tab.find_element(locator.chrome.lincdoc.text_last_name).set_text("jack")
tab.find_element(locator.chrome.lincdoc.text_phone_number).set_text("12345")
tab.find_element(locator.chrome.lincdoc.text_ou_email).set_text("[email protected]")
tab.find_element(locator.chrome.lincdoc.text_student_id).set_text("Junior")

tab.find_element(locator.chrome.lincdoc.checkbox_topic, {'topic':'Academic Contract'}).set_checkbox()

print('done')