Scrolling through page in Selenium Python

Question

Can you please explain/break-down the following syntax on how 'scrollTop' works in Selenium-Python:

inner_window = driver.find_element(By.XPATH, "XPATH")
driver.execute_script('arguments[0].scrollTop = arguments[0].scrollTop + 
document.getElementById("ID").scrollHeight;',
inner_window,)

Above code scrolls down directly to the bottom of the scroll bar and picks the values at the bottom, whereas I need to scroll through and pick values from top, middle and bottom of the page.

Also, how would I know when the bottom of the scroll is reached??

I also tried the below, but didn't work:

ht= driver.execute_script("return document.getElementById('ID').scrollHeight")
print(ht)
nht= ht//3
print(nht)
sht = str(nht)



inner_window = driver.find_element(By.XPATH, "XPATH")


for items in ree:
    time.sleep(1)
    wer=items.text.replace(' ','')
    #print(wer)
    arr1.append(wer)
    driver.execute_script('arguments[0].scrollTop = arguments[0].scrollTop + '+sht+';',
    inner_window,
    )


time.sleep(10)

print(arr1)

Answer 1

If you want to scroll to the end of an page with Selenium, the safest way isn't to use JavaScript, but to simulate pressing the End key:

from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys

body_element = driver.find_element(By.TAG_NAME, "body")
body_element.send_keys(Keys.END)

But there's an even easier way. In full disclosure, I'm the author of the Browserist package. Browserist is lightweight, less verbose extension of the Selenium web driver that makes browser automation even easier. Simply install the package with pip install browserist.

Browserist has several options for scrolling, just a few lines of code is needed. Examples:

from browserist import Browser

browser = Browser()
browser.open.url("https://stackoverflow.com")
browser.scroll.into_view("/html/body/div[3]/div[2]/div[1]/div[3]/div/div/div[6]")
browser.scroll.page.to_end()
browser.scroll.page.to_top()
browser.scroll.page.down()
browser.scroll.down_by(100)
browser.scroll.up_by(50)

Here's what I get (slowed down as Browserist finishes the job quickly). I hope this is helpful. Let me know if you have questions?