Rotating proxies that require authentication using selenium undetected chrome driver

43 Views Asked by At

Hello everyone I want to scrape this website but I need to rotate proxies since I always get blocked. [Please note that I am a beginner ]

Also I need to use Selenium webdriver (I tried to use only the requests library with BeautifulSoup but I get the following error 'Please enable JS and disable adblocker' , I also heard that I could use the request-html library [a more recent library , that enables javascript] I tried but with no success... ).

Moreover I use the undetected-chromedriver from this library (I read on forums that this library allowed a better headers optimization [am I right ? ])

So my problematic is the following : how can I rotate proxies using paid proxies (that need a username and a password , I bought datacenter proxies from SmartProxy ) with Selenium undetected chrome driver ?

Here is what I tried :

My proxy list ( paid proxies ) look something like this (they are in a text file called : smartproxies.txt )

http://USERNAME:[email protected]:10001
http://USERNAME:[email protected]:10002
http://USERNAME:[email protected]:10003
http://USERNAME:[email protected]:10004
http://USERNAME:[email protected]:10005

Then I used a for loop to iterate through all my proxies and go to this page : https://api.ipify.org/ a page that allows you to know your ip (just to see if i have correctly rotated proxies ) here is my script :

import undetected_chromedriver as uc


def sele(url, using_proxies=True):
    with open('smartproxies.txt', 'r') as f:
        proxies_list = f.read().splitlines()
    for proxy in proxies_list:
        options = uc.ChromeOptions()
        options.add_argument(f'--proxy-server={proxy}')
        driver = uc.Chrome(options=options)
        driver.get(url)
        sleep(3)
        driver.close()

if __name__ == "__main__":
    sele(url="https://api.ipify.org/")

However when the script runs , the page that open at each iteration of the loop shows me the same IP address ! I don't see my error

Also , is it really necessary to use the undetected chromedriver for that or would a regular chrome driver work ?

Thanks !

0

There are 0 best solutions below