I have a small webscrapping script built with selenium, most specifically the undetected chromedriver and python on a ubuntu droplet in digital ocean, but i'm running through some issues.
When i'm running this in my computer, it works all fine, my environment:
- Mac Os Catalina version 10.15.7
- Python3.11.5, pip24.0
- Selenium4.16.0, undetected_chromedriver3.5.5
- Both GoogleChrome and ChromeDriver versions are 122.0.6261.94 (altho i'm not really using the chrome driver in my script, was before but then deleted it for reassons i don't remenber but was still working)
The script:
import undetected_chromedriver as uc
import time
from random import randint
import undetected_chromedriver as uc
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
chrome_options = uc.ChromeOptions()
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--headless=new')
chrome_options.add_argument('--enable-javascript')
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('--headless=new')
chrome_options.add_argument('--incognito')
user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.0 Safari/605.1.15'
chrome_options.add_argument('User-Agent={0}'.format(user_agent))
chrome_options.add_experimental_option("excludeSwitches", ["enable-automation"])
chrome_options.add_experimental_option('useAutomationExtension', True)
driver = uc.Chrome(chrome_options=chrome_options)
try:
driver.get('https://google.com')
wait = WebDriverWait(driver, 10)
wait.until(EC.presence_of_element_located((By.TAG_NAME, 'body')))
time.sleep(5)
except Exception as e:
print(f"Coudn't go to the website: ", e)
Environment in my digital ocean droplet:
- OS is Ubuntu 23.10 x64
- Python3.11.6, pip23.3
- Selenium4.18.1, Undetected_chromedriver3.5.5, webdriver-manager4.0.1
- Both GoogleChrome and ChromeDriver versions are 122.0.6261.94
Running locally this works 100%, but when i run in the DO droplet, it throws this error:
selenium.common.exceptions.WebDriverException: Message: unknown error: cannot connect to chrome at 127.0.0.1:46297
from chrome not reachable
Stacktrace:
#0 0x55a2a485be93 <unknown>
#1 0x55a2a4553b37 <unknown>
#2 0x55a2a453ea13 <unknown>
#3 0x55a2a458d601 <unknown>
#4 0x55a2a458416b <unknown>
#5 0x55a2a45cd71c <unknown>
#6 0x55a2a45c0c53 <unknown>
#7 0x55a2a4591db3 <unknown>
#8 0x55a2a459277e <unknown>
#9 0x55a2a48217cb <unknown>
#10 0x55a2a48257e5 <unknown>
#11 0x55a2a480f0e1 <unknown>
#12 0x55a2a4826372 <unknown>
#13 0x55a2a47f31bf <unknown>
#14 0x55a2a484a488 <unknown>
#15 0x55a2a484a683 <unknown>
#16 0x55a2a485b044 <unknown>
#17 0x7f656e497b5a <unknown>
Things i've tried to solve this:
- Adding some arguments to the chrome_options, such as --no-sandbox, --incognito, --headless=new, ...
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--headless=new')
chrome_options.add_argument('--enable-javascript')
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('--headless=new')
chrome_options.add_argument('--incognito')
chrome_options.add_experimental_option("excludeSwitches", ["enable-automation"])
chrome_options.add_experimental_option('useAutomationExtension', True)
Certifying that the chrome version installed in the machine "google-chrome --version", and the downloaded chromedriver is the same version. Both where downloaded from official sources.
Using the commands bellow to uninstall and again installing the undetected_chromedriver:
pip uninstall undetected_chromedriver
pip install undetected_chromedriver
- Tried using xfvb
from xvfbwrapper import Xvfb
vdisplay = Xvfb(width=800, height=1280)
vdisplay.start()
- Tried explicitly calling the executable_path when calling the chrome instance, but didn't work.
This error is still persisting, any tips or help on how to solve this will be welcome, thanks in advance.