AWS Lambda - (The process started from chrome location /opt/chrome/chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)

Question

I am running a Lambda in AWS written in Python with selenium. This is lambda will open a chrome browser and navigate to a website. This lambda functions perfectly for initial few invocations and later to that the lambda errors out. Any help would be really appreciated. I have been spending hours in trying out different things.

Error message,

Message: session not created: Chrome failed to start: exited normally.
(chrome not reachable)
(The process started from chrome location /opt/chrome/chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)

Following is my configuration to create the driver,

chrome_options = Options()

service = Service("/opt/chromedriver")
chrome_options.binary_location = '/opt/chrome/chrome'
chrome_options.add_argument("--no-sandbox") # No protection needed
chrome_options.add_argument("--headless=new") # Hide the GUI
chrome_options.add_argument("--single-process") # Lambda only give us only one CPU
chrome_options.add_argument("--disable-dev-shm-usage") 
chrome_options.add_argument("--disable-extensions"); # disabling extensions
chrome_options.add_argument("--disable-gpu"); # applicable to windows os only
chrome_options.add_argument("--disable-infobars"); # disabling infobars
chrome_options.add_argument('--remote-debugging-port=9222')    

web_driver = webdriver.Chrome(options=chrome_options, service=service)

My Dockerfile configuration is,

FROM public.ecr.aws/lambda/python@sha256:a8770bd841c9caa98557ef7dc590bbd2707eb5ecab13bc8edbc81bf4961dca61 AS build
RUN yum install -y unzip && \
    curl -Lo "/tmp/chromedriver-linux64.zip" "https://storage.googleapis.com/chrome-for-testing-public/122.0.6261.69/linux64/chromedriver-linux64.zip" && \
    curl -Lo "/tmp/chrome-linux64.zip" "https://storage.googleapis.com/chrome-for-testing-public/122.0.6261.69/linux64/chrome-linux64.zip" && \
    unzip /tmp/chromedriver-linux64.zip -d /opt/ && \
    unzip /tmp/chrome-linux64.zip -d /opt/

FROM public.ecr.aws/lambda/python@sha256:a8770bd841c9caa98557ef7dc590bbd2707eb5ecab13bc8edbc81bf4961dca61
RUN yum install atk cups-libs gtk3 libXcomposite alsa-lib \
    libXcursor libXdamage libXext libXi libXrandr libXScrnSaver \
    libXtst pango at-spi2-atk libXt xorg-x11-server-Xvfb \
    xorg-x11-xauth dbus-glib dbus-glib-devel -y

# Copying requirements.txt and installing them
COPY requirements.txt ./
RUN pip install -r requirements.txt

# Copy the chrome binary and driver
COPY --from=build /opt/chrome-linux64 /opt/chrome
COPY --from=build /opt/chromedriver-linux64 /opt/

#Copy the python program file
COPY selenium_python_lambda.py ./

#Run the handler
CMD [ "selenium_python_lambda.lambda_handler" ]

requirements.txt is

selenium==4.10.0
unidecode==1.3.4
bs4==0.0.1