Running Pydoll in a Docker Container

[misotrnka]

Hi,

I'm experimenting with Pydoll and I'm trying to make it run in a Docker container. I was able to start the Chrome browser with Xvfb, I'm able to ping it (via browser._connection_handler.ping()), but when I try to navigate somewhere (e.g. await page.go_to('https://www.example.com')), it hits a time out with this:

pydoll/browser/page.py", line 208, in go_to
    raise TimeoutError('Page load timed out')

Is there anyone here that has experience with containerizing Pydoll apps? Is there any caveat that I'm not aware of?

[thalissonvs]

Hi @misotrnka , the xvfb is really necessary? You can just set the headless to True. Try this dockerfile:

FROM python:3.11.0

# Install dependencies
RUN apt -y update
RUN apt install -y gnupg
RUN apt install -y wget
RUN apt install -y iproute2
RUN apt install -y wkhtmltopdf
RUN apt install -y libx11-xcb1
RUN apt install -y libdbus-glib-1-2
RUN apt install -y git
RUN apt install -y tini


# Install google chrome browser
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
RUN wget -O /tmp/chrome.deb https://dl.google.com/linux/chrome/deb/pool/main/g/google-chrome-stable/google-chrome-stable_$CHROMEDRIVER_VERSION-1_amd64.deb
RUN apt install -y /tmp/chrome.deb
RUN rm /tmp/chrome.deb

# APP
WORKDIR /app

RUN pip install --upgrade pip
RUN pip install --upgrade pip wheel

RUN pip install pydoll-python

COPY . /app/

ENV PYTHONPATH=/app

[FriendlyUser]

I have tried to use pydoll in docker but if I use docker I cant bypass cloudflare reCAPTCHA is this expected? Or is it something wrong on my end.

[thalissonvs]

To successfully bypass detection in Docker containers, you need to be careful with a few things. First, make sure to change the user-agent, since the default headless user-agents are problematic. Some browser options are also important. I especially recommend using --enable-webgl.

options = ChromiumOptions()
options.add_argument("--user-agent=your-user-agent")
options.add_argument("--enable-webgl")
options.add_argument("--no-sandbox")

If that doesn't solve the issue, you might need to resort to more aggressive alternatives, such as using Xvfb and setting headless to False, to simulate a more realistic environment.

[misotrnka]

Thank you @thalissonvs , but I don't think Xvfb is really the problem. The behaviour is the same when I run in headless mode without Xvfb. I've tracked the issue down to this so far:

• There are pages where the page never reaches "loaded" state, which is something the go_to() method is waiting for. I've modified this so I could investigate further by waiting for the "interactive" state. Then I try to select an element on the page, I send the CDP command but never get a response from the browser and then I get a timeout.

Here's how it looks:

Command: {'method': 'Runtime.evaluate', 'params': {'expression': 'window.location.href', 'returnByValue': False}}
Response: {"id": 1, "result": {"result": {"type": "string", "value": "chrome://new-tab-page/"}}}

Command: {'method': 'Page.navigate', 'params': {'url': 'https://www.gerulata.com'}}
Response: {"id": 2, "result": {"frameId": "E70E639CFD9E6CCB01768737DB01D665", "loaderId": "203A14857893D63D9011A18D4A2A4478"}}

Command: {'method': 'Runtime.evaluate', 'params': {'expression': 'document.readyState', 'returnByValue': False}}
Response: {"id": 3, "result": {"result": {"type": "string", "value": "loading"}}}

... waiting for the "loaded" state

Command: {'method': 'Runtime.evaluate', 'params': {'expression': 'document.readyState', 'returnByValue': False}}
Response: {"id": 5, "result": {"result": {"type": "string", "value": "interactive"}}}

Command: {'method': 'Runtime.evaluate', 'params': {'expression': 'document.readyState', 'returnByValue': False}}
Response: {"id": 6, "result": {"result": {"type": "string", "value": "interactive"}}}

Command: {'method': 'Runtime.evaluate', 'params': {'expression': 'document.readyState', 'returnByValue': False}}
Response: {"id": 7, "result": {"result": {"type": "string", "value": "interactive"}}}

Command: {'method': 'Runtime.evaluate', 'params': {'expression': 'document.readyState', 'returnByValue': False}}
Response: Never received.

TimeoutError: Page load timed out

With the modified code, where I proceed when reached the "interactive" state (should be enough since I only need the DOM parsed), it will proceed to selecting an element on the page, but again the CDP protocol times out:

...
Command: {'method': 'Runtime.evaluate', 'params': {'expression': 'document.readyState', 'returnByValue': False}}
Response: {"id": 5, "result": {"result": {"type": "string", "value": "interactive"}}}

Command: {'method': 'Runtime.evaluate', 'params': {'expression': 'document.querySelector("#contact-us");', 'returnByValue': False}}
Response: Never received.

TimeoutError: Page load timed out

I haven't encountered this behavior on pages that do not linger in the "interactive" state and properly reach "loaded" within some reasonable time. However, there are many pages that will never be fully loaded for a number of reasons. So perhaps Pydoll should have a mechanism to detect whether the page loading actualy finished and don't rely on the page "loaded" state.

Still however, running the selector on a document in "interactive" state SHOULD work afaik. So maybe there is some other problem with the CDP. Since this only happens when running in a container, I believe the issue is actualy somehow related to CDP communication.

[cleitonleonel]

Thank you @thalissonvs , but I don't think Xvfb is really the problem. The behaviour is the same when I run in headless mode without Xvfb. I've tracked the issue down to this so far:

• There are pages where the page never reaches "loaded" state, which is something the go_to() method is waiting for. I've modified this so I could investigate further by waiting for the "interactive" state. Then I try to select an element on the page, I send the CDP command but never get a response from the browser and then I get a timeout.

Here's how it looks:

Command: {'method': 'Runtime.evaluate', 'params': {'expression': 'window.location.href', 'returnByValue': False}}
Response: {"id": 1, "result": {"result": {"type": "string", "value": "chrome://new-tab-page/"}}}

Command: {'method': 'Page.navigate', 'params': {'url': 'https://www.gerulata.com'}}
Response: {"id": 2, "result": {"frameId": "E70E639CFD9E6CCB01768737DB01D665", "loaderId": "203A14857893D63D9011A18D4A2A4478"}}

Command: {'method': 'Runtime.evaluate', 'params': {'expression': 'document.readyState', 'returnByValue': False}}
Response: {"id": 3, "result": {"result": {"type": "string", "value": "loading"}}}

... waiting for the "loaded" state

Command: {'method': 'Runtime.evaluate', 'params': {'expression': 'document.readyState', 'returnByValue': False}}
Response: {"id": 5, "result": {"result": {"type": "string", "value": "interactive"}}}

Command: {'method': 'Runtime.evaluate', 'params': {'expression': 'document.readyState', 'returnByValue': False}}
Response: {"id": 6, "result": {"result": {"type": "string", "value": "interactive"}}}

Command: {'method': 'Runtime.evaluate', 'params': {'expression': 'document.readyState', 'returnByValue': False}}
Response: {"id": 7, "result": {"result": {"type": "string", "value": "interactive"}}}

Command: {'method': 'Runtime.evaluate', 'params': {'expression': 'document.readyState', 'returnByValue': False}}
Response: Never received.

TimeoutError: Page load timed out

With the modified code, where I proceed when reached the "interactive" state (should be enough since I only need the DOM parsed), it will proceed to selecting an element on the page, but again the CDP protocol times out:

...
Command: {'method': 'Runtime.evaluate', 'params': {'expression': 'document.readyState', 'returnByValue': False}}
Response: {"id": 5, "result": {"result": {"type": "string", "value": "interactive"}}}

Command: {'method': 'Runtime.evaluate', 'params': {'expression': 'document.querySelector("#contact-us");', 'returnByValue': False}}
Response: Never received.

TimeoutError: Page load timed out

I haven't encountered this behavior on pages that do not linger in the "interactive" state and properly reach "loaded" within some reasonable time. However, there are many pages that will never be fully loaded for a number of reasons. So perhaps Pydoll should have a mechanism to detect whether the page loading actualy finished and don't rely on the page "loaded" state.

Still however, running the selector on a document in "interactive" state SHOULD work afaik. So maybe there is some other problem with the CDP. Since this only happens when running in a container, I believe the issue is actualy somehow related to CDP communication.

Hello @misotrnka , could you provide the codes for your experiment and also the links to the pages used in the tests?

[rastarr]

@misotrnka Did you ever get this working?
i'd love to see a docker container that successfully operates with pydoll

[misotrnka]

Yes, in fact I did. I've used this Zendriver-Docker repo for the image. You just need to provide the correct binary location:

from pydoll.browser.options import Options

options = Options()
options.binary_location = "google-chrome-stable"

Then I've found that some additional browser configuration is needed, so I'm using these options. Not sure if all of these are required to make this work (probably not), but this is what I've ended up after some experimentations and also leveraging experience of my colleagues from other projects.

options.add_argument("--enable-features=UseOzonePlatform")
options.add_argument("--ozone-platform=wayland")
options.add_argument("--disable-background-networking")
options.add_argument("--disable-background-timer-throttling")
options.add_argument("--disable-blink-features AutomationControlled")
options.add_argument("--disable-dev-shm-usage")
options.add_argument("--disable-extensions")
options.add_argument("--disable-gpu")
options.add_argument("--disable-notifications")
options.add_argument("--disable-sync")
options.add_argument("--disable-translate")
options.add_argument("--disable-web-security")
options.add_argument("--js-flags")
options.add_argument("--max-old-space-size=500")

Hope this helps a bit.

[rastarr]

i've spent a few days on trying to get this to work, without sucess. mostly using chatgpt to assist but keep getting the same bunch of errors and no progress unfortunately. my coding skills aren't nearly up to the task for getting a working pydoll docker image working and then i'm sure my app.py is totally borked too. maybe at some point there'll be a pydoll docker image and app.py that will become available, as a starting point. Great you got yours working though, kudos to you