Robot Framework test case fails with “Element not found” when running selenium library based test case with headless chrome inside a DOCKER container

CHIDAMBARANATHAN RAMACHANDRAN

unread,

Mar 2, 2021, 5:29:31 PM3/2/21

to robotframework-users

Below is the test case that I am trying to execute inside the docker container.

Login To GUI

[Documentation] To open GUI and login with valid credentials

${chrome_options}= Evaluate sys.modules['selenium.webdriver'].ChromeOptions() sys, selenium.webdriver

Call Method ${chrome_options} add_argument --no-sandbox

Call Method ${chrome_options} add_argument --headless

Call Method ${chrome_options} add_argument --disable-dev-shm-usage

Call Method ${chrome_options} add_argument --ignore-certificate-errors-spki-list

Call Method ${chrome_options} add_argument --ignore-ssl-errors

Open Browser ${url} chrome options=${chrome_options} executable_path=/usr/lib/chromium/chromedriver

Set Browser Implicit Wait 5

Input Text id=username ${username}

Input Text id=password ${password}

Click Button //input[@value='Sign in']

The test case passed successfully when I tried to execute it directly from IDE (Pycharm) in the MAC terminal. But, When I tried to perform the same via docker container, it fails with error “Element with locator 'id=username' not found” and a blank white screen is attached as part of screenshot in logs. The page I request should get redirected to an authentication page (key cloak) with the username password field, but I am getting blank page in the docker container.

I checked the log file inside container “/usr/lib/chromium/chrome_debug.log”

[0302/115225.286372:WARNING:dns_config_service_posix.cc(342)] Failed to read DnsConfig.

[0302/115226.149284:ERROR:cert_issuer_source_aia.cc(32)] Error parsing cert retrieved from AIA (as DER): ERROR: Failed parsing Certificate SEQUENCE ERROR: Failed parsing Certificate

[0302/115226.345313:ERROR:cert_issuer_source_aia.cc(32)] Error parsing cert retrieved from AIA (as DER): ERROR: Failed parsing Certificate SEQUENCE ERROR: Failed parsing Certificate

[0302/115226.345462:ERROR:cert_issuer_source_aia.cc(104)] AiaRequest::OnFetchCompleted got error -301

[0302/115226.346040:ERROR:ssl_client_socket_impl.cc(960)] handshake failed; returned -1, SSL error code 1, net_error -202

Then I tried the below command inside the container and I got:

/usr/lib/chromium # chromium-browser --headless --no-sandbox --ignore-certificate-errors --ignore-ssl-errors https://<url>

[0302/115903.090501:ERROR:bus.cc(393)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory

[0302/115903.091302:WARNING:dns_config_service_posix.cc(342)] Failed to read DnsConfig.

[0302/115903.152546:WARNING:dns_config_service_posix.cc(342)] Failed to read DnsConfig.

[0302/115903.631311:ERROR:cert_issuer_source_aia.cc(32)] Error parsing cert retrieved from AIA (as DER): ERROR: Failed parsing Certificate SEQUENCE ERROR: Failed parsing Certificate

[0302/115903.633207:ERROR:cert_issuer_source_aia.cc(32)] Error parsing cert retrieved from AIA (as DER): ERROR: Failed parsing Certificate SEQUENCE ERROR: Failed parsing Certificate

[0302/115903.633315:ERROR:cert_issuer_source_aia.cc(104)] AiaRequest::OnFetchCompleted got error -301

[0302/115904.273717:INFO:CONSOLE(27)] "Mixed Content: The page at 'https://<url>/auth/realms/ml/protocol/openid-connect/auth?client_id=ml-client&redirect_uri=https%3A%2F%2F<url>%2Foauth%2Fcallback&response_type=code&scope=ml-scope+openid+email+profile&state=6d35f7-add8-40b-a8e7-b169876cfc' was loaded over a secure connection, but contains a form that targets an insecure endpoint 'http://ml-sec-access-mgmt-http:8080/auth/realms/ml/login-actions/authenticate?session_code=mrjXrpjeadGywFIIgkHhddBag74tDnWV6FHA3Qk&execution=f19849-6670-406c-a1b0-139bb1f1dc05&client_id=ml-client&tab_id=vGTrJ7OI8'. This endpoint should be made available over a secure connection.", source: https://<url>/auth/realms/ml/protocol/openid-connect/auth?client_id=ml-client&redirect_uri=https%3A%2F%2F<url>%2Foauth%2Fcallback&response_type=code&scope=ml-scope+openid+email+profile&state=6d85f7-add8-40db-a8e7-b16239876cfc (27)

I even download the chromium browser in my MAC and tried opening the URL it works fine.

Docker File [Reference: https://github.com/ppodgorsek/docker-robot-framework/blob/master/Dockerfile]:

#Base image

FROM python:3.9.0-alpine3.12

# Set the reports directory environment variable

ENV ROBOT_REPORTS_DIR /opt/robotframework/reports

# Set the tests directory environment variable

ENV ROBOT_TESTS_DIR /opt/robotframework/tests

# Set the working directory environment variable

ENV ROBOT_WORK_DIR /opt/robotframework/temp

# Set number of threads for parallel execution

# By default, no parallelisation

ENV ROBOT_THREADS 1

# Install system dependencies

RUN apk update \

&& apk --no-cache upgrade \

&& apk --no-cache --virtual .build-deps add \

gcc \

libffi-dev \

linux-headers \

make \

musl-dev \

openssl-dev \

which \

wget \

curl \

vim \

ca-certificates \

git \

jq \

chromium \

chromium-chromedriver

#Install robotframework and required libraries from the requirements file

ADD requirements.txt /

RUN pip3 install \

--no-cache-dir \

-r requirements.txt

# Create the default report and work folders with the default user to avoid runtime issues

# These folders are writeable by anyone, to ensure the user can be changed on the command line.

RUN mkdir -p ${ROBOT_REPORTS_DIR} \

&& mkdir -p ${ROBOT_WORK_DIR} \

&& chmod ugo+w ${ROBOT_REPORTS_DIR} ${ROBOT_WORK_DIR}

# Installing product related utilities inside the container

XXXXX<contents are not >

# Allow any user to write logs

RUN chmod ugo+w /var/log

# Update system path

ENV PATH=/opt/robotframework/bin:$PATH

# A dedicated work folder to allow for the creation of temporary files

WORKDIR ${ROBOT_WORK_DIR}

Requirements.txt file contents:

#Required robot framework packages

robotframework==3.2.2

robotframework-requests==0.7.2

robotframework-seleniumlibrary==4.5.0

robotframework-jsonlibrary==0.3.1

robotframework-kubelibrary==0.2.0

I even referred the link Getting empty page running selenium in headless chrome Docker.

I could not figure out what could be the issue. Is it really a redirect issue or certificate issue or Mixed content? I am quite confused. Any ideas?

Dave Amies

unread,

Mar 3, 2021, 3:59:35 AM3/3/21

to robotframework-users

The error from “/usr/lib/chromium/chrome_debug.log” is complaining about a https certificate error

But this error seems to be the key:

[0302/115904.273717:INFO:CONSOLE(27)] "Mixed Content: The page at 'https://<url>/auth/realms/ml/protocol/openid-connect/auth?client_id=ml-client&redirect_uri=https%3A%2F%2F<url>%2Foauth%2Fcallback&response_type=code&scope=ml-scope+openid+email+profile&state=6d35f7-add8-40b-a8e7-b169876cfc' was loaded over a secure connection, but contains a form that targets an insecure endpoint 'http://ml-sec-access-mgmt-http:8080/auth/realms/ml/login-actions/authenticate?session_code=mrjXrpjeadGywFIIgkHhddBag74tDnWV6FHA3Qk&execution=f19849-6670-406c-a1b0-139bb1f1dc05&client_id=ml-client&tab_id=vGTrJ7OI8'. This endpoint should be made available over a secure connection.", source: https://<url>/auth/realms/ml/protocol/openid-connect/auth?client_id=ml-client&redirect_uri=https%3A%2F%2F<url>%2Foauth%2Fcallback&response_type=code&scope=ml-scope+openid+email+profile&state=6d85f7-add8-40db-a8e7-b16239876cfc (27)

What it seems to be saying is that the login form of you application was loaded from a secure site (https) but when you submit the form it will submit the form data to an insecure (http) site.

Not sure if "http://ml-sec-access-mgmt-http:8080/" and "https://<url>/" are the same site or not? I would suggest you load the login form manually on your machine and then using the browser's dev tools check the form's action attribute, if it is indeed a http:// url you should discuss with your development/deployment team.

Hope this helps,

Dave.

CHIDAMBARANATHAN RAMACHANDRAN

unread,

Mar 3, 2021, 6:09:41 AM3/3/21

to robotframework-users

Hi Dave,

Thanks for the response.

Test Case Description: The test case tries to load a URL via browser and it connects to keycloak for authentication and then keycloak loads the keycloak login page with username, password and other elements. Then the credentials are entered and submit button is clicked.

I am able to successfully run the test case in my local machine, whereas I am facing only when I run inside the docker container. Here it seems application code is working fine.

Just a comparison below between local and docker container environment,

The login page "view page source" looks like below:

.

.

</html>

Regards,

Nathan

Dave Amies

unread,

Mar 3, 2021, 5:59:03 PM3/3/21

to robotframework-users

Hi Nathan,

I noticed that you are running Chrome on the Mac and Chromium in the docker image (do you have the versions for both?)

Chromium and Chrome are not the same, they are very similar but not the same, you said you ran this on the docker container and got the error outside RF:

/usr/lib/chromium # chromium-browser --headless --no-sandbox --ignore-certificate-errors --ignore-ssl-errors https://<url>

I suggest you try downloading Chromium on your Mac and try running the same command on your Mac and see what the result is, this might narrow it down to a Chromium setting.

If we can confirm it's Chromium, it simplifies things so you have 2 options:

1) figure out how to make the site work with Chromium

2) use Chrome on the docker image

CHIDAMBARANATHAN RAMACHANDRAN

unread,

Mar 7, 2021, 11:38:06 AM3/7/21

to robotframework-users

Hi Dave,

Unfortunately it seems chrome is not available for alpine linux. So, I changed to ubuntu 20:04 as a base image for the docker container and used chrome.

But again I am getting the same blank page issue.

root@a4ac8fd9a950:/opt/google/chrome# google-chrome --headless --no-sandbox https://<URL>

[0306/152227.264852:WARNING:headless_content_main_delegate.cc(530)] Cannot create Pref Service with no user data dir.

[0306/152227.265234:WARNING:discardable_shared_memory_manager.cc(194)] Less than 64MB of free space in temporary directory for shared memory files: 63

[0306/152227.269687:ERROR:bus.cc(393)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory

[0306/152228.160231:ERROR:cert_issuer_source_aia.cc(32)] Error parsing cert retrieved from AIA (as DER):

ERROR: Failed parsing Certificate SEQUENCE

ERROR: Failed parsing Certificate

[0306/152228.363766:ERROR:cert_issuer_source_aia.cc(32)] Error parsing cert retrieved from AIA (as DER):

ERROR: Failed parsing Certificate SEQUENCE

ERROR: Failed parsing Certificate

[0306/152228.363958:ERROR:cert_issuer_source_aia.cc(104)] AiaRequest::OnFetchCompleted got error -301

[0306/152228.364625:ERROR:ssl_client_socket_impl.cc(924)] handshake failed; returned -1, SSL error code 1, net_error -202

But I made this thing working, by using Xvfb [Xvfb (short for X virtual framebuffer) is an in-memory display server for UNIX-like operating system (e.g., Linux). It enables you to run graphical applications without a display (e.g., browser tests on a CI server) while also having the ability to take screenshots.]. It worked without any problem. But Xvfb is not there with alpine and so I used Ubuntu. Rewritten the entire docker file contents with ubuntu base image and it worked after minor environmental hiccups.

Thanks a lot for your support. It helped me a lot in thinking from different perspective. Hope my solution is fine. Please feel free share your views/suggestions.

New docker file content for reference: [Giving it for others reference]

FROM ubuntu:20.04

# Set the reports directory environment variable

ENV ROBOT_REPORTS_DIR /opt/robotframework/reports

# Set the tests directory environment variable

ENV ROBOT_TESTS_DIR /opt/robotframework/tests

# Set the working directory environment variable

ENV ROBOT_WORK_DIR /opt/robotframework/temp

# Set number of threads for parallel execution

# By default, no parallelisation

ENV ROBOT_THREADS 1

ENV DEBIAN_FRONTEND=noninteractive

# Install system dependencies

RUN apt-get update \

&& apt-get install --quiet --assume-yes \

python3-pip \

unzip \

firefox \

wget \

curl \

vim \

ca-certificates \

git \

jq \

xvfb

# Install chrome package

RUN wget --no-verbose https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb

RUN dpkg --install google-chrome-stable_current_amd64.deb; apt-get --fix-broken --assume-yes install

#Install robotframework and required libraries from the requirements file

ADD requirements.txt /

RUN pip3 install \

--no-cache-dir \

-r requirements.txt

# Install webdrivers for chrome and firefox

RUN CHROMEDRIVER_VERSION=`wget --no-verbose --output-document - https://chromedriver.storage.googleapis.com/LATEST_RELEASE` && \

wget --no-verbose --output-document /tmp/chromedriver_linux64.zip http://chromedriver.storage.googleapis.com/$CHROMEDRIVER_VERSION/chromedriver_linux64.zip && \

unzip -qq /tmp/chromedriver_linux64.zip -d /opt/chromedriver && \

chmod +x /opt/chromedriver/chromedriver && \

ln -fs /opt/chromedriver/chromedriver /usr/local/bin/chromedriver

RUN GECKODRIVER_VERSION=`wget --no-verbose --output-document - https://api.github.com/repos/mozilla/geckodriver/releases/latest | grep tag_name | cut -d '"' -f 4` && \

wget --no-verbose --output-document /tmp/geckodriver.tar.gz https://github.com/mozilla/geckodriver/releases/download/$GECKODRIVER_VERSION/geckodriver-$GECKODRIVER_VERSION-linux64.tar.gz && \

tar --directory /opt -zxf /tmp/geckodriver.tar.gz && \

chmod +x /opt/geckodriver && \

ln -fs /opt/geckodriver /usr/local/bin/geckodriver

# Create the default report and work folders with the default user to avoid runtime issues

# These folders are writeable by anyone, to ensure the user can be changed on the command line.

RUN mkdir -p ${ROBOT_REPORTS_DIR} \

&& mkdir -p ${ROBOT_WORK_DIR} \

&& chmod ugo+w ${ROBOT_REPORTS_DIR} ${ROBOT_WORK_DIR}

# Installing product related utilities inside the container

XXXXXXXX

# Allow any user to write logs

RUN chmod ugo+w /var/log

# Update system path

ENV PATH=/opt/robotframework/bin:$PATH

# A dedicated work folder to allow for the creation of temporary files

WORKDIR ${ROBOT_WORK_DIR}

Regards,

Nathan

Dave Amies

unread,

Mar 8, 2021, 7:19:19 AM3/8/21

to robotframework-users

Hi Nathan,

I actually built a docker image for rfswarm agent and chrome, I used ubuntu 20.04 as well for the same reason, of no support for Chrome in Alpine.

One thing I found was that it was necessary to configure the docker file to switch to a non-root user:

```

USER 1001

```

This is because chrome will error and crash out when run as root (this appears to be intentional by google) and would explain these errors you got:

```

[0306/152227.264852:WARNING:headless_content_main_delegate.cc(530)] Cannot create Pref Service with no user data dir.

[0306/152227.265234:WARNING:discardable_shared_memory_manager.cc(194)] Less than 64MB of free space in temporary directory for shared memory files: 63

[0306/152227.269687:ERROR:bus.cc(393)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory

```

but you may find as a non-root user you may have other permissions issues and might need to configure permissions for your user before switching (chmod)

I also needed to have a shell script as my entry point to run the following:

```

GEOMETRY="${SCREEN_WIDTH}x${SCREEN_HEIGHT}x${SCREEN_DEPTH}"

Xvfb :99 -screen 0 $GEOMETRY &

export DISPLAY=:99

```

before robot runs so that chrome can find the Xvfb display. (I got that from this example https://github.com/eficode/robotframework-selenium/tree/master/robotframework-selenium-chrome)

While I got it working I needed to use the --no-sandbox option to get chrome to run, and even then chrome would still crash on me if I ran more than 4 chrome browsers on each docker container because the non sandboxed chrome sessions would trip over each other, so it was not a useful solution for me.

From my experience chrome & chromium don't behave very well in a docker container, FireFox on the other hand plays well.

I have not yet tried to run windows or mac docker containers so no idea if they behave any better. if you really need to test with Chrome you might find life easier to use virtual machines (e.g. virtualbox) rather than docker containers.

Finally I also noticed you are still getting SSL Cert errors, but I don't know if that ir related to the earlier errors about not having a temporary directory or if these are the same issue as you had with chromium.

Hope this helps,

Dave.

Wentao Ma

unread,

Jul 12, 2022, 10:40:55 AM7/12/22

to robotframework-users

The same error in Mac m1, It works on firefox. Can not work on chromium and webkit , different errors .

------------------------------------------

*** Settings ***

Library Browser

*** Variables ***

${browser} chromium

*** Test Cases ***

Example Test

Log To Console Using ${browser}

New Browser ${browser} headless=true

New Page https://playwright.dev

Sleep 1s

Get Text h1 contains Playwright

------------------------------------------------

mawentao@MacBook-Pro UItest % docker run --rm --name rfbrowser -v /Users/mawentao/PycharmProjects/UItest/docker/test/:/test --ipc=host --user pwuser --security-opt seccomp=docker/seccomp_profile.json marketsquare/robotframework-browser:latest bash -c "robot --outputdir /test/output /test"

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

==============================================================================

Test

==============================================================================

Test.Chromium Test

==============================================================================

Example Test Using chromium

[ WARN ] Keyword 'Take Screenshot fail-screenshot-{index}' could not be run on failure:

Error: Tried to take screenshot, but no page was open.

| FAIL |

Error: browser.newContext: Browser closed.

==================== Browser output: ====================

<launching> /home/pwuser/.local/lib/python3.8/site-packages/Browser/wrapper/node_modules/playwright-core/.local-browsers/chromium-1012/chrome-linux/chrome --disable-field-trial-config --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disable-background-timer-throttling --disable-backgrounding-occluded-windows --disable-back-forward-cache --disable-breakpad --disable-client-side-phishing-detection --disable-component-extensions-with-background-pages --disable-default-apps --disable-dev-shm-usage --disable-extensions --disable-features=ImprovedCookieControls,LazyFrameLoading,GlobalMediaControls,DestroyProfileOnBrowserClose,MediaRouter,DialMediaRouteProvider,AcceptCHFrame,AutoExpandDetailsElement,CertificateTransparencyComponentUpdater,AvoidUnnecessaryBeforeUnloadCheckSync --allow-pre-commit-input --disable-hang-monitor --disable-ipc-flooding-protection --disable-popup-blocking --disable-prompt-on-repost --disable-renderer-backgrounding --disable-sync --force-color-profile=srgb --metrics-recording-only --no-first-run --enable-automation --password-store=basic --use-mock-keychain --no-service-autorun --export-tagged-pdf --headless --hide-scrollbars --mute-audio --blink-settings=primaryHoverType=2,availableHoverTypes=2,primaryPointerType=4,availablePointerTypes=4 --no-sandbox --user-data-dir=/tmp/playwright_chromiumdev_profile-z6v6Nl --remote-...

[ Message content over the limit has been removed. ]

...onnect to socket /var/run/dbus/system_bus_socket: No such file or directory

[pid=260][err] [0712/143415.784430:ERROR:bus.cc(398)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory

[pid=260][err] [0712/143415.792297:ERROR:file_path_watcher_inotify.cc(329)] inotify_init() failed: Function not implemented (38)

[pid=260][err] qemu: unknown option 'type=utility'

[pid=260][err] [0712/143415.957269:WARNING:bluez_dbus_manager.cc(247)] Floss manager not present, cannot set Floss enable/disable.

[pid=260][err] [0712/143415.959141:ERROR:gpu_process_host.cc(959)] GPU process launch failed: error_code=1002

[pid=260][err] [0712/143415.959644:WARNING:gpu_process_host.cc(1267)] The GPU process has crashed 1 time(s)

[pid=260][err] [0712/143415.991067:ERROR:gpu_process_host.cc(959)] GPU process launch failed: error_code=1002

[pid=260][err] [0712/143415.991176:WARNING:gpu_process_host.cc(1267)] The GPU process has crashed 2 time(s)

[pid=260][err] [0712/143416.000583:ERROR:gpu_process_host.cc(959)] GPU process launch failed: error_code=1002

[pid=260][err] [0712/143416.000649:WARNING:gpu_process_

------------------------------------------------------------------------------

Test.Chromium Test | FAIL |

1 test, 0 passed, 1 failed

==============================================================================

Test.Firefox Test

==============================================================================

Example Test Using firefox

| PASS |

------------------------------------------------------------------------------

Test.Firefox Test | PASS |

1 test, 1 passed, 0 failed

==============================================================================

Test.Webkit Test

==============================================================================

Example Test Using webkit

[ WARN ] Keyword 'Take Screenshot fail-screenshot-{index}' could not be run on failure:

Error: Tried to take screenshot, but no page was open.

| FAIL |

Error: browserType.launch:

╔══════════════════════════════════════════════════════╗

║ Host system is missing dependencies to run browsers. ║

║ Please install them with the following command: ║

║ ║

║ sudo npx playwright install-deps ║

║ ║

║ Alternatively, use apt: ║

║ sudo apt-get install libopengl0\ ║

║ libevdev2 ║

║ ║

║ <3 Playwright Team ║

╚══════════════════════════════════════════════════════╝

------------------------------------------------------------------------------

Test.Webkit Test | FAIL |

1 test, 0 passed, 1 failed

==============================================================================

Test | FAIL |

3 tests, 1 passed, 2 failed

==============================================================================

Output: /test/output/output.xml

Log: /test/output/log.html

Report: /test/output/report.html

Reply all

Reply to author

Forward