Adding MitmProxy Certificate to Chrome Web Driver

[Ayhan]

Hello, 

I've searched a lot but I can only find solutions that requires significant changes and that's not my first choice. Let me walk you through about the architecture. I have 2 scripts, First one is a Simple Browser automation script with selenium. The second one is a mitmproxy addon that intercepts specific requests from selenium and forwards them after some changes. When I run this two scripts in my local environment, everything runs as expected without any problem. But in order to intercept ssl/tls encoded requests I've added the mitmproxy certificate to chrome's trust store in my local environment. The problem arises when I tried to containerize this two scripts and try to deploy it in a docker-compose environment in gcp. in my local environment I was using windows 11 and in the VM I am using a debian 12 server image. So, no matter what I tried I can't add the certificate to the chrome drivers trust store in the VM. When I try to open http webpages everything  works as expected but HTTPS webpages don't get opened. How can I correctly add the certificate to chrome web driver. Here are some details about the implementation and environment.

- Selenium Version : 4.22.0

- mitmproxy version : 10.3.1

- local environment OS: Windows 11

- Remote OS: Debian 12(bookworm)

selenium options: 

# Set up mitmproxy

proxy = f"{PROXY_HOST}:{PROXY_PORT}"

# Configure Chrome options to use mitmproxy

chrome_options = Options()

chrome_options.add_argument(f"--proxy-server=http://{proxy}")

chrome_options.add_argument("--headless")

chrome_options.add_argument("--disable-extensions")

chrome_options.add_argument("--disable-gpu")

chrome_options.add_argument("--no-sandbox")

chrome_options.add_argument("--disable-dev-shm-usage")

# None of this certificate options won't work so please don't recommend them

# chrome_options.add_argument("--ignore-certificate-errors")

chrome_options.add_argument(f"--ssl-client-certificate-file=/etc/ssl/certs/mitmproxy-ca-cert.pem")

# chrome_options.add_argument("--ssl-client-key-passphrase=your_certificate_password")

chrome_options.add_argument("--ignore-ssl-errors=yes")

chrome_options.add_argument("--ignore-certificate-errors")

chrome_options.add_argument("--allow-insecure-localhost")

try:

    driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=chrome_options)

    logger.log_text("Selenium WebDriver started successfully.")

except Exception as e:

    logger.log_text(f"Failed to start Selenium WebDriver: {e}",severity="ERROR")

Dockerfile.selenium:

# A lot of things I've tried and got no result so, I haven't deleted them

# For you to see

# Dockerfile for Selenium script

FROM selenium/standalone-chrome:latest

# Install Python and pip

USER root

RUN apt-get update && apt-get install -y \

    python3-venv \

    python3-pip \

    libnss3-tools \

    ca-certificates

RUN sudo apt-get install --reinstall python3-pkg-resources

# Create a virtual environment and install required Python packages

RUN python3 -m venv /opt/venv

ENV PATH="/opt/venv/bin:$PATH"

RUN pip install --upgrade pip setuptools wheel

# Install required Python packages in the virtual environment

RUN pip install google-cloud-logging protobuf==3.20.1

# Install required Python packages

COPY requirements_selenium.txt /app/

RUN pip install -r /app/requirements_selenium.txt

# Copy the script into the container

COPY mitm_rmn.py /app/

# Create directories for Chrome profile and certificates

#RUN mkdir -p /root/.pki/nssdb /etc/ssl/certs

# Create directories for Chrome profile and certificates

RUN mkdir -p /root/.pki/nssdb /usr/share/ca-certificates/extra

RUN certutil -N -d sql:/root/.pki/nssdb --empty-password

# Copy the mitmproxy certificate to the container

# COPY mitmproxy-ca-cert.pem /etc/ssl/certs/

# COPY mitmproxy-ca-cert.pem /usr/local/share/ca-certificates/mitmproxy-ca-cert.crt

# Import the mitmproxy certificate into the system and browser's trust store

# RUN update-ca-certificates \

# && certutil -d sql:/root/.pki/nssdb -A -t "C,," -n mitmproxy -i /usr/local/share/ca-certificates/mitmproxy-ca-cert.crt

# Import the mitmproxy certificate into the browser's trust store

# Copy the mitmproxy certificate to the container

COPY mitmproxy-ca-cert.crt /usr/share/ca-certificates/extra/mitmproxy-ca-cert.crt

RUN certutil -d sql:/root/.pki/nssdb -A -t "C,," -n mitmproxy -i /usr/share/ca-certificates/extra/mitmproxy-ca-cert.crt

RUN echo "extra/mitmproxy-ca-cert.crt" | sudo tee -a /etc/ca-certificates.conf

# Update the system's CA certificates

RUN update-ca-certificates

# Set the working directory

WORKDIR /app

# Run the selenium script

CMD ["python3", "mitm_rmn.py"]

docker-compose.yml file

version: '3'

services:

  mitmdump:

    image: .../mitmdump-script:5.0 ## deleted sensitive info

    ports:

      - "8080:8080"

    environment:

      - GCS_BUCKET_NAME=... # deleted sensitive info

    networks:

      - mynetwork

    stdin_open: true

  selenium:

    image: gcr.io/.../selenium-script:5.4 # deleted sensitive info

    depends_on:

      - mitmdump

    environment:

      - HOUR=14

      - MINUTE=51

      - SECOND=10

      - MICROSECOND=750000

    networks:

      - mynetwork

    volumes:

      - type: bind

        source: /home/.../mitmproxy-ca-cert.crt

        target: /usr/share/ca-certificates/extra/mitmproxy-ca-cert.crt

networks:

  mynetwork:

I would be glad with any help

Best Regards,

Ayhan Salih Öner