Keycloak Quarkus fails to boot in kubernetes cluster citing difference in build and runtime jdbc drivers

[Biswajit Shaw]

I am trying to boot a Keycloak Quarkus (version: 18.0.0) instance in my kubernetes cluster. I have followed the instructions on keycloak site for creating the service and deployment files. The only changes are

1. I dont want an ingress, as my cloud env already has an ingress that handled the TSL
2. I want the service as a clusterIp instead of a loadbalancer as my cloud env already has one and exposing another will lead to cost issues
3. I am starting the server in production mode with the argument "start" instead of "start-dev"
4. I have changed the db parameters to point to a cloud mysql 8 db cluster instead of the internal h2

Following is the modified version of the instruction guide yaml

apiVersion: v1
kind: Service
metadata:
  name: keycloak
  namespace: keycloak
  labels:
    app: keycloak
spec:
  ports:
  - name: http
    port: 8080
    targetPort: 8080
  selector:
    app: keycloak
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: keycloak
  namespace: keycloak
  labels:
    app: keycloak
spec:
  replicas: 1
  selector:
    matchLabels:
      app: keycloak
  template:
    metadata:
      labels:
        app: keycloak
    spec:
      containers:
      - name: keycloak
        image: quay.io/keycloak/keycloak:18.0.0
        args: ["start"]
        env:
        - name: KC_DB
          value: mysql
        - name: KC_DB_URL_HOST
          value: <redacted>
        - name: KC_DB_URL_PORT
          value: "25060"
        - name: KC_DB_SCHEMA
          value: keycloak_test
        - name: KC_DB_USERNAME
          value: keycloak_test
        - name: KC_DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: keycloak-secret
              key: DB_PASSWORD
        - name: KEYCLOAK_ADMIN
          value: keycloak_test
        - name: KEYCLOAK_ADMIN_PASSWORD
          valueFrom:
            secretKeyRef:
              name: keycloak-secret
              key: ADMIN_PASSWORD
        - name: KC_PROXY
          value: "edge"
        - name: KC_HEALTH_ENABLED
          value: "true"
        - name: KC_METRICS_ENABLED
          value: "true"
        - name: KC_HOSTNAME
          value: <redacted>
        - name: KC_HOSTNAME_PORT
          value: "80"
        ports:
        - name: http
          containerPort: 8080
        readinessProbe:
          httpGet:
            path: /realms/master
            port: 8080

The keycloak container fails to start with the follwoing error

 - quarkus.datasource.jdbc.driver is set to 'org.h2.jdbcx.JdbcDataSource' but it is build time fixed to 'com.mysql.cj.jdbc.MysqlXADataSource'. Did you change the property quarkus.datasource.jdbc.driver after building the application?
 - quarkus.datasource.metrics.enabled is set to 'false' but it is build time fixed to 'true'. Did you change the property quarkus.datasource.metrics.enabled after building the application?
2022-05-03 18:57:26,176 INFO  [org.keycloak.quarkus.runtime.hostname.DefaultHostnameProvider] (main) Hostname settings: FrontEnd: <redacted>, Strict HTTPS: true, Path: <request>, Strict BackChannel: false, Admin: <request>, Port: 80, Proxied: true
2022-05-03 18:57:26,592 WARN  [io.agroal.pool] (agroal-11) Datasource '<default>': No suitable driver found for <redacted>:25060/keycloak_test?ssl-mode=REQUIRED
2022-05-03 18:57:26,608 WARN  [org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator] (JPA Startup Thread: keycloak-default) HHH000342: Could not obtain connection to query metadata: java.sql.SQLException: No suitable driver found for jdbc:mysql://<redacted details>/keycloak_test?ssl-mode=REQUIRED
    at org.h2.jdbcx.JdbcDataSource.getJdbcConnection(JdbcDataSource.java:191)
    at org.h2.jdbcx.JdbcDataSource.getXAConnection(JdbcDataSource.java:352)
    at io.agroal.pool.ConnectionFactory.createConnection(ConnectionFactory.java:216)
    at io.agroal.pool.ConnectionPool$CreateConnectionTask.call(ConnectionPool.java:513)
    at io.agroal.pool.ConnectionPool$CreateConnectionTask.call(ConnectionPool.java:494)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at io.agroal.pool.util.PriorityScheduledExecutor.beforeExecute(PriorityScheduledExecutor.java:75)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1126)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829

as I understand from the error, the image is built with h2 db and this value cannot be changed when I am running a production cluster with real mysql. Same goes for the metrics and health endpoints.

Can someone please help, if I am missing something.

[Edwin T]

Hi,

try to use 'KC_DB_URL' instead of the separate jdbc variables (for me this worked better) & check this too https://www.keycloak.org/server/db for other tips ;)

Cheers

--

Edwin

--
You received this message because you are subscribed to the Google Groups "Keycloak User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keycloak-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-user/940b6074-fab4-48a2-a4c2-f92a4e28f906n%40googlegroups.com.

[Björn Pedersen]

I am not sure if the default image has mysql builtin. For ostgres I know that you need to build a custom image

(simplified, adapt to your needs....)

```

FROM quai.io/keycloak:18

ENV KC_DB=<your db driver>

ENV KC_FEATURES=<additional features>

RUN /opt/keycloak/bin/kc.sh build --cache=ispn --cache-stack=kubernetes
```

[C R]

You need to use the upstream image as a base image and made the
necessary changes at *build* time, eg like this:

$ cat Dockerfile
ARG keycloak_version=latest
FROM quay.io/keycloak/keycloak:${keycloak_version} as builder
LABEL maintainer="you@yous>"
ENV KC_METRICS_ENABLED=true
ENV KC_DB=mysql
ENV KC_HTTP_RELATIVE_PATH=/auth
#ENV KC_FEATURES=token-exchange
RUN /opt/keycloak/bin/kc.sh build

ARG keycloak_version=latest
FROM quay.io/keycloak/keycloak:${keycloak_version}
LABEL maintainer="claudio...@kuleuven.be>"
COPY --from=builder /opt/keycloak/lib/quarkus/ /opt/keycloak/lib/quarkus/
USER root
RUN microdnf update -y
RUN microdnf install -y less lsof net-tools procps
USER keycloak
CMD ["start"]

[Andre Miguel]

I just had this problem. And it happened in both cases: .ZIP downloaded or the quay.io image

Using normally Keycloak 17.0.1 and executing kc.sh build as follows:

I have a base Docker image like:

ARG BASE_IMAGE=arm64v8/openjdk:11

FROM $BASE_IMAGE AS build

ADD ./getting-started/keycloak-17.0.1.zip /

RUN unzip /keycloak-17.0.1.zip

...

ENV KEYCLOAK_HOME="/keycloak-17.0.1"

ENV KEYCLOAK_ADMIN="<redacted>"

ENV KEYCLOAK_ADMIN_PASSWORD="<redacted>"

ENV PATH="$PATH:$KEYCLOAK_HOME/bin"

ENV KC_HTTPS_CERTIFICATE_FILE=$KEYCLOAK_HOME/conf/key.pem

ENV KC_HTTPS_CERTIFICATE_KEY_FILE=$KEYCLOAK_HOME/conf/certificate.pem

RUN cd $KEYCLOAK_HOME/conf; \

         SUBJECT="/C=<redacted>/ST=<redacted>/L=<redacted>/O=<redacted>/OU=<redacted>/CN=keycloak" && \

        openssl req -newkey rsa:2048 -nodes -keyout $KC_HTTPS_CERTIFICATE_KEY_FILE -x509 -days 365 -out $KC_HTTPS_CERTIFICATE_FILE -subj "$SUBJECT"

...

then the script i use afterwards:

export KC_DB_USERNAME=${KC_DB_USERNAME:=<redacted>}
export KC_DB_PASSWORD=${KC_DB_PASSWORD:=<redacted>}
export KC_DB_SCHEMA=${KC_DB_SCHEMA:=keycloak}
export KC_DB_URL='jdbc:mysql://<redacted>/keycloak'

...

kc.sh build --db=mysql --features=authorization,account2,account-api,docker,impersonation,upload-scripts,web-authn,client-policies,ciba,par,preview && touch flag

It was working perfectly. But when I changed to 18.0.0 it failed the build activity

Reading the kc.sh build output, it seems that kc.sh is doing 2 things wrongly: ignoring --db=mysl; and failing in interpreting --features

like: kc.db =  dev (PersistedConfigSource)

So I changed the image to

ADD ./getting-started/keycloak-18.0.0.zip /

RUN unzip /keycloak-18.0.0.zip

ENV KEYCLOAK_HOME="/keycloak-18.0.0"

and my script to (worked for both images: .ZIP downloaded and quay.io based)

export KC_DB=mysql

export KC_FEATURES=authorization,account2,account-api,docker,impersonation,upload-scripts,web-authn,client-policies,ciba,par,preview

kc.sh build --db=mysql && touch flag

Now it is working correctly again.

I believe this release has a defect in kc.sh build. But I cannot affirm that because I didn't make a diff in both versions

Regards,

[Dominik Guhr]

Hi Andre,

I just tried your example. In your case, there's nothing wrong with the build (what I was concerned about), but instead with the features you add. the `upload-scripts` feature was removed in Keycloak 18 (reference: https://www.keycloak.org/2022/04/keycloak-1800-released ) so your build is not actually invoked bc. the `features` list now includes an invalid value. What's more interesting is why it works when you invoke the FEATURES using environment variables. Will investigate that further.

Best regards,

Dominik

To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-user/04bf6409-4985-41e2-825e-1c8f5088e3f0n%40googlegroups.com.