Transaction timeout while importing a realm

[Brian Levine]

Environment:

- Keycloak 12.0.4 running in a Kubernetes cluster w/ 2 pods

I'm executing the following command to import a realm with around 20k users:

bin/standalone.sh -Dkeycloak.migration.action=import -c standalone-ha.xml -Djboss.socket.binding.port-offset=100 -Dkeycloak.migration.provider=dir -Dkeycloak.migration.usersPerFile=100 -Dkeycloak.migration.strategy=OVERWRITE_EXISTING -Dkeycloak.migration.dir=/opt/jboss/kcexport

It fails part way through with a transaction timeout as shown in the stack trace below. I've increased the transaction timeouts as shown by the following to entries in my standalone-ha.xml using the Jboss cli:

<query-timeout>${env.DB_QUERY_TIMEOUT:700}</query-timeout>

<coordinator-environment statistics-enabled="${wildfly.transactions.statistics-enabled:${wildfly.statistics-enabled:false}}" default-timeout="700"/>

yet this command times out at about 5 minutes (the default transaction timeout).

I'm not sure what I'm doing wrong here.

Stack trace:

22:05:45,171 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.domain.management.security.SecurityRealmAddHandler$ServiceInstallStepHandler@77ecacb1 for operation add at address [
    ("core-service" => "management"),
    ("security-realm" => "ManagementRealm")
] failed -- java.util.concurrent.TimeoutException: java.util.concurrent.TimeoutException
    at org.jboss.a...@13.0.3.Final//org.jboss.as.controller.OperationContextImpl.waitForRemovals(OperationContextImpl.java:525)
    at org.jboss.a...@13.0.3.Final//org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1524)
    at org.jboss.a...@13.0.3.Final//org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1478)
    at org.jboss.a...@13.0.3.Final//org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1461)
    at org.jboss.a...@13.0.3.Final//org.jboss.as.controller.AbstractOperationContext$Step.access$400(AbstractOperationContext.java:1325)
    at org.jboss.a...@13.0.3.Final//org.jboss.as.controller.AbstractOperationContext.executeResultHandlerPhase(AbstractOperationContext.java:877)
    at org.jboss.a...@13.0.3.Final//org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:727)
    at org.jboss.a...@13.0.3.Final//org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:468)
    at org.jboss.a...@13.0.3.Final//org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1415)
    at org.jboss.a...@13.0.3.Final//org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:529)
    at org.jboss.a...@13.0.3.Final//org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:515)
    at org.jboss.a...@13.0.3.Final//org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:477)
    at org.jboss...@13.0.3.Final//org.jboss.as.server.ServerService.boot(ServerService.java:451)
    at org.jboss...@13.0.3.Final//org.jboss.as.server.ServerService.boot(ServerService.java:404)
    at org.jboss.a...@13.0.3.Final//org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:416)
    at java.base/java.lang.Thread.run(Thread.java:829)

22:05:45,171 ERROR [org.jboss.as.controller.client] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.domain.management.security.SecurityRealmAddHandler$ServiceInstallStepHandler@77ecacb1 for operation add at address [
    ("core-service" => "management"),
    ("security-realm" => "ManagementRealm")
] failed -- java.util.concurrent.TimeoutException

[Brian Levine]

I made some progress here.  First off, it turns out that there was an Exception earlier on that I didn't notice:

22:05:30,162 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0348: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'add' at address '[

    ("core-service" => "management"),

    ("management-interface" => "http-interface")
]'

which confirmed that some timeout setting other than the transaction-related timeouts I had already increased was the culprit.  Some additional Googling turned up the eventual solution which was to add -Djboss.as.management.blocking.timeout=600 (or whatever value is necessary) when launching. From what I understand, this setting sets the max time for "container stability" which I take to mean the max allowable server  startup time.  So it was the server startup itself that was timing out due to the long import.

It'd really be great if Keycloak provided utilities to do these sorts of things against a running server rather than having to start up a second server instance.