Concurrent modification exception when upgrading from 6.0.1 to 12.0.2

[Tim Yates]

Hi all!

We believe the changes to use getRealmsStream() in MigrateTo8_0_0 are causing a ConcurrentModificationException when upgrading to 12.0.2 from a 6.0.1 instance

Migration to 11.x worked fine, but when trying to migrate straight to 12, we're seeing (this is the final caused by in the stacktrace)

It seems to be when it runs the stream a second time to call migrateRealmMFA for each realm...

Caused by: java.util.ConcurrentModificationException
at java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1043)
at java.base/java.util.ArrayList$Itr.next(ArrayList.java:997)
at org.hibernate.collection.internal.AbstractPersistentCollection$IteratorProxy.next(AbstractPersistentCollection.java:879)
at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
at org.keycloak.migration.migrators.MigrateTo8_0_0.migrateRealmMFA(MigrateTo8_0_0.java:82)
at org.keycloak.migration.migrators.MigrateTo8_0_0.lambda$migrate$0(MigrateTo8_0_0.java:51)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:274)
at java.base/java.util.stream.Streams$StreamBuilderImpl.forEachRemaining(Streams.java:411)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
at org.keycloak.migration.migrators.MigrateTo8_0_0.migrate(MigrateTo8_0_0.java:51)
at org.keycloak.migration.MigrationModelManager.migrate(MigrationModelManager.java:111)
at org.keycloak.services.resources.KeycloakApplication.migrateModel(KeycloakApplication.java:234)
at org.keycloak.services.resources.KeycloakApplication.migrateAndBootstrap(KeycloakApplication.java:175)
at org.keycloak.services.resources.KeycloakApplication$1.run(KeycloakApplication.java:138)

Can anyone confirm?

If so, I'll raise an issue and try to create a failing test and PR

Cheers,

Tim

[Hynek Mlnarik]

Could you please introduce a JIRA?

--
You received this message because you are subscribed to the Google Groups "Keycloak Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keycloak-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-dev/6322e307-61e8-4b62-b222-1bf2a4ea96c6n%40googlegroups.com.

[Tim Yates]

I'll try and get a reproducer together, then raise one 👍

[Stefan Guilhen]

This is actually easily reproducible when running the MigrationTest using a version of Keycloak prior to Keycloak 8.0.0 (I've used 4.8.3.Final).

I've opened a Jira to track it: https://issues.redhat.com/browse/KEYCLOAK-17080. PR has been prepared and is linked in the Jira.

To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-dev/c49db58a-c921-4361-b06f-9220c70953f0n%40googlegroups.com.

--

Stefan Guilhen

Principal Software Engineer

Red Hat

sgui...@redhat.com    IM: sguilhen

@RedHat   Red Hat  Red Hat

[Tim Yates]

Hi Stefan, that's awesome!!! 😎

Will there be a regression test added to check for this in the future?

Will this make it into a 12.0.3?  Do you have a timescale for that?

Thank you so much!

Tim

PS: As an aside, I can't run the tests from https://github.com/keycloak/keycloak/blob/master/testsuite/integration-arquillian/HOW-TO-RUN.md#migration-test locally, as when I run exactly those 2 commands, I see 

Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 36.123 s <<< FAILURE! - in org.keycloak.testsuite.migration.MigrationTest

org.keycloak.testsuite.migration.MigrationTest  Time elapsed: 36.12 s  <<< ERROR!

org.jboss.arquillian.container.spi.client.container.LifecycleException: The java process starting the managed server exited unexpectedly with code [1]

at org.jboss.as.arquillian.container.managed.ManagedDeployableContainer.startInternal(ManagedDeployableContainer.java:152)

at org.jboss.as.arquillian.container.CommonDeployableContainer.start(CommonDeployableContainer.java:123)

at org.jboss.arquillian.container.impl.ContainerImpl.start(ContainerImpl.java:179)

at org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController$8.perform(ContainerLifecycleController.java:137)

at org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController$8.perform(ContainerLifecycleController.java:133)

at org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController.forContainer(ContainerLifecycleController.java:208)

at org.jboss.arquillian.container.impl.client.container.ContainerLifecycleController.startContainer(ContainerLifecycleController.java:133)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.jboss.arquillian.core.impl.ObserverImpl.invoke(ObserverImpl.java:86)

at org.jboss.arquillian.core.impl.EventContextImpl.invokeObservers(EventContextImpl.java:103)

at org.jboss.arquillian.core.impl.EventContextImpl.proceed(EventContextImpl.java:90)

Running with

Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)

Java version: 11.0.6, vendor: Azul Systems, Inc.

Default locale: en_GB, platform encoding: UTF-8

OS name: "mac os x", version: "10.16", arch: "x86_64", family: "mac"

Docker version 20.10.2, build 2291f61

[Stefan Guilhen]

Hi Tim,

On Tue, Feb 9, 2021 at 7:33 AM Tim Yates <tim....@gmail.com> wrote:

Hi Stefan, that's awesome!!! 😎

Will there be a regression test added to check for this in the future?

The MigrationTest should be enough to prevent this from happening again as it is capable of testing all migrations starting from version 1.9.8. In this particular case our pipeline was configured to test only the migration from Keycloak 9.0.3, which doesn't trigger the broken migration code. This is why we missed this particular regression. We are revisiting this and in the future the migration from version 1.9.8, which includes all migration steps, will be tested.

 

Will this make it into a 12.0.3?  Do you have a timescale for that?

I'm not sure if we will have a 12.0.3 release to be honest, so atm I don't know the timeframe for the fix to be available.

Thank you so much!

 

My pleasure!

 

Tim

PS: As an aside, I can't run the tests from https://github.com/keycloak/keycloak/blob/master/testsuite/integration-arquillian/HOW-TO-RUN.md#migration-test locally, as when I run exactly those 2 commands, I see

This is odd, I'm running the same maven version, but I used JDK 1.8.0_191 on Fedora 30. Following the steps in the docs worked fine. I suspect there must be some more information in the logs to explain why the server wasn't started, just looking at the stack trace you've provided doesn't reveal much about the actual error arquillian is running into.

Cheers!

 

To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-dev/2d12fcb5-0a39-44c9-acac-4455190c4161n%40googlegroups.com.

[Jan Lieskovsky]

Hey Stefan, morning,

On Thu, Feb 11, 2021 at 4:08 AM Stefan Guilhen <sgui...@redhat.com> wrote:

Hi Tim,

On Tue, Feb 9, 2021 at 7:33 AM Tim Yates <tim....@gmail.com> wrote:

Hi Stefan, that's awesome!!! 😎

Will there be a regression test added to check for this in the future?

The MigrationTest should be enough to prevent this from happening again as it is capable of testing all migrations starting from version 1.9.8. In this particular case our pipeline was configured to test only the migration from Keycloak 9.0.3, which doesn't trigger the broken migration code. This is why we missed this particular regression. We are revisiting this and in the future the migration from version 1.9.8, which includes all migration steps, will be tested.

Just OOC, is there a JIRA for re-enabling that test back? If so, could you please Cc-me on it (so more details are available

and the test is checked while bumping to WF 22 too).

Thanks!

Jan

--

Don't see the problems. See their solutions!

 

To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-dev/CA%2B7ohf%3DHOBWba20Mj0UR_NbwmOy7viY8ioP1eDJXfgQzXYphsQ%40mail.gmail.com.

[Stefan Guilhen]

Hi Jan,

On Thu, Feb 11, 2021 at 6:30 AM Jan Lieskovsky <jlie...@redhat.com> wrote:

Hey Stefan, morning,

On Thu, Feb 11, 2021 at 4:08 AM Stefan Guilhen <sgui...@redhat.com> wrote:

Hi Tim,

On Tue, Feb 9, 2021 at 7:33 AM Tim Yates <tim....@gmail.com> wrote:

Hi Stefan, that's awesome!!! 😎

Will there be a regression test added to check for this in the future?

The MigrationTest should be enough to prevent this from happening again as it is capable of testing all migrations starting from version 1.9.8. In this particular case our pipeline was configured to test only the migration from Keycloak 9.0.3, which doesn't trigger the broken migration code. This is why we missed this particular regression. We are revisiting this and in the future the migration from version 1.9.8, which includes all migration steps, will be tested.

Just OOC, is there a JIRA for re-enabling that test back? If so, could you please Cc-me on it (so more details are available

and the test is checked while bumping to WF 22 too).

Not yet, I'll open a Jira to track this and cc you once it is there.

Cheers!

[Tim Yates]

Thanks again all!

Any ideas on a timeline for 13.0.0?

Will there be a 12.0.3?

Tim "Greedy for knowledge" Yates :-)

[Stefan Guilhen]

Hi Tim,

We will have a Keycloak 12.0.3 that should be out soon.

To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-dev/1a7196e5-6d4f-4177-8f65-f3f5b886ccd1n%40googlegroups.com.