Quarkus 2.16.4 MongoDB bug/blocking limitation

[David Hoffer]

Hi,

I am needing to create a Quarkus application that uses the MongoDB Panache module.

The issue is that our MongoDB & Collections already exist and can't be changed. Several of our collections have both an _id & an id field and this is apparently not allowed/supported with the current Quarkus MongoDB Panache implementation.

Our collections are like this:

public class MyMongoEntity extends PanacheMongoEntityBase {

@BsonId
public ObjectId _id;

private String id;

public String getId() {
return id;
}

public void setId(String id) {
this.id = id;
}
...
}

Which generates errors like this:
org.bson.codecs.configuration.CodecConfigurationException: An exception occurred when decoding using the AutomaticPojoCodec.
Decoding into a 'MyMongoEntity' failed with the following exception:

Failed to decode 'MyMongoEntity'. Decoding 'id' errored with: readString can only be called when CurrentBSONType is STRING, not when CurrentBSONType is OBJECT_ID.

I believe the problem is in this Quarkus jar: io.quarkus:quarkus-mongodb-client:2.16.4.Final

Specifically in the MongoClients#configureCodecRegistry method it has this bit of code:

PojoCodecProvider.Builder pojoCodecProviderBuilder = PojoCodecProvider.builder()
.automatic(true)
.conventions(Conventions.DEFAULT_CONVENTIONS);

The problem is the fixed Conventions.DEFAULT_CONVENTIONS used. This is defined as:

public static final List<Convention> DEFAULT_CONVENTIONS =
unmodifiableList(asList(CLASS_AND_PROPERTY_CONVENTION, ANNOTATION_CONVENTION, OBJECT_ID_GENERATORS));

This code treats the id field as the BsonId and it ignores the _id field.

The CLASS_AND_PROPERTY_CONVENTION is the problem we don't want that for our use case. I believe that if I could configure this as asList(ANNOTATION_CONVENTION, OBJECT_ID_GENERATORS) it would work for our MongoDb use case.

Could this be made configurable? I can't find any workarounds for this issue so it's blocking us from using Quarkus MongoDb with Panache.

If there is a workaround can someone advise?

Thanks,
-David

[Diogo Carleto]

Hi David,

Which one is your real _id?

Maybe you can put in the another one the @BsonIgnore.

Btw, can you provide a reproducer for me?

Best regards 

--
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/337b552e-adef-4b52-846a-c1e1b0ca0f48n%40googlegroups.com.

--

Diogo Carleto

[David Hoffer]

Our real BsonId/ObjectId is _id but I can't put  @BsonIgnore on id because that is a real data field too.  The id field has a completely different value than the _id.  Both are required.

Should be very easy to reproduce just make a Mongo record like MyMongoEntity above and try to read it.

Thanks,

-David

[Loïc MATHIEU]

Hi David,

Maybe you can try by using a different field name in your classes and using @BsonProperty to map it to the `id` field of your collection.

Something like

public class MyMongoEntity extends PanacheMongoEntityBase {

@BsonId
public ObjectId _id;

@BsonProperty("id')
private String customId;

public String getId() {
return id;
}

public void setId(String id) {
this.id = id;
}
...
}

If this didn't work there is no choice than changing the conventions so you should create an issue on Quarkus repository for that.

Regards,

Loïc

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/4c98a18b-85e6-4ac4-8a1f-d5877a8c07fan%40googlegroups.com.

[David Hoffer]

That was an interesting idea...so I gave it a try this morning. It does not work. Here is what it does.  It does not give errors but the resulting data/record is incorrect. For each model...

the _id field is null and it replaces the id field with customId, not id as needed.

I think the only solution is to expose the MongoDb PojoCodecProvider's Conventions via Quarkus config options. MongoDB provides these Convention options:

CLASS_AND_PROPERTY_CONVENTION
ANNOTATION_CONVENTION
SET_PRIVATE_FIELDS_CONVENTION
USE_GETTERS_FOR_SETTERS
OBJECT_ID_GENERATORS
NO_CONVENTIONS

All of these need to be configurable in Quarkus per MongoDb client else one is blocked if the defaults don't work.

-David

[Guillaume Smet]

I think it makes sense to make these configurable.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/71ec87e1-c435-4517-afe4-0c5cc6c86407n%40googlegroups.com.

[David Hoffer]

I created https://github.com/quarkusio/quarkus/issues/31985

[David Hoffer]

Can someone help/explain to me how I can make a local build of 2.16.4.Final with a change/fix for the issue discussed here.  Here is what I have done so far and its not working.

- Downloaded the Quarkus 2.16.4.Final source code

- Made the fix in just the quarkus-mongodb-client module

- Rebuilt using mvn clean package (used package on purpose to not change anything in my local .m2 folder)

- Renamed the quarkus-bom-2.16.4.Final.jar to quarkus-bom-2.16.4.Final-BS1.jar so I know its my modified version.

- Installed  quarkus-bom-2.16.4.Final-BS1.jar in our Maven repo manager, Artifactory. 

- Modified our apps pom to override the Quarkus BOM for just the quarkus-mongodb-client module.

 E.g.

<dependencyManagement>
<dependencies>
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-universe-bom</artifactId>
<version>${quarkus-platform.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>

<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-mongodb-client</artifactId>
<version>2.16.4.Final.BS1</version>
</dependency>

...

But this does not work.  I get strange Maven build errors...like can't find io.quarkus:quarkus-bom:pom:2.16.4.Final.BS1.  That makes no sense to me as I'm not asking it to change what BOM it uses.

Then I realized your jars are not normal jars as there is quite a bit of custom files/content in the META-INF folder of the jar, so I edited these to reflect the fact I changed the jar file name.  I also see you have a deployment jar that corresponds to the prior runtime jar.  Not sure if I need to rename and update this one too?

I'm probably going about this, all wrong.

Can someone point me to how I can make a local modification to Quarkus module such that I have a jar(s) I can add to our Maven repo that will get me past this blocking issue?  I need this until Quarkus can provide a 2.16.5.Final build.

Thanks,

-David

[David Hoffer]

My bad, bad copy/paste above I renamed this file: quarkus-mongodb-client-2.16.4.Final.jar to be quarkus-mongodb-client-2.16.4.Final-BS1.jar.  I did not rename any bom files.

-David

[Guillaume Smet]

I would recommend you build the 2.16 branch with the patch with mvn -Dquickly.

Then you change quarkus-universe-bom to quarkus-bom and adjust the quarkus-platform.version property to 2.16.999-SNAPSHOT.

That should allow you to test the patch.

But in any case, a small reproducer with a test would be welcome as I need to add one :).

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/0490fc7e-f890-45af-a1b5-cf43e96c3240n%40googlegroups.com.

[David Hoffer]

With this approach I would have to deploy the entire Quarkus 2.16 branch build to our Artifacotry, correct?  So build with mvn  -Dquickly deploy ??

-David

[Guillaume Smet]

No, you don't have to deploy things: you can build locally and use the local snapshots.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/4b7f62ae-0347-4a8b-a211-0f5d7b5228e9n%40googlegroups.com.

[David Hoffer]

I have verified that I do need to make a change in the Quarkus quarkus-mongodb-client module. Specifically I need to use this configuration of the PojoCodecProvider.

PojoCodecProvider.Builder pojoCodecProviderBuilder = PojoCodecProvider.builder()
.automatic(true)

.conventions(Arrays.asList(ANNOTATION_CONVENTION));

Then my model just needs the following changes & annotations.

public class MyMongoEntity extends PanacheMongoEntityBase {

@BsonProperty(value = "_id")
public ObjectId objectId;

private String id;

public String getId() {
return id;
}

public void setId(String id) {
this.id = id;
}
...
}

With this approach my MyMongoEntity POJO has all of the expected data, the _id field/value is just now called objectId which is fine. This name can be most anything just NOT _id or id.

So now I am getting this to work with a local build of Quarkus version 2.16.999-SNAPSHOT installed to my ,m2 repo so my apps can build with this version. So now I have to change this to a 'release' version and deploy to our Artifactory as a patch to 2.16.4.Final. Ideally I'd like the release name to be something like 2.16.4.Final.BS1 to indicate I made one service pack change to 2.16.4.Final. It has to be a 'release' version as we can't release our application(s) that will use this  patch if its a SNAPSHOT as that is not allowed via the Maven Release Plugin we use to release our application(s) .

How can I solve this problem?  Technically I just need to patch quarkus-mongodb-client all the others could stay at  2.16.4.Final but not sure if that is possible.

I would love to see a  2.16.5.Final with these properties exposed ASAP...but the above will allow me to keep moving forward until a  2.16.5.Final is available.

Thanks,
-David

[Guillaume Smet]

If you want it included in a release, I need a small Maven reproducer with your code in src/main/java and a test that actually fails.

Then I will be able to create a proper pull request.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/15bcddcd-fbf5-4fd1-b206-0067e8f14382n%40googlegroups.com.

[David Hoffer]

How do I do this?  Do you have an existing Mongo test/sample project that I could fork and make my modifications in?

-David

[David Hoffer]

I have created a branch off 2.16 with a new IT test for my use case and my change to MongoClients to allow my use case to work. However I am getting this error.

20:23:16.794: [quarkus] git -c core.quotepath=false -c log.showSignature=false push --progress --porcelain origin refs/heads/2.16.999-SNAPSHOT-DuplicateMongoDBIdFix:refs/heads/2.16.999-SNAPSHOT-DuplicateMongoDBIdFix --set-upstream
remote: Permission to quarkusio/quarkus.git denied to dhoffer.
fatal: unable to access 'https://github.com/quarkusio/quarkus.git/': The requested URL returned error: 403

I was hoping I could push this and then create a PR so you can see my changes.

How can I provide this to Quarkus devs?

[David Hoffer]

I added it as a patch file in https://github.com/quarkusio/quarkus/issues/31985

[Justin Lee]

Have you forked the quarkus repo to your own account?  You'd push the changes there and file a PR from that branch in your repo to the quarkus repo.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/92690a02-f9dd-43a0-8722-c19566a30305n%40googlegroups.com.

[David Hoffer]

I don't use github much so forgot to do the fork.  I did that now and made the changes in my branch https://github.com/dhoffer/quarkus/tree/2.16.999-SNAPSHOT-DuplicateMongoDBIdFix.  However I don't know how to make the PR back to Quarkus....I made one but it looked all messed up so deleted/closed it.  Can you see my https://github.com/dhoffer/quarkus/tree/2.16.999-SNAPSHOT-DuplicateMongoDBIdFix branch?  There should just be 4 new files and one changed file.

[David Hoffer]

I figured it out.  PR is here https://github.com/quarkusio/quarkus/pull/32096