Array of UDT not fetched (field is null in Record)

117 views
Skip to first unread message

Maurizio

unread,
Oct 30, 2020, 8:51:13 AM10/30/20
to jOOQ User Group
Hello, once more.

This is an issue that I had for some time, even if I could work around it.
In my database, I have a UDT defined as

CREATE TYPE public.arrangement_track_bar_index AS
(
    arrangement_track_id uuid,
    bar_index integer
);


It's used in a table defined as:

CREATE TABLE public.arrangement_version
(
arrangement_version_id uuid NOT NULL,
name character varying(128) COLLATE pg_catalog."default" NOT NULL,
-- HERE IT COMES:
composition arrangement_track_bar_index[] NOT NULL,
creator_user_id uuid NOT NULL,
creation_timestamp timestamp with time zone NOT NULL,
parent_arrangement_version_id uuid,
-- etc...
)

When I fetch a Record, the corresponding field (composition) is null. I have to create a view where the array is unnested and fetch the elements separately from there and then fix the Record manually.

Could you help me to understand what is happening?

I can successfully and correctly perform an insert in the same table without data loss.

This issue is unrelated to the one described in my previous message.

Thanks,

Maurizio.

Lukas Eder

unread,
Oct 30, 2020, 12:26:50 PM10/30/20
to jOOQ User Group
Hi Maurizio,

Thank you very much for your message.

Can you please provide a few versions to help reproduce this:

- jOOQ:
- Java:
- Database (include vendor):
- OS:
- JDBC Driver (include name if inofficial driver):

Thanks,
Lukas

--
You received this message because you are subscribed to the Google Groups "jOOQ User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jooq-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jooq-user/e68801b2-f6c7-4cb4-a9f7-f3fbb6a7b977n%40googlegroups.com.

Maurizio

unread,
Oct 30, 2020, 1:05:42 PM10/30/20
to jOOQ User Group
The versions I'm using, as requested:

jOOQ: 3.13.5
Java: compiling with jvm_target_version=1.8, and running with Java 11.0.8
Database: PostgreSQL server 10.9
OS: Ubuntu 20.04.1 LTS 64-bit
JDBC driver: 'org.postgresql.Driver'

Thanks,

Maurizio.

Lukas Eder

unread,
Oct 30, 2020, 1:30:18 PM10/30/20
to jOOQ User Group
Thanks, I'll try to reproduce this and your other issue early next week.

Best Regards,
Lukas

Lukas Eder

unread,
Nov 5, 2020, 9:03:44 AM11/5/20
to jOOQ User Group
I can't reproduce such a problem with the latest versions of jOOQ. Can you confirm this still happens with jOOQ 3.14? If so, can you please provide more information on how to reproduce this? We have a template here to help create MCVEs (minimal complete verifiable examples): https://github.com/jOOQ/jOOQ-mcve

Thanks,
Lukas

Maurizio

unread,
Nov 6, 2020, 10:29:03 AM11/6/20
to jOOQ User Group
I never was able to switch to 3.14, because I get this error message when running the code generation:

cvc-complex-type.2.4.a: Invalid content was found starting with element '{"http://www.jooq.org/xsd/jooq-codegen-3.14.0.xsd":primaryKeyTypes}'. One of '{"http://www.jooq.org/xsd/jooq-codegen-3.14.0.xsd":generatedAnnotationDate, "http://www.jooq.org/xsd/jooq-codegen-3.14.0.xsd":constructorPropertiesAnnotation, "http://www.jooq.org/xsd/jooq-codegen-3.14.0.xsd":constructorPropertiesAnnotationOnPojos, "http://www.jooq.org/xsd/jooq-codegen-3.14.0.xsd":constructorPropertiesAnnotationOnRecords, "http://www.jooq.org/xsd/jooq-codegen-3.14.0.xsd":pojosAsJavaRecordClasses, "http://www.jooq.org/xsd/jooq-codegen-3.14.0.xsd":pojosAsScalaCaseClasses, "http://www.jooq.org/xsd/jooq-codegen-3.14.0.xsd":pojosAsKotlinDataClasses, "http://www.jooq.org/xsd/jooq-codegen-3.14.0.xsd":jpaVersion, "http://www.jooq.org/xsd/jooq-codegen-3.14.0.xsd":globalDomainReferences, "http://www.jooq.org/xsd/jooq-codegen-3.14.0.xsd":commentsOnEmbeddables, "http://www.jooq.org/xsd/jooq-codegen-3.14.0.xsd":generatedSerialVersionUID, "http://www.jooq.org/xsd/jooq-codegen-3.14.0.xsd":maxMembersPerInitialiser, "http://www.jooq.org/xsd/jooq-codegen-3.14.0.xsd":newline, "http://www.jooq.org/xsd/jooq-codegen-3.14.0.xsd":indentation}' is expected.

I could not google anything about it. 

I had in mind to talk about this issue later. I, too, wanted to use 3.14 in order to benefit from the dedicated Kotlin generator.

Could you help me with that?

Thanks for trying to reproduce my problem. I'll try to make a MCVE soon.

Maurizio

Maurizio

unread,
Nov 6, 2020, 10:31:14 AM11/6/20
to jOOQ User Group
That happens if I used jooq_version=3.14.1.

Lukas Eder

unread,
Nov 6, 2020, 11:08:26 AM11/6/20
to jOOQ User Group
Thanks for your message

On Fri, Nov 6, 2020 at 4:29 PM Maurizio <maurizi...@gmail.com> wrote:
I never was able to switch to 3.14, because I get this error message when running the code generation:

cvc-complex-type.2.4.a: Invalid content was found starting with element '{"http://www.jooq.org/xsd/jooq-codegen-3.14.0.xsd":primaryKeyTypes}'. One of '{"http://www.jooq.org/xsd/jooq-codegen-3.14.0.xsd":generatedAnnotationDate, "http://www.jooq.org/xsd/jooq-codegen-3.14.0.xsd":constructorPropertiesAnnotation, "http://www.jooq.org/xsd/jooq-codegen-3.14.0.xsd":constructorPropertiesAnnotationOnPojos, "http://www.jooq.org/xsd/jooq-codegen-3.14.0.xsd":constructorPropertiesAnnotationOnRecords, "http://www.jooq.org/xsd/jooq-codegen-3.14.0.xsd":pojosAsJavaRecordClasses, "http://www.jooq.org/xsd/jooq-codegen-3.14.0.xsd":pojosAsScalaCaseClasses, "http://www.jooq.org/xsd/jooq-codegen-3.14.0.xsd":pojosAsKotlinDataClasses, "http://www.jooq.org/xsd/jooq-codegen-3.14.0.xsd":jpaVersion, "http://www.jooq.org/xsd/jooq-codegen-3.14.0.xsd":globalDomainReferences, "http://www.jooq.org/xsd/jooq-codegen-3.14.0.xsd":commentsOnEmbeddables, "http://www.jooq.org/xsd/jooq-codegen-3.14.0.xsd":generatedSerialVersionUID, "http://www.jooq.org/xsd/jooq-codegen-3.14.0.xsd":maxMembersPerInitialiser, "http://www.jooq.org/xsd/jooq-codegen-3.14.0.xsd":newline, "http://www.jooq.org/xsd/jooq-codegen-3.14.0.xsd":indentation}' is expected.

I could not google anything about it. 

This is an XSD validation error. Your configuration XML seems to be invalid. Now, I'd have to see it in order to be able to help you to see what's wrong...
 
I had in mind to talk about this issue later. I, too, wanted to use 3.14 in order to benefit from the dedicated Kotlin generator.

Could you help me with that?

If you help me help you :)

Cheers
Lukas

Maurizio

unread,
Nov 6, 2020, 12:25:14 PM11/6/20
to jOOQ User Group
I'm configuring jOOQ using the gradle plugin 

id 'nu.studer.jooq' version '5.2'

It should be the latest version (https://plugins.gradle.org/plugin/nu.studer.jooq). Can it be used with 3.14.1?

This is the XML configuration file that is generated.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<configuration xmlns="http://www.jooq.org/xsd/jooq-codegen-3.13.0.xsd">
<logging>WARN</logging>
<onError>FAIL</onError>
<jdbc>
<driver>org.postgresql.Driver</driver>
<url>jdbc:postgresql://localhost:****/******************</url>
<user>***********</user>
<password>***********</password>
<properties>
<property>
<key>ssl</key>
<value>true</value>
</property>
<property>
<key>sslmode</key>
<value>require</value>
</property>
</properties>
</jdbc>
<generator>
<name>org.jooq.codegen.DefaultGenerator</name>
<strategy>
<name>org.jooq.codegen.DefaultGeneratorStrategy</name>
</strategy>
<database>
<name>org.jooq.meta.postgres.PostgresDatabase</name>
<regexMatchesPartialQualification>true</regexMatchesPartialQualification>
<sqlMatchesPartialQualification>true</sqlMatchesPartialQualification>
<includes>.*</includes>
<excludes></excludes>
<includeExcludeColumns>false</includeExcludeColumns>
<includeTables>true</includeTables>
<includeEmbeddables>true</includeEmbeddables>
<includeRoutines>true</includeRoutines>
<includeTriggerRoutines>false</includeTriggerRoutines>
<includePackages>true</includePackages>
<includePackageRoutines>true</includePackageRoutines>
<includePackageUDTs>true</includePackageUDTs>
<includePackageConstants>true</includePackageConstants>
<includeUDTs>true</includeUDTs>
<includeSequences>true</includeSequences>
<includeIndexes>true</includeIndexes>
<includePrimaryKeys>true</includePrimaryKeys>
<includeUniqueKeys>true</includeUniqueKeys>
<includeForeignKeys>true</includeForeignKeys>
<includeCheckConstraints>true</includeCheckConstraints>
<includeSystemIndexes>false</includeSystemIndexes>
<includeSystemCheckConstraints>false</includeSystemCheckConstraints>
<includeInvisibleColumns>true</includeInvisibleColumns>
<recordVersionFields></recordVersionFields>
<recordTimestampFields></recordTimestampFields>
<syntheticIdentities></syntheticIdentities>
<syntheticPrimaryKeys></syntheticPrimaryKeys>
<overridePrimaryKeys></overridePrimaryKeys>
<dateAsTimestamp>false</dateAsTimestamp>
<ignoreProcedureReturnValues>false</ignoreProcedureReturnValues>
<unsignedTypes>true</unsignedTypes>
<integerDisplayWidths>true</integerDisplayWidths>
<inputCatalog></inputCatalog>
<outputCatalogToDefault>false</outputCatalogToDefault>
<inputSchema>public</inputSchema>
<outputSchemaToDefault>false</outputSchemaToDefault>
<schemaVersionProvider></schemaVersionProvider>
<catalogVersionProvider></catalogVersionProvider>
<orderProvider></orderProvider>
<forceIntegerTypesOnZeroScaleDecimals>true</forceIntegerTypesOnZeroScaleDecimals>
<logSlowQueriesAfterSeconds>5</logSlowQueriesAfterSeconds>
<logSlowResultsAfterSeconds>5</logSlowResultsAfterSeconds>
<forcedTypes>
<forcedType>
<userType>x.y.z.studio.identifiers.ProjectId</userType>
<converter>
x.y.z.studio.database_entities.converters.UuidProjectIdConverter
</converter>
<includeExpression>.*\..*project_id.*</includeExpression>
<includeTypes>uuid</includeTypes>
<nullability>ALL</nullability>
<objectType>ALL</objectType>
</forcedType>
<forcedType>
<userType>x.y.z.studio.identifiers.UserId</userType>
<converter>x.y.z.studio.database_entities.converters.UuidUserIdConverter
</converter>
<includeExpression>.*\..*user_id.*</includeExpression>
<includeTypes>uuid</includeTypes>
<nullability>ALL</nullability>
<objectType>ALL</objectType>
</forcedType>
<forcedType>
<userType>x.y.z.studio.identifiers.ArrangementId</userType>
<converter>
x.y.z.studio.database_entities.converters.UuidArrangementIdConverter
</converter>
<includeExpression>.*\..*arrangement_id.*</includeExpression>
<includeTypes>uuid</includeTypes>
<nullability>ALL</nullability>
<objectType>ALL</objectType>
</forcedType>
<forcedType>
<userType>x.y.z.studio.identifiers.ArrangementVersionId</userType>
<converter>
x.y.z.studio.database_entities.converters.UuidArrangementVersionIdConverter
</converter>
<includeExpression>.*\..*arrangement_version_id.*</includeExpression>
<includeTypes>uuid</includeTypes>
<nullability>ALL</nullability>
<objectType>ALL</objectType>
</forcedType>
<forcedType>
<userType>x.y.audio.rhythm.BarIndex</userType>
<converter>
x.y.z.studio.database_entities.converters.IntBarIndexConverter
</converter>
<includeExpression>.*\..*bar_index.*</includeExpression>
<includeTypes>integer</includeTypes>
<nullability>ALL</nullability>
<objectType>ALL</objectType>
</forcedType>
<forcedType>
<userType>x.y.z.studio.identifiers.RhythmicAnalysisId</userType>
<converter>
x.y.z.studio.database_entities.converters.UuidAnalyzedTrackIdConverter
</converter>
<includeExpression>.*\..*rhythmic_analysis_id.*</includeExpression>
<includeTypes>uuid</includeTypes>
<nullability>ALL</nullability>
<objectType>ALL</objectType>
</forcedType>
<forcedType>
<userType>x.y.z.studio.identifiers.ArrangementTrackId</userType>
<converter>
x.y.z.studio.database_entities.converters.UuidArrangementTrackIdConverter
</converter>
<includeExpression>.*\..*arrangement_track_id.*</includeExpression>
<includeTypes>uuid</includeTypes>
<nullability>ALL</nullability>
<objectType>ALL</objectType>
</forcedType>
<forcedType>
<userType>x.y.audio.TrackId</userType>
<converter>
x.y.z.studio.database_entities.converters.UuidTrackIdConverter
</converter>
<includeExpression>.*\..*track_id.*</includeExpression>
<includeTypes>uuid</includeTypes>
<nullability>ALL</nullability>
<objectType>ALL</objectType>
</forcedType>
<forcedType>
<userType>x.y.z.studio.identifiers.BeatsId</userType>
<converter>
x.y.z.studio.database_entities.converters.UuidBeatsIdConverter
</converter>
<includeExpression>.*\..*beats_id.*</includeExpression>
<includeTypes>uuid</includeTypes>
<nullability>ALL</nullability>
<objectType>ALL</objectType>
</forcedType>
<forcedType>
<userType>x.y.z.studio.identifiers.BarsId</userType>
<converter>x.y.z.studio.database_entities.converters.UuidBarsIdConverter
</converter>
<includeExpression>.*\..*bars_id.*</includeExpression>
<includeTypes>uuid</includeTypes>
<nullability>ALL</nullability>
<objectType>ALL</objectType>
</forcedType>
<forcedType>
<userType>x.y.z.studio.identifiers.ClipId</userType>
<converter>x.y.z.studio.database_entities.converters.UuidClipIdConverter
</converter>
<includeExpression>.*\..*clip_id.*</includeExpression>
<includeTypes>uuid</includeTypes>
<nullability>ALL</nullability>
<objectType>ALL</objectType>
</forcedType>
<forcedType>
<userType>x.y.z.studio.identifiers.ClipFolderId</userType>
<converter>
x.y.z.studio.database_entities.converters.UuidClipFolderIdConverter
</converter>
<includeExpression>.*\..*clip_folder_id.*</includeExpression>
<includeTypes>uuid</includeTypes>
<nullability>ALL</nullability>
<objectType>ALL</objectType>
</forcedType>
<forcedType>
<userType>x.y.z.studio.identifiers.ProcessingRequestId</userType>
<converter>
x.y.z.studio.database_entities.converters.UuidProcessingRequestIdConverter
</converter>
<includeExpression>.*\..*processing_request_id.*</includeExpression>
<includeTypes>uuid</includeTypes>
<nullability>ALL</nullability>
<objectType>ALL</objectType>
</forcedType>
<forcedType>
<userType>x.y.z.color.Color</userType>
<converter>
x.y.z.studio.database_entities.converters.StringColorConverter
</converter>
<includeExpression>.*\..*color.*</includeExpression>
<nullability>ALL</nullability>
<objectType>ALL</objectType>
</forcedType>
<forcedType>
<userType>java.time.Instant</userType>
<converter>
x.y.z.studio.database_entities.converters.OffsetDateTimeInstantConverter
</converter>
<includeExpression>.*\..*creation_timestamp.*</includeExpression>
<nullability>ALL</nullability>
<objectType>ALL</objectType>
</forcedType>
</forcedTypes>
</database>
<generate>
<indexes>true</indexes>
<relations>true</relations>
<sequenceFlags>true</sequenceFlags>
<implicitJoinPathsToOne>true</implicitJoinPathsToOne>
<deprecated>false</deprecated>
<deprecationOnUnknownTypes>true</deprecationOnUnknownTypes>
<instanceFields>true</instanceFields>
<generatedAnnotation>false</generatedAnnotation>
<generatedAnnotationType>DETECT_FROM_JDK</generatedAnnotationType>
<nonnullAnnotation>false</nonnullAnnotation>
<nonnullAnnotationType>javax.annotation.Nonnull</nonnullAnnotationType>
<nullableAnnotation>false</nullableAnnotation>
<nullableAnnotationType>javax.annotation.Nullable</nullableAnnotationType>
<routines>true</routines>
<sequences>true</sequences>
<udts>true</udts>
<queues>true</queues>
<links>true</links>
<keys>true</keys>
<tables>true</tables>
<embeddables>true</embeddables>
<records>true</records>
<recordsImplementingRecordN>true</recordsImplementingRecordN>
<pojos>true</pojos>
<pojosEqualsAndHashCode>false</pojosEqualsAndHashCode>
<pojosToString>true</pojosToString>
<immutablePojos>false</immutablePojos>
<serializablePojos>true</serializablePojos>
<interfaces>false</interfaces>
<immutableInterfaces>false</immutableInterfaces>
<serializableInterfaces>true</serializableInterfaces>
<daos>false</daos>
<jpaAnnotations>false</jpaAnnotations>
<validationAnnotations>false</validationAnnotations>
<springAnnotations>false</springAnnotations>
<globalObjectReferences>true</globalObjectReferences>
<globalCatalogReferences>true</globalCatalogReferences>
<globalSchemaReferences>true</globalSchemaReferences>
<globalTableReferences>true</globalTableReferences>
<globalSequenceReferences>true</globalSequenceReferences>
<globalUDTReferences>true</globalUDTReferences>
<globalRoutineReferences>true</globalRoutineReferences>
<globalQueueReferences>true</globalQueueReferences>
<globalLinkReferences>true</globalLinkReferences>
<globalKeyReferences>true</globalKeyReferences>
<globalIndexReferences>true</globalIndexReferences>
<javadoc>true</javadoc>
<comments>true</comments>
<commentsOnCatalogs>true</commentsOnCatalogs>
<commentsOnSchemas>true</commentsOnSchemas>
<commentsOnTables>true</commentsOnTables>
<commentsOnColumns>true</commentsOnColumns>
<commentsOnUDTs>true</commentsOnUDTs>
<commentsOnAttributes>true</commentsOnAttributes>
<commentsOnPackages>true</commentsOnPackages>
<commentsOnRoutines>true</commentsOnRoutines>
<commentsOnParameters>true</commentsOnParameters>
<commentsOnSequences>true</commentsOnSequences>
<commentsOnLinks>true</commentsOnLinks>
<commentsOnQueues>true</commentsOnQueues>
<commentsOnKeys>true</commentsOnKeys>
<sources>true</sources>
<sourcesOnViews>true</sourcesOnViews>
<fluentSetters>true</fluentSetters>
<javaBeansGettersAndSetters>false</javaBeansGettersAndSetters>
<varargSetters>false</varargSetters>
<fullyQualifiedTypes></fullyQualifiedTypes>
<emptyCatalogs>false</emptyCatalogs>
<emptySchemas>false</emptySchemas>
<javaTimeTypes>true</javaTimeTypes>
<primaryKeyTypes>false</primaryKeyTypes>
<newline>\n</newline>
</generate>
<target>
<packageName>x.y.z.studio.database_entities.jooq</packageName>
<directory>
/A/B/C/build/generated/source/jooq/main
</directory>
<encoding>UTF-8</encoding>
<clean>true</clean>
</target>
</generator>
</configuration>

The namespace is xmlns="http://www.jooq.org/xsd/jooq-codegen-3.13.0.xsd". Is this correct?
This is the file that I found in tmp/generateJooq after trying to build with jOOQ 3.14.1 and getting that error message.

Thanks,

Maurizio

Lukas Eder

unread,
Nov 11, 2020, 7:53:45 AM11/11/20
to jOOQ User Group
I'm looking into this again now, because someone else seems to have had the same issue:


Things work when I add

buildscript {
    configurations['classpath'].resolutionStrategy.eachDependency {
if (requested.group == 'org.jooq') {
useVersion '3.14.3'
}
}
}
Some additional inspiration here:
https://github.com/etiennestuder/gradle-jooq-plugin/blob/master/example/specify_jooq_config_xml_schema_version/build.gradle
I'll be happy to look into further issues, but again, showing what you did *exactly* will be very helpful in helping you debug your issues.

Thanks,
Lukas
--
You received this message because you are subscribed to the Google Groups "jOOQ User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jooq-user+...@googlegroups.com.

Maurizio

unread,
Nov 12, 2020, 12:35:54 PM11/12/20
to jOOQ User Group
I'll try this fix and see if I can switch to 3.14.

I'm happy to report that the problem in the title and at the origin of this discussion has been fixed with https://github.com/jOOQ/jOOQ/issues/10928.

As a side note, I suspect there is an occasional tendency of jOOQ to swallow exceptions and, in so doing, obfuscating the problems. This is one case: I got no exception in my application, only in the MCVE. Running my application, what I saw was that a field was "null" when it should have not been. It was fetched, an exception was thrown during the mapping, but I had no trace of it.

Another problem that I currently have, is that, if I run several concurrent read-only transactions, many of them will fail. But if I run the same multi-step queries without placing them inside a transaction (it's not really needed, at this moment, it's just future-proofing), they work just fine. When debugging the transaction, I see there is an exception somewhere, but I still have to manage put my hands on the original one: I just see successive, generic, exceptions thrown by jOOQ. More on this topic in future.

Thanks again for helping me!

Maurizio.

Lukas Eder

unread,
Nov 12, 2020, 2:38:03 PM11/12/20
to jOOQ User Group
On Thu, Nov 12, 2020 at 6:35 PM Maurizio <maurizi...@gmail.com> wrote:
I'll try this fix and see if I can switch to 3.14.

I'm happy to report that the problem in the title and at the origin of this discussion has been fixed with https://github.com/jOOQ/jOOQ/issues/10928.

Thanks again for the confirmation.
 
As a side note, I suspect there is an occasional tendency of jOOQ to swallow exceptions and, in so doing, obfuscating the problems. This is one case: I got no exception in my application, only in the MCVE. Running my application, what I saw was that a field was "null" when it should have not been. It was fetched, an exception was thrown during the mapping, but I had no trace of it.

Yes, that design flaw might be addressed in a future major release 4.0. org.jooq.tools.Convert tends to turn values that cannot be converted to the target type into nulls, rather than throwing exceptions. There are a few open issues discussing this. Nothing we can do in the short term without breaking many applications out there.
 
Another problem that I currently have, is that, if I run several concurrent read-only transactions, many of them will fail. But if I run the same multi-step queries without placing them inside a transaction (it's not really needed, at this moment, it's just future-proofing), they work just fine. When debugging the transaction, I see there is an exception somewhere, but I still have to manage put my hands on the original one: I just see successive, generic, exceptions thrown by jOOQ. More on this topic in future.

Well, you know the drill :) Not much I can say about that from this description, except, I don't think that sounds like something we're aware of.

Best Regards,
Lukas
Reply all
Reply to author
Forward
0 new messages