| The issue isn’t related to BQ source nested records but it’s a reference issue, we have tested this within Wrangler and as a File Source and the issue is when there are referenced records, the parent record is saved as a known record and is used for every record that has the same name, causing it to fail. While debugging we found that the issue may be with StructuredRecord when there are nested records, so we performed some other tests: 1. Created a JSON dataset with nested records where the child has same name as parent and tried to parse from Wrangler, the output was wrong,
2. Created a JSON dataset with nested records where the child has same name as parent and tried to parse with File Plugin as source, if you press Validate button, there will be issue in Output Schema,
3. Created a Unit Test with StructuredRecord where the child has same name as parent and the output was wrong,
Schema mainSchema = Schema.recordOf(“record”, Schema.Field.of(“Field”, Schema.of(Schema.Type.STRING)));
Schema subSchema = Schema.recordOf(“record”, Schema.Field.of(“record”, mainSchema));
Output: {“type”:“record”,“name”:“record”,“fields”:[{“name”:“record”,“type”:“record”}]} and if we read the source code of StructuredRecord we can see that this is related with the ability to reference other schemas:[ |https://github.com/cdapio/cdap/blob/83798c0cc09cb616a54d1065962e07c2d04c21ce/cdap-api-common/src/main/java/io/cdap/cdap/internal/io/SchemaTypeAdapter.java#L367-L370][https://github.com/cdapio/cdap/blob/83798c0cc09cb616a54d1065962e07c2d04c21ce/cdap-api-common/src/main/java/io/cdap/cdap/internal/io/SchemaTypeAdapter.java#L367-L3+|https://github.com/cdapio/cdap/blob/83798c0cc09cb616a54d1065962e07c2d04c21ce/cdap-api-common/src/main/java/io/cdap/cdap/internal/io/SchemaTypeAdapter.java#L367-L370]+70 |
