Jackson, SequenceWriter: example of how to incrementally build a complex JSON?

[Rodrigo Ottero]

Hi folks,

Apologies in advance if I am not clear enough, happy to provide clarifications if required.

I am having a hard time finding information about how to use SequenceWriter to output a custom POJO to a JSON file in an incremental way.

I found Bozhidar's solution to incrementally output data as a JSON array (https://dzone.com/articles/writing-big-json-files-with-jackson); that is natively supported by SequenceWriter through the init(true) method.

In my use case, we have a service that receives a SQL query and outputs it as a CSV file. 

Since some queries may produce millions of rows, this CSV is written incrementally as the result set is read, line by line, to prevent out of memory issues.

I was asked to add the option of outputting a JSON file instead of a CSV file, but in the same incremental way.

The gotcha is that I need to use a legacy JSON structure, something like:

{
    "metadata": {
        "query": "SELECT * FROM ZZZ",
        "processingTime": 100,
    },
    "result": {
        "columns": [
            "Column1",
            "Column2",
            "..."
        ],
        "result": [
            [
                "abc",
                "def",
                "..."
            ],
            [
                "foo",
                "bar",
                "..."
            ],
            ...millions of lines...
        ]
    }
}

The part in red is the part I want to incrementally write/update on the JSON file, while keeping the blue parts fixed.

I was browsing the javadocs for SequenceWriter (https://developer.adobe.com/experience-manager/reference-materials/6-5/javadoc/com/fasterxml/jackson/databind/SequenceWriter.html) and saw that the constructor accepts a DefaultSerializerProvider and a JsonGenerator object, but I am not sure how I would use these resources to achieve my objective.

Would you have a piece of documentation or an example of how I could use SequenceWriter to incrementally write the JSON file I need, based on a custom POJO? 

Or could you please tell me if I should be looking for another approach, in case SequenceWriter is not the answer for my use case?

Many thanks,

Best regards,

Rodrigo

[Rodrigo Ottero]

TL;DR:

Is there any way I could stream data to a huge json file in small chunks, but adding/appending said data to one field only? 

I mean, this json file represents one single pojo object, and I need to keep appending data to one of its fields. 

Thanks! 

(sent by smartphone / enviado pelo celular)

--
You received this message because you are subscribed to a topic in the Google Groups "jackson-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jackson-user/kXvRr6b7rg4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jackson-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jackson-user/01796d7e-5b37-4180-94ea-a7be908d4efan%40googlegroups.com.

[Tatu Saloranta]

On Wed, Jun 15, 2022 at 2:34 PM Rodrigo Ottero <ott...@gmail.com> wrote:
>
> TL;DR:
>
> Is there any way I could stream data to a huge json file in small chunks, but adding/appending said data to one field only?
>
> I mean, this json file represents one single pojo object, and I need to keep appending data to one of its fields.

Incremental output is indeed possible, but if output is not a sequence
of elements, SequenceWriter is probably not the way to go.
You can still use `ObjectMapper` (and `ObjectWriter`) `writeValue()`
methods that take `JsonGenerator`, however.

So usually you will need to write "outer" part of content straight
using `JsonGenerator` (or CsvGenerator etc) methods,
and then for serializing values within, use ObjectMapper. Or use
`JsonGenerator`, depending on case.

I hope this helps,

-+ Tatu +-

> You received this message because you are subscribed to the Google Groups "jackson-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jackson-user...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/jackson-user/CAOmZcH%2BxxXQoSTw7cTMsnZ1eHtcQtmuMgbkYDxkWjYMHeoeH_Q%40mail.gmail.com.