How to append single json strings to an existing file?

[MV]

Hi there,

I am using Akka Actor framework to receive messages as Java objects. When I receive the message, I do the following:

(1) Use ObjectMapper.withPrettyWriter().writeValueAsString(<java pojo>)

(2) Open a file and use java.nio.Files API to write to the file. When I write to the file, I use Arrays.asList(json_string_from step_1) with file in append mode.

I am having a few issues:

(1) I am unable to write the first json string as an array. When I write it the first time, I would like to see:

[

 {

    "name" : "test"

    "age" : 29

},  --> I don't get this "comma" with closing "]"

]  ---> don't get this.

All I get is :

{

   "name" : "test"

   "age" : 29

}

(2) While calling successive Files.write (...), the output in the file is 

{

   "name" : "test"

   "age" : 29

}

{

   "name" : "test2"

    "age" : 30

}

This results in malformed JSON file and I cannot use objectMapper.readValue as a list. Looks like I have to use Files API to load the data.

Qn) How do I write a single json as an array with ", ]" and then append to this list as data starts flowing in through the messaging architecture?

I did see some solution on reading the file first, add new data to the array and then write the whole list again to the file each time. I think with messaging architecture this is not very efficient as we will see a lot of messages. 

Is there someway to open the file and write the first data and then keep appending to the file?

Thanks

MV

[Tatu Saloranta]

On Sat, Dec 29, 2018 at 3:01 PM MV <mee...@gmail.com> wrote:
>
> Hi there,
>
> I am using Akka Actor framework to receive messages as Java objects. When I receive the message, I do the following:
> (1) Use ObjectMapper.withPrettyWriter().writeValueAsString(<java pojo>)
> (2) Open a file and use java.nio.Files API to write to the file. When I write to the file, I use Arrays.asList(json_string_from step_1) with file in append mode.

Ok, a few questions. Let's see if I can help.

>
> I am having a few issues:
> (1) I am unable to write the first json string as an array. When I write it the first time, I would like to see:
> [
> {
> "name" : "test"
> "age" : 29
> }, --> I don't get this "comma" with closing "]"
> ] ---> don't get this.
>
> All I get is :
> {
> "name" : "test"
> "age" : 29
> }
>
> (2) While calling successive Files.write (...), the output in the file is
> {
> "name" : "test"
> "age" : 29
> }
> {
> "name" : "test2"
> "age" : 30
> }
> This results in malformed JSON file and I cannot use objectMapper.readValue as a list. Looks like I have to use Files API to load the data.
>
> Qn) How do I write a single json as an array with ", ]" and then append to this list as data starts flowing in through the messaging architecture?
> I did see some solution on reading the file first, add new data to the array and then write the whole list again to the file each time. I think with messaging architecture this is not very efficient as we will see a lot of messages.
> Is there someway to open the file and write the first data and then keep appending to the file?
>
> Thanks
> MV

Ok, first things first. I think that your writing actually makes
sense, and although you can not read it with
`ObjectMapper.readValue()` (since it is a sequence of JSON values and
not a single JSON value), you can read it with something like:

----

MappingIterator<POJO> it = mapper.readerFor(POJO.class)
.readValues(file); // note it is "values" NOT "value"
while (it.hasNextValue()) {
POJO p = it.nextValue();
}
it.close();

-----

Now; if you really do want a JSON array with values, there are ways to
achieve that. You would probably want to create a `JsonGenerator`
directly (from `JsonFactory`), call `writeStartArray()`, and then for
each POJO either:

1. Unless you really have to do `writeValueAsString()` separately (if
so, why?), just use

ObjectMapper.writeValue(jsonGenerator, value)

which will handle addition of separators etc

2. If you do have to create intermediate JSON string or byte array,
instead use generator directly

jsonGenerator.writeRawValue(encodedJson)

and then at the end of output, call `jsonGenerator.writeArray()` to
get the close marker.

But as I said, first solution seems like a better choice to me given
information here.

-+ Tatu +-

[MV]

Hi Tatu

Thanks for your quick response. So I decided to try option #1 as you suggested i.e. continue writing single messages to a file in "append" mode (without using JsonGenerator) for now. 

I am still having trouble because it seems like my stream closes with some sort of NPE and circular reference. This only seems to occur when I use:

objectMapper.writeValue(BufferedWriter (FileWriter), pojo)

It is the same error when I use writeValueAsString().

Instead, If I used these sequence of steps, it works.

FileWrite fw  = new FileWriter("myMessages.json", true);

BufferedWriter bw = new BufferedWriter(fw);

String jsonStr = objectMapper.writeValueAsString(pojo)

bw.write(jsonStr).

Of course opening "myMessages.json" file has a lot of red squiggles because the format is incorrect. It is just a list of messages.

I do have these settings for ObjectMapper:

objectMapper.setSerializationInclusion(JsonInclude.Include.Non_Null)

objectMapper.setDateFormat(new SimpleDateFormat("yyyy-MM-dd HH:mm a z"));

It does not look like these settings are taking effect.

I am not able to find the NPE because when I look in the debugger through IDE (IntelliJ), nothing is null.

The error is along the lines of NPE from circular reference. But the rest of the code uses the same class for future calculations and there is no NPE anywhere in the code.

I would like to use objectMapper.writeValue(bufferedWriter, pojo) if possible but don't know how to resolve this circular NPE. 

I am going to try loading the values as you mentioned with "readValues()" and will keep you posted here.

Thanks a lot for your timely responses. I really appreciate it.

~MV

[MV]

Hi Tatu

I made a few changes to my POJO to avoid the circular reference issue. Now I can write the objects to  file and the output in the file is in the format:

{ pojo } 

{ pojo }

I am unable to deserialize using your technique and from the error message, I think it is because the individual fields inside the POJO are lists (ArrayList) and some items in the ArrayList also contain a Map (e.g. travelDistanceMap item is a LinkedHashMap in the file attached).

The error I am seeing right now, is that when deserializing "homeList", it says it cannot find the property "homeId" when it is clearly found in the output and defined in the POJO. I am guessing it is because it is a list and that is not how I am reading it back. 

By the way, I am using your suggested technique of MappinIterator and objectMapper.readValues() to read these. I am attaching 2 screenshots:

(1) IS the JSON output

(2) My deserializer code

After you take a peek at it, here are the questions:

(1) When I get the next token, should I actually check if the token == "homeList" and then call objectMapper.readValue(token, TypeReference)?

(2) Should I be using annotations in the class on the fields that are a list? What about the case of travelDistanceMap where it is a Map field inside a token?

(3) Writing this to a file with *.json and opening in an editor complains about malformed JSON. I understand this is because I am writing out individual JSON objects instead of a proper JSON list. Should I just rename the file to *.txt?

Thanks in advance for your time,

MV

[Tatu Saloranta]

I think at this point I would need actual unit test. Reading code seemed ok, but without actual exception message it's hard to know what might be wrong.

-+ Tatu +-

--
You received this message because you are subscribed to the Google Groups "jackson-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jackson-user...@googlegroups.com.
To post to this group, send email to jackso...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[MV]

Hi Tatu

I am attaching 3 more images:

(1) The error message

(2) How I am writing out the message to the file

(3) the object mapper setup I have for writing to the file. Please note that I am using 2 object mapper instances, one in the Actor for writing the data and the other in the test file to read/load the dat.

Any help is really appreciated. Thanks in advance for your time.