Stream closed before data is read and written to JSON

[Oliver Hancock]

I am taking in a JSON file that is an array of objects. I am trying to unnest it and serialize it however memory management is very import, meaning that the json being read and unested needs to be done through through a stream. The given input would look like an array of json objects looking like this:

{"a": "some \"string\" here"}


With the desired outcome looking like

{"data": "{\"a\":\"some \\\"string\\\" here\"}"}


for the reading and writing of the data object i am using the jackson streaming library as it is the least memory intensive library fitting with my requirements. To serialize the data i am use an chararray library called fastUtils however the problem persists without this so it is not the library or the serialization itself. I have also increased the buffer size of both the pipedReader object and the char buffer but again there is no benefit.

Current code:

Reader reader = new InputStreamReader(inputStream, StandardCharsets.UTF_8);
Writer writer = new OutputStreamWriter(outputStream, StandardCharsets.UTF_8);

final JsonFactory jsonFactory = new JsonFactory();
final JsonGenerator jsonGenerator = jsonFactory.createGenerator(writer);
final JsonParser jsonParser = jsonFactory.createParser(reader);

jsonParser.nextValue();

jsonGenerator.writeStartObject();
jsonGenerator.writeFieldName("dataItem");
jsonGenerator.flush();
writer.flush();
writer.write(58)

PipedReader pipedReader = new PipedReader(4096);
PipedWriter pipedWriter = new PipedWriter(pipedReader);
JsonGenerator jsonGeneratorPiped = jsonFactory.createGenerator(pipedWriter);

new Thread(
    new Runnable(){
    public void run() {
        jsonGeneratorPiped.copyCurrentStructure(jsonParser);
        jsonGeneratorPiped.flush();
        pipedWriter.close();
    }
    }
).start();

char[] buffer = new char[2048];
int len = reader.read(buffer);
char quote = '"';
char escape = '\\';
CharArrayList listBuffer;
//listBuffer = new CharArrayList();
while (len != -1) {
    listBuffer = new CharArrayList(buffer,0,len);
    //listBuffer.addElements(0,buffer,0,len);
    for (CharListIterator iter = listBuffer.iterator(); iter.hasNext();) {
        if (iter.nextChar() == quote) {
            iter.set(escape);
            iter.add(quote);
        }
    }

    writer.write(listBuffer.elements(), 0, listBuffer.size());

    //toCharArray uses lot of memory
    //writer.write(listBuffer.toCharArray(), 0, listBuffer.size());

    //listBuffer.trim();
    len = reader.read(buffer);
}

writer.flush();
pipedReader.close();
jsonGeneratorPiped.close();

jsonGenerator.writeEndObject();
jsonGenerator.flush();

jsonGenerator.close();
jsonParser.close();


that is the current code i have implemented and it works fine for files that are small, however with larger files >10mb from my testing, the stream closes mid way through resulting in this error

java.io.IOexception: stream is closed

Any help would be greatly appreciated

[Tatu Saloranta]

Since Jackson does not do any such limits checking currently, I'd
guess use of PipedReader / PipedWriter is problematic.
Your Runnable may be throwing an exception that you are not catching

Alternatively if you are reading from/writing to some network resource
(reading from a server via URL), it's likely some limit by that
system. 10 megs sounds like the kind of limit many systems impose for
maximum requests/responses.

I hope this helps,

-+ Tatu +-