What is the most efficient way to get a line delimited JSON network response into CBL?

[Jared McFarland]

I am getting line delimited JSON from Mixpanel.com's raw export API and I'm looking for a more efficient way to handle this task than what I'm currently doing.

For reference here is the API: https://mixpanel.com/docs/api-documentation/exporting-raw-data-you-inserted-into-mixpanel

And if you aren't already familiar with LDJSON: http://en.wikipedia.org/wiki/Line_Delimited_JSON

Right now I'm just waiting for the entire response to complete and then transforming the data into an NSArray of NSDictionaries like so...

    NSError *jsonError;

    NSString *jsonString = [[NSString alloc] initWithData:data encoding:NSUTF8StringEncoding]; // data is returned by my NSURLSessionDataTask's completion handler

    

    NSMutableArray *resultsArray = [[jsonString componentsSeparatedByString:@"\n"] mutableCopy];

    [resultsArray removeObjectAtIndex:[resultsArray count]-1]; // remove empty last object

    

    NSMutableArray *jsonArray = [NSMutableArray array];

    

    for (NSString *result in resultsArray) {

        NSDictionary *jsonEvent = [NSJSONSerialization JSONObjectWithData:[result dataUsingEncoding:NSUTF8StringEncoding] options:0 error:&jsonError];

        [jsonArray addObject:jsonEvent];

    }

Then I just take jsonArray and...

    BOOL transaction = [database inTransaction:^BOOL{

        for (NSDictionary *event in jsonArray) {

            CBLDocument *document = [database createDocument];

             NSError *documentError;

             NSMutableDictionary *mutableEvent = [event mutableCopy];

             [mutableEvent setObject:kMPCBLDocumentTypeEvent forKey:kMPCBLDocumentKeyType];

             [document putProperties:mutableEvent error:&documentError];

             if (documentError) {

                 return NO;

             }

        }

        return YES;

    }];

However, the whole point of LDSON is that you don't have to wait for the entire response, so I'm trying to figure out the best way to go about parsing the data as it streams in, and probably calling inTransaction: on batches of 1000 (or so) event objects. These responses can be 100,000's of documents, or even millions in rare cases. 

The NSURLSessionDataDelegate protocol has the didReceiveData: method but I'm struggling to properly parse the results and wondering if there is already some well established pattern for serializing LDSON as it is received.

Any input would be greatly appreciated!

[Jens Alfke]

On Apr 11, 2015, at 6:30 AM, Jared McFarland <jared...@gmail.com> wrote:

However, the whole point of LDSON is that you don't have to wait for the entire response, so I'm trying to figure out the best way to go about parsing the data as it streams in

Just accumulate the incoming data into an NSMutableData object. After you append new bytes, look for a newline in the data (probably using memchr or something like that). If you find one, copy the bytes from the start to just before the newline into a new NSData, then use NSJSONSerialization to parse it. Then delete the consumed bytes (including the newline) from the mutable data buffer.

—Jens

[Jared McFarland]

I just discovered SBJSON which handles chunked JSON data and it's actually working quite well. I also just discovered CBLJSONReader but I failed to get YAJL to build, due to "LLVM 6.0 Garbage collection is no longer allowed"

[Jared McFarland]

Hi Jens,

Thanks for this, makes sense. I didn't see your response before I made my post below. SBJSON seems to be doing the trick though.

Thanks again,

Jared