Merging datasets

19 views
Skip to first unread message

jsc...@twitter.com

unread,
May 23, 2017, 9:17:26 PM5/23/17
to json-stat
  Is there a way to merge datasets?

  I have a dataset response that typically includes the values for a range of dates. However, there are times when the size gets large enough (20MB+) that I convert the response into a collection of datasets. 

  Something like:

{
"version" : "2.0",
"class" : "collection",
"link" : {
"item" : [
{
"class" : "dataset",
"href" : "/time-series/histogram?bin_width=250&client=android&country=All&start_time=2017-04-22T00%3A00%3A00Z&end_time=2017-04-23T00%3A00%3A00Z"
},
{
"class" : "dataset",
"href" : "/time-series/histogram?bin_width=250&client=android&country=All&start_time=2017-04-23T00%3A00%3A00Z&end_time=2017-04-24T00%3A00%3A00Z"
},
{
"class" : "dataset",
"href" : "/time-series/histogram?bin_width=250&client=android&country=All&start_time=2017-04-24T00%3A00%3A00Z&end_time=2017-04-25T00%3A00%3A00Z"
}
                        ...
                ]
        }
}

  How can I merge N datasets into 1 with the correct indices/values for dimension/id/role/value/etc?

  If this isn't the correct approach, can you suggest another way to deal with merging new data? We're doing this because of file size, but it seems there are other situations (polling for new data in an app) where you'd need to merge in new data.

John

Xavier Badosa

unread,
May 24, 2017, 5:12:28 AM5/24/17
to json-stat
The JSON-stat Javascript Utilities Suite (JJUS), which is a helper library built on top of the JSON-stat Javascript Toolkit (JJT), has something called "join":


Some examples:

Sequential Dataset Join

Dataset Join by Concept

Dataset Join by Sex

Dataset Join by Sex and Concept

Let me know if join() is useful to your purposes,

X.

jsc...@twitter.com

unread,
May 25, 2017, 1:38:22 PM5/25/17
to json-stat
Xavier,

  Thank you. I forgot about `join`.

  It does do what I want, but the performance is an issue. I was able to work around it but it'd be nice to investigate ways to speed up `.join`. I'll open a ticket and we can discuss there.

  Thanks for the links & examples.

John
Reply all
Reply to author
Forward
0 new messages