getting multiple json objects from one file

[Andy A]

What function should I use to get multiple json objects from one file?

For example, a plain-text file with 3 objects, each object containing 4 key/value pairs, the values of each key vary in size.

I'm already learning how to access the keys and values in a single object, so that is mostly under control ;) But how do I "process" each object?

I've mostly experimented with json_tokener_parse_ex() so far, but that requires the length of a single object to be known, if I understand correctly.

I can read in the file with fread of course, so basically it's from that point, when the entire contents of a the file is a char* that I want to parse it.

[Andy A]

Re-reading my post, I'm not sure if I was clear enough. I want to read in a file and be able to see each object individually so I can get the values from each object.

[Andy A]

I apologize for the multiple posts.

This is the code I'm experimenting with. There are 3 objects in the file. The values from only one object are being printed:

#include <stdio.h>
#include <json.h>
#include <stdlib.h>

#define BUF_SIZE 2048

enum json_tokener_error jerr;

int
main (int argc, char *argv[])
{
  FILE *fp = fopen ("units.json", "r");
  if (fp == NULL)
  {
    fprintf (stderr, "Unable to open file.");
    exit (EXIT_FAILURE);
  }

  fseek (fp, 0, SEEK_END);
  int f_size = ftell (fp);
  rewind (fp);

  char buffer[BUF_SIZE];

  size_t result = fread (buffer, 1, f_size, fp);

  printf ("%s", buffer);
  json_tokener *tok;

  do
  {
    tok = json_tokener_new_ex (JSON_TOKENER_DEFAULT_DEPTH);

    json_object *new_object = json_tokener_parse_ex (tok, buffer, f_size);

    json_object_object_foreach (new_object, key, val)
    {
      printf ("key: '%s' type of val:..", key);
      int val_type = json_object_get_type (val);
      printf ("%d  value is ", val_type);
      switch (val_type)
      {
      case json_type_boolean:
        printf ("%d\n", json_object_get_boolean (val));
        break;
      case json_type_string:
        printf ("%s\n", json_object_get_string (val));
        break;
      case json_type_int:
        printf ("%d\n", json_object_get_int (val));
        break;
      }
    }
  }
  while ((jerr = json_tokener_get_error (tok)) == json_tokener_continue);

  return 0;
}

[Eric Haszlakiewicz]

On Mon, Nov 26, 2018 at 9:45 PM Andy A <andy0...@gmail.com> wrote:
>
> I apologize for the multiple posts.
>
> This is the code I'm experimenting with. There are 3 objects in the file. The values from only one object are being printed:
>

...

> size_t result = fread (buffer, 1, f_size, fp);

First problem: you're only calling fread once, and with a size that
could be larger than BUF_SIZE.
If your file happens to be small enough to fit in BUF_SIZE, then
you're probably ok, but this won't work in general.
For larger inputs you'll need to loop on reading no more than BUF_SIZE
bytes into the buffer, then incrementally feeding that to the json
tokener.

> json_object *new_object = json_tokener_parse_ex (tok, buffer, f_size);

Problem 2: you're passing the full file size here, not however actual
bytes fread() read. (i.e. result)

> json_object_object_foreach (new_object, key, val)

Problem 3: you're assuming parse_ex actually returned an object. If
it hasn't yet parsed a complete object then it will return NULL and
this foreach will crash.

> ...

> while ((jerr = json_tokener_get_error (tok)) == json_tokener_continue);

Problem 4: if you've parsed a complete object, get_error will indicate
json_tokener_success instead, but actually you need to check for both
values.
If you get NULL and json_tokener_continue, read more bytes from the
file and call parse_ex again.
If you get non-NULL and json_tokener_success, do whatever you need to
with the returned object, then adjust your offset into your buffer and
the bytes available and pass the remaining bytes into parse_ex to
start the next object.

> do
> {
> tok = json_tokener_new_ex (JSON_TOKENER_DEFAULT_DEPTH);

> ...
> while (...)

Problem 5: you're leaking tokener objects each time you go through this loop.

In brief, you need to adjust how you're calling
json_tokener_parse_ex() so you keep giving it more bytes to parse the
next object by using tok->char_offset and arranging to re-fill your
buffer as needed.

Eric

[Eric Haszlakiewicz]

On Mon, Nov 26, 2018 at 9:22 PM Andy A <andy0...@gmail.com> wrote:
>
> What function should I use to get multiple json objects from one file?
>
> For example, a plain-text file with 3 objects, each object containing 4 key/value pairs, the values of each key vary in size.
>
> I'm already learning how to access the keys and values in a single object, so that is mostly under control ;) But how do I "process" each object?
>
> I've mostly experimented with json_tokener_parse_ex() so far, but that requires the length of a single object to be known, if I understand correctly.

No, it doesn't require that. It will parse as much of a json string
as you give it, either telling you that it needs more data
(json_tokener_continue) or returning the object that it parsed
(json_tokener_success). When it parses a complete object, the
internal state of the tokener (ie. tok->char_offset) will tell you
where it stopped at and you can start parsing the next object from
there (or handle that as an error, if you only expect to get a single
object).

Eric

[Andy A]

Thank you for the explanation. With your help, I got the code to work as intended. I added some extra error-checking as well; although it's just my "experimental" code, better for users who read this later when it comes up in a search.

#include <stdio.h>
#include <json.h>
#include <stdlib.h>

#include <string.h>

#define BUF_SIZE 80

enum json_tokener_error jerr;

int
main (int argc, char *argv[])
{
  FILE *fp = fopen ("units.json", "r");
  if (fp == NULL)
  {
    fprintf (stderr, "Unable to open file.");
    exit (EXIT_FAILURE);
  }

  char buffer[BUF_SIZE];

  json_tokener *tok = NULL;
  json_object *new_object = NULL;
  tok = json_tokener_new_ex (JSON_TOKENER_DEFAULT_DEPTH);

  char *result = NULL;

  do
  {
    do
    {
      result = fgets (buffer, BUF_SIZE, fp);
      if (result == NULL)
        break;
      new_object = json_tokener_parse_ex (tok, buffer, strlen (buffer));

    }
    while ((jerr = json_tokener_get_error (tok)) == json_tokener_continue);

    if (jerr != json_tokener_success)
    {
      fprintf (stderr, "Error: %s\n", json_tokener_error_desc (jerr));
      // Handle errors, as appropriate for your application.
    }

    if (new_object == NULL)
    {
      fprintf (stderr, "Incomplete or invalid object\n");
      exit (EXIT_FAILURE);

    }

    json_object_object_foreach (new_object, key, val)
    {
      printf ("key: '%s' type of val:..", key);
      int val_type = json_object_get_type (val);
      printf ("%d  value is ", val_type);
      switch (val_type)
      {
      case json_type_boolean:
        printf ("%d\n", json_object_get_boolean (val));
        break;
      case json_type_string:
        printf ("%s\n", json_object_get_string (val));
        break;
      case json_type_int:
        printf ("%d\n", json_object_get_int (val));

       dd break;
      }
    }
    printf ("\n");
  }
  while (tok->char_offset < strlen (buffer));

  if (fclose (fp) == EOF)
  {
    fprintf (stderr, "Error closing file\n");
    exit (EXIT_FAILURE);
  }
  return 0;
}

[Andy A]

Hmm.. after I formatted my code in the post above, I couldn't edit the post anymore... continuing...

This is the contents of the JSON file:

{
  "name": "diamond mine",
  "cost": 1000,
  "is_destructible": true,
  "armor": 150
},
{
  "name": "barracks",
  "cost": 250,
  "is_destructible": true,
  "armor": 300
},
{
  "name": "defense tower",
  "cost": 300,
  "is_destructible": true,
  "armor": 400
}

and my output

key: 'name' type of val:..6  value is diamond mine
key: 'cost' type of val:..3  value is 1000
key: 'is_destructible' type of val:..1  value is 1
key: 'armor' type of val:..3  value is 150

key: 'name' type of val:..6  value is barracks
key: 'cost' type of val:..3  value is 250
key: 'is_destructible' type of val:..1  value is 1
key: 'armor' type of val:..3  value is 300

key: 'name' type of val:..6  value is defense tower
key: 'cost' type of val:..3  value is 300
key: 'is_destructible' type of val:..1  value is 1
key: 'armor' type of val:..3  value is 400

[Andy A]

I forgot I wanted to increase BUF_SIZE to compensate for lines longer than 80

#define BUF_SIZE 512

[Eric Haszlakiewicz]

On Tue, Nov 27, 2018 at 11:31 AM Andy A <andy0...@gmail.com> wrote:
>
> I forgot I wanted to increase BUF_SIZE to compensate for lines longer than 80
>
> #define BUF_SIZE 512

That's completely irrelevant. There are no "lines" in json, it's all
objects, arrays or strings delimited by {, [ or " (plus a few
constants). If you're expecting literal line breaks to mean anything
within json input you're going about it wrong.

Eric

[Eric Haszlakiewicz]

On Tue, Nov 27, 2018 at 11:14 AM Andy A <andy0...@gmail.com> wrote:
>
> Thank you for the explanation. With your help, I got the code to work as intended. I added some extra error-checking as well; although it's just my "experimental" code, better for users who read this later when it comes up in a search.
>

...
There's a notable limitations in your code that it only works with
input where top-level json objects are separated with a newline
(because of your use of fgets). However, if you can guarantee your
input is formatted in that way, then that seems like probably a
reasonable tradeoff to avoid the extra work to actually use the
tok->char_offset value for starting the next object. Glad to hear you
got it working!

Eric

[Andy Alt]

Thanks for the code review and feedback, Eric. I'll keep your comments
in mind as I move forward with more complex objects. Not only am I new
to the json-c API, I'm also new to working with JSON.

--
-Andy