Memory leak (json_tokener_parse || json_object_get_string)?

[Da Beave]

I have a strange situation.   I've been using json-c to parse some data that I receive via libcurl.   It is a multi-threaded application (pthreads),  and basically

works like this:

{

..<snip>...

/* Get data from the server */

curl_easy_setopt(curl, CURLOPT_WRITEDATA, &response);  /* "response" storage the JSON response from the server */ 

..<snip>...

json = json_tokener_parse(response);   

/* "cat" is the only thing I'm looking for at this time! */

cat  = json_object_get_string(json_object_object_get(json, "cat"));

.. <snip - do some stuff with "cat"> ..

free(json); 

}

Basically,  "response" has the JSON data from my server.  json-c correctly pulls the data I need from the JSON string (pulling "cat").  We do

some stuff with the json-c results,  then "free" the json struct.   However,  memory usage seems go up and up. If I replace the json calls with a home made (and ugly!) strtok_r() and manually pull the data without json-c,  my memory usage seems to remain constant.   Over about 20 minutes of run time,  the strtok_r version uses about 1/2 of the memory the json-c uses. 

Is there something I'm not doing to clean up/garbage collect properly?  The strtok_r() isn't an option for me,  as I do have other fields and more complex JSON queries I'll need to parse.  

I've tested with json-c from the git tree and json-0.9 .   

Thank you!

- Champ Clark III

[Eric Haszlakiewicz]

On Nov 1, 2012 12:39 PM, "Da Beave" <dab...@gmail.com> wrote:
> {
> ..<snip>...
> /* Get data from the server */
> curl_easy_setopt(curl, CURLOPT_WRITEDATA, &response);  /* "response" storage the JSON response from the server */ 
> ..<snip>...
> json = json_tokener_parse(response);   
>
> /* "cat" is the only thing I'm looking for at this time! */
>
> cat  = json_object_get_string(json_object_object_get(json, "cat"));
>
> .. <snip - do some stuff with "cat"> ..
>
> free(json); 

"free"?  Is that a typo or are you really just calling the free() function?  With json_object structures you need to use json_object_put() because it's more than just a flat block of memory.

Eric

[Da Beave]

struct json_object *json = NULL;

[Da Beave]

I should probably rename that.   No, it's not "freeing" the function,  but the struct.  

I'm trying the json_object_put() now.  

[Vijay Maddula]

Hi Da Beave,

I am also facing same problem. I am using json_object_put(jsonobject) but it is not freeing the memory.

is json_object_put() worked for you?

Thanks

Vijay

[Eric Haszlakiewicz]

On Thu, Aug 23, 2018, 9:01 AM Vijay Maddula <mvijayk...@gmail.com> wrote:

Hi Da Beave,

I am also facing same problem. I am using json_object_put(jsonobject) but it is not freeing the memory.

is json_object_put() worked for you?

Thanks

Vijay

On Friday, November 2, 2012 at ...

You're responding to a very old email, so I'd be surprised if you got any useful answer specifically answering whether the put call worked for him.

However, plenty of other people are using it successfully.  If you're have some issues, perhaps you could post a snippet of your code and we can see if there's anything apparently wrong with it.

Eric

[Vijay Maddula]

Hi Eric,

My code is thread based and like this:

in one thread i am calling

JSONobj = json_tokener_parse(chunk.memory);

chunk.memory contains data which is downloaded using curl and is about 2MB of data.

After this i am immediately calling  

json_object_put(JSONobj);

after this exiting the thread. but my memory is increasing when i call json_tokener_parse and after json_object_put,  memory is not coming back.

I have observed memory details in top command.

Thanks

Vijay

[Eric Haszlakiewicz]

On Thu, Aug 23, 2018, 9:45 AM Vijay Maddula <mvijayk...@gmail.com> wrote:

My code is thread based and like this:

in one thread i am calling

JSONobj = json_tokener_parse(chunk.memory);

chunk.memory contains data which is downloaded using curl and is about 2MB of data.

After this i am immediately calling  

json_object_put(JSONobj);

after this exiting the thread. but my memory is increasing when i call json_tokener_parse and after json_object_put,  memory is not coming back.

I have observed memory details in top command.

That should work fine, but top is far too coarse of a tool to use to figure out what's going on.  Can you run you test through valgrind to try to narrow down where the memory is leaking from?

Or, alternately, trim this down to a minimal test case that shows the problem?  i.e. cut the threading, cut the curl, etc... and just hard code the contents of chunk.memory and the json-c calls.

Eric

[Michael Clark]

On 24/08/2018, at 9:19 AM, Eric Haszlakiewicz <haw...@gmail.com> wrote:

On Thu, Aug 23, 2018, 9:45 AM Vijay Maddula <mvijayk...@gmail.com> wrote:

My code is thread based and like this:

in one thread i am calling

JSONobj = json_tokener_parse(chunk.memory);

chunk.memory contains data which is downloaded using curl and is about 2MB of data.

After this i am immediately calling  

json_object_put(JSONobj);

after this exiting the thread. but my memory is increasing when i call json_tokener_parse and after json_object_put,  memory is not coming back.

I have observed memory details in top command.

That should work fine, but top is far too coarse of a tool to use to figure out what's going on.  Can you run you test through valgrind to try to narrow down where the memory is leaking from?

Ya valgrind is a good idea.

Or, alternately, trim this down to a minimal test case that shows the problem?  i.e. cut the threading, cut the curl, etc... and just hard code the contents of chunk.memory and the json-c calls.

Ya allocate an object in one thread queue it to another and get the other thread to free it.

You could slap it into test_queue.cc here:

https://github.com/michaeljclark/queue_atomic

The test code sends items via a queue to many threads and the threads send the items back and the master thread counts them to make sure they all arrived. You could change it to parse a JSON object in the master thread, scatter then to other threads over queues and have the other threads free them.

We should find out the malloc implementation being used. 

Some malloc implementations have per thread pools so allocating in one thread and releasing in another could indeed make the memory usage grow.

Assuming a completely memory clean thread-safe library, i.e. the use of appropriate locking when dealing with objects allocated in one thread and modified in another, the library still has no control over what malloc and free do when free is called in another thread.

Some mallocs use queues to defer expensive inter-thread synchronisation, some mallocs may just keep the free memory in the other thread waiting for a subsequent malloc.

It’s surely out of scope for a library. If one wants to ensure correct behaviour one should queue the free back to the thread that owns the memory.

The test case would probably be a test for the malloc implementation assuming ref counting is working as it should be.

mc

[Vijay Maddula]

[Vijay Maddula]

Hi,

I have attached my sample code here. But i am not able to parse the data, may be the json data formate is not correct. I have given actual json data in text formate in that file. can you please help how to go with this.

Thanks

Vijay

[Vijay Maddula]

Hi Eric and Michael J Clark,

Thank you for you time.

I have attached a test-code(as you said) of my scenario here. As I have Observed Json is allocating memory for parsing, and after json_object_put It is not releasing memory to Heap  and from next parsing onwards it is not creating any extra memory and is using the memory which is allocated at first time. And when Process exiting it is clearing all the memory. While process is executing json is not releasing the memory.

I ran valgrind and it shows 0 memory leaks. Valgrind giving results after process exited.

So what i have to do to clear the total json allocated memory while executing a process.

I have observed memory usage using top and htop commands also.

Thanks

Vijay

[Vijay]

Sorry, Forget to attach the test-code file. Now i attached the file.

[Eric Haszlakiewicz]

On Fri, Aug 24, 2018, 6:46 AM Vijay Maddula <mvijayk...@gmail.com> wrote:

LpI have attached a test-code(as you said) of my scenario here. As I have Observed Json is allocating memory for parsing, and after json_object_put It is not releasing memory to Heap  and from next parsing onwards it is not creating any extra memory and is using the memory which is allocated at first time. And when Process exiting it is clearing all the memory. While process is executing json is not releasing the memory.

I ran valgrind and it shows 0 memory leaks. Valgrind giving results after process exited.

So what i have to do to clear the total json allocated memory while executing a process.

I have observed memory usage using top and htop commands also.

That's just how mallow usually works.  Unless you allocate and free large (many MB) chunks at a time, the malloc library will generally not release the memory back to the OS.  The json-c code doesn't do anything special when allocating and releasing memory, it just calls malloc and free, but it's working with small sizes on the order of tens to hundreds of bytes, so it would be hugely inefficient for the malloc code to map and unmap memory from the OS for each call.

If you have a usage pattern of allocate, free all, allocate, free all, etc... then whatever chunks of memory are initially requested will just be reused, and it's not a problem unless you see usage growing over time.

I'll take a peek at your code, but I suspect there's nothing wrong with it, nor with json-c.

Eric

[Vijay]

Hi Eric,

Thank you for your reply and time.

Malloc creates the memory and releases immediately whenever we call free(). That we can observe with top/htop/other tools. I know json is also calling free but in case of Json, after json_object_put() it is not releasing the memory(observed with top/htop). You can observe this with my code using top command. My requirement is whenever i done with json data, the memory should get free.

Thanks

Vijay

[Eric Haszlakiewicz]

On Fri, Aug 24, 2018 at 7:29 AM Vijay <mvijayk...@gmail.com> wrote:
> Thank you for your reply and time.
> Malloc creates the memory and releases immediately whenever we call free(). That we can observe with top/htop/other tools. I know json is also calling free but in case of Json, after json_object_put() it is not releasing the memory(observed with top/htop). You can observe this with my code using top command. My requirement is whenever i done with json data, the memory should get free.

That seems a bit unusual, but even if whatever malloc implementation
you're using happens to release the memory back to the OS for your
test, the fact that that doesn't happen with the way that json-c calls
malloc&free is not a bug. It is entirely normal behavior for a
process to keep the memory regions mapped for later use.
I see that your (very large!) test case just run through things once.
Put your code in a loop. If the memory usage *grows*, then _that_ is
indicative of a problem.

Also, you said that valgrind didn't show any errors, which indicates
that the json-c code is working fine.

However, I notice that your test case uses a path to the include files
("json/json.h") that was changed 6 years ago (see commit 30dd367)
instead of the current path of json-c/json.h. Are you using an
ancient version of json-c? If so, you should update.

Eric

> On Friday, August 24, 2018 at 4:48:09 PM UTC+5:30, Eric wrote:
>>
>>
>>
>> On Fri, Aug 24, 2018, 6:46 AM Vijay Maddula <mvijayk...@gmail.com> wrote:
>>>
>>> LpI have attached a test-code(as you said) of my scenario here. As I have Observed Json is allocating memory for parsing, and after json_object_put It is not releasing memory to Heap and from next parsing onwards it is not creating any extra memory and is using the memory which is allocated at first time. And when Process exiting it is clearing all the memory. While process is executing json is not releasing the memory.
>>> I ran valgrind and it shows 0 memory leaks. Valgrind giving results after process exited.
>>> So what i have to do to clear the total json allocated memory while executing a process.
>>> I have observed memory usage using top and htop commands also.
>>
>>
>> That's just how mallow usually works. Unless you allocate and free large (many MB) chunks at a time, the malloc library will generally not release the memory back to the OS. The json-c code doesn't do anything special when allocating and releasing memory, it just calls malloc and free, but it's working with small sizes on the order of tens to hundreds of bytes, so it would be hugely inefficient for the malloc code to map and unmap memory from the OS for each call.
>> If you have a usage pattern of allocate, free all, allocate, free all, etc... then whatever chunks of memory are initially requested will just be reused, and it's not a problem unless you see usage growing over time.
>>
>> I'll take a peek at your code, but I suspect there's nothing wrong with it, nor with json-c.
>>
>> Eric
>>

> --
> You received this message because you are subscribed to the Google Groups "json-c" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to json-c+un...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Vijay]

Hi Eric,

If I run my code in a loop it will not increase the memory, This case already tested and mentioned in this thread.

But in my case, once in a while i will only use huge amount of data. Remaining all the time it is very small amount of data. So my problem is whenever i get huge data, json is locking memory with its. So then remaining processes in the system will not run properly due to short ram. Json releasing data only when process exits but mine is continuous process. Thats the reason i want free the memory whenever i done with json data.

Thanks

Vijay

[Michael Clark]

On 27/08/2018, at 10:21 PM, Vijay <mvijayk...@gmail.com> wrote:

Hi Eric,

If I run my code in a loop it will not increase the memory, This case already tested and mentioned in this thread.

But in my case, once in a while i will only use huge amount of data. Remaining all the time it is very small amount of data. So my problem is whenever i get huge data, json is locking memory with its. So then remaining processes in the system will not run properly due to short ram. Json releasing data only when process exits but mine is continuous process. Thats the reason i want free the memory whenever i done with json data. 

I think this will be your malloc implementation not JSON.

[Vijay]

Hi Eric,

Even I think same like, my malloc implementation is not clearing the memory. But for that i have ran test cases and found it is not the malloc. You can observe the same in my test code like, allocate memory with malloc and free it. As per my observation I can say this is because of JSON and same thing i have found in some forums.

And one more thing is the file i attached second time don't have any malloc(if there also we can comment and see) calls, still it is increasing the memory after tokener_parse and not freeing with json_object_put. you can observe that by simply compiling and run that test code.

Thanks

Vijay

[Eric Haszlakiewicz]

AFAIK, none of the messages in the forums have been memory leaks in json-c itself, just mistakes on the part of the callers.

You're able to show a memory leak with the file 5.c (the one you sent most recently)?  Or 6.c?  Where does valgrind say the leak occurs?

I didn't realize you had sent two different files and only tried 5.c, which worked just fine for me.

Eric

--

[Vijay]

Hi Eric,

I didnt say json-c is leaking memory. I said how it is performing on the memory as my observation. And also i mentioned valgrind shows 0 errors. My problem is "json freezing memory with its until process exited".

ie, To parse 2mb of data, json-c is taking some "x"mb. After json_object_put  call the amount of data json taken is not available to the process. But the same "x"mb  is also using for next parsing (if it is in that boundary), means it is not increasing the memory for next parse. But my requirement is whenever i call json_object_put, it should clear all the memory ie, "x"mb of the memory, which is allocated by json.

And i am saying about the file 5.c. In that you can observe after json_object_put also the memory created by json parse is not getting free(observed with top and htop cammand).

Thanks

Vijay