Is Jansson for small embedded platform?

[pozz]

Dear group,

I'm learning more about JSON data format and I'm studying the
possibility to use it in my embedded applications for storing
configuration in a non-volatile memory.

Because I work on an embedded platform with small resources, I need a
C implementation of JSON with very small requirements. But the most
important thing is that I don't have malloc()/free() facilities, but
only static variables?

Could the Jansson library good for me?

[Akos Polster]

Maybe JSON is overkill for such a small system. The dynamic nature of
the format is the main reason why dynamic memory management is
required by implementations, like Jansson.

~ Akos.

--
Akos Polster
http://tftfy.blogspot.com/

> --
> Jansson users mailing list
> jansso...@googlegroups.com
> http://groups.google.com/group/jansson-users

[Petri Lehtinen]

Jansson relies heavily on malloc() and free() to manage its memory. It
uses very little stack space or static variables, almost everything
happens in heap.

It could be possible to overcome this by using custom memory
allocation functions, see

http://www.digip.org/jansson/doc/2.2/apiref.html#custom-memory-allocation

For example, you could reserve a static buffer and return chunks of
memory from that buffer. It would possibly be quite hard to get it
right, though, and you'd have to deal with fragmentation, etc.

Another solution would be to use some allocator library for embedded
systems. See the link above for examples on how to use a custom
allocator library.

Petri

[pozz]

On 6 Set, 14:16, Akos Polster <a...@pipacs.com> wrote:
> Maybe JSON is overkill for such a small system. The dynamic nature of
> the format is the main reason why dynamic memory management is
> required by implementations, like Jansson.

Thank you for your answer. Evidently I have to look at other
configuration file
format or serialization library. Do you have any suggestion? What do
you think
about BSON, MessagePack or similar?

Usually my embedded application uses static structure to store
configuration parameters
in RAM, but it needs to save these parameters in non-volatile memory.

Until now, I used to copy the in-memory image of the structure
(including padding bytes)
to the non-volatile memory, simply using a byte-to-byte copy. But
this is very uncomfortable
when the configuration file format changes with a new software
version. After the
upgrade, the new version should be able to read the old configuration
and transform it
to new layout. So I was thinking about a more flexible and dynamic
data file format.

[pozz]

On 6 Set, 14:37, Petri Lehtinen <pe...@digip.org> wrote:
> pozz wrote:
> > Dear group,
>
> > I'm learning more about JSON data format and I'm studying the
> > possibility to use it in my embedded applications for storing
> > configuration in a non-volatile memory.
>
> > Because I work on an embedded platform with small resources, I need a
> > C implementation of JSON with very small requirements. But the most
> > important thing is that I don't have malloc()/free() facilities, but
> > only static variables?
>
> > Could the Jansson library good for me?
>
> Jansson relies heavily on malloc() and free() to manage its memory. It
> uses very little stack space or static variables, almost everything
> happens in heap.
>
> It could be possible to overcome this by using custom memory
> allocation functions, see
>

>  http://www.digip.org/jansson/doc/2.2/apiref.html#custom-memory-alloca...

>
> For example, you could reserve a static buffer and return chunks of
> memory from that buffer. It would possibly be quite hard to get it
> right, though, and you'd have to deal with fragmentation, etc.
>
> Another solution would be to use some allocator library for embedded
> systems. See the link above for examples on how to use a custom
> allocator library.

I already tried to implement my own dynamic allocator and I won't try
again the future :-)

[Max Shaffer]

Pozz,
I am using Jansson with success for exactly the type of application
you described. My processor is an ARM Cortex M3 with 64k RAM. My heap
is 16k, which is probably overkill. You need to careful about loading
JSON objects during runtime, but the new dump callback functionality
in Jansson-2.2 is perfect for dumping objects to static buffers in
order to avoid large dynamic allocations. If you only load objects
during power-on, you can get away with a "normal" sized heap.

As far as storing objects in non-volatile memory, you need to encode
Jansson objects back into JSON strings before saving them. You cannot
simply save out Jansson objects as they include function pointers and
other non-data members.

Hope this helps!
~Max

Sent from my private jet

[pozz]

On 6 Set, 15:57, Max Shaffer <max.shaf...@gmail.com> wrote:
> Pozz,
> I am using Jansson with success for exactly the type of application
> you described.

Bingo! :-)

> My processor is an ARM Cortex M3 with 64k RAM. My heap
> is 16k, which is probably overkill.

:-(
My platform is based on a 16-bits microcontroller with 256KB of Flash
memory, 8KB of serial I2C EEPROM and 6KB of RAM!!!

> You need to careful about loading JSON objects during runtime, but the
> new dump callback functionality
> in Jansson-2.2 is perfect for dumping objects to static buffers in
> order to avoid large dynamic allocations.

Can you make some example? I'm new to Jansson so I don't know how it
works well.
Does your platform have malloc()/free()?

> If you only load objects
> during power-on, you can get away with a "normal" sized heap.

Can you explain better this point?

> As far as storing objects in non-volatile memory, you need to encode
> Jansson objects back into JSON strings before saving them. You cannot
> simply save out Jansson objects as they include function pointers and
> other non-data members.

If you can write down some simple example, it'll be of great help form
me,
thank you.

[Max Shaffer]

Hey Pozz,

Here is a simple example, which should help you evaluate Jansson for your needs.

As Petri noted earlier, Jansson relies heavily on dynamic memory allocation, so you will need to define a heap. For normal operations such as getting and setting values (json_object_get and json_object_set), the allocations are generally small, so the heap size needed for these sort of methods does not need to be very large. The space required to hold your json data is also allocated on the heap, but as long as your object size is more or less constant, you can simply increase your heap size instead of allocating a static buffer for the data. For this reason, importing JSON data should only be done during initialization, unless you are sure that you have enough space on the heap for your data.

~Max

--

[pozz]

On 6 Set, 21:31, Max Shaffer <max.shaf...@gmail.com> wrote:
> Hey Pozz,
>
> Here is a simple example, which should help you evaluate Jansson for your
> needs.

Thank you Max for spending your time to help me.

> As Petri noted earlier, Jansson relies heavily on dynamic memory allocation,
> so you will need to define a heap.

What do you mean with heap? Are you thinking about a static allocated
array of
unsigned chars and a couple of custom malloc/free functions that use
this array?
Do you have an example of this kind of malloc/free? Are you care about
fragmentation
and similar?

> For normal operations such as getting and
> setting values (json_object_get and json_object_set), the allocations are
> generally small, so the heap size needed for these sort of methods does not
> need to be very large. The space required to hold your json data is also
> allocated on the heap, but as long as your object size is more or less
> constant, you can simply increase your heap size instead of allocating a
> static buffer for the data. For this reason, importing JSON data should only
> be done during initialization, unless you are sure that you have enough
> space on the heap for your data.

I have two different requirements.

First, I often save big tables in the configuration
file, but only a small section of them is loaded in RAM (depending on
other
parameters). With your example, the overall JSON object is loaded in
RAM
with json_loads(). Is there a way to read the JSON object step by
step, like
an incoming stream, and decide on the fly to copy or to discard
parameters?

Second, many parts of the configuration are not loaded at startup, but
during
run-time. Do you remember the tables above? If the user changes the
configuration
at run-time, it could be needed to load a different part of a table
from the
saved configuration.

Do you think Jansson and JSON approach could be used with my
requirements?

[Petri Lehtinen]

pozz wrote:
> First, I often save big tables in the configuration file, but only a
> small section of them is loaded in RAM (depending on other
> parameters). With your example, the overall JSON object is loaded in
> RAM with json_loads(). Is there a way to read the JSON object step
> by step, like an incoming stream, and decide on the fly to copy or
> to discard parameters?

Jansson's JSON decoder always parses the whole input to memory, so you
cannot read step-by-step. "yajl" is a widely used streaming JSON
library, see http://lloyd.github.com/yajl/

[PARESY Antoine]

Hi all

I'm trying to compile jansson for an embedded device (Renesas RX63N) in a RTOS environment.

I'm using Renesas RX compiler in v.1.02.00 and E2Studio IDE.

I include all source files to my project but I got this error at link and can't find the origin of it ...

** L2310 (E) Undefined external symbol "_abort" referenced in ".\JSON\dump.obj"

** L2310 (E) Undefined external symbol "_abort" referenced in ".\JSON\load.obj"

** L2310 (E) Undefined external symbol "_abort" referenced in ".\JSON\strconv.obj"

Do you have an idea ? I can't find any reference of this "abort" in the code...

Thanks in advance for your help

[Petri Lehtinen]

PARESY Antoine wrote:
> Hi all
>
> I'm trying to compile jansson for an embedded device (Renesas RX63N) in a RTOS
> environment.
> I'm using Renesas RX compiler in v.1.02.00 and E2Studio IDE.
>
> I include all source files to my project but I got this error at link and can't
> find the origin of it ...
> ** L2310 (E) Undefined external symbol "_abort" referenced in ".\JSON\dump.obj"
> ** L2310 (E) Undefined external symbol "_abort" referenced in ".\JSON\load.obj"
> ** L2310 (E) Undefined external symbol "_abort" referenced in ".\JSON\
> strconv.obj"
>
> Do you have an idea ? I can't find any reference of this "abort" in the code...
>
> Thanks in advance for your help

It's probably caused by the assert() calls. They're most easily
removed by defining a macro called NDEBUG before including assert.h,
e.g. by using the -DNDEBUG compiler switch (works on gcc, not sure
about the Renesas compiler).

You can also safely remove the asserts from the code.

Petri

[Rasmus Fink]

-or make a function next to main() called abort(). That function can be empty or include your own favorite debugging info.

Havn't tried it myself but it sounds good :-)

/Race

>--

>--
>Jansson users mailing list
>jansso...@googlegroups.com
>http://groups.google.com/group/jansson-users

>---
>You received this message because you are subscribed to the Google Groups "Jansson users" group.
>To unsubscribe from this group and stop receiving emails from it, send an email to jansson-user...@googlegroups.com.
>For more options, visit https://groups.google.com/groups/opt_out.

[PARESY Antoine]

Yes ! Thanks for your help. 

That's what I saw here too : http://members.boardhost.com/hew_forum/msg/896.html

Compilation OK, let's try to play now !

[PARESY Antoine]

Hi !

It's me again, since the library is working now, I would like to adapt some parts of the code.

The reason is :

Since today, I tried with strings only but I would like to work with files.

The problem is that I am using a File System dedicated to emebedded system (Segger) on a SD Card.

I don't have the standard FILE format.

I had a look on json_loadf function but can't undersant it.

The solution should be in json_loadf and json_dumpf function but no idea of how lex_init function works ...

Can I have a tip to go ahead with this problem ?

Thanks

Best Regards

[Petri Lehtinen]

You could try json_load_callback(). It takes a function pointer as an
argument, and calls that function when it needs more data. That way
you don't need to have a standard FILE pointer.

Petri

[PARESY Antoine]

Thanks ! I made a small test with this call back :

size_t test_callback(void *buffer, size_t buflen, void *data)

{

unsigned long nb_read;

nb_read = FS_Read(buffer,buflen,data); //data = FS_FILE pointer from json_load_callback arg

return nb_read;

}

And it's working :) I will perform more advanced test but it's a good start