No dynamic memory at all: newlib and sprintf

[pozz]

I'd like to avoid using dynamic memory (heap) at all. I'm using gcc for
ARM and newlib (really I'm using NXP MCUXpresso IDE, but I don't think
this is important).

When the code is minimal, I noticed heap related functions (malloc, free
and so on) aren't included in the output binary file.
However, as soon as I use sprintf-like functions, heap is used.

I looked at the newlib source code and indeed I noticed __ssputs_r(), a
common shared function between sprintf-like functions, that uses malloc.
I think it uses malloc for asprintf-like functions that I'm not using.

How to remove completely the heap with sprintf functions? Maybe by
implementing a custom minimal sprintf function?

[David Brown]

Check how __ssputs_r is using the heap. Maybe you can get away with
just adding these to your code somewhere:

static char heap[100];
void * malloc(size_t size) {
(void) size;
return heap;
}

void free(void *ptr) {
(void) ptr;
}

If the __ssputs_r function only ever calls malloc once while it works,
and you don't use it from different threads or contexts (the "_r" in
newlib is for reentrant code), that might do the job, while saving you
from the effort of making your own printf stuff.

[pozz]

IMHO, I will have an error from the linker, because of multiple
definition of malloc and free.

[pozz]

Il 30/08/2019 09:57, David Brown ha scritto:

> [...]while saving you

> from the effort of making your own printf stuff.

I found this:
https://github.com/mpaland/printf

[Paul Rubin]

pozz <pozz...@gmail.com> writes:
> How to remove completely the heap with sprintf functions? Maybe by
> implementing a custom minimal sprintf function?

Use snprintf instead of sprintf on general principles, but both might
allocate some temp space for conversions. All I can say is see if you
can check the source code for the implementation. You could put in a
fake malloc that crashes immediately, and see if sprintf actually calls
it with the data that your program passes it. If it doesn't, that's not
a guarantee, but it is a point of information.

[David Brown]

No, you won't.

The linker first loads the object files you have specified for linking.
It resolves all the symbols there when it can, and collects the
unresolved ones. Then it pulls in the libraries, and uses them to
resolved those symbols. That can lead to new unresolved symbols, which
are again pulled from the libraries (and so on).

Symbols defined in your object code - or in a library that is loaded
before newlib - will get priority.

It is not uncommon to have your own stub definitions of library
functions like "exit" or "atexit" in order to avoid extra code being
pulled in from libraries when you know your code will never call them.

[pozz]

I didn't know this, yes it works.

However I tried to replace sprintf() with a custom sprintf(), but in
this case the linker complains with:

multiple definition of `sprintf'

It sees two definitions in my printf.c and in libc_nano.a. Could you
argue why this happens with sprintf() and not with malloc()?

[Dave Nadler]

Pozz, you may want to have a look here:
http://www.nadler.com/embedded/newlibAndFreeRTOS.html
Hope that helps!
Best Regards, Dave

[pozz]

Il 30/08/2019 14:21, pozz ha scritto:
> Il 30/08/2019 10:48, David Brown ha scritto:

>[...]

> However I tried to replace sprintf() with a custom sprintf(), but in
> this case the linker complains with:
>
>   multiple definition of `sprintf'
>
> It sees two definitions in my printf.c and in libc_nano.a. Could you
> argue why this happens with sprintf() and not with malloc()?

After seen that -O1 linker option solves this problem, I found the option

--allow-multiple-definition

that works even with -O0, that I use in debug build.

[David Brown]

On 30/08/2019 15:39, pozz wrote:
> Il 30/08/2019 14:21, pozz ha scritto:
>> Il 30/08/2019 10:48, David Brown ha scritto:
> >[...]
>> However I tried to replace sprintf() with a custom sprintf(), but in
>> this case the linker complains with:
>>
>>    multiple definition of `sprintf'
>>
>> It sees two definitions in my printf.c and in libc_nano.a. Could you
>> argue why this happens with sprintf() and not with malloc()?

Are you sure? Have you checked the map file (and given the linker
settings for showing detailed information in the map file) ?

>
> After seen that -O1 linker option solves this problem, I found the option
>
>   --allow-multiple-definition
>
> that works even with -O0, that I use in debug build.
>

It seems risky to allow multiple definitions. It's not an option I have
used - I'd need to look more deeply into documentation before I can be
more helpful.

I never recommend using -O0, for any purpose. Code at -O1 (on gcc at
least) is much easier for debugging, with far clearer assembly. More
importantly, it enables a lot more analysis on the compiler and
therefore gives far more accurate static warnings.

I usually compile at approximately -O2 (by "approximately", I mean I
often have some additional specific flags added). It can make
single-step debugging difficult because it re-arranges code a lot, so
occasionally I will go down to -O1 for a file if I need to use a lot of
single-step or assembly level debugging on it.

I am strongly against the whole concept of "debug build" and "release
build" if it encourages you to use different optimisation levels and
settings. Enabling or disabling verbose outputs for different bits of
the code is okay.

[pozz]

Il 01/09/2019 22:47, David Brown ha scritto:
> On 30/08/2019 15:39, pozz wrote:
>> Il 30/08/2019 14:21, pozz ha scritto:
>>> Il 30/08/2019 10:48, David Brown ha scritto:
>>  >[...]
>>> However I tried to replace sprintf() with a custom sprintf(), but in
>>> this case the linker complains with:
>>>
>>>    multiple definition of `sprintf'
>>>
>>> It sees two definitions in my printf.c and in libc_nano.a. Could you
>>> argue why this happens with sprintf() and not with malloc()?
>
> Are you sure?  Have you checked the map file (and given the linker
> settings for showing detailed information in the map file) ?
>
>>
>> After seen that -O1 linker option solves this problem, I found the option
>>
>>    --allow-multiple-definition
>>
>> that works even with -O0, that I use in debug build.
>>
>
> It seems risky to allow multiple definitions.  It's not an option I have
> used - I'd need to look more deeply into documentation before I can be
> more helpful.

You're right, but it is the only solution I found to replace standard
pritnf with a custom implementation.

> I never recommend using -O0, for any purpose.  Code at -O1 (on gcc at
> least) is much easier for debugging, with far clearer assembly.  More
> importantly, it enables a lot more analysis on the compiler and
> therefore gives far more accurate static warnings.

much easier... far clearer... a lot more analysis... far more
accurate... than what? Are you comparing -O1 with -O0?

> I usually compile at approximately -O2

for production/release build? For this I use -Os, because many times I
have small Flash memory.

> (by "approximately", I mean I
> often have some additional specific flags added).  It can make
> single-step debugging difficult because it re-arranges code a lot, so
> occasionally I will go down to -O1 for a file if I need to use a lot of
> single-step or assembly level debugging on it.
>
> I am strongly against the whole concept of "debug build" and "release
> build" if it encourages you to use different optimisation levels and
> settings.

Why? Debug buil is used for... debug, usually during developing. It's
much easier to debug with -O0 than -O2 or -O1.

> Enabling or disabling verbose outputs for different bits of
> the code is okay.

Yes, this is another goal of debug build (enabling a serial port for
some outputs). And I usually disable watchdog in debug buid, because
watchdog doesn't work well during debugging.

[David Brown]

Yes. Compare the generated assembly codes, and see which optimisation
gives you code that is easiest to follow. (The "more analysis" and
"more accurate warnings" bits I hope you agree with.)

>
>
>> I usually compile at approximately -O2
>
> for production/release build? For this I use -Os, because many times I
> have small Flash memory.

I find that often, -Os makes little difference in code size compared to
-O2. Sometimes it is even smaller. But that does vary a bit between
targets, and also according to the type of code you write.

And remember that the size of the flash and the size of the program is
not the important issue. What is important is that the program fits in
the flash, perhaps with some space for future changes to the code. If
you have a little microcontroller with 8K flash, then a 7K program that
runs 1% faster, can sleep 1% longer, and therefore has 1% more battery
life, is /better/ than a 3K program that is 1% slower.

>
>
>> (by "approximately", I mean I often have some additional specific
>> flags added).  It can make single-step debugging difficult because it
>> re-arranges code a lot, so occasionally I will go down to -O1 for a
>> file if I need to use a lot of single-step or assembly level debugging
>> on it.
>>
>> I am strongly against the whole concept of "debug build" and "release
>> build" if it encourages you to use different optimisation levels and
>> settings. 
>
> Why? Debug buil is used for... debug, usually during developing. It's
> much easier to debug with -O0 than -O2 or -O1.
>

No, it is not. Single-stepping through -O2 can be confusing as code
jumps around a bit, but -O1 is fine for that. And if you are doing a
lot of single-stepping debugging, you might want to think about your
development process to see if it can be made more efficient. (There is
never one right answer, and I don't want to give that impression - in
this business, there is always a great amount of variation. I can only
talk about generalities, and my own experiences.)

With -O0 compiles, gcc puts (almost) everything on the stack. The
assembly code for even simple operations is horrible - there are many
times as many instructions as needed. This makes it very difficult to
follow what is being done. And it is totally unrealistic in terms of
performance and code size from "real" complied code.

-O1 compiles generate code that is simpler and neater. -O2 has similar
code generation in many cases, but does far more re-arrangement and
re-ordering, which can make debugging difficult.


A big problem with -O0 for "debug builds" that I have seen, is people
get the wrong impression of their code. They write code that looks okay
to them, but is in fact bad - they have misunderstood pointer aliasing,
missed out essential "volatile", relied on integer overflow wraps, and
all sorts of other mistakes. They do all their testing and debugging
with -O0. Then for the "release" build, they have -O1 or -O2 (or -Os),
their asserts and run-time checks are disabled, and their test code is
removed - they compile and use their "working" and "tested" code. And
then things go subtly wrong in ways they don't understand or expect,
since the code is "tested". It is classic "it worked when I tested it"
code, and is the cause of a lot of problems people have.

Remember - test what your ship, ship what you test. Splitting "debug"
and "release" builds just doubles your workload for all your testing and
debugging.

>
>> Enabling or disabling verbose outputs for different bits of the code
>> is okay.
>
> Yes, this is another goal of debug build (enabling a serial port for
> some outputs). And I usually disable watchdog in debug buid, because
> watchdog doesn't work well during debugging.
>

Watchdogs don't work well in release either - at least, not for what
many people seem to think they should do. But they cause more problems
during debugging.