[lwip-users] memory pools question
jblackarty
I faced with problem and founded that this issue was discussed
vigorously at "[lwip-users] about memory problem" in 2008 and still remained
opened !
On Thu, 17 Jul 2008 11:14:32 +0100 Kieran Mansley wrote:
> On Thu, 2008-07-17 at 17:56 +0800, yueyue papa wrote:
>> If onlye enable
>> */
>> #ifndef MEM_USE_POOLS
>> #define MEM_USE_POOLS 1
>> #endif
>>
>> /**
>> * MEMP_USE_CUSTOM_POOLS==1: whether to include a user file lwippools.h
>> * that defines additional pools beyond the "standard" ones required
>> * by lwIP. If you set this to 1, you must have lwippools.h in your
>> * inlude path somewhere.
>> */
>> #ifndef MEMP_USE_CUSTOM_POOLS
>> #define MEMP_USE_CUSTOM_POOLS 0
>> #endif
>
> I've had a look at the code, and that combination is not supported. You
> should see an error like this one when you try and compile it:
>
> ../../../../../lwip/src/core/init.c:156:4: #error "MEM_USE_POOLS
> requires custom pools (MEMP_USE_CUSTOM_POOLS) to be enabled in your
> lwipopts.h"
>
>> If i switch enable MEMP_USE_CUSTOM_POOLS , default windows build is ok.
>
> That is the right thing to do, and as it builds OK then it sounds like
> there's no error. You'll also need to write an lwippools.h if you've
> not already done so; see here for documentation:
>
> http://lwip.scribblewiki.com/Custom_memory_pools
>
> Kieran
WHY "MEM_USE_POOLS requires custom pools", rather than vice versa ???
It's not right thing to provide unnecessary user-defined pools to get standard
pools working. Please explain, or my brain will be broken.
P.S.: I think yueyue papa wanted to get answer exactly on this
question.
Regards,
Artem Pisarenko
_______________________________________________
lwip-users mailing list
lwip-...@nongnu.org
https://lists.nongnu.org/mailman/listinfo/lwip-users
jblackarty
It's totally unclear are mem and memp independent mechanisms, or mem
utilizes memp, or memp uses mem, or they are cross-references each
other ? Custom pools are part of pools (memp), or pools are part of
custom pools ?
There are complicated set of options:
MEM_LIBC_MALLOC,
MEMP_MEM_MALLOC,
MEM_USE_POOLS,
MEMP_USE_CUSTOM_POOLS,
MEMP_NUM_*,
MEM_SIZE,
PBUF_POOL_SIZE.
How they related to each other ?
For MEMP_MEM_MALLOC a comment says "Use mem_malloc/mem_free instead of
the lwip pool allocator.". But mem_malloc is part of pool mechanisms.
How it can be disabled ? Also "Especially useful with MEM_LIBC_MALLOC
..." confuses. Isn't libc allocator completely disables pool mechanism
? These options must exclude each other.
Is it possible disable allocating standard structs in pool at all ?
Does it makes sense configure MEMP_NUM_* options if MEM_LIBC_MALLOC
enabled ?
MEM_SIZE includes only pool of standart structures or also pbufs pool ?
Questions may be continued indefinitely. lwip source code doesn't
describe architecture because it's heavily stuffed with complex macro conditions
nested each in other. "Custom_memory_pools" article on wiki doesn't
help too because it doesn't describe whole memory mechanism completely.
I will appreciate if someone throw light upon this subject.
Simon Goldschmidt
> It's totally unclear are mem and memp independent mechanisms, or mem
> utilizes memp, or memp uses mem, or they are cross-references each
> other ? Custom pools are part of pools (memp), or pools are part of
> custom pools ?
>
> There are complicated set of options:
Yes, that's the nature of lwIP being adjustable to the needs of your (possibly small or otherwise restricted) target hardware.
I'll start explaining the 'standard' options. Basically, lwIP has 2 strategies for memory allocation: pools and a heap. The heap is mainly only used for outgoing PBUFs allocated with the type PBUF_RAM. Then there is one memory pool per structure type:
> MEMP_NUM_*,
These control the amount of structures of that type (e.g. MEMP_NUM_UDP_PCB sets the amount of available udp connections).
> MEM_SIZE,
This is the size of the heap (used for PBUF_RAM allocation).
> PBUF_POOL_SIZE.
As it says in opt.h (did you read that file???), the number of pbufs in the pbuf pool (where PBUF_POOL_BUFSIZE controls the size of these pbufs). The pbufs from the pool are of type PBUF_POOL.
Nnow to the more advanced options:
> MEM_LIBC_MALLOC,
Some people don't care about the danger of heap fragmentation and want to use the C library's malloc/free functions instead of the lwIP-internal memory allocation (if only to save code because the C library's functions are included in their binary, anyway). Set this to 1 to do so.
> MEMP_MEM_MALLOC,
This is mainly only usefule with the previous option: when set to 1, memp pool allocation uses the heap allocation functions (this was introduced to let memp allocations be using the C library's malloc.
> MEM_USE_POOLS,
This is the other way round than the previous: when you're scared of heap fragmentation, let mem_malloc() allocate from specific pools. The size of these pools (and the number of entries in them) needs to be configured *and* you have to enable the next option.
> MEMP_USE_CUSTOM_POOLS,
This enables custom memp pools. They are needed for the previous option but you can also define more pools, for example to put your semaphores or mboxes into.
> How they related to each other ?
> For MEMP_MEM_MALLOC a comment says "Use mem_malloc/mem_free instead of
> the lwip pool allocator.". But mem_malloc is part of pool mechanisms.
No, memp_malloc (not the 'p') allocates from pools, mem_malloc allocates from the heap!
> How it can be disabled ? Also "Especially useful with MEM_LIBC_MALLOC
> ..." confuses. Isn't libc allocator completely disables pool mechanism
> ? These options must exclude each other.
No, why should they???
> Is it possible disable allocating standard structs in pool at all ?
Yes, see above
> Does it makes sense configure MEMP_NUM_* options if MEM_LIBC_MALLOC
> enabled ?
No.
> MEM_SIZE includes only pool of standart structures or also pbufs pool ?
None of them. It controls the heap size and has nothing todo with pools.
> Questions may be continued indefinitely.
Is that so?
> lwip source code doesn't
> describe architecture because it's heavily stuffed with complex macro
> conditions
> nested each in other.
lwIP relies on people's contribution, so if there's anything missing for a beginner, please don't hesitate to contribute. Writing something on the wiki is the easiest part to help future beginners getting started faster.
> "Custom_memory_pools" article on wiki doesn't
> help too because it doesn't describe whole memory mechanism completely.
Right, it only describes how to enable custom memory pools. I guess it's rather targeted at people already using lwIP as a guideline of how to get custom pools running.
> I will appreciate if someone throw light upon this subject.
Hope I did. :-)
Simon
--
Empfehlen Sie GMX DSL Ihren Freunden und Bekannten und wir
belohnen Sie mit bis zu 50,- Euro! https://freundschaftswerbung.gmx.de
jblackarty
The only thing remained in a shadow is combination MEM_USE_POOLS +
MEMP_USE_CUSTOM_POOLS. As I understand MEMP_USE_CUSTOM_POOLS have only
one exclusive dependency on MEM_USE_POOLS. Also you said directly that
user have to enable second one if enables first one. Why not remove
this redundant option ? They have equal meaning. Or maybe it's
possible to have MEMP_USE_CUSTOM_POOLS enabled without MEM_USE_POOLS ?
This married couple has another issue I asked about in thread "System objects management (semaphores, mailboxes, mutexes, threads, etc.): pool size ?"
To use MEMP pool efficiently in sys_arch I need to know a kind of mailbox in
sys_mbox_new() rather then size. I cannot reverse-determine kind from
size, because sizes are not different generally. At the same time,
pool-based mem_malloc won't help in case when mailbox itself allocated
in RTOS memory and RTOS return just pointer (sys_mbox_t) of fixed
size. In such a way, I have to implement own independed pool mechanism included
in port for that RTOS.
>> MEMP_NUM_*,
>> MEM_SIZE,
>> PBUF_POOL_SIZE.
>> MEM_LIBC_MALLOC,
>> MEMP_MEM_MALLOC,
>> MEM_USE_POOLS,
>> MEMP_USE_CUSTOM_POOLS,
> No, why should they???
> Yes, see above
> No.
> Is that so?
> Hope I did. :-)
> Simon
Simon Goldschmidt
> The only thing remained in a shadow is combination MEM_USE_POOLS +
> MEMP_USE_CUSTOM_POOLS. As I understand MEMP_USE_CUSTOM_POOLS have only
> one exclusive dependency on MEM_USE_POOLS. Also you said directly that
> user have to enable second one if enables first one. Why not remove
> this redundant option ? They have equal meaning. Or maybe it's
> possible to have MEMP_USE_CUSTOM_POOLS enabled without MEM_USE_POOLS ?
Exactly. For example, your port can create custom pools to put semaphores or mboxes in it and allocate from that pool when calling sys_sem_new().
> This married couple has another issue I asked about in thread "System
> objects management (semaphores, mailboxes, mutexes, threads, etc.): pool size
> ?"
> To use MEMP pool efficiently in sys_arch I need to know a kind of mailbox
> in
> sys_mbox_new() rather then size. I cannot reverse-determine kind from
> size, because sizes are not different generally. At the same time,
> pool-based mem_malloc won't help in case when mailbox itself allocated
> in RTOS memory and RTOS return just pointer (sys_mbox_t) of fixed
> size. In such a way, I have to implement own independed pool mechanism
> included
> in port for that RTOS.
I don't think I have thought that through fully, but mboxes are only created in 5 different sizes (at maximum), so it shouldn't be too hard to switch over mbox size to know the pool to allocate from?
Simon
--
Empfehlen Sie GMX DSL Ihren Freunden und Bekannten und wir
belohnen Sie mit bis zu 50,- Euro! https://freundschaftswerbung.gmx.de
_______________________________________________
jblackarty
> only created in 5 different sizes (at maximum), so it shouldn't be
> too hard to switch over mbox size to know the pool to allocate from?
No! In general case, sizes are NOT different. At compile-time macro
expansions define each size, but at run-time sys_mbox_new() cannot match given
numeric value to one of macros if their values are equal. That's the
problem ! I suggest define enumeration of mbox types and change
interface of sys_mbox_new() to pass that type instead size. This
solution will be more flexible in order to enable engineers make ports
with efficient memory usage. Of course, if not to mention memory
waste, then I can allocate all mboxes having size of the biggest one and
place them in one common pool and do not worry about nothing.
Moreover, I found that default values for all mboxes sizes are zero in
opt.h. Does it mean that user must define them in lwipopts.h ? But user
generally don't know workflows in lwip such precisely in order to
determine optimal values.
Simon Goldschmidt
> No! In general case, sizes are NOT different. At compile-time macro
> expansions define each size, but at run-time sys_mbox_new() cannot match given
> numeric value to one of macros if their values are equal. That's the
> problem !
What's the problem there? You can just use one pool then. The number of mboxes allocated shouldn't be limited by the sys port: instead, you should have enough mboxes to allocate all raw, top and UDP connections. Limiting the number of these is the expected way to limit memory usage.
> I suggest define enumeration of mbox types and change
> interface of sys_mbox_new() to pass that type instead size. This
> solution will be more flexible
Yes, that would be more flexible, but I don't see a pressing need to change the Port API yet another time.
Simon
jblackarty
>> expansions define each size, but at run-time sys_mbox_new() cannot match given
>> numeric value to one of macros if their values are equal. That's the
>> problem !
> What's the problem there? You can just use one pool then. The
> number of mboxes allocated shouldn't be limited by the sys port:
> instead, you should have enough mboxes to allocate all raw, top and
> UDP connections. Limiting the number of these is the expected way to limit memory usage.
Ok. I give up. I just tried to explain that I have to implement own
memory pool mechanism since lwip's one can't be used.