String class operations: bug in class or in realloc?

[TK]

Following test code bases on a small change in WString.cpp/.h (see https://groups.google.com/a/arduino.cc/forum/#!topic/developers/ryrMAJiqwf0, setReserverPercentage() added, see my github fork of the Arduino master at https://github.com/TobiasKnauss/Arduino), but this should not be the cause for the strange behaviour I have seen. I haven't found out yet where it comes from. I suppose either a bug in the String class or in realloc():

#include <MemoryFree.h>
#include <Streaming.h>

String m_sText1;
String m_sText2;

void setup()
{
  Serial.begin (9600);
  m_sText1.setReservePercentage(100);
  Serial << F("cap=") << m_sText1.capacity << F(" len=") << m_sText1.len << F(" addrS=0x") << _HEX((long)&m_sText1) << F(" addrBuf=0x") << _HEX((long)m_sText1.buffer) << endl;
  m_sText2.setReservePercentage(100);
  Serial << F("cap=") << m_sText2.capacity << F(" len=") << m_sText2.len << F(" addrS=0x") << _HEX((long)&m_sText2) << F(" addrBuf=0x") << _HEX((long)m_sText2.buffer) << endl;
  Serial << endl;
}

void loop()
{
  String sText ("a");
  Serial << F("cap=") << sText.capacity << F(" len=") << sText.len << F(" addrS=0x") << _HEX((long)&sText) << F(" addrBuf=0x") << _HEX((long)sText.buffer) << endl;
 
  m_sText1 += "a";
  Serial << F("cap=") << m_sText1.capacity << F(" len=") << m_sText1.len << F(" addrS=0x") << _HEX((long)&m_sText1) << F(" addrBuf=0x") << _HEX((long)m_sText1.buffer) << F(" ") << m_sText1 << endl;

  m_sText2 += "b";
  Serial << F("cap=") << m_sText2.capacity << F(" len=") << m_sText2.len << F(" addrS=0x") << _HEX((long)&m_sText2) << F(" addrBuf=0x") << _HEX((long)m_sText2.buffer) << F(" ") << m_sText2  << endl;
 
  Serial << F("mem @ loop end:   ") << freeMemory() << endl << endl;
 
  delay (100);                                                 
}

############################## output ##############################

cap=0 len=0 addrS=0x133 addrBuf=0x1E7
cap=0 len=0 addrS=0x12C addrBuf=0x1EB

cap=1 len=1 addrS=0x8DF addrBuf=0x1EF
cap=1 len=1 addrS=0x133 addrBuf=0x1E7 a
cap=1 len=1 addrS=0x12C addrBuf=0x1EB b
mem @ loop end:   1759

cap=1 len=1 addrS=0x8DF addrBuf=0x1EF       << Question 1: 0x1E7 is free now. Why not created there?
cap=2 len=2 addrS=0x133 addrBuf=0x1F3 aa
cap=2 len=2 addrS=0x12C addrBuf=0x1F8 bb
mem @ loop end:   1757

cap=1 len=1 addrS=0x8DF addrBuf=0x1EF
cap=4 len=3 addrS=0x133 addrBuf=0x1E7 aaa   << m_sText1 goes back to 0x1E7.
cap=4 len=3 addrS=0x12C addrBuf=0x1F8 bbb
mem @ loop end:   1752

cap=1 len=1 addrS=0x8DF addrBuf=0x1F4
cap=4 len=4 addrS=0x133 addrBuf=0x1E7 aaaa
cap=4 len=4 addrS=0x12C addrBuf=0x1F8 bbbb
mem @ loop end:   1752

cap=1 len=1 addrS=0x8DF addrBuf=0x1F4
cap=8 len=5 addrS=0x133 addrBuf=0x1E7 aaaaa
cap=8 len=5 addrS=0x12C addrBuf=0x1F8 bbbbb
mem @ loop end:   1743

...

cap=1 len=1 addrS=0x8DF addrBuf=0x1F4
cap=16 len=9 addrS=0x133 addrBuf=0x203 aaaaaaaaa
cap=16 len=9 addrS=0x12C addrBuf=0x216 bbbbbbbbb
mem @ loop end:   1729

cap=1 len=1 addrS=0x8DF addrBuf=0x1FF       << still Question 1: address rises further. Why?
cap=16 len=10 addrS=0x133 addrBuf=0x203 aaaaaaaaaa
cap=16 len=10 addrS=0x12C addrBuf=0x216 bbbbbbbbbb
mem @ loop end:   1729

...

cap=1 len=1 addrS=0x8DF addrBuf=0x1FF
cap=32 len=17 addrS=0x133 addrBuf=0x229 aaaaaaaaaaaaaaaaa
cap=32 len=17 addrS=0x12C addrBuf=0x24C bbbbbbbbbbbbbbbbb
mem @ loop end:   1697

cap=1 len=1 addrS=0x8DF addrBuf=0x225
cap=32 len=18 addrS=0x133 addrBuf=0x229 aaaaaaaaaaaaaaaaaa
cap=32 len=18 addrS=0x12C addrBuf=0x24C bbbbbbbbbbbbbbbbbb
mem @ loop end:   1697

...

final faulty behaviour:
Question 2: why does this occur? m_sText2 stays at 256 for one time, but is not cleared. If realloc failed (no memory block of required size available), then it should have been erased by invalidate()
And why is the text cleared later?

cap=1 len=1 addrS=0x8DF addrBuf=0x331
cap=256 len=256 addrS=0x133 addrBuf=0x335 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
cap=256 len=256 addrS=0x12C addrBuf=0x438 bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
mem @ loop end:   1249

cap=1 len=1 addrS=0x8DF addrBuf=0x331
cap=512 len=257 addrS=0x133 addrBuf=0x53B aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
cap=256 len=256 addrS=0x12C addrBuf=0x438 bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
mem @ loop end:   993

cap=1 len=1 addrS=0x8DF addrBuf=0x434
cap=512 len=258 addrS=0x133 addrBuf=0x53B aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
cap=512 len=257 addrS=0x12C addrBuf=0x231
mem @ loop end:   737

cap=1 len=1 addrS=0x8DF addrBuf=0x22D
cap=512 len=259 addrS=0x133 addrBuf=0x53B aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
cap=512 len=258 addrS=0x12C addrBuf=0x231 bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
mem @ loop end:   737

cap=1 len=1 addrS=0x8DF addrBuf=0x22D
cap=512 len=260 addrS=0x133 addrBuf=0x53B aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
cap=512 len=259 addrS=0x12C addrBuf=0x231 bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
mem @ loop end:   737

cap=1 len=1 addrS=0x8DF addrBuf=0x22D
cap=512 len=261 addrS=0x133 addrBuf=0x53B
cap=512 len=260 addrS=0x12C addrBuf=0x231 bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
mem @ loop end:   737

cap=1 len=1 addrS=0x8DF addrBuf=0x22D
cap=512 len=262 addrS=0x133 addrBuf=0x53B
cap=512 len=261 addrS=0x12C addrBuf=0x231
mem @ loop end:   737

... text missing from here on

[Andrew Kroll]

Better to use an avr dragon to debug things. Give me an example that is short as possible and i'll look at it w/mine.

--
You received this message because you are subscribed to the Google Groups "Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to developers+...@arduino.cc.

[TK]

Am Samstag, 23. April 2016 00:10:38 UTC+2 schrieb Andrew Kroll:

Better to use an avr dragon to debug things. Give me an example that is short as possible and i'll look at it w/mine.

I agree. Debugging on the Arduino boards is almost impossible.
This is the "shortest" example I have that produces the bug. If you want less loops and less output, you could use  += "aaaaaaaa" instead of  += "a" (and same for "b"), which will create the same failure.
The WString.cpp/.h can be taken from my github Arduino branch.

[Matthijs Kooijman]

Hi TK,

I've looked over your code and output, but I can't find anything wrong
with it. I suspect that the problem is in your expectations, not the
actual behaviour. I'll expand on that below.

> cap=1 len=1 addrS=0x8DF addrBuf=0x1EF
> cap=1 len=1 addrS=0x133 addrBuf=0x1E7 a
> cap=1 len=1 addrS=0x12C addrBuf=0x1EB b
> mem @ loop end: 1759
>
> cap=1 len=1 addrS=0x8DF addrBuf=0x1EF << Question 1: 0x1E7 is free now. Why not created there?
> cap=2 len=2 addrS=0x133 addrBuf=0x1F3 aa
> cap=2 len=2 addrS=0x12C addrBuf=0x1F8 bb
> mem @ loop end: 1757

0x1E7 is used for m_sText1 at that point. Only when m_sText1 += "a"
runs, will m_sText1 be reallocated. In your code, you print the first
line before appending the "a", so at that point, 0x1E7 is still used.

I believe this also explains the other questions you put in the first
section.

> final faulty behaviour:
> Question 2: why does this occur? m_sText2 stays at 256 for one time, but is
> not cleared. If realloc failed (no memory block of required size
> available), then it should have been erased by invalidate()
> And why is the text cleared later?
>
> cap=1 len=1 addrS=0x8DF addrBuf=0x331
> cap=256 len=256 addrS=0x133 addrBuf=0x335
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
> cap=256 len=256 addrS=0x12C addrBuf=0x438
> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
> mem @ loop end: 1249
>
> cap=1 len=1 addrS=0x8DF addrBuf=0x331
> cap=512 len=257 addrS=0x133 addrBuf=0x53B
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
> cap=256 len=256 addrS=0x12C addrBuf=0x438
> bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
> mem @ loop end: 993
>
> cap=1 len=1 addrS=0x8DF addrBuf=0x434
> cap=512 len=258 addrS=0x133 addrBuf=0x53B
> aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
> cap=512 len=257 addrS=0x12C addrBuf=0x231
> mem @ loop end: 737

You speak of "invalidate", but where is that called according to you?
From what I can see, if a realloc fails, the += operation just ends up
as a no-op. See:

https://github.com/arduino/Arduino/blob/master/hardware/arduino/avr/cores/arduino/WString.cpp#L155-L164
https://github.com/arduino/Arduino/blob/master/hardware/arduino/avr/cores/arduino/WString.cpp#L268

This is consistent with the above: the capacity of m_sText2 stays at
256, as well as its contents. The reason that the allocation fails once,
but not later, is that the sText variable splits the available memory
in two. If sText would not be present, m_sText2 could expand downwards
to grow to 512 bytes, into the area just vacated by m_sText1. In the
next iteration, sText is allocated in a different place, so expanding
m_sText2 is possible then.

Gr.

Matthijs

[TK]

Hi Matthijs,
thanks for you answer and your work on this.
Sorry that I didn't have time to come back to this earlier.

Am Montag, 25. April 2016 11:01:25 UTC+2 schrieb Matthijs Kooijman:

Hi TK,

I've looked over your code and output, but I can't find anything wrong
with it. I suspect that the problem is in your expectations, not the
actual behaviour. I'll expand on that below.

> cap=1 len=1 addrS=0x8DF addrBuf=0x1EF
> cap=1 len=1 addrS=0x133 addrBuf=0x1E7 a
> cap=1 len=1 addrS=0x12C addrBuf=0x1EB b
> mem @ loop end:   1759
>
> cap=1 len=1 addrS=0x8DF addrBuf=0x1EF       << Question 1: 0x1E7 is free now. Why not created there?
> cap=2 len=2 addrS=0x133 addrBuf=0x1F3 aa
> cap=2 len=2 addrS=0x12C addrBuf=0x1F8 bb
> mem @ loop end:   1757
0x1E7 is used for m_sText1 at that point. Only when m_sText1 += "a"
runs, will m_sText1 be reallocated. In your code, you print the first
line before appending the "a", so at that point, 0x1E7 is still used.

I don't agree yet. In my opinion it works as follows:

cap=0 len=0 addrS=0x133 addrBuf=0x1E7   << these 2 lines are from setup()
cap=0 len=0 addrS=0x12C addrBuf=0x1EB

cap=1 len=1 addrS=0x8DF addrBuf=0x1EF    << first creation of 'sText', at 0x1EF because the mem below is used by m_sText1 and ..2
cap=1 len=1 addrS=0x133 addrBuf=0x1E7 a  << status of m_sText1 AFTER the 1st 'a' was added. The print does NOT come before the += operation.
cap=1 len=1 addrS=0x12C addrBuf=0x1EB b  << same here with sText2. I already wonder here why the buffer isn't realloc. Obviously there is space for at least 1 char.

mem @ loop end:   1759

cap=1 len=1 addrS=0x8DF addrBuf=0x1EF       << You're right, 0x1E7 isn't free yet. sText still there.
cap=2 len=2 addrS=0x133 addrBuf=0x1F3 aa    << both buffers of m_sText1 & ..2 reallocated due to non sufficient remaining space.

cap=2 len=2 addrS=0x12C addrBuf=0x1F8 bb
mem @ loop end:   1757

cap=1 len=1 addrS=0x8DF addrBuf=0x1EF       << 0x1E7 is free now. Why isn't sText put there? Does the compiler optimize things and keep sText although defined locally? It should be removed from stack and heap when loop() is exited.
cap=4 len=3 addrS=0x133 addrBuf=0x1E7 aaa   << instead, m_sText1 goes back to 0x1E7.

cap=4 len=3 addrS=0x12C addrBuf=0x1F8 bbb
mem @ loop end:   1752

I don't know how the internal memory handling works. Afaik, freed memory is put into a list and new requests take from that list first before looking for unused blocks. Is that right?

  m_sText2 += "b";
uses
  String & operator += (const char *cstr)   {concat(cstr);return (*this);}
where concat() uses
  return concat(cstr, strlen(cstr));
where concat(,) is
unsigned char String::concat(const char *cstr, unsigned int length)
{
  unsigned int newlen = len + length;
  if (!cstr) return 0;
  if (length == 0) return 1;
  if (!reserve(newlen)) return 0;     <<< if reserve() fails, then strcpy is not executed, the 'b' is not added, and the length stays the same. This is the case in the line where m_sText1 adds 1 'a' (new heap loc at 0x53B) and it's length becomes 257, but m_sText2 stays at length 256.
  strcpy(buffer + len, cstr);
  len = newlen;
  return 1;
}

reserve() can only fail at
  if (changeBuffer(size))
changeBuffer() can only fail at
  char *newbuffer = (char *)realloc(buffer, maxStrLen + 1);
and keeps everything as is if failed.
So you're right, invalidate() is not called. I somehow mixed it up with copy() where it is used if it fails.
I also agree to your further explanations.

But why is the content completely lost if the length exceeds 260 chars?

Regards
Tobias

[Matthijs Kooijman]

Hi TK,

> cap=1 len=1 addrS=0x12C addrBuf=0x1EB b << same here with sText2. I
> already wonder here why the buffer isn't realloc. Obviously there is space
> for at least 1 char.

I'm not sure why that is, but perhaps it is a side-effect of the
block-based allocation changes you applied to the String class (at least
I believe you mentioned that, right?).

> cap=1 len=1 addrS=0x8DF addrBuf=0x1EF << 0x1E7 is free now. Why isn't
> sText put there? Does the compiler optimize things and keep sText although
> defined locally? It should be removed from stack and heap when loop() is
> exited.

Ah, you're right. 0x1E7 is indeed free and malloc *could* have used it for
allocating sText, I believe.

> I don't know how the internal memory handling works. Afaik, freed memory is
> put into a list and new requests take from that list first before looking
> for unused blocks. Is that right?

Yup, that's pretty much what happens. Some merging of adjacent blocks
also happens, though I'm not sure when. It could be that merging only
happens opportunistically and for some time, multiple adjacent freed
blocks live separetely on the free blocks chain. I am pretty sure malloc
will prefer a memory region that fits the requested memory size exactly,
which could explain why 0x1EF is reused over 0x1E7. Alternatively,
malloc could decide to allocate from the top part of the freed block of
memory, though I'm not sure what the policy for that is.

If you want to see details, here's the source code for malloc on the AVR
architecture:

http://svn.savannah.nongnu.org/viewvc/trunk/avr-libc/libc/stdlib/malloc.c?root=avr-libc&view=markup

Note that I linked to the trunk version, you might want to look through
the tags to find the source for the avr-libc version you're actually
using (dunno what version is shipped with Arduino off-hand).

> But why is the content completely lost if the length exceeds 260 chars?

Oh, seems I missed that part. That is indeed weird. I wonder if this is
somehow a bug in the serial printing code or something like that? Does
it also happen when you just allocate a long String statically and print
that?

Gr.

Matthijs

[TK]

Am Samstag, 30. April 2016 21:41:44 UTC+2 schrieb Matthijs Kooijman:

Hi TK,

> cap=1 len=1 addrS=0x12C addrBuf=0x1EB b  << same here with sText2. I
> already wonder here why the buffer isn't realloc. Obviously there is space
> for at least 1 char.
I'm not sure why that is, but perhaps it is a side-effect of the
block-based allocation changes you applied to the String class (at least
I believe you mentioned that, right?).

I don't think so. When creating that text, I don't set any reserve. You'd see that on the capacity value.
Whatever, I just take it as is.
 

> cap=1 len=1 addrS=0x8DF addrBuf=0x1EF       << 0x1E7 is free now. Why isn't
> sText put there? Does the compiler optimize things and keep sText although
> defined locally? It should be removed from stack and heap when loop() is
> exited.
Ah, you're right. 0x1E7 is indeed free and malloc *could* have used it for
allocating sText, I believe.

malloc() obviously has some weird implementation.
When m_sText.. grows more and more, they are (due to the small, blocking sText) stored at higher mem locations. But also sText itself is put just right underneath the high mem loc of m_sText..  weird. As if it would priorize freed RAM and use that from top to bottom.

> But why is the content completely lost if the length exceeds 260 chars?
Oh, seems I missed that part. That is indeed weird. I wonder if this is
somehow a bug in the serial printing code or something like that? Does
it also happen when you just allocate a long String statically and print
that?

I can't test that right now. I bought a Geniuno Uno and a Mega2560 for a special project. I didn't knew before how many pins and mem I would require, so I bought both. Uno was sufficient, so that one is installed and gone. I need to buy a new one first, Mega2560 has much more RAM and behaves differently therefore.
 

[Matthijs Kooijman]

Hi TK,

> > I'm not sure why that is, but perhaps it is a side-effect of the
> > block-based allocation changes you applied to the String class (at least
> > I believe you mentioned that, right?).
> I don't think so. When creating that text, I don't set any reserve. You'd
> see that on the capacity value.
> Whatever, I just take it as is.

I was thinking it might have malloc'd more than the capacity value, but
not so imporant indeed.

> malloc() obviously has some weird implementation.
> When m_sText.. grows more and more, they are (due to the small, blocking
> sText) stored at higher mem locations. But also sText itself is put just
> right underneath the high mem loc of m_sText.. weird. As if it would
> priorize freed RAM and use that from top to bottom.

Yes, that's *exactly* what it does. See the comment at line 134 of
http://svn.savannah.nongnu.org/viewvc/trunk/avr-libc/libc/stdlib/malloc.c?annotate=2149&root=avr-libc

Apparently putting a new entry at the top of a freed area saves some
work, so that's what happens.

This is exactly why using dynamic memory extensively on a
microcontroller is not always a good idea: If your usage pattern doesn't
match the allocation pattern well, things can get fragmented and memory
is effectively lost.

> > But why is the content completely lost if the length exceeds 260 chars?

> I can't test that right now.

I got curious and tried your code on my Uno and indeed, it also fails
(it shows the strings as empty). It takes a bit longer (I don't have
your changes to String, so it reallocates byte-by-byte), so it fails
when the length is 519. I can't see any failed allocations here until
the strings reach 565 characters, though, so that does not seem to be
the trigger.

I have no more time to look more closely, but this smells like a bug in
String or Print. I seem to be able to reliably reproduce this, so I'll
try to find some time to figure out what happens exactly.

Gr.

Matthijs

[Andrew Kroll]

Have you tried to see if my implementation also suffers?
https://github.com/felis/UHS30/blob/master/libraries/UHS_host/malloc.c

--
You received this message because you are subscribed to the Google Groups "Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to developers+...@arduino.cc.

--

Visit my github for awesome Arduino code @ https://github.com/xxxajk