mmap failing with EINVAL inside valgrind
Guido Reina
I am having problems trying to run an application with valgrind
v3.4.1. The application mmap()s a big file (around 200 MB) with the
flag MAP_FIXED and some fixed address, the size is multiple of 4096.
map fails with the error EINVAL when I run it with valgrind. If I run
it with gdb, it works fine. Is there something I can change to make it
work?
Thanks in advance,
Guido
Noob
It always helps to provide a small compilable example to demonstrate your
problem, so that others may reproduce it on their system.
You should also state OS and kernel revision.
David Schwartz
MAP_FIXED is supposed to return EINVAL if you specify an address that
cannot be mapped. It *IS* working.
You are asking the wrong question. The right question is "why does
this application fail when its mmap fails, since the mmap failing is
perfectly routine and ordinary".
DS
Moi
> Thank you for the answers.
>
> The thing is that it works when I run it "normally" and it also works
> when I run it with gdb, but not when I run it valgrind.
>
> I have just read the subject of my post... I don't want to mean that
> valgrind is not working. I am having this problem and I want to
> understand why this particular call to mmap is failing when I run it
> with valgrind. Finding memory leaks without valgrind is too difficult,
> that's why I have started this post.
Well, if mmap() returns without an error without valgrind, it probably
means that mmap() succeeded.
When it fails with valgrind it might be valgrind-related. Maybe it trips
a guard page ? Maybe valgrind is confused by mmap()?
It seems you use a c++ wrapper around mmap. Maybe it hits a (memory
related) bug, and errno is not updated accordingly. (not all erros change
errno)
Maybe your fixed mapped address range overlaps with memory addresses in
use (such as the heap) which trips valgrind ?
For testing:
1) mmap only half of the file (if the application permits)
2) mmap at another address (if the application permits)
HTH, YMMV,
AvK
David Schwartz
> Thank you for the answers.
>
> The thing is that it works when I run it "normally" and it also works
> when I run it with gdb, but not when I run it valgrind.
This may be true, but I don't think so, and you haven't presented any
evidence. Why do you think that 'mmap' shouldn't return an error? In
other words, you have showed that it works differently, but not that
it doesn't work.
> I have just read the subject of my post... I don't want to mean that
> valgrind is not working. I am having this problem and I want to
> understand why this particular call to mmap is failing when I run it
> with valgrind. Finding memory leaks without valgrind is too difficult,
> that's why I have started this post.
Can you give any reason why it shouldn't fail? Is it guaranteed to
succeed by some standard? For example, is the address passed to
MAP_FIXED an address it got from a previous 'mmap' call?
> I want to do what Noob suggested and write a test program to see where
> the problem is. The original program is using POST++ to mmap a file
> and it fails in the file::open method. The base address for mmap is:
> 0x62500000, and the size: 200000000. POST++ converts the size to a
> multiple of 4096.
Where did that address and size come from? If it didn't come from a
previous 'mmap', then failure is perfectly reasonable.
> So, the call is something like:
> void* p = mmap(base, mapped_size, PROT_READ|PROT_WRITE, MAP_VARIABLE|
> MAP_SHARED|MAP_FILE, fd, 0)
> where base = 0x62500000
> and mapped_size is a bit bigger than 200000000, so it is multiple of
> 4096
>
> I thought that the base address could be in use when running it with
> valgrind, so I checked: /proc/<pid>/maps and chose another address,
> but this didn't work either.
So what? Again, why does this constitute not working? Picking random
addresses for mappings is not something that is ever guaranteed to
work.
DS
David Schwartz
> I don't like specifying a memory address for mmap, but the program is
> like this and I cannot change it. It has to load in that memory
> address. Don't ask me from where they have got that memory address
> for mmap, I have never liked magic numbers.
It sounds like the program works by pure luck and has just run out of
luck. The original programmer probably picked a value that "worked for
him" and didn't care whether it worked for you. This program is pegged
to those assumptions, and now that they're not valid, neither is the
program.
If you can figure out what specifically is wrong with that address,
you may be able to clear that address. It may help to do your
debugging and testing on another machine. What OS and version is this?
DS
Moi
> The file which does the mmap is:
>
> http://www.garret.ru/post/file.cxx
I browsed through the source (and noticed that the first 2/3 was
conditionally compiled for Widows.;-)
It seems there already is a debugger hook present (the program tries to
fire up a debugger when catching a signal).
this MIGHT interfere with running under a debugger or valgrind.
(BTW: what IS it ? some kind of bufferpool + transaction log processing
for a DBMS ???)
>
>
> I will try Moi's approach, just to see whether it can, at least, mmap
> part of the file (which won't make the program work, but at least I can
> see whether mmap succeeds), just to see if it is a problem with the
> size. Moi, you might be right with the memory range, that's what I
> wanted to see looking at the memory ranges... I don't know why they want
> the program to mmap the file in that memory address.
>
> Guido
I must admit: It is only my gut feeling. I don't know c++ enough, I don't
know valgrind at all.
The phenomenon you observe MIGHT be due to interference.
BTW can you find any REASON for the code to use a fixed map-adress ?
The code looks reasonable good. But it is only one small part. In some
other part malloc/new/free/delete MAY have been overloaded to make use of
mmapped memory or some other clever scheme ...
(no, I don't like C++ ;-)
Good luck,
AvK
Rainer Weikusat
> On Sep 3, 7:48�am, guidoreina <guidore...@gmail.com> wrote:
>> Thank you for the answers.
[...]
>> I have just read the subject of my post... I don't want to mean that
>> valgrind is not working. I am having this problem and I want to
>> understand why this particular call to mmap is failing when I run it
>> with valgrind. Finding memory leaks without valgrind is too difficult,
>> that's why I have started this post.
>
> Can you give any reason why it shouldn't fail? Is it guaranteed to
> succeed by some standard?
Specifications document implementation requirements. No properties of
actual implementations follow from that.
> For example, is the address passed to
> MAP_FIXED an address it got from a previous 'mmap' call?
[...]
> So what? Again, why does this constitute not working? Picking random
> addresses for mappings is not something that is ever guaranteed to
> work.
Quoting the current UNIX(*)-standard on this (from the mmap
rationale):
The MAP_FIXED address treatment is likely to fail for non-page-aligned
values and for certain architecture-dependent address
ranges. Conforming implementations cannot count on being able to
choose address values for MAP_FIXED without utilizing non-portable,
implementation-defined knowledge. Nonetheless, MAP_FIXED is provided
as a standard interface conforming to existing practice for utilizing
such knowledge when it is available.
And this basically turns the issue around: Can you quote a formal
specification of any kind which specifically prohibits use of this
particular value as an address on any implementation which supports
MAP_FIXED? If so, is this relevant here, ie is this the implementation
used by the person with the problem?
Scott Lurndal
>Morning,
>POST++ is "Persistent Object Storage for C++". "My" application uses
>POST++ to open and use a database of objects. All the applications
>which need to use the database, map it to the same address... weird.
>Why don't they just let the application load the file where mmap
>thinks is the right way? I don't know.
Because they probably use absolute pointers in the mmap region
rather than using offsets from the mmap base. Lazy programmers.
scott
Rainer Weikusat
Or people with different preferences. Designed in such a way, the
'object database' on disk can just be mapped into the address space of
an application and immediatly yields a set of useful 'life'
C++-objects. Some people always believe certain features really
shouldn't exist. But insofar they do, there is obviously no consensus.
Some people believe that everyone should eat only vegetables for
greater purity ...
Moi
> Morning,
>
> Yes, it is a bit confusing to have Windows at the beginning of the file,
> it would have been better to have it in another file, as they don't
> share anything. The POST++ catches SIGSEGV to detect when it is trying
> to access a page which is not loaded. When this program crashes, POST++
> tries to start the debugger... but the program is launched by another
> program and then we never get a core dump nor we can start the debugger.
>
> POST++ is "Persistent Object Storage for C++". "My" application uses
> POST++ to open and use a database of objects. All the applications which
> need to use the database, map it to the same address... weird. Why don't
> they just let the application load the file where mmap thinks is the
> right way? I don't know.
>
> I have done some other experiments. The application has a configuration
> file where you can define the base address for mmap and the size. I have
> reduced the size but still doesn't work, that is because POST++ uses
> then the size of the file. I have just run it with gdb. It opens the
> file as O_RDWR and it uses for mmap: prot = PROT_READ | PROT_WRITE.
> flags = MAP_PRIVATE | MAP_FIXED. mmap returns the base address =
> 0x62500000. I will try later (now they want me to do something else) to
> change it so it doesn't use a fixed address. I want to see whether it
> works without fixed address... I don't know whether the application will
> work.... if it does, I make the change temporarily to get rid of the >
> 250 MB of memory leaked per day ;).
I have been reading some more in POST++'s source.
I appears POST needs the signals. It uses them to create a copy of a
"page" (for rollback purposes) once it is accessed.
The application stores objects into the mmapped memory pool, since these
objects can contain pointers (to other objects in the pool), the adresses
of the objects need to be reliable between runs.
If more applications are accessing the same mmapped file, containing the
objects, they should all mmap the file to the *same* address. The easiest
way to guarantee this is to use a *fixed* address.
In other words: you should probably keep the fixed address.
In the last part of POST's .html documentation there are some hints for
debugging. Specifically the need to instruct gdb to *not* intercept
SIGSEGV. I presume valgrind does something similar to gdb, so you'll have
to find a way to instruct valgrind to ignore some of the signals, too.
That still does not solve the mystery where the EINVAL comes from, but
maybe the EINVAL is an artifact caused by the program's interaction with
valgrind. IMHO valgrind is the problem. (not of the memory leak, of
course)
HTH,
AvK
Scott Lurndal
>sc...@slp53.sl.home (Scott Lurndal) writes:
>> guidoreina <guido...@gmail.com> writes:
>>>Morning,
>>
>>>POST++ is "Persistent Object Storage for C++". "My" application uses
>>>POST++ to open and use a database of objects. All the applications
>>>which need to use the database, map it to the same address... weird.
>>>Why don't they just let the application load the file where mmap
>>>thinks is the right way? I don't know.
>>
>> Because they probably use absolute pointers in the mmap region
>> rather than using offsets from the mmap base. Lazy programmers.
>
>Or people with different preferences. Designed in such a way, the
>'object database' on disk can just be mapped into the address space of
>an application and immediatly yields a set of useful 'life'
>C++-objects.
You mean like Microsoft did with Office file formats? We all know
how well that worked.
Fact is, that using fixed addresses is neither portable nor future-proof;
the next release of the OS may not support mapping at an address a prior
version did.
Not to mention the issues noted by the OP.
scott
Rainer Weikusat
> Rainer Weikusat <rwei...@mssgmbh.com> writes:
>>sc...@slp53.sl.home (Scott Lurndal) writes:
>>> guidoreina <guido...@gmail.com> writes:
>>>>Morning,
>>>
>>>>POST++ is "Persistent Object Storage for C++". "My" application uses
>>>>POST++ to open and use a database of objects. All the applications
>>>>which need to use the database, map it to the same address... weird.
>>>>Why don't they just let the application load the file where mmap
>>>>thinks is the right way? I don't know.
>>>
>>> Because they probably use absolute pointers in the mmap region
>>> rather than using offsets from the mmap base. Lazy programmers.
>>
>>Or people with different preferences. Designed in such a way, the
>>'object database' on disk can just be mapped into the address space of
>>an application and immediatly yields a set of useful 'life'
>>C++-objects.
>
> You mean like Microsoft did with Office file formats? We all know
> how well that worked.
I have no idea what 'Microsoft did with the Office file formats'. I
recognize this type of 'argument', however, and happen to know a
nice parody coming from some Peanuts-strip, where Linus, during a
discussion of the relative merits of Beethoven, insofar I remember
this correctly, suddenly cries: "Beethoven never supported Hitler!".
> Fact is, that using fixed addresses is neither portable
For a suitable definition of 'portable', certainly. It is an
XSI-mandated UNIX(*)-extension and chances are that this is already
much more 'portable' than what would be needed. And a file created in
this way is obviously unsuitable for data interchange between
different systems. And it wreaks havoc onto the OpenBSD 'security
concept' of 'make programming more difficult in the hope that "they"
will never figure out how to pull our trowsers down in public again'.
But all of this is, first and foremost, hypothetical and may or may
not be relevant for any given situation. Files are not necessarily
used for data interchange, they may just provide 'persistent
storage'.
> nor future-proof; the next release of the OS may not support mapping
> at an address a prior version did.
In a strict sense, nothing is future-proof.
> Not to mention the issues noted by the OP.
Another valgrind-quirk? OMG ...
David Schwartz
> Or people with different preferences. Designed in such a way, the
> 'object database' on disk can just be mapped into the address space of
> an application and immediatly yields a set of useful 'life'
> C++-objects. Some people always believe certain features really
> shouldn't exist. But insofar they do, there is obviously no consensus.
That something can be done correctly does not excuse when it is done
incorrectly. This is the latter case, since the program fails when the
'mmap' fails. This is a case where the 'mmap' is allowed to fail and
the program is not.
DS
Rainer Weikusat
SUS documents this as
The mmap() function may fail if:
[EINVAL]
The addr argument (if MAP_FIXED was specified) or off is not a
multiple of the page size as returned by sysconf(), or is
considered invalid by the implementation.
'the implementation' is Linux (judging from the original posting) or
could be (according to http://www.garret.ru/post.html) 'Linux, Digital
Unix, Solaris, AIX' or 'Windows NT 4.0/95'). valgrind isn't
sufficiently compatible with the used supported platform to be used
together with the POST++-library. This can be considered a missing
feature or even bug of valgrind, but not of the library since noone
ever claimed that the 'UNIX(*)-implementation named valgrind' was a
supported platform.
But all of this is actually completely besides the point, because the
library doesn't use MAP_FIXED. According to the comments in
valgrind-3.4.1/coregrind/m_aspacemgr/aspacemgr-linux.c
this is supposed to work such that either the 'hinted' mapping is
placed at the intended address (by the kernel) or that valgrind picks
a suitable replacement address. The file linked-to by the OP doesn't
contain MAP_FIXED, either.
David Schwartz
> But all of this is actually completely besides the point, because the
> library doesn't use MAP_FIXED. According to the comments in
>
> valgrind-3.4.1/coregrind/m_aspacemgr/aspacemgr-linux.c
>
> this is supposed to work such that either the 'hinted' mapping is
> placed at the intended address (by the kernel) or that valgrind picks
> a suitable replacement address. The file linked-to by the OP doesn't
> contain MAP_FIXED, either.
I think you're drawing a distinction between the library and the
program that I'm not. In any event, I'm a bit confused now. The OP
specifically stated that an mmap call with MAP_FIXED was failing, and
that it was used in a case where it is not guaranteed to succeed. If
the library doesn't call mmap with MAP_FIXED and the program doesn't,
why is mmap getting called with MAP_FIXED? If valgrind is doing it, it
would be interesting to know the circumstances -- you may be right, it
maybe be a bug in valgrind.
DS
Moi
relevant part of post++
....
file_header hdr;
hdr.base_address = base;
hdr.file_size = 0;
read(fd, &hdr, sizeof hdr);
base = (char*)hdr.base_address;
allocated_size = size = ALIGN(hdr.file_size, page_size);
mapped_size = size > max_file_size ? size : max_file_size;
void* p = mmap(base, mapped_size, PROT_READ|PROT_WRITE,
MAP_VARIABLE|MAP_SHARED|MAP_FILE, fd, 0);
if (p == (char*)MAP_FAILED) {
error_code = errno;
TRACE_MSG(("post_file::open: mmap failed: base=%p, size=%ld: %
s\n",
base, mapped_size, strerror(error_code)));
::close(fd);
return false;
}
base = (char*)p;
n_locked_pages = 0;
...
, so it reads the intended map addres and size from the file before
mapping it to that "fixed" address.
Both of these _could_ of course contain garbage, causing EINVAL.
AvK
Rainer Weikusat
This isn't a mapping to a fixed address, for the simple reason that
MAP_FIXED is not among the MAP_-flags to the mmap call (MAP_VARIABLE
is #defined to 0). It is a request to mmap the file at the address
given as base if possible and to mmap it elsewhere if not. Valgrind
intercepts this call and should perform exactly this task: If mapping
at this address isn't possible because it is occupied by a
valgrind-owned mapping, treat this as so-called 'floating mmap', ie
map in some possible location.
The way valgrind seems to treats this is that it always tries to do a
fixed mapping (code is in coregrind/m_syswrap/syswrap-generic and
coreground/m_aspacemgr/aspacemgr-linux) it just selects and address it
'knows' to be good. Quote from am_get_advisory (comments added)
switch (req->rkind) {
case MFixed: /* MAP_FIXED */
if (fixedIdx >= 0) {
*ok = True;
return req->start;
} else {
*ok = False;
return 0;
}
break;
case MHint: /* address w/o MAP_FIXED */
if (fixedIdx >= 0) {
*ok = True;
return req->start;
}
if (floatIdx >= 0) {
*ok = True;
return nsegments[floatIdx].start;
}
*ok = False;
return 0;
case MAny: /* no address in original call */
if (floatIdx >= 0) {
*ok = True;
return nsegments[floatIdx].start;
}
*ok = False;
return 0;
default:
break;
}
Moi
> Moi <ro...@invalid.address.org> writes:
>> On Thu, 10 Sep 2009 05:24:47 -0700, David Schwartz wrote:
>>> On Sep 9, 8:47 am, Rainer Weikusat <rweiku...@mssgmbh.com> wrote:
>>>
>>>> But all of this is actually completely besides the point, because the
>>>> library doesn't use MAP_FIXED. According to the comments in
>>>>
>>>> valgrind-3.4.1/coregrind/m_aspacemgr/aspacemgr-linux.c
>>>>
>>>> this is supposed to work such that either the 'hinted' mapping is
>>>> placed at the intended address (by the kernel) or that valgrind picks
>>>> a suitable replacement address. The file linked-to by the OP doesn't
>>>> contain MAP_FIXED, either.
>>>
>>> I think you're drawing a distinction between the library and the
>>> program that I'm not. In any event, I'm a bit confused now. The OP
>>> specifically stated that an mmap call with MAP_FIXED was failing, and
>>> that it was used in a case where it is not guaranteed to succeed. If
>>> the library doesn't call mmap with MAP_FIXED and the program doesn't,
>>> why is mmap getting called with MAP_FIXED? If valgrind is doing it, it
>>> would be interesting to know the circumstances -- you may be right, it
>>> maybe be a bug in valgrind.
>>
>> relevant part of post++
void* p = mmap(base,
>> mapped_size, PROT_READ|PROT_WRITE,
>> MAP_VARIABLE|MAP_SHARED|MAP_FILE, fd, 0);
>> if (p == (char*)MAP_FAILED) {
>
> This isn't a mapping to a fixed address, for the simple reason that
> MAP_FIXED is not among the MAP_-flags to the mmap call (MAP_VARIABLE is
> #defined to 0). It is a request to mmap the file at the address given as
> base if possible and to mmap it elsewhere if not. Valgrind intercepts
> this call and should perform exactly this task: If mapping at this
> address isn't possible because it is occupied by a valgrind-owned
> mapping, treat this as so-called 'floating mmap', ie map in some
> possible location.
My bad.
I had not even checked the flags, seeing that arg1 was non null.
I stand corrected.
AvK
David Schwartz
> The way valgrind seems to treats this is that it always tries to do a
> fixed mapping (code is in coregrind/m_syswrap/syswrap-generic and
> coreground/m_aspacemgr/aspacemgr-linux) it just selects and address it
> 'knows' to be good. Quote from am_get_advisory (comments added)
Nice detective work. So we might have a workaround for the OP -- if
the mmap fails, try it again without specifying an address.
DS
Rainer Weikusat
> I tried with the google performance tools library, which I have read
> they can also be used to detect memory leaks, but I get a SIGBUS when
> POST++ is doing its stuff, it seems I will have to find the memory
> leaks the hard way...
Assuming that you really saw a MAP_FIXED, eg, in strace-output, this
call has been done by valgrind which always uses MAP_FIXED on this
codepath, no matter if the application originally specified it. For a
so-called 'hinted mapping', valgrind tries either the given address if
the area in question isn't used by valgrind itself or an area valgrind
considers to be free if not. It returns EINVAL if no suitable area
could be found. Otherwise, it calls into the kernel with the start
address returned by its search. Should the kernel deny the mapping,
valgrind again searches for a suitable free area, this time ignoring
the address given in the application call, and tries to mmap that if
such an area is found (using MAP_FIXED). Should the kernel deny the
second mapping, too, EINVAL is returned.
This means that either an area large enough to be used for the
POST++-mapping wasn't available according to the memory map managed by
valgrind or 'something strange' (and probably, unintended) is going on
within valgrind itself. It should be possible to research this further
by inserting suitable diagnotisc output statements into the two
involved valgrind-routines, generic_PRE_sys_mmap in
coregrind/m_syswrap/syswrap_generic.c and am_get_advisory in
coregrind/m_aspacemgr/aspacemgr-linux.c
Rainer Weikusat
> On Sep 15, 2:09�pm, Rainer Weikusat <rweiku...@mssgmbh.com> wrote:
>> guidoreina <guidore...@gmail.com> writes:
>> > I tried with the google performance tools library, which I have read
>> > they can also be used to detect memory leaks, but I get a SIGBUS when
>> > POST++ is doing its stuff, it seems I will have to find the memory
>> > leaks the hard way...
>>
>> Assuming that you really saw a MAP_FIXED, eg, in strace-output, this
>> call has been done by valgrind which always uses MAP_FIXED on this
>> codepath,
[...]
> Yes, the POST++ code has MAP_FIXED.
The POST++-code does not have MAP_FIXED. Neither the file you linked to
nor the original package. It uses a so-called hinted mapping, meaning,
it requests that its database is mapped at the specificed address if
possible and elswhere if not.