memory exhausted

[Alex Krasnov]

My server configuration is:

CVS 1.12.13 on Solaris 11.

My client configuration is:

CVS 1.12.13 on Debian 4.0

I am trying to check out a project containing multiple binary files each less
than 512 MB in size. The server reports the following error:

Nov 12 10:01:24 root cvs[879]: [ID 233912 daemon.emerg] error (1, 0) called recursively. Original message was:
Nov 12 10:01:24 root cvs[879]: [ID 702911 daemon.emerg] cvs [checkout aborted]: memory exhausted
Nov 12 10:01:24 root cvs[879]: [ID 587406 daemon.emerg] error (1, 0) called recursively. Second message was:
Nov 12 10:01:24 root cvs[879]: [ID 702911 daemon.emerg] cvs [checkout aborted]: memory exhausted
Nov 12 10:01:24 root cvs[879]: [ID 286756 daemon.emerg] Aborting.

The server reports sufficient physical memory available. Before:

Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 100193 391 10%
ZFS File Data 56597 221 5%
Anon 42156 164 4%
Exec and libs 10991 42 1%
Page cache 1053 4 0%
Free (cachelist) 4824 18 0%
Free (freelist) 830328 3243 79%

Total 1046142 4086
Physical 1046141 4086

After:

Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 127140 496 12%
ZFS File Data 249138 973 24%
Anon 41993 164 4%
Exec and libs 10799 42 1%
Page cache 1053 4 0%
Free (cachelist) 421502 1646 40%
Free (freelist) 194517 759 19%

Total 1046142 4086
Physical 1046141 4086

No per-process memory limits appear to be set. However, the cvs process is
unable to memory-map the file:

mmap64(0x00000000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0) = 0x08430000
mmap64(0x00000000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0) = 0x08420000
mmap64(0x00000000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0) Err#12 ENOMEM
mmap64(0x00000000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0) Err#12 ENOMEM

Questions:

(0) Why is mmap64 failing in this case?
(1) Can the memory usage be limited?
(2) Can the number of files memory-mapped simultaneousely be limited?

Alex

[Alex Krasnov]

[Paul Sander]

We just had a similar question yesterday. When CVS reads RCS files,
it mmaps them into virtual memory. When CVS fails, how large is the
RCS file it's processing at the time, in comparison to the metrics
you indicate below?

[Alex Krasnov]

The RCS file is essentially of the same size as is the working copy, since
there is only one revision. One particular file is 360 MB in size, which
is approximately the resident memory size of the server CVS process
immediately before it fails.

The behavior of the server CVS process makes sense to me. I am more
interested in why it leads to a memory allocation failure and whether
there are ways to configure CVS to require a smaller memory footprint or
the OS to allow a larger memory footprint.

On a side note, I actually sent my question before the amazingly similar
one by Kevin Bulgrien--not entirely certain why it took so long to
deliver. Also, sorry about the duplicate message; I sent it in hope that
the first message simply got dropped somewhere along the way.

Alex

[Arthur Barrett]

Alex,

> My server configuration is:
>
> CVS 1.12.13 on Solaris 11.
>
> My client configuration is:
>
> CVS 1.12.13 on Debian 4.0
>

Just like with Kevin I think the first thing to do is be 100% sure that
it is the client or the server failing. Try coping the file to the
server and doing a commit with :local: if that fails it'll be easier to
debug too.

> No per-process memory limits appear to be set. However, the
> cvs process is
> unable to memory-map the file:

All the memory map stuff is new in 1.12 I think - you could try going
back to 1.11...

If it is the server that is the problem then there are probably quite a
lot of solaris parameters that could affect it - particularly if the
memory maps in 1.12 are using shared memory. I'm no Solaris expert and
AFAICT Solaris 11 is not even a released product.
http://en.wikipedia.org/wiki/Solaris_(operating_system)#Versions

Regards,

Arthur

[Paul Sander]

I found the mmap in CVS 1.11.23, and would not be at all surprised to
find it also in the unstable release that Alex is using. I'm certain
it's been in CVS since CVS absorbed RCS.

Once Alex has determined which part of CVS is crashing (client or
server), the first thing I would check is ulimit for the account
running the CVS binary. Also try running side-by-side something that
monitors the allocation of swap space and a tracing facility on the
CVS binary. This should give you some idea of what in CVS is
consuming the most space.

[Todd Denniston]

Paul Sander wrote, On 11/13/2008 05:28 PM:
> I found the mmap in CVS 1.11.23, and would not be at all surprised to
> find it also in the unstable release that Alex is using. I'm certain
> it's been in CVS since CVS absorbed RCS.

I don't think it has been quite that long, at least on solaris.
http://lists.gnu.org/archive/html/bug-cvs/2003-11/msg00152.html

mmap happened between 1.11.1p1 and 1.11.2.
the solaris mmap patches I applied for solaris 2.6 did not help.

At least in the 1.11.x series you could drop mmap by:
after you configure (but before compile) of cvs Version >1.11.1p1 make the
following change in config.h:

diff config.h.normal config.h
199c199
< #define HAVE_MMAP 1
---
> /*#define HAVE_MMAP 1*/

then `make; make install`

and if you felt like it, you could follow some of my other suggestions about
having mmap problems :)
http://lists.gnu.org/archive/html/info-cvs/2003-12/msg00300.html

--
Todd Denniston
Crane Division, Naval Surface Warfare Center (NSWC Crane)
Harnessing the Power of Technology for the Warfighter

[Paul Sander]

On Nov 13, 2008, at 4:00 PM, Todd Denniston wrote:

> Paul Sander wrote, On 11/13/2008 05:28 PM:
>> I found the mmap in CVS 1.11.23, and would not be at all surprised
>> to find it also in the unstable release that Alex is using. I'm
>> certain it's been in CVS since CVS absorbed RCS.
>
> I don't think it has been quite that long, at least on solaris.
> http://lists.gnu.org/archive/html/bug-cvs/2003-11/msg00152.html
>
> mmap happened between 1.11.1p1 and 1.11.2.
> the solaris mmap patches I applied for solaris 2.6 did not help.
>
> At least in the 1.11.x series you could drop mmap by:
> after you configure (but before compile) of cvs Version >1.11.1p1
> make the
> following change in config.h:
>
> diff config.h.normal config.h
> 199c199
> < #define HAVE_MMAP 1
> ---
> > /*#define HAVE_MMAP 1*/
>
> then `make; make install`
>
> and if you felt like it, you could follow some of my other
> suggestions about having mmap problems :)
> http://lists.gnu.org/archive/html/info-cvs/2003-12/msg00300.html

Keep in mind that with mmap turned off, CVS reads the RCS file into a
malloc buffer that gets realloc'ed as it overflows. Under conditions
where the entire RCS file is read, which is just about anything that
reads beyond the initial admin section (including every commit and tag
operation, and most log operations), this can be inefficient and cause
a swap space overflow even if much of the heap is unused.

If I recall correctly, RCS had a large-memory non-mmap option that
would stat the RCS file and allocate a single buffer large enough to
accommodate it, and read it all in at once. This was in addition to
the slower small-memory method that never read the entire RCS file
into memory. It would be nice if CVS incorporated these capabilities,
too. A bonus would be to fall back to the small memory method if it
anticipates a shortage of memory based on current consumption and
system limits.

[Alex Krasnov]

I am reasonably certain that the problem is on the server, although the
local method does not trigger it. On the other hand, the local method
seems to use different memory allocation method.

More significantly, downgrading to CVS 1.11.22 on the server appears to
have eliminated the problem. The server CVS process now grows to over 1.2
GB of resident memory but does not fail for the 360 MB file, since this
version appears to not use mmap. Thanks for this suggestion--I did not
realize that the version that I was using is the unstable feature release.

Alex

On Thu, 13 Nov 2008, Paul Sander wrote:

[Larry Jones]

Paul Sander writes:
>
> I found the mmap in CVS 1.11.23, and would not be at all surprised to
> find it also in the unstable release that Alex is using. I'm certain
> it's been in CVS since CVS absorbed RCS.

No, CVS originally read the file into memory using normal I/O. It was
changed to use mmap instead, where available, in 1.11.2. Then in
1.11.23 it was enhanced to fall back to normal I/O if mmap fails. The
fallback code is also checked in to the 1.12 branch, but there hasn't
been a release that includes it yet.
--
Larry Jones

You don't get to be Mom if you can't fix everything just right. -- Calvin