Re: mmap and large files
Lew Pitcher
(saptars...@gmail.com) wrote:
> Hello,
> On linux, suppose I have a 64 bit system with 32GB ram. If I were to
> call mmap
> mmap(void *start, size_t length, int prot, int flags, int fd, off_t
> offset);
>
> offset equal to 0 and length the length of a 10 gb file, will mmap map
> the *entire* file into memory?
>
> Or will it (behind the scenes) load in chunks?
> So e.g were the file to be 35GB, would a call to mmap fail? And the
> way around would be to load say X mb chunks by shifting the offset.
comp.os.linux.development.aps is a better forum for this question. F'ups set
to there.
<remark status="off-topic in comp.lang.c">
AFAICT, mmap will, if it /can/, map the entire file into process memory.
That /does not/ mean that it /loads/ the file into memory, either wholly,
or in chunks. What it does mean is that directed access into that memory
region will cause the kernel to retrieve and manipulate pages of the file,
such that the file /appears/ to be mapped contiguously.
However, that behaviour is contingent on mmap() returning a success value.
It /could/ return ((char *) -1) and set errno to EINVAL, indicating that
the requested length was too big.
</remark>
HTH
--
Lew Pitcher
Master Codewright & JOAT-in-training | Registered Linux User #112576
Me: http://pitcher.digitalfreehold.ca/ | Just Linux: http://justlinux.ca/
---------- Slackware - Because I know what I'm doing. ------
Fishtank
> <remark status="off-topic in comp.lang.c">
> AFAICT, mmap will, if it /can/, map the entire file into process memory.
> That /does not/ mean that it /loads/ the file into memory, either wholly,
> or in chunks. What it does mean is that directed access into that memory
> region will cause the kernel to retrieve and manipulate pages of the file,
> such that the file /appears/ to be mapped contiguously.
>
> However, that behaviour is contingent on mmap() returning a success value.
> It /could/ return ((char *) -1) and set errno to EINVAL, indicating that
> the requested length was too big.
> </remark>
>
Regards
Sapsi
David Schwartz
> > Hello,
> > On linux, suppose I have a 64 bit system with 32GB ram. If I were to
> > call mmap
> > mmap(void *start, size_t length, int prot, int flags, int fd, off_t
> > offset);
> > offset equal to 0 and length the length of a 10 gb file, will mmap map
> > the *entire* file into memory?
Yes, if it can.
> > Or will it (behind the scenes) load in chunks?
It will, behind the scenes, load in chunks as necessary to make the
mapping work. Loading occurs after mapping.
> > So e.g were the file to be 35GB, would a call to mmap fail? And the
> > way around would be to load say X mb chunks by shifting the offset.
If there was sufficient linear address space in the process address
space and no resource limit prohibited it, it would succeed.
> AFAICT, mmap will, if it /can/, map the entire file into process memory.
> That /does not/ mean that it /loads/ the file into memory, either wholly,
> or in chunks. What it does mean is that directed access into that memory
> region will cause the kernel to retrieve and manipulate pages of the file,
> such that the file /appears/ to be mapped contiguously.
Exactly.
> However, that behaviour is contingent on mmap() returning a success value.
> It /could/ return ((char *) -1) and set errno to EINVAL, indicating that
> the requested length was too big.
Which is quite likely on a 32-bit OS. On a 64-bit OS, you typically
can map multi-gigabyte files without any problems. Heck, a process
with suitable privileges can 'mmap' a 1TB hard drive partition if it
wants to.
DS
Fishtank
> Which is quite likely on a 32-bit OS. On a 64-bit OS, you typically
> can map multi-gigabyte files without any problems. Heck, a process
> with suitable privileges can 'mmap' a 1TB hard drive partition if it
> wants to.
>
> DS
Thanks to all.
Regards
S