[Boost-users] [iostream] Problems opening large mapped_file

[Heiko Klein]

Hi,

I wanted to test the mapped_file_source iostream device. My code is
working well with a normal file_source device and files >= 6GB. When I
change the constructor to a mapped_file_source, it works still well for
smaller files (e.g. 150MB), but for files > 2GB I get (tested with 2.5GB):

exception occured: failed getting file size: Value too large for defined
data type


And for files > 4GB (tested with 6GB) I get:

exception occured: failed opening file: Bad file descriptor



My constructor is as follows:

using namespace boost::iostreams;
typedef stream<mapped_file_source> mmStream;
mmStream* mmistream = new mmStream();
mmistream->open(mapped_file_source(fileName));



I've tested against boost 1.44 and a 32bit linux (Ubuntu 10.04).


Are these general mmap problems, or am I missing some flags for
mapped_file_source?

Best regards,

Heiko
_______________________________________________
Boost-users mailing list
Boost...@lists.boost.org
http://lists.boost.org/mailman/listinfo.cgi/boost-users

[Vitaly Budovski]

Hello,

32bit systems have only approximately 3GB of user addressable memory.
You should upgrade to a 64bit system.


Regards,

Vitaly

[Heiko Klein]

On 2012-08-07 06:38, Vitaly Budovski wrote:
> On 7 August 2012 02:43, Heiko Klein <Heiko...@met.no> wrote:
>> Hi,
>>
>> I wanted to test the mapped_file_source iostream device. My code is working
>> well with a normal file_source device and files >= 6GB. When I change the
>> constructor to a mapped_file_source, it works still well for smaller files
>> (e.g. 150MB), but for files > 2GB I get (tested with 2.5GB):
>>
>> exception occured: failed getting file size: Value too large for defined
>> data type
>>
>>
>> And for files > 4GB (tested with 6GB) I get:
>>
>> exception occured: failed opening file: Bad file descriptor
>>
>>
>>
>> My constructor is as follows:
>>
>> using namespace boost::iostreams;
>> typedef stream<mapped_file_source> mmStream;
>> mmStream* mmistream = new mmStream();
>> mmistream->open(mapped_file_source(fileName));
>>
>>
>>
>> I've tested against boost 1.44 and a 32bit linux (Ubuntu 10.04).
>>
>>
>> Are these general mmap problems, or am I missing some flags for
>> mapped_file_source?
>>
>> Best regards,
>>
>> Heiko
>

> Hello,
>
> 32bit systems have only approximately 3GB of user addressable memory.
> You should upgrade to a 64bit system.
>
>
> Regards,
>
> Vitaly

Vitaly,

thanks for your explanation. This makes sense for the problems with the
6GB file. But it does not explain the 'failed getting file size: Value
too large for defined data type' with the 2.5GB file. Do you have an
idea why this operation fails?

Since I cannot upgrade to 64bit everywhere, I catch now the exception
and open the stream again as file_source.

[Viatchesla...@h-d-gmbh.de]

Looking with debugger where the exception is thrown you should be able to
understand the root issue.

-- Slava

[Heiko Klein]

I checked now the boost-sources. My version was 1.40 not 1.44 as I
thought previously. 1.40 does not have build-in large-file support,
restricting it to 2GB files, this has been corrected in later version,
e.g. from a diff to 1.41:

#ifdef _LARGEFILE64_SOURCE
flags |= O_LARGEFILE;
#endif


Heiko

--
Dr. Heiko Klein Tel. + 47 22 96 32 58
Development Section / IT Department Fax. + 47 22 69 63 55
Norwegian Meteorological Institute http://www.met.no
P.O. Box 43 Blindern 0313 Oslo NORWAY