[boost] Boost Serialization: Serializing large objects

[Adlai Shawareb via Boost]

We are trying to text serialize a 700 MB object. We are getting an archive_exception: input stream error.

This is on a 32-bit QNX system.


* Is there a limit to the size of the object that can be serialized?
* Is there anything we can do to fix or work around this?

Here is the code we are using:

std::stringstream ss_;
boost::archive::text_oarchive oa(ss_, boost::archive::no_header);
oa << &object;

Adlai

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Robert Ramey via Boost]

There is no known limitation on file size in the boost serialization
library.

Robert Ramey

[Robert Ramey via Boost]

On 10/5/17 10:41 AM, Adlai Shawareb via Boost wrote:

suggestion: Try using a text file stream rather than a stringstream.

Robert Ramey

[Adlai Shawareb via Boost]

> We are trying to text serialize a 700 MB object. We are getting an archive_exception: input stream error.
>
> This is on a 32-bit QNX system.
>

> Is there a limit to the size of the object that can be serialized?

> Is there anything we can do to fix or work around this?
>
> Here is the code we are using:
>
> std::stringstream ss_;
> boost::archive::text_oarchive oa(ss_, boost::archive::no_header);
> oa << &object;
>
> Adlai

Thanks for your help. When serializing into a stringstream, how does Boost add or allocate memory to the stringstream?

[Robert Ramey via Boost]

On 10/9/17 4:21 PM, Adlai Shawareb via Boost wrote:

> Thanks for your help. When serializing into a stringstream, how does Boost add or allocate memory to the stringstream?

Boost doesn't do it. std::stringstream does it.

It's not too hard to make your own archive which is just a giant memory
buffer. I think there's an example in the documentation which does
this. But I might be wrong.

Robert Ramey

[Adlai Shawareb via Boost]

>
> Boost doesn't do it. std::stringstream does it.
>
> It's not too hard to make your own archive which is just a giant memory
> buffer. I think there's an example in the documentation which does
> this. But I might be wrong.

Sorry Robert I wasn't clear. What I meant to ask was which calls to std::stringstream does Boost use to add memory to the stringstream?

I looked at the Serialization Tutorial and didn't see an example of making my own archive. If you are inclined, I would appreciate an example.

[Robert Ramey via Boost]

On 10/10/17 10:41 AM, Adlai Shawareb via Boost wrote:
>>
>> Boost doesn't do it. std::stringstream does it.
>>
>> It's not too hard to make your own archive which is just a giant memory
>> buffer. I think there's an example in the documentation which does
>> this. But I might be wrong.
>
> Sorry Robert I wasn't clear. What I meant to ask was which calls to std::stringstream does Boost use to add memory to the stringstream?

as far as I know there are no such calls supported by std::stringstream.
But maybe I'm wrong about that. Boost serialization uses the
interface provide by std::basic_stream - no more than that.

[Adlai Shawareb via Boost]

> as far as I know there are no such calls supported by std::stringstream.
> But maybe I'm wrong about that. Boost serialization uses the
> interface provide by std::basic_stream - no more than that.

Thanks Robert. Does Boost use operator<< to add data to the stringstream? If not, how is data added?
I would like to know so that I can look into why using that method we get an error.

[Robert Ramey via Boost]

On 10/10/17 11:29 AM, Adlai Shawareb via Boost wrote:
>> as far as I know there are no such calls supported by std::stringstream.
>> But maybe I'm wrong about that. Boost serialization uses the
>> interface provide by std::basic_stream - no more than that.
>
> Thanks Robert. Does Boost use operator<< to add data to the stringstream? If not, how is data added?
> I would like to know so that I can look into why using that method we get an error.
>
> Adlai

I'm sorry, I can't give short answers to these questions. You'll have
to study the documentation.

Robert Ramey

[Chris Glover via Boost]

>
>
> Thanks Robert. Does Boost use operator<< to add data to the stringstream?
> If not, how is data added?
> I would like to know so that I can look into why using that method we get
> an error.
>
> Adlai
>

You say the object is 700 megs. Is that 700 megs in memory or in text? The
text representation is almost certainly bigger than the in memory
representation so you might just be running out of memory.

Alternatively, you could just be running out of address space if
stringstream requires a contiguous buffer. That's an implementation detail,
but I would not be surprised if stringstream allocates a contiguous buffer,
in which case you almost certainly would be bumping into a memory
fragmentation issue due to the 32bit address space.

If you serialize directly to a file instead of stringstream, as Robert
suggested, the problem might go away.

-- chris

[Adlai Shawareb via Boost]

> You say the object is 700 megs. Is that 700 megs in memory or in text? The text representation is almost certainly bigger than the in memory
> representation so you might just be running out of memory.
>
> Alternatively, you could just be running out of address space if stringstream requires a contiguous buffer. That's an implementation detail, but I
> would not be surprised if stringstream allocates a contiguous buffer, in which case you almost certainly would be bumping into a memory
> fragmentation issue due to the 32bit address space.
>
> If you serialize directly to a file instead of stringstream, as Robert suggested, the problem might go away.
>
> -- chris

Thanks for responding Chris. The unserialized data is 700 megs. You are correct, the text representation is closer to 1.2 GB. We are using extended addressing on QNX, so I haven't focused on the 32bit address space / fragmentation issue, but it can't be ruled out. The contiguous buffer issue seems likely. The questions I was asking of Robert were to look for something that we could take to QNX to get some feedback.

Adlai

________________________________

[Seth via Boost]

On 05-10-17 19:41, Adlai Shawareb via Boost wrote:
> std::stringstream ss_;
> boost::archive::text_oarchive oa(ss_, boost::archive::no_header);
> oa << &object;

If you can't write directly to a file, just use `back_insert_device`
with a properly reserved container (std::string or std::vector<char> for
example):
http://www.boost.org/doc/libs/1_65_1/libs/iostreams/doc/classes/back_inserter.html