[boost] Boost XML Serialization Stability across library versions

[Adlai Shawareb]

Hi,

We want to generate an Serialized XML archive now, and be able to deserialize it and get the same data it in the future.

Is Boost XML Serialization guaranteed to be compatible across versions of the library?

The XML serialized archive would be stored in a memory chip and, in the future, read back and deserialized. I don't believe that would impact the answer, but want to offer that.

Adlai

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Robert Ramey]

On 1/24/17 3:52 PM, Adlai Shawareb wrote:
> Hi,
>
> We want to generate an Serialized XML archive now, and be able to deserialize it and get the same data it in the future.
>
> Is Boost XML Serialization guaranteed to be compatible across versions of the library?

It is certainly meant to be. But it is not explicity tested so it can't
be guaranteed. If someone wanted to make tests for this, we would look
into incorporating them into the test suite.

>
> The XML serialized archive would be stored in a memory chip and, in the future, read back and deserialized. I don't believe that would impact the answer, but want to offer that.

Hmmmm. Remember that the XML generated by the serialization library
includes special tags to support the recover of the original C++ data
structure which produced the archive. So it's not particularly well
suited to "external" applications and, as far as I know, used to recover
the data by a C++ program. So if one is going to do this, I don't see
much point in using xml. One could get the same portability while using
plain text with a lot less overhead.

Robert Ramey

[Jeff Garland]

Actually unless something is changed I believe it's not compatible.
There's a library version number burned into the archive that prevents this
from working unfortunately. I believe I tested this between 1.54 and 1.58
versions of boost.

Jeff

[Adlai Shawareb]

Thank you. We are really looking at Boost Serialization as a way to manage changes in the data stored in the memory chip. In future versions we may add a float or remove an int, and need to be backward compatible with those changes.

Is there a feature in Boost that would support managing data in that way?

The only alternative to serialization we've come up with is to use unions and structs to selectively include or exclude data as the versions change.

Thanks,
Adlai

________________________________

[Robert Ramey]

On 1/24/17 6:29 PM, Adlai Shawareb wrote:
> Thank you. We are really looking at Boost Serialization as a way to manage changes in the data stored in the memory chip. In future versions we may add a float or remove an int, and need to be backward compatible with those changes.
>
> Is there a feature in Boost that would support managing data in that way?
>
> The only alternative to serialization we've come up with is to use unions and structs to selectively include or exclude data as the versions change.
>
> Thanks,
> Adlai

I think you're confusing two different questions

a) The library maintains backward compatibility to your previous data.
So if your data changes, you can still retrieve data from older
archives. This is referred to as "versioning".

b) The same system is applied to maintaining backward compatibility to
structs from the standard library, - vector, optional, etc....

c) some structures from the standard library don't have a version number
encoded in the data. versioning is maintained by a library version
number encoded in the header record.

d) This also maintains backward compatibility when you change your
compiler to another version.

e) This is all described in the documenation along with examples.

f) This has worked well. In a couple of very rare instances it has
occurred this this versioning between libraries has been messed up for
one or another data type. I regret this. I would like to see an
expanded test suite to address this and hence be able to guarantee that
this problem would never occur. Unfortunately, no one has expressed the
view that this is sufficiently necessary in order to justify the
expenditure required.

g) In your case, I would recommend creating some tests of your own that
you could run on every new version of the library so that you would
absolutly that backward compatibility has not been broken.

I thought you were referring to abi compatibility - which I don't
believe can be guaranteed. But now I see this is not your question.

Robert Ramey

[Richard Hodges]

It's occurred to me before that boost.serialization could really use a json
archive.

[Hans Dembinski]

Dear Robert,

I am interested in implementing a portable binary archive for boost.serialization and would like to discuss this with you. I have a personal interest in that for my histogram library, which would profit from a full implementation of portable_binary_archive that supports floating point numbers.

I had a look into the boost serialisation code, in particular, the incomplete portable_binary_archive implementation in the examples directory and EOS, the external implementation of a portable binary archive. I also read the notes in the documentation of boost.serialization about the portable binary archive. Is it still correct that portable_binary_archive only needs a portable implementation of reading and writing floats and doubles?

I read that the author of EOS tried to submit to boost a while back. Apparently, this never happened. Do you remember what the issue was?

I read up on floating point representation in binary, and I would approach the problem of serialising floats by implementing the IEEE754 standard in a portable way. Does that make sense?

Best regards,
Hans

[Bjorn Reese]

On 01/25/2017 09:43 AM, Richard Hodges wrote:
> It's occurred to me before that boost.serialization could really use a json
> archive.

https://github.com/breese/trial.protocol

[Richard Hodges]

Nice, thank you.

[Robert Ramey]

On 1/25/17 12:43 AM, Richard Hodges wrote:
> It's occurred to me before that boost.serialization could really use a json
> archive.
>

Right. No one as under taken that task. Should be fairly straight
forward using spirit v3.

Robert Ramey

[Hans Dembinski]

Dear Robert,

I am interested in implementing a portable binary archive for boost.serialization and would like to discuss this with you. I have a personal interest in that for my histogram library, which would profit from a full implementation of portable_binary_archive that supports floating point numbers. I also know that the IceCube Neutrino Observatory would love to have a portable_binary_archive.

I had a look into the boost serialisation code, in particular, the incomplete portable_binary_archive in the examples directory and the external implementation of a portable binary archive, EOS. I also read the notes in the documentation of boost.serialization that you made about the incomplete implementation. It is understanding, that all it needs is a portable implementation of reading and writing floats and doubles. Is that correct?

I read that the author of EOS tried to submit to boost a while back. Apparently, this never happened. What was the issue?

I read up on floating point representation in binary, and I would approach the problem by implementing the IEEE754 standard in a portable way. Does that make sense?

Best regards,
Hans

[Adlai Shawareb]

> Thanks Robert,
>
> Regarding your suggestion to use text serialization. Since, over time, we will be adding and deleting data from the data set that we serialize, would the following be the right approach to managing the changes in the data that is de/serialized. I'm > > thinking that we would set the BOOST_CLASS_VERSION before de/serializing.
>
> {
> ar & mInt;
> If(version >= 2)
> ar & mFloat;
> If(version >= 3)
> ar & mInt2;
> ar & mFloat2;
> }
>
> Regards,

[Adlai Shawareb]

Have you had a chance to review this? I did look at the Outlook QuoteFix link on your website, but the Download section of that site is broken.

Regards,
Adlai

[Adlai Shawareb]

Thanks Robert,

Regarding your suggestion to use text serialization. Since, over time, we will be adding and deleting data from the data set that we serialize, would the following be the right approach to managing the changes in the data that is de/serialized. I'm thinking that we would set the BOOST_CLASS_VERSION before de/serializing.

{
ar & mInt;
If(version >= 2)
ar & mFloat;
If(version >= 3)
ar & mInt2;
ar & mFloat2;
}

Regards,
Adlai

On 1/25/17 12:04:00, Robert Ramey wrote:
> Right. No one as under taken that task. Should be fairly straight
> forward using spirit v3.
> Robert Ramey

[Robert Ramey]

On 1/26/17 1:20 PM, Adlai Shawareb wrote:
> Thanks Robert,
>
> Regarding your suggestion to use text serialization. Since, over time, we will be adding and deleting data from the data set that we serialize, would the following be the right approach to managing the changes in the data that is de/serialized. I'm thinking that we would set the BOOST_CLASS_VERSION before de/serializing.
>
> {
> ar & mInt;
> If(version >= 2)
> ar & mFloat;
> If(version >= 3)
> ar & mInt2;
> ar & mFloat2;
> }

This is the way that versioning is intended to be used. Note that this
verisoning is implemented for all archives types. So using XML wouldn't
add anything. I would say the only think you need to do is not use
binary archive. This might create difficulties if over the life of the
product, the compiler was upgraded. So if were me I'd use the text archive.

I've you're strapped for space, you can compress the text stream on the
fly. This makes for a small, fast, guaranteed portable archive.

I also can't stress enough that for this application, in addition to
running the serialization library test suite, you should add you're own
set of tests to test serialization on your own data types.

[Adlai Shawareb]

Thank you Robert. Is there a compression library that you think would be well suited for this?

Regards,
Adlai

On 1/29/17 10:03 AM, Robert Ramey wrote:
> This is the way that versioning is intended to be used. Note that this
> verisoning is implemented for all archives types. So using XML wouldn't
> add anything. I would say the only think you need to do is not use
> binary archive. This might create difficulties if over the life of the
> product, the compiler was upgraded. So if were me I'd use the text archive.
>
> I've you're strapped for space, you can compress the text stream on the
> fly. This makes for a small, fast, guaranteed portable archive.
>
> I also can't stress enough that for this application, in addition to
> running the serialization library test suite, you should add you're own
> set of tests to test serialization on your own data types.

[Robert Ramey]

On 1/31/17 6:21 PM, Adlai Shawareb wrote:
> Thank you Robert. Is there a compression library that you think would be well suited for this?

use boost.iostreams for this. The serialization library can a
boost.stream so you stackup any combination of modules from there. One
is a zip stream.