Serialization of primitive types

[gabor.dicso]

Hi all,

I would like to be able to serialize primitive types platform-
independently. I have hand-written Java data classes and I want to
serialize their primitive fields using a cross-platform framework.
These classes can not be generated, they must be written by hand,
additional code is generated based upon them. Also, serializing the
object as a whole isn't an option either, because the fields sometimes
have to be processed before serializing their values. I have to
serialize the fields separately. It must be made cross-platform
because the values will be stored in a database and they may be read
from other platforms. Creating wrapper PB-objects for each primitive
type is an overhead I must avoid because the operation will be done
very frequently and with large amounts of data.

I found that Protocol Buffers addresses cross-platform serialization
of objects, but I could not figure out how to use it as a
serialization framework for primitive types (without having
created .proto descriptors). Is it possible to use PB as a cross-
platform serializer-deserializer framework for primitive types?
Thanks,

Gabor Dicso

[Marc Gravell]

It sounds to me like you are over-complicating things. It is not uncommon to have a separate DTO model for serialization, so simply write a little code to map from your domain model (the comple model described above) to the DTO model (close to the serialization format).

It is possible to write a code-first protobuf implementation, which might also be a good fit here - but I don't know of one for java.

Marc

> --
> You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
> To post to this group, send email to prot...@googlegroups.com.
> To unsubscribe from this group, send email to protobuf+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
>

[Miguel Muñoz]

I agree with Marc. When things get complicated, it's a good idea to
separate your tasks. It seems like your java class, which generates
some of the data based on other data, is one issue, and your
serialization is a separate issue. (I know it would be nice to just
make that class serializable, but that may be where you make things
complicated.)

When I want to serialize my classes with protobufs, I create a
separate protobuf object to just handle serialization. Then I create a
utility class that transfers data between my protobuf object and my
java class. Then it's easy to add a constructor to my java class that
takes a protobuf object and defers the work to the utility class.

When I transfer data using protobufs, I don't convert to the protobuf
format until the last possible moment before sending, and I
immediately convert to the java class on receiving data. That lets me
put my protobuf objects behind a facade, so I don't need to know the
serialization details.

-- Miguel Muñoz

[Christopher Smith]

I think Gabor wants to avoid the overhead of implementing all that
additional bookkeeping as it'd slow down development. Something that
would effectively generate a protobuf descriptor so that it'd stay
consistent with changes in the Java code.

I would suggest looking at the protostuff project:

http://code.google.com/p/protostuff/

I think it has all that is needed to achieve the goals Gabor is looking for.

--Chris

2011/6/16 Miguel Muñoz <SwingG...@yahoo.com>:

> --
> You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
> To post to this group, send email to prot...@googlegroups.com.
> To unsubscribe from this group, send email to protobuf+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
>
>

--
Chris

[gabor.dicso]

Hello, thanks for all the responses. Sorry if my goals were not
perfectly clear. To put it simple, what I am looking for is
essentially something like "byte[] intBytes =
ProtoBuf.intToBytes(100);" / "int i =
ProtoBuf.readInt(bytesOfSerializedObject);" so that the values are
serialized and deserialized platform-independently. Such functionality
is obviously present in the ProtoBuf codebase. It would be great if it
was made public so I could use it in my code. To save CPU time and
memory, I want to avoid wrapping primitive types or any similar
workarounds, it's going to be a very frequent operation.

The reason why I need all this is that I must map the information
stored in the fields of my data classes to a list of special data
classes from third-party code. It's not a simple field-by-field
mapping, e.g. two fields may need to be mapped to the same third-party
class serialized after each other. This mapping is where I require
cross-platform binary serialization. All these operations must be made
by generated code based on annotations in my data classes. What at
first may seem like over-complicating things is actually my effort to
make things as simple and effective as possible. :)

As for now, I finally decided to go with Thrift's binary
serialization. But if it ever becomes possible to use ProtoBuf's
serialization functionality in a similar way, I'd definitely like to
hear about it. :)

Gabor

On Jun 16, 8:29 pm, Christopher Smith <cbsm...@gmail.com> wrote:
> I think Gabor wants to avoid the overhead of implementing all that
> additional bookkeeping as it'd slow down development. Something that
> would effectively generate a protobuf descriptor so that it'd stay
> consistent with changes in the Java code.
>
> I would suggest looking at the protostuff project:
>
> http://code.google.com/p/protostuff/
>
> I think it has all that is needed to achieve the goals Gabor is looking for.
>
> --Chris
>

> 2011/6/16 Miguel Muñoz <SwingGuy1...@yahoo.com>:

[Jason Hsueh]

I didn't really go through the whole thread, so I might have missed something, but it's not clear to me how you plan to distinguish different values in the data stream. But protobuf's serialization primitives are in CodedInputStream/CodedOutputStream.

[gabor.dicso]

Hi Jason, that's what I was looking for. Thanks for the info!

Gabor