[protobuf] Avoid attributes/ Cyclic support
501 views
Skip to first unread message
Fabio Maulo
May 4, 2010, 2:29:18 AM5/4/10
to Protocol Buffers
- Is there a way to avoid attributes ?
I mean something like an "internal" metadata where we can define the
identifier of each member.
- protobuff supports cyclic references?
--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To post to this group, send email to prot...@googlegroups.com.
To unsubscribe from this group, send email to protobuf+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
I mean something like an "internal" metadata where we can define the
identifier of each member.
- protobuff supports cyclic references?
--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To post to this group, send email to prot...@googlegroups.com.
To unsubscribe from this group, send email to protobuf+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Jason Hsueh
May 4, 2010, 12:37:44 PM5/4/10
to Fabio Maulo, Protocol Buffers
On Mon, May 3, 2010 at 11:29 PM, Fabio Maulo <fabio...@gmail.com> wrote:
- Is there a way to avoid attributes ?
I mean something like an "internal" metadata where we can define the
identifier of each member.
I'm not sure what you mean by this. By attributes do you mean field options? Can you describe in more detail what you want to do?
- protobuff supports cyclic references?
Yes, but mutual recursion is only supported if the types are defined within the same file. You aren't allowed to have cycles in file imports.
--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To post to this group, send email to prot...@googlegroups.com.
To unsubscribe from this group, send email to protobuf+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
Marc Gravell
May 4, 2010, 3:42:53 PM5/4/10
to Fabio Maulo, Protocol Buffers
You mention attributes; what implementation are you using? protobuf-net (one of the .NET implementations) uses .NET attributes as one mechanism for describing fields; is this what you mean? If it is, then I have a soon-to-be-released "v2" of protobuf-net which indeed allows independent metadata (as well as supporting the attributes for "v1" usage and .proto for code-generation).
Re cyclic references; the protobuf format is intrinsically a "tree" serializer (rather than a "graph" serializer), but this is something that occasionally comes up. I plan on trying a spike after "v2" to see what might be possible (it would be a legal protobuf stream, but would not be much use for interop unless something wider could be agreed, ideally coming from Kenton et al).
If you /aren't/ referring to protobuf-net, then just ignore me ;-p
Marc
--
Regards,
Marc
Regards,
Marc
Fabio Maulo
May 4, 2010, 4:08:44 PM5/4/10
to Protocol Buffers
I can't ignore you ;)
What I mean is exactly a "independent" metadata.
Something like
Serializer.Serialize<T>(Stream dataStream, T object, IDictionary<Type,
SerializerMetaData> metaHolder);
Because protobuf-net supports different set of meta-data (embedded,
DataContract and so on) the most easy way to be DRY is converting any
"meta-source" to a common source and then perform the serialization/
deserialization.
protobuf-net may hold these converted-meta-data in some place to avoid
the usage of reflection to investigate the meta-data declared by
attributes.
At the same time, the user can use some alternative ways to define
meta-data of serialization using:
- available attributes of v1
- his custom attributes
- fluent-configuration
- conventions
- whatever his fantasy can produce
These stuff are not so important... well... the concept of "Don't
touch my existing implementation" is pretty important for me ;)
On 4 mayo, 16:42, Marc Gravell <marc.grav...@gmail.com> wrote:
> You mention attributes; what implementation are you using? protobuf-net (one
> of the .NET implementations) uses .NET attributes as one mechanism for
> describing fields; is this what you mean? If it is, then I have a
> soon-to-be-released "v2" of protobuf-net which indeed allows independent
> metadata (as well as supporting the attributes for "v1" usage and .proto for
> code-generation).
>
> Re cyclic references; the protobuf format is intrinsically a "tree"
> serializer (rather than a "graph" serializer), but this is something that
> occasionally comes up. I plan on trying a spike after "v2" to see what might
> be possible (it would be a legal protobuf stream, but would not be much use
> for interop unless something wider could be agreed, ideally coming from
> Kenton et al).
>
> If you /aren't/ referring to protobuf-net, then just ignore me ;-p
>
> Marc
>
What I mean is exactly a "independent" metadata.
Something like
Serializer.Serialize<T>(Stream dataStream, T object, IDictionary<Type,
SerializerMetaData> metaHolder);
Because protobuf-net supports different set of meta-data (embedded,
DataContract and so on) the most easy way to be DRY is converting any
"meta-source" to a common source and then perform the serialization/
deserialization.
protobuf-net may hold these converted-meta-data in some place to avoid
the usage of reflection to investigate the meta-data declared by
attributes.
At the same time, the user can use some alternative ways to define
meta-data of serialization using:
- available attributes of v1
- his custom attributes
- fluent-configuration
- conventions
- whatever his fantasy can produce
These stuff are not so important... well... the concept of "Don't
touch my existing implementation" is pretty important for me ;)
On 4 mayo, 16:42, Marc Gravell <marc.grav...@gmail.com> wrote:
> You mention attributes; what implementation are you using? protobuf-net (one
> of the .NET implementations) uses .NET attributes as one mechanism for
> describing fields; is this what you mean? If it is, then I have a
> soon-to-be-released "v2" of protobuf-net which indeed allows independent
> metadata (as well as supporting the attributes for "v1" usage and .proto for
> code-generation).
>
> Re cyclic references; the protobuf format is intrinsically a "tree"
> serializer (rather than a "graph" serializer), but this is something that
> occasionally comes up. I plan on trying a spike after "v2" to see what might
> be possible (it would be a legal protobuf stream, but would not be much use
> for interop unless something wider could be agreed, ideally coming from
> Kenton et al).
>
> If you /aren't/ referring to protobuf-net, then just ignore me ;-p
>
> Marc
>
Marc Gravell
May 4, 2010, 4:22:29 PM5/4/10
to Fabio Maulo, Protocol Buffers
"v2" does all of this, and more. The existing ("v1") attributes essentially just become the /default/ implementation. There is a new "TypeModel" class (for want of a better name), which functions a bit like "XmlSerializer", "DataContractSerializer", etc.
- You can have as many TypeModel instances as you like, representing different data or the same data with different schemas.
- TypeModel is abstract; there is a concrete RuntimeTypeModel, which you can configure with types (classes and structs) / fields / callbacks / inheritance / surrogates / enums / etc to your heart's content
- There is *full* inbuilt meta-programming, so you can use CompileInPlace() to compile a RuntimeTypeModel in-situ (useful for conventions etc)
- Or you can even use Compile() to write the model to a standalone dll with a separate concrete TypeModel implementation matching your model - useful for things like iPhone, Phone 7
The "v2" stuff is incomplete but fairly stable. It should be ready in a few weeks. It is available in SVN if you want to try it, but there are still some gaps (primarily extension data, mapped enums [unmapped enums are fine, though] and the *WithLengthPrefix etc variants).
Does that serve?
Marc
Fabio Maulo
May 4, 2010, 4:37:35 PM5/4/10
to Protocol Buffers
For sure! Marc.
I'm working in a new commercial prj for win-azure.
The persistence will look like a doc DB (well... pardon me... it will
be my interpretation).
I have a base test for IDocumentSerializer<TDocument> and I have tried
some others serializer and, so far, I'm stuck with .NET binary
formatter.
Looking forward for protobuf-net V2.
Thanks Marc.
On 4 mayo, 17:22, Marc Gravell <marc.grav...@gmail.com> wrote:
> "v2" does all of this, and more. The existing ("v1") attributes essentially
> just become the /default/ implementation. There is a new "TypeModel" class
> (for want of a better name), which functions a bit like "XmlSerializer",
> "DataContractSerializer", etc.
>
> - You can have as many TypeModel instances as you like, representing
> different data or the same data with different schemas.
> - TypeModel is abstract; there is a concrete RuntimeTypeModel, which you can
> configure with types (classes and structs) / fields / callbacks /
> inheritance / surrogates / enums / etc to your heart's content
> - There is *full* inbuilt meta-programming, so you can use CompileInPlace()
> to compile a RuntimeTypeModel in-situ (useful for conventions etc)
> - Or you can even use Compile() to write the model to a standalone dll with
> a separate concrete TypeModel implementation matching your model - useful
> for things like iPhone, Phone 7
>
> The "v2" stuff is incomplete but fairly stable. It should be ready in a few
> weeks. It is available in SVN if you want to try it, but there are still
> some gaps (primarily extension data, mapped enums [unmapped enums are fine,
> though] and the *WithLengthPrefix etc variants).
>
> Does that serve?
>
> Marc
>
> > protobuf+u...@googlegroups.com<protobuf%2Bunsubscribe@googlegroups.c om>
> > .
> > .
I'm working in a new commercial prj for win-azure.
The persistence will look like a doc DB (well... pardon me... it will
be my interpretation).
I have a base test for IDocumentSerializer<TDocument> and I have tried
some others serializer and, so far, I'm stuck with .NET binary
formatter.
Looking forward for protobuf-net V2.
Thanks Marc.
On 4 mayo, 17:22, Marc Gravell <marc.grav...@gmail.com> wrote:
> "v2" does all of this, and more. The existing ("v1") attributes essentially
> just become the /default/ implementation. There is a new "TypeModel" class
> (for want of a better name), which functions a bit like "XmlSerializer",
> "DataContractSerializer", etc.
>
> - You can have as many TypeModel instances as you like, representing
> different data or the same data with different schemas.
> - TypeModel is abstract; there is a concrete RuntimeTypeModel, which you can
> configure with types (classes and structs) / fields / callbacks /
> inheritance / surrogates / enums / etc to your heart's content
> - There is *full* inbuilt meta-programming, so you can use CompileInPlace()
> to compile a RuntimeTypeModel in-situ (useful for conventions etc)
> - Or you can even use Compile() to write the model to a standalone dll with
> a separate concrete TypeModel implementation matching your model - useful
> for things like iPhone, Phone 7
>
> The "v2" stuff is incomplete but fairly stable. It should be ready in a few
> weeks. It is available in SVN if you want to try it, but there are still
> some gaps (primarily extension data, mapped enums [unmapped enums are fine,
> though] and the *WithLengthPrefix etc variants).
>
> Does that serve?
>
> Marc
>
> > .
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Protocol Buffers" group.
> > To post to this group, send email to prot...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > protobuf+u...@googlegroups.com<protobuf%2Bunsubscribe@googlegroups.c om>
> > You received this message because you are subscribed to the Google Groups
> > "Protocol Buffers" group.
> > To post to this group, send email to prot...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > .
Marc Gravell
May 4, 2010, 5:09:55 PM5/4/10
to Fabio Maulo, Protocol Buffers
Please allow me to say something loudly:
**Do not use BinaryFormatter for persistence.** (IMO)
That road is full of problems; even assuming you never want to do anything except MS/.NET, it is still hugely limiting around your assemblies. I've lost track of people asking "I rebuilt my dll; now none of my data loads; what broke?". BinaryFormatter is OK *at a stretch* for comms (in particular inter-AppDomain in the same process), as you aren't going to get bitten (as often) by binary problems - but it is very brittle around versioning.
It sounds like "v2" will do what you want; /please/ don't go BinaryFormatter... (for your sake, not mine).
Marc
Marc Gravell
May 4, 2010, 5:16:27 PM5/4/10
to Fabio Maulo, Protocol Buffers
Oh, I should also add that BinaryFormatter is slow(er) and uses more bandwitdh (very important when IO is your biggest bottleneck). This is based on "v1":
--
Regards,
Marc
"v2" should give the same bandwidth numbers (else I've borked it), but should be faster again. The problem, of course, is that protobuf doesn't support object references (meaning: full graph support). This is something I plan to play with, but:
a: I can't guarantee when (I do this on my own time)
b: it would be an implementation-specific feature
Marc
On 4 May 2010 21:37, Fabio Maulo <fabio...@gmail.com> wrote:
--
Regards,
Marc
Fabio Maulo
May 4, 2010, 5:18:41 PM5/4/10
to Protocol Buffers
LOL!!
I know Marc, I know... at this moment the dll version is not the
problem and the DocumentSerializer is injectable.
What I need is an efficient, high speed but overall not invasive and
supporting cyclic reference serializer.
Cyclic as:
Parent
IEnumerable<Child> Children
Child
Parent Parent
I'll checkout the protobuff-net V2 at some moment.
Thanks again.
On 4 mayo, 18:09, Marc Gravell <marc.grav...@gmail.com> wrote:
> Please allow me to say something loudly:
>
> **Do not use BinaryFormatter for persistence.** (IMO)
>
> That road is full of problems; even assuming you never want to do anything
> except MS/.NET, it is still hugely limiting around your assemblies. I've
> lost track of people asking "I rebuilt my dll; now none of my data loads;
> what broke?". BinaryFormatter is OK *at a stretch* for comms (in particular
> inter-AppDomain in the same process), as you aren't going to get bitten (as
> often) by binary problems - but it is very brittle around versioning.
>
> It sounds like "v2" will do what you want; /please/ don't go
> BinaryFormatter... (for your sake, not mine).
>
> Marc
>
> > <protobuf%2Bunsubscr...@googlegroups.c om>
I know Marc, I know... at this moment the dll version is not the
problem and the DocumentSerializer is injectable.
What I need is an efficient, high speed but overall not invasive and
supporting cyclic reference serializer.
Cyclic as:
Parent
IEnumerable<Child> Children
Child
Parent Parent
I'll checkout the protobuff-net V2 at some moment.
Thanks again.
On 4 mayo, 18:09, Marc Gravell <marc.grav...@gmail.com> wrote:
> Please allow me to say something loudly:
>
> **Do not use BinaryFormatter for persistence.** (IMO)
>
> That road is full of problems; even assuming you never want to do anything
> except MS/.NET, it is still hugely limiting around your assemblies. I've
> lost track of people asking "I rebuilt my dll; now none of my data loads;
> what broke?". BinaryFormatter is OK *at a stretch* for comms (in particular
> inter-AppDomain in the same process), as you aren't going to get bitten (as
> often) by binary problems - but it is very brittle around versioning.
>
> It sounds like "v2" will do what you want; /please/ don't go
> BinaryFormatter... (for your sake, not mine).
>
> Marc
>
> > > > .
> > > > > For more options, visit this group athttp://
> > > > groups.google.com/group/protobuf?hl=en.
>
> > > > --
> > > > You received this message because you are subscribed to the Google
> > Groups
> > > > "Protocol Buffers" group.
> > > > To post to this group, send email to prot...@googlegroups.com.
> > > > To unsubscribe from this group, send email to
> > > > protobuf+u...@googlegroups.com<protobuf%2Bunsubscribe@googlegroups.c om>
> > <protobuf%2Bunsubscr...@googlegroups.c om>
> > > > > For more options, visit this group athttp://
> > > > groups.google.com/group/protobuf?hl=en.
>
> > > > --
> > > > You received this message because you are subscribed to the Google
> > Groups
> > > > "Protocol Buffers" group.
> > > > To post to this group, send email to prot...@googlegroups.com.
> > > > To unsubscribe from this group, send email to
> > > > protobuf+u...@googlegroups.com<protobuf%2Bunsubscribe@googlegroups.c om>
Reply all
Reply to author
Forward
0 new messages