C# alternative implementation

[Marc Gravell]

Short version: I have something that works and allows
[de]serialization and RPC, although it doesn't support the full
feature-set yet.

Detail:

I've been taking a look at protocol buffers from the C# (or
general .NET) angle. I know Jon is looking at a port, but I wanted to
see what would be involved in a ground-up approach using .NET concepts
from the outset. In particular, it seems that there is a lot of cross-
over between .proto and WCF data-contracts (data) / service-contracts
(RPC). By tying into the data-contract core, this would allow:

* existing data-contracts can use the proto binary format without
changes
* compatibility with "mex" generated classes
* compatibility with LINQ classes (LINQ-to-SQL can emit data-
contracts)
* simple code generation from .proto files
* use within WCF as an RPC layer
* existing WCF services can use the proto binary format without
changes

I've written an attribute based C# implementation that uses a one-time
(static ctor) discovery step to avoid lots of reflection, and I'm
after people's feedback.

Example C# code (entity and service - note DataContract etc are WCF
attributes):

[DataContract] // note optional Name etc
public class Test1 { // keep original names in meta
[DataMember(Name="a", IsRequired = true, Order = 1)]
public int A { get; set; }
}
[DataContract]
public class Test3 {
[DataMember(Name="c", IsRequired = true, Order = 3)]
public Test1 C { get; set; }
}
[ServiceContract(Name="ProtoService")]
public interface IProtoService {
[OperationContract, ProtoBehavior]
Test3 Get(Test1 request);
}

Note the ProtoBehaviourAttribute is responsible for swapping the
serialization engine, allowing granular control of which individual
methods are suitable for proto usage. Under the hood, since WCF is xml-
based it is using a base-64 element for the content, but I'm writing
it as a raw buffer; the hope is that this will allow the MTOM
formatter to detect this and swap for MTOM (not fully tested).

There are some other bespoke attributes for controlling (for example)
signed (zigzag) variants etc.

Anyway, it works [although isn't a complete proto implementation], and
is quick and small. The WCF trick is a handy way of getting a well-
tested RPC channel for free, although obviously the raw binary could
be used in any custom RPC stack.

Thoughts?

Marc

[Marc Gravell]

btw, example client and server:

=== client ===

static class Program {
static void Main() {
Test1 request = new Test1 { A = 12345};
Test3 response;
using (var proxy = new WcfProxy<IProtoService>()) {
response = proxy.Service.Get(request);
}
Console.WriteLine(response.C.A);
}
}

=== server ===

public class MyService : IProtoService {
public Test3 Get(Test1 request) {
return new Test3 { C = request };
}
}

=== end examples ===

where WcfProxy<T> is a simple subclass (unaware of proto) of WCF's
ClientBase<T>; it would also work with a "mex"-generated client - just
the subclass is simpler for demonstration.

Actually - the WCF specifics aren't too important (don't worry if you
aren't a .NET 3.0 person - obviously this is a mixed group!); I'm just
wondering if people think that this has potential...

Marc

[Torbjörn Gyllebring]

Seems like a really nice idea and personally I think it's neat to see
multiple implementations gives ample oppurtunity for cross-pollination
and let us (the community)
experiment with diffrent designs and tradeoffs.

@any googler:
Would be really nice if some sort of compliance test suite was
available for implementors is that possible?

[Jon Skeet]

On Jul 15, 10:12 am, Torbjörn Gyllebring

<torbjorn.gyllebr...@gmail.com> wrote:
> Seems like a really nice idea and personally I think it's neat to see
> multiple implementations gives ample oppurtunity for cross-pollination
> and let us (the community) experiment with diffrent designs and tradeoffs.

Agreed. (It helps that I've worked with Marc in the past, and enjoyed
the fruits of his labour :)

> @any googler:
> Would be really nice if some sort of compliance test suite was
> available for implementors is that possible?

That suggests more of a common API than may be the case. What may be
more feasible is to have some common artifacts that enable each
implementor to easily write their own compliance test suite :)

An obvious example would be a set of .protos, a description of values
to include in them, and then golden files to say "this is what the
serialized version should look like". From there it's pretty easy to
write unit tests to verify that each implementation can read and write
correctly. I believe some of this exists already (there are certainly
protos geared towards unit tests) but I don't know how comprehensive
it is. This would also go hand-in-hand with a performance benchmarking
suite.

Jon

[Torbjörn Gyllebring]

> That suggests more of a common API than may be the case. What may be
> more feasible is to have some common artifacts that enable each
> implementor to easily write their own compliance test suite :)
>
> An obvious example would be a set of .protos, a description of values
> to include in them, and then golden files to say "this is what the
> serialized version should look like". From there it's pretty easy to
> write unit tests to verify that each implementation can read and write
> correctly. I believe some of this exists already (there are certainly
> protos geared towards unit tests) but I don't know how comprehensive
> it is. This would also go hand-in-hand with a performance benchmarking
> suite.
>
> Jon

I couldn't agree more, thanks for reading my mind and actually
figuring out what I ment not what I said :)

[Marc Gravell]

For info, during a break I managed to get repeated elements working
(IList<T> in .NET parlance), allowing me to run some performance tests
against regular WCF; results are encouraging: ~x2 for local usage, and
~x2.5 for remote usage. Not bad for adding a single attribute.

Test uses a fairly large payload - a list of 1000 items (up and down
the wire), using http over a local link (same machine) and remote
(internet) pipe. I haven't got as far as enabling MTOM yet.

Of course, the proto serializer could also potentially be used to hook
into BinaryFormatter to boost remoting performance (I haven't tried).

Some numbers below; the only difference between the "proto" and
"standard" test is the addition of the [ProtoBehavior] marker.

Marc

[anybody is welcome to the code if they are interested / want to
validate / etc]

---- results ----

Standard WCF serialization (local):

500 calls; 21811ms
500 calls; 20721ms
500 calls; 21121ms

Proto serialization (local):

500 calls; 11886ms
500 calls; 12779ms
500 calls; 11642ms

Standard WCF serialization (remote):

20 calls; 34069ms
20 calls; 33959ms
20 calls; 33950ms

Proto serialization (remote):

20 calls; 13631ms
20 calls; 13672ms
20 calls; 14148ms

[Radik]

It looks good!
Can you share the source code?

[sweet...@gmail.com]

Oh yes please! I have a project that could use that _right_now_! :)

-(e)

[Marc Gravell]

Re sharing, absolutely [in fact it is already available]. It is very
encouraging to see interest ;-p

But I am aware of several critical bugs that need urgent fixing:
* the base-128 is borked for large numbers
* my test rig shows that somehow I've broken sub-entity
deserialization [it worked yesterday...]

I also need to come up with a standalone WCF example [my previous rig
needed 3 projects and 2 machines]

I hope to have something a bit more robust after the weekend; until
then, you are welcome to look just to see the rough approach used, but
please: it is *very* early at the moment...

The repo is here: http://code.google.com/p/protobuf-net/
You can grab the trunk via svn, but there aren't any formal releases
yet (for obvious reasons).

But the short version is that if you are familiar with data-contracts,
this code will be immediately familiar.

I have lots of plans to improve/extend this, but all feedback
welcome...

Marc

[Jon Skeet]

On Jul 17, 9:22 pm, Marc Gravell <marc.grav...@gmail.com> wrote:
> Re sharing, absolutely [in fact it is already available]. It is very
> encouraging to see interest ;-p
>
> But I am aware of several critical bugs that need urgent fixing:
> * the base-128 is borked for large numbers

I know we're trying to avoid each other's codebases at the moment (to
avoid confusion, not due to legal or NIH concerns) but you could
probably lift my port of the base-128 stuff quite easily. It's in
CodedInputStream and CodedOutputStream.

Jon

[Marc Gravell]

That will probably save me some brain-ache; I'll take a peek later,
cheers. [I'm not used to counting in 7s...]

Hopefully that will also provide a quick fix to the "negative int32
deserializes as 8 bytes" issue...

Marc

[Marc Gravell]

> I hope to have something a bit more robust after the weekend

Fixed the base-128 and sub-entity deserialization, which makes it
usable for tentative investigations. Still lots to do, though... but
it will get your messages in/out, and will work happily with WCF/MTOM.

Marc