[Fwd: [realXtend-dev] KristalliProtocol work-in-progress specification]

[Toni Alatalo]

For your information,

there's a first draft of a specification of another protocol with
similar goals than in MXP - it's called Kristalli and Jukka has made it
earlier (a year or couple ago) for games and some personal projects of
his. We have been talking about MXP, enet, sirikata / google protobufs
etc. as possible alternatives to the SLUDP which currently use in
realXtend, due to the Second Life legacy. I asked Jukka to document his
protocol also 'cause knew that you people were specifying and
implementing a udp transport layer, which Kristalli also includes.

I think this documents it quite nicely, and is not huge, so perhaps
interesting for you folks to check out and give feedback and insight on
similarities/differences w.r.t how things are in MXP now. After reading
this myself, I read some of the ENet docs, but they don't actually
document the networking -- features are largely similar though, a
notable difference being that enet has channels whereas Kristalli
doesn't. Haven't had a chance to read current MXP docs yet, someone here
will at some point - this is not anything urgent for us, we'll be using
the current networking code in reX at least some months still.

Am hoping that we can perhaps merge this effort with MXP or if MXP
already does the same things as well or better, just switch to MXP - or
to enet .. for serialization I think we probably want to use google
protobufs in any case.

~Toni

[Tommi Laukkanen]

Hi

I have been following Kirstalli-thread in your list but have not yet
found time to read through the spec and participate. I believe that
merge of Kristalli and MXP transport layer would launch the virtual
world protocol development to useful direction by combining the
strengths of our projects. We could start by creating a simple feature
level comparison chart between MXP 0.5, planned MXP 0.6, Kristalli and
ENET. This would give us high level view on the situation and a
starting point for merge considerations. From our point of view this
is a good time as we have managed to create pretty solid 0.6
specification with features we wanted to put in but the implementation
has not been started yet. It is good time to evaluate the feature set
and see what kind of improvements can be done by merge of these two
independent approaches.

I will try and read through the specification by next Monday and
produce skeleton of the feature set comparison matrix. We can also add
list of protocol level innovations which should be specifically noted
when considering the final product.

Apologies about cross posting but I thought this would be interesting
for readers in both projects.

Yours sincerely,
Tommi Laukkanen

[Tommi Laukkanen]

Hello

Here is link to the initial comparison matrix:

http://www.bubblecloud.org/wiki/-/wiki/Main/Comparison

Please notify me of any errors or ambiguities. You can also register and modify the wiki yourself and copy the matrix to your own materials.

We could also add SLUDP there but I would need someone with experience to contribute that column or help me with the analysis. 

Regards,

Tommi

--
You received this message because you are subscribed to the Google Groups "Metaverse eXchange Protocol" group.
To post to this group, send email to metaverseexc...@googlegroups.com.
To unsubscribe from this group, send email to metaverseexchangep...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/metaverseexchangeprotocol?hl=en.

[Kripken]

Some notes on ENet:

1. ENet splits large messages into fragments of optimal size and automatically reassembles them, and vice versa clumps small messages together, also automatically. Is this perhaps relevant to what is meant by "message frames"?

2. ENet is well documented, both in the code and out. I guess it lacks a particularly formal 'spec' document, though. It can probably be quickly written, though.

3. ENet works on C++ on Macs (we use it there, in fact), so that should be corrected.

4. ENet has Python bindings, http://code.google.com/p/pyenet/

5. I think the maturity of the implementations, whether they have been used in production and for how long, etc., is a feature the comparison should mention.

- kripken

[Tommi Laukkanen]

Hi Kripken

Thanks for the clarifications I have updated the list accordingly. The
specification of the ENET protocol would be valuable addition to ENET
and it would be great if it could be put together.

The message frames refers a feature where message frame size is
limited to be considerable smaller than packet size so that each
packet carries multiple frames even if the message sizes are large. I
am not saying this is the best approach but a differentiating feature.
Is ENET behaving this way? For example if you would send ten large
messages simultaneously then all the outgoing packets would carry
frames from all of these messages simultaneously. Maybe this feature
could be named differently like message frame multiplexing.

-tommi

On Jan 30, 11:17 am, Kripken <kripkenste...@gmail.com> wrote:
> Some notes on ENet:
>
> 1. ENet splits large messages into fragments of optimal size and
> automatically reassembles them, and vice versa clumps small messages
> together, also automatically. Is this perhaps relevant to what is meant by
> "message frames"?
>
> 2. ENet is well documented, both in the code and out. I guess it lacks a
> particularly formal 'spec' document, though. It can probably be quickly
> written, though.
>
> 3. ENet works on C++ on Macs (we use it there, in fact), so that should be
> corrected.
>

> 4. ENet has Python bindings,http://code.google.com/p/pyenet/

> >> metaverseexchangep...@googlegroups.com<metaverseexchangepro tocol%2Bunsu...@googlegroups.com>

> >> .
> >> For more options, visit this group at
> >>http://groups.google.com/group/metaverseexchangeprotocol?hl=en.
>
> >  --
> > You received this message because you are subscribed to the Google Groups
> > "Metaverse eXchange Protocol" group.
> > To post to this group, send email to
> > metaverseexc...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > metaverseexchangep...@googlegroups.com<metaverseexchangepro tocol%2Bunsu...@googlegroups.com>

[Kripken]

Hmm, I don't completely understand what you mean, sorry.

But here is how ENet works, AFAIK: If you have a packet size of 1024, and messages of sizes 512 (message A), and 2048 (message B), they will be sent out as follows:

packet 1: A + first 512 of B
packet 2: middle 1024 of B
packet 3: last 512 of B

The example shows both the automatic aggregation and splitting up features. (It ignores headers, etc.)

- kripken

To unsubscribe from this group, send email to metaverseexchangep...@googlegroups.com.

[Tommi Laukkanen]

Could you demonstrate what happens if you had message C with size of 2048 as well added as third message to that example?

-tommi

[Kripken]

With messages:

A - 512 bytes
B - 2048 bytes
C - 2048 bytes

If the packet size is 1024 then the packets ENet would send are

1: A + 1/4 of B
2: 1/2 of B
3: 1/4 of B and 1/4 of C
4: 1/2 of C
5: 1/4 of C

- kripken

[Tommi Laukkanen]

Ok

So the messages are sent in a queue where as in multiplexing approach they are split to multiple small fragments and interleaved to same packets:

1) A (F1-256) B (F1-256) C (F1-256) A (F2-256)

2) B (F2-256) C (F2-256) B (F3-256) C (F3-256) 

3) B (F3-256) C (F3-256)

In MXP 0.6 the packets being multiplexed are chosen from different channels if possible so this allows different channels to share packets in a fair manner. I changed the feature name to multiplexing in the matrix to avoid confusion.

- tommi

[Kripken]

Something is odd here. If the order is A,B,C, and you interleave them as so, then (for example) B's final fragment may arrive later than it would if you sent all of B before you send any of C. So this adds latency. What is the benefit of such interleaving inside a channel?

- kripken

[Tommi Laukkanen]

If you had just single channel there would be no benefit. The benefit is that multiple channels can be multiplexed to the same packet stream fairly. Otherwise single long message in one channel can jam the entire connection and prevent other channels from transmitting.

-tommi

[Kripken]

Oh, ok, now I understand you. My example was with A,B,C in a single channel, not separate ones as in yours.

I don't know how ENet handles the multiple channel issue. However, for the type of games it is usually used for, it probably doesn't matter anyhow (for the same reason prioritization doesn't matter - if you get to the point where they do make a difference, you are already lagged).

- kripken

On Sun, Jan 31, 2010 at 6:48 AM, Tommi Laukkanen <tommi.s.e...@gmail.com> wrote:

If you had just single channel there would be no benefit. The benefit is that multiple channels can be multiplexed to the same packet stream fairly. Otherwise single long message in one channel can jam the entire connection and prevent other channels from transmitting.

-tommi

--

[Tommi Laukkanen]

True most games do not need transmit large chunks in the middle of map. Virtual worlds may have occasionally need to transmit large state messages for objects or even media. Personally I prefer moving media over http but I can see that low priority media pushes from server to client could be useful in some situations.

-tommi

[Ryan McDougall]

You should include the sirikata guys, or maybe use the kyoryoku list,
to get a wider audience.

Cheers,