Fwd: [sirikata-devs] Re: KristalliProtocol work-in-progress specification]

[Ryan McDougall]

Forwarding Daniel's comments to kyoryoku so everyone can post.

Cheers,

---------- Forwarded message ----------
From: Daniel Reiter Horn <dani...@graphics.stanford.edu>
Date: Sun, Jan 31, 2010 at 9:41 PM
Subject: Re: [sirikata-devs] Re: KristalliProtocol work-in-progress specification]
To: platfo...@googlegroups.com
Cc: metaverseexc...@googlegroups.com, realxt...@googlegroups.com

I can't figure out how to edit the page, so I've included it in an HTML email to your list--
You guys forgot a row that's really important: native javascript support, because there's no magical "binding layer" in javascript that you can just paste in C++ code, and javascript does not allow for truly binary strings, so you have to base128 or base64 encode things or use firefox extensions for binary unicode.
This is why simply TCP support is insufficient for javascript support

Transport Layer Comparison Matrix

Feature	MXP0.5	MXP0.6	ENET	Kristalli	SLUDP	Raknet	TCPSST(Sirikata)
Connectivity	+	+	+	+	+	+	+
Optional guaranteed delivery	+	+	+	+	+	+	+
Message assembly	+	+	+	+		+	+
Message fragmentation	+	+	+	+		+	+
Multiplexing	+	+					+
Sequencing		+	+	+		+	+
Channels		+	+	+		+	+
Prioritization				+		+
Congestion avoidance		*	+	+		+	+
Compression					+	+
Encryption						+
Separated transport layer		+	+	+	+	+	+
Open Source License	+	+	+	?	+		+
Open Specification	+	+		+	+		+(WebSockets)
UDP support	+	+	+	+	+	+
TCP support		*		+		+	+
C++ Windows support	+	*	+	+	+	+	+
C++ Linux support	+	*	+		+	+	+
C++ Solaris support	^	*	+		?	?	+
C++ Mac support	+	*	+		+	+	+
Native C# support	+	*			+	^	(easy)
Native Java support	+	*			?
Native Javascript Support							+ (webSockets)
Native Python support	^	*			?
Binding Python support			+		?	+	+
	Mature and widely adopted implementation			+		+	+

'+' Supported, ' ' Not supported, '^' Developed, '*' Planned

On Sun, 2010-01-31 at 20:19 +0200, Ryan McDougall wrote:

You should include the sirikata guys, or maybe use the kyoryoku list,
to get a wider audience.

Cheers,

On Sat, Jan 30, 2010 at 9:24 AM, Tommi Laukkanen
<tommi.s.e...@gmail.com> wrote:
> Hello
> Here is link to the initial comparison matrix:
> http://www.bubblecloud.org/wiki/-/wiki/Main/Comparison
> Please notify me of any errors or ambiguities. You can also register and
> modify the wiki yourself and copy the matrix to your own materials.
> We could also add SLUDP there but I would need someone with experience to
> contribute that column or help me with the analysis.
> Regards,
> Tommi
>
>
> On Wed, Jan 27, 2010 at 5:52 PM, Tommi Laukkanen
> <tommi.s.e...@gmail.com> wrote:
>>
>> Hi
>>
>> I have been following Kirstalli-thread in your list but have not yet
>> found time to read through the spec and participate. I believe that
>> merge of Kristalli and MXP transport layer would launch the virtual
>> world protocol development to useful direction by combining the
>> strengths of our projects. We could start by creating a simple feature
>> level comparison chart between MXP 0.5, planned MXP 0.6, Kristalli and
>> ENET. This would give us high level view on the situation and a
>> starting point for merge considerations. From our point of view this
>> is a good time as we have managed to create pretty solid 0.6
>> specification with features we wanted to put in but the implementation
>> has not been started yet. It is good time to evaluate the feature set
>> and see what kind of improvements can be done by merge of these two
>> independent approaches.
>>
>> I will try and read through the specification by next Monday and
>> produce skeleton of the feature set comparison matrix. We can also add
>> list of protocol level innovations which should be specifically noted
>> when considering the final product.
>>
>> Apologies about cross posting but I thought this would be interesting
>> for readers in both projects.
>>
>> Yours sincerely,
>> Tommi Laukkanen
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Metaverse eXchange Protocol" group.
>> To post to this group, send email to
>> metaverseexc...@googlegroups.com.
>> To unsubscribe from this group, send email to
>> metaverseexchangep...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/metaverseexchangeprotocol?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Metaverse eXchange Protocol" group.
> To post to this group, send email to
> metaverseexc...@googlegroups.com.
> To unsubscribe from this group, send email to
> metaverseexchangep...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/metaverseexchangeprotocol?hl=en.
>

--
You received this message because you are subscribed to the Google Groups "Sirikata Dev List" group.
To post to this group, send email to platfo...@googlegroups.com.
To unsubscribe from this group, send email to platformtalk...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/platformtalk?hl=en.

[Daniel Reiter Horn]

strange---I had trouble getting it to the
metaverseexc...@googlegroups.com too

> +unsub...@googlegroups.com.

> For more options, visit this group at
> http://groups.google.com/group/platformtalk?hl=en.
>
>
>
>

> --
> http://groups.google.com/group/kyoryoku

[Daniel Reiter Horn]

Tommi wanted me to take this discussion to the kyoryoku list, replies
inline about the new protocol

On Sun, 2010-01-31 at 22:02 +0200, Tommi Laukkanen wrote:
> Hi Daniel
>
>
> Thank you for the TCPSST data. It looks very well shaped. Is TCPSST
> your existing protocol you use in Sirikata currently or a new thing
> being developed?
It is currently in use in sirikata! We've revved the master branch to
include the websocket specification
>
>
> Some clarifications:
>
>
> Optional guaranteed delivery is something you can do only with UDP as
> you need to be able to mark some of the messages unreliable so that
> those packets can be dropped by the network when they are in flight.
The API lets you mark packets as optional and the plan was to
potentially favor dropping them if the send queues are full and new data
needs to be added on the application side, but none of that has been
codified, and so it's probably not fair for us to check that--- but
there are some queues for double buffering during async send, so you
could imagine dropping there optionally. Right now we do have a
simulation for this functionality so you can develop against it, but
it's not good ;-)

> This is important for rapid resolving of congestion in the
> connection. TCP just keeps retransmitting all data which has been send
> to the wire. I need to clarify this to the wiki page.
We work around the head of line blocking issue by optionally allowing
more than 1 TCP connection to a place, so that in the case of loss at
least other streams can send. Then packets marked unordered will be
placed on the queues of less congested streams. So we have some
functionality but it's not directly comparable to the UDP analogs.

A reminder that we chose the TCP route so we could support all languages
including javascript (which does not have access to UDP for the
foreseeable future). We are developing a UDP component to the TCPSST
protocol that will allow for higher performing access to the same API,
but the implementation is still in its very early stages and right now
limited to our research code only.

>
>
> Could you provide me with link to WebSockets specification so I can
> add it to the wiki page?
http://tools.ietf.org/html/draft-hixie-thewebsocketprotocol-70

>
>
> You have ported your entire network library to pure JavaScript?
As of last night (I think) yes. If you need immediate access to the
code I can point it to you, but the webserver is my dev machine so I
wouldn't want to post my IP to a public list for everyone to hammer.
We'll be uploading in less than a week to github and it's already under
the BSD license, so if you ask me for early access and host it yourself
you can redistribute.
>
>
> I would appreciate if you replied here and quoted my question so we
> can move this discussion to kyoryoku list:
done
>
>
>
> Another thing which came to my mind: What kind of encoding you are
> using for in wire format as you need to be able to decode it with
> javascript? Do you have some json/xml like explicit encoding syntax or
> message/packet structure contract (like TCP/UDP packet structure for
> instance)?
> http://groups.google.com/group/kyoryoku/browse_thread/thread/8bf0f57e888480f5
>

So TCPSST, like websockets has null/0xff delimited packets and length
delimited packets. If a TCPSST compliant server receives a null/0xff
delimited packet then TCPSST expects items to be base64 encoded
(otehrwise it's raw binary)--Either way we use the exact google protocol
buffers format. We wrote a native javascript protobuf parser protojs
( http://github.com/sirikata/protojs ) that we use to encode these items
directly to base64.

The protobuf spec is well documented and open source BSD Licensed
implementations exist for all major languages that we know of (with our
contribution there being the notably missing javascript implementation)

Hope that answers your questions!
-Daniel
>
> Best regards,
> Tommi Laukkanen
>
>

[Ryan McDougall]

It seems to me that the differences between implementations are
relatively minor (excluding say SLUDP or RakNet). Is it not possible
to just pick one of them and use them uniformly? Or are those minor
differences worth it?

Cheers,

> --
> http://groups.google.com/group/kyoryoku

[Tommi Laukkanen]

Hi Ryan

At least we can say that there is quite good correlation on what is
the preferred feature set picked by people who have been developing
these transport protocols. However there is no alternative which
reaches all the key requirements including quality, performance, good
platform/language coverage, open source license and open
specification. The technical requirements are there for a reason and
those are last I would sacrifice as they will ultimately set the
boundaries for the virtual world experience which can be created by
using this transport protocol. Please do not say I am idealist :). I
believe I am pragmatist who can see problems if certain corners are
cut.

For near term everyone will continue with their chosen transport
protocol but on the long term I honestly believe some high quality
groundwork needs to be done. Best probability for good end result we
get by combining our forces to create at minimum a joined wish list
for this kind of protocol. It may be that we can actually reach that
by doing a bit of extra work to some existing alternative.

Sometimes it is not easy to look beyond next quarter but free agents
of open source world are perfect executors of long term development
exercises as they do not request investments from affiliated
companies. It may be enough that everyone just gives a small, honest,
encouraging push to get movement to correct direction.

-tommi

[Tommi Laukkanen]

Hi Daniel

I finally got to updating the comparison matrix with your data. Couple
of new thoughts and questions came to my mind:

Might need to divide some of the features to UDP specific section as
they are not applicable with TCP.

Message assembly and fragmentation are UDP specific. With TCP you
cannot affect how messages are placed network packets as this is done
automatically by TCP stack.

Double checking: Multiplexing in this comparison means that the long
messages are split to multiple small frames which are interleaved with
frames from other messages in the connection. Does TCPSST have this
kind of functionality currently?

What does channel concept mean in TCPSST and what kind benefits they
provide? I am trying to figure out how TCPSST channels align with the
channels in UDP implementations.

-tommi

[Daniel Reiter Horn]

On Mon, 2010-02-01 at 12:21 -0800, Tommi Laukkanen wrote:
> Hi Daniel
>
> I finally got to updating the comparison matrix with your data. Couple
> of new thoughts and questions came to my mind:
>
> Might need to divide some of the features to UDP specific section as
> they are not applicable with TCP.

ya perhaps...though since we're the only TCP finding the best analogy
might make sense

>
> Message assembly and fragmentation are UDP specific. With TCP you
> cannot affect how messages are placed network packets as this is done
> automatically by TCP stack.

ya so I guess don't check it (or check it but it's not under the control
of our app)

>
> Double checking: Multiplexing in this comparison means that the long
> messages are split to multiple small frames which are interleaved with
> frames from other messages in the connection. Does TCPSST have this
> kind of functionality currently?

well with more than 1 TCP stream, if you send unordered then the smaller
packets will find their way onto streams without the massive packet to
send--so it's somewhat analogous.

>
> What does channel concept mean in TCPSST and what kind benefits they
> provide? I am trying to figure out how TCPSST channels align with the
> channels in UDP implementations.

Basically if you have N TCP connections and M logical streams of control
(say messages from an object about position) then they can be
multiplexed in an N:M fashion. And if the packets are unordered then
they can be sent to any of the N streams (i.e. whichever has the least
packets). The other nice thing is that when we build a UDP layer
underneath we can use structured streams which will give us M actual
streams of control rather than N.
Let me link you to the structured streams paper since that has a lot of
ideas about how our API would work if it had UDP underneath
http://pdos.csail.mit.edu/uia/sst/
We tried to emulate that API but with TCP underneath--meaning we
couldn't perform as well or have all the features, but applications
using our API enable themselves to be optimized when someone writes the
UDP version under our API

>
> -tommi
>

[Tommi Laukkanen]

Very interesting page and already 2007.

Is either of these implemented in the version you are using in Sirikata currently:

• Dynamic prioritization of application's streams
• Optional end-to-end cryptographic security comparable to SSL

Here are some tough questions:

Have you had problems with keeping the aggregate connection (multiple TCP connections) going if network is performing poorly? What happens if single TCP connection gets disconnected?

What if large messages appear only occasionally in ordered streams/channels. Then we can not direct the stream or packet to TCP connection reserved for large messages and it would jam the current TCP connection and all other streams using it? You could work around this by implementing frame multiplexing so that you can push through the TCP connections multiple packets simultaneously in parallel rather than serial manner.

 

-tommi

[Tommi Laukkanen]

ya so I guess don't check it (or check it but it's not under the control
of our app)

As long as we do not separate UDP and TCP we should keep them checked as TCP does that automatically for you as many other useful features. Thats the nice thing about TCP.

-tommi