Pysage networking interface restrictive?

[bigj...@gmail.com]

Currently, pysage "outsources" the networking transport to RakNet.

There are a couple of reasons it is used:

1. RakNet is built on top of UDP, which is lightweight (designed for
games)
2. it supports optionally: in order and/or reliable packet delivery on
top of UDP
3. I would like to focus my effort in enhancing pysage instead of
writing a UDP network library

However, this may be a little restrictive since RakNet is not free for
commercial use (although not expensive).

I'm thinking to provide several options to the network stack and let
the end user choose which one to use. Thus, users can choose to
plugin a TCP stack, or a RakNet stack, or plugin user's own defined
network interface.

Pysage already has this interface somewhat defined. After some code
cleanup and change RakNet as an optional dependency, it should be good
to go.

[Cosmin Stejerean]

On Sat, Mar 14, 2009 at 9:58 PM, <bigj...@gmail.com> wrote:

Currently, pysage "outsources" the networking transport to RakNet.

There are a couple of reasons it is used:

1. RakNet is built on top of UDP, which is lightweight (designed for
games)
2. it supports optionally: in order and/or reliable packet delivery on
top of UDP
3. I would like to focus my effort in enhancing pysage instead of
writing a UDP network library

However, this may be a little restrictive since RakNet is not free for
commercial use (although not expensive).

I'm thinking to provide several options to the network stack and let
the end user choose which one to use.  Thus, users can choose to
plugin a TCP stack, or a RakNet stack, or plugin user's own defined
network interface.

I like the idea of making dependency on RakNet optional and allowing users to utilize alternative network backends that are completely free.

--
Cosmin Stejerean
http://offbytwo.com

[arschles]

I like that idea, too. I guess that means that pysage would have to
have a UDP abstraction layer. Could this layer abstract not only the
network backend but also the protocol (so that pysage could utilize
TCP later, or even higher level protocols like HTTP)?

On Mar 16, 11:10 am, Cosmin Stejerean <cstejer...@gmail.com> wrote:

[John Yang]

Currently, the network abstraction layer interface is pretty high
level. I believe that it is a good idea to leave the protocol as
implementation.

Take a look at "Transport" class here:
http://code.google.com/p/pysage/source/browse/trunk/pysage/transport.py

If the user needs a different protocol aside from the common ones,
he/she just needs to implement that interface and plugin user's own
transport class. A TCP and a http/XML stack would be very good
additions, especially the TCP one, as some users may not need the
performance of UDP.

John

[arschles]

Sorry, I spoke too soon on the "high level API" think. I just saw the
transport class today.

What do you think about an HTTP/JSON message passing stack? In this
stack, it would be fairly easy to serialize and deserialize messages
using simplejson (or json for later python distros).

[John Yang]

Actually, thinking about this again, I'm not so sure why the end users
would want to use other serialization methods than binary packing (how
it is currently). Binary packing is much more efficient than
ascii-like methods or pickle. However, it is a bit less convenient.
For example, you do need to specify the attribute types in terms of c
types for every message that you intend to send across processes or
network. Pysage also provides a few custom types on top of ctypes.

Refer to:

http://code.google.com/p/pysage/wiki/NetworkMessage

The idea is to provide a high level API for defining a message, but
underneath provide the most efficient way for serialization.
Optimally, the end users only interact with message instances end to
end.

Having said that, we do need an abstraction layer on top of the
network protocol, for which the serialized data is delivered. The
scenarios can be if the user wants to avoid a UDP library like raknet
for licensing reasons, and use raw UDP or TCP; or to provide
encryption on top of the data, etc.

[arschles]

> Actually, thinking about this again, I'm not so sure why the end users
> would want to use other serialization methods than binary packing (how
> it is currently). Binary packing is much more efficient than
> ascii-like methods or pickle. However, it is a bit less convenient.
> For example, you do need to specify the attribute types in terms of c
> types for every message that you intend to send across processes or
> network. Pysage also provides a few custom types on top of ctypes.

Agreed. However, the place that I could see an HTTP transport method
(ie: that sends JSON data) would be for easy interoperability between
pysage and other code that wants to communicate with pysage actors.
Sounds like that is something that can wait until we get the
networking stack in order. What do you think?

> Having said that, we do need an abstraction layer on top of the
> network protocol, for which the serialized data is delivered

As I understand, a transport derivative could be created that doesn't
use RakNet at all and implements "raw" UDP passing. I'm going to look
at implementing that.

[John Yang]

On Fri, Apr 10, 2009 at 12:34 AM, arschles <arsc...@gmail.com> wrote:
>
>> Actually, thinking about this again, I'm not so sure why the end users
>> would want to use other serialization methods than binary packing (how
>> it is currently).  Binary packing is much more efficient than
>> ascii-like methods or pickle.  However, it is a bit less convenient.
>> For example, you do need to specify the attribute types in terms of c
>> types for every message that you intend to send across processes or
>> network.  Pysage also provides a few custom types on top of ctypes.
>

> Agreed. However, the place that I could see an HTTP transport method
> (ie: that sends JSON data) would be for easy interoperability between
> pysage and other code that wants to communicate with pysage actors.
> Sounds like that is something that can wait until we get the
> networking stack in order. What do you think?

Sounds good.

>
>> Having said that, we do need an abstraction layer on top of the
>> network protocol, for which the serialized data is delivered
>
> As I understand, a transport derivative could be created that doesn't
> use RakNet at all and implements "raw" UDP passing. I'm going to look
> at implementing that.

Yea, both a raw TCP and UDP transport would be valuable, especially
the TCP transport since it's much more stable than UDP in
"non-realtime" applications such as first person shooter games.

I'm going to look into cleaning up the API a little bit in my next
tasks. After that, let's discuss how to implement a way to send
simple messages without crafting a message class like you had
mentioned.

[John Yang]

On Fri, Apr 10, 2009 at 10:09 PM, John Yang <bigj...@gmail.com> wrote:
> On Fri, Apr 10, 2009 at 12:34 AM, arschles <arsc...@gmail.com> wrote:
>>
>>> Actually, thinking about this again, I'm not so sure why the end users
>>> would want to use other serialization methods than binary packing (how
>>> it is currently).  Binary packing is much more efficient than
>>> ascii-like methods or pickle.  However, it is a bit less convenient.
>>> For example, you do need to specify the attribute types in terms of c
>>> types for every message that you intend to send across processes or
>>> network.  Pysage also provides a few custom types on top of ctypes.
>>
>> Agreed. However, the place that I could see an HTTP transport method
>> (ie: that sends JSON data) would be for easy interoperability between
>> pysage and other code that wants to communicate with pysage actors.
>> Sounds like that is something that can wait until we get the
>> networking stack in order. What do you think?
>
> Sounds good.
>
>>
>>> Having said that, we do need an abstraction layer on top of the
>>> network protocol, for which the serialized data is delivered
>>
>> As I understand, a transport derivative could be created that doesn't
>> use RakNet at all and implements "raw" UDP passing. I'm going to look
>> at implementing that.
>
> Yea, both a raw TCP and UDP transport would be valuable, especially
> the TCP transport since it's much more stable than UDP in
> "non-realtime" applications such as first person shooter games.

I meant applications other than realtime apps like first person
shooter games could use TCP.

[arschles]

I created one new issue for the "raw" UDP transport layer (not sure
how to change the issue type or owner).

I'm almost done with the UDP code, and I'll send a patch when I get it
tested a bit more.

On Apr 10, 10:15 pm, John Yang <bigjh...@gmail.com> wrote:
> On Fri, Apr 10, 2009 at 10:09 PM, John Yang <bigjh...@gmail.com> wrote:

[John Yang]

On Sat, Apr 11, 2009 at 1:49 PM, arschles <arsc...@gmail.com> wrote:
>
> I created one new issue for the "raw" UDP transport layer (not sure
> how to change the issue type or owner).

I changed it to "enhancement".

>
> I'm almost done with the UDP code, and I'll send a patch when I get it
> tested a bit more.

Sounds good.

[arschles]

John,

Can you clarify what the Transport.listen method is used for? As far
as I can tell, it listens for connections (which always correspond to
messages) and calls connection_handler in a separate thread whenever
it receives a new connection. Is this correct?

Also, I'm not sure what the purpose of the id parameter in
Transport.send is for? Is this a UID internal to the Transport layer?

Overall, I think that each transport layer should implement exactly
the Transport interface (for instance, the IPCTransport layer does not
implement packet_type_info - albeit for good reason) or there may be
some maintenance headaches later. This will also make documentation
easier

Thanks,
Aaron

On Apr 11, 2:48 pm, John Yang <bigjh...@gmail.com> wrote:

[arschles]

John,

Can you clarify what the Transport.listen method is used for? As far
as I can tell, it listens for connections (which always correspond to
messages) and calls connection_handler in a separate thread whenever
it receives a new connection. Is this correct?

Also, I'm not sure what the purpose of the id parameter in
Transport.send is for? Is this a UID internal to the Transport layer?

Overall, I think that each transport layer should implement exactly
the Transport interface (for instance, the IPCTransport layer does not
implement packet_type_info - albeit for good reason) or there may be
some maintenance headaches later.

Thanks,
Aaron



On Apr 11, 2:48 pm, John Yang <bigjh...@gmail.com> wrote:

[John Yang]

On Mon, Apr 13, 2009 at 8:37 PM, arschles <arsc...@gmail.com> wrote:
>
> John,
>
> Can you clarify what the Transport.listen method is used for? As far
> as I can tell, it listens for connections (which always correspond to
> messages) and calls connection_handler in a separate thread whenever
> it receives a new connection. Is this correct?

Right. "listen" takes in the port that the server is listening on and
a function that handles new connections. The "connection_handler"
part has not been used. "connection_handler" makes sense with
"accept".

>
> Also, I'm not sure what the purpose of the id parameter in
> Transport.send is for? Is this a UID internal to the Transport layer?

ID uniquely identifies a peer. The user will need to tell pysage
where the message is going, be it an IP address, port or an IPC file
descriptor.

>
> Overall, I think that each transport layer should implement exactly
> the Transport interface (for instance, the IPCTransport layer does not
> implement packet_type_info - albeit for good reason) or there may be
> some maintenance headaches later. This will also make documentation
> easier

Yea. I agree. The interface still needs some work, like it's missing
"accept", "disconnect", and "packet_type_info". For now, use
IPCTransport as an example. I believe the RakNet one needs some more
testing.

[arschles]

> Right. "listen" takes in the port that the server is listening on and
> a function that handles new connections. The "connection_handler"
> part has not been used. "connection_handler" makes sense with
> "accept".

does listen ever return? if not, does it listen for connections
forever? if so, when? also, is a connection defined as a single
message?

> ID uniquely identifies a peer. The user will need to tell pysage
> where the message is going, be it an IP address, port or an IPC file
> descriptor.

Judging from the IPCTransport code, it looks like id and address are
synonymous, and a caller adds a new address to a Transport by
explicitly connecting to it (with the connect method) or accepting a
message (using the accept method) and using the address of the message
sender as the new address. Is that right?

Once I get these questions ironed out, I'll add some documentation to
the transport module and also send you a patch with the "raw" UDP
implementation.

thanks,
Aaron

On Apr 13, 9:24 pm, John Yang <bigjh...@gmail.com> wrote:

[John Yang]

I got a chance to finish the UDP transport with unit tests. It is
committed in trunk. I also cleaned up the transport interface a
little bit. It should make more sense now.

My implementation uses the "select" system call currently. We may
want to add a "poll" version later for better performance.

I'm also looking to revamp the pysage networking API so that I can add
documentation on how the different transports can be used.

Feel free to tackle a TCP version if interested.

John S. Yang