security and authentication

[K Livingston]

is there anywhere I can find discussion on security of the Orbited
daemon. can I control or otherwise authenticate an orbited daemon
connection?

I'm thinking about scenarios where malicious users might either
connect to non-existant channels or invalidly to existing channels,
thus wasting resources holding file handles open (even if I'm not
communicating to them).

[Michael Carter]

The security burden fall on the app layer, not orbited. That is, the application provides the client with a unique identifier, like a session key, to use when connecting to orbited. The Orbited daemon has no idea what the identifier means, or if its valid. But, if someone connects with an invalid identifier then they won't ever be sent any messages from orbited (via the app), because the app only dispatches messages to known identifiers.

For publish/subscribe communication, Orbited 0.3 has a system known as revolved. It is very similar to Bayeux in terms of functionality, and like Bayeux there are no built in mechanisms for channel based security. This is being corrected in Orbited 0.4. We are working on a security plugin system for publish subscribe with some pre-built backends, like a database server or memcached. I'm not fully sure how this would work, but there should be a prototype in the next few weeks.

 

[K Livingston]

On Jan 22, 5:52 am, "Michael Carter" <cartermich...@gmail.com> wrote:
> But, if someone connects with an
> invalid identifier then they won't ever be sent any messages from orbited
> (via the app), because the app only dispatches messages to known
> identifiers.

So that keeps bad guys from getting data they shouldn't and that makes
sense, but what keeps bad guys from wasting your resources?

For example, I see you are running an Orbited server, so I spin up 100
sockets to connect to it just using arbitrary identification numbers.
Sure I won't get any data, but I'm wasting your Orbited server's time
(ok maybe not that much), but worse you server has now dedicated 100
file handles to this malicious machine. Now lets say it's 500 or
1000's etc. someone could eat all your file handles, right?

That's a patological case, and something that would need addressing,
but even the non-pathological case, say a client get's hung, and does
not gracefully take down it's side of the Comet connection, will the
Orbited server ever terminate it's end of it and reclaim the
resources? If not that could have long term stability problems,
operating like a slow crawl to the pathological condition listed
above.

I don't know what the solutions are? but the problems are significant.

For the first, possibly an authentication payload sent to the Orbited
server inside the request to establish the Comet connection gets
forwarded from Orbited to a/the backend application server. It then
responds with accept or deny. accept sets up as expected, and deny
results in the Orbited server closing the connection. Actually I
think you can do this without change to the client, the Orbited server
can just ask the application server using the user id and number,
since the app server will be aware of what user ids and session
numbers it doled out and which it didn't.

For the second, I think you mentioned that the connections are
periodically re-established by the client, so perhaps the Orbited
server should kill connections at 2 times the expected re-establish
rate. Allowing for picking up of dead resources. (perhaps you
already do this)


For the record, this is great - keep up the work - not having to put
all this logic into an application is good for the community.

Kevin

[Jacob Rus]

Kevin Livingston wrote:
> That's a patological case, and something that would need addressing,
> but even the non-pathological case, say a client get's hung, and does
> not gracefully take down it's side of the Comet connection, will the
> Orbited server ever terminate it's end of it and reclaim the
> resources? If not that could have long term stability problems,
> operating like a slow crawl to the pathological condition listed
> above.

Yes, Orbited will terminate its end of the connection and reclaim
resources. If you find cases where it doesn't, those are bugs.

> For [bad guys wasting your resources, eating all your file handles],

> possibly an authentication payload sent to the Orbited
> server inside the request to establish the Comet connection gets
> forwarded from Orbited to a/the backend application server. It then
> responds with accept or deny. accept sets up as expected, and deny
> results in the Orbited server closing the connection. Actually I
> think you can do this without change to the client, the Orbited server
> can just ask the application server using the user id and number,
> since the app server will be aware of what user ids and session
> numbers it doled out and which it didn't.

One potential problem with such a solution is that it sort of depends
on an event-based application server, and complexifies porting ORBIT
clients. I also wonder whether this attack would be particularly
worse on the server than just repeatedly opening and closing
connections as fast as possible; we should benchmark it I suppose, but
it is not clear to me that open file handles will be as much a
bottleneck as CPU and bandwidth, here.

-Jacob

[K Livingston]

> Kevin Livingston wrote:
> > possibly an authentication payload sent to the Orbited
> > server inside the request to establish the Comet connection gets
> > forwarded from Orbited to a/the backend application server.  It then
> > responds with accept or deny.  accept sets up as expected, and deny
> > results in the Orbited server closing the connection.  Actually I
> > think you can do this without change to the client, the Orbited server
> > can just ask the application server using the user id and number,
> > since the app server will be aware of what user ids and session
> > numbers it doled out and which it didn't.

On Jan 22, 3:44 pm, "Jacob Rus" <jacobo...@gmail.com> wrote:
> One potential problem with such a solution is that it sort of depends
> on an event-based application server, and complexifies porting ORBIT
> clients.  I also wonder whether this attack would be particularly
> worse on the server than just repeatedly opening and closing
> connections as fast as possible; we should benchmark it I suppose, but
> it is not clear to me that open file handles will be as much a
> bottleneck as CPU and bandwidth, here.

It wouldn't require an event based server to handle, you could do the
authentication over HTTP or any number of other things. And this
should be thought out more, I don't have a formal proposal, but a
couple of ideas I can envision is in the configuration have a setting
for connection authentication mode:

NONE: does what it does now

HTTP: (I would envision this being the default for those that want it)
in this case Orbited could sent a HTTP Get request to a specified URL
with a parameter ?userkey set to the URL encoded form of the user key
attempting connection. The server can then do whatever it wants to
decide, and send :accept or :deny as a response to Orbited. I don't
know if this could be done fast enough before accepting the socket
connection, or if it would have to be accepted into a holding pen
first, and then depending on response from server dropped or
transferred to being an Orbited connection.

SOCKET: stable socket kept open between Orbited and another
application to implement basically the same as above, but without
requiring socket tare down and reconstruction and HTTP overhead.
Orbited sends ID, server responds ID+accept, or ID+deny

PYTHON: some other python function to determine eligibility

some other methods you could think up...


My application has to have a list of Orbited user ids somewhere
anyway, right, because it has to know who it's talking to for normal
functionality, so it should be easy to know if an ID you give it is
someone it's talking to or not.

I would think repeated open/close or SYN type attacks (if experienced)
could be addressed by something outside of the Orbited protocol. It's
fairly standard for firewalls to protect against that (and/or throttle
interaction). I was more addressing was I saw as attack vectors
specific to the Orbited server and it's protocol. I saw it as Orbited
opening a potentially new attack vector on my server.


thanks again.

[Michael Carter]

I don't really see having multiple open connections as a dangerous attack vector. The most resource intensive part of Orbited is accepting and closing TCP connections. Becuase libevent uses edge triggered network IO libararies like epoll and kqueue, there is practically no cost to an additional idle connection. Besides, the firewall could just block an ip address that opens hundreds of idle connections.

Overall though, it makes little sense to put authentication directly in the transport layer of a Comet application. But Orbited is currently just the transport layer. We are attempting to add a connection/session layer with the upcoming CSP  protocol. You wouldn't build authentication into IP or TCP, and i see orbited as a layer not unlike IP and CSP as a layer similar to TCP. In my mind, the real place for authentication would be in the pubsub layer or in the app.
 

-Michael Carter

[Jacob Rus]

K Livingston wrote:
> I would think repeated open/close or SYN type attacks (if experienced)
> could be addressed by something outside of the Orbited protocol. It's
> fairly standard for firewalls to protect against that (and/or throttle
> interaction). I was more addressing was I saw as attack vectors
> specific to the Orbited server and it's protocol. I saw it as Orbited
> opening a potentially new attack vector on my server.

The ORBIT protocol (i.e. the capability to *send* events) definitely
shouldn't be exposed to the general internet.

But I don't really see Comet communications being much more attackable
than any other type. Your firewall is better equipt to handle such
vulnerabilities than any authentication of the kind you suggest built
into Orbited, which would additionally at least double the complexity
of Orbited–app communications, and take not insignificant server
processing time to deal with (every connection from a browser must
first be checked with the app).

We need to do some benchmarking to figure out exactly how many idle
connections an Orbited node can handle, etc., but I don't think
idle-connection DDOS attacks against Orbited will be a significant
problem, and adding per-connection checking is as likely to aggravate
as improve matters.

So to summarize, IMO such constraints are better placed in a layer
outside Orbited itself, such as in your interaction-throttling firewall.

-Jacob

[Michael Carter]

We need to do some benchmarking to figure out exactly how many idle
connections an Orbited node can handle, etc., but I don't think
idle-connection DDOS attacks against Orbited will be a significant
problem, and adding per-connection checking is as likely to aggravate
as improve matters.

 

One thing I'll add to this discussion is that I did a benchmark of Orbited 0.1.3 and found an 11% reduction in throughput when I increased the number of concurrent connections from 1,000 to 10,000. Its not a bad figure, but I think it can be further optimized.

-Michael Carter

[K Livingston]

On Jan 23, 1:16 am, "Jacob Rus" <jacobo...@gmail.com> wrote:
> The ORBIT protocol (i.e. the capability to *send* events) definitely
> shouldn't be exposed to the general internet.

oh of course, and I'm not worried about that at all.

> But I don't really see Comet communications being much more attackable

> of Orbited-app communications,

My concern was/is that it is a state-full transaction with the
server. (but I don't know the life of that state, so I'll plead a
little ignorance there, - see below) And for the most part I do see
other transport layers requiring application level authentication (or
at least awareness). For example, I can't just open a TCP/IP
connection to your computer it has to first be listening on that port,
and then the listening application will accept that connection, and
can see where it came from, either do something with it, or drop it,
if it's so inclined. But the application is well aware of the
connection. Orbited insulates me from that awareness, in many cases
for good - that's kind of the point, right? but the case I present is
at least one for bad, and I'm just trying to learn how bad, that's
all. (the profiling data below is I think important first steps in
understanding that - thank you)

> and take not insignificant server
> processing time to deal with (every connection from a browser must
> first be checked with the app).

I guess, here is where I expose some more of my ignorance on the
matter, and probe for more information. What is the typical time to
live of an Orbited connection, before the server will clean it up?
Your default config file shows a 5, I think, but I don't know what the
units are. 5 seconds? 5 m? h? if it is something like 5s then I
guess the trade off for overhead to authenticate vs. wait for death/
cleanup, probably isn't worth it. If it's minutes+ it might be.
(time to authenticate could be quick (check against a hash table of
expected connections), but if it has to happen every few seconds per
user - right, that might be too much, if it's minutes, actually I just
did some back of the envelope, and I didn't like the results of
putting that check in... so I guess I'm beginning to come to your side
of the fence.

On Jan 23, 7:57 am, "Michael Carter" <cartermich...@gmail.com> wrote:
> One thing I'll add to this discussion is that I did a benchmark of Orbited
> 0.1.3 and found an 11% reduction in throughput when I increased the number
> of concurrent connections from 1,000 to 10,000. Its not a bad figure, but I
> think it can be further optimized.

that's not as bad as I would have thought, it would be good to know
the shape of that curve too. (if that code is easily deployable,
others could regression test their installations/hardware-config too
(and if they are feeling kind report it), to get a feel for how much
load Orbited can support)



to be clear, I'm just trying to fully understand this mode of
interaction before I can comfortably commit a lot of resources to
prototyping with it, much less deploying it. But I am excited about
what it offers, and am very interested in using it. And I think you
guys are doing great work. So please don't take this thread as
inflammatory, but rather exploratory -- as I think it's important to
know these properties of the technology, and I greatly appreciate your
work in helping me understand it.

-Kevin

[K Livingston]

On Jan 23, 1:16 am, "Jacob Rus" <jacobo...@gmail.com> wrote:

> of Orbited-app communications, and take not insignificant server

> processing time to deal with (every connection from a browser must
> first be checked with the app).

each key is used repeatedly, right? a client will connect with a key,
use the Orbited connection for a bit, then tear-down and then
reconnect with the same key, over and over again, right? So the
Orbited server wouldn't have to check each connection, just each key,
when it sees a new one it would have to phone home, but otherwise, it
would just be "oh you again, welcome back", right?


On a completely different note, what happens if two Orbited
connections come in attempting to use the same key at the same time?

thanks,
-Kevin

[Michael Carter]

Hi Kevin,

This discussion is much welcomed. I like nothing more than to delve into architecture/design issues for distributed comet applications / servers. If I sound defensive its only because I'm being terser than I'd like due to limited time to respond to email.

So the main two questions as I see it are 1) what are the benefits to checking back with the application per connection to authenticate, and 2) What are the penalties? We seem to have come to these two ideas:

Benefit: No idle-connection-as-attack-vector possibility


Penalty: Performance hit of calling back to the application per connection


Given this trade-off, I say, make it an optinal hook in the ORBIT protocol.

But that introduces some more complexities:

1) What if multiple app nodes are connected to a given orbited node. who handles the auth?

2) What if the application is using ORBIT protocol as a synchronous protocol? how does it handle callbacks?


To shed some more light on the subject and answer earlier questions, let me describe the Orbited session/connection model.

When a connection is opened with a previously unknown identifier, then a new Session object is created. That session object looks at the connection and creates a transport object as specified by the transport arg in the query string. The transport object then notifies the session that it is ready to dispatch events. At some point the transport connection may be closed.  The session object then buffers events for X seconds until a new connection is opened. When a new connection is opened, it is always passed to the session which either hands it off to the current transport object, or creates a new transport object.

After there has been no open connection for X seconds, the session is closed and the identifier is deleted. A new connection with that key will be treated as an unknown connection and a new Session object will be created.

So one of the proposals for Orbited 0.4 is to create a more complex version of the ORBIT protocol. The current ORBIT/1.0 protocol would be renamed to ORBIT/basic and we'd have a new mode identified by ORBIT/advanced. The advanced protocol would alert the application when a session was created or ended. I think it would be fine to allow the app to respond to a 'session created' notice with 'not allowed' or something to implement the security model as described.

Personally though, I don't think this is that big a benefit. I would probably just let an attacker keep an idle connection open. Is there a more compelling reason than performance to add this functionality? Peace of mind? Maybe its a good idea, I'm just not convinced yet from a technical standpoint. I also don't know how to deal with the additional complexities as outlined above. Most app developers are using frameworks like rails or django that provide no mechanism for app-side callback authentication.

-Michael Carter

[Marcus Cavanaugh]

On Jan 23, 2008, at 6:02 PM, Michael Carter wrote:
> So one of the proposals for Orbited 0.4 is to create a more complex
> version of the ORBIT protocol. The current ORBIT/1.0 protocol would
> be renamed to ORBIT/basic and we'd have a new mode identified by
> ORBIT/advanced. The advanced protocol would alert the application
> when a session was created or ended. I think it would be fine to
> allow the app to respond to a 'session created' notice with 'not
> allowed' or something to implement the security model as described.

I think that would be a beneficial step in the right direction.

IMHO, one of the most important things to consider as Orbited develops
is simplicity. In keeping with the Zen of Python, I think it's
critical to consider various possibilities and whittle those down
until you're left with the best way forward.

Orbited, as a simple event-delivery system (as I originally met it
back at 0.2), is architecturally great: I think that design is worth
merit. Where it _can_ get complicated is when we need something more
than just blind event delivery. Again, the more we think through the
problem to come up with the best solution, the better off Orbit (and
comet in general) will be in the future.

Related to connection notification: The more I think about it, a full
connection and disconnection notification system, integrated with
Orbited, would be quite complex.

[K Livingston]

On Jan 23, 6:02 pm, "Michael Carter" <cartermich...@gmail.com> wrote:
> This discussion is much welcomed.

cool. thanks.

> So the main two questions as I see it are 1) what are the benefits to
> checking back with the application per connection to authenticate, and 2)
> What are the penalties? We seem to have come to these two ideas:
> Benefit: No idle-connection-as-attack-vector possibility
> Penalty: Performance hit of calling back to the application per connection

I think I agree, although, it would only need to be per id / per
session object, not per transport connection, right?

> Given this trade-off, I say, make it an optional hook

If it was decided to add it, I think optional is appropriate (and
could really just be another property of the existing server, see
below), and hook is I think the right way to think about it.

> 1) What if multiple app nodes are connected to a given orbited node. who
> handles the auth?

hm.. I didn't think about that. I guess which app could be encoded in
the key value, or channel name etc. That could certainly be a problem
though -- which would require more thinking.

> 2) What if the application is using ORBIT protocol as a synchronous
> protocol? how does it handle callbacks?

I guess I'm not really sure of the distinction here?

> To shed some more light on the subject and answer earlier questions, let me
> describe the Orbited session/connection model.

thank you this was informative.
how many seconds have you found is typically appropriate between
reconnecting, and also for terminating a seemingly expired connection?

> When a connection is opened with a previously unknown identifier, then a new
> Session object is created. That session object looks at the connection and
> creates a transport object as specified by the transport arg in the query
> string. The transport object then notifies the session that it is ready to
> dispatch events. At some point the transport connection may be closed. The
> session object then buffers events for X seconds until a new connection is
> opened. When a new connection is opened, it is always passed to the session
> which either hands it off to the current transport object, or creates a new
> transport object.
>
> After there has been no open connection for X seconds, the session is closed

> So one of the proposals for Orbited 0.4 is to create a more complex version
> of the ORBIT protocol.

If this change was desired you wouldn't need to change the anything in
the existing setup, it would only involve adding a way for Orbited to
check identifier with the app. before accepting it and creating the
session object (or at least before pulling the session object into the
official "herd"). Everything else could stay the same, right?

> advanced protocol would alert the application when a session was created or
> ended.

I didn't think about ending, that may also be beneficial to know
about.

> Personally though, I don't think this is that big a benefit. I would
> probably just let an attacker keep an idle connection open. Is there a more
> compelling reason than performance to add this functionality? Peace of mind?
> Maybe its a good idea, I'm just not convinced yet from a technical
> standpoint.

It's definitely a trade off that should be made empirically, what
costs less resources. (or less of the resources you can afford to
trade, there may be a curve there, and applications may lie on
different places on the curve). Although, peace of mind seems nice
and I think about it sometimes, they are just machines (when the
terminators come I'll regret that remark), but it's really a cost-
benefit decision. (no point in spending a lot of resources to save a
few -- that would make a worse attack vector, wouldn't it?)

> Most app developers are using frameworks like rails or
> django that provide no mechanism for app-side callback authentication.

Sure, if they just added another page in their framework, (less
efficient than a direct connection), but the Orbited server could just
call that Ruby on Rails page or whatever (which could be specified in
the configuration file) the Orbited server would pass it the ID in
question as an HTTP variable (like submitting a form). The web page
handler does whatever it needs to to look it up and passes a response
back to the Orbited server telling it what to do. It's very
possible. Necessary though, that seems to have become the more
important question.

thanks,
Kevin

[K Livingston]

On Jan 24, 12:36 am, Marcus Cavanaugh <marcuscavana...@gmail.com>
wrote:
> The biggest problem I've had with Orbited, which I have not found an
> elegant solution to, is how to have the Application Server know when
> clients are active.

Until your post I didn't even realize how important disconnect notices
might be,.. If we are talking resource wasting though, thinking about
say a simple chat server example (like you said), if there are a whole
bunch of disconnected clients the server will be wasting a whole bunch
of time trying to send messages to clients that don't exist any more.

There are at least two options for disconnect notices, one active, as
soon as the session object goes away phone home. The other is
passive, if the application server attempts to send to a non-exist ant
client it get informed that this client no longer exists.


> I don't think it's possible for Orbited to *both* reliably notify an
> application server about connections/disconnections *and* scale
> effectively, primarily because the moment you couple the Orbit node to
> an Application server, you lose the properties of Orbit that
> distinguish it from the other Comet apps: Elegant -- effectively
> linear -- scalability, thanks to the decoupling of the Comet layer
> from the Application layer.

I haven't thought about it enough, but this may be a false dichotomy.
there are ways the orbited servers could communicate to the app
servers without having to be coupled 1 to 1 or otherwise. For example
(and I'm not saying this is best), the communications from the orbited
server to the app servers could go through the same load balancers
that the outside word does when talking to the app servers.

> But the problem with pings to the Application Server is this: Imagine
> 1000 clients, pinging an Application Server every 30 seconds. That's
> 2000 requests a minute -- 33 requests a second. That's a huge load for
> such a relatively small number of connections. 66 requests a second
> for 2000 connections... and it only gets worse.

I agree active pinging is bad, or rather redundant, since the clients
are already actively pinging the Orbited server, right? (presuming
tear-down and re-create of the transport connection happens with some
regularity) So leveraging this knowledge seems only appropriate, as
you seem to be saying too.


> - What if the application server becomes disconnected from Orbited,

some communication guarantee would be necessary, or on reconnect or
polling, the app server could also get a list of active connections
(as an alternative, I'm not advocating, just thinking alternatives)

> IMHO, the more distinct we can keep the tiers of an application, the
> better.

I think I agree, but that doesn't say anything about what knowledge
the two layers could share, or how they go about communicating the
state of that shared knowledge.


> -- and whether or not such a feature should be in the Orbited layer.

but it already has some of that knowledge, right?
for example it knows what orbited connections are active.


Kevin
(I hope that was coherent, it's getting late, and proofreading isn't
working well for me right now)

[Jacob Rus]

Kevin Livingston wrote:
> > 2) What if the application is using ORBIT protocol as a synchronous
> > protocol? how does it handle callbacks?
>
> I guess I'm not really sure of the distinction here?

As opposed to an event-based server. A synchronous app does blocking
I/O, and can't call a callback based on information coming back over
the ORBIT protocol. So it would need another mechanizm. Which leaves
us with different mechanisms depending on the type of app server.

> benefit decision. (no point in spending a lot of resources to save a
> few -- that would make a worse attack vector, wouldn't it?)

Yes, that's exactly right. :)

> > Most app developers are using frameworks like rails or
> > django that provide no mechanism for app-side callback authentication.
>
> Sure, if they just added another page in their framework, (less
> efficient than a direct connection), but the Orbited server could just
> call that Ruby on Rails page or whatever (which could be specified in
> the configuration file) the Orbited server would pass it the ID in

This strikes me as complex, counter-intuitive, and somewhat fragile,
but it might work.

-Jacob

[K Livingston]

On Jan 24, 1:11 am, "Jacob Rus" <jacobo...@gmail.com> wrote:
> Kevin Livingston wrote:
> > > 2) What if the application is using ORBIT protocol as a synchronous
> > > protocol? how does it handle callbacks?
> > I guess I'm not really sure of the distinction here?
> As opposed to an event-based server. A synchronous app does blocking
> I/O, and can't call a callback based on information coming back over
> the ORBIT protocol. So it would need another mechanizm. Which leaves
> us with different mechanisms depending on the type of app server.

I mean i understand synchronous communication, but wasn't sure in this
context. So to give this a concrete example, we're talking about
something like a single threaded application server? Then yes there
could be some complicated situation which might lead to deadlocks,
which I thought about before, but then promptly forgot about. But I
think then the Orbited callbacks could just get interleaved into the
single event queue, and I think the order would be such that deadlock
would not be created.

Or if there were other situations, providing a concrete example might
facilitate the discussion better (at least for me, I like examples).

thanks,
Kevin

[Michael Carter]

Great email Marcus, I don't have enough time to adequately respond. I'll go over it again 12-16 hours from now.

In particular, I see three options for implementing a good connect / disconnect notification model. One I just built this past weeked for a threaded app. It was basically just a matter of creating another thread that handled all the orbited communication and also ran a timer to dispatch pings and such. We could start putting this sort of functionality into the client libraries, as one approach.

But as you say, that would be even more of a strain on the app server, and makes it harder to scale orbited nodes horizontally. For that reason, I tend to agree that we should have ORBIT/advanced protocol, and possibly even implement it in such a way as to guarantee delivery of notices. But again, this could cause problems by sharing too much state between layers thus making it hard to distribute.

A third option is to have orbited make actual HTTP requests to the web framework as callbacks when these connect/disconnect/auth events take place. It suffers from similar problems of guaranteed delivery, as well as binding particular application nodes to particular orbited nodes, but it has the advantage of keeping the ORBIT protocol simple. It has the additional advantage of being incredibly simple to implement on the application side. In RoR or django you'd just implement a /orbited/on_connect or /on_disconnect callback that orbited calls whenever someone disconnects or connects. We could also address Kevin's ideas of authentication by providing a /orbited/authentication callback option for the application.

I would be much more willing to put a HTTP callback mechanism in Orbited than change the protocol because it would keep everythign simple still and the ORBIT protocol would continue to be backwards compatabile (a minor issue only.) But most importantly, its the first semi-sensible way of creating real-time web applications with synchronous app servers that I've yet considered. The major concern would be mapping Orbited nodes to app nodes for the purpose of scaling. But at worst we could start with a A-B config based mapping mechanism.

-Michael Carter

events, so it can't be certain that it knows all clients which have
connected to Orbited. (Unless there's an external authentication
mechanism, which would therefore _require_ that an Application Server
receive those notifications, which IMO is a different, separate
concern.)

The biggest problem I've had with Orbited, which I have not found an
elegant solution to, is how to have the Application Server know when
clients are active.

I don't think it's possible for Orbited to *both* reliably notify an
application server about connections/disconnections *and* scale
effectively, primarily because the moment you couple the Orbit node to
an Application server, you lose the properties of Orbit that
distinguish it from the other Comet apps: Elegant -- effectively
linear -- scalability, thanks to the decoupling of the Comet layer
from the Application layer.

Take a simple chat server: Unless a client "pings" the Application
server regularly to tell the Application server that the client is
still "connected", there isn't a way for the Application server to
know whether or not that client is active unless Orbited can tell the
Application server that that specific client has connected. So how do
you determine when a user has left the chat room? By either the client
explicitly sending a "PART" message, or by an Application-Server-
induced ping timeout. (That's what differentiates UDP from TCP...
that's essentially what we're working with in terms of client-to-
server requests for an application server over HTTP.)

But the problem with pings to the Application Server is this: Imagine
1000 clients, pinging an Application Server every 30 seconds.  That's
2000 requests a minute -- 33 requests a second. That's a huge load for
such a relatively small number of connections. 66 requests a second
for 2000 connections... and it only gets worse.

Session _creation_ is dead-simple. The first time a client talks to
the application server, the application server knows it's available.
The only reason we need pings is to detect when a client *disconnects*.

If the connection/notification mechanism (as I was trying to describe
in that graphic) is added to Orbited, it provides a way for the
Application Server to track disconnections. However, there are a few
concerns:

- What if the application server becomes disconnected from Orbited,

and during that time a client disconnects from Orbited? The
application would miss the notification, and the Application would
forever think that the client was still connected.
- A workaround for that would be for Orbited to implement a queuing
mechanism to ensure reliable delivery of all connection and
disconnection messages, for any and all application servers which
request that capability. (But that's just a by-the-book message
queue... it seems that would be better suited to an application-
dependent message queuing or pub-sub layer...)

Bah. All-in-all, I guess this can be summarized in two paragraphs:

To track client connection status, one must implement pinging to the
Application server, which (from the numbers I've calculated) could
take a heavy toll on performance of an app server.

Alternatively, Michael has mentioned that Orbited will be getting a
connection/disconnection notification mechanism, which _could_ solve
some of those problems, but I'd be interested in knowing if that will
be a reliable delivery mechanism or not. If it's reliable, I suppose
that could be leveraged effectively -- but if it's not reliable, it
wouldn't do much good in the situation where the App Server(s) could
temporarily lose their connection to the Orbited node(s).

IMHO, the more distinct we can keep the tiers of an application, the
better.

Now that I've hashed that all out in my head, I'm actually a lot
clearer on how a *reliable* connection-notification facility in
Orbited *could* be feasible without overcomplicating matters. But I'm
still uncertain as to whether or not that would be the best solution

-- and whether or not such a feature should be in the Orbited layer.

Does that make sense at all?

Marcus


--
Marcus Cavanaugh
marcusc...@gmail.com

[Michael Carter]

On Jan 23, 2008 11:20 PM, K Livingston <kevinliving...@gmail.com> wrote:

On Jan 24, 1:11 am, "Jacob Rus" <jacobo...@gmail.com> wrote:
> Kevin Livingston wrote:
> > > 2) What if the application is using ORBIT protocol as a synchronous
> > > protocol? how does it handle callbacks?
> > I guess I'm not really sure of the distinction here?
> As opposed to an event-based server.  A synchronous app does blocking
> I/O, and can't call a callback based on information coming back over
> the ORBIT protocol.  So it would need another mechanizm.  Which leaves
> us with different mechanisms depending on the type of app server.

I mean i understand synchronous communication, but wasn't sure in this
context.  So to give this a concrete example, we're talking about
something like a single threaded application server?  Then yes there
could be some complicated situation which might lead to deadlocks,
which I thought about before, but then promptly forgot about.  But I
think then the Orbited callbacks could just get interleaved into the
single event queue, and I think the order would be such that deadlock
would not be created.

The synchronous version of the ORBIT protocol is very weak in the context of orbited sessions as added in 0.3 The reason is that its conceivable to have a session timeout of 30 seconds, but that means that an ORBIT requests may take 30 seconds before a response comes back. So your thread in the web app is left hanging for half a minute, which is unacceptable. The only good solution is to set the timeout to 0 and implement your own presence system via pings to/from the app.

In the context of asynchronous callbacks the synchronous protocol is a liability for a second reason -- there is no way to implement async callbacks as the client will be blocking whenever it waits for network IO. So there is no way to actually listen for a callback and perform request/responses in the app worker threads.

Its not just a matter of deadlocks; I don't see how to do it besides stick the orbited client in its own thread and make it an asynchronous implementation. But that adds unacceptable complexity for a class of applications that don't need to scale up to large numbers of users.
 

-Michael Carter

[Marcus Cavanaugh]

On Jan 24, 2008, at 7:36 AM, Michael Carter wrote:
> Great email Marcus, I don't have enough time to adequately respond.
> I'll go over it again 12-16 hours from now.
>>

Thanks.

> I would be much more willing to put a HTTP callback mechanism in
> Orbited than change the protocol because it would keep everythign
> simple still and the ORBIT protocol would continue to be backwards
> compatabile (a minor issue only.) But most importantly, its the
> first semi-sensible way of creating real-time web applications with
> synchronous app servers that I've yet considered. The major concern
> would be mapping Orbited nodes to app nodes for the purpose of
> scaling. But at worst we could start with a A-B config based mapping
> mechanism.

Yes, an HTTP callback could be one way of doing it. That would be the
only reasonable way to do it with synchronous frameworks I think, but
I wouldn't be a fan of using that when you have asynchronous
frameworks; that would require an asynchronous app to learn the HTTP
protocol, and if the App server (or Orbit) doesn't know about keep-
alive, there'd be a pretty big overhead for each notification.

In the case of multiple Application Server / Orbited nodes:

Just like it's the Application Servers' responsibility to determine
which Orbit node(s) to communicate with (i.e. via client-token
hashing), I think it also falls within the scope of the Application
Server to notify Orbit about what types of events it wants to receive.

If an Application node wanted to receive connect/disconnect
notifications, it could specify so when it connects to Orbited. If
there are multiple App nodes connected to the Orbit node, the
Application Server would need to be aware that BOTH nodes would
receive notification (it would need to discard one), unless only *one*
of the app servers explicitly requested notification. Alternatively,
Orbit could implement a type of notification where it will just send
the notification to ONE of the nodes, even if there are multiple nodes
connected.

If it was done that way, a distributed application could choose to
select one node to be a "notification handler" -- that node could
handle ALL notifications while never sending _any_ events to Orbited
itself. That would be useful if it wanted to push the notifications
onto the distributed application's message queuing system.

On a related note:

The easier it is for clients to implement Orbited, the greater chance
Orbited has for widespread adoption. If the "ORBIT/advanced" protocol
is homegrown, custom clients will need to be written for every target
language; this has the advantage that the protocol is completely
tailored to our needs, but at the cost of interoperability.

Have you seen STOMP? (http://stomp.codehaus.org/) STOMP is a text-
based protocol used with Message Queue software, and there are
numerous clients in different languages for it. Its protocol reminds
me very much of Orbited's protocol: simple and HTTP-like. I think it
would be worth considering whether or not Orbited would benefit from
using *that* protocol -- if it did, we'd instantly have clients done
and written in 12 languages.

That's just a thought, but I would strongly encourage you to at least
look at such a possibility. By reusing a protocol like STOMP, we get
immense interoperability and a much shallower curve of entry for
different programming languages. I think that alone would be worth the
consideration, not to mention the other benefits of using a standard
protocol.

I forget where, but I read a story not too long ago where someone
implemented a Message Queue system over Memcached's protocol: They
instantly had the advantage that they could use _any_ of the already-
created Memcached clients to dispatch messages; that seems like a huge
benefit.

Maybe it wouldn't work, but I think it's worth an investigation. The
potential benefits are strong and IMHO worth consideration.

Marcus

[Michael Carter]

Have you seen STOMP? (http://stomp.codehaus.org/) STOMP is a text-
based protocol used with Message Queue software, and there are
numerous clients in different languages for it. Its protocol reminds
me very much of Orbited's protocol: simple and HTTP-like. I think it
would be worth considering whether or not Orbited would benefit from
using *that* protocol -- if it did, we'd instantly have clients done
and written in 12 languages.

That's just a thought, but I would strongly encourage you to at least
look at such a possibility. By reusing a protocol like STOMP, we get
immense interoperability and a much shallower curve of entry for
different programming languages. I think that alone would be worth the
consideration, not to mention the other benefits of using a standard
protocol.

I looked at STOMP very thoroughly, and was excited about it at first but now I'm not so sure. There are a couple problems that make me hesitate.

1) It only has a notion of publish/subscribe. This would work well for Revolved, but for plain old Orbited message dispatch we'd also want a "send" option (in STOMP their "send" is actually a publish command, and there is no command named "publish" )
2) the Reicept frame is used to acknowledge that the server received a particular command, but not the outcome of the command. In fact, there is no good way of returning a 'success' or 'failure' of a particular command. There is an error message type, but it contains NO provisions for specifiying in the protocol what particular command caused the error. It seems to be used to report malformed requests more than anything.

For the first problem i could accept implementing send as a specific case of publish. if the app publishes a message to "user:heyadayo" then that could mean "send a message to heyadayo", but the second problem is a big one because the protocol doesn't have a real way to acknowledge the success or failure of any given action, so therefore we'd have to hack our own meaning into the protocol which would somewhat defeat the purpose of having clients in many languages.

I'll think about it a but more though, and any suggestions are welcome.
-Michael Carter

[Marcus Cavanaugh]

On Feb 6, 2008, at 2:06 AM, Michael Carter wrote:
> I looked at STOMP very thoroughly, and was excited about it at first
> but now I'm not so sure. There are a couple problems that make me
> hesitate.
>
> 1) It only has a notion of publish/subscribe. This would work well
> for Revolved, but for plain old Orbited message dispatch we'd also
> want a "send" option (in STOMP their "send" is actually a publish
> command, and there is no command named "publish" )

You're right; a STOMP-based Orbit protocol would need to give some
type of semantic meaning (e.g. recipients names are separated via
commas or something), so it couldn't be used verbatim.

> 2) the Reicept frame is used to acknowledge that the server received
> a particular command, but not the outcome of the command. In fact,
> there is no good way of returning a 'success' or 'failure' of a
> particular command. There is an error message type, but it contains
> NO provisions for specifiying in the protocol what particular
> command caused the error. It seems to be used to report malformed
> requests more than anything.

Also a good point. The server could probably behave differently, but
you'd have to make sure the clients are OK with that behavior.

Given those considerations, I think you're right that STOMP wouldn't
be a very good fit for the Orbited server. It's close, but some
nuances mean that an equally simple Orbited protocol might make more
sense.

Provided the Orbited protocol remains simple, straightforward, and
consistent, while keeping room for extensibility in the future, I
suppose a homegrown protocol would be fine... especially since the
Orbited protocol is used somewhat internally to an application rather
than as a public-facing interface.

[Michael Carter]

Marcus,

I've been thinking some more about your idea, and I realized that it could work nicely. My issue of publish/subscribe versus send isn't that big a deal -- after all, OP doesn't currently do pubsub, nor do I actually want it to. So the send frame in stomp could just be used to send a message to a set of recipients.

As for the error messages, I've reworked the way that OP deals with errors and successes. That is, whenever you issue a SEND command you can get a receipt back, but in order to actually be informed of a success or failure you must first register a callback. The callback can be dispatched either through OP itself, or as a remote url (for frameworks like django, rails, pylons, etc.) Other callbacks include signon and signoff.

In the context of STOMP, it makes sense to just change the syntax of callbacks such that its really a subscribe request. "I'm subscribing the success events" sort of thing.

You can see my current documention on the ORBIT protocol here: http://orbited.org/svn/orbited/branches/0.4/daemon/orbited/op/DOCUMENTATION

I plan on implementing this first because it provides a better framework for me to think about the problems. But I want to include a STOMP daemon in orbited 0.4. I don't suppose you're interested in helping with that? (or anyone else)

-Michael Carter

[Marcus Cavanaugh]

On Feb 12, 2008, at 1:25 AM, Michael Carter wrote:
> Marcus,
>
> I've been thinking some more about your idea, and I realized that it
> could work nicely. My issue of publish/subscribe versus send isn't
> that big a deal -- after all, OP doesn't currently do pubsub, nor do
> I actually want it to. So the send frame in stomp could just be used
> to send a message to a set of recipients.
>
> As for the error messages, I've reworked the way that OP deals with
> errors and successes. That is, whenever you issue a SEND command you
> can get a receipt back, but in order to actually be informed of a
> success or failure you must first register a callback. The callback
> can be dispatched either through OP itself, or as a remote url (for
> frameworks like django, rails, pylons, etc.) Other callbacks include
> signon and signoff.
>
> In the context of STOMP, it makes sense to just change the syntax of
> callbacks such that its really a subscribe request. "I'm subscribing
> the success events" sort of thing.
>

> You can see my current documentation on the ORBIT protocol here:

> http://orbited.org/svn/orbited/branches/0.4/daemon/orbited/op/DOCUMENTATION
>
> I plan on implementing this first because it provides a better
> framework for me to think about the problems. But I want to include
> a STOMP daemon in orbited 0.4. I don't suppose you're interested in
> helping with that? (or anyone else)

Sure, I'd be up for working on that. Let me know where to start and
I'll take a look.