lightweight presences & xmpp
Toni Alatalo
Coincidentally we were also conducting stress & scalability tests with Erno here at Playsign yesterday. Not with Tundra-kNet, but with XMPP, using ejabberd as the server and a Flash-based candidate for a browser based realXtend client. Don't have much time to report now, and the results are also still inconclusive, but brief note about the idea and interim results. The service will go live next week, along with a Tundra based implementation of the same service, so you can all login and test how it works soon.
With 'lightweight presence' I mean a avatar configuration which is suitable for large user amounts for social apps (not racing or shooting hard realtime games). This boils down to:
- no physics (for avatars) on server
- rare net sync (e.g. max 1 msg/sec per presence)
- use of client side logic for movements and other actions, to smooth moves with rare messages
I tested the idea originally with Tundra & kNet in January, by modifying the moving objects test in the repo to have 1000 moving objects, which moved with about once-a-sec average interval. That is available in https://github.com/antont/tundra/commits/raremove (the only commit in that branch, resulting in https://github.com/antont/tundra/blob/raremove/bin/scenes/EntityMoveTest/EntityMoveTest.js and https://github.com/antont/tundra/blob/raremove/bin/scenes/EntityMoveTest/scene-raremove.txml
Result was promising, as expected: 1000 such rare movers caused about the same load (net bandwidth) as tens of constant movers (sorry can't recall if it was 20, 30 or 60).
Later we developed a Flash based client, using the open source Away3d lib which was featured in the report & public demos available via http://www.realxtend.org/webnaali/research/ . First as a standalone single user app, which then in the second step needed chat too. We decided to use XMPP for the chat as the overall service benefits from support personnel being able to use normal IM clients, whereas the end users have the integrated chat in the 3d client. Igniterealtime.org provides a as3 xmpp client lib called XIFF which we are using.
Further on we were asked to add presence / avatars to the application. We had already tested using normal socket connections to connect to Tundra from Flash, in so-called 'FlashNaali' featured in the report. However, here we already had the XMPP connection so decided to test with that first. Also because at this stage there was no free movement, but just automated navigation to clicked targets in the 3d scene. That is best done with a single network message, instead of streaming the whole path as hundreds of net messages, as the clients needed the animation path code anyway for own movement so can also use it for remote presences. This worked well for a few clients, so we proceeded to automated stress testing with bots.
Results were pretty good - could run 150 frequently (~1/8secs average) moving bots on the server, without ejabberd having any problems. XMPP is *very* expressive though so the bandwidth use was crazy (>100MB/s outgoing from the server). Ejabberd has throttling / traffic shapers on by default, capped to 1k/client by default, so everything still continued to run fine. Just a lot of the messages were not delivered to all the clients. Which is actually not critical in this app even, the customer figured it's no problem.
Luckily the application / scene has been designed for easy partitioning: it consists of 50 rooms, which are in the same world and immediately accessible from each other, but you can only see the inside of one at a time. So to scale up we changed the configuration to have a XMPP room for each 3d room. This seems to allow thousands of users on one server ok. I ran with 50rooms * 30clients => 1,500 total on last Saturday with server load of only 20-30% on a puny Amazon small instance. However the machines running the bots got choked so they were not sending movement messages as much as intended. We ran a more reliable test with Erno yesterday, with 1000 bots moving at even 8sec intervals (which is well realistic for this app with the relative rare & long running actions). Now we got ejabberd even crash once, and have cpu use at 50-60% much of the time. There is a server bot which is on all the 50 channels, and it is configured so that ejabberd's throttling is disabled. Is running in the same Amazon place as the server so should be in the same LAN and getting all the messages cheaply. We logged the test on that server and results are pretty good: almost all the messages from all the bots came through.
This is probably enough for this service - 1000 concurrent users is more than expected, and even if 1500-2000 come, they probably don't cause messages as often. And even if they do, everything keeps working, but just some of the non-critical messages may get dropped. In case even more come, plan is to use sharding - just start a new ejabberd for the next set of users (similar to what Adminotech does with Tundra instances automatically, but there typically at 20 users with the heavy avatars, not at 1000-2000 like here with the light ones).
There is now free movement with avatars in this app too. The bandwidth saving is made simply so that the movement is client side, and they send positions at max 1sec interval. Movement is again smooth thanks to client side interpolation. This seems to work and feel ok for this peaceful meeting like app.
The application has actually originally been developed with Tundra, and now been ported to this yet-unnamed XMPP+Flash combo. Uses the same data for the scene which is converted semi-automatically (we get object positions from TXML to the Flash client etc). It will go live next week with both - the Tundra version being labeled the 'full experience' and the browser based flash&xmpp client some kind of 'light version'. Interesting to see what happens :o
XMPP is after all the Internet standard for presence, what virtual worlds largely are about, so considering the use of it within realXtend seems worthwhile. It does authentication simply, has Internet-wide federation with server-to-server talks, is used by both Google gtalk and Facebook chat etc. We had client side use of it in Naali days already, but it can be interesting from a server side perspective with Tundra too.
One possibility is a combination of XMPP and Tundra: have XMPP rooms for Tundra scenes so that people can chat with IM clients too (from mobile phones etc), but also run Tundra processes for those so that running server side logic and physics are possible, and use of the efficient movement messages with kNet. Flash allows use of UDP so a normal Tundra client could be implemented. Possibly by using emscripten to compile kNet & parts of TundraLogic (the movement messages, movement extrapolation logic etc) to Javascript. XMPP supports defining auxiliary services for the rooms so could well have e.g. a Tundra URL there. Could still allow this 'lightweight presence' style movement for XMPP-only clients too. Chiru at CIE is actively working with XMPP and Tundra using the QT XMPP lib, we'll talk about it more soon.
On the other hand, there is also a XMPP extension for stream compression, which can quite very well for streams such as movement. Unfortunately it is not (yet) featured in the lib we are using with Flash, and also the native built-in zip support in Flash does not support reusing the compression frame which the stream compression needs. Ejabberd on the server side does support it and there is at least a Java client lib that supports it so testing it would be easy. On the plugin-free front for browsers directly, Google is working on having identical stream compression directly in Chrome for websockets, that can become nice too (first-person shooter style networking was in their planning talks about it).
And of course this idea of lightweight presences can be just applied with Tundra too - with the optimized kNet messages we can perhaps already have thousands of rare moving / lazily synched presences (e.g. avatars) in a single scene?
~Toni
P.S. that Flash app is proprietary, made for a customer, but the idea and agreement is to open source the generic and generally useful parts of it under the realXtend umbrella. The whole XMPP chat and presence/avatar stuff falls clearly on the open side. The 3d scene things are Away3d which is already open source. We'll get back to this in May. Ideas for names are welcome, what could there be alongside Taiga and Tundra? (if not Suo, which is 'swamp' in Finnish :)
Jeroen van Veen
Interesting study! Did you check out webrtc support in Chrome Canary
yet? It has limited p2p and media
support(http://www.webrtc.org/running-the-demos) buildin the browser.
cheers,
Jeroen
Op 19 april 2012 09:14 heeft Toni Alatalo <to...@playsign.net> het
volgende geschreven:
> --
> http://groups.google.com/group/realxtend-dev
> http://wiki.realxtend.org
> http://dev.realxtend.org