High Concurrency Low Overhead JS Embedding

[MikeM]

I'm writing this post at the urging of David Mandelin.

It's purpose is to document our current Spider Money use case and
start a discussion on how we can transition
to newer versions of Spider Money and still meet these use case
requirements.

We use SM for many different server side embeddings. It's been a
wonderful tool and we love it!

One of our applications is as a web server.
In this embedding we use JS as the core scripting language of the web
server.
Most requests (except 100% file based GET's) run server side JS to do
their work.
The web server is used to maintain rich client side web applications
(not static pages).

We support various methods of authorization and authentication in the
web server.
Once a client has been authenticated, we can keep a session for
purposes of maintaining session state information.
This allows the server side JS code to set properties and functions on
the session and retrieve them on subsequent visits.
Basic state information is also persisted and replicated to other
servers in the web farm to facilitate auto fail over if the web server
should die.
(The load balancer pushes each client back to the same web server each
time for speed reasons)

Almost all of the hits to our web server that run JS also need to hit
the database.
The web server is Synchronous (not async like node.js).
However we DO support Workers (synchronous or asynchronous), but
workers cannot respond back to the http client directly (at least not
yet)

Speed is extremely important. On average requests to the server
involving JS complete 1-10 ms (even with SQL database access). File
requests are measured in micro-seconds.

------------------
Here is a simplified explanation of our software architecture:

1) A single process (NT Service on Windows)
2) A single multi-threaded JS runtime (when not debugging)
3) A pool of request threads (already running but suspended)
4) 1-n pools of compiled JSScripts (one pool per server web root per
incoming connector and port)
5) 1-n pools of compiled JSFunctions (same scope as above used for
security related callouts)
6) A pool of JSContext's
7) A pool of database connections (already connected to the DB)
8) A hash of sessions by session id (UUID)

Each session can be accessed by multiple requests and JS scripts
simultaneously (we provide locking primitives to our scripts outside
spidermonkey when needed)
Session state is maintained in a special global object, the "Session
Global".
The session global is rooted via JS_AddNamedRootRT() and lives as long
as the session does.
Each request that hits the server also has another "Request Global"
that lives only as long as the request itself.
The request global is used to hold temporal vars and functions that
must die when the request dies and must not pollute
the session global.
These two globals are linked together via prototype chain using
JS_SetPrototype(cx, requestGlobal, ressionGlobal);

We support the including of sub JS Scripts, both at compile time and
dynamically at run time.
These sub scripts can be executed on either of the two globals
(request or session) or on a new sandboxed global, the latter of which
we tend to use for security check purposes.
We use resolver hooks (for each global) to lazy initialize anything we
can.
JSScripts are automatically re-compiled & re-cached if script source
changes on disk.
We use JS_SetOperationCallback() to handle long running scripts.
We Suspend/Resume the request around anything that could possibly at a
long time (I/O or DB access etc)

We initialize standard classes only once on the session global.
The standard classes are reachable from the request global via the
prototype chain.
Initializing standard classes only once is extremely important to
prevent "instanceOf" from being broken on return trips back to the
server, and for speed.

--------------------------------------------------
Each request that hits the server does following:

1) Grab a random request thread from the thread pool to handle the
Socket accept (if not already pipelining)
2) If a script needs to be run then grab a free JSContext from the
pool.
3) Associate context with thread JS_SetContextThread() &
JS_BeginRequest()
4) Create a request global
5) Check for session cookie and grap the session object from the hash
(and its session global) or create a new one if a session is needs to
be started.
6) Link the globals via JS_SetPrototype() and NULL the parent of the
request via (cx, requestGlobal, NULL);
7) Lookup compiled script to run from script pool (create, compile &
cache if not already done)
8) Run the script via JS_ExecuteScript() using chosen thread and
request global (1-n sub scripts may run too)
9) Push JSON/HTTP response down the socket
10) Cleanup using JS_ClearNewbornRoots(cx), JS_ClearRegExpStatics(cx)
JS_ClearPendingException(cx), JS_SetGlobalObject(cx, NULL),
JS_EndRequest(cx), JS_SetOperationCallback(cx, NULL),
JS_ClearContextThread(cx), JS_SetContextPrivate(cx, NULL)
11) Put all objects back into their appropriate pools & suspend the
request thread (if not piplining)

Drawbacks to Current Embedding Setup
------------------------------
1) Stop the world GC (doesn't scale well)
2) Staying on trace is hard (I believe we clone properties between
globals sometimes or other magic scary hacks)

Design Goals
--------------------
1) Support 10000 concurrent sessions while maintaining session state
2) Support 500 concurrent JS Scripts running at a time on 500 threads.


The Future
-----------------
At this point the future of SpiderMonkey if very unclear for server
side embeddings such as this.
It seems that many good things have been lost.
We are looking for ideas on how can we make the current SpiderMonkey
tip work.
Based on our understanding of the current world, here are some things
we may need to make it work.

1) Immutable scripts (no baked in JIT pointers or such)
2) Way to keep these scripts in a single hash/cache that can be used
in any context or compartment (or runtime if contexts die out)
3) Compartmental GC. Or any variation that is NOT stop the world GC.
4) The ability to associate any thread with any compartment or
runtime. Mandating the same thread be used with the same compartment
or runtime
would require a thread pool of 10,000 instead of 500 to support
10,000 concurrent sessions.
5) A way to support session persistence of vars, properties and
functions between executions and still be able to have vars,
properites and functions that die with each request.
Prototype chaining with 2 globals works well now. Can we keep it?
6) If multiple runtimes are required now (sounds like they will be),
be able to control set amount of memory used by a runtime. In the
current world 500 x 2MB = 1Gig. That doesn't work in a 32 bit world.
A 1 GB memory footprint for the engine alone leaves no headroom for
application code. We have some services wrapped by this http server
that can easily eat 1GB all by themselves without JS.

We would be more than happy to write any patches that would be
required to keep using spidermonkey.
Please let me know your thoughts.

Thanks!

Mike M