A New Approach to Databases and Data Processing — Simulating 10Ks of Spaceships on My Laptop

[pron]

Hi.

Last week I posted an article on our company's blog, called "A New Approach to Databases and Data Processing — Simulating 10Ks of Spaceships on My Laptop"

It discusses the use of an in-memory database to schedule application logic. The general idea is submitting a query to the DB alongside a callback that will process the results. The database is in a unique position to schedule not only internal database transactions/queries but also the business logic processing the data. If the data structure used by the database models the domain as well as the data access patterns closely enough, this results in significant scaling capabilities, transparent to the application. The database ensures no deadlocks and races, and parallelizes the application.

In the article I mention that conceptually this is akin to writing the entire application as stored procedures, but this may be confusing as the database assumes a new role: not just in charge of storing and retrieving data, but in charge of scheduling and parallelizing all data processing. This approach is, obviously not entirely novel, but it takes the database to the extreme: the database and the application become one (in fact, I mention that original RDBMSs had a similar role).

The demo is 10Ks spaceship battle simulation we've built on top of our spatial DB, which was built in Java. The point of the post is not to show how fast the simulation can run because there are faster ways to do it. The idea is to show how even a naive solution running on top of the right kind of database can achieve decent performance and good scaling. 

Internally we employ a ForkJoinPool in asyncMode, with our hand-rolled locks (we use both optimistic and pessimistic locking) that don't park/unpark the running thread but cause the current ForkJoinTask to quit and be rescheduled when the lock becomes available (essentially we use a poor man's continuations on top of ForkJoin). I was fortunate to receive the help of Nir Shavit, Cliff Click and Doug Lea when I was first considering the design. 

I'm sure some of you may find this interesting, and I'd love to hear what you think.

[Martin Thompson]

What is the relevance of this to mechanical sympathy?  My first scan just makes me think this is spam promoting your database.  Can you please make it more specific and point to a useful development technique that you want to share or have a discussion around?

[pron]

Oh. Sorry. I thought people would be interested in a concurrent data structure, by the use of fork-join to traverse said structure, and by the general idea of the article, namely using a database as a parallelization mechanism for application code. Reading this group's posts (religiously, I might add) since its inception (I was the first one to post here), I was under the impression that it's devoted to JVM techniques and data-structures designed to chiefly for performance, and built to fully take advantage of the peculiarities of modern hardware. Well, the article presents an approach to better exploit multi-core hardware in large-scale applications, implemented on the JVM as a cache-aware, concurrent data structure (a concurrent R-tree variant, in the example), and using the new jsr166e fork-join (a big improvement over Java 7's FJ, BTW) for scheduling, with locks specifically built for fork-join (to be open-sourced soon). I just thought this might be of interest, and I'll be happy to answer questions people may have, about the concept or the implementation.

If this is not enough, I have a very specific GC-related question as well. Occasionally when running the demo -- every 10 or 20 runs or so -- I get what I can only describe as "a bad GC run". During normal execution, our code allocates a relatively large amount of very short-lived objects, none of it promoted to old gen (except for the actual data structure nodes, of course). But occasionally, as the program starts, the GC spends an inordinate amount of time collecting and continues to behave this way throughout the program's lifetime. If it starts behaving this way when the program launches, it continues, and if not -- this behavior is not observed at all. Any ideas? 

[Michael Barker]

> If this is not enough, I have a very specific GC-related question as well.
> Occasionally when running the demo -- every 10 or 20 runs or so -- I get
> what I can only describe as "a bad GC run". During normal execution, our
> code allocates a relatively large amount of very short-lived objects, none
> of it promoted to old gen (except for the actual data structure nodes, of
> course). But occasionally, as the program starts, the GC spends an
> inordinate amount of time collecting and continues to behave this way
> throughout the program's lifetime. If it starts behaving this way when the
> program launches, it continues, and if not -- this behavior is not observed
> at all. Any ideas?

We had something similar happen in a production system recently, it
also exhibited large amounts of the pause time where spent in sys.
We're not sure of the specific cause, however the following things
helped:

- We normally have a System.gc() call occur in our overnight window.
This was missed the evening before we saw the problems, we checked to
make sure it happened during our next window.
- We added CPU pinning via taskset to ensure that the Java application
did not run on the same cores that were handling system interrupts,
which were restricted to a different few cores using cpu banning in
irqbalance.

Mike.