finance application using graphs

[Meisam Shahid Sorkhabi]

Hi Folks,

Looking to solicit your thoughts about how to architect a graph db system to ingest real-time, bursty, high velocity financial transaction data.

Need to be able to handle about 30,000 - 50,000 messages a second during peak and about 2000-5000 msg/sec avg. From this data, the state of the graph needs to be built in real time i.e. new vertex added, edges established, on the fly. The real-time ingest constraint is "soft" at about 60 seconds. That means no more than 60 seconds can pass before any new transaction message can be properly linked with all other relevant information already recorded in the graph.

The nature of the data is such that node connectedness is low - it would look more like a tree with less than 5-6 height but potentially hundreds of thousands of leaves.

I have access to 6 enterprise grade servers - each sporting between 16-24 cores and anywhere from 400-800 GB RAM. HDD available would be in the 20-30 TB range.

Now, the clients of such a hypothetical system, would need to be able to query the latest state of the system and that "state" needs to be consistent with reality (pursuant to the max 60 seconds delay).

I believe the hardest part would be keeping up with (ingest) the incoming data at a high rate, while giving clients a consistent view of the latest state.

What do you think? Is such a system achievable? What would you recommend as the Titan storage backends? Any input is highly appreciated!