Financial application storage. If not Raven, then what?

[georgiosd]

Hey all,

Some of you will have seen me on the list quite a lot from late 2012 to early 2014 and less so recently.

In that time range, I was involved in a large-scale web project where we used RavenDB and I can say that as the main dev/architect, I didn't regret though I didn't see the project to production.

Anyway, since then, I am doing some research in financial markets and have ran into the need to store tick data for several instruments (stocks/currencies/whatever).

On one hand the ability to create map-reduce indices is very handy to automatically calculate min/max/volume values for a range of dates...

On the other hand, the amount of data is very large and I'm wondering whether RavenDB is the best fit for this kind of application.

Indicatively, EUR/USD tick data for 1 year is 15Μ samples (bid/ask/time/volume) which is only 80MB on disk but gets blown in JSON in Raven.

I did try to create per-day documents but they turned up too large to load quickly. 

Of course there are other configurations possible, such as hourly but I still don't think it would work when you start adding symbols.

I think it would be a different matter if Raven used BSON but...

So, any ideas on simple (i.e. not enterprise level beasts) storage systems that can support queries that would be good for this application?

Thanks

Georgios

[Oren Eini (Ayende Rahien)]

What kind of queries are you looking at?

This is a time series system, and that is something that we'll be adding to RavenDB post 3.0 release.

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--
You received this message because you are subscribed to the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ravendb+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Felipe Leusin]

I've stumbled upon http://geteventstore.com/ before and maybe it's a fit for your problem...

[Georgios Diamantopoulos]

One example would be to generate candles using indices for standard values (do it in-memory for non-standard values)

Using the same indices you can find min/max/average values for specific ranges etc.

What exactly do you mean by you'll be adding it?

Georgios

---

From: aye...@ayende.com
Date: Sun, 12 Oct 2014 18:17:34 +0300
Subject: Re: [RavenDB] Financial application storage. If not Raven, then what?
To: rav...@googlegroups.com

You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/yyyOivDzcVU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

[Oren Eini (Ayende Rahien)]

I don't understand. Candles? indices? 

Can you show what you mean?

We are going to be adding time series capabilities to ravendb.

[Georgios Diamantopoulos]

Thanks! I looked at the event store a while ago, not sure if it would do it but perhaps something has changed since

---

Date: Sun, 12 Oct 2014 15:30:05 -0700
From: felipe...@gmail.com
To: rav...@googlegroups.com

Subject: Re: [RavenDB] Financial application storage. If not Raven, then what?

You received this message because you are subscribed to a topic in the Google Groups "RavenDB - 2nd generation document database" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ravendb/yyyOivDzcVU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ravendb+u...@googlegroups.com.

[Georgios Diamantopoulos]

Most people nowadays trade on Japanese candlestick charts

e.g. on a 5min candle chart, you get one candle (like a column) for ticks contained in a 5minute period with 5 values: open (first value), close (last value), low, high and volume (count).

If you only store ticks on a DB like Raven, you can create an index that will map-reduce to 5min candles. Makes sense?

But, what does it mean to you to add time series capabilities? BSON storage? An API that is friendlier to those systems? Something else?

Georgios

---

From: aye...@ayende.com
Date: Mon, 13 Oct 2014 16:03:29 +0300

[Oren Eini (Ayende Rahien)]

Not really, remember, I don't actually have any idea about stock trading, or candle charts.

Can you talk in data?

USD - 09:00 = 1.39, 09:01 = 1.38, etc

And then show what you want to get out?

I'm talking about dedicated storage that is optimized for this scenario. That is why I'm trying to see what you need, to work it into our plans.

[Georgios Diamantopoulos]

EUR/USD simplified ticks example with periods set to 5 minutes:

09:01 1.33

09:02 1.34

09:03 1.32
...

09:05 1.35

The chart would show two candles:

Candle1 (09:00-09:05) - Low: 1.32, High: 1.34, Open: 1.33, Close: 1.32, Volume: 3

Candle2 (09:05-09:10) - Low = High = Open = Close = 1.35 (gets updated as new ticks arrive)

There are standard period sizes (1min, 5min, 15min, 1h, 4h, 1d, 1w, 1month) but some software offers the ability to create arbitrary periods (e.g. 3 minutes)

More info and visuals: http://en.wikipedia.org/wiki/Candlestick_chart

Any better?

I think that's the most you can delegate to the database along with min/max values etc.

From a quick Google search I did on time series DBs, InfluxDB looks quite mature, you can probably get some ideas from there.

Georgios

---

From: aye...@ayende.com
Date: Mon, 13 Oct 2014 16:39:57 +0300

[Oren Eini (Ayende Rahien)]

Hm..

You can probably do that pretty easily by establishing a 1 minute rollup and then just aggregating from there, since the data is very small at that rate.

Thanks for the information, we'll take that into account.

[Georgios Diamantopoulos]

No problem, happy to talk to you further if you need it at some point.

What's your approx. ETA for this?

Georgios

---

From: aye...@ayende.com
Date: Tue, 14 Oct 2014 12:36:28 +0300

[Oren Eini (Ayende Rahien)]

About 2 months after the 3.0 release, I'm guessing.

[Georgios Diamantopoulos]

InfluxDB does it by grouping: http://influxdb.com/docs/v0.8/api/query_language.html

---

From: aye...@ayende.com
Date: Tue, 14 Oct 2014 12:36:28 +0300

[georgiosd]

Oren, hi,

Is this still in your plans?

Thanks

Georgios

[Oren Eini (Ayende Rahien)]

Can you open a new thread about this?

Hibernating Rhinos Ltd  

Oren Eini l CEO l Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

 

--