Store time series data with mapdb

[Raphael Geissler]

Hi mapdb experts,

I already asked this question on Stackoverflow (for unknown reaons I got a down vote :/ ), but maybe someone here can consult me. I couldn't solve this be myself.

I am trying to find out if mapdb would be a good solution for my usecase, but I am not sure, if my requirements can be handle by mapdb.

I have huge amount of timebased log data that looks simplified as this:

new Object[]{1001L, "source1", "some text 1"} 
new Object[]{1000L, "source2", "some text 2"} 
new Object[]{1002L, "source3", "some text 3"} 
...

So my requirements are

1. Store logs in off-heap memory
   
2. Logs must be sorted by the timestamp (long value)
3. Get sorted logs filtered by source

So a query would look like this:

Give me all objects sorted by timestamp between timestamp 1000 and timestamp 1002 where source is source2 or source3.

Is there an efficient solution to handle that with mapdb?

Thanks for your help!

Raphael

P.S.: If you found this a bad question, then please tell me why, that I can learn for the future. :)

[Nicholas DiPiazza]

Solr would work better.

--
You received this message because you are subscribed to the Google Groups "MapDB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapdb+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Raphael Geissler]

Hi.

Thanks for the hint.

But do you know if this can also be done with mapdb?

Raphael

[Nicholas DiPiazza]

If you can figure it out on your own. Then sure. Its not very well supported anymore. Look at the issues list. 

[Raphael Geissler]

Ok. Thanks for the answer, then I will search for a better solution.

[alla...@utoronto.ca]

This can be done by MapDB.  As to the question of support, it's one person supporting it.  Having said that, Mapdb just keeps working for us in a business environment with billions of records.

Anyway, In mapdb create a TreeMap in external memory.  Load your stuff.  For an exaple of how to get a list of items filtered by something, see https://www.mkyong.com/java8/java-8-filter-a-map-examples/

[Nicholas DiPiazza]

When you "load your stuff" if you are loading it in bulk all at once, make sure you use "mapdb sinks" when doing so. or else the load will be slow slow slow. http://www.mapdb.org/book/btreemap/import/

--