Warp 10 on EC2 instances with "cold storage" on S3

[Adam Makhlouf]

Hi everyone,

I am considering using Warp 10 (standalone) in the AWS ecosystem on 2 EC2 instances (with datalog replication).

I've seen on this blog article that the S3 WarpScript extension can help archive data into S3.

Now if I want to keep "hot data" for 15 days on Warp 10 and then archive older data to S3. If I run a fetch for a timespan that is for example from 5 days ago to 20 days ago (so a mix of data on Warp 10 and data on S3) will the platform be able to manage it. Or will I need to split the request?

I hope this was clear, happy to try to clarify if not!

Thanks for your help!

[mathias....@gmail.com]

Hi,

you could craft a macro which would handle the various storage tiers depending on the time range or other criteria.

Mathias.

[Adam Makhlouf]

Hi,

Thank you Mathias. Any idea on the execution time difference (ie. twice as slow, 10 times slower?)  for the same operation being executed on LocalDB as compared to S3 ?

Have a great week-end!

--
You received this message because you are subscribed to a topic in the Google Groups "Warp 10 users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/warp10-users/yVhp8sT-Kd4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to warp10-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/warp10-users/74f1a0cd-a015-476a-9f3e-f27faf8aaac1n%40googlegroups.com.

[mathias....@gmail.com]

The objects stored in S3 are wrapped GTS, so a single object can pack millions of data points. The actual fetching of the object is in most cases pretty fast, so the fetch + unwrapping can even be faster than fetching the same data from the LevelDB data store. If you are intending to load data for multiple series then most of the cost will be in the fetching of the objects from S3, as fetching multiple objects may need multiple requests to the S3 store.

By storing data in S3 you loose flexibility since what you are storing are chunks of series, so the smallest part you can fetch is a single chunk. If you store weekly chunks but only need 10s of data, you will still have to fetch the whole week and discard most of what you fetched.

You should chose your chunk size according to your use cases, don't store monthly chunks if your queries only need a day worth of data for example.

[Adam Makhlouf]

Thank you very much Mathias! Very helpful!