Thought experiment

[Greg Jackson]

So I think I'm going to have to build a system to record multiple current measurements at a 1ms interval for 48 hours. Or recording once a second (max,min,average), I still have to think about the implications of that.

I know nothing about databases.

So what I would like to do is record the values, shove them off in Json or similar over mqtt or similar to a listener that shoves them into MySQL or similar (that's where I get hazy). Then I can perform some basic analysis and graphing using Python (which I've done before so that should be grand).

In terms of storage it would be multiple experiments I would like to compare, each experiment consisting of tags describing the experiment and its setup and then tables for each measurement (current a,b,c temperature a,b light level etc...). I guess I could do it all with nested Json but it's 518,400 measurements/sensor @1second and 172,000,000 measurements/sensor @1ms so it's impractical to process.

Does anyone have any thoughts for appropriate technologies?

[John Maguire]

I would say use Mongodb, that way you can stick with Json and you can then do powerful analytical queries with a lot less work later.

I would do each reading as a separate "row" in a single collection. Is there multiple locations for the readings? Or just a single sensor?

Using Mongo you will get up and running a lot quicker and its already in the format you want plus if you want to adjust your model later its very easy to do an will still keep your data intact.

Send me a model and I'd happily help you with the set up. I believe there is a python, Java, c, and many more libraries but i can help set u up with node which won't require much if you just want an endpoint api

[Martin Kennedy]

Yep sounds like a no-sql DB is what you're after, so mongo would be good, cos there's a lot of data. If you have multiple experiment locations or need public access you could always spin up an AWS server for the 48 hrs and then move the data afterwards if necessary.

[Greg Jackson]

Cool. Cheers all. I'll start playing with an arduino recording temperature or something and sending to mongo. Then I'll write a script to display the average from the last 5 minutes or so. That should give me enough to get started with and get my head around the components of the system. Thanks for the help!!!

Once I know what questions to ask I may come back with more.