HTTP POST to google spreadsheet
740 views
Skip to first unread message
MRE
May 16, 2012, 1:17:10 PM5/16/12
to TokyoHackerSpace
So I finally got a cheap and dirty alternative to Pachube working
(they changed their name by the way)..
I can not claim to be the originator of the code or idea, just tweaker
of various related projects to do what I want.
I originally found some arduino net shield code to post to a sheet,
but I suspect it would not work in all instances.
I tried to convert the same process to Processing with no - little
success.
I also found a good example of using the java apache.http libraries to
do http POSTs. The Libraries are not installed by default, so you have
to grab them yourself.
I then smashed the two together and I was finally up and running.
Here is a google spreadsheet which is reporting soil moisture in a
test tank. It updates twice a minute:
https://docs.google.com/spreadsheet/ccc?key=0ArBwm5DI5eb5dGZNTnVOR2VPQU9WZlhTZlM1YmY2RUE
Regarding the method:
The arduino reports via serial to my PC, running Processing.
Processing takes over, and does an http POST to a Google form.
The Google form is linked to the sheet (when you create a form, it
automatically creates the sheet with the links to push data onto it).
Re the soil moisture sensor:
It is the typical stainless steel nails, mounted in a piece of plastic
(cut from a chopping board).
To reduce galvanic action (which coats the nails in bubbles, reducing
sensitivity) I am doing a pseudo AC method, using two digital outputs.
The nails are tied to the digital outs, while one is also tied through
a resistor to the analog input. I toggle the digital pins back and
forth between high and low, reading on the appropriate phase.
Just for kicks, I tried out the nails on my tongue. A fun zap that was!
(they changed their name by the way)..
I can not claim to be the originator of the code or idea, just tweaker
of various related projects to do what I want.
I originally found some arduino net shield code to post to a sheet,
but I suspect it would not work in all instances.
I tried to convert the same process to Processing with no - little
success.
I also found a good example of using the java apache.http libraries to
do http POSTs. The Libraries are not installed by default, so you have
to grab them yourself.
I then smashed the two together and I was finally up and running.
Here is a google spreadsheet which is reporting soil moisture in a
test tank. It updates twice a minute:
https://docs.google.com/spreadsheet/ccc?key=0ArBwm5DI5eb5dGZNTnVOR2VPQU9WZlhTZlM1YmY2RUE
Regarding the method:
The arduino reports via serial to my PC, running Processing.
Processing takes over, and does an http POST to a Google form.
The Google form is linked to the sheet (when you create a form, it
automatically creates the sheet with the links to push data onto it).
Re the soil moisture sensor:
It is the typical stainless steel nails, mounted in a piece of plastic
(cut from a chopping board).
To reduce galvanic action (which coats the nails in bubbles, reducing
sensitivity) I am doing a pseudo AC method, using two digital outputs.
The nails are tied to the digital outs, while one is also tied through
a resistor to the analog input. I toggle the digital pins back and
forth between high and low, reading on the appropriate phase.
Just for kicks, I tried out the nails on my tongue. A fun zap that was!
Akiba
May 16, 2012, 8:50:23 PM5/16/12
to tokyohac...@googlegroups.com
Nice. Is there any limit to the amount of data that can be held in a single
document?
Akiba
FreakLabs Open Source Wireless
Web: http://www.freaklabs.org
Shop:http://www.freaklabsstore.com
Twitter: http://www.twitter.com/freaklabs
> --
> You received this message because you are subscribed to the
> Google Groups "TokyoHackerSpace" group.
> To post to this group, send email to
> tokyohac...@googlegroups.com.
> To unsubscribe from this group, send email to
> tokyohackerspa...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/tokyohackerspace?hl=en.
>
document?
Akiba
FreakLabs Open Source Wireless
Web: http://www.freaklabs.org
Shop:http://www.freaklabsstore.com
Twitter: http://www.twitter.com/freaklabs
> You received this message because you are subscribed to the
> Google Groups "TokyoHackerSpace" group.
> To post to this group, send email to
> tokyohac...@googlegroups.com.
> To unsubscribe from this group, send email to
> tokyohackerspa...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/tokyohackerspace?hl=en.
>
Kalin KOZHUHAROV
May 16, 2012, 9:47:30 PM5/16/12
to tokyohac...@googlegroups.com
On Thu, May 17, 2012 at 9:50 AM, Akiba <ch...@freaklabs.org> wrote:
> Nice. Is there any limit to the amount of data that can be held in a single
> document?
>
Number of Cells: Total of 400,000 cells across all sheets
> Nice. Is there any limit to the amount of data that can be held in a single
> document?
>
Number of Columns: 256 columns per sheet
http://support.google.com/docs/bin/answer.py?hl=en&answer=37603
But since you are using a PC to upload the data, use POST to a fusion table:
https://developers.google.com/fusiontables/docs/developers_guide#Inserting
Then your only limit will be the sky! Well 100MB per table actually, a
bit of cloud on the sky, LoL.
I haven't seen Arduino library for doing that directly (e.g. with
Ethernet shield), but hacking the pachube one will not be that
difficult I guess.
Cheers,
Kalin.
Mikele
May 16, 2012, 11:15:03 PM5/16/12
to tokyohac...@googlegroups.com
that's super interesting!
been able to easily POST data from a sensor.
\m/
been able to easily POST data from a sensor.
\m/
MRE
May 17, 2012, 12:57:25 AM5/17/12
to TokyoHackerSpace
In theory, no.
the post method a ccepts a field name (|column value, found in the
html source of the goole form... its an odd naming, not one you
create) plus the value to enter.
So you can have as many columns as google spreadsheet allows. First is
always googles own timestamp.
I am unaware of any column or row limits enforced on the google sheet.
I am going to experiment to figure out a post rate limit.
also found a drupal vis module for google docs. Have not tried yet.
Setting up a site under my diy-scib.org site for data. If it works
out, ill do the same on the ths beta site.
the post method a ccepts a field name (|column value, found in the
html source of the goole form... its an odd naming, not one you
create) plus the value to enter.
So you can have as many columns as google spreadsheet allows. First is
always googles own timestamp.
I am unaware of any column or row limits enforced on the google sheet.
I am going to experiment to figure out a post rate limit.
also found a drupal vis module for google docs. Have not tried yet.
Setting up a site under my diy-scib.org site for data. If it works
out, ill do the same on the ths beta site.
MRE
May 17, 2012, 12:58:39 AM5/17/12
to TokyoHackerSpace
Will look into that.
We can hack a lib to do it somehow.
On May 17, 10:47 am, Kalin KOZHUHAROV <me.ka...@gmail.com> wrote:
> On Thu, May 17, 2012 at 9:50 AM, Akiba <ch...@freaklabs.org> wrote:
> > Nice. Is there any limit to the amount of data that can be held in a single
> > document?
>
> Number of Cells: Total of 400,000 cells across all sheets
> Number of Columns: 256 columns per sheet
>
> http://support.google.com/docs/bin/answer.py?hl=en&answer=37603
>
> But since you are using a PC to upload the data, use POST to a fusion table:https://developers.google.com/fusiontables/docs/developers_guide#Inse...
We can hack a lib to do it somehow.
On May 17, 10:47 am, Kalin KOZHUHAROV <me.ka...@gmail.com> wrote:
> On Thu, May 17, 2012 at 9:50 AM, Akiba <ch...@freaklabs.org> wrote:
> > Nice. Is there any limit to the amount of data that can be held in a single
> > document?
>
> Number of Cells: Total of 400,000 cells across all sheets
> Number of Columns: 256 columns per sheet
>
> http://support.google.com/docs/bin/answer.py?hl=en&answer=37603
>
MRE
May 17, 2012, 1:00:21 AM5/17/12
to TokyoHackerSpace
Well, still useful for garden and weather data..and school
experiments.
100 mb is better though ;)
> On Thu, May 17, 2012 at 9:50 AM, Akiba <ch...@freaklabs.org> wrote:
> > Nice. Is there any limit to the amount of data that can be held in a single
> > document?
>
> Number of Cells: Total of 400,000 cells across all sheets
> Number of Columns: 256 columns per sheet
>
> http://support.google.com/docs/bin/answer.py?hl=en&answer=37603
>
> But since you are using a PC to upload the data, use POST to a fusion table:https://developers.google.com/fusiontables/docs/developers_guide#Inse...
> > Nice. Is there any limit to the amount of data that can be held in a single
> > document?
>
> Number of Cells: Total of 400,000 cells across all sheets
> Number of Columns: 256 columns per sheet
>
> http://support.google.com/docs/bin/answer.py?hl=en&answer=37603
>
Akiba
May 17, 2012, 1:04:10 AM5/17/12
to tokyohac...@googlegroups.com
I'll try fusion tables and see how that goes once I get some time. We've
been using that for Safecast data for awhile now. If the database is
unlimited, then it's probably the ideal way to post data to a google
service.
Akiba
FreakLabs Open Source Wireless
Web: http://www.freaklabs.org
Shop:http://www.freaklabsstore.com
Twitter: http://www.twitter.com/freaklabs
> -----Original Message-----
> From: tokyohac...@googlegroups.com
> [mailto:tokyohac...@googlegroups.com] On Behalf Of MRE
been using that for Safecast data for awhile now. If the database is
unlimited, then it's probably the ideal way to post data to a google
service.
Akiba
FreakLabs Open Source Wireless
Web: http://www.freaklabs.org
Shop:http://www.freaklabsstore.com
Twitter: http://www.twitter.com/freaklabs
> -----Original Message-----
> From: tokyohac...@googlegroups.com
> [mailto:tokyohac...@googlegroups.com] On Behalf Of MRE
mikele
May 17, 2012, 1:05:18 AM5/17/12
to tokyohac...@googlegroups.com
post to a PHP script to fill a MySQL table is simple.
then using a javascript library to show graphs?
http://www.highcharts.com/
am I missing something?
then using a javascript library to show graphs?
http://www.highcharts.com/
am I missing something?
Akiba
May 17, 2012, 1:21:11 AM5/17/12
to tokyohac...@googlegroups.com
Nice. The google hack is just a way to upload data if you
don't want to maintain your own server. It's also possible to put together a
simple api and use PHP to store the data into a db. Didnt know about the
highcharts thing. Gonna have to check it out sometime.
From: tokyohac...@googlegroups.com [mailto:tokyohac...@googlegroups.com] On Behalf Of mikele
Sent: Thursday, May 17, 2012 2:05 PM
To: tokyohac...@googlegroups.com
Subject: [THS:17624] Re: HTTP POST to google spreadsheet
--
You received this message because you are subscribed to the Google Groups "TokyoHackerSpace" group.
To view this discussion on the web visit https://groups.google.com/d/msg/tokyohackerspace/-/cnTa3tuNqaIJ.
Mikele
May 17, 2012, 1:33:49 AM5/17/12
to tokyohac...@googlegroups.com
there is plenty of js library to handle data and output charts, I linked the one that I liked better for the look. Free to use for non commercial purpose.
uploading to a google spreadsheet make sense for a quick and dirty test and if you need an easy interface to edit the data afterwards.
tonight I can try to put together a couple of scripts and a page on my server for fun.
I'll use the data in the spreadhseet to make a CVS, I will write a script to parse that and POSTs to a webpage that renders the graph.
My challenge will be having the graph page to 'listen' and refresh, that is something I had to work on anyway, better sooner than later.
I bet it won't take me longer than a bottle of wine.
uploading to a google spreadsheet make sense for a quick and dirty test and if you need an easy interface to edit the data afterwards.
tonight I can try to put together a couple of scripts and a page on my server for fun.
I'll use the data in the spreadhseet to make a CVS, I will write a script to parse that and POSTs to a webpage that renders the graph.
My challenge will be having the graph page to 'listen' and refresh, that is something I had to work on anyway, better sooner than later.
I bet it won't take me longer than a bottle of wine.
Akiba
May 17, 2012, 1:36:36 AM5/17/12
to tokyohac...@googlegroups.com
Be careful. You might end up hosting all the air quality
data we'll be collecting soon ;)
From: tokyohac...@googlegroups.com [mailto:tokyohac...@googlegroups.com] On Behalf Of Mikele
Sent: Thursday, May 17, 2012 2:34 PM
To: tokyohac...@googlegroups.com
Subject: Re: [THS:17625] Re: HTTP POST to google spreadsheet
Mikele
May 17, 2012, 1:44:26 AM5/17/12
to tokyohac...@googlegroups.com
:) I would go as far as renting another server just for that if the project kicks off.
Torsten Wagner
May 17, 2012, 3:29:07 AM5/17/12
to tokyohac...@googlegroups.com
Telling my students over and over again that Excel is not the way to
do scientific data calculation and visualisation, this hack leaves a
bit goosebumps.
How about an alternatives?
Arduino+Ethernet-Shield+Dropbox API
I checked dropbox. They have a low level API to down and upload files.
So your data would be in the cloud. Getting an API for a uC to write
files to dropbox would be nice. Any other program could fetch the file
and create a graph.
However, dropbox requires a developer-key and does not allow appending
of files. Would be bad to upload and download each file over and over
again... well one could create a folder and save e.g. every X
measurements in one csv file. The receiver application can put this
together on the other side.
One step further, I wonder how difficult it would be to write a
library which can create a (subset) of a vector file format (e.g. svg)
for microcontroller. With this, one could generate images and send
them or store them locally. Easily readable by any svg-aware program
(webbrowser, etc.).
Totti
do scientific data calculation and visualisation, this hack leaves a
bit goosebumps.
How about an alternatives?
Arduino+Ethernet-Shield+Dropbox API
I checked dropbox. They have a low level API to down and upload files.
So your data would be in the cloud. Getting an API for a uC to write
files to dropbox would be nice. Any other program could fetch the file
and create a graph.
However, dropbox requires a developer-key and does not allow appending
of files. Would be bad to upload and download each file over and over
again... well one could create a folder and save e.g. every X
measurements in one csv file. The receiver application can put this
together on the other side.
One step further, I wonder how difficult it would be to write a
library which can create a (subset) of a vector file format (e.g. svg)
for microcontroller. With this, one could generate images and send
them or store them locally. Easily readable by any svg-aware program
(webbrowser, etc.).
Totti
MRE
May 17, 2012, 8:24:12 AM5/17/12
to TokyoHackerSpace
Why ae you against spredsheets for holding data?
Sure some overhead compared to a csv, but in my opinion worth it on
the digestability front.
If i were planning a lot of manipulation, Id opt for csv or a nosql
db.
This is low rate, simple garden data.
mikele
May 18, 2012, 9:51:31 AM5/18/12
to tokyohac...@googlegroups.com
it took more than 1 bottle.
but it works and it is pretty.
a webpage showing a list of sensors and graphs using DB stored data.
it refresh each second, and shows if data have been added (i.e. via _POST).
I'll show Tuesday, by then I will rent a server to use for this demo and more. (after long thinking I decided to use my other testing server)
that said, I know nothing about sensors... if the project would scale up, it would need at least interfaces for:
sensors's profiles - more details about each sensor (i.e. description, geolocation, data range, chart type, min interval, etc, etc)
accounts management - where users with account can change, add, delete their sensor's profile, or just reset the log or download it...
still, it was fun, I learned a thing or two and I'll use it to log data from those smart environmental sensors I do want ...
but it works and it is pretty.
a webpage showing a list of sensors and graphs using DB stored data.
it refresh each second, and shows if data have been added (i.e. via _POST).
I'll show Tuesday, by then I will rent a server to use for this demo and more. (after long thinking I decided to use my other testing server)
that said, I know nothing about sensors... if the project would scale up, it would need at least interfaces for:
sensors's profiles - more details about each sensor (i.e. description, geolocation, data range, chart type, min interval, etc, etc)
accounts management - where users with account can change, add, delete their sensor's profile, or just reset the log or download it...
still, it was fun, I learned a thing or two and I'll use it to log data from those smart environmental sensors I do want ...
> >> From: tokyohackerspace@googlegroups.com
> >> [mailto:tokyohackerspace@googlegroups.com] On Behalf Of Mikele
> >> Sent: Thursday, May 17, 2012 2:34 PM
> >> To: tokyohackerspace@googlegroups.com
> >> Subject: Re: [THS:17625] Re: HTTP POST to google spreadsheet
>
> >> there is plenty of js library to handle data and output charts, I linked
> >> the one that I liked better for the look. Free to use for non commercial
> >> purpose.
>
> >> uploading to a google spreadsheet make sense for a quick and dirty test
> >> and if you need an easy interface to edit the data afterwards.
>
> >> tonight I can try to put together a couple of scripts and a page on my
> >> server for fun.
> >> I'll use the data in the spreadhseet to make a CVS, I will write a script
> >> to parse that and POSTs to a webpage that renders the graph.
> >> My challenge will be having the graph page to 'listen' and refresh, that
> >> is something I had to work on anyway, better sooner than later.
> >> I bet it won't take me longer than a bottle of wine.
>
> >> On Thu, May 17, 2012 at 2:21 PM, Akiba <ch...@freaklabs.org> wrote:
>
> >>> Nice. The google hack is just a way to upload data if you don't want to
> >>> maintain your own server. It's also possible to put together a simple api
> >>> and use PHP to store the data into a db. Didnt know about the highcharts
> >>> thing. Gonna have to check it out sometime.
>
> >>> Akiba
> >>> FreakLabs Open Source Wireless
> >>> Web:http://www.freaklabs.org
> >>> Shop:http://www.freaklabsstore.com
> >>> Twitter:http://www.twitter.com/freaklabs
>
> >>> ________________________________
> >>> From: tokyohackerspace@googlegroups.com
> >>> [mailto:tokyohackerspace@googlegroups.com] On Behalf Of mikele
> >>> Sent: Thursday, May 17, 2012 2:05 PM
> >>> To: tokyohackerspace@googlegroups.com
> >>> To post to this group, send email to tokyohackerspace@googlegroups.com.
> >>> To unsubscribe from this group, send email to
> >>> tokyohackerspace+unsubscribe@googlegroups.com.
> >>> For more options, visit this group at
> >>>http://groups.google.com/group/tokyohackerspace?hl=en.
>
> >>> --
> >>> You received this message because you are subscribed to the Google Groups
> >>> "TokyoHackerSpace" group.
> >>> To post to this group, send email to tokyohackerspace@googlegroups.com.
> >>> To unsubscribe from this group, send email to
> >>> tokyohackerspace+unsubscribe@googlegroups.com.
> >>> For more options, visit this group at
> >>>http://groups.google.com/group/tokyohackerspace?hl=en.
>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "TokyoHackerSpace" group.
> >> To post to this group, send email to tokyohackerspace@googlegroups.com.
> >> To unsubscribe from this group, send email to
> >> tokyohackerspace+unsubscribe@googlegroups.com.
> >> For more options, visit this group at
> >>http://groups.google.com/group/tokyohackerspace?hl=en.
>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "TokyoHackerSpace" group.
> >> To post to this group, send email to tokyohackerspace@googlegroups.com.
> >> To unsubscribe from this group, send email to
> >> tokyohackerspace+unsubscribe@googlegroups.com.
> >> For more options, visit this group at
> >>http://groups.google.com/group/tokyohackerspace?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "TokyoHackerSpace" group.
> > To post to this group, send email to tokyohackerspace@googlegroups.com.
> > To unsubscribe from this group, send email to
> > tokyohackerspace+unsubscribe@googlegroups.com.
Akiba
May 18, 2012, 10:52:57 AM5/18/12
to tokyohac...@googlegroups.com
Sounds good. Let's kick off the project. Two ideas for data
collection is air and radiation data. I'm also starting up the weather monitor
prototype design for the UN project so we can send the data from the prototypes
to your server.
From: tokyohac...@googlegroups.com [mailto:tokyohac...@googlegroups.com] On Behalf Of mikele
Sent: Friday, May 18, 2012 10:52 PM
To: tokyohac...@googlegroups.com
Subject: [THS:17657] Re: HTTP POST to google spreadsheet
To view this discussion on the web visit https://groups.google.com/d/msg/tokyohackerspace/-/f3Kg4UQ_i2YJ.
To post to this group, send email to tokyohac...@googlegroups.com.
To unsubscribe from this group, send email to tokyohackerspa...@googlegroups.com.
Torsten Wagner
May 18, 2012, 11:32:15 AM5/18/12
to tokyohac...@googlegroups.com
Well I am not exactly against spredsheets. But I saw to many bad excel
plots in my life.... which is why I get mentioned goosebumps...
Albeit I would be interested in a solution which generates graphs
without using a PC in between.
Arduino->Cloud
As said, it might be rather nice to have a lib, which can create a
vector graphic format on the Arduino and hence upload an image. Since
it would not need any knowledge on the receiver side, this combined
with other libraries would allow you to send your data in many many
different ways, as email, upload it to dropbox, flicker, to the THS
website, fax it, print it, store it, 3d print it, etc.
plots in my life.... which is why I get mentioned goosebumps...
Albeit I would be interested in a solution which generates graphs
without using a PC in between.
Arduino->Cloud
As said, it might be rather nice to have a lib, which can create a
vector graphic format on the Arduino and hence upload an image. Since
it would not need any knowledge on the receiver side, this combined
with other libraries would allow you to send your data in many many
different ways, as email, upload it to dropbox, flicker, to the THS
website, fax it, print it, store it, 3d print it, etc.
MRE
May 18, 2012, 11:37:09 AM5/18/12
to TokyoHackerSpace
You can HAVE cake, or you can EAT cake, but you cant HAVE your EATEN
cake.
Unless you get really drunk and put your body in reverse.
On May 19, 12:32 am, Torsten Wagner <torsten.wag...@gmail.com> wrote:
> Well I am not exactly against spredsheets. But I saw to many bad excel
> plots in my life.... which is why I get mentioned goosebumps...
> Albeit I would be interested in a solution which generates graphs
> without using a PC in between.
> Arduino->Cloud
> As said, it might be rather nice to have a lib, which can create a
> vector graphic format on the Arduino and hence upload an image. Since
> it would not need any knowledge on the receiver side, this combined
> with other libraries would allow you to send your data in many many
> different ways, as email, upload it to dropbox, flicker, to the THS
> website, fax it, print it, store it, 3d print it, etc.
>
cake.
Unless you get really drunk and put your body in reverse.
On May 19, 12:32 am, Torsten Wagner <torsten.wag...@gmail.com> wrote:
> Well I am not exactly against spredsheets. But I saw to many bad excel
> plots in my life.... which is why I get mentioned goosebumps...
> Albeit I would be interested in a solution which generates graphs
> without using a PC in between.
> Arduino->Cloud
> As said, it might be rather nice to have a lib, which can create a
> vector graphic format on the Arduino and hence upload an image. Since
> it would not need any knowledge on the receiver side, this combined
> with other libraries would allow you to send your data in many many
> different ways, as email, upload it to dropbox, flicker, to the THS
> website, fax it, print it, store it, 3d print it, etc.
>
Torsten Wagner
May 18, 2012, 1:39:33 PM5/18/12
to tokyohac...@googlegroups.com
Hey MRE,
check this... this is what I get by looking at the SVG specs for a
couple of minutes. Graph is generated by a little program written in
ANSI C... basically, it writes a header with all the graph stuff, adds
the data and writes a foot line.
Shouldn't be much of a problem to get this into an uC. Open the file
in a text editor and you can see how simple it is.
Good night
Totti
check this... this is what I get by looking at the SVG specs for a
couple of minutes. Graph is generated by a little program written in
ANSI C... basically, it writes a header with all the graph stuff, adds
the data and writes a foot line.
Shouldn't be much of a problem to get this into an uC. Open the file
in a text editor and you can see how simple it is.
Good night
Totti
Kalin KOZHUHAROV
May 18, 2012, 9:48:03 PM5/18/12
to tokyohac...@googlegroups.com
Hi Totti,
I've been following this thread but too busy to reply :-|
On Sat, May 19, 2012 at 2:39 AM, Torsten Wagner
<torsten...@gmail.com> wrote:
> Hey MRE,
>
> check this... this is what I get by looking at the SVG specs for a
> couple of minutes. Graph is generated by a little program written in
> ANSI C... basically, it writes a header with all the graph stuff, adds
> the data and writes a foot line.
> Shouldn't be much of a problem to get this into an uC. Open the file
> in a text editor and you can see how simple it is.
>
I agree it is very easy to write SVG and it is relatively easy to view
it (e.g. Firefox).
However I still don't get your point in doing so (except because you can)...
It is trivial to make the same SVG on the server you are uploading
your data to, so unless you are serving the SVG directly off the uC
board, I don't see much use. (e.g.
http://arduino.cc/en/Tutorial/WebServer)
Or I am missing something?
Kalin.
I've been following this thread but too busy to reply :-|
On Sat, May 19, 2012 at 2:39 AM, Torsten Wagner
<torsten...@gmail.com> wrote:
> Hey MRE,
>
> check this... this is what I get by looking at the SVG specs for a
> couple of minutes. Graph is generated by a little program written in
> ANSI C... basically, it writes a header with all the graph stuff, adds
> the data and writes a foot line.
> Shouldn't be much of a problem to get this into an uC. Open the file
> in a text editor and you can see how simple it is.
>
it (e.g. Firefox).
However I still don't get your point in doing so (except because you can)...
It is trivial to make the same SVG on the server you are uploading
your data to, so unless you are serving the SVG directly off the uC
board, I don't see much use. (e.g.
http://arduino.cc/en/Tutorial/WebServer)
Or I am missing something?
Kalin.
MRE
May 18, 2012, 10:43:48 PM5/18/12
to TokyoHackerSpace
Viewing in my phone browser, that is just an ugly line of text.
But googles graph looks real nice.
On May 19, 2:39 am, Torsten Wagner <torsten.wag...@gmail.com> wrote:
> Hey MRE,
>
> check this... this is what I get by looking at the SVG specs for a
> couple of minutes. Graph is generated by a little program written in
> ANSI C... basically, it writes a header with all the graph stuff, adds
> the data and writes a foot line.
> Shouldn't be much of a problem to get this into an uC. Open the file
> in a text editor and you can see how simple it is.
> Good night
> Totti
>
> ...
>
> read more »
>
> graph.svg
> 10KViewDownload
But googles graph looks real nice.
On May 19, 2:39 am, Torsten Wagner <torsten.wag...@gmail.com> wrote:
> Hey MRE,
>
> check this... this is what I get by looking at the SVG specs for a
> couple of minutes. Graph is generated by a little program written in
> ANSI C... basically, it writes a header with all the graph stuff, adds
> the data and writes a foot line.
> Shouldn't be much of a problem to get this into an uC. Open the file
> in a text editor and you can see how simple it is.
> Good night
> Totti
>
>
> read more »
>
> graph.svg
> 10KViewDownload
Mikele
May 18, 2012, 11:01:38 PM5/18/12
to tokyohac...@googlegroups.com
I was having the same thoughts.
if I had to store data in a human and machine readable text form I would use JSON.
pretty much any server or client language can easily parse it and output a graph or do computation.
if I had to store data in a human and machine readable text form I would use JSON.
pretty much any server or client language can easily parse it and output a graph or do computation.
Kalin.
--
You received this message because you are subscribed to the Google Groups "TokyoHackerSpace" group.
Mikele
May 18, 2012, 11:05:48 PM5/18/12
to tokyohac...@googlegroups.com
guys, what are you POSTing exactly?
timestamp should be from the microcontroller obviously... how does it look? is a UNIX timestamp?
sensor reading.
what is the range -256 to 256?
more, less, integer, floating type?
geotags?
???
timestamp should be from the microcontroller obviously... how does it look? is a UNIX timestamp?
sensor reading.
what is the range -256 to 256?
more, less, integer, floating type?
geotags?
???
Akiba
May 18, 2012, 11:41:48 PM5/18/12
to tokyohac...@googlegroups.com
If its going directly to a server, it's probably best for
the server to timestamp it since the server can use NTP to synchronize time. A
microcontroller RTC (real time clock) will drift over time. However if the
data will be buffered on MCU memory and then transmitted/uploaded in batches
then an RTC is needed on the PCB.
Mikele: My suggestion is to assume the server will provide
the timestamp for now. That may change as we understand the systems we're
building better but that should get things started.
From: tokyohac...@googlegroups.com [mailto:tokyohac...@googlegroups.com] On Behalf Of Mikele
Sent: Saturday, May 19, 2012 12:06 PM
To: tokyohac...@googlegroups.com
Subject: Re: [THS:17664] Re: HTTP POST to google spreadsheet
Mikele
May 18, 2012, 11:50:06 PM5/18/12
to tokyohac...@googlegroups.com
that's the easy way.
I was thinking data transmission will cause a (non-constant) lag between reading and server timestamping the record
I am waiting and email with FTP credential for the server.
Crazy that since its Saturday their customer service is not working.
I was thinking data transmission will cause a (non-constant) lag between reading and server timestamping the record
I am waiting and email with FTP credential for the server.
Crazy that since its Saturday their customer service is not working.
Akiba
May 19, 2012, 12:15:50 AM5/19/12
to tokyohac...@googlegroups.com
Most values are changing very slowly compared to server
time lag. MRE's values are posting 1/minute and the values probably need around
5-10 minutes to change significantly (soil moisture). Even a 1 second lag isn't
very significant. The drift over time on an embedded clock that has no way to
synchronize itself would probably be more significant than any lag. The solution
to this would be to use a JJY receiver (Japan atomic time standard, similar to
WWVB in the US) or to use a GPS module on at least one node with a PPS output
that can be used to sync all other nodes. But things get more complex as you try
to sync more accurately. Since that precision is not needed, then its best to
just use the server timestamp.
If you're interested in time accuracy, you should check out
www.leapsecond.com
Those guys are crazy. I was actually researching building a
rubidium disciplined GPS wall clock which would theoretically be accurate to ~10
femto-seconds. Of course, it'd just be an over-engineered wall clock, but it
would have been crazy :)
Sent: Saturday, May 19, 2012 12:50 PM
To: tokyohac...@googlegroups.com
Subject: Re: [THS:17666] Re: HTTP POST to google spreadsheet
Mikele
May 19, 2012, 12:38:59 AM5/19/12
to tokyohac...@googlegroups.com
ok.. you see... I was thinking reading would have been every few mS :)
like Kalin suggested we should take a video of the leap second... maybe using a 360 lens... and lasers
like Kalin suggested we should take a video of the leap second... maybe using a 360 lens... and lasers
Torsten Wagner
May 19, 2012, 8:11:48 AM5/19/12
to tokyohac...@googlegroups.com
Well the idea is that you keep any kind of post-processing of the measurement data out. Its not related to any specific project.
I like MREs project (as usual), but it make me a little bit wonder about the used toolchain. Using a 8bit uC to get the data, feed this to a local PC, crunch numbers to make them ready for upload to the cloud, use a billion dollar cloud framework to get the data online and visualize them.
So first of all, I asked myself would it be possible to reduce this toolchain?!
The other problem I see with the present solution, you need to change the toolchain at several different points (e.g., in the arduino, at the PC, and in the google cloud), if the type of your recorded data change. Sure a server machine is more then capable to generate the graph but you would need to tell it how to do it. What is the exact input, what is the max value, what the min value, is an averaging necessary, should there be an interpolation for missing data points, how do deal with obvious measurement errors? Foremost, you want to convert the AD-value (e.g. 0-255 for 8bit) to the physical unit representation (e.g. temperature in degC). If you operate your own server machine, you can get along with sending metadata too. But using a free service most likely would not allow this kind of detailed settings.
On the other hand, if the uC can generate a svg graph with the current measurement data, you could use many different ways to get it online.
A dedicated webserver on the uC would be only one possibility. Other would be uploading to different web-services (Dropbox & Co, Flicker, Google Docs, etc.), emailing, FTP, scp, etc. All this methods need to have no idea of what data they are actually showing and hence make it totally independent from the used services. Hence one and the same library for a uC which is capable to generate graphs could be used for many different measurement/visualisation projects. Since you know what kind of data your sensor node is going to get, you can make sure the graph is generated on the uC with the correct parameters and that would be the only place where you need to take care of it.
A nice example would be Akibas radiation counter boards. Instead of uploading to pachube, there could be a minimal web-server running and a html page with a refresh rate of e.g. 30s displays a dynamically generated SVG-based graph of the counter measurements. This would make it a stand-alone counter which allows you to have graphs, buttons and inputs in a web-mask format similar as we know from network routers; OS independent accessible from every computer within the network.
So my idea was not directly related against MREs goggle spreadsheet approach but simply my thoughts of how to get more stand-alone operated sensor nodes.
Totti
Kalin KOZHUHAROV
May 19, 2012, 10:41:01 AM5/19/12
to tokyohac...@googlegroups.com
On Sat, May 19, 2012 at 9:11 PM, Torsten Wagner
<torsten...@gmail.com> wrote:
> Well the idea is that you keep any kind of post-processing of the
> measurement data out. Its not related to any specific project.
> I like MREs project (as usual), but it make me a little bit wonder about
> the used toolchain. Using a 8bit uC to get the data, feed this to a local
> PC, crunch numbers to make them ready for upload to the cloud, use a billion
> dollar cloud framework to get the data online and visualize them.
>
> So first of all, I asked myself would it be possible to reduce this
> toolchain?!
>
> The other problem I see with the present solution, you need to change the
> toolchain at several different points (e.g., in the arduino, at the PC, and
> in the google cloud), if the type of your recorded data change. Sure a
> server machine is more then capable to generate the graph but you would need
> to tell it how to do it. What is the exact input, what is the max value,
> what the min value, is an averaging necessary, should there be an
> interpolation for missing data points, how do deal with obvious measurement
> errors? Foremost, you want to convert the AD-value (e.g. 0-255 for 8bit) to
> the physical unit representation (e.g. temperature in degC). If you operate
> your own server machine, you can get along with sending metadata too. But
> using a free service most likely would not allow this kind of detailed
> settings.
>
Well, I agree about the toolchain, especially having a desktop in the
<torsten...@gmail.com> wrote:
> Well the idea is that you keep any kind of post-processing of the
> measurement data out. Its not related to any specific project.
> I like MREs project (as usual), but it make me a little bit wonder about
> the used toolchain. Using a 8bit uC to get the data, feed this to a local
> PC, crunch numbers to make them ready for upload to the cloud, use a billion
> dollar cloud framework to get the data online and visualize them.
>
> So first of all, I asked myself would it be possible to reduce this
> toolchain?!
>
> The other problem I see with the present solution, you need to change the
> toolchain at several different points (e.g., in the arduino, at the PC, and
> in the google cloud), if the type of your recorded data change. Sure a
> server machine is more then capable to generate the graph but you would need
> to tell it how to do it. What is the exact input, what is the max value,
> what the min value, is an averaging necessary, should there be an
> interpolation for missing data points, how do deal with obvious measurement
> errors? Foremost, you want to convert the AD-value (e.g. 0-255 for 8bit) to
> the physical unit representation (e.g. temperature in degC). If you operate
> your own server machine, you can get along with sending metadata too. But
> using a free service most likely would not allow this kind of detailed
> settings.
>
chain is NG for my understanding (these things tend to be rebooted or
get stuck).
> On the other hand, if the uC can generate a svg graph with the current
> measurement data, you could use many different ways to get it online.
> A dedicated webserver on the uC would be only one possibility. Other would
> be uploading to different web-services (Dropbox & Co, Flicker, Google Docs,
> etc.), emailing, FTP, scp, etc. All this methods need to have no idea of
> what data they are actually showing and hence make it totally independent
> from the used services. Hence one and the same library for a uC which is
> capable to generate graphs could be used for many different
> measurement/visualisation projects. Since you know what kind of data your
> sensor node is going to get, you can make sure the graph is generated on the
> uC with the correct parameters and that would be the only place where you
> need to take care of it.
>
> A nice example would be Akibas radiation counter boards. Instead of
> uploading to pachube, there could be a minimal web-server running and a html
> page with a refresh rate of e.g. 30s displays a dynamically generated
> SVG-based graph of the counter measurements. This would make it a
> stand-alone counter which allows you to have graphs, buttons and inputs in a
> web-mask format similar as we know from network routers; OS independent
> accessible from every computer within the network.
>
need one (few) types of output, more or less for direct consumption.
If you need a graph of MRE's garden soil humidity - fine. But if you
want to plot that against temperature you get from WMA, then you'd
better look at numeric data (yes, it may be possible to extract the
digital data from the SVG).
(OMG, this is one ugly way:
http://josiah.berkeley.edu/Instructional/NumbersFromGraphs/ )
To have both camps happy, it will be best if a SVG is generated (may
be on the server, may be on the uC) and numeric data is also sent.
Kalin.
Akiba
May 19, 2012, 11:22:09 AM5/19/12
to tokyohac...@googlegroups.com
Why not just have some canned visualizations and have the full data set for
download. Everybody will want to slice and dice the data according to their
own interest. I want to integrate my air data with wind speed data. Raw
numbers is always the best because then people just have to learn how to use
Excel...oops...errr, I mean gnuplot.
Akiba
FreakLabs Open Source Wireless
Web: http://www.freaklabs.org
Shop:http://www.freaklabsstore.com
Twitter: http://www.twitter.com/freaklabs
download. Everybody will want to slice and dice the data according to their
own interest. I want to integrate my air data with wind speed data. Raw
numbers is always the best because then people just have to learn how to use
Excel...oops...errr, I mean gnuplot.
Akiba
FreakLabs Open Source Wireless
Web: http://www.freaklabs.org
Shop:http://www.freaklabsstore.com
Twitter: http://www.twitter.com/freaklabs
> -----Original Message-----
> From: tokyohac...@googlegroups.com
> [mailto:tokyohac...@googlegroups.com] On Behalf Of Kalin
> KOZHUHAROV
> Sent: Saturday, May 19, 2012 11:41 PM
> To: tokyohac...@googlegroups.com
> From: tokyohac...@googlegroups.com
> [mailto:tokyohac...@googlegroups.com] On Behalf Of Kalin
> KOZHUHAROV
> Sent: Saturday, May 19, 2012 11:41 PM
> To: tokyohac...@googlegroups.com
MRE
May 19, 2012, 12:12:10 PM5/19/12
to TokyoHackerSpace
Interesting ideas Totti and Kalin...
So.. here is my rationalizations ;)
1: I did it in a few hours, with no prior knowledge of how, simply by
mashing together two different ideas that related to each other. Quick
and Dirty. Sometimes, this really is the best way.
2: I dont really envision people using such a method as a replacement
to something like patchube. But it does offer certain advantages over
patchube. I envision a method like this as a great way to do short to
medium term experiments. I set it up because I was curious just how
long it would take for a certain amount of clay pellets to dry out,
after having 2 cups of water dumped on it. Also, I wanted to verify
the long term viability of the method by which I am measuring soil
moisture. After a week, I can throw that data away. It will no longer
have meaning to me, once the experiment is complete. In the case of a
high school student doing some sort of experiment (recording light
data, weather, the number of times their brother goes to the toilet..
etc), they want the data in an easy to digest manner. It is very easy
for people to copy and paste, and one click to graph.
Finally, given that the data is key driven, it is not publicly browse-
able data. Only my friends, whom I have given the link, can read it.
It is private file. But it is a private file on the web, so Mikele
can take my data (because he has the key) and use it for his own
visualization project.
3: This simple idea has gotten Mikele to start working on building a
data server for THS to record our own sensors (and possibly I can put
student projects on it, or replicate his work on a school's server for
their private science use.. etc etc).
4: Why a PC in between? I admit it is not always ideal. However,
again there are certain cases which make it useful. Ultimately, the
design would be:
Multiple wireless nodes communicate to a central node.
Central node has access to the internet, and posts for all.
OR Central node is a my database server.
I am slowly learning about linux with the help of Chris Shannon, and
have three older fanless mini-itx boards with solid state drives, in
the form of SD or CF cards. These will form cheap central hubs for
wireless data nodes. Now, I could put an arduino with netshield or
beaglebone or something else as the central node, and upload data to
google spreadsheets, or that data can be stored on the central node..
lots of options here.
I like having my PC in the chain as a temporary solution as I
experiment and try stuff out. It an intermediate step. I dont like
leaving my laptop on all the time, so it is not my permanent solution.
5: Thinking from a casual builder/experimenter's view: As I said,
Pachube account may not be desireable for a number of reasons (also,
you can get higher post rates to google than pachube). This is an easy
method to do, even if you dont have an ethernet shield. Newbie says "I
have an arduino, and a sensor. I want to get data on my cell phone
when I am at work." This sort of method would work well for them.
Arduino to PC to Google. It is real easy to build up. They can always
add complexity later.
So.. here is my rationalizations ;)
1: I did it in a few hours, with no prior knowledge of how, simply by
mashing together two different ideas that related to each other. Quick
and Dirty. Sometimes, this really is the best way.
2: I dont really envision people using such a method as a replacement
to something like patchube. But it does offer certain advantages over
patchube. I envision a method like this as a great way to do short to
medium term experiments. I set it up because I was curious just how
long it would take for a certain amount of clay pellets to dry out,
after having 2 cups of water dumped on it. Also, I wanted to verify
the long term viability of the method by which I am measuring soil
moisture. After a week, I can throw that data away. It will no longer
have meaning to me, once the experiment is complete. In the case of a
high school student doing some sort of experiment (recording light
data, weather, the number of times their brother goes to the toilet..
etc), they want the data in an easy to digest manner. It is very easy
for people to copy and paste, and one click to graph.
Finally, given that the data is key driven, it is not publicly browse-
able data. Only my friends, whom I have given the link, can read it.
It is private file. But it is a private file on the web, so Mikele
can take my data (because he has the key) and use it for his own
visualization project.
3: This simple idea has gotten Mikele to start working on building a
data server for THS to record our own sensors (and possibly I can put
student projects on it, or replicate his work on a school's server for
their private science use.. etc etc).
4: Why a PC in between? I admit it is not always ideal. However,
again there are certain cases which make it useful. Ultimately, the
design would be:
Multiple wireless nodes communicate to a central node.
Central node has access to the internet, and posts for all.
OR Central node is a my database server.
I am slowly learning about linux with the help of Chris Shannon, and
have three older fanless mini-itx boards with solid state drives, in
the form of SD or CF cards. These will form cheap central hubs for
wireless data nodes. Now, I could put an arduino with netshield or
beaglebone or something else as the central node, and upload data to
google spreadsheets, or that data can be stored on the central node..
lots of options here.
I like having my PC in the chain as a temporary solution as I
experiment and try stuff out. It an intermediate step. I dont like
leaving my laptop on all the time, so it is not my permanent solution.
5: Thinking from a casual builder/experimenter's view: As I said,
Pachube account may not be desireable for a number of reasons (also,
you can get higher post rates to google than pachube). This is an easy
method to do, even if you dont have an ethernet shield. Newbie says "I
have an arduino, and a sensor. I want to get data on my cell phone
when I am at work." This sort of method would work well for them.
Arduino to PC to Google. It is real easy to build up. They can always
add complexity later.
MRE
May 19, 2012, 12:17:49 PM5/19/12
to TokyoHackerSpace
A key point.
The google graph is like.. "click here" and its done. Dead simple. It
can be attached to a blog post.
Having the data available in list format is very useful, since people
can play their own way.
The only processing I am doing is in the Arduino, scaling the ADC
value to a number from 0 to 100 percent.
The scaling is based on raw values I read directly (put it in dry
soil, read the ADC, and called it zero. Put it in a cup of water for
full saturation, called it 100).
Processing on the PC and google neither one alter the value.
I like that I can check in on the sensor while I am at work, from my
phone, WITHOUT needing to download or heat my phone up processing a
java applet. I can read numbers just fine. ;)
Its a temporary experiment. So I kept it simple.
The google graph is like.. "click here" and its done. Dead simple. It
can be attached to a blog post.
Having the data available in list format is very useful, since people
can play their own way.
The only processing I am doing is in the Arduino, scaling the ADC
value to a number from 0 to 100 percent.
The scaling is based on raw values I read directly (put it in dry
soil, read the ADC, and called it zero. Put it in a cup of water for
full saturation, called it 100).
Processing on the PC and google neither one alter the value.
I like that I can check in on the sensor while I am at work, from my
phone, WITHOUT needing to download or heat my phone up processing a
java applet. I can read numbers just fine. ;)
Its a temporary experiment. So I kept it simple.
MRE
May 20, 2012, 7:26:13 AM5/20/12
to TokyoHackerSpace
So I figured soil moisture data was pretty boring.
So I pulled out my DHT11 humidity/temperature sensor and got it
working. I am calculating dew point on the arduino before sending data
out.
Also, I found a very detailed tutorial online for using the TLS230R
light to frequency sensor, so I slapped that on as well.
Again, the arduino is calculating light power and lux. This was all
dreamed up by a camera guy, but I am not sure how accurate those
calculations are.
Anyway, I thought it would be fun to run it tomorrow and check out the
dip as the sun goes black, and the world ends.
Here is my new sheet:
https://docs.google.com/spreadsheet/ccc?key=0ArBwm5DI5eb5dGJ1YmlZMWVUaWM4ZjRYQzVreHBpSWc#gid=0
So I pulled out my DHT11 humidity/temperature sensor and got it
working. I am calculating dew point on the arduino before sending data
out.
Also, I found a very detailed tutorial online for using the TLS230R
light to frequency sensor, so I slapped that on as well.
Again, the arduino is calculating light power and lux. This was all
dreamed up by a camera guy, but I am not sure how accurate those
calculations are.
Anyway, I thought it would be fun to run it tomorrow and check out the
dip as the sun goes black, and the world ends.
Here is my new sheet:
https://docs.google.com/spreadsheet/ccc?key=0ArBwm5DI5eb5dGJ1YmlZMWVUaWM4ZjRYQzVreHBpSWc#gid=0
Matt
May 20, 2012, 4:54:42 PM5/20/12
to tokyohac...@googlegroups.com
MRE, would you point something at the damn hundred mile long cloud parked in front of the sin and make it go away please!
MRE
May 20, 2012, 5:13:17 PM5/20/12
to TokyoHackerSpace
Lasers set to full power
On May 21, 5:54 am, Matt <matthew.em...@gmail.com> wrote:
> MRE, would you point something at the damn hundred mile long cloud parked in front of the sin and make it go away please!
>
On May 21, 5:54 am, Matt <matthew.em...@gmail.com> wrote:
> MRE, would you point something at the damn hundred mile long cloud parked in front of the sin and make it go away please!
>
> On 20/05/2012, at 8:26 PM, MRE <epreme...@gmail.com> wrote:
>
>
>
>
>
>
>
> > So I figured soil moisture data was pretty boring.
> > So I pulled out my DHT11 humidity/temperature sensor and got it
> > working. I am calculating dew point on the arduino before sending data
> > out.
>
> > Also, I found a very detailed tutorial online for using the TLS230R
> > light to frequency sensor, so I slapped that on as well.
> > Again, the arduino is calculating light power and lux. This was all
> > dreamed up by a camera guy, but I am not sure how accurate those
> > calculations are.
>
> > Anyway, I thought it would be fun to run it tomorrow and check out the
> > dip as the sun goes black, and the world ends.
>
> > Here is my new sheet:
> >https://docs.google.com/spreadsheet/ccc?key=0ArBwm5DI5eb5dGJ1YmlZMWVU...
>
>
>
>
>
>
>
> > So I figured soil moisture data was pretty boring.
> > So I pulled out my DHT11 humidity/temperature sensor and got it
> > working. I am calculating dew point on the arduino before sending data
> > out.
>
> > Also, I found a very detailed tutorial online for using the TLS230R
> > light to frequency sensor, so I slapped that on as well.
> > Again, the arduino is calculating light power and lux. This was all
> > dreamed up by a camera guy, but I am not sure how accurate those
> > calculations are.
>
> > Anyway, I thought it would be fun to run it tomorrow and check out the
> > dip as the sun goes black, and the world ends.
>
> > Here is my new sheet:
MRE
May 21, 2012, 6:49:58 AM5/21/12
to TokyoHackerSpace
ok.. so here is the day's data...
interestingly, the temperature and humidity were mostly constant
through the day...
The graph shows light. Right around 730 it drops significantly.
Throughout the remainder of the day, the spikes and dips are due to
cloud cover.
https://docs.google.com/spreadsheet/ccc?key=0ArBwm5DI5eb5dGJ1YmlZMWVUaWM4ZjRYQzVreHBpSWc#gid=0
interestingly, the temperature and humidity were mostly constant
through the day...
The graph shows light. Right around 730 it drops significantly.
Throughout the remainder of the day, the spikes and dips are due to
cloud cover.
https://docs.google.com/spreadsheet/ccc?key=0ArBwm5DI5eb5dGJ1YmlZMWVUaWM4ZjRYQzVreHBpSWc#gid=0
Akiba
May 21, 2012, 6:53:35 AM5/21/12
to tokyohac...@googlegroups.com
Graph looks good. Cool that you also caught the eclipse. One thing I'd
probably recommend is to average the raw data a bit when charting to smooth
things out. That way, its easier to see trends.
Akiba
FreakLabs Open Source Wireless
Web: http://www.freaklabs.org
Shop:http://www.freaklabsstore.com
Twitter: http://www.twitter.com/freaklabs
> -----Original Message-----
> From: tokyohac...@googlegroups.com
probably recommend is to average the raw data a bit when charting to smooth
things out. That way, its easier to see trends.
Akiba
FreakLabs Open Source Wireless
Web: http://www.freaklabs.org
Shop:http://www.freaklabsstore.com
Twitter: http://www.twitter.com/freaklabs
> -----Original Message-----
> From: tokyohac...@googlegroups.com
MRE
May 21, 2012, 12:42:05 PM5/21/12
to TokyoHackerSpace
Yeah.
Mikele was bugging me about posting to his new site, but I was
literally hacking the light sensor code to get something up and
posting SOMEWHERE before I went to bed, so that I would have it going
during the eclipse.
It was my only goal.
I really wanted to see that dip.
Pulled the plug this evening.
I am really digging using google spreadsheets for short term, one shot
data collection.
I haven't done any averaging yet on the raw data. I can play with that
in post, but didnt want to do it on the hardware end. As it is, the
light sensor needs a certain amount of time to count up clicks for
frequency, which naturally does some averaging to the data. The
original code went for 1000ms, but I really did not want to post every
second. During my first test I dropped several thousand data points
before I stopped that horse. Good thing is google doesnt seem to mind,
at least for short bursts.
Anyway, I ended up hacking the code up a bit (that 1000mS was hard
coded and unforgiving. Increase the sensor read time and the readings
go up with it.. fail). I rewrote it such that it averages properly,
regardless of the read period. So, I could set it for 10 seconds and
the average will still report properly. In fact, it is a bit more
accurate over time. The original setting was pretty granular at lower
light levels. For example, it is either 0, or 100, or 400. With a
larger window, you get more time to count clicks, which means less
gets thrown away due to rounding to whole numbers. So, with a 10
second averaging window, Im looking at 10 rather than 100 as increment
units of frequency.
I am sure there is ways to clean that up even more.. but the existing
code is pretty complex. I could simplify it a lot, and skip out on all
the calculations. But the other point is to make sure it plays well
with others. I could get highly accurate frequency counts at high
sample rates, but that would require a trade off. But its not my code
to begin with (only tweaks)... so.. room to grow I suppose.
Mikele was bugging me about posting to his new site, but I was
literally hacking the light sensor code to get something up and
posting SOMEWHERE before I went to bed, so that I would have it going
during the eclipse.
It was my only goal.
I really wanted to see that dip.
Pulled the plug this evening.
I am really digging using google spreadsheets for short term, one shot
data collection.
I haven't done any averaging yet on the raw data. I can play with that
in post, but didnt want to do it on the hardware end. As it is, the
light sensor needs a certain amount of time to count up clicks for
frequency, which naturally does some averaging to the data. The
original code went for 1000ms, but I really did not want to post every
second. During my first test I dropped several thousand data points
before I stopped that horse. Good thing is google doesnt seem to mind,
at least for short bursts.
Anyway, I ended up hacking the code up a bit (that 1000mS was hard
coded and unforgiving. Increase the sensor read time and the readings
go up with it.. fail). I rewrote it such that it averages properly,
regardless of the read period. So, I could set it for 10 seconds and
the average will still report properly. In fact, it is a bit more
accurate over time. The original setting was pretty granular at lower
light levels. For example, it is either 0, or 100, or 400. With a
larger window, you get more time to count clicks, which means less
gets thrown away due to rounding to whole numbers. So, with a 10
second averaging window, Im looking at 10 rather than 100 as increment
units of frequency.
I am sure there is ways to clean that up even more.. but the existing
code is pretty complex. I could simplify it a lot, and skip out on all
the calculations. But the other point is to make sure it plays well
with others. I could get highly accurate frequency counts at high
sample rates, but that would require a trade off. But its not my code
to begin with (only tweaks)... so.. room to grow I suppose.
Torsten Wagner
May 21, 2012, 9:02:34 PM5/21/12
to tokyohac...@googlegroups.com
Hey Emery
nice that it worked out.
> It was my only goal.
Really! You should get a bit higher aims in live ;)
> I really wanted to see that dip.
> Pulled the plug this evening.
> I am really digging using google spreadsheets for short term, one shot
> data collection.
I checked the graph. Yeah its seems to work nice and I agree that its
a good solution if you look out for some quick and dirty tricks. Just
to make it clear again, I am not against the google spreadsheet stuff.
For me doing measurements for a living, I consider the longterm
usability rather important. Well, I have every now and then some
colleagues asking they need the data from measurements 10 years ago
and they saved it in foo2000 which was soooooooo great at that time.
Now, they can't open the file anymore, since foo2000 did stop a year
later and was proprietary. Sometimes I can help by extracting the data
(embedded as ASCII in the file), sometimes I can't (binary blob). The
last resort is then a graph digitizer, kind of OCR for graphs, sure
accuracy is rather bad.
> I haven't done any averaging yet on the raw data. I can play with that
> in post, but didnt want to do it on the hardware end. As it is, the
> light sensor needs a certain amount of time to count up clicks for
> frequency, which naturally does some averaging to the data. The
> original code went for 1000ms, but I really did not want to post every
> second. During my first test I dropped several thousand data points
> before I stopped that horse. Good thing is google doesnt seem to mind,
> at least for short bursts.
Actually, I was going to ask you what is this frequency thing?
> Anyway, I ended up hacking the code up a bit (that 1000mS was hard
> coded and unforgiving. Increase the sensor read time and the readings
> go up with it.. fail). I rewrote it such that it averages properly,
> regardless of the read period. So, I could set it for 10 seconds and
> the average will still report properly. In fact, it is a bit more
> accurate over time. The original setting was pretty granular at lower
> light levels. For example, it is either 0, or 100, or 400. With a
> larger window, you get more time to count clicks, which means less
> gets thrown away due to rounding to whole numbers. So, with a 10
> second averaging window, Im looking at 10 rather than 100 as increment
> units of frequency.
Hmm, I know many people love the Arduino, but I was wondering if a
certain (more capable) sensor node design e.g., based on a ARM Cortex
M3, would be a good idea. This gives you more power to crunch numbers
in the uC and getting some sort of wireless for data transmission and
you are ready to go.
Time to get some dedicated hardware?!
What would be needed?
* Enough computational power to get more complex math done (e.g. FFT,
floating point calculations, etc.)
* A real-time clock for precise timestamps
* Low-Power save mode to allow long operation times e.g. on solar
panels or batteries
* Inputs for all kind of sensors (voltage, resistance, current,
frequency, time, impedance?)
* Basic outputs for switching (Digital, PWM, DAC?)
* Wireless communication including bootloader (*Akibaaaaaaa*)
* Alternative power-inputs, solar, battery, wallet, maybe some energy
harvesting
* Enough memory to save data in between pulls via wireless (e.g.
measure every 5 min but send data over only every few hours to
preserve energy).
Which controller might be up to this task. I could see a PSoC3 or
PSoC5 for this design. Only drawback would be the proprietary (but
free to download) IDE to get the analogue and CPLD parts programmed.
But it would give a great freedom in terms of all kind of in- and
outputs.
Parallax Propeller is another nice uC which is heavily ignored. 8
cores @ 32 bit gives you enough power to get many things done in
parallel. Drawback here would be the fact that it comes only with
digital I/Os and counters. ADC/DAC and other stuff would require some
implementation on the PCB (which could be as easy as a resistor
ladder, etc)
What do you think?
Totti
nice that it worked out.
> It was my only goal.
> I really wanted to see that dip.
> Pulled the plug this evening.
> I am really digging using google spreadsheets for short term, one shot
> data collection.
a good solution if you look out for some quick and dirty tricks. Just
to make it clear again, I am not against the google spreadsheet stuff.
For me doing measurements for a living, I consider the longterm
usability rather important. Well, I have every now and then some
colleagues asking they need the data from measurements 10 years ago
and they saved it in foo2000 which was soooooooo great at that time.
Now, they can't open the file anymore, since foo2000 did stop a year
later and was proprietary. Sometimes I can help by extracting the data
(embedded as ASCII in the file), sometimes I can't (binary blob). The
last resort is then a graph digitizer, kind of OCR for graphs, sure
accuracy is rather bad.
> I haven't done any averaging yet on the raw data. I can play with that
> in post, but didnt want to do it on the hardware end. As it is, the
> light sensor needs a certain amount of time to count up clicks for
> frequency, which naturally does some averaging to the data. The
> original code went for 1000ms, but I really did not want to post every
> second. During my first test I dropped several thousand data points
> before I stopped that horse. Good thing is google doesnt seem to mind,
> at least for short bursts.
> Anyway, I ended up hacking the code up a bit (that 1000mS was hard
> coded and unforgiving. Increase the sensor read time and the readings
> go up with it.. fail). I rewrote it such that it averages properly,
> regardless of the read period. So, I could set it for 10 seconds and
> the average will still report properly. In fact, it is a bit more
> accurate over time. The original setting was pretty granular at lower
> light levels. For example, it is either 0, or 100, or 400. With a
> larger window, you get more time to count clicks, which means less
> gets thrown away due to rounding to whole numbers. So, with a 10
> second averaging window, Im looking at 10 rather than 100 as increment
> units of frequency.
certain (more capable) sensor node design e.g., based on a ARM Cortex
M3, would be a good idea. This gives you more power to crunch numbers
in the uC and getting some sort of wireless for data transmission and
you are ready to go.
Time to get some dedicated hardware?!
What would be needed?
* Enough computational power to get more complex math done (e.g. FFT,
floating point calculations, etc.)
* A real-time clock for precise timestamps
* Low-Power save mode to allow long operation times e.g. on solar
panels or batteries
* Inputs for all kind of sensors (voltage, resistance, current,
frequency, time, impedance?)
* Basic outputs for switching (Digital, PWM, DAC?)
* Wireless communication including bootloader (*Akibaaaaaaa*)
* Alternative power-inputs, solar, battery, wallet, maybe some energy
harvesting
* Enough memory to save data in between pulls via wireless (e.g.
measure every 5 min but send data over only every few hours to
preserve energy).
Which controller might be up to this task. I could see a PSoC3 or
PSoC5 for this design. Only drawback would be the proprietary (but
free to download) IDE to get the analogue and CPLD parts programmed.
But it would give a great freedom in terms of all kind of in- and
outputs.
Parallax Propeller is another nice uC which is heavily ignored. 8
cores @ 32 bit gives you enough power to get many things done in
parallel. Drawback here would be the fact that it comes only with
digital I/Os and counters. ADC/DAC and other stuff would require some
implementation on the PCB (which could be as easy as a resistor
ladder, etc)
What do you think?
Totti
MRE
May 22, 2012, 3:01:38 AM5/22/12
to TokyoHackerSpace
My response to your response of my response:
On May 22, 10:02 am, Torsten Wagner <torsten.wag...@gmail.com> wrote:
> Hey Emery
>
Hey Torsten!
> nice that it worked out.my response to your response of my response:
>
> > I really wanted to see that dip.
> > Pulled the plug this evening.
> > I am really digging using google spreadsheets for short term, one shot
> > data collection.
>
> I checked the graph. Yeah its seems to work nice and I agree that its
> a good solution if you look out for some quick and dirty tricks. Just
> to make it clear again, I am not against the google spreadsheet stuff.
> For me doing measurements for a living, I consider the longterm
> usability rather important. Well, I have every now and then some
> colleagues asking they need the data from measurements 10 years ago
> and they saved it in foo2000 which was soooooooo great at that time.
> Now, they can't open the file anymore, since foo2000 did stop a year
> later and was proprietary. Sometimes I can help by extracting the data
> (embedded as ASCII in the file), sometimes I can't (binary blob). The
> last resort is then a graph digitizer, kind of OCR for graphs, sure
> accuracy is rather bad.
Yeah, could see that as an issue. Google could tap out on docs
anytime.
Hopefully Mikele's data server wont be flakey like google ;)
As for long term storage... sounds like csv is the safest bet.
>
> > I haven't done any averaging yet on the raw data. I can play with that
> > in post, but didnt want to do it on the hardware end. As it is, the
> > light sensor needs a certain amount of time to count up clicks for
> > frequency, which naturally does some averaging to the data. The
> > original code went for 1000ms, but I really did not want to post every
> > second. During my first test I dropped several thousand data points
> > before I stopped that horse. Good thing is google doesnt seem to mind,
> > at least for short bursts.
>
> Actually, I was going to ask you what is this frequency thing?
The sensor is a light to frequency converter. More light = higher
frequency.
The cpu just counts interrupt clicks for 1 second. But in very low
light, the frequncy may be in single digit hz or even 0.whatever hz.
Yet the minimum value out the uC is 10. Thus its real "steppy" in the
low end.
For dedicated sensors, DSPIC is the defacto standard. A good
compromise between performance, power consumption, and scaled to task.
Can do filtering and lots of other software-analog goodness.
ARM and Propeller are way over spec for sensors.
Thus if you can live without computation analysis onboard, arduino is
actually a good choice, athough it could perform better in power
management.
On May 22, 10:02 am, Torsten Wagner <torsten.wag...@gmail.com> wrote:
> Hey Emery
>
Hey Torsten!
> nice that it worked out.my response to your response of my response:
>
> > It was my only goal.
>
> Really! You should get a bit higher aims in live ;)
Nah.. that smacks of effort.
> > It was my only goal.
>
> Really! You should get a bit higher aims in live ;)
>
> > I really wanted to see that dip.
> > Pulled the plug this evening.
> > I am really digging using google spreadsheets for short term, one shot
> > data collection.
>
> I checked the graph. Yeah its seems to work nice and I agree that its
> a good solution if you look out for some quick and dirty tricks. Just
> to make it clear again, I am not against the google spreadsheet stuff.
> For me doing measurements for a living, I consider the longterm
> usability rather important. Well, I have every now and then some
> colleagues asking they need the data from measurements 10 years ago
> and they saved it in foo2000 which was soooooooo great at that time.
> Now, they can't open the file anymore, since foo2000 did stop a year
> later and was proprietary. Sometimes I can help by extracting the data
> (embedded as ASCII in the file), sometimes I can't (binary blob). The
> last resort is then a graph digitizer, kind of OCR for graphs, sure
> accuracy is rather bad.
anytime.
Hopefully Mikele's data server wont be flakey like google ;)
As for long term storage... sounds like csv is the safest bet.
>
> > I haven't done any averaging yet on the raw data. I can play with that
> > in post, but didnt want to do it on the hardware end. As it is, the
> > light sensor needs a certain amount of time to count up clicks for
> > frequency, which naturally does some averaging to the data. The
> > original code went for 1000ms, but I really did not want to post every
> > second. During my first test I dropped several thousand data points
> > before I stopped that horse. Good thing is google doesnt seem to mind,
> > at least for short bursts.
>
> Actually, I was going to ask you what is this frequency thing?
frequency.
The cpu just counts interrupt clicks for 1 second. But in very low
light, the frequncy may be in single digit hz or even 0.whatever hz.
Yet the minimum value out the uC is 10. Thus its real "steppy" in the
low end.
compromise between performance, power consumption, and scaled to task.
Can do filtering and lots of other software-analog goodness.
ARM and Propeller are way over spec for sensors.
Thus if you can live without computation analysis onboard, arduino is
actually a good choice, athough it could perform better in power
management.
Mikele
May 22, 2012, 3:29:47 AM5/22/12
to tokyohac...@googlegroups.com
I keep thinking about it...
and I believe for long term storage, a JSON object would be idea...
it can be quite a complex object, and still human readable... like xml but more compact and better suited for transmission.
I think this:
sensor(s) send its sensed input to a microcontroller
microcontroller does all the concertation between sensors and compute the output(s) that it is expected to give
outputs should be stored somehow, locally (an sd card on spot) or remotely (on a pc disk as a file, on a DB...)
in both case I think outputting JSON would be ideal.. for both having a text file with the data that can be viewed in a text editor and for transmission to another machine.
I would bet any server or client side language has a library to deal with JSON
rendering into graph or presentation should always be done on client side, using the data provided.
Clients like we browser are technology aware... in time they evolve and gets better at doing stuff...
data should be simple and well structured... no need to go back and change the microcontroller settings or data archives if in future we will all have implants in our cortex to project images directly in our retina.
What I did last week-end was a proof of concept, still need to write off a bug (tonight with MRE help) or two and a bare bone webpage+DB to store and sensor display readings can be available for anybody....
BUT
I am sure every user will come up with features requests, different sets of needs etc etc
evolving that page into something that can fit any need etc would require some programming...
In my head PHP and Javascript are simple enough for anybody to customize the scripts themselves...
BUT
after I watched MRE google docs files and having followed this thread... it might be that for an electronic technician dealing with sensors, it is less time consuming to play with a spreadsheet than coding his way out.
for what I did I see 3 possibility
1) we make it work, and that was it... a learning experience
2) we make it work, and it stays limited to THS members use, for their instant tests (live view on a iPad while soldering stuff)
3) we make it work, and we set up a developers' team for the project... we move it to a public server where people can collaborate on the software and see where it leads.
in any case... my point is: JSON is awesome.
and I believe for long term storage, a JSON object would be idea...
it can be quite a complex object, and still human readable... like xml but more compact and better suited for transmission.
I think this:
sensor(s) send its sensed input to a microcontroller
microcontroller does all the concertation between sensors and compute the output(s) that it is expected to give
outputs should be stored somehow, locally (an sd card on spot) or remotely (on a pc disk as a file, on a DB...)
in both case I think outputting JSON would be ideal.. for both having a text file with the data that can be viewed in a text editor and for transmission to another machine.
I would bet any server or client side language has a library to deal with JSON
rendering into graph or presentation should always be done on client side, using the data provided.
Clients like we browser are technology aware... in time they evolve and gets better at doing stuff...
data should be simple and well structured... no need to go back and change the microcontroller settings or data archives if in future we will all have implants in our cortex to project images directly in our retina.
What I did last week-end was a proof of concept, still need to write off a bug (tonight with MRE help) or two and a bare bone webpage+DB to store and sensor display readings can be available for anybody....
BUT
I am sure every user will come up with features requests, different sets of needs etc etc
evolving that page into something that can fit any need etc would require some programming...
In my head PHP and Javascript are simple enough for anybody to customize the scripts themselves...
BUT
after I watched MRE google docs files and having followed this thread... it might be that for an electronic technician dealing with sensors, it is less time consuming to play with a spreadsheet than coding his way out.
for what I did I see 3 possibility
1) we make it work, and that was it... a learning experience
2) we make it work, and it stays limited to THS members use, for their instant tests (live view on a iPad while soldering stuff)
3) we make it work, and we set up a developers' team for the project... we move it to a public server where people can collaborate on the software and see where it leads.
in any case... my point is: JSON is awesome.
Nava Whiteford
May 22, 2012, 3:40:49 AM5/22/12
to tokyohac...@googlegroups.com
> For dedicated sensors, DSPIC is the defacto standard. A good
> compromise between performance, power consumption, and scaled to task.
> Can do filtering and lots of other software-analog goodness.
> ARM and Propeller are way over spec for sensors.
> Thus if you can live without computation analysis onboard, arduino is
> actually a good choice, athough it could perform better in power
> management.
As a matter of interest, what kind of power supply/battery life are you after?
> compromise between performance, power consumption, and scaled to task.
> Can do filtering and lots of other software-analog goodness.
> ARM and Propeller are way over spec for sensors.
> Thus if you can live without computation analysis onboard, arduino is
> actually a good choice, athough it could perform better in power
> management.
MRE
May 22, 2012, 3:44:56 AM5/22/12
to TokyoHackerSpace
another is MSP430 by TI.... again, a small micro, acceptable amount of
storage, good power management, and mixed signal processing.
storage, good power management, and mixed signal processing.
Torsten Wagner
May 22, 2012, 4:41:30 AM5/22/12
to tokyohac...@googlegroups.com
Yeah the MSP430 is a good candidate indeed.
ultra-low-power (it might run just from sticking some metal wires into
the soil ;)
enough computational power and (I guess this is important too) a nice
gcc compiler (something dsPIC miss)
There were some with wireless too if I remember correctly...
Totti
ultra-low-power (it might run just from sticking some metal wires into
the soil ;)
enough computational power and (I guess this is important too) a nice
gcc compiler (something dsPIC miss)
There were some with wireless too if I remember correctly...
Totti
Nava Whiteford
May 22, 2012, 5:03:35 AM5/22/12
to tokyohac...@googlegroups.com
In my limited experience they are indeed a good time.
Sounds like a fun project, if there's anything I can help with let me know.
Sounds like a fun project, if there's anything I can help with let me know.
Kalin KOZHUHAROV
Jun 6, 2012, 10:57:43 PM6/6/12
to tokyohac...@googlegroups.com
Talking about air-quality...
http://www.shef.ac.uk/news/nr/kitchen-pollution-air-university-sheffield-city-1.183749
Kalin.
http://www.shef.ac.uk/news/nr/kitchen-pollution-air-university-sheffield-city-1.183749
Kalin.
Mikele
Jun 6, 2012, 11:31:47 PM6/6/12
to tokyohac...@googlegroups.com
full study is not free to download... not much, but need to register and all...
Kalin.
Reply all
Reply to author
Forward
0 new messages