Data logging

[eppie]

Hello,

I was wondering how you guys have implemented the logging of data and how to update/delete records after communication.

At the moment I am using a recordstore for saving data and when a record is sended to the server it will be deleted from the recordstore. This approach is working but I am not really satisfied by this approach because of the high change of data corruption and the performance. 

Also I am saving the data as string and I found out that this uses much memory. (A string of 2000 characters takes up a huge amount of memory!)

If I going with storing it into files what would be a good approach? Multiple files? 

I am looking forward to share some idea's here. 

Thanks,

Eppie20

[ejw]

Hi,

 

i wouldn't touch a record store unless you absolutely must.

 

On my first build of our project I captured data and put it into record stores, but they are very inconsistent. I switched all my data handling out to vector files and now add and delete data through the vector, and have a write to text file option of power goes down.

 

Personally, if you can use vectors, do so rather than recordstores.

 

ejw

[Florent Clairambault]

Hi Eppie,

For that purpose I created a "SafeQueue" component. We define a max number of files, a maximum size per file, a maximum number of lines in memory, a minimum free size on device, we add data as lines and read data as lines. By default, it is set to 20 files of 8KB each. We add data to a memory buffer (a Vector), we transmit in real-time or at once, if it reached a certain amount of data, we acknowledge the data. If we could do it, we empty this memory buffer. If we couldn't we ask the safequeue to save it. 

We can also use this to send binary data by encoding it in base64 (but this is outside the scope of the "safequeue").

Once some data could be acknowledged and we are in a "good" situation, we ask the safequeue if we have some more data to send.

I've used this on top of HTTP (disconnected) or M2MP (kept alive connection) but this idea works pretty much everywhere.

Best regards,

--
Florent Clairambault

--
Florent Clairambault

Cell : +33 6 86 95 54 05
Home : +33 1 83 64 24 64
Fax : +33 9 55 53 76 87

--
 
 

[eppie]

Thanks for the replies!

I will dump the recordstore and will go with files. I have to write it to storage instead of memory because of the size of the received message. (2kb and up).

Ejw, with vector files you mean a vector backupped by a file? If so how do you handle the removal by index? 

Florent, that SafeQueue looks very promissing, is it a public available library?

Op dinsdag 18 december 2012 11:49:51 UTC+1 schreef eppie het volgende:

[ejw]

Hi eppie,

I don't backup the vector file normally. Data flows through the vector, as it is added to the file the count of records increases. Our code reads the vector(0) record, transmits it then deletes vector(0) (see below*). This way the data is always sent out in order of first saved first out basis. We have a monitor on a pin that checks the power level. On power failure we use the few miiliseconds we have to write the contents of the vector into a txt file on the non-volatile area. So it is a sort of backup, but only in a power out situation.

* This is all done via UDP so we have a check routine that makes sure that the packets transmitted reach the database server before the actual records are deleted from the vector.

ejw

[Ahmed ADNANE]

Hello,

Find attached the source code of the PersistentVector class that I built myself, it allows saving data in flash memory in very safe and optimal way, I used it in many projects and the results were great.

Hope that will help,

2012/12/19 ejw <sup...@virtualict.co.uk>

--
 
 

--
ADNANE Ahmed

[eppie]

Ahmed thank you for you code!

Op woensdag 19 december 2012 21:40:38 UTC+1 schreef Ahmed ADNANE het volgende:

[eppie]

Ahmed,

Does your implementation also really delete bytes from the file?

Op vrijdag 21 december 2012 11:05:19 UTC+1 schreef eppie het volgende:

[Ahmed ADNANE]

Eppie,

No the implementation doesn't delete any bytes from the file, it's about a FIFO file that contains recordstores as Strings, when yo add elements to the PersistantVector and it's physically (associated file) empty, then new Data is written to the file and its size is growing up until the maximum size you specified, when the file reaches its maximum size it keeps it forever and never changes, now when you delete an element from a full PersistantVector, it's only the size field in the file that is modified, so your PersistantVector is not logically full but it is physically, when you add new element then, new data replaces former data in the file and the size field is modified. All that is done in a transparent way vis-a-vis the programmer.

2012/12/21 eppie <epp...@gmail.com>

--
 
 

--
ADNANE Ahmed

[eppie]

Ok now I understand your approach! Nice and really a good concept. I didn't think about overwriting instead of deleting.

Op vrijdag 21 december 2012 12:59:20 UTC+1 schreef Ahmed ADNANE het volgende:

[Florent Clairambault]

Hi,

Sorry but no, this library is not opensource, I was giving you the ideas behind a component of it. 

About the file vector, it's definitely useful and well thought but:

* Writing frequently might not be good for the flash (but we haven't found any proof to support that). 

* There really is no point in making everything static. It would be useful to have more than one instance of your file vector. You could drop the static and allow some constructors like: PersistantVector( String fileName, int frameSize );

* It uses a fixed-width 256 bytes cell. Which means 4 elements consumes 1KB. This isn't space efficient and if you do need to save a string of 2000 chars you need to increase this to at least 2000 (hoping you won't use unicode chars). I think storing the space of each element before writing it would be a lot more space efficient. If you need to quickly append data, you could keep the end offset in the beginning of the file.

Currently it's:

[int: number of elements] [256 bytes: element 1] [256 bytes: element 2] [256 bytes: element 3]

You could have:

[int: number of elements] [int: offset after last element (the next byte to write if you prefer)] [int: size of element 1] [n bytes: element 1] [int: size of element 2] [n bytes: element 2] [int: size of element 3] [n bytes: element 3]

Best regards,

--
Florent

--
 
 

[eppie]

Hi Florent,

Thanks for your reply and I also modified the provided code to support dynamic size structures. At the moment the implementation is way slower then the recordstore so I think I have to make some optimalisations. :)

Op zaterdag 29 december 2012 02:57:59 UTC+1 schreef Florent het volgende: