Rpi long-term stability and SD card wear minimizing

[Remigiusz Zukowski]

Hello,

is there anybody running weewx on Rpi for some reasonably long time (e.g. half year)? I am curious how stable this system is, especially if there's some problem with SD card wear caused by excessive writes.
I still haven't tuned my weewx Rpi configuration but I did something  to minimize SD card writes. First of all I installed ramlog: http://www.tremende.com/ramlog/ which moves /var/log to a ramdisk and synchronizes its hdd copy upon system startup/shutdown. Another thing I did is mounting 10M ramdisk into public_html directory, so the generated webpages resides on ram, not on the disk. The only drawback is that after restart it requires max. 5 minutes to generate webpage but this is fully acceptable for me. I haven't touched database files, they are on SD card. I hope that database writes are very small and thus negligible. One question to the weewx developers: is weewx creating any temporary files during its job?

best regards

Remek

[Alex Woo]

The SD Card is definitely the weak link to a R-Pi.   The cards are write limited but some are better than others.   At the suggestion of someone on this list, I'm using Samsung cards exclusively and, so far, no failures.

Alex

[Andrew Milner]

My pi has been running 24/7 since about January with no problems.  I only use the SD card when rebooting as I use an external USB HD for the system and datbases etc so as to not have any SD problems that WILL occur at some time or another.

Using RAM disk is certainly one way to minimise SD card accessing, although I would run a CRON job to do the transferring to SD as you will probably not be booting/restarting the Pi that often!!!  There are many views on SD cards etc and I suspect every Pi user has different viewpoints!!!!!

[Thomas Keffer]

Database writes happen once per archive interval. The database transaction creates a journaling file. No other writes, no other temporary files.

-tk

--
You received this message because you are subscribed to the Google Groups "Weewx user's group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[William Phelps]

I have 2 RPi's running weewx, one at home and one at the nearby elementary school. Both have been up for over 11 months now. The school's has a 5 minute archive interval, mine is set to 1 minute. I had problems at first with cheap SD cards, so I started doing some research. I think a lot of the problems people have with SD cards on the Pi are because they use cheap cards. I bought several different cards and did some testing, mostly for read & write performance. The best card I found at the time was the Samsung Pro Class 10 SDHC. Class 6 cards are rated at 1 million write cycles, but class 10 is higher, not sure how much though, it's difficult to find specifications for this from the manufacturers.

When I first started having problems, I noticed that when the SD card started failing, the kernel IO wait times went up significantly, due to error recovery on the card, so I added a weewx extension to collect and graph kernel cpu times. If the SD card starts to fail again I should be able to see it in the data.

I also take backups of the database every night to a NAS. You could get cheap USB memory and back up to that if you don't have network attached storage. It's very tempting now to try backing up to something like Amazon S3 space, it's so inexpensive.

I use 16 GB SD memory cards, and my file system is currently at 25%. That gives the card 12 GB of space to use for wear leveling,  I'm going to run these until they fail and see how long that takes. Samsung guarantees the card for 10 years, for what that's worth. Given the prices of these cards now, I think this is the way to go. It's a lot less hassle, and less $$ than running an external HD (and they fail too). 

William

[Alex Woo]

William,

I wish more schools had weather stations.   Mine's near JLS.

Could you send me your extensions to collect and graph kernel cpu times?   

Thanks,

Alex Woo

http://www.linkedin.com/in/alexwoo

--
You received this message because you are subscribed to a topic in the Google Groups "Weewx user's group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/JxWgTEAikbA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.

[vds]

On Wednesday, December 4, 2013 10:04:22 PM UTC-8, William Phelps wrote:

I use 16 GB SD memory cards, and my file system is currently at 25%. That gives the card 12 GB of space to use for wear leveling,  I'm going to run these until they fail and see how long that takes. Samsung guarantees the card for 10 years, for what that's worth. Given the prices of these cards now, I think this is the way to go. It's a lot less hassle, and less $$ than running an external HD (and they fail too). 

I've had horrible luck with USB sticks and SD cards in the past on multiple platforms, but most recently the Pi has been running nicely for me as a webcam server (only) running 'motion' off SD only.  Uptime now over a month running current Raspbian.  Previously it was fine for a month or so but we had some bouncy power here in a storm and after the 6th power bounce the SD didn't come back.  If I had a UPS then it would have probably just stayed up. 

Just a suggestion for the pi, but getting your system set up and running a nice controlled 'dd' to create+save a gold image is easy and effective.  If the SD card goes, just dd the gold image back.  Ten minute restore.  If I was running weewx on the box, it would catch up to any data logger records automagically.  Probably my favorite feature :-)

I've been running just short of 3 years now on a Seagate Dockstar (essentially a Sheeva Plug) using a Dockstar spinning laptop drive.  100% uptime other than power failures and administrator stupidity.  I back up my databases to Dropbox when I remember to do so.  Probably should just rsync them up to S3 once a week or so as you suggest.  That's pretty hands-off too.

[William Phelps]

There is a very nice tool "rpi-clone" which will clone your SD card for you - using a USB SD card adapter

https://github.com/billw2/rpi-clone

[david murphy]

Would you mind sharing the dd script. I was under the impression you needed to umount the partition before you could reliably use dd. I am aware of rpi-clo Ne but would prefer to use my NAS as a repository.

Ta.

[david murphy]

[William Phelps]

Why you would want to dd the entire file system to a NAS? If the card fails you would need to install a working system and then dd the file system back over the top of that? Can you even do that?

Why not make a one time backup of the card (once you've got everything working the way you like) and then just back up just the database and any other files that might be changed? I run a simple shell script nightly that backs up the database to my NAS, and once a week another one runs that backs up /home/weewx to the NAS. I used win32diskimager to make a copy of the SD card once I had the everything working the way I wanted. Once in a while I run win32diskimager or rpi-clone again and copy the image to the NAS.

William

[david murphy]

Ta William,

I'm looking at a total hands of approach. Familiar with all you point out but would like to be in a situation where if I lose an sd card all I want to do is replace stuffed sd card with a Virgin install and restore backup over that.

Only time I want to visit (under the house) is when system is (pi / weewx software) dead. No intervention.

[vds]

On Saturday, December 7, 2013 4:33:51 AM UTC-8, david murphy wrote:

I'm looking at a total hands of approach. Familiar with all you point out but would like to be in a situation where if I lose an sd card all I want to do is replace stuffed sd card with a Virgin install and restore backup over that.

Perhaps you should as your question on the Raspi forums, as what you're asking has nothing to do with the application software (weewx) you're using the Pi for. 

Regardless, I load my SD card from a Macbook Air which has an internal SD slot. The following are the mac commands basically verbatim from the RasPi website.  

To save the golden image once you have your SD card correct, shutdown the pi in an orderly manner and then dd 'from' the SD card to a saved image file

# insert card, do a df, verify that it's mounted as /dev/disk1s1

#  - unmount the card and dd it, then eject

  sudo diskutil unmount /dev/disk1s1

  sudo dd bs=1m if=/dev/rdisk1 of=raspi-configured-stashed-image.img

  sudo diskutil eject /dev/rdisk1

To restore the saved image to a new SD card, just switch the "if" and "of" values above.  For a image=>SD restore your input file (if) would be the saved image file.  Your output file would be the device the SD card is at. 

The only totally hands-off approach I can think of is network booting the Pi off a fileserver in your LAN.  You might want to look into some google queries similar to "network boot raspberry pi" and experiment  a bit (again, this is 'not' the forum for those questions), but one pointer to get you started might be http://www.whaleblubber.ca/boot-raspberry-pi-nfs/

 

[Adrian Chandler]

You may be interested in this - Raspberry Pi - Auto Backups - that seems to do what you ask. I've not tried it myself, but I intend to give it a go.
 

[david murphy]

Adrian,

Great find, Thanks for sharing.

[david murphy]

Vds,

Much appreciate your reply. I had a look in raspberrypi.org but couldn't find exactly what I thought would be a common type script. Between what you and Adrian have provided I hope to get a solution up and running shortly.

Thanks again.

[vds]

On Monday, December 9, 2013 3:46:38 AM UTC-8, david murphy wrote:

Vds,

Much appreciate your reply. I had a look in raspberrypi.org but couldn't find exactly what I thought would be a common type script. Between what you and Adrian have provided I hope to get a solution up and running shortly.

Try http://www.raspberrypi.org/downloads and http://elinux.org/RPi_Easy_SD_Card_Setup for more pointers

 

[Branislav Vedel]

As several people wrote already, by far the weakest point of Raspberry Pi is are the writing cycles to the SD card. While the risk of a corrupted SD card can be decreased with using a good SD card, the risk stays there.
My set-up is the following:
- I mounted the whole log folder to RAM
- I pointed all files generated by weewx (html, images, logs) to a dedicated folder (actually /var/weewx), which is also mounted to RAM
- I switched from sqlite to mysql dB, which is located on a "proper" server on my LAN
- I don't use any RP overclocking, as it seems to contribute to SD corruptions

So far, the system seems stable and running 24x7, going for reboots only in case of power outages.

[Steve]

Hi all,

I'm running weewx on an RPi.

I already run ramlog and have all the html pages and images generated to a directory in RAM to try and help minimise the SD card wear.

But is there anyway of limiting weewx messages to only syslog? (even though all log are in RAM) At the moment I can see 'most' of what weewx is doing in user.log and messages as well as syslog, the information is being logged into 3 separate log files. (It might be a Pi issue, but user.log seems exclusively weewx data, messages entries are 99.9% weewx data, syslog is a mix of events, but I thought I'd ask here).

Regards,

Steve.

[mwall]

On Monday, December 23, 2013 10:08:23 AM UTC-5, Steve wrote:

But is there anyway of limiting weewx messages to only syslog? (even though all log are in RAM) At the moment I can see 'most' of what weewx is doing in user.log and messages as well as syslog, the information is being logged into 3 separate log files. (It might be a Pi issue, but user.log seems exclusively weewx data, messages entries are 99.9% weewx data, syslog is a mix of events, but I thought I'd ask here).

steve,

this is an operating system issue, more specifically a syslog/rsyslog issue.  redhat does it one way, debian does it another, and even different versions of debian do it differently.

if you are running a debian variant on your pi, then rsyslog is probably controlling which log messages go to which files.

take a look in /etc/rsyslog.conf and /etc/rsyslog.d/

m

[Steve]

Thanks Matthew,

Another learning curve to climb.

Looks like all the fun is in /etc/rsyslog.conf as there's nothing in  /etc/rsyslog.d/

FWI I'm running Raspbian, a Debian wheezy port for the RPi.

Regards,

Steve.

[Branislav Vedel]

Hi Steve,
the /etc/rsyslog.d folder is reserved for configuration files, per application; e.g. if you put file weewx.conf there with specific syntax, all the log messages generated by weewx will "obey" the rules.
I'm attaching my conf file, basically I redirected all log messages generated by weewx to a dedicated file weewx.log, located in /var/weewx/log folder. I believe you get the idea...
Then the whole folder /var/weewx is mounted to RAM space. Of course, if your RP crashes, the files from RAM are gone. Then you may want to rsync/ftp them to a remote server or a USB HDD/stick.

[William Phelps]

I am curious why you would go to all this trouble. Why not just use a good Class 10 SD card and be done with it?

William

[Steve]

Thank you very much for that.

I have it up and running, and it appears to be working perfectly. As I'm already running ramlog, I just rely on it copying the log files to the SD card once a day and for the time being that is good enough - especially as the weewx data copied is now 1/3rd of what it used to be.

Never a dull moment. Who would have thought simply wanting to log my weather data would give me so much.

[Steve]

Hi William,

I hope I am using a good Class 10 SD card. It is oversized and I'm hoping the wearing algorithm works well, but my RPi is also logging my energy consumption, 10 inputs at 6 second intervals means my MySQL database cops a hammering (I have weewx writing to it too). It's been running measureit since July and weewx since the middle of November without drama and I'd like to keep it that way. Less is more ;)

I hope the poor Pi can handle it: http://wotid.dyndns.org/

Regards,

Steve.

On Tuesday, 24 December 2013 04:07:40 UTC+11, William Phelps wrote:

[Branislav Vedel]

Obvious reason would be that I want to keep the writes to the SD card to the absolute minimum.
Not-so-obvious reason is that I have a proper server up & running, I'm trying to improve my linux skills and remote backing up with rsync with password-less authentication ;)

[vds]

On Monday, December 23, 2013 9:16:05 AM UTC-8, Steve wrote:

I hope the poor Pi can handle it: http://wotid.dyndns.org/

I'm running weewx on a Seagate Dockstar (essentially a Pogoplug with half the memory) and it's not even breathing hard running weewx using sqlite3, plus syncing data up to AWS every few minutes.

Try adding the monitoring extension to your weeex to see how hard the Pi is working...

http://sourceforge.net/p/weewx/wiki/monitor/

There are a number of command line monitoring tools including lm-sensors, but I don't know firsthand if the Pi supports any of that.

Your energy monitoring stuff looks pretty cool, incidentally.

 

[William Phelps]

I'm running a Vantage Pro 2 with the interval set to 1 minute, with extensions to collect data on processor usage, temperature, etc, and also collecting data from a TED5000 energy monitor. I converted the school farm station (also running weewx on an R-PI) to 5 minute interval months ago, but left mine at 1 minute on purpose to stress test the SD card. It's been up since April and the kernal IO Wait times are still flat line at or very near 0. (when the older SD card I was using started to go, the IO Wait time was a clear indicator of problems).

None of this proves that all Class 10 cards won't have problems of course. I just question spending much time on efforts to reduce SD card writes when newer cards don't seem to have the problems the older ones did. If anyone is running a good Class 10 and does experience card failure I have yet to hear about it. Personally I want to see if the SD card really is still an issue with the PI or not.

I have 3 other R-PI's running 24/7 in the house, two with security cameras collecting stills and video. All are running class 10 cards and I have not had any problems so far. The 2 running security cameras have much higher SD card write rates than the one running Weewx since they are collecting continuous video data and then deleting anything older than a preset number of days. I do back them all up to a NAS though just in case...

William

[Thomas Keffer]

+1. Wise words.

I'm reminded of Donald Knuth's words, "Premature optimization is the root of all evil."

Before launching an effort to improve write optimization, it would be worth looking for some data to see if it's really a problem. 

-tk

--

You received this message because you are subscribed to the Google Groups "Weewx user's group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

[vds]

William - there are a lot of class-10 SD card manufacturers out there.  It would be a big help if you'd share the precise model(s) you've had so much luck with.  Perhaps something for the wiki ?

[Branislav Vedel]

I think this is a bit off-topic as it's more related to Raspberry Pi as such rather than weewx. Plus there are numerous dedicated forums where this has been discussed. I think there is even a wiki directly on the RP site.

[Steve]

Back on the 1st page, about the 5th post William wrote about some of his experiences with SD cards. "Samsung Pro Class 10 SDHC"

[Thomas Keffer]

I just discovered this talk on hacking SD cards, given at the Chaos Computer Club. It's absolutely fascinating:

http://www.bunniestudios.com/blog/?p=3554

Here are their slides. I'm wowed by the amount of work these two put into this project!

http://bunniefoo.com/bunnie/sdcard-30c3-pub.pdf

It's easy to see why quality varies so tremendously across SD cards.

-tk

[William Phelps]

Tom,

Wow! Great find, and fascinating reading. Especially the part about using an SD card for an MITM attack. The mind boggles at the potential hacking that could go on using this... "hey, can I get a copy of that document? here's my SD card..."

Of course spinning disk drives have embedded mcu's too, and the firmware is often field upgradeable.

William