want to have /home/weewx/public_html in a ramdisk

[Eric K]

Currently running weewx 4.5.1 in a Raspberry Pi 3B+ with the Bullseye version of Raspberry Pi OS.

For a Raspberry Pi (using a microSD card as the OS system drive) I want to create a ramdisk for the /home/weewx/public_html directory.  

I followed examples from various webpages on the ramdisk topic.  

What I've tried thus far doesn't fully work.

How are others implementing this?

I started by renaming my /home/weewx/public_html directory to /home/weewx/public_html_backup, so there would be no conflict when the ramdisk was created at bootup.  

Then, I put this line in the /etc/fstab file and rebooted.

tmpfs   /home/weewx/public_html   tmpfs   defaults,noatime,size=100M   0 0

I tested it with sudo mount -a and the new partition was visible by using the df command.

After a reboot, a /home/weewx/public_html directory was created as a tmpfs volume.

To test the functionality, I manually ran wee_reports to force the webpage to be created.

When wee_reports ran, the process crashed with errors, because some of the files don't exist from previous runs of wee_reports.

pi@rpi3b:/home/weewx $ sudo bin/wee_reports
Using configuration file /home/weewx/weewx.conf
Generating for all time
Traceback (most recent call last):
  File "/home/weewx/bin/user/belchertown.py", line 1390, in get_extension_list
    with open(forecast_file, "wb+") as file:
FileNotFoundError: [Errno 2] No such file or directory: '/home/weewx/public_html/json/forecast.json'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/weewx/bin/weewx/reportengine.py", line 196, in run
    obj.start()
  File "/home/weewx/bin/weewx/reportengine.py", line 281, in start
    self.run()
  File "/home/weewx/bin/weewx/cheetahgenerator.py", line 152, in run
    ngen = self.generate(gen_dict[section_name], self.gen_ts)
  File "/home/weewx/bin/weewx/cheetahgenerator.py", line 222, in generate
    ngen += self.generate(section[subsection], gen_ts)
  File "/home/weewx/bin/weewx/cheetahgenerator.py", line 222, in generate
    ngen += self.generate(section[subsection], gen_ts)
  File "/home/weewx/bin/weewx/cheetahgenerator.py", line 310, in generate
    searchList = self._getSearchList(encoding, timespan,
  File "/home/weewx/bin/weewx/cheetahgenerator.py", line 387, in _getSearchList
    searchList += obj.get_extension_list(timespan, db_lookup)
  File "/home/weewx/bin/user/belchertown.py", line 1399, in get_extension_list
    raise Warning(
Warning: Error writing forecast info to /home/weewx/public_html/json/forecast.json. Reason: [Errno 2] No such file or directory: '/home/weewx/public_html/json/forecast.json'  

I found that if I copy the contents of my /home/weewx/public_html_backup folder into the /home/weewx/public_html ramdisk, then wee_reports completes and creates the fully populated /home/weewx/public_html directory.

Is everyone else populating the ramdisk with a backup of the puclic_html folder at bootup time?

Thanks,

Eric

[vince]

I would try 'mkdir /home/weewx/public_html/forecast' so the parent directory exists when weewx tries to write it.

I'd also add that doing it your way means your NOAA files will be recreated every time it boots, which could take ages if you have many years of info like many of us do.

One way would be to put something in your rc.local which runs after things mount ala:

    if [ -d /home/weewx/public_html ]
    then
       # prepopulate things here

       mkdir /home/weewx/public_html/forecast

    else

       logger "error - rc.local could not mkdir for weewx"

    fi

You might run into timing issues if you do too much this way since rc.local tends to run toward the end of the startup sequence, but a quick mkdir should work.

[p q]

If you're wanting a ramdisk because SD cards are unreliable, I wouldn't bother. There is plenty of history of people running Weewx on Raspberry Pis for many years without a problem with corrupted SD cards. 

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/71a87cdb-be9b-4cfc-b67b-e72802faceean%40googlegroups.com.

--

Peter Quinn
(415)794-2264

[Eric K]

Hi Peter.

Correct, extending the life of the SD card is my goal.

I am currently rebuilding my weewx Raspberry Pi system because of a suspected SD card failure.

I had 2 local Linux users tell me that the symptoms are consistent with an SD card corruption.

The card is only 9 months old, and it's not bottom-of-the barrel quality!

I reached out for help 2 weeks ago and got zero responses.

see:
https://forums.raspberrypi.com/viewtopic.php?p=1979221&hilit=kernel+panic#p1979221

[Doug Jenkins]

Eric, if you have pi4 with firmware older than 09/2021, you can boot from a usb3 drive. This is what I have been doing with my RPI as I have lost 2 good sdcards due to write failures.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/d8e90467-dfcd-46fc-8a2d-af534c0f1dcdn%40googlegroups.com.

[Eric K]

Its' a Raspberry Pi 3B+ 

[vince]

Personally I wouldn't bother doing ramdisk on a pi3, pi3+ or pi4, and I'd instead just shoot a backup of your /home/weewx tree once (to baseline it) and then occasionally grab your archive and NOAA directories via cron or something to another computer or dropbox or something.

When you get down to it, all that is important is the archive and your weewx.conf file - you can rebuild anything else in just a few minutes if it ever becomes necessary.

[Doug Jenkins]

I am sorry, I didn't read the entire post. 

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/39197852-6c84-4cae-9a37-d74320f4abe7n%40googlegroups.com.

[Eric K]

No problem, Doug. 
Your intention to share helpful information is appreciated!  

[Eric K]

Vince, thanks for your idea of manually creating the /home/weewx/public_html/json directory.

I tried it and it did allow Belchertown to work with the first run of wee_reports.

Later, I found if I ran wee_reports a 2nd time, the Belchertown skin completes the creation of the web page in the ramdisk, because the first run of wee_reports (that fails) creates the /home/weewx/public_html/json directory.  That's good enough to succeed after that.

At present, the public_html directory is running inside a ramdisk.

Since it works, I'm going to leave it like this.

I'm using the imagemagick package to add date/time info to each webcam grab.

If nothing else, my webcam pictures are likely loading and processing faster.

Eric

[Graham Eddy]

if you need a directory to exist (be created if necessary) on boot up (such as a subdirectory inside a ramdisk), i configure it in /etc/tmpfiles.d/.

in my case, running weewx as user ‘weewx’, i have a directory /run/weewx/ (/run/ is a ramdisk) owned by weewx for runtime stuff like pidfiles, so have file /etc/tmpfiles.d/weewx.conf containing

d  /run/weewx  0755 weewx weewx -

⊣ Graham Eddy ⊢

On 13 Mar 2022, at 11:13 am, Eric K <eric.k...@gmail.com> wrote:

Vince, thanks for your idea of manually creating the /home/weewx/public_html/json directory.

I tried it and it did allow Belchertown to work with the first run of wee_reports.

Later, I found if I ran wee_reports a 2nd time, the Belchertown skin completes the creation of the web page in the ramdisk, because the first run of wee_reports (that fails) creates the /home/weewx/public_html/json directory.  That's good enough to succeed after that.

At present, the public_html directory is running inside a ramdisk.

Since it works, I'm going to leave it like this.

I'm using the imagemagick package to add date/time info to each webcam grab.

If nothing else, my webcam pictures are likely loading and processing faster.

Eric

[snip]

[Greg Troxel]

vince <vince...@gmail.com> writes:

> Personally I wouldn't bother doing ramdisk on a pi3, pi3+ or pi4, and I'd
> instead just shoot a backup of your /home/weewx tree once (to baseline it)
> and then occasionally grab your archive and NOAA directories via cron or
> something to another computer or dropbox or something.
>
> When you get down to it, all that is important is the archive and your
> weewx.conf file - you can rebuild anything else in just a few minutes if it
> ever becomes necessary.

While I see your point, I've done something in between, avoiding writes
while not going to the trouble of a sytem image that runs out of RAM
entirely after boot.

My setup is a RPI3 running NetBSD, with weewx installed in /usr/weewx,
on the uSD.

However, I have in /usr/weewx two symlinks

lrwxr-xr-x 1 gdt wheel 16 May 22 2020 public_html -> /tmp/public_html
lrwxr-xr-x 1 gdt wheel 23 May 22 2020 public_metric_html -> /tmp/public_metric_html

which means that while my software and my weewx database is on the uSD,
my generated html pages are in /tmp which is a RAM-type fs. The html
pages are then rsynced to another machine for web access.

I also do backups every week. Perhaps should be more often of the database.

Whether this is extending the life of my uSD card is a good question.
But it was easy and avoids a lot of writes every 5m cycle. Certainly I
do not need to retain the gnerated html and graph files.

[vince]

Agree.   I have a couple pi that I run most things in ramdisk on but I don't care about logs on those pi.

Here's a snippet from /etc/fstab for the box I use to shoot weathercam snaps and timelapse into /mnt/ramdisk, with cron jobs on other systems to pull the results back onto something the weewx system that runs off spinning disk.

#---- run motion in ramdisk ---
# obviously do 'mkdir -p /mnt/ramdisk' before this can work

#
tmpfs /mnt/ramdisk tmpfs size=260M 0 0

#

#---- put logs and tmp in ramdisk ---
tmpfs           /tmp            tmpfs   defaults,nosuid,mode=0755,nodev,noatime   0       0
tmpfs           /var/log        tmpfs   defaults,nosuid,mode=0755,nodev,noatime   0       0
tmpfs           /var/tmp        tmpfs   defaults,nosuid,mode=0755,nodev,noatime   0       0

Here's a similar one for a pi that used to run just nginx to serve up a webpage for my weewx system to grab via a custom extension.  I later switched to just publishing to MQTT but left the pi fstab alone since it was great for uptime.

tmpfs           /tmp            tmpfs   defaults,nosuid,mode=0755,nodev,noatime   0       0
tmpfs           /var/log        tmpfs   defaults,nosuid,mode=0755,nodev,noatime   0       0
tmpfs           /var/log/nginx  tmpfs   defaults,nosuid,mode=0755,nodev,noatime   0       0
tmpfs           /var/tmp        tmpfs   defaults,nosuid,mode=0755,nodev,noatime   0       0
tmpfs           /var/www/html   tmpfs   defaults,nosuid,mode=0755,nodev,noatime   0       0

Note - these are model-B that are particularly sensitive to power issues.  I've never had to do this on any pi that uses the micro SD card from the original zero all the way up to a current pi4.

[Graham Eddy]

as well as doing nightly system backups (rsnapshot) to a separate on-site server, i also specifically save weewx.sdb *often* to on-site backup and nightly to off-site backup:

graham@messmate:~$ sudo -u weewx crontab -l

# local backup production weewx.sdb every 15 mins, rollover one hour

2-59/15 * * * * scp -Cq paperbark:/home/weewx/archive/weewx.sdb /srv/cache/weewx/weewx.sdb_$(date '+\%M')

# offsite backup production weewx.sdb every day - picked up in *its* backups to off-site storage

2 5 * * * scp -3Cq paperbark:/home/weewx/archive/weewx.sdb weather.geddy.id.au:backups/weewx.sdb

⊣ Graham Eddy ⊢

[Rainer Lang]

There are only two ways to be really safe:
1. boot your RPi3+ from a (n external) HDD/SSD - instructions how to make this work can be found in the internet

2. use an industrial standard SLC microSD card - it will hold for at least 10 years.

(and make regular (!) backups of your important weewx files like weewx.sdb and weewx.conf - and your skin files if you have made adaptations and maybe your crontab and fstab files if you made modifications there)

I'm using a 16 GB SanDisk/WD SDSDQED-016G-XI micro SD card. Cost me about 27 EUR/30 USD those days (1 1/2 years ago).

Nowadays prices have gone up and there are still big delays in delivery due to the "chip crisis". (26 weeks right now)

You need to buy them from special stores like mouser.com or arrow.com - you won't get them in normal, consumer stores.

e.g. https://www.mouser.com/ProductDetail/SanDisk/SDSDQED-016G-XI?qs=gZXFycFWdANR2bR%2Fc%252B7h0w%3D%3D

Don't let yourself be fooled by product names like  High Endurance, Max Endurance - they are only good for video stream recordings.
Weewx (and other applications with high database activity) will wear them down fast.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/d8e90467-dfcd-46fc-8a2d-af534c0f1dcdn%40googlegroups.com.

[Tom Keffer]

High-end consumer SD cards are better than you think. As an experiment, I spun up a WeeWX instance using full logging on a Sandisk Extreme Plus SD card, thinking it would last a year or so. That was over seven years ago and it's doing fine.

http://www.threefools.org/weewx/status/index.html

Incidentally, the instance now has over 3 years of uptime, with no growth in memory usage!

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/2219b881-a206-daa0-551d-c6dac8170bf1%40gmail.com.

[Graham Eddy]

the UPS is the key thing. i expect that without a UPS the sd card would have died after a year. as was mentioned earlier, power glitches and outages (and i’ll add pulling the power plug!) are sd card killers

nonetheless on principle i like to put high-activity hotspots in ramdisks where feasible

⊣ Graham Eddy ⊢

[DR]

Thank you for these real-world experiences.  A few more bucks when buying is easily made up in having hours of re-building a system and finding all the personal tweaks one puts in.

What a great group to share their info and experience.  I am appreciative. Dale

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/CAPq0zEBkzELBNJw-sc9a2%3DNPboze7%3D10isOGGBNkQe2y0NCQQg%40mail.gmail.com.