How to revive 'frozen' WeeWX?

[Ton vanN]

Since WeeWX4.4 available have been using it without much problems running on a Raspberry3B with OS=Raspian_Buster. Memory max. expanded through Raspi-Config.
Setup of WeeWX to (re)start when (re)booting.
Occasionally stopping, but either by restart through PuttySSH, or (as last resort) by an 'emergency' hard reset, was always able to restart.
Most used CLI through PuttySSH is sudo /etc/init.d/weewx restart

It still happily reports: [ ok ] Restarting weewx (via systemctl): weewx.service

However, this weekend such simple measures did not help, and database and many files stuck/frozen at 14-10-23 10:15:00, while last FTP-upload at 14-1023 22:17:36, but related files all 0kb. Date&Time of that FTP-upload also valid for all external uploads coming from WeeWX.

Fresh reinstall&setup (using copies of latest sdb-file etc.) obviously a pragmatic remedy,
but would shorter route be available to revive this existing configuration?

Observed 2nd effect: cannot access crontab, getting report that no space left on device.
But df shows plenty of space

[p q]

What does the log file say?

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/89069d87-3803-429c-ba6c-3cc15a574f96n%40googlegroups.com.

[vince]

Lets see the output of "df -h" because it sure sounds like /var or /tmp is full...

[Greg from Oz]

What does df -i show?

You could have run out of inodes.

[Ton vanN]

Question of Vince triggers other observation.

Subject Raspberry should have a RAM-fs, but when viewing the folders, don’t find /var/tmp and it’s subfolders.

Sign for corruption?

Op maandag 16 oktober 2023 om 00:58:30 UTC+2 schreef Greg from Oz:

[Ton vanN]

Have run 3 variants of df with following results.

raspberrypi9:~ $ df
Bestandssysteem 1K-blokken Gebruikt Beschikbaar Geb% Aangekoppeld op
/dev/root         30358348 30341964           0 100% /
devtmpfs            439400        0      439400   0% /dev
tmpfs               472680        0      472680   0% /dev/shm
tmpfs               472680    47740      424940  11% /run
tmpfs                 5120        4        5116   1% /run/lock
tmpfs               472680        0      472680   0% /sys/fs/cgroup
/dev/mmcblk0p1      258095    49395      208700  20% /boot
tmpfs                94536        0       94536   0% /run/user/1000
raspberrypi9:~ $ df -i
Bestandssysteem I-nodes  IGebr   IVrij IGeb% Aangekoppeld op
/dev/root       1899328 116283 1783045    7% /
devtmpfs         109850    379  109471    1% /dev
tmpfs            118170      1  118169    1% /dev/shm
tmpfs            118170    523  117647    1% /run
tmpfs            118170      3  118167    1% /run/lock
tmpfs            118170     15  118155    1% /sys/fs/cgroup
/dev/mmcblk0p1        0      0       0     - /boot
tmpfs            118170     13  118157    1% /run/user/1000
raspberrypi9:~ $ df -h
Bestandssysteem Grootte Gebruikt Besch Geb% Aangekoppeld op
/dev/root           29G      29G     0 100% /
devtmpfs           430M        0  430M   0% /dev
tmpfs              462M        0  462M   0% /dev/shm
tmpfs              462M      47M  415M  11% /run
tmpfs              5,0M     4,0K  5,0M   1% /run/lock
tmpfs              462M        0  462M   0% /sys/fs/cgroup
/dev/mmcblk0p1     253M      49M  204M  20% /boot
tmpfs               93M        0   93M   0% /run/user/1000
raspberrypi9:~ $

Op maandag 16 oktober 2023 om 09:05:07 UTC+2 schreef Ton vanN:

[Ton vanN]

@ p q

Which logfile meant? At what location in the WeeWX-folder&filetree?

Op maandag 16 oktober 2023 om 13:48:00 UTC+2 schreef Ton vanN:

[Ton vanN]

Correction related to RAM-filing system:

folder /var/tmp/ is present, but completely empty, where expecting at least a tmpfs .......

Checking on /etc/fstab miss the additions as described at https://domoticproject.com/extending-life-raspberry-pi-sd-card/#Mounting_Filesystems_in_RAM_Disk

Seems to imply that no RAM-filing system has been activated for this Raspberry.

Additional info:

file /home/weewx/archive/weewx.sdb has size of 41.956 KB

Op maandag 16 oktober 2023 om 09:05:07 UTC+2 schreef Ton vanN:

Question of Vince triggers other observation.

[vince]

Your / partition is full.     You have 29G Size and 29G Used and 0 Available.

raspberrypi9:~ $ df -h
Bestandssysteem Grootte Gebruikt Besch Geb% Aangekoppeld op
/dev/root           29G      29G     0 100% /
devtmpfs           430M        0  430M   0% /dev
tmpfs              462M        0  462M   0% /dev/shm
tmpfs              462M      47M  415M  11% /run
tmpfs              5,0M     4,0K  5,0M   1% /run/lock
tmpfs              462M        0  462M   0% /sys/fs/cgroup
/dev/mmcblk0p1     253M      49M  204M  20% /boot
tmpfs               93M        0   93M   0% /run/user/1000

Check your /var/log partition to see what its size is:

sudo du -sm /var/log

If it is not many GB, check your /home/pi and /tmp and /var/tmp directories the same way.

[Ton vanN]

Vince,

The results of the suggested checks.

raspberrypi9:~ $ sudo du -sm /var/log

25977   /var/log
raspberrypi9:~ $ sudo du -sm /tmp
1       /tmp
raspberrypi9:~ $ sudo du -sm /home/pi
53      /home/pi
raspberrypi9:~ $ sudo du -sm /var/tmp
1       /var/tmp

IMHO contents of checked files are minimal ....

Wondering why /dev/root full, with this Raspberry only running WeeWX plus 2 auxiliary Python-scripts (sized 2KB and 1KB), serving periodic upload of weewx.sdb to a remote, backup server

Guessing/speculation:
might expansion of the file system (via raspi-config) have had some effects?

Op maandag 16 oktober 2023 om 20:58:56 UTC+2 schreef vince:

[Tom Keffer]

The "-m" option to du means that sizes are in megabytes. Your /var/log directory has 26 gigabytes in it.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/2311e7cb-34e4-40f2-a659-5463b414dffcn%40googlegroups.com.

[Ton vanN]

Was on wrong leg, assuming KBs .....
Unpleasant surprise, which seems to lead to conclusion that this install cannot be saved, but new start required.
What is best practical approach to make new, small configuration while 'borrowing' from the old configuration the sdb-file and various conf-files?

Somewhere a description for that kind of clean-up/restart?

Obviously keen to avoid the 'error' that lead to this 'saturated' install:
any hint?

Op dinsdag 17 oktober 2023 om 13:06:53 UTC+2 schreef Tom Keffer:

[vince]

Just delete the big logs from /var/log and reboot - that's all you need to do.

With you providing no data on which files in /var/log are the big ones, no we can't help you much.

[Ton vanN]

Vince,

As advised, deleted all .1-files and .gz-files to make space an for clean-up
Back in operation, but still pensive on the daemon.log which has a size of 22.649875 KB
Is that 'healthy', or should it be reduced in some way as precaution against new 'saturation'?

Op dinsdag 17 oktober 2023 om 22:15:54 UTC+2 schreef vince:

[Ton vanN]

After removal of .1-files and .gz-files for checking reran df, showing

Bestandssysteem 1K-blokken Gebruikt Beschikbaar Geb% Aangekoppeld op

/dev/root         30358348 28786676      287176 100% /

devtmpfs            439400        0      439400   0% /dev
tmpfs               472680        0      472680   0% /dev/shm

tmpfs               472680    47796      424884  11% /run

tmpfs                 5120        4        5116   1% /run/lock
tmpfs               472680        0      472680   0% /sys/fs/cgroup
/dev/mmcblk0p1      258095    49395      208700  20% /boot

tmpfs                94536        4       94532   1% /run/user/1000

/dev/root at 100% occupancy does not seem healthy, and breeding new problems  .......

Op dinsdag 17 oktober 2023 om 22:15:54 UTC+2 schreef vince:

Just delete the big logs from /var/log and reboot - that's all you need to do.

[Dale Chatham]

Suggestion to determine where al the space is being used.

Go to the mount point of the filesystem n question, in this case, /

Run the command du -s *  | sort -n
The directories/files with the largest usage will show up at the bottom of the list.

If a directory, cd into that directory and run du -s * | sort again.

Eventually you will find it.

Note that a file open for writing but not yet closed may not appear.  A reboot may either make it visible or get rid of it.

A good start if it is in /var/log is to blow away all the gzipped log files.  Those have been archived and compressed by logwatch and should not affect anything.  Per my above comment, deleting an open file will merely delete the directory entry, not the file.  If you want to remove the space used by those, try "> filename".

[Rainer Lang]

you can install (unless already installed) the ncdu tool:
sudo apt-get install ncdu
and you'll have to run it in the root directory in a console window on your RPi
it will give you a nice and complete overview which directories use which storage capacity down to the files

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/80e35d3b-a3a2-4a42-ac38-19899dcf59f9n%40googlegroups.com.

[vince]

I know we have a language issue here, but I think you might need a little Linux training too.

We have already established that your /var/log directory is filling up.  That is very unusual.  You need to determine what is causing the big logs.

Please run the following and provide the output:

ls -al /var/log

[cric...@pobox.com]

This might be a bit more comprehensive:

> sudo find / -xdev -size +1000000 -ls

This will report any "large" files.  If I'm reading the manpage correctly, the number is the count of 512 byte

blocks by default, so you can fiddle with the number and/or tack on a scale character.  See "man find" and see

the options for the -size test.

The sudo prevents permissions warnings.

Chris

[Ton vanN]

Vince,

Since more than 10 years retired from professional day-to-day Linux applications,
indeed some of my knowledge of Linux has faded, if not vaporated .....

As requested, ran the suggested CLI with results shown below

raspberrypi9:~ $ ls -al /var/log
totaal 25394612
drwxr-xr-x  7 root root        4096 okt 19 00:00 .
drwxr-xr-x 11 root root        4096 okt 14 11:52 ..
-rwxrwxr-x  1 root root           0 jul 24 00:00 alternatives.log
drwxr-xr-x  2 root root        4096 jul 24 00:00 apt
-rwxrwxr-x  1 root adm       283792 okt 19 14:06 auth.log
-rwxrwxr-x  1 root adm     89212455 okt 19 00:00 auth.log.1
-rwxrwxr-x  1 root root      170555 okt 14 11:52 boot.log
-rwxrwxr-x  1 root root           0 sep 26  2019 bootstrap.log
-rw-rw----  1 root utmp           0 okt  1 00:00 btmp
drwxr-xr-x  2 root root        4096 okt 19 00:00 cups
-rwxrwxr-x  1 root adm    113695224 okt 19 14:06 daemon.log
-rwxrwxr-x  1 root adm  23122752257 okt 19 00:00 daemon.log.1
-rw-r-----  1 root adm      2756977 okt 19 14:05 debug
-rw-r-----  1 root adm   1048983611 okt 18 23:55 debug.1
-rwxrwxr-x  1 root root           0 jul 24 00:00 dpkg.log
-rw-r--r--  1 root root        2784 feb 12  2023 faillog
-rwxrwxr-x  1 root root        2847 dec  4  2022 fontconfig.log
drwxr-xr-x  3 root root        4096 dec  4  2022 hp
-rwxrwxr-x  1 root adm          818 okt 19 13:00 kern.log
-rwxrwxr-x  1 root adm     13796027 okt 18 20:31 kern.log.1
-rw-rw-r--  1 root utmp      292292 okt 19 14:04 lastlog
drwx--x--x  2 root root        4096 okt 14 11:52 lightdm
-rw-r-----  1 root adm       265264 okt 19 14:05 messages
-rw-r-----  1 root adm    119656811 okt 19 00:00 messages.1
drwx------  2 root root        4096 sep 26  2019 private
-rw-r-----  1 root adm    116739711 okt 19 14:06 syslog
-rw-r-----  1 root adm    196420717 okt 19 00:00 syslog.1
-rw-r-----  1 root adm       228016 okt 18 00:00 syslog.2.gz
-rwxrwxr-x  1 root adm      3022087 okt 19 14:05 user.log
-rwxrwxr-x  1 root adm   1175652544 okt 18 23:55 user.log.1
-rwxrwxr-x  1 root adm            0 feb 16  2019 vsftpd.log
-rwxrwxr-x  1 root adm          304 jul 25 17:34 vsftpd.log.2
-rwxrwxr-x  1 root adm          304 jul  5 17:06 vsftpd.log.3
-rwxrwxr-x  1 root adm          304 jun 20 13:59 vsftpd.log.4
-rw-rw-r--  1 root utmp      162048 okt 19 14:04 wtmp
-rw-r--r--  1 root root        8703 okt 14 11:52 Xorg.0.log
-rw-r--r--  1 root root        8775 feb 17  2019 Xorg.0.log.old

Op woensdag 18 oktober 2023 om 23:49:32 UTC+2 schreef cric...@pobox.com:

[vince]

Do a "sudo rm daemon.log.1 debug.1 kern.log.1 user.log.1" to get immediate relief.

Then look at the daemon.log, debug, kern.log, and user.log files that remain to see what is in them.  This is very unusual. You have something in your os that is logging something you need to investigate.    Even if you do a "tail -n 20" of each file it will help you determine what is happening.

On Thursday, October 19, 2023 at 5:15:18 AM UTC-7 Ton vanN wrote:

raspberrypi9:~ $ ls -al /var/log

-rwxrwxr-x  1 root adm     89212455 okt 19 00:00 auth.log.1

-rwxrwxr-x  1 root adm    113695224 okt 19 14:06 daemon.log
-rwxrwxr-x  1 root adm  23122752257 okt 19 00:00 daemon.log.1
-rw-r-----  1 root adm      2756977 okt 19 14:05 debug
-rw-r-----  1 root adm   1048983611 okt 18 23:55 debug.1

-rwxrwxr-x  1 root adm     13796027 okt 18 20:31 kern.log.1

-rw-r-----  1 root adm    119656811 okt 19 00:00 messages.1

-rw-r-----  1 root adm    196420717 okt 19 00:00 syslog.1

[Ton vanN]

Is surprisingly effective remedy:
all files now significantly smaller and WeeWX running smoothly.
But 'digging' to find root-cause for this unpleasant 'growing'.

Op donderdag 19 oktober 2023 om 19:08:55 UTC+2 schreef vince:

[Ton vanN]

No root-cause found and therefore continued with the 'minimized' /var/log with fingers-crossed.

After almost 2 years must revisit this topic, because again the subject raspberry on strike. 

Via PuttySSH tried the same remedy as hinted then, but to no effect.
Changed path to /var/log/

CLI:
sudo rm daemon.log.1

Translated reporting is:
rm: cannot remove ''daemon.log.1' Filing system is read-only

As remedy tried to change the attributes of files by issue of 
sudo chmod 775 *.*
at several levels, but resulting report is same:
Filing system is read-only

Weird, because with WinScp and 
ls -l 
the attributes for all log-files show as rwxrwsr-x

Any ide what has happened, and how to remedy?

Op woensdag 1 november 2023 om 12:10:26 UTC+1 schreef Ton vanN:

[Ton vanN]

Correction/addition, see below extract from PuttySSH-readout on files:

-rwxrwxr-x 1 root adm   805154629 mrt  6 11:47 daemon.log

-rwxrwxr-x 1 root adm  1251904421 mrt  2 00:02 daemon.log.1
-rwxrwxr-x 1 root adm    34392792 feb 23 00:02 daemon.log.2.gz
-rwxrwxr-x 1 root adm    34421901 feb 16 00:02 daemon.log.3.gz
-rwxrwxr-x 1 root adm    34334370 feb  9 00:02 daemon.log.4.gz
-rw-r----- 1 root adm     5318820 mrt  6 11:47 debug
-rw-r----- 1 root adm     8165348 mrt  2 00:01 debug.1
-rw-r----- 1 root adm      286230 feb 23 00:01 debug.2.gz
-rw-r----- 1 root adm      149939 feb 16 00:01 debug.3.gz
-rw-r----- 1 root adm      254542 feb  9 00:01 debug.4.gz
-rw-r--r-- 1 root root          0 mrt  1 00:00 dpkg.log
-rw-r--r-- 1 root root       1850 feb 21 14:04 dpkg.log.1
-rw-r--r-- 1 root root        322 feb 13 22:56 dpkg.log.2.gz
-rw-r--r-- 1 root root        937 nov  7 14:46 dpkg.log.3.gz
-rw-r--r-- 1 root root       7777 jul 27  2024 dpkg.log.4.gz
-rwxrwxr-x 1 root root       1665 jul 27  2024 dpkg.log.5.gz

-rw-r--r-- 1 root root       2784 feb 12  2023 faillog
-rwxrwxr-x 1 root root       2847 dec  4  2022 fontconfig.log
drwxr-xr-x 3 root root       4096 dec  4  2022 hp

-rwxrwxr-x 1 root adm     1954376 mrt  6 11:47 kern.log
-rwxrwxr-x 1 root adm       32165 feb 22 09:36 kern.log.1
-rwxrwxr-x 1 root adm        8440 jan 30 13:02 kern.log.2.gz
-rwxrwxr-x 1 root adm        1178 jan 25 23:23 kern.log.3.gz
-rwxrwxr-x 1 root adm        1237 jan 18 21:54 kern.log.4.gz
-rw-rw-r-- 1 root utmp     292292 mrt  6 11:46 lastlog
drwx--x--x 2 root root       4096 mrt  6 11:46 lightdm
-rw-r----- 1 root adm     3292608 mrt  6 11:47 messages
-rw-r----- 1 root adm     3131508 mrt  2 00:00 messages.1
-rw-r----- 1 root adm      158279 feb 23 00:00 messages.2.gz
-rw-r----- 1 root adm       82455 feb 16 00:00 messages.3.gz
-rw-r----- 1 root adm      137821 feb  9 00:00 messages.4.gz

drwx------ 2 root root       4096 sep 26  2019 private

-rw-r----- 1 root adm    93744258 mrt  6 11:47 syslog
-rw-r----- 1 root adm   182939786 mrt  6 00:01 syslog.1
-rw-r----- 1 root adm     5800444 mrt  5 00:00 syslog.2.gz
-rw-r----- 1 root adm     5804241 mrt  4 00:00 syslog.3.gz
-rw-r----- 1 root adm     5805256 mrt  3 00:00 syslog.4.gz
-rw-r----- 1 root adm     5802534 mrt  2 00:00 syslog.5.gz
-rw-r----- 1 root adm     5817293 mrt  1 00:00 syslog.6.gz
-rw-r----- 1 root adm     5820100 feb 28 00:00 syslog.7.gz
-rwxrwxr-x 1 root adm    13973209 mrt  6 11:47 user.log
-rwxrwxr-x 1 root adm    21557861 mrt  2 00:01 user.log.1
-rwxrwxr-x 1 root adm      832412 feb 23 00:01 user.log.2.gz
-rwxrwxr-x 1 root adm      444514 feb 16 00:01 user.log.3.gz
-rwxrwxr-x 1 root adm      791921 feb  9 00:01 user.log.4.gz

In addition, since the events of 2023 the configuration of WeeWX stayed unchanged 4.8.0.

and the O.S.-status is equally unchanged (although updated&upgraded as far as possible):
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

Op woensdag 9 juli 2025 om 20:12:42 UTC+2 schreef Ton vanN:

[Ton vanN]

Reran CLI with shown result:
sudo du -sm /var/log
2420    /var/log
Much smaller than in 2023 ......

Op woensdag 9 juli 2025 om 20:30:43 UTC+2 schreef Ton vanN:

[Ton vanN]

Also weird:
CLI =>   /var/log $ sudo crontab -e
Report:
/tmp/crontab.spPpVc: Bestandssysteem is alleen-lezen
Creation of temporary crontab file failed - aborting

Op woensdag 9 juli 2025 om 20:37:18 UTC+2 schreef Ton vanN:

[Ton vanN]

Considering age of the installation perhaps appropriate & easier to make a fresh install on another Raspberry with fresh SD-card, with recent version of Raspian and with v5.1 for WeeWX, and make setup equivalent to old one for the readers and for the outputs.
Pity for loss of data, or exists an easy path to merge the old/latest sdb in the new WeeWX-install?

Op woensdag 9 juli 2025 om 20:41:16 UTC+2 schreef Ton vanN:

[vince]

If you can do so, imaging a new SD card and starting over might be your simplest path forward. Save a copy of your current weewx.sdb and later use it to catch up your new installation.

[Anton vanNwnhzn@GMail]

Digging deeper, (blankly) observe that reading Weatherflow_Tempest and output towards CWOP, HWA, Weathercloud and WUnderground still running without problems.
Reason to keep alive, and study what is still (re)usable.

Op 10-7-2025 om 19:50 schreef vince:

--

You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/47a4ef1e-1ac9-49b4-aad4-6dd767c858fen%40googlegroups.com.

-- 

===============================================================
Contactinfo voor Anton van Nieuwenhuijzen:
Email    = ton...@gmail.com   
===============================================================
Deze E-mail en eventuele aanhangende files zijn 
alleen bestemd voor de geadresseerde(n). 
Als je deze E-mail ten onrechte hebt ontvangen, 
dan aub verwijderen en de afzender informeren.

[Ton vanN]

Main problem seems corruption of the O.S. resulting that it has become read-only for all files, not so much corruption of WeeWX.
Example.
input at PuttySSH of CLI = sudo /home/weewx/bin/weewxd /home/weewx/weewx.conf

yields a 'normal' reaction until writing is required towards the publichtml-folder, causing an O.S.error

Unfortunately no simple remedy to correct the O.S. 
=> solution = setup of fresh, second weewx at other Raspberry to get new, operational skins etc.

Op donderdag 10 juli 2025 om 21:24:37 UTC+2 schreef Anton vanNwnhzn@GMail:

[Peter Fletcher]

A 'dying' SD will frequently report that all the files on it are readonly and will not allow new data to be written to it. Arguably, this is a feature, rather than a bug, since it gives you a chance to copy wanted data off it before it completely fails. 

[Anton vanNwnhzn@GMail]

Thanks for the hint:
it might explain why the data-passthrough to WUnderground etc. still OK, because using this Raspberry with RAM-filingsystem as workarea, not the SD-card.
Indeed, then urgent need to save/backup all valuable data from this SD-card [if access still possible]

Op 15-7-2025 om 21:40 schreef 'Peter Fletcher' via weewx-user:

To view this discussion visit https://groups.google.com/d/msgid/weewx-user/553843bb-a04c-42a4-84cb-2457dc258b75n%40googlegroups.com.