Weewx 5.0.2, saving sql-backups on a usb stick fails.

[Pierre-Yves]

Hello,
I am running Weewx on a Rpi4-2GB (user.sdr driver + BME280 and as3935 extensions). That works very well.
I am able to save sql-backups on the default directory (/var/tmp/backups) but I fail to save to a USB stick connected to the Pi USB2 port, with an [errno 13] error Permission denied: '/media/pi/WEEWX-BU'
The same configuration was working properly with Weewx 4.10.2

I read and followed the Github wiki but have still the same error.

Below, the log, the first part of the sql-backup skin.conf and details on the usb disk connected.

Any clue ?

Thanks,

Pierre-Yves

[vince]

Permission non accordée: '/media/pi/WEEWX-BU'

You need to give the user running the weewx process to write to the usb stick.   This has come up many many times regarding usb devices and v5.

The command is likely "sudo usermod -aG dialout weewx" if your weewx is running as user 'weewx'.

[Pierre-Yves]

Thanks Vince for feedback but I remember I already sent this comment. However, I sent it again. Same result...

How can I check I am running weewx as user weewx ?

Pierre-Yves

Mar  3 23:30:31 raspberrypi weewxd[13195]: INFO user.sqlbackup: version is 0.5
Mar  3 23:30:31 raspberrypi weewxd[13195]: DEBUG user.sqlbackup:  starting from 2024-02-25 23:15:31
Mar  3 23:30:31 raspberrypi weewxd[13195]: DEBUG user.sqlbackup:  databases, sqlite 1 named ['weewx.sdb']
Mar  3 23:30:31 raspberrypi weewxd[13195]: ERROR user.sqlbackup: [Errno 13] Permission non accordée: '/media/pi/WEEWX-BU'
Mar  3 23:30:31 raspberrypi weewxd[13195]: DEBUG weewx.manager: Daily summary version is 4.0
Mar  3 23:30:31 raspberrypi weewxd[13195]: /usr/lib/python3/dist-packages/Cheetah/Compiler.py:1686: UserWarning: You supplied an empty string for the source!
Mar  3 23:30:31 raspberrypi weewxd[13195]:   warnings.warn("You supplied an empty string for the source!", )
Mar  3 23:30:31 raspberrypi weewxd[13195]: ERROR weewx.cheetahgenerator: Evaluation of template /etc/weewx/skins/sqlbackup/index.html.tmpl failed with exception '<class 'FileNotFoundError'>'
Mar  3 23:30:31 raspberrypi weewxd[13195]: ERROR weewx.cheetahgenerator: **** Ignoring template /etc/weewx/skins/sqlbackup/index.html.tmpl
Mar  3 23:30:31 raspberrypi weewxd[13195]: ERROR weewx.cheetahgenerator: **** Reason: [Errno 2] Aucun fichier ou dossier de ce type: '/tmp/sqlbackup/syslinks.inc'
Mar  3 23:30:31 raspberrypi weewxd[13195]: ERROR weewx.cheetahgenerator: ****  Traceback (most recent call last):
Mar  3 23:30:31 raspberrypi weewxd[13195]: ERROR weewx.cheetahgenerator: ****    File "/usr/share/weewx/weewx/cheetahgenerator.py", line 334, in generate
Mar  3 23:30:31 raspberrypi weewxd[13195]: ERROR weewx.cheetahgenerator: ****      unicode_string = compiled_template.respond()
Mar  3 23:30:31 raspberrypi weewxd[13195]: ERROR weewx.cheetahgenerator: ****    File "_etc_weewx_skins_sqlbackup_index_html_tmpl.py", line 202, in respond
Mar  3 23:30:31 raspberrypi weewxd[13195]: ERROR weewx.cheetahgenerator: ****    File "/usr/lib/python3/dist-packages/Cheetah/Template.py", line 1685, in _handleCheetahInclude
Mar  3 23:30:31 raspberrypi weewxd[13195]: ERROR weewx.cheetahgenerator: ****      nestedTemplateClass = compiler.compile(source=source,
Mar  3 23:30:31 raspberrypi weewxd[13195]: ERROR weewx.cheetahgenerator: ****    File "/usr/lib/python3/dist-packages/Cheetah/Template.py", line 738, in compile
Mar  3 23:30:31 raspberrypi weewxd[13195]: ERROR weewx.cheetahgenerator: ****      fileHash += str(os.path.getmtime(file))
Mar  3 23:30:31 raspberrypi weewxd[13195]: ERROR weewx.cheetahgenerator: ****    File "/usr/lib/python3.9/genericpath.py", line 55, in getmtime
Mar  3 23:30:31 raspberrypi weewxd[13195]: ERROR weewx.cheetahgenerator: ****      return os.stat(filename).st_mtime
Mar  3 23:30:31 raspberrypi weewxd[13195]: ERROR weewx.cheetahgenerator: ****  FileNotFoundError: [Errno 2] Aucun fichier ou dossier de ce type: '/tmp/sqlbackup/syslinks.inc'
Mar  3 23:30:31 raspberrypi weewxd[13195]: INFO weewx.cheetahgenerator: Generated 0 files for report sqlbackup in 0.06 seconds
Mar  3 23:30:31 raspberrypi weewxd[13195]: INFO weewx.reportengine: Copied 4 files to /var/www/html/weewx/sqlbackup

[Glenn McKechnie]

As the author of sql-backup, I should probably step up here.

I haven't been able to set up a V5 installation yet, so therefore I'm unfamiliar with the changes required but that group change Vince mentioned should be enough - the disk needs rw (read write) for the weewx user, or dialout group.

Did you login/logout to set the group changes that vince mentioned.

https://github.com/weewx/weewx/wiki/Understanding-permissions#how-to-fix-device-permissions

further comments inline below...

On Mon, 4 Mar 2024 at 09:39, Pierre-Yves <pyb...@gmail.com> wrote:

Thanks Vince for feedback but I remember I already sent this comment. However, I sent it again. Same result...

How can I check I am running weewx as user weewx ?

  ps -adelf | grep [Ww]eewxd

The 3rd column will show the user.

Pierre-Yves

Mar  3 23:30:31 raspberrypi weewxd[13195]: INFO user.sqlbackup: version is 0.5
Mar  3 23:30:31 raspberrypi weewxd[13195]: DEBUG user.sqlbackup:  starting from 2024-02-25 23:15:31
Mar  3 23:30:31 raspberrypi weewxd[13195]: DEBUG user.sqlbackup:  databases, sqlite 1 named ['weewx.sdb']
Mar  3 23:30:31 raspberrypi weewxd[13195]: ERROR user.sqlbackup: [Errno 13] Permission non accordée: '/media/pi/WEEWX-BU'

and also,  what's the details for the USB drive

  ls -al '/media/pi/WEEWX-BU'

You could always manually mount that drive as the weewx user under (for eg:) /mnt/sql-backup and add an entry to fstab - with suitable permissions. And reflect that change in the skin.conf file

p.s. I found thumb drives unreliable for backups. They eventually go read-only - the Sans-disk ones I had did that.

 

Mar  3 23:30:31 raspberrypi weewxd[13195]: DEBUG weewx.manager: Daily summary version is 4.0

[...]

 

Mar  3 23:30:31 raspberrypi weewxd[13195]: ERROR weewx.cheetahgenerator: ****      return os.stat(filename).st_mtime
Mar  3 23:30:31 raspberrypi weewxd[13195]: ERROR weewx.cheetahgenerator: ****  FileNotFoundError: [Errno 2] Aucun fichier ou dossier de ce type: '/tmp/sqlbackup/syslinks.inc'

That error should disappear after the first run - it's due to the manual steps in configuration.

Cheers
 Glenn

rorpi - read only raspberry pi & various weewx addons
https://github.com/glennmckechnie

[Pierre-Yves]

Thanks Glenn for feedback.

Below the results:

- Even after a restart of the Pi, I still have the same error.

- ps -adelf | grep [Ww]eewxd returns:

pi@raspberrypi:~ $ ps -adelf | grep [Ww]eewxd
4 S weewx    14866     1  2  80   0 - 26178 -      mars03 ?       00:14:55 python3 /usr/share/weewx/weewxd.py /etc/weewx/weewx.conf

ls -al '/media/pi/WEEWX-BU' returns:

pi@raspberrypi:~ $ ls -al '/media/pi/WEEWX-BU'
total 273976
drwxr-xr-x   5 pi   pi        4096  1 janv.  1970  .
drwxrwxrwx+  3 root root      4096  3 mars  17:59  ..
-rw-r--r--   1 pi   pi        8622 22 déc.  16:46 'as3935(min_strikes).py'
drwxr-xr-x  12 pi   pi        4096  2 mars  14:18  Belchertown
drwxr-xr-x  12 pi   pi        4096  1 mars  18:27  Belchertown-old
-rw-r--r--   1 pi   pi        4156  6 févr. 23:27  bme280wx.py
-rw-r--r--   1 pi   pi        1616  7 févr. 00:00  Commandes-usuelles.txt
-rw-r--r--   1 pi   pi       40148  1 mars  21:50  graphs.conf
-rw-r--r--   1 pi   pi      148331 28 juin   2023 'sdr(last).py'
drwxr-xr-x   5 pi   pi        4096 28 oct.  18:04  Seasons202310281704
-rw-r--r--   1 pi   pi       40047  1 mars  21:28  skin.conf
-rw-r--r--   1 pi   pi       36467  1 mars  09:00  weewx.conf
-rw-r--r--   1 pi   pi   280235008  2 mars  15:26  weewx.sdb.backup

PS: I use also this usb drive for manually saving various files and folders without any problem (skins, database...)

It seems, in view of the table above, that the drive is not properly mounted with regards to the weewx 5.0.2 rules. I tried to change using chown, but I got an "operation denied"...

Cheers,
Pierre-Yves

[Glenn McKechnie]

On Mon, 4 Mar 2024 at 19:07, Pierre-Yves <pyb...@gmail.com> wrote:

[...]

pi@raspberrypi:~ $ ps -adelf | grep [Ww]eewxd
4 S weewx    14866     1  2  80   0 - 26178 -      mars03 ?       00:14:55 python3 /usr/share/weewx/weewxd.py /etc/weewx/weewx.conf

ls -al '/media/pi/WEEWX-BU' returns:

pi@raspberrypi:~ $ ls -al '/media/pi/WEEWX-BU'
total 273976
drwxr-xr-x   5 pi   pi        4096  1 janv.  1970  .
drwxrwxrwx+  3 root root      4096  3 mars  17:59  ..
-rw-r--r--   1 pi   pi        8622 22 déc.  16:46 'as3935(min_strikes).py'
drwxr-xr-x  12 pi   pi        4096  2 mars  14:18  Belchertown
drwxr-xr-x  12 pi   pi        4096  1 mars  18:27  Belchertown-old
-rw-r--r--   1 pi   pi        4156  6 févr. 23:27  bme280wx.py
-rw-r--r--   1 pi   pi        1616  7 févr. 00:00  Commandes-usuelles.txt
-rw-r--r--   1 pi   pi       40148  1 mars  21:50  graphs.conf
-rw-r--r--   1 pi   pi      148331 28 juin   2023 'sdr(last).py'
drwxr-xr-x   5 pi   pi        4096 28 oct.  18:04  Seasons202310281704
-rw-r--r--   1 pi   pi       40047  1 mars  21:28  skin.conf
-rw-r--r--   1 pi   pi       36467  1 mars  09:00  weewx.conf
-rw-r--r--   1 pi   pi   280235008  2 mars  15:26  weewx.sdb.backup

PS: I use also this usb drive for manually saving various files and folders without any problem (skins, database...)

It seems, in view of the table above, that the drive is not properly mounted with regards to the weewx 5.0.2 rules. I tried to change using chown, but I got an "operation denied"...

This has me perplexed.

The /media directory seems to be treated differently by the system. I've tried to change ownership / permissions on a test drive and it's not behaving as I expect. 

I'm able to make those same changes under /mnt so I'm missing something. I need an up-slap to the back of the head, a far stronger cup of coffee or probably just a hefty dose of RTFM. In the meantime, I'll blame systemd :P

If you don't want to relocate that drive (understandable), and you have a spare USB slot then I'd be plugging a fresh drive in under /mnt and mounting it via fstab. You can then manually set the permissions to weewx and they'll stick.

And, if someone knows the answer. I'm all ears

[Pierre-Yves]

Thanks a lot Glenn.

I need to dig a little deeper into this story of connection under mnt and mounting via fstab. My knowledge of these tools is a little limited ;-)

Pierre-Yves

[Glenn McKechnie]

And that research will benefit you, so it's worth doing - always.

But as you've found this issue, and as I need to add some notes to the README for weewx-sqlbackup - that cover this new issue. You get the first draft and if it needs adjusting, tell me.

So; In brief, and as the root user (or sudo it) .

mkdir   /mnt/sqlbackup

chown weewx:weewx   /mnt/sqlbackup (or chmod whatever)

plug your usb drive in, ( and hopefully the system auto mount will get out of our way).

Use dmesg to identify the partition on this backup drive, then use fdisk -l to confirm it (optional). 

This example uses /dev/sd?1 - you will have something more meaningful.

test it by mounting the drive.

mount   /dev/sd?1   /mnt/sqlbackup

ls -al   /mnt/sqlbackup

All okay? Is it what you expect? Then ...

umount   /mnt/sqlbackup.

Now, add the following line to your /etc/fstab  file; perhaps with an appropriate comment

# added a weewx sql backup drive. RTFM with man fstab

/dev/sd?1   /mnt/sqlbackup   ext4   errors=remount-ro   0   0

Save the file and mount it (and everything else - redundantly) with  mount -a

I'm assuming it's formatted as ext4 - adjust as required; and I've specified 0 0 on that fstab entry to remind me that there are no file / boot up checks on that drive.

That's the gist of it. Tweak as you like.

Ah, and change the skin.conf entry to point to it.

Cheers
 Glenn

rorpi - read only raspberry pi & various weewx addons
https://github.com/glennmckechnie

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/0914053a-b235-4f51-b660-ca432d9ba46en%40googlegroups.com.

[Pierre-Yves]

Hi Glenn,

I prepared a USB drive formatted as ext4 and did all the steps you described. That worked except I had to run an additional "sudo chown weewx:weewx /mnt/sqlbackup" after "sudo mount /dev/sdb1 /mnt/sqlbackup" in order to have the right group and user (weewx:weewx).

Finally, I succeded to write a backup on the drive.

A strange behavior however: if I umount the drive and mount it again, I do not see anymore the backup sub-folder in the file manager but, if I run a "ls -al   /mnt/sqlbackup", I can see it on the terminal... File manager indicates no sub-folder in sqlbackup directory

pi@raspberrypi:~ $ ls -al   /mnt/sqlbackup
total 28
drwxr-xr-x 4 weewx weewx  4096  4 mars  14:20 .
drwxr-xr-x 3 root  root   4096  4 mars  13:33 ..
drwxr-xr-x 2 weewx weewx  4096  4 mars  14:20 20240304
drwx------ 2 root  root  16384  4 mars  13:22 lost+found

If I connect the USB drive to another computer, the backup folder including the 2 usual files is present (visible)...

If I stop weewx and restart the Rpi, I recover a normal behavior and the File manager shows all files and folder.

As a conclusion I would say that the problem seems to be fixed, though I found the solution a bit complicated and fastidious. I would never have found myself how to proceed.

Thanks a lot for your help.

Cheers,
Pierre-Yves