pywws files being written as user root and group root

[John Preston]

Hi

I am getting errors reading files when running pywws-hourly. pywws-hourly -vvv ~/weather/data. If I use sudo it fails differently.

08:13:28:pywws.logdata:Fetching data

08:13:28:pywws.logdata:1 catchup records

08:13:28:pywws.process:Generating summary data

08:13:28:pywws.calib:Using default calibration

08:13:28:pywws.process:daily: 2020-12-08 09:00:00

08:13:28:pywws.process:monthly: 2020-12-01 09:00:00

08:13:28:pywws.regulartasks:doing task sections ['logged', 'hourly', 'daily', '12 hourly']

08:13:28:pywws.regulartasks:Graphing 7days.png.xml

08:13:28:pywws.storage:flushing

Traceback (most recent call last):

  File "/usr/local/bin/pywws-hourly", line 10, in <module>

    sys.exit(main())

  File "/usr/local/lib/python2.7/dist-packages/pywws/hourly.py", line 97, in main

    return hourly(args[0])

  File "/usr/local/lib/python2.7/dist-packages/pywws/hourly.py", line 68, in hourly

    pywws.regulartasks.RegularTasks(context).do_tasks()

  File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__

    self.gen.throw(type, value, traceback)

  File "/usr/local/lib/python2.7/dist-packages/pywws/storage.py", line 271, in pywws_context

    ctx.flush()

  File "/usr/local/lib/python2.7/dist-packages/pywws/storage.py", line 259, in flush

    self.hourly_data.flush()

  File "/usr/local/lib/python2.7/dist-packages/pywws/filedata.py", line 375, in flush

    self._flush(self._wr_cache)

  File "/usr/local/lib/python2.7/dist-packages/pywws/filedata.py", line 404, in _flush

    with open(cache.path, **kwds) as csvfile:

IOError: [Errno 13] Permission denied: '/home/pi/weather/data/hourly/2020/2020-12/2020-12-09.txt'

pi@raspberrypi:~/weather/data/calib/2020/2020-12 $ cd ~/weather/data/hourly/2020/2020-12

pi@raspberrypi:~/weather/data/hourly/2020/2020-12 $ ls

2020-12-01.txt  2020-12-02.txt  2020-12-03.txt  2020-12-04.txt  2020-12-05.txt  2020-12-06.txt  2020-12-07.txt  2020-12-08.txt  2020-12-09.txt

pi@raspberrypi:~/weather/data/hourly/2020/2020-12 $ ls -alt

total 44

drwxrwxrwx 2 root root    4096 Dec  9 08:00 .

-rw-r--r-- 1 root root     484 Dec  9 08:00 2020-12-09.txt

-rwxrwxrwx 1 pi   weather 1451 Dec  9 08:00 2020-12-08.txt

-rwxrwxrwx 1 pi   weather 1443 Dec  8 07:28 2020-12-07.txt

-rwxrwxrwx 1 pi   weather 1459 Dec  7 15:07 2020-12-06.txt

-rwxrwxrwx 1 pi   weather 1749 Dec  6 14:28 2020-12-05.txt

-rwxrwxrwx 1 pi   weather 1815 Dec  6 14:28 2020-12-04.txt

-rwxrwxrwx 1 pi   weather 1721 Dec  4 07:27 2020-12-03.txt

-rwxrwxrwx 1 pi   weather 1573 Dec  3 12:56 2020-12-02.txt

-rwxrwxrwx 1 pi   weather 1515 Dec  2 09:12 2020-12-01.txt

drwxrwxrwx 4 root root    4096 Dec  1 09:58 ..

I manually changed user and group for files shown uo to yesterday then ran pywws-houly and todays files written as root root. This then fails later in the program.

This also happens with Calib files and raw files.  PI user is part of sudo, weather and other groups. I am running pi headless and executing .bashrc. I seem to have sone inconsistent results from commands when using sudo and not.  

Ive tried lots of things over 3 days so now run out of ideas help!!!

Thanks John

[Jim Easterbrook]

On 09/12/2020 08:39, John Preston wrote:
>
> I am getting errors reading files when running
> pywws-hourly. pywws-hourly -vvv ~/weather/data. If I use sudo it fails
> differently.
>

> pi@raspberrypi:~/weather/data/hourly/2020/2020-12 $ ls -alt
> total 44
> drwxrwxrwx 2 root root    4096 Dec  9 08:00 .
> -rw-r--r-- 1 root root     484 Dec  9 08:00 2020-12-09.txt
> -rwxrwxrwx 1 pi   weather 1451 Dec  9 08:00 2020-12-08.txt
> -rwxrwxrwx 1 pi   weather 1443 Dec  8 07:28 2020-12-07.txt
> -rwxrwxrwx 1 pi   weather 1459 Dec  7 15:07 2020-12-06.txt
> -rwxrwxrwx 1 pi   weather 1749 Dec  6 14:28 2020-12-05.txt
> -rwxrwxrwx 1 pi   weather 1815 Dec  6 14:28 2020-12-04.txt
> -rwxrwxrwx 1 pi   weather 1721 Dec  4 07:27 2020-12-03.txt
> -rwxrwxrwx 1 pi   weather 1573 Dec  3 12:56 2020-12-02.txt
> -rwxrwxrwx 1 pi   weather 1515 Dec  2 09:12 2020-12-01.txt
> drwxrwxrwx 4 root root    4096 Dec  1 09:58 ..
>

> Ive tried lots of things over 3 days so now run out of ideas help!!!

1/ Stop pywws.

2/ Change the ownership of all files and directories to the pi user:

sudo chown -R pi:weather /home/pi/weather/data

3/ Restart pywws, running as pi user.

4/ Never run pywws as root.

--
Jim Easterbrook <http://www.jim-easterbrook.me.uk/>

[John Preston]

Thanks for the quick response

I did as you suggested although not sure i had stopped pywws.

This error was generated

    pywws.regulartasks.RegularTasks(context).do_tasks()

  File "/usr/local/lib/python2.7/dist-packages/pywws/regulartasks.py", line 221, in do_tasks

    self._do_common(now, sections)

  File "/usr/local/lib/python2.7/dist-packages/pywws/regulartasks.py", line 143, in _do_common

    self.do_plot(template)

  File "/usr/local/lib/python2.7/dist-packages/pywws/regulartasks.py", line 232, in do_plot

    input_xml = pywws.plot.GraphFileReader(input_file)

  File "/usr/local/lib/python2.7/dist-packages/pywws/plot.py", line 575, in __init__

    self.doc = GraphNode(xml.dom.minidom.parse(input_file))

  File "/usr/lib/python2.7/xml/dom/minidom.py", line 1918, in parse

    return expatbuilder.parse(file)

  File "/usr/lib/python2.7/xml/dom/expatbuilder.py", line 922, in parse

    fp = open(file, 'rb')

IOError: [Errno 2] No such file or directory: 'home/pi/weather/graph_templates/7days.png.xml'

Here is directory of files and permissions

    -rwxrwxrwx 1 pi weather 3216 Dec  6 15:30 24hrs_full_features.png.xml

-rwxrwxrwx 1 pi weather 2304 Dec  6 15:30 24hrs.png.xml

-rwxrwxrwx 1 pi weather 2614 Dec  6 15:30 24hrs.txt

-rwxrwxrwx 1 pi weather 2498 Dec  6 15:30 28days.png.xml

-rwxrwxrwx 1 pi weather 1959 Dec  6 15:30 6hrs_html_cp.txt

-rwxrwxrwx 1 pi weather 2857 Dec  6 15:30 6hrs.txt

-rwxrwxrwx 1 pi weather 2291 Dec  6 15:30 7days.png.xml

-rwxrwxrwx 1 pi weather 2182 Dec  6 15:30 7days.txt

-rwxrwxrwx 1 pi weather 2393 Dec  6 15:30 allmonths.txt

-rwxrwxrwx 1 pi weather 1748 Dec  6 15:30 dmi_borgervejr_1080.txt

-rwxrwxrwx 1 pi weather 1792 Dec  6 15:30 dmi_borgervejr.txt

-rwxrwxrwx 1 pi weather 2619 Dec  6 15:30 feed_daily.xml

-rwxrwxrwx 1 pi weather 2750 Dec  6 15:30 feed_hourly.xml

-rwxrwxrwx 1 pi weather 1734 Dec  6 15:30 forecast_9am.txt

-rwxrwxrwx 1 pi weather 1577 Dec  6 15:30 forecast_tweet.txt

-rwxrwxrwx 1 pi weather 1840 Dec  6 15:30 forecast_week.txt

-rwxrwxrwx 1 pi weather 2304 Dec  6 15:30 humidex.png.xml

pi@raspberrypi:~/weather/data/graph_templates $ 

here is the weather.ini

paths]

work = /temp/pywws

datastoretype = filedata

templates = home/pi/weather/templates/

graph_templates = home/pi/weather/graph_templates/

modules = home/pi/weather/modules/

[config]

usb activity margin = 3.0

ws type = 1080

pressure offset = -7.1

logdata sync = 1

day end hour = 9, False

rain day threshold = 0.2

frequent writes = False

gnuplot version = 4.2

gnuplot encoding = iso_8859_1

[live]

services = []

plot = []

text = []

[logged]

services = []

text = []

plot = []

[hourly]

services = []

plot = ['7days.png.xml', '24hrs.png.xml', 'rose_24hrs.png.xml']

text = ['tweet.txt', '24hrs.txt', '6hrs.txt', '7days.txt']

[12 hourly]

services = []

text = []

plot = []

[daily]

services = []

text = []

plot = []

[ftp]

directory =

password =

port = 21

site =

user =

Any ideas?

Thanks

John

[Jim Easterbrook]

On 09/12/2020 14:26, John Preston wrote:
>
> I did as you suggested although not sure i had stopped pywws.
>
> This error was generated

> IOError: [Errno 2] No such file or directory:
> 'home/pi/weather/graph_templates/7days.png.xml'

There should be a / at the start of that path.

> here is the weather.ini
> paths]

Missing [

> work = /temp/pywws
> datastoretype = filedata
> templates = home/pi/weather/templates/
> graph_templates = home/pi/weather/graph_templates/
> modules = home/pi/weather/modules/

Missing / at the start of templates, graph_templates, and modules paths.

[John Preston]

Im an idiot, couldnt see for looking. It ran fine this time.

Thanks Jim

[John Preston]

Sorry to bother you again. I spoke too soon. I fialed this morning after I logged in again. SSH access through .bashrc.

Whats happening is pywws-hourly fails when trying to access usb or usb1 with the error below unless I use sudo however if i do use sudo hourly files generated writeen  as root and then they can not be  accessed. Thats why I had the error yesterday.  The pi user is in sudo group.

pywws-testweatherstation

17:15:40:pywws.logger:pywws version 20.1.0, build 1673 (92d8868)

Traceback (most recent call last):

  File "/usr/local/bin/pywws-testweatherstation", line 10, in <module>

    sys.exit(main())

  File "/usr/local/lib/python2.7/dist-packages/pywws/testweatherstation.py", line 117, in main

    ws = pywws.weatherstation.WeatherStation()

  File "/usr/local/lib/python2.7/dist-packages/pywws/weatherstation.py", line 464, in __init__

    self.cusb = CUSBDrive()

  File "/usr/local/lib/python2.7/dist-packages/pywws/weatherstation.py", line 318, in __init__

    self.dev = USBDevice(0x1941, 0x8021)

  File "/usr/local/lib/python2.7/dist-packages/pywws/device_libusb1.py", line 77, in __init__

    self.dev = self.context.openByVendorIDAndProductID(idVendor, idProduct)

  File "/home/pi/.local/lib/python2.7/site-packages/usb1/__init__.py", line 2267, in openByVendorIDAndProductID

    return result.open()

  File "/home/pi/.local/lib/python2.7/site-packages/usb1/__init__.py", line 2034, in open

    mayRaiseUSBError(libusb1.libusb_open(self.device_p, byref(handle)))

  File "/home/pi/.local/lib/python2.7/site-packages/usb1/__init__.py", line 127, in mayRaiseUSBError

    __raiseUSBError(value)

  File "/home/pi/.local/lib/python2.7/site-packages/usb1/__init__.py", line 119, in raiseUSBError

    raise __STATUS_TO_EXCEPTION_DICT.get(value, __USBError)(value)

usb1.USBErrorAccess: LIBUSB_ERROR_ACCESS [-3]

Ideas??

Thanks John

[Jim Easterbrook]

On 10/12/2020 17:22, John Preston wrote:
>
> Whats happening is pywws-hourly fails when trying to access usb or usb1
> with the error below unless I use sudo however if i do use sudo hourly
> files generated writeen  as root and then they can not be  accessed.
> Thats why I had the error yesterday.  The pi user is in sudo group.
>
> pywws-testweatherstation
> 17:15:40:pywws.logger:pywws version 20.1.0, build 1673 (92d8868)

> usb1.USBErrorAccess: LIBUSB_ERROR_ACCESS [-3]
>
> Ideas??

This is saying your pi user does not have permission to access the
weather station's USB port. The cure is not to run as root, but to set
up a 'udev' rule that changes ownership of the USB port when the station
is plugged in. The documentation mentions this.
https://pywws.readthedocs.io/en/latest/guides/getstarted.html#test-the-weather-station-connection

Instead of changing the 'group' of the port in the udev rule you can
change the owner. This saves having to set up a "weatherstation" group.

ACTION!="add|change", GOTO="weatherstation_end"
SUBSYSTEM=="usb", ATTRS{idVendor}=="1941", ATTRS{idProduct}=="8021",
OWNER="pi"
LABEL="weatherstation_end"