WeeWX 5.0.0b17 removes existing ~/weewx-data/scripts directory

[pa...@pauland.net]

pip update to 5.0.0b17 deletes ~/weewx-data/scripts if it exist, then creates new ~/weewx-data/scripts directory to hold setup-daemon.systemd and setup-daemon.mac. On previous V4 setup.py installs and newer V5 pip installs I have always created a scripts subdirectory within the WeeWX directory to hold custom scripts related to WeeWX. The scripts directory and it's subdirectories were always safe when updating to a new version of WeeWx. I bet I'm not the only user that normally creates a custom scripts directory and will be surprised when their by custom scripts disappear. I guess it's resonable to assume WeeWX has controll of all files under ~/weewx-data/ and user is at own peril by adding custom files. I was just surprised that the existing ~/weewx-data/scripts directory was removed without warning.

[Tom Keffer]

We could timestamp the old 'scripts' subdirectory instead of deleting it.

On Wed, Dec 13, 2023 at 4:29 AM pa...@pauland.net <pa...@pauland.net> wrote:

pip update to 5.0.0b17 deletes ~/weewx-data/scripts if it exist, then creates new ~/weewx-data/scripts directory to hold setup-daemon.systemd and setup-daemon.mac. On previous V4 setup.py installs and newer V5 pip installs I have always created a scripts subdirectory within the WeeWX directory to hold custom scripts related to WeeWX. The scripts directory and it's subdirectories were always safe when updating to a new version of WeeWx. I bet I'm not the only user that normally creates a custom scripts directory and will be surprised when their by custom scripts disappear. I guess it's resonable to assume WeeWX has controll of all files under ~/weewx-data/ and user is at own peril by adding custom files. I was just surprised that the existing ~/weewx-data/scripts directory was removed without warning.

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/2fd8ae9a-fc15-4694-95b6-14f1ee220e31n%40googlegroups.com.

[Paul R Anderson]

That would be better than just deleting it. Wondering if  the new 'scripts' subdirectory could be located at ~/weewx-data/util/scripts ?

Paul

[Joel Bion]

Is it possible to have a formally defined place where users can put in their local customizations? 

Advantages:

1) location is standardized for both users and developers. 

2) because of (1), it is a directory that will be known to be preserved across upgrades.

3) because of (1), an upgrade could look for that directory existing and if so, telling the upgrading user “private_extensions” directory non empty. You need to see if anything in it needs changing due to the upgrade” etc

4) I have mine, and I do call it “private_extensions”

Sent from my iPhone

On Dec 13, 2023, at 4:47 AM, Paul R Anderson <pa...@pauland.net> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/CAOAVAefsPGHaYoG1Z0DD5jfVG2tk%2BEaPy-jSXZom4Q%3DZwuT19A%40mail.gmail.com.

[Tom Keffer]

On Wed, Dec 13, 2023 at 4:47 AM Paul R Anderson <pa...@pauland.net> wrote:

That would be better than just deleting it. Wondering if  the new 'scripts' subdirectory could be located at ~/weewx-data/util/scripts ?

That's something I thought of, but then it gets buried amongst some unrelated subdirectories, making them harder to find.

How about this: just upgrade the scripts within the subdirectory 'scripts'? Then they can peacefully coexist with your scripts.

[Paul R Anderson]

That sounds great!

Paul

[Tom Keffer]

Commit da31d16.

[Greg]

I upgraded from 5.0.0.b16 to 17 and got this error:

Dec 14 10:15:55 moonbi weewxd[491310]: ERROR weewx.engine: StdCalibration loop error unsupported operand type(s) for /: 'NoneType' and 'int'

Dec 14 10:15:21 moonbi weewxd.py: restx: MQTT: binding to loop
Dec 14 10:15:21 moonbi weewxd.py: restx: MQTT: topic is jedwood/raw/weather
Dec 14 10:15:21 moonbi weewxd.py: restx: MQTT: desired unit system is METRIC
Dec 14 10:15:21 moonbi weewxd.py: restx: MQTT: data will be uploaded to mqtt://192.168.1.164:1883/
Dec 14 10:15:21 moonbi weewxd[491310]: INFO weewx.engine: 'pyephem' detected, extended almanac data is available
Dec 14 10:15:21 moonbi weewxd[491310]: INFO __main__: Starting up weewx version 5.0.0b17
Dec 14 10:15:21 moonbi weewxd[491310]: INFO weewx.engine: Using binding 'wx_binding' to database 'weewx'
Dec 14 10:15:21 moonbi weewxd[491310]: INFO weewx.manager: Starting backfill of daily summaries
Dec 14 10:15:21 moonbi weewxd[491310]: INFO weewx.manager: Daily summaries up to date
Dec 14 10:15:22 moonbi weewxd[491310]: INFO weewx.drivers.fousb: synchronising to the weather station (quality=0)


Dec 14 10:15:52 moonbi weewxd[491310]: INFO weewx.engine: Starting main packet loop.
Dec 14 10:15:55 moonbi weewxd[491310]: INFO weewx.drivers.fousb: station status {'rain_overflow': 0, 'lost_connection': 0, 'unknown': 0} (0)
Dec 14 10:15:55 moonbi weewxd[491310]: ERROR weewx.engine: StdCalibration loop error unsupported operand type(s) for /: 'NoneType' and 'int'
Dec 14 10:15:55 moonbi weewxd[491310]: INFO weewx.restx: MQTT: Published record 2023-12-14 10:15:56 AEDT (1702509356)
^[^[Dec 14 10:16:56 moonbi weewxd[491310]: ERROR weewx.engine: StdCalibration loop error unsupported operand type(s) for /: 'NoneType' and 'int'
Dec 14 10:16:56 moonbi weewxd[491310]: INFO weewx.restx: MQTT: Published record 2023-12-14 10:16:56 AEDT (1702509416)

I downgraded back to 16 and I didn't get that error.

Python 3.11.2

[Tom Keffer]

The error was always there, it's just that previous versions swallowed it. 

We should probably continue to ignore 'None' type errors. It's difficult for a newbie to write an expression that can handle them.

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/fbbcd4d1-ca19-40fe-9b39-aa418bc0e2c2n%40googlegroups.com.

[Graham Eddy]

first reaction is: ok for folk’s own code but a nightmare for supported code

⊣GE⊢

[Tom Keffer]

Changed in commit 4b969ce.

For TypeError involving None, the error gets logged if debug>1, then the program presses on. Any other kind of error results in program termination.

This applies only to expressions used in StdCalibrate.

-tk

[Greg]

Does a commit mean if I download it again the fix is in there or do I wait until the version number increases?

[Tom Keffer]

Not sure what you're asking, but try this. The current version is always available as the "development" branch of the repository. 

On PyPi, once a release has been submitted, it cannot be changed. So, new releases have the version number bumped.

Is that what you were asking?

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/26ced7a1-b087-42eb-b31f-4b8cca3d5edcn%40googlegroups.com.

[Greg]

I guess I am asking is the commit a change that is in the version 17? But I think what you are saying is all the commits are done and then you release the next version with all the commits in it?

I am not a programmer so I was wondering how it works.

Thanks

[Tom Keffer]

No. Version 5.0.0b17 never changes. Version 5.0.0b18 will have any new changes incorporated into it.

However, the code repository always has the latest code in it. It just hasn't been organized into a formal release.

Does that help?

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/c69b202e-5e92-4d95-a304-7679d652c135n%40googlegroups.com.

[Greg]

OK I will just wait until there is a release change.

Thanks for the explanation.

[Chris Alemany]

Hi all,

Stumped here:  Yesterday at 4:10PM weewx v5.0b15 (pip install on Debian) stopped updating the archive.sdb. It turns out that an hour before that, it stopped updating the forecast.sdb file. Some other reports are still running (purple, Wundergound, divumWX/Weather34) but anything that depends on info from the archive is stuck at 4:10PM. As I mentioned, there are no errors reported. It is simply silent when the 5min archive time comes and it runs through reports with the old data while the loop packets update independantly.  Are there any additional logging options I can turn on that deal with archive generation?

I decided to try updating to b17 just to see if it would kickstart anything but I have the same behaviour.

I’m using a Davis Vantage Pro 2 with serial data logger -> ethernet bridge.

I double checked the Davis side, restarted the bridge with no change.

All Debian 11 packages are up to date.

Any thoughts welcome.

https://alberniweather.ca

Thanks

Chris

Chris Alemany
3854 6th Avenue
Port Alberni, BC
V9Y4M2
On Stolen Traditional Territory of c̓išaaʔath and hupačasatḥ First Nations.
250-731-7930

Web: https://chrisalemany,ca

Social: https://mstdn.chrisalemany.ca/@chris

Other Blogs and Socials:

https://alberniweather.ca

https://mstdn.ca/@alberniweather.ca

https://vibyrail.ca

https://mstdn.ca/@vibyrail

[Vince Skahan]

Sigh.  Usual thoughts.

set debug=1 and provide the logs that result the next time it happens

or if you already had debug=1, provide the logs you have up to when it went bad

We can't read minds.

[Chris Alemany]

I was basically asking if there is anything more than debug=1 that I could enable.

There is nothing in the debug messages that I’ve been able to detect other than silence. The archive should be updated around Dec 16 12:05:21

[Vince Skahan]

My guess is you are far doing too much far too often.  Are you really querying purpleair and also rsync'ing somewhere every TWO seconds ?  That's a bit much.

Disable everything you added to vanilla core weewx (temporarily) and run just unaltered weewx and see if it works.  Then turn your additions/customizations back on one-by-one and see what is breaking things.

[Chris Alemany]

Do you also have a problem with people using wunderground rapid fire?

I’ve been doing this for years.

Thanks for the advice. I’ll figure it out.

--
You received this message because you are subscribed to the Google Groups "weewx-development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-developm...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/4f9af1bf-0702-4751-821d-61a105d38518n%40googlegroups.com.

[Vince Skahan]

Not really, but your rsync and purple air stuff querying and uploading every two seconds  looks very very unusual.  I'd certainly start there.

[Chris Alemany]

I’m noticing this in the log, is this normal. It seems to me it is talking to the Vantage, knows it has records it needs to get, but tries twice and then gives up. Is that “empty record” a problem?

"Dec 16 13:42:16 footiMac weewxd[1741196]: DEBUG weewx.drivers.vantage: Getting archive packets since 2023-12-15 16:10:00 PST (1702685400)

Dec 16 13:42:18 footiMac weewxd[1741196]: DEBUG weewx.drivers.vantage: Successfully woke up Vantage console

Dec 16 13:42:19 footiMac weewxd[1741196]: DEBUG weewx.drivers.vantage: Retrieving 224 page(s); starting index= 0

Dec 16 13:42:19 footiMac weewxd[1741196]: DEBUG weewx.drivers.vantage: Empty record page 0; index 0

Dec 16 13:42:19 footiMac weewxd[1741196]: INFO weewx.engine: Starting main packet loop.

Dec 16 13:42:21 footiMac weewxd[1741196]: DEBUG weewx.drivers.vantage: Successfully woke up Vantage console

Dec 16 13:42:21 footiMac weewxd[1741196]: DEBUG weewx.drivers.vantage: Requesting 200 LOOP packets.

Dec 16 13:42:23 footiMac weewxd[1741196]: DEBUG weewx.drivers.vantage: Successfully woke up Vantage console”

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/c873934b-4206-46db-854e-3ed96f35bdeen%40googlegroups.com.

[John Kline]

Vince, the log is not querying a purple air device every two seconds.  A purple air could not handle that.  It is inserting purple air values from the most reading in every loop packet.

As for rsyncing every two seconds, I do it ALL THE TIME.  Between machines and also to the Google cloud.  Two seconds is an enormous amount of time.  In fact, in a single weewx instance, I’m rsyncing every two seconds to various devices (perhaps four or more places) every two seconds.  I’ve done this on Raspberry Pi 3s, 4s and now 5s.

Because this is my code, I can tell you that the rsync is on a separate thread and typically a short connection timeout.  For longer connection timeouts, if a queue develops (slow CPU, slow network), it throws away queue entries older than n seconds (configurable).  It never builds up a long queue and, if it skips a loop record, it is recorded in the log.

Looking at my public weewx instance, which rsyncs to a Google compute instance every two seconds, I can see how many skips I’ve had so far today.  I use a 1s timeout with a connection timeouts also of 1 s.

   rsync: conn timeouts                               7

   rsync: conn timeouts (banner exchg)   6

   rsync: io timeouts                                    1

I’ve also contributed this rsync code to Gary’s real time gauges.  Again, it rsyncs on every loop packet (which, for vantage, is every two seconds).

John

On Dec 16, 2023, at 1:52 PM, Chris Alemany <chri...@gmail.com> wrote:

I’m noticing this in the log, is this normal. It seems to me it is talking to the Vantage, knows it has records it needs to get, but tries twice and then gives up. Is that “empty record” a problem?

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/68E3BA4F-DB82-4F56-88EF-4C5DE78DADAF%40gmail.com.

[Tom Keffer]

Chris, you have a classic case of corrupted memory in your logger. 

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-development/68E3BA4F-DB82-4F56-88EF-4C5DE78DADAF%40gmail.com.

[Chris Alemany]

Thanks for this Tom.

Thankfully the first option, unplugging/de-batterying the Vantage Console for 2 minutes to reset it worked.  It doesn’t appear that any information was lost.

Thanks all.

Cheers

Chris

[Tom Keffer]

Quick! Buy a lottery ticket! That almost never works.!