FineOffsetUSB lockup - Possible solution for unattended reset

[Jarom Hatch]

So, I had been dealing with the lockup issue on my WS-2090 for a while ever since before switching to weewx from wview.  As the wiki page states (http://sourceforge.net/p/weewx/wiki/folockup/), the only way for me to get out of it was to power cycle the receiver.  I thought that was pretty lame.

Then I switched to wview and under ver 2.7.0 I found what I consider a little gem.  Let me illustrate:

When the lockup is happening I get the following output in wee_config_fousb --info:

Using configuration file /etc/weewx/weewx.conf

Driver version 1.6

Querying the station...

Fine Offset station settings:

                  local time: 2014.12.16 17:00:33 MST

                polling mode: ADAPTIVE

                  abs_pressure: 14.0

                 current_pos: 0

                data_changed: 8

                  data_count: 2348

                   date_time: 2021-00-00 00:00

               hum_in_offset: 0

              hum_out_offset: 0

                          id: 49152

               lux_wm2_coeff: 68.0

                     magic_1: 0x87

                     magic_2: 0x 0

                       model: 48384

                   rain_coef: 21772

           ***** read_period: None *****

                rel_pressure: 86.0

              temp_in_offset: 8559

             temp_out_offset: 10167

                    timezone: 1

                  unknown_01: 71

                  unknown_18: 33

                     version: 0

                   wind_coef: 65450

                   wind_mult: 320

While locked up the read_period is always "None" and sometimes the data_count is super high.  From what I can gather the data_count is the number of stored records in the station.  The station date also is reporting from the year 2021.  

So I tried something.  I ran wee_config_fousb --clear-memory to see if that brought it back, as I figured clearing the memory is the same as power cycling the thing.  Turns out it did something unexpected.  It fixed the read_period but also kept records in memory, though the number changed to something much more believable:

Using configuration file /etc/weewx/weewx.conf

Driver version 1.6

Querying the station...

Fine Offset station settings:

                   local time: 2014.12.16 22:20:36 MST

                 polling mode: ADAPTIVE

                  abs_pressure: 856.1

                  current_pos: 8384

                 data_changed: 0

                   data_count: 509

                    date_time: 2014-12-16 22:20

                hum_in_offset: 8226

               hum_out_offset: 5650

                           id: None

                lux_wm2_coeff: 0

                      magic_1: 0x55

                      magic_2: 0xaa

                        model: None

                    rain_coef: None

                  read_period: 1

                 rel_pressure: 1016.9

               temp_in_offset: 5120

              temp_out_offset: 0

                     timezone: -7

                   unknown_01: 0

                   unknown_18: 0

                      version: 255

                    wind_coef: None

                    wind_mult: 0

Note:  The times between the two samples are a few hours apart.  This is because I now run this on a cron every 10 minutes and output the latest info to a file.  But you can see that even after a few hours the numbers are good still.  I haven't had to power cycle my station in a few months now while before it was at least once per week.

My script that I cron every 10 minutes (written late at night while in a bad mood):

#!/bin/bash

sleep 30

readPeriod=$(/usr/bin/wee_config_fousb /etc/weewx/weewx.conf --info | tee /root/last_wx_info | grep read_period | tr -d ' ' | cut -d ':' -f 2)

pid=$(cat /var/run/weewx.pid)

resetWX() {

        /etc/init.d/weewx stop

        sleep 10

        if ps -p $pid > /dev/null

        then

                echo "weewx didn't terminate.  Killing PID $pid."

                kill -9 $pid

        fi

        /usr/bin/wee_config_fousb /etc/weewx/weewx.conf --clear-memory -y

        /etc/init.d/weewx start

}

if [ "$readPeriod" = "None" ]

then

        cp /root/last_wx_info /root/last_wx_error

        echo "Resetting WX on $(date)" >> /root/resetWX.log

        resetWX >> /root/resetWX.log

fi

Hope this helps someone out.

[mwall]

jarom,

i suspect that you are experiencing corruption, but not lockup.  when the lockup happens, *nothing* can communicate with the station, so you cannot even tell it to clear its memory.

one thing you could do to make your installation more robust is to change back to the default polling mode of PERIODIC instead of ADAPTIVE.  the ADAPTIVE mode works for pywws because it has a single thread that syncs to the station, then stays synced.  in weewx, ADAPTIVE does not work as designed because the sync to the station gets disrupted by the way the engine and driver interact (i intend to fix this at some point with a complete rewrite of the fousb driver).  as a result, using ADAPTIVE in weewx will cause much, much more usb traffic (and thus greater chance of lockup, and possibly corruption) as the fousb driver tries to re-sync.

this is also why it is safer to use record_generation=software for fine offset stations.  if you use record_generation=hardware (the default), then the driver must re-sync on every archive interval, just to read a single record.

m

(my original reply incorrectly stated that PERIODIC generates much, much more usb traffic)

[Jarom Hatch]

I see.  I guess I've never experienced lockup as described then.  Back when I was running wview it would seemingly lock up but I think it was just the corruption issue as I could communicate, just not get anything.  I found the lockup thing while googling for the symptoms I was seeing so I figured it was the same thing.

I see the corruption regardless of what polling method I use and the cron I set up seems to remedy the problem.  Is a memory reset something that weewx can just do automatically when it encounters corruption instead of having to run a script?

[Jarom Hatch]

What do you give up switching to record_generation=software?  Is the data going to be identical in both cases?

[mwall]

On Wednesday, December 17, 2014 4:15:15 PM UTC-5, Jarom Hatch wrote:

What do you give up switching to record_generation=software?  Is the data going to be identical in both cases?

with record_generation=hardware, the station firmware does the averaging of 'live' sensor readings.  with record_generation=software, weewx does it.

the data should be similar, but not identical, since we have no idea what algorithms the firmware actually uses.

m

[mwall]

On Wednesday, December 17, 2014 11:24:40 AM UTC-5, Jarom Hatch wrote:

I see the corruption regardless of what polling method I use and the cron I set up seems to remedy the problem.  Is a memory reset something that weewx can just do automatically when it encounters corruption instead of having to run a script?

you'll have to run your own script.

m

[mwall]

On Wednesday, December 17, 2014 11:24:40 AM UTC-5, Jarom Hatch wrote:

I see the corruption regardless of what polling method I use and the cron I set up seems to remedy the problem.  Is a memory reset something that weewx can just do automatically when it encounters corruption instead of having to run a script?

let me rephrase my last posting.

if you can determine a consistent indicator of corruption, and a reliable way to eliminate the corruption, then we could incorporate that into the driver.

in my experience, the case where read_period=None is apparently rather rare.  i have experienced it maybe twice in over 3 years of running 4 different fine offset stations.  and jim easterbrook's pywws did not even check for it (he might have added the check in the past year or so).  on the other hand, it sounds like it happens frequently on your station.

it would help if you could run with record_generation=software and polling_mode=PERIODIC, then document how often you see read_period=None.

also track changes to the 'magic numbers' - the fousb driver will log any changes to these two values.

m

[Jarom Hatch]

First, if the read_period=None issue and the lockup issue are indeed different issues, I'd much rather be experiencing the first.  I have never had the station completely lock up on me, but I have had the corruption issue keep me from sending data to WU for a few hours at a time.  Since I've been running that script I've not encountered the problem again.  If it's true that my condition is rare, I'm fine with continuing with the cron to check.  If it were more common it might be nice to check for the condition and fix it before it becomes a problem.

I've switched the polling mode back to periodic (not sure what adaptive would benefit me anyway, it seemed like a better option when I was reading about it).  I'll switch the record generation to software and keep an eye on the data.

[Cancunia]

I'm not sure how many Fine Offset installs are based on Raspberry Pi, but thought that I'd mention a way around the problem. It only works on the B+ model, but a simple reboot does actually power cycle the USB ports. There's also a utility called hub-ctrl on GitHub that can switch off USB ports on the B+ , but it seems to only work on all ports or none.

[Gazza]

It's good to know that the B+ Pi supports PPPP, I'm doing some testing of the B+ Pi and had not got that far as I am waiting on an updated version of Matthew's usb_control.py to continue.

usb_control.py is based on hub-ctrl and the functionality is in the Fine Ofset driver so all we need is a readily available device that supports PPPS and maybe the B+ will do the job.

There is some more info here: https://groups.google.com/forum/#!topic/weewx-user/Hl_bvmYQs4U


Gary

[Matías Laporte]

Hello everybody. I've been googling for a while but I'm not able to find an answer to this, or someone that has this setup and can confirm what I suppose. 

If the Raspberry model B+ supports power-cycling the USBs (although some people it turns down ON/OFF all USBs at the same time, some say that it can be done on a port per port basis -last post here-), then It could be used for connecting a Fine Offset station directly to it (without a powered USB HUB) and powercycle the station (using hub-ctrl.c) to 'solve' the lockups? 

Thank you very much!

[mwall]

On Tuesday, September 29, 2015 at 10:34:02 AM UTC-4, Matías Laporte wrote:

Hello everybody. I've been googling for a while but I'm not able to find an answer to this, or someone that has this setup and can confirm what I suppose. 

If the Raspberry model B+ supports power-cycling the USBs (although some people it turns down ON/OFF all USBs at the same time, some say that it can be done on a port per port basis -last post here-), then It could be used for connecting a Fine Offset station directly to it (without a powered USB HUB) and powercycle the station (using hub-ctrl.c) to 'solve' the lockups? 

Thank you very much!

the first step is to make sure that you can turn power on/off via software.  use hub-ctrl.c or usb_control.py to experiment (choose your poison - c or python).  when you have the minimal code and/or invocation that works, post it.

hopefully talking to the rpi built-in hub is not much different than talking to an external, powered hub.

the fousb.py driver includes code to reset usb power, but it has only been tested on some standalone usb hubs.  it would be better to abstract that function into a class and put it elsewhere in weewx so that it could be used with any weather station, not just fine offset.

m

[Matías Laporte]

Hi Matthew, thank you very much for your answer.

I cannot test it yet because I was deciding what model of the raspberry pi to buy, depending on if it could be used to "solve" or not the problem. Besides I'm in Argentina, and apparently there aren't a lot of (or none) rpi 2 b+ models for sale.

In the link I posted someone using hub-ctrl.c could reset USB 2, 3 and 4 separately, and all the USBs + Ethernet (it uses an USB interface apparently) together , but it was just one user (recently) and no one else has posted there for a while. Also, I don't know what he was using it for, so I wanted to know if maybe someone used it with a fine offset station. e.g., the code for restarting USB 4 was:

sudo ./hub-ctrl -h 0 -P 3 -p 0 ; sleep 3; sudo ./hub-ctrl  -h 0 -P 3 -p 1

I'll see if I can get my hands on a rpi 1/2 b+ and try it. Be certain that if I get it working I'll post here the results.

Thank you. 

[Matías Laporte]

Well, if it is of any use, I couldn't get the station's console working again after (what I suppose is) a lockup with the previous command. But, I got it working with a different solution. I have a simple script added to the cronjob that checks when was the last weewx record added in the MySQL table and, if it was a "long time ago" (half an hour), it just restarts the raspberry pi, which effectively cuts the power from the station's console. After the restart weewx works OK again.

[pon...@gmail.com]

I'm a bit confused. I have a FineOffset WH1080 powered by AA batteries connected via USB to a Raspberry Pi B+ and, in 2 years, have experienced one lock-up of the console which required it to be power cycled by having the batteries removed. We've had occasional power cuts in that time so the RPi has had occasional resets and has never locked up. As far as I remember the symptom of the WH1080 lockup was simply that it stopped feeding data to the RPi (i.e., the LCD display was still updated). Power cycling the RPi did not resolve, nor did removing and reconnecting the USB cable. I tried to avoid resetting the console as it had uncollected data but in the end I had to do so, or so it seemed at the time.

What I'm not clear on is which device is considered to be locked up when these lockup incidents happen. If there's a way to avoid losing data if it happens again I'd like to know.

[Alexandros Bagos]

Hi Matias,

Can you post the script that you added to cron? Will it work with a sqlite db? Additionally, after the power cycle of the console, it's not a problem if you don't set the time/date/timezone on it?

Thanks,

Alex.

[Aldo Bord]

On Tuesdayì 29 settembre 2015 16:46:38 UTC+2, mwall ha scritto:

On Tuesday, September 29, 2015 at 10:34:02 AM UTC-4, Matías Laporte wrote:

Hello everybody. I've been googling for a while but I'm not able to find an answer to this, or someone that has this setup and can confirm what I suppose. 

If the Raspberry model B+ supports power-cycling the USBs (although some people it turns down ON/OFF all USBs at the same time, some say that it can be done on a port per port basis -last post here-), then It could be used for connecting a Fine Offset station directly to it (without a powered USB HUB) and powercycle the station (using hub-ctrl.c) to 'solve' the lockups? 

hopefully talking to the rpi built-in hub is not much different than talking to an external, powered hub.

 

Hi, this would be a pretty useful feature, since USB hub with power switching are hard to find and the Pi3 is also able to switch power on individual ports, as reported in the above link.

Has anyone already added this function to weewx ? I could not find any other update on this matter.

Thanks

Aldo

[Alexandros Bagos]

Hello,

I have tried to setup weewx on a raspberry Pi2 B+ which supports per-port USB power on/off to reset the port of a locked Fine Offset, but it didn't work as expected. From the logs I could see that the port was actually being controlled but after puting the power back on it wouldn't initialize properly. So the station would still not send data, in fact it didn't "beep" as it does when you power it on. It did seem to work maybe once or twice but it was not consistent.

I think the option to reboot the Pi completely is better.

Br,

Alex.

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/TCEYaNEqLOI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Alexandros Bagos

[Aldo Bord]

Il giorno mercoledì 12 aprile 2017 14:52:38 UTC+2, Alexandros Bagos ha scritto:

Hello,

I have tried to setup weewx on a raspberry Pi2 B+ which supports per-port USB power on/off to reset the port of a locked Fine Offset, but it didn't work as expected. From the logs I could see that the port was actually being controlled but after puting the power back on it wouldn't initialize properly. So the station would still not send data, in fact it didn't "beep" as it does when you power it on.

This sounds a bit strange, are you sure the USB power was actually cutted and the station had no batteries ?

Afaik it's not necessary to reboot the pi, but just the weather station.

[Alexandros Bagos]

Yes, the station had no batteries. I was also hoping it would work without having to reboot te pi.

I don't know if this makes any difference, but I have a WH-3080. This type suffers from the USB lockout but has one advantage: if you are quick in removing and puting back the USB cable, the memory is retained. The "lost" data are retrieved after this reset.

It was a little confusing because the first time it did seem to work (I was there when it happened), but every other time it didn't manage to reset the station, although I could see in the logs that it was constantly trying.

Alex.

--

You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/TCEYaNEqLOI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Alexandros Bagos

[Aldo Bord]

Il giorno mercoledì 12 aprile 2017 16:09:40 UTC+2, Alexandros Bagos ha scritto:

Yes, the station had no batteries. I was also hoping it would work without having to reboot te pi.

 

Yes, it should work this way, there is no need to reboot the pi, yo ujust need to power cycle the weather station.  

 

I don't know if this makes any difference, but I have a WH-3080.

I also has a 3080, connected to a USB hub with port switching. The reset procedure works, but I would like to get rid of the USB hub and connect the weather station directly to my new Pi3. 

 

It was a little confusing because the first time it did seem to work (I was there when it happened), but every other time it didn't manage to reset the station, although I could see in the logs that it was constantly trying.

I think something is not working in the power cut procedure. 

How did you implemented it ? Using hub-ctrl ?

As Matthew suggested, it would be great to implement it in the FO driver, or even better as a class... any python programmer around willing to add this function ? 

Aldo

[Alexandros Bagos]

I used Matthew's implementation as it is already in the FO driver. I got the values I needed to use from the output of "lsusb -v".

I didn't try to reset the station with the external application (bug-ctrl), but can try that too.

Alex.

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/TCEYaNEqLOI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Alexandros Bagos

[Alexandros Bagos]

Hello,

The reason this does not work is because the Pi does not actually cut the power from the USB port, it cuts the data signal. You can only cut the power from all USB ports including the Ethernet port. This is the way exactly as I found it elsewhere:

To shut off power on USB ports (this shuts power on ethernet as well):
echo '1-1' |sudo tee /sys/bus/usb/drivers/usb/unbind

To turn power back on
echo '1-1' |sudo tee /sys/bus/usb/drivers/usb/bind

Network functions and USB drivers appear to recover just fine after turning power back on.

Alex.