Re: [pywws] Troubling with Raspberry Pi and WH 1080: no data stored

[Jim Easterbrook]

On 20/01/13 10:39, Alessandro Grechi wrote:
> Hello,
> I'm trying to setup pywws on my Raspberry Pi with PCE-FWS 20 (which is a
> WH-1080).
> I follow guides and tutorials, but I get stuck on storing data from
> LogData in /weather/data folder.
>
> Link with the station seems good:
>
> pi@raspberrypi ~/weather/code $ sudo python TestWeatherStation.py
>
> 0000 55 aa 00 01 24 c0 00 4d 01 00 41 25 03 07 00 06 01 20 02 20
> 09 00 00 00 01 00 00 01 00 00 00 01

Your station's 'logging interval' is set to one minute. pywws assumes
values larger than this, typically 5 minutes. I don't know how well
pywws handles one minute logging.

> So, then LogData:
>
> pi@raspberrypi ~/weather/code $ python RunModule.py LogData -vvv
> /home/pi/weather/data
>
> 11:30:13:pywws.WeatherStation.CUSBDrive:using pywws.device_cython_hidapi
>
> 11:30:14:pywws.LogData:Computer and weather station clocks disagree
> by 0:10:00 (H:M:S).
>
> 11:30:14:pywws.LogData:Synchronising to weather station
>
> 11:30:14:pywws.weather_station:delay 0, pause 0.5
>
[snip]
>
> 11:32:00:pywws.weather_station:delay 0, pause 0.5

It looks as if your station isn't moving on to a new memory location.
pywws is waiting for a pointer change which never happens.

> And this is my weather.ini:
>
> [fixed]
>
> ws type = 1080
>
> station clock = None
>
> sensor clock = None
>
> pressure offset = 63.7
>
> fixed block = {'data_changed': 0, 'unknown_18': 0, 'timezone': 1,
> 'data_count': 1, 'min': {'hum_out': {'date': '2010-01-01 12:00',
> 'val': 79}, 'wind$

'data_count' is 1 - your station has no historical data to read. It's
definitely not incrementing the pointer.

Try resetting the base station - disconnect USB and pull out the
batteries, wait 10 or 20 seconds then put the batteries back in. Wait
for readings from the outside sensors to be displayed before operating
the touch screen or connecting to a computer.
--
Jim Easterbrook <http://www.jim-easterbrook.me.uk/>

[Alessandro Grechi]

I've done the reset.

How can I change the logging interval?

I've found and tried this command:

python SetWeatherStation.py -r 5

Then tried LogData again, with same (no) results:

pi@raspberrypi ~/weather/code $ python RunModule.py LogData -vvv /home/pi/weather/data

12:15:16:pywws.WeatherStation.CUSBDrive:using pywws.device_cython_hidapi

12:15:18:pywws.LogData:Computer and weather station clocks disagree by 1115 days, 0:15:00 (H:M:S).

12:15:18:pywws.LogData:Re-read fixed block

12:15:19:pywws.LogData:Pressure offset change: 63.7 -> 64

12:15:19:pywws.LogData:Synchronising to weather station

12:15:19:pywws.weather_station:delay 0, pause 0.5

12:15:20:pywws.weather_station:delay 0, pause 0.5

12:15:20:pywws.weather_station:delay 0, pause 0.5

Thank you,

Alessandro 

[Jim Easterbrook]

On 20/01/13 11:21, Alessandro Grechi wrote:
> I've done the reset.
> How can I change the logging interval?
> I've found and tried this command:
>
> python SetWeatherStation.py -r 5
>
>
> Then tried LogData again, with same (no) results:
>
> pi@raspberrypi ~/weather/code $ python RunModule.py LogData -vvv
> /home/pi/weather/data
>
> 12:15:16:pywws.WeatherStation.CUSBDrive:using pywws.device_cython_hidapi
>
> 12:15:18:pywws.LogData:Computer and weather station clocks disagree
> by 1115 days, 0:15:00 (H:M:S).
>
> 12:15:18:pywws.LogData:Re-read fixed block
>
> 12:15:19:pywws.LogData:Pressure offset change: 63.7 -> 64
>
> 12:15:19:pywws.LogData:Synchronising to weather station
>
> 12:15:19:pywws.weather_station:delay 0, pause 0.5
>
> 12:15:20:pywws.weather_station:delay 0, pause 0.5
>
> 12:15:20:pywws.weather_station:delay 0, pause 0.5

You need to let it run for 5 minutes or more. If the 'delay' value
doesn't change after a minute or two then the station is still not
incrementing.

After leaving the station running for 30 minutes or so (no computer
connected) you should be able to see some logged data on the station
console (see instruction book). If there is no logged data then your
station is broken.

[Alessandro Grechi]

Yes, I've arrived at your same conclusion.

Using EasyWeather software I can read realtime values from the station, but the records are empty.

Moreover, in the setup config I can see that the logger interval of the weather station is set to 1 min, but when I try to change it the saving fails (nothing is send).

Running the station without computer link doesn't make the station memory full up. So I must conclude it's somehow corrupted...

I'll contact the assistance to get it repaired.

Thank you anyway for your support, you have helped me to figure out which was the problem.

Greetings,

A.G.

[Alessandro Grechi]

Hello Jim and everyone,

I want to share with you my (happy ending) story with the WH1080 and Raspberry Pi.

After solving the above problems power cycling batteries and using EasyWeather to reset memory and settings, I get back running.

But the USB connection freeze issue still came up randomly. After testing every piece of software around here, I finally SOLVED definitively all the problems with swpi.

This is a piece of great opensource project managed by Tonino Tarsi of http://www.vololiberomontecucco.it, based also on pywws code, I think.

In the python code which reads live data from the station he has inserted a function which reads the time from weather station and updates the computer's one according to it. This prevents the software to read data from memory while station is writing in it. 

No more usb hangs so far... :)

I suggest you to look inside the code in https://github.com/ToninoTarsi/swpi ad add similar hack to pywws also! ;)

Ps: look the function "SetTimeFromWeatherStation" in https://github.com/ToninoTarsi/swpi/blob/master/sensor_wh1080.py

greetigs,

A.G.

[Jim Easterbrook]

On 16/02/13 15:25, Alessandro Grechi wrote:
>
> After solving the above problems power cycling batteries and using
> EasyWeather to reset memory and settings, I get back running.
> But the USB connection freeze issue still came up randomly. After
> testing every piece of software around here, I finally SOLVED
> definitively all the problems with swpi.
> This is a piece of great opensource project managed by Tonino Tarsi
> of http://www.vololiberomontecucco.it, based also on pywws code, I think.
> In the python code which reads live data from the station he has
> inserted a function which reads the time from weather station and
> updates the computer's one according to it. This prevents the software
> to read data from memory while station is writing in it.

As introduced to pywws last November:
http://code.google.com/p/pywws/source/detail?r=560
Note that swpi is based on a recent version of pywws that has those changes.

> No more usb hangs so far... :)
>
> I suggest you to look inside the code
> in https://github.com/ToninoTarsi/swpi ad add similar hack to pywws also! ;)
>
> Ps: look the function "SetTimeFromWeatherStation"
> in https://github.com/ToninoTarsi/swpi/blob/master/sensor_wh1080.py

This function waits for the 'fixed block' datetime to change. This has
several disadvantages: 1/ Not all stations have datetime stored in their
fixed block. 2/ Those that do may go for two or three minutes without
updating it. 3/ This gets the 'station clock' (60 second period) but
misses the 'sensor clock' (48 second period).

Setting the computer time to the weather station is a mistake if your
computer has a network connection. NTP does a far better job.

Lastly, I note that Tonino Tarsi makes no acknowledgement of pywws on
his GitHub site, despite his code being obviously based on my work, and
including a significant part of it. That stinks, and is a clear breach
of the GPL.

[Alessandro Grechi]

As introduced to pywws last November:
http://code.google.com/p/pywws/source/detail?r=560
Note that swpi is based on a recent version of pywws that has those changes.

Yes I've noted that. But unfortunately that doesn't clue the issue with my wh1080.

This function waits for the 'fixed block' datetime to change. This has
several disadvantages: 1/ Not all stations have datetime stored in their
fixed block. 2/ Those that do may go for two or three minutes without
updating it. 3/ This gets the 'station clock' (60 second period) but
misses the 'sensor clock' (48 second period).

Setting the computer time to the weather station is a mistake if your
computer has a network connection. NTP does a far better job.

I understand your point of view, I just reported a solution that resolved completely my problem (and I think could solve the others', too), after I had tested several which don't. 

In my case this clock sync works like a charm and does not corrupt the memory. Station clock of 60 second sounds fine to me, since I need to update services or send ftp data only once every 2-5 minutes.

Also, I see that the weather station clock (with DFT sync enabled) differs from NTP time only by few seconds. It seems acceptable.

Over all this code just works fine, and it's the only thing that counts to me :)

Lastly, I note that Tonino Tarsi makes no acknowledgement of pywws on
his GitHub site, despite his code being obviously based on my work, and
including a significant part of it. That stinks, and is a clear breach
of the GPL.

I can assure you that the developer is a kind person and I think could understand your regrets and agree with you if you contact him. I suppose it's just a misunderstanding.

Greetings,

A.G.

 

[Jim Easterbrook]

On 16/02/13 18:30, Alessandro Grechi wrote:
>
> I understand your point of view, I just reported a solution that
> resolved completely my problem (and I think could solve the others',
> too), after I had tested several which don't.

Have you tried a recent version of pywws since your weather station
started working again? If you remember, your station wasn't logging any
data at all last month.

> Over all this code just works fine, and it's the only thing that counts
> to me :)

It works because pywws works. It is not substantially different to
pywws. Setting your computer's clock to the station clock is irrelevant.

[Alessandro Grechi]

Have you tried a recent version of pywws since your weather station
started working again? If you remember, your station wasn't logging any
data at all last month.

Yes, I have tried the most recent version of pywws just some days after my last message here. Either livelog or hourly worked fine for few hours before "bad magic block" error happening again.

And I see this problem is still happening to a lot of people with a recent wh1080... 

It works because pywws works. It is not substantially different to
pywws. Setting your computer's clock to the station clock is irrelevant.
--

Probably you're are right, I don't know exactly what is relevant. Maybe the clock is not. I can tell you only that with swpi code I run smoothly since days without issues, with pywws I can't achieve that.

I would like to figure out what the difference is, in order to help other people in my situation... 

Maybe a difference might be that swpi only reads live data, not "logged" data at all. And don't clear (write) station memory, too. This perhaps could prevent station memory corruption better than other.

Greetings,

A.G.

[Jim Easterbrook]

On 17/02/13 01:18, Alessandro Grechi wrote:
>
> Have you tried a recent version of pywws since your weather station
> started working again? If you remember, your station wasn't logging any
> data at all last month.
>
>
> Yes, I have tried the most recent version of pywws just some days after
> my last message here. Either livelog or hourly worked fine for few hours
> before "bad magic block" error happening again.

"Unrecognised 'magic number'" is not the USB hangup problem I thought
was the issue, and it doesn't stop pywws working. It means your station
is trashing the first two bytes of its memory - usually a sign that it's
not logging data properly.

> And I see this problem is still happening to a lot of people with a
> recent wh1080...

Which problem?

> Maybe a difference might be that swpi only reads live data, not "logged"
> data at all. And don't clear (write) station memory, too. This perhaps
> could prevent station memory corruption better than other.

pywws does not write to station memory in normal operation. swpi writes
to it, to set the logging interval to one minute.

Not reading logged data could be a solution if you have a broken station
that does not log any data.

[Alessandro Grechi]

"Unrecognised 'magic number'" is not the USB hangup problem I thought
was the issue, and it doesn't stop pywws working. It means your station
is trashing the first two bytes of its memory - usually a sign that it's
not logging data properly.

I said that because it was the only error appeared in log before communication stops and thinking was related.

 And with this I mean no data were read anymore, even if restarting daemons or trying TestWeatherStation.py (no response / stuck).

Pywws can no longer communicate with the console and I have to cycle the power before I can connect to it again.

 

> And I see this problem is still happening to a lot of people with a
> recent wh1080...

Which problem?

If you see past messages in this group (or in other, also) there are several users with a recent wh1080 who have the same strange behavior.

Software stop reading after a while, no matter what, and they need console reset. And there was no clue so far.

 

pywws does not write to station memory in normal operation. swpi writes
to it, to set the logging interval to one minute.

Not reading logged data could be a solution if you have a broken station
that does not log any data.

My station is not broken and now correctly log data to his memory. 

[Jim Easterbrook]

On 17/02/13 10:05, Alessandro Grechi wrote:
> "Unrecognised 'magic number'" is not the USB hangup problem I thought
> was the issue, and it doesn't stop pywws working. It means your station
> is trashing the first two bytes of its memory - usually a sign that
> it's
> not logging data properly.
>
>
> I said that because it was the only error appeared in log before
> communication stops and thinking was related.

It's interesting that this happens just before the USB lockup. I haven't
heard of this happening to anyone else. It certainly hasn't happened to
me when I've had USB lockups.

> And with this I mean no data were read anymore, even if restarting
> daemons or trying TestWeatherStation.py (no response / stuck).
> Pywws can no longer communicate with the console and I have to cycle the
> power before I can connect to it again.
>
> > And I see this problem is still happening to a lot of people with a
> > recent wh1080...
>
> Which problem?
>
>
> If you see past messages in this group (or in other, also) there are
> several users with a recent wh1080 who have the same strange behavior.
> Software stop reading after a while, no matter what, and they need
> console reset. And there was no clue so far.

Every case I know of has been effectively cured by using the new 'avoid
reading at certain times' algorithm implemented in pywws and Cumulus.
There is still a small chance of lockups when the software resyncs every
24 hours, but they are greatly reduced. I haven't had one in 4 months of
live logging.

The algorithm does rely on the computer clock being reasonably stable. I
wonder if this is why it isn't working for you?

I've been looking a bit more at swpi. It doesn't use pywws's live_data
algorithm, but simply reads the current data every 30 seconds. This does
nothing to avoid the 48 second outside sensor period (as far as I can
tell) so I am surprised this prevents USB lockups. The low rate of
reading should make them less frequent though.

> My station is not broken and now correctly log data to his memory.

I'm glad to hear that. Does TestWeatherStation.py now show the correct
'magic number' in the first two bytes of memory?

[Alessandro Grechi]

Ok, I'll make some new test with pywws in the next days and I'll write you the results and logs.

Thank you for your interest.

A.G.

[Jim Easterbrook]

On 17/02/13 10:27, Jim Easterbrook wrote:
> On 17/02/13 10:05, Alessandro Grechi wrote:
>> "Unrecognised 'magic number'" is not the USB hangup problem I thought
>> was the issue, and it doesn't stop pywws working. It means your
>> station
>> is trashing the first two bytes of its memory - usually a sign that
>> it's
>> not logging data properly.
>>
>>
>> I said that because it was the only error appeared in log before
>> communication stops and thinking was related.
>
> It's interesting that this happens just before the USB lockup. I haven't
> heard of this happening to anyone else. It certainly hasn't happened to
> me when I've had USB lockups.

I've been thinking about this today and I think I've worked out what's
happening. My guess is that the order of events is this:

1/ Station stops logging data. (We don't know why this happens. I
suspect a fault.) A symptom of this might be the corrupted 'magic number'.

2/ pywws attempts to sync to the logging time by waiting for a pointer
change, reading from USB every half second, continuously. Sooner or
later, pywws reads at the critical time and triggers the USB lockup.

I could change pywws so that if there is no pointer change after a
minute or two it stops waiting, and stops logging data. This shouldn't
affect normal operation, it just adds a bit of complexity.

[Alessandro Grechi]

 I've been thinking about this today and I think I've worked out what's 

happening. My guess is that the order of events is this:

1/ Station stops logging data. (We don't know why this happens. I
suspect a fault.) A symptom of this might be the corrupted 'magic number'.

2/ pywws attempts to sync to the logging time by waiting for a pointer
change, reading from USB every half second, continuously. Sooner or
later, pywws reads at the critical time and triggers the USB lockup.

I could change pywws so that if there is no pointer change after a
minute or two it stops waiting, and stops logging data. This shouldn't
affect normal operation, it just adds a bit of complexity.

Yes, maybe the problem could be something like that, I guess.

Software I'm using now, in fact, doesn't read logged data (and so does not wait for it) but only live data. 

If you make pywws working as you said, what happens after it stops waiting for logging data? It hangs out or live data are still read?

[Jim Easterbrook]

I expect it'll be difficult to do anything other than print out a
message and exit the program. It is assumed throughout the pywws
software that there is logged data to process, and I really don't see
much point in rewriting it to cope with a few broken weather stations.
If my station stopped logging data I'd want to know immediately, rather
than risk running a week without storing any data because I didn't spot
a message in a log file.