Vanage Pro 2 crashes WeeWx daily

[Timothy Bryan]

I have a Vantage Pro 2 and have been  running weewx on an Rpi.  However it will run ok till around midnight (as early as about 10PM and as late as about 1 or 2AM) it will go offline.  I am also uploading to WU.  I know just enough cli to get me in trouble, probably even less. Can someone please interpret this and offer me some advise.  I have been watching this user group but have not seen this particular problem. I have included here an excerpt from the syslog of the latest incident-I think. Mahalo.

[vince]

How is your VP2 connected to the pi ?    Are you using some non-Davis logger or the like ?

[Timothy Bryan]

It's connected via USB to the Davis logger

[Tom Keffer]

We have seen this a few times. Things to try:

1. Try unplugging the logger, then plugging it back in.

2. If that doesn't work, try purging "modemmanager":

sudo apt purge modemmanager

3. If that doesn't work, try clearing the logger memory. 

If that doesn't work, let us know, but this time include more of the log, and please do not use a screenshot. See the wiki article Help! Posting to weewx user.

-tk

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/3499e270-2f85-4661-9e39-01a284855fc6n%40googlegroups.com.

[Timothy Bryan]

Thank you, Will do. I'll get back to you...

[Timothy Bryan]

I tried all the above with no success.  When I tried purging modemmanager, I got a message saying it was not installed and so not purged.  After that weewx was not able to 'wake' the console.  I then cleared the memory of the logger by disconnecting the power from the console (AC and battery) and continued to get the error messages.  I think you should be able to see those messages in the log file I included here...if I did everything right.  Thank you for your patience.

Tim

[vince]

What model pi ?

Do you have other stuff attached to your pi's USB ports ?

I'd try the following:

• run 'lsusb' to see what it sees
  • unplug the logger from the pi
  • run 'dmesg' to see if it detected that
  • then run 'lsusb' and see if the output is any different
• then plug the logger back into the pi
  • and run 'dmesg' again to see if it detected the plugging back in
  • and run 'lsusb' to see if it sees the logger again

[Tom Keffer]

In addition to Vince's suggestions, check and see if there is a competing process trying to access the serial port. Try:

sudo lsof /dev/ttyUSB0

Finally, if none of that works, make sure you have connectivity to the logger in the Vantage. See the wiki article Establishing connectivity.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/e911691d-e324-4f3b-82d0-f56a81a1d91fn%40googlegroups.com.

[Timothy Bryan]

I have a rpi 3 +B with a usb keyboard, a bluetooth mouse and ther logger connected. I ran Vince's procedure and as far as I can interpret the readouts, they showed  logger connected, unconnected then reconnected again.

I then ran 'sudo lsof /dev/ttyusb0'  and got this message:

 
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
      Output information may be incomplete.
COMMAND PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
python3 616 root    4u   CHR  188,0      0t0  256 /dev/ttyUSB0
pi@raspberrypi:~ $ sudo tail -f /var/log/syslog

Dunno what that means.

At about that point weewx had started getting data from the logger again and continues to do so now.  We'll see if it continues to hang in there.

Incidentally, I also tried running 'screen' and 'minicom' utilities according to the 'Establishing connectivity' instructions provided in the Trouble Shooting article and got 'command not found' messages.

Thank YOU GUYS for hanging in there with me,

Tim

[Tom Keffer]

It's quite possible that at some point you had two instances of weewxd running simultaneously. 

In any case, glad to hear that you are up and running.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/6b2aada8-e4e8-48c8-862e-7bbfc0c2cc28n%40googlegroups.com.

[Timothy Bryan]

Update:  She crashed again last night right on time and since then I discovered I did not purge the data-logger as I wrongly reported.  I can now say with confidence after following the manufacturers instructions the data-logger has been purged and weewx is up and running again.   We'll see if this works tomorrow morning when I check.

Mahalos again,

Tim

[vince]

"right on time" ????   What time ?

Do you have some cron job running at a particular time of day that might be messing with the USB bus ?

[Timothy Bryan]

No.  Nothing else is running on the pi except weewx.  It seems the crashes only occur roughly between 10pm and 2am either side of midnight.

On Jan 12, 2022, at 2:53 PM, vince <vince...@gmail.com> wrote:

"right on time" ????   What time ?

Do you have some cron job running at a particular time of day that might be messing with the USB bus ?

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/O45YbcJaKjM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/f31ecdb9-64e1-4c7b-9406-03e9be9703e1n%40googlegroups.com.

[Tom Keffer]

I guess I'm not following. The log excerpt you posted had a crash around 0830. Was this atypical? If so, you should post the log for your "late night" crashes.

You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/8453CE42-41FE-4057-A5B1-7E4508F35647%40gmail.com.

[vince]

On Wednesday, January 12, 2022 at 5:26:22 PM UTC-8 timot...@gmail.com wrote:

 It seems the crashes only occur roughly between 10pm and 2am either side of midnight.

Well when it happens again, we need to see /var/log/messages and /var/log/syslog for the period before+after it fails, and also run "dmesg --ctime" and give us that output for the period around when it fails please.

 

[Timothy Bryan]

It happening again at at around 1am.  I woke and noticed it at 4am.  Though I am creating a log as described in "Help! Posting To Weewx Users," I don't know how to attach "/var/log/messages" log. 

Sorry for the ignorance but I am learning, all-be it slowly,

Tim

[Tom Keffer]

It looks like an instance of weewx was running, but, for some reason, systemd stopped it. Can we see earlier in the log? It may give us a clue as to why.

--

You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/de672f72-3a75-4bf9-adfb-a47eb8b73c92n%40googlegroups.com.

[Timothy Bryan]

Not sure if this is what you need.

[Timothy Bryan]

[vince]

Please run:

     dmesg --ctime

I see it went whacko at 00:44 or so but I sure don't see anything saying why.

[Timothy Bryan]

This is the output from "dmesg --ctime" 

[vince]

Are you plugging stuff in and unplugging it a lot ?

Do you have 'just' the usb keyboard plugged in ?

Is there anything else plugged into the USB slots ?

How are you powering your pi ?

Most issues on a pi are from a slow SD card, or an overburdened USB subsystem or the pi is being powered with a weak power supply.

Do you have a different/better power supply you might try to power the pi ?

[Tom Keffer]

In addition to Vince's questions, here's one more: are you using a serial-to-usb converter? If so, do you have another one you can try? They are notoriously flaky.

You also have a lot of network time synchronizations, which always follow a DHCP router advertisement. I have no idea what that means, but there are a lot more than I would have expected.

--
You received this message because you are subscribed to the Google Groups "weewx-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/27243bda-0d0d-40c5-9c3a-6f0d0a48eac6n%40googlegroups.com.

[Timothy Bryan]

The only things I have plugged in besides the logger is the keyboard.  The mouse is Bluetooth. I have on occasion shared the keyboard with another computer, but not very often.  I thought about the power supply and it was a 2.4 amp cube but to be safe I changed it to a 3 amp cube and that is what’s powering it now.

The SD card is a PNY 64gb, micro XC I, which I believe is a fast enough card but I could be wrong.

I do have a rpi 4 and if it would make since, I could install weewx on that one with the goal of eliminating weewx and the pi as the culprit?  Just a thought and I’ve got the time to do it...

--
You received this message because you are subscribed to a topic in the Google Groups "weewx-user" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/weewx-user/O45YbcJaKjM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/27243bda-0d0d-40c5-9c3a-6f0d0a48eac6n%40googlegroups.com.

[vince]

All I can suggest is that if you can swap in a different pi/SD/power-supply and it goes away, that would point to the pi3 being the problem.  If the problem continues with a different computer, that would point to your datalogger probably.

Your logger will save a week of readings, so there's little risk of losing readings if you swap in a different pi temporarily for a couple days.

You didn't answer Tom's question - is it a Davis USB logger, or a serial logger and serial2usb adaptor ?

[Timothy Bryan]

Sorry Tom, didn't see your question.  The logger is a USB logger not a serial with a USB adaptor.

I'll try the pi4 and see if we can maybe eliminate the pi(s) as being the culprits.  I am also trying to locate another logger.

Don't know what to say about the network time synchronizations.

Thanks again you guys..

Tim

[vince]

Change one thing at a time please.

[John Kline]

I don’t the time syncs are an issue.  I believe the time syncs after DHCP adverts are because of the following file:

/etc/dhcp/dhclient-exit-hooks.d/timesyncd

On each DHCP renewal, this script is called which causes a restart with the NTP servers passed by DHCP (which probably haven’t changed).  It’s not the greatest for stable time, but it’s good enough for most.  I’ve ripped this out on my machines.

On Jan 13, 2022, at 1:49 PM, vince <vince...@gmail.com> wrote:

Change one thing at a time please.

--

You received this message because you are subscribed to the Google Groups "weewx-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to weewx-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/614627b7-692c-4db3-af74-1973267627ean%40googlegroups.com.

[Tom Keffer]

I should have known you'd have the answer, John!

To view this discussion on the web visit https://groups.google.com/d/msgid/weewx-user/64A79A3F-00D7-43B4-B2F8-6A615E95D401%40johnkline.com.

[vince]

On Thursday, January 13, 2022 at 3:57:17 PM UTC-8 jo...@johnkline.com wrote:

It’s not the greatest for stable time, but it’s good enough for most.  I’ve ripped this out on my machines.

Me too.  Just install ntp on all your systems which just plain works.   You don't even need to disable timesyncd, systemd uses ntp preferentially if it's installed and enabled.

sudo apt-get install ntp

But I disable timesyncd too just in case :-) 

[Timothy Bryan]

I installed ntp. I also disabled timesyncd.

I also switched out the computer to a pi4, using the same sd card.

The system came back and worked fine until 12 midnite, then, wait for it...it crashed.

Next I plan to change the card and reinstall weewx.  More and more I am suspecting the logger as the problem.

[Greg Troxel]

Timothy Bryan <timot...@gmail.com> writes:

> I installed ntp. I also disabled timesyncd.
>
> I also switched out the computer to a pi4, using the same sd card.
>
> The system came back and worked fine until 12 midnite, then, wait for
> it...it crashed.
>
> Next I plan to change the card and reinstall weewx. More and more I am
> suspecting the logger as the problem.

If you can run without systemd, that might be worth trying. It seems
like something is happening that shouldn't in a cron kind of way. I
have only been dimly following, but some of your logs show tons of
starting and stopping activity at midnight.

[Timothy Bryan]

Hi Greg(and all), just read up on systemd though don't really understand it. I will study more and try to figure out how to run without it.  In the meantime, I installed weewx on a different card and it ran ok for awhile but then crashed and issued its last report at 20:00.  I have included here the syslog from 1800 to 2100. Notice the crash time is outside the range of times the crash has occurred in the past.

Mahalos, Tim

[Timothy Bryan]

Addendum to last message.  This just happened when I Stopped weewx then started it again.

[vince]

If you changed out the pi, sd card, and pi power supply then you've probably proven your logger is ill.

[Greg Troxel]

Timothy Bryan <timot...@gmail.com> writes:

> Addendum to last message. This just happened when I Stopped weewx then
> started it again.

I see a lot of starting and stopping and it's really hard to separate
what happened before you did that, or if systemd did that by itself.

I am suggesting that you change things so that:

you are really really sure that only one thing is trying to read the console

systemd does not start and stop weewx

nothing does anything about changing USB device attachment

It may well be that your logger is broken. But there's a lot going on
in the logs that makes me suspicious.

[Timothy Bryan]

At this point after installing weewx on different Rpis on several occasions with the same results, like Vince I am fairly convinced the problem is with the logger and for now I'm going wait till I can swap it out and see if it makes a difference.  If I'm lucky my friend will loan me an extra logger he has.  Standby.  I might also explore setting up weewx to take data from my WeatherLink Live device but from what I have read so far that could be an even deeper rabbit hole to dive into with my limited (but growing) knowledge base...At any rate I really appreciate everyone's help. Thank you so much! 

Aloha and Mahalo,

Tim

[Timothy Bryan]

Update: Since my last post, I tried making a wireless connection to my WeatherLinkLive unit but could not get it to work.

I have recently started running weewx again via the console and the (faulty?) data logger.  All works well for a while (about 12 hrs) then it stops and I get a message saying weewx ‘can’t wake the console.’ As stated above, I usually discover the failure upon waking.  I then stop weewx then start it again and again getting  the same ‘can’t wake the console’ error message. However, I have discovered that if I allow the pi and console/ logger to sit (still powered up) for about 12hrs., then stop and start weewx, it will work properly for about 12 hrs until the routine repeats itself, back filling the data from the previous down time.  This pattern has repeated itself for 4 days.

[Timothy Bryan]

Update: PROBLEM IDENTIFIED AND SOLVED!  After following the above routine for several days and re-reading the Davis Vantqge Pro 2 Console manual I  noted it recommends the console not be located outside.  Though my mine is located in my upstairs bedroom, away from the elements, it does get a bit chilly there.  During winter months it can drop down into the upper 50s in my bedroom.  So with that in mind I decided to inclose the console in a foam box with the lamp on to see if I could raise and stabilize the temperature of the console and it worked.  I was able to keep the temp within the box between 72 and 66 degrees and the crashes stopped. This also explains why I could not bring the system back up till later in the day when the temps rose. Don't know if this is particular to my console or if all Davis consoles act this way but I will share this with Davis support.  Yippie!  Add this to the tool kit.  Thanks everyone for you input and suggestions.