goesproc suddenly failing

[Larry Roux]

I tried rebooting but both machines have failed with the following error:

terminate called after throwing an instance of 'std::invalid_argument'
what(): stoi
Aborted

command line: goesproc --config /usr/local/share/goestools/goesproc-goesr.conf --mode packet --subscribe tcp://0.0.0.0:5004

These have been running since, oh august or september without issues. This started today around 7pm East coast time.

Any ideas?

[Larry Roux]

Sorry, this is for GOES 16. both seem to fail as soon as they try to write a file or something.

This is the chain that led up to the error

I almost never get packet drops so I am thinking this is something on the GOES 16 side?

Writing: ./goes16/fd/ch08_enhanced/2019-05-15/GOES16_FD_CH08_enhanced_20190515T193033Z.jpg (took 3.192s)
VC 9: VCDU drop! (lost 371118; prev: 5375676; packet: 5746795)
VC 9: Detected TP_PDU drop (skipped 4722 packet(s) on APID 294; prev: 5668, packet: 10391)
VC 9: Unable to append to S_PDU on APID 294
VC 9: Finished S_PDU for APID 294 (OR_ABI-L2-CMIPF-M6C09_G16_s20191351930335_e20191351940049_c20191351940134.lrit)
VC 22: VCDU drop! (lost 244062; prev: 3212379; packet: 3456442)
VC 22: Detected TP_PDU drop (skipped 9300 packet(s) on APID 708; prev: 14182, packet: 7099)
VC 20: VCDU drop! (lost 472; prev: 482951; packet: 483424)
VC 31: VCDU drop! (lost 154758; prev: 4061575; packet: 4216334)
VC 31: Detected TP_PDU drop (skipped 7068 packet(s) on APID 992; prev: 10717, packet: 1402)
VC 32: VCDU drop! (lost 169287; prev: 3414047; packet: 3583335)
VC 32: Detected TP_PDU drop (skipped 16004 packet(s) on APID 1025; prev: 11911, packet: 11532)
VC 22: Detected TP_PDU drop (skipped 12403 packet(s) on APID 707; prev: 133, packet: 12537)
VC 20: Detected TP_PDU drop (skipped 6750 packet(s) on APID 642; prev: 12497, packet: 2864)

terminate called after throwing an instance of 'std::invalid_argument'
what(): stoi
Aborted

Restarting just gives the error after a short time

pi@goesperm:~/GOES $ goesproc --config /usr/local/share/goestools/goesproc-goesr.conf --mode packet --subscribe tcp://0.0.0.0:5004

[Pieter Noordhuis]

Thanks for the report Larry!

To dial in on the issue, can you run: goespackets --subscribe tcp://127.0.0.1:5004 --record (or whatever the right address is), and send me a minute worth of packets or so? Or whatever the duration is before the issue happens? I'm sure it's not a big deal to fix, but NOAA must have changed something in the stream composition that trips over goesproc.

Cheers,

Pieter

--
You received this message because you are subscribed to the Google Groups "goestools-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to goestools-use...@googlegroups.com.
To post to this group, send email to goestoo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/goestools-users/af36cfdc-5dbe-458b-a043-2be47d3ccfa4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Pieter Noordhuis]

I'm pretty sure the following line trips the exception: https://github.com/pietern/goestools/blob/a806bbf1ea77c586c015a134b6130cc61df90053/src/goesproc/handler_goesr.cc#L59

But it's hard to say why, or what they changed in the channel numbering for this exception to happen.

I took down my receivers so I don't have access to the feed right now, which is why a couple minutes worth of packets would be immensely helpful here.

Cheers,

Pieter

[Larry Roux]

Will try to do that tonight if possible. I am off on vacation tomorrow for 4 days so, of course, it happened at the worst time.
It seemed to do the text file fine but is failing on images is my guess. I have goespackets running now but have to wait for an image to come through.

[Larry Roux]

OK, nothing came out from goespackets. At least not in the folder I ran it from

goespackets --subscribe tcp://0.0.0.0:5004

goespackets --help doesnt show an --out parm so I assume the files would go into the directory I am in?

[Pieter Noordhuis]

You'll have to run it with the --record option.

Thanks,

Pieter

--
You received this message because you are subscribed to the Google Groups "goestools-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to goestools-use...@googlegroups.com.
To post to this group, send email to goestoo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/goestools-users/e374c2d4-1cae-41ab-ad4b-ade38073ba53%40googlegroups.com.

[Larry Roux]

never mind. forgot the --record parm

[Brad Bowers]

Started having the same issues, looked like it started around 3pm est or so.

[Larry Roux]

Here are 2 files that came out before the failure:

https://www.dropbox.com/s/qfzqw2mdsmmfx7a/packets-2019-05-16T03_42_00.raw?dl=0
https://www.dropbox.com/s/2lg91yiy7anj08s/packets-2019-05-16T03_43_00.raw?dl=0

[Larry Roux]

I think the error happened when they made a change at UTC 00:00

[Pieter Noordhuis]

Thanks, Larry. Reproduced the issue. The problem is that the ground station started sending "Channel = N/A" in the ancillary text header of the image file. This is not correct obviously and goestools doesn't know how to deal with it. I'll see if I can fix by extracting the channel number from the file name.

Cheers,

Pieter

--
You received this message because you are subscribed to the Google Groups "goestools-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to goestools-use...@googlegroups.com.
To post to this group, send email to goestoo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/goestools-users/a8fb39fe-9b1e-4344-bd78-36b5045286ea%40googlegroups.com.

[Larry Roux]

Call NOAA and tell them to stop it! he he he

[Larry Roux]

I am away for 4 days but I am sure you will have volunteers to test the fix. lots of goestools fans out there. Great software! Good luck and thanks for all your hard work!

[Pieter Noordhuis]

Thanks again for the packet dump, Larry.

I have a fix at https://github.com/pietern/goestools/commit/96dc5544290e6f2742564a8795b960103a09a513. Didn't try it out with the full stream as I don't have one, but it doesn't crash for the short set of packets that you posted. Please update your goestools installation to see if it fixes the problem.

Thanks,

Pieter

On Wed, May 15, 2019 at 8:59 PM Larry Roux <larr...@gmail.com> wrote:

Call NOAA and tell them to stop it! he he he

--
You received this message because you are subscribed to the Google Groups "goestools-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to goestools-use...@googlegroups.com.
To post to this group, send email to goestoo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/goestools-users/4b8fe960-9a2c-4f44-8ada-3b75e122ca28%40googlegroups.com.

[Larry Roux]

So far so good. Will check it in the morning. Thanks for your quick work!

[Larry Roux]

Sorry, it eventually failed

Writing: ./goes16/fd/ch14_enhanced/2019-05-16/GOES16_FD_CH14_enhanced_20190516T043033Z.jpg (took 3.220s)
terminate called after throwing an instance of 'std::runtime_error'
what(): Assertion `num >= 1 && num <= 16` failed at /home/pi/goestools/src/goesproc/handler_goesr.cc:66
Aborted

[Pieter Noordhuis]

Thanks, Larry. Looks like the mesoscale images now don't have the channel encoded nor the product sub ID. Last resort is to pull the channel identification from the filename.

Cheers,

Pieter

--
You received this message because you are subscribed to the Google Groups "goestools-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to goestools-use...@googlegroups.com.
To post to this group, send email to goestoo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/goestools-users/33fbabe3-46e6-44ee-8dec-dbf8875b9d9d%40googlegroups.com.

[Pieter Noordhuis]

Added another commit that falls back on extracting the channel from the filename. Hopefully this fixes the issue.

Cheers,

Pieter

[Larry Roux]

Installed and running. Will check in a while! <fingers crossed>

[Larry Roux]

Running about an hour now. I think you got it!
Thank you once again

[Timothy Elwell]

Pieter

I haven't been paying too close attention to the exact product
differences between 16 and 17, but the strange thing is this is only
happening on my goes 16 feed. The goesproc ingest from goes 17 hasn't
died from this problem. So, it's definitely the specific image feed.
Pieter, I'm around off and on the next several days and will try to test
as you submit patches. I need to look again, but do both GOES get fed
from the same ground station? Is it possible NOAA updated something at
GOES16 ground station that's forgetting to put the product ID in? From
what I'm noticing on my GOES 16, it's every single image that comes down
is killing the process. However, GOES 17 is running without an problems
on images. Very strange.

Good catch, Larry!

On 5/16/19 7:20 AM, Larry Roux wrote:
> Running about an hour now. I think you got it!
> Thank you once again
>

--
Tim
KG1GEM
Flower Mound, TX
Denton County SkyWarn Spotter

[Brad Bowers]

I'm back up and running also, I'll monitor and let you know. 

So, on the topic of monitoring... I'm currently ingesting all the data from statsd via telegraf/influxdb/grafana... the receiver is running fine. I'm using a separate virtual machine to process the data and dump it to the web. Would there be any stats from goesproc that could be dumped to a similar monitoring setup? Only way I noticed there was an issue was no images being uploaded, then checking the console to find goesproc crapped itself.

[Pieter Noordhuis]

Good point Brad! Could you create an issue on GitHub so we don't forget about it? I imagine we can add statsd counters per handler or something like that.

I reached out to Seth at NOAA last night and he filled me in on what happened, which I'll copy below.

----

Thanks for the email, a few other users have reached out to me as well with the exact same issue. NESDIS just installed PDA release 3.3 on Tuesday at the backup acquisition site. Yesterday, Operations failed over GOES-16's broadcast over to the backup site, thus the problem showed up. I just swapped the broadcast back over and the problem is alleviated due to the fact the main operational PDA here at NSOF has yet to be patched (planned for next week). 

We're looking into the root cause at the moment and trying to determine the plan forward for next week.

----

Cheers,

Pieter 

--
You received this message because you are subscribed to the Google Groups "goestools-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to goestools-use...@googlegroups.com.
To post to this group, send email to goestoo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/goestools-users/be77f874-1a2b-452a-8c5b-057963716cbd%40googlegroups.com.

[tom]

According to a NOAA contact, the header should be back to normal after
their testing of a new product addition. I have not made any changes to
Goestools and I'm back up with GOES16 though I did have to reboot first.

Tom

> --
> You received this message because you are subscribed to the Google
> Groups "goestools-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to goestools-use...@googlegroups.com

> <mailto:goestools-use...@googlegroups.com>.

> To post to this group, send email to goestoo...@googlegroups.com

> <mailto:goestoo...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/goestools-users/be77f874-1a2b-452a-8c5b-057963716cbd%40googlegroups.com
> <https://groups.google.com/d/msgid/goestools-users/be77f874-1a2b-452a-8c5b-057963716cbd%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Brad Bowers]

All set! Thanks! https://github.com/pietern/goestools/issues/49

To unsubscribe from this group and stop receiving emails from it, send an email to goestoo...@googlegroups.com.

[Hernán Freschi]

Can confirm

terminate called after throwing an instance of 'std::invalid_argument'
  what():  stoi

Last image received was 20190515T193033Z

A solar storm is hitting the earth right now. Could it be related?

On Wed, May 15, 2019 at 10:10 PM Larry Roux <larr...@gmail.com> wrote:

Sorry, this is for GOES 16.  both seem to fail as soon as they try to write a file or something. 

This is the chain that led up to the error

I almost never get packet drops so I am thinking this is something on the GOES 16 side?

Writing: ./goes16/fd/ch08_enhanced/2019-05-15/GOES16_FD_CH08_enhanced_20190515T193033Z.jpg (took 3.192s)
VC 9: VCDU drop! (lost 371118; prev: 5375676; packet: 5746795)
VC 9: Detected TP_PDU drop (skipped 4722 packet(s) on APID 294; prev: 5668, packet: 10391)
VC 9: Unable to append to S_PDU on APID 294
VC 9: Finished S_PDU for APID 294 (OR_ABI-L2-CMIPF-M6C09_G16_s20191351930335_e20191351940049_c20191351940134.lrit)
VC 22: VCDU drop! (lost 244062; prev: 3212379; packet: 3456442)
VC 22: Detected TP_PDU drop (skipped 9300 packet(s) on APID 708; prev: 14182, packet: 7099)
VC 20: VCDU drop! (lost 472; prev: 482951; packet: 483424)
VC 31: VCDU drop! (lost 154758; prev: 4061575; packet: 4216334)
VC 31: Detected TP_PDU drop (skipped 7068 packet(s) on APID 992; prev: 10717, packet: 1402)
VC 32: VCDU drop! (lost 169287; prev: 3414047; packet: 3583335)
VC 32: Detected TP_PDU drop (skipped 16004 packet(s) on APID 1025; prev: 11911, packet: 11532)
VC 22: Detected TP_PDU drop (skipped 12403 packet(s) on APID 707; prev: 133, packet: 12537)
VC 20: Detected TP_PDU drop (skipped 6750 packet(s) on APID 642; prev: 12497, packet: 2864)
terminate called after throwing an instance of 'std::invalid_argument'
  what():  stoi
Aborted

Restarting just gives the error after a short time

pi@goesperm:~/GOES $ goesproc --config  /usr/local/share/goestools/goesproc-goesr.conf --mode packet --subscribe tcp://0.0.0.0:5004

terminate called after throwing an instance of 'std::invalid_argument'
  what():  stoi
Aborted

--
You received this message because you are subscribed to the Google Groups "goestools-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to goestools-use...@googlegroups.com.

To post to this group, send email to goestoo...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/goestools-users/af36cfdc-5dbe-458b-a043-2be47d3ccfa4%40googlegroups.com.

[Timothy Elwell]

Pieter,

Is it possible to run two rtl-sdr's with goestools on one machine? I can
see 2 problems:

1) naming the individual receivers such that during a reboot, they
always show up the same (ie, the dongle attached to my goes16 dish is
always goes16 to the computer let's say);

2) making sure the correct instance of goesrecv attaches to the correct
dongle (ie my goes16 config'd instance attaches to the goes16 dongle.

In regards to 1, I remember there's a way in linux to specify for sound
cards the specific order of the cards, such that a certain card is
always card 0 and another card is always card 1. This, of course, makes
it easy to attach a specific instance to a specific card. I don't know
that there's a way this could be done with serial/usb.

For 2, it would be much more trivial so long as we could figure out #1.

Anyway, it's a curiosity for being able to run multiple instances of
this off say a single laptop or desktop. In this such way, you could
have redundancy without having multiple machines or pi's.

Thanks for all the great work on everything!

Tim

[Timothy Elwell]

So as to not confuse different threads with multiple items, i thought
I'd ask this as another thread:

In the EMWIN file naming, is it possible to use the same naming
structures as the other items? I have experimented with it some and was
able to write a new method to mostly handle things, but it gets terribly
confused with the zip files. Basically, I'm trying to shorten all my
names. I'd like do something like HWOFWD-180236Z.txt or
???RAD-180236Z.gif (jpg, png), etc. My method handled non-zipped files
fine, but just got confused on finding the suffix of the file.
Ultimately, it would append lrit to anything that was a zip originally.

Thanks again!

[Tim Davis]

What was the fix for the error. Just started up my goes system. and got the : terminate called after throwing an instance of 'std::invalid_argument'

  what():  stoi

Thanks

Tim

On Wednesday, May 15, 2019 at 9:03:15 PM UTC-4, Larry Roux wrote:

I tried rebooting but both machines have failed with the following error:

terminate called after throwing an instance of 'std::invalid_argument'
  what():  stoi
Aborted

command line: goesproc --config  /usr/local/share/goestools/goesproc-goesr.conf --mode packet --subscribe tcp://0.0.0.0:5004

These have been running since, oh august or september without issues.  This started today around 7pm East coast time.

Any ideas?

[randy bracy]

Im not sure if there is an easier way but i had this error last week and just ended up reinstalling goestools. Its been running non stop ever since. 

--

You received this message because you are subscribed to the Google Groups "goestools-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to goestools-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/goestools-users/c25b7a10-1d97-4d2d-8288-602a15a2cad8%40googlegroups.com.

[Tim Davis]

Hi Randy,

Did you download a new updated version or just reinstall old version?

On Sunday, September 1, 2019 at 12:25:15 PM UTC-4, randy bracy wrote:

Im not sure if there is an easier way but i had this error last week and just ended up reinstalling goestools. Its been running non stop ever since. 

On Sun, Sep 1, 2019, 12:02 PM Tim Davis <kj4...@gmail.com> wrote:

What was the fix for the error. Just started up my goes system. and got the : terminate called after throwing an instance of 'std::invalid_argument'

  what():  stoi

Thanks

Tim

On Wednesday, May 15, 2019 at 9:03:15 PM UTC-4, Larry Roux wrote:

I tried rebooting but both machines have failed with the following error:

terminate called after throwing an instance of 'std::invalid_argument'
  what():  stoi
Aborted

command line: goesproc --config  /usr/local/share/goestools/goesproc-goesr.conf --mode packet --subscribe tcp://0.0.0.0:5004

These have been running since, oh august or september without issues.  This started today around 7pm East coast time.

Any ideas?

--
You received this message because you are subscribed to the Google Groups "goestools-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to goestoo...@googlegroups.com.

[randy bracy]

I first deleted the goestools folder in the home/pi directory and then i followed the tutorial below again. I skipped steps 2, 3 and 4 as i didnt think they were necessary for this problem. 

https://gist.github.com/lxe/c1756ca659c3b78414149a3ea723eae2#file-goes16-rtlsdr-md

To unsubscribe from this group and stop receiving emails from it, send an email to goestools-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/goestools-users/7a80e077-af1e-4e75-a391-48e4bad52fc3%40googlegroups.com.

[Tim Davis]

Randy,

Thanks for the info. I done that edit in the file but it crashed again. I just deleted and reinstalled goestools will wait and see what happens.

Tim

To view this discussion on the web visit https://groups.google.com/d/msgid/goestools-users/7a80e077-af1e-4e75-a391-48e4bad52fc3%40googlegroups.com.

[Tim Davis]

Well looks like reinstall done the trick. Been running 12hrs and no

: terminate called after throwing an instance of 'std::invalid_argument'

  what():  stoi.

Thanks Randy for the tip.

Got this pic from GOES 16 yesterday of Dorian.

[tom]

Is anyone using an SDRplay RSP1a with Goestools?

Tom