How to monitor GQRX crash on RPi3

[Andy Webster]

Hi all,

I have GQRX set up and running on a Raspberry Pi 3 (Raspbian/Pixel) and am trying to set it all up as an automatic weather satellite receiver/decoder.  Some (incomplete) notes here:  http://alloutput.com/amateur-radio/wxsat-reception/

For the most part this is working well with GPredict controlling GQRX recording and some Python scripts resampling the recorded audio, feeding it into Wxtoimg and then publishing the images on WordPress.  However, the intent is to run the system 24/7 (yes, file management will become a problem on the RPi and in WordPress...!) and GQRX is crashing for some reason after something like 12-16 hours of running.  Symptoms are images not being published as expected, the RPi3 is deathly slow when I try to log in via SSH or VNC and then, eventually, the RPi3 appears to return to normal behaviour, I can log in to VNC and I see that GQRX is no longer running (but the GPredict and the scripts are still up and running).

What can I monitor to see why GQRX is crashing?  I'm now running it through an xterm window that is set to -hold so that may yield some clues next time it happens.  Are there any log files I can check after the next crash?

Thanks,

Andy

[Andy Webster]

Oh, the GQRX version is 2.6-RPi3-1 by the way.

[Alexandru Csete]

Hi Andy,

Gqrx does not create any log files but you can pipe its stdout and
stderr to a file if you wish. There is also the linux syslog you can
look at to see if anything got logged there.

My guess is that a connection problem with the SDR hardware happens,
which most certainly will crash or freeze gqrx. You could also monitor
memory usage to see if there might be any leak, though I would expect
that to have longer term effect only after 24+ hours.

If the crash occurs while gqrx is recording, it could be due to disk IO issues.

Alex

> --
> You received this message because you are subscribed to the Google Groups
> "Gqrx SDR" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to gqrx+uns...@googlegroups.com.
> To post to this group, send email to gq...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/gqrx/86247e78-3527-422a-9061-4b02d19876b3%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Andy Webster]

Thanks for the quick response Alex, piping stdout and stderr to a log file now, will see what this turns up...

Happy new year! :-)

[Andy Webster]

Hi Alex,

The stdout and stderr messages show a frequent instance of this message:

portaudio_sink::work(): Error writing to audio device: Output underflowed

Probably more interesting is the kernel log appears to show gqrx being killed when the system runs out of memory, see the end of the attached logfile.  That's not to say gqrx is the problem, it may simply be the victim of my hacky python scripts (I have one script running every 2 minutes to detect new audio recordings and scripts running every 5 minutes to decode the resampled audio and publish to WordPress, the log file also shows some repeated messages at these intervals although not all the time...).

Is there anything in there that helps pinpoint the problem?  I'm now looking into how to log the memory usage on the RPi3 so that should show up if it really is a memory leak problem.

Thanks

Andy

[Alexandru Csete]

Andy,

The 1+ GB memory consumed by gqrx before it's killed strongly suggest
a leak in gqrx, or one of the libraries used by it. There are very
efficient ways to find leaks, but it's not something I can or want to
spend time on explaining. It also requires instrumented code that
doesn't run fast, so it should be done on a PC and not the RPI.

What you can do for sure is to try simpler setups to see when the
crash occurs. Does it occur when you let gqrx run on its own without
touching it or does it come only with the AOS/LOS triggered
recordings?

In any case, you should open an issue on github describing the setup
so that we can debug it later.

Alex

> --
> You received this message because you are subscribed to the Google Groups
> "Gqrx SDR" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to gqrx+uns...@googlegroups.com.
> To post to this group, send email to gq...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/gqrx/6f1f4a36-27a9-4304-bfab-fc18f92f68a1%40googlegroups.com.

[Lloyd Brown]

Andy,

I hope this isn't out of line or anything, but if you're talking about an unattended, automated system, maybe a GUI-based software solution like gpredict/gqrx, isn't the best option.  

Have you considered SATNOGS (satnogs.org)? They have a decent community of users, and a service-oriented software setup.  They've essentially built a distributed network of ground stations. I think you have to put up a station and let others schedule observations, in order to access other stations for observations. But once you do all that, all you'd need to do is access the data files from their network, and process them.  You could even schedule observations on other network members' stations.  Granted that's not many stations yet, but you could help that.

Don't get me wrong. I love gpredict and gqrx.  But something more server-like might make more sense. Match the tool to the job.

Lloyd

> email to gqrx+unsubscribe@googlegroups.com.

> To post to this group, send email to gq...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/gqrx/6f1f4a36-27a9-4304-bfab-fc18f92f68a1%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Gqrx SDR" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gqrx+unsubscribe@googlegroups.com.

To post to this group, send email to gq...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gqrx/CAHG%3DS_fdQx6AWH6YZV6c56VMVLJvKxKFJCKJtV%3Dp5Vv-tzqDCQ%40mail.gmail.com.

[Alexandru Csete]

Lloyd,

You are not out of line and I agree with you 100%

Gpredict + gqrx is useful if you want something pretty to look at. If
you don't need the visuals then simple command line tools and scripts
might work better. I bookmarked this post on Reddit few weeks ago:

https://www.reddit.com/r/RTLSDR/comments/5iuzbl/fully_automated_raspberry_pi_noaa_satellite/

For rtlsdr devices this may be the best option right now.

On the other hand, I'm happy that people test gqrx in various
unintended scenarios :-)

Alex

>> > email to gqrx+uns...@googlegroups.com.

>> > To post to this group, send email to gq...@googlegroups.com.
>> > To view this discussion on the web visit
>> >
>> > https://groups.google.com/d/msgid/gqrx/6f1f4a36-27a9-4304-bfab-fc18f92f68a1%40googlegroups.com.
>> >
>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Gqrx SDR" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to gqrx+uns...@googlegroups.com.

>> To post to this group, send email to gq...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/gqrx/CAHG%3DS_fdQx6AWH6YZV6c56VMVLJvKxKFJCKJtV%3Dp5Vv-tzqDCQ%40mail.gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>

> --
> You received this message because you are subscribed to the Google Groups
> "Gqrx SDR" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to gqrx+uns...@googlegroups.com.

> To post to this group, send email to gq...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/gqrx/CAKvBWagzjwN4fVzgc%3DMEaxYbw8sTQ4adixjR2Mq%3Dx6_9y_Z3pw%40mail.gmail.com.

[Andy Webster]

Alex, thanks again, I'll do a bit more digging and try to isolate the behaviour before logging an issue on github.

Lloyd, thanks for chipping in, no problem at all!  The system I'm trying to put together is intended to be automated but not entirely unattended... that actual end result I have in mind is a RPi-based system that can be wall-mounted in a school corridor with a monitor attached.  The monitor can display GPredict, GQRX and the latest image received by the system alongside a poster explaining what's going on (the underlying theme being satellites use radio to get data back down to Earth and doesn't it all look enticing...).  RPi in a transparent case, flashy lights, interesting things moving around on the display.  I sketched the idea up at some point to discuss with the teacher I'm working with:

(not sure how the image comes out in Google Groups...)

I've been running a school STEM project with the GPredict/ GQRX/ Wxtoimg combo with some success and this seems like a nice way to tidy it all up and bring in a bit of Raspberry Pi...

I've also posted up another thread on this group yesterday about restoring filter settings on GQRX startup and this desire comes from two angles: firstly a desire to have as few things for the students to set correctly each time they play with their system and secondly from a desire to have the whole automated system set itself up after a scheduled (or indeed unscheduled!) reboot.  I've worked with systems that drive public monitors before and there's nothing worse than having your crashed screen on display for all to point and laugh at!  I'm thinking a scheduled reboot each night would be "a good thing"....

Having said all that, it may be a bit too much to ask and alternative suggestions are always welcome!  I'd come across SATNOGs before and skipped over it because it seemed aimed at producing useful data rather than hobby-level performance (our antennas tend to be disadvantaged and in the UK there is terrible interference to NOAA APT from pagers) but I'll have another look with this in mind, thanks.

Andy

> email to gqrx+uns...@googlegroups.com.

> To post to this group, send email to gq...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/gqrx/6f1f4a36-27a9-4304-bfab-fc18f92f68a1%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Gqrx SDR" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gqrx+uns...@googlegroups.com.

[Alexandru Csete]

Andy,

Thanks for the update and the info. The setup looks cool.

Restoring the filter settings is not too much to ask, I just can't do it right now because all my stuff is in boxes and I don't even have a development environment setup. It's just very bad timing for me, sorry. 

The situation may be different next week or the week after, but if the request only exists in my mailbox, I will most certainly forget about it. THatø's why I recommended creating a feature request on Github.

As a workaround, you could try receiving using wide FM mono - it might just work with strong signals.

Alex

To unsubscribe from this group and stop receiving emails from it, send an email to gqrx+unsubscribe@googlegroups.com.

To post to this group, send email to gq...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gqrx/bd2d19fd-8312-416c-909f-bad1b7517676%40googlegroups.com.

[Andy Webster]

Stripped the setup down to just RPi3/Raspbian/Pixel and gqrx and monitoring the memory usage over 24 hrs shows a steady climb once gqrx is started.  I've posted an issue on github with all the details I can think of now:

https://github.com/csete/gqrx/issues/472

Thanks again,

Andy

[Alexandru Csete]

On Wed, Jan 4, 2017 at 12:07 AM, Andy Webster <andywe...@gmail.com> wrote:
> Stripped the setup down to just RPi3/Raspbian/Pixel and gqrx and monitoring
> the memory usage over 24 hrs shows a steady climb once gqrx is started.
> I've posted an issue on github with all the details I can think of now:
>
> https://github.com/csete/gqrx/issues/472

Thanks for testing Andy! I should be able to reproduce it on a desktop
setup unless it occurs in a RPI-specific library. I will try to
resolve this quickly and also look into restoring the filter and FM
settings between sessions.

Alex

[Andy Webster]

My pleasure, happy to help.  Alex you are a true gent.

Andy

--
You received this message because you are subscribed to a topic in the Google Groups "Gqrx SDR" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/gqrx/HrV0AaKIG4o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to gqrx+unsubscribe@googlegroups.com.

To post to this group, send email to gq...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gqrx/CAHG%3DS_cKEVDVPei19w5uzQxLawnDqAROJA9tGbiYg5ZabmJM5w%40mail.gmail.com.