Odd time slippage with WSJT-X over 24 hours

[Freeman Pascal, N5FPP]

I have started leaving gqrx monitoring my SDRPlay with WSJT-X over night and even for 24 hours.  Tonight I noticed a really odd problem that eventually was resolved by restarting gqrx.  WSJT-X started to not be able to decode and the audio appeared to be delayed by twelve seconds.  At first I thought it might be the clock on my iMac, but it looked fine.  I even went as far as monitoring WWV on 5Mhz and noticed that the iMac was synced fine but the audio from all of the JT65 signals on 80m were delayed. On a hunch I stopped and restarted gqrx and the problem went away. I can only imagine the problem being with gqrx or the SDRPlay drivers.  Any ideas?

I'm running a local build from the latest code in the git repository.

-Freeman, N5FPP

[Alexandru Csete]

On Wed, Feb 10, 2016 at 6:55 AM, Freeman Pascal, N5FPP
<analias.n...@gmail.com> wrote:
> I have started leaving gqrx monitoring my SDRPlay with WSJT-X over night and
> even for 24 hours. Tonight I noticed a really odd problem that eventually
> was resolved by restarting gqrx. WSJT-X started to not be able to decode
> and the audio appeared to be delayed by twelve seconds. At first I thought
> it might be the clock on my iMac, but it looked fine. I even went as far as
> monitoring WWV on 5Mhz and noticed that the iMac was synced fine but the
> audio from all of the JT65 signals on 80m were delayed. On a hunch I stopped
> and restarted gqrx and the problem went away. I can only imagine the problem
> being with gqrx or the SDRPlay drivers. Any ideas?

Hi Freeman,

It could be sample rate mismatch between the input and the output. For
example, if the oscillator of the SDRPlay runs at a higher rate than
it is supposed to, it will generate more samples than what is being
consumed at the audio output end, so the output buffer will grow.

This does not mean that there is anything wrong with the device. Even
a very small clock mismatch can accumulate over time. Ideally, gqrx
would adaptively correct this as time goes on, but this has not been
implemented so far.

> I'm running a local build from the latest code in the git repository.

That's cool because if you can build gqrx from source you can try to
correct this in the source code. If you can figure out how far off the
SDRPlay oscillator is you can try to add that correction in the source
code. Perhaps SDRPlay API already has a function for doing that,
otherwise I can show you where in gqrx you could do that.

However, this is just a guess, and the error could be something
completely different. But it's worth a try in my opinion.

Alex

[Tim Howe]

Hi all,

I am observing significant latency creep (~1s per hour) in the buffer between a Perseus (USB) SDR front end and gqrx.This latency seems to be between the radio hardware and the FFT/waterfall,

in that after tuning the hardware front end, the delay is visible in the FFT/waterfall changing. Delay in the audio path seems much more reasonable (i.e. tuning the DSP passband

has a fairly immediate affect on the audio output).

This latency does not clear when I stop and restart gqrx DSP - it only goes away when I completely exit gqrx and re start the program (which restarts the Perseus front end as well).

 It appears as though the whole gqrx chain is running at a rate derived from the audio sink device, rather than being slave to the rate provided by the IQ source??

Ideally, I would hope that the signal processing path is synchronous with the sample rate provided by the IQ source, which in my case is a Perseus SDR which contains its own sampling clock master.

The SDR signal path is free to drop the sample rate rate as it sees fit (and I would imagine this would be by integer factors) but would remain synchronous with the clock master in the IQ source.

Ultimately this will result in an audio output stream at some given sample rate.

Now my question is this: IF the gqrx signal path were clocked by the IQ source/master, how is the inevitable slip between the audio samples produced (from the IQ source clock) and the audio sink device,

which contains its OWN clock master be accommodated? A dynamic re-sampling block is usually necessary to achieve this.

Is this being done at all?? I have not found anything online that explains in detail how gqrx accommodates 2 clock masters.

Many thanks

Tim

[Dave Baxter]

Hi.

I have experienced much the same in the recent past, with GQRX on Linux (LMDE 5) working with a RSP1 SDR.

Using any of the simple RTL based dongles, no such problem.  (Except the FCD+, that is not recognized any more, so I can't use them.)

Unable to re-test, as so far I've failed to even get GQRX running in any way on LMDE 6.

(There is something very odd, about the LMDE 6 repositories of much installable software, and also  circular unmet dependency library issues if one tries to build older applications from source!)

Not urgent for me.  73

Dave G8KBV.  (Seriously considering another distro' due to the above.)

--
You received this message because you are subscribed to the Google Groups "Gqrx SDR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gqrx+uns...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/gqrx/af8eb278-e871-44ca-834a-8282c13042dbn%40googlegroups.com.

[righthal...@gmail.com]

The RSP1 and RSP2 have serious timing problems - Mine lose 12 seconds every twenty minutes.

[Bruce]

I've been looking into this with my RSP1 as I have a timing critical application. I don't mind how long the total time delay is (within limits), but I do need it to be a constant delay.
I primarily use the S-meter readings, via the remote interface.

I setup a test jig. A small transmitter transmits for 50ms every 5s, and the TX enable is also fed into the Linux box via a serial port control line.
In an app that listens to the S-meter readings I can time in ms how long it takes between the serial port line transition, and the S-meter registering the TX signal by exceeding a threshold.
This shows about 150ms+/-50ms normally.

However, I have been able to catch a changed delay, but I'm not yet convinced it is a delay that is gradually increasing due to mismatched clocks drifting.
It's really hard to reproduce reliably, but it seems to maybe have more to do with the screensaver (or leaving it) doing something to Gqrx??
I can sit there observing the delay for an hour, and though it isn't constant (Gqrx is actually always a bit lumpy in outputting S-meter readings, sometimes pausing for a few samples then outputting say 5 quickly), on average I can't see a gradual increase happening at all.
However I can come back an hour later, still all the same, then another hour later, it's jumped up, on average, 500ms!  Then, hours later, I'll observe yet another jump of 500ms more (+1s compared to original delay).

The other thing I can report is that there is more than one delay involved. There is the delay I have measured above between signal and S-meter, then sometimes (and quite independently) I see a step change of delay from the S-meter till the audio can 'hear' the signal. I have so far heard this up to 1s extra. So that's a total of 2s extra delay observed if you are listening to the audio, but contributed to via two entirely separate mechanisms!

Do those who have reported RSP timing delays know for sure it is definitely due to gradual clock mismatches, or they just suspect this might be the case?
Can you maybe heat or cool the RSP to trigger extra delay?
Though my tests don't disprove this, the weird step changes I've seen, and the fact there are two independent delay mechanisms at play here, would perhaps appear to point to something else.

I'm also wondering if on Gqrx startup somehow you can sometimes end up with something extra in the (USB?) buffer, and then you never catch up?

Maybe I'll have to feed all the measured delays into a file so i can see if there's a step change or a gradual change?

Why wouldn't any receiver have a drift issue? Do some use a clock derived from the USB master to sample? Or is this an issue in the RSP driver that others don't have?
Seems like GQRX needs to very very slightly oversample to gobble up any slack in the case the receiver is running faster than the Linux CPU crystal.

Also, the original reply below: Perhaps SDRPlay API already has a function for doing that, otherwise I can show you where in gqrx you could do that. The problem is that the difference will not be a constant. it will vary with temperature. Safer just to always slightly oversample (and if no samples waiting just wait for next sample time - that way it handles the RX clock being either fast or slow compared to CPU. The extreme case would be to sample at double the input rate (but almost all the time every 2nd sample is a miss), and so the effective sample rate will end up being close to the original.

To view this discussion visit https://groups.google.com/d/msgid/gqrx/9a151586-49e3-4900-8e84-487abf5162een%40googlegroups.com.

-- 
Cheers,
Bruce

[Bruce]

See original below for context. I'm using the RSP1 SDR, but this applies to any SDR that display this delay issue with Gqrx.

I have now recorded the measured delay every 5s into a file over 24hrs or so.
My measurement is a bit ropey, so it varies a bit due to relying on a non-realtime-os to measure it. I've eliminated all the occasional impossible times under 100ms.
However, the trend is very clear when plotted out (see attached image).

Yes; there is a jump in delay when the screen is brought to life. You can see this in the occasional huge time delay points at the top of the graph. However, this is temporary and GQRX recovers. It may be purely a measurement error (I'm relying on Qt Timers).

I suspect it doesn't show a drift at all for the first few hours because it may have been drifting the other way then?; if the RSP1 clock is slower than the CPU clock then Gqrx will have an underrun and just miss a sample occasionally - no big deal. It's only an issue when the RSP1 clock drifts faster then the CPU clock, and a buffer somewhere starts to fill.

I see a number of possible solutions to this problem (perhaps some may wish to comment?):

1. Fix the problem #1. There should not be an infinite buffer involved; there should be a re-clocking stage instead. This could be as simple as a semaphore protected location(s) where the RSP clock domain writes the (IQ?) samples to, and the CPU clock domain reads from, effectively a single sample buffer. However, this problem might be a USB buffer, or a buffer inside the RSP binary driver, both of which are hard to 'get at'. Does anyone actually know/suspect where this problem is?
2. Fix the problem #2. Have the RSP1 clock be master clock for everything. Nice to have, but likely not a realistic fix.
   
3. Somehow arrange that the CPU clocking is fractionally faster then the RSP clock, but with a difference that exceeds the maximum drift. That way you'll get occasional underruns but never overruns/buffering.
4. If #3 not possible, arrange that that the CPU clocking is twice the RSP clock. Mostly every 2nd sample will be an underrun (use last sample again), but it should work as long as Gqrx is told about the double sampling. Not sure if underruns will just use last sample value again though, inside binary driver? If underruns give a zero sample instead, that's a problem.
   
5. Periodically restart the whole DSP and RSP1 and USB, so the drift doesn't get too far. eg. every 10mins? Though this is a bit of a non-solution, it may end up being the most practical!
6. If #5, also look at restarting the sound side as well, as I noted below there also seems to be an independent increasing delay issue there too (between waterfall & sound card output). I don't know if this 2nd problem is due to another independent clock domain (eg. CPU and SoundCard), or due to some event (changing sound outputs, wakeup, or something).
   

-- 
Cheers,
Bruce

[Bruce]

Here's an update in case anyone is interested. Zero feedback from the mailing list so I'm not sure if anyone is?

I've discovered I made an incorrect assumption. Tim Howe, in the original post, said "This latency does not clear when I stop and restart gqrx DSP".
I assumed the extra latency Tim was seeing, was much the same thing I've been seeing with the RSP. I guess it seemed a reasonable assumption?
That meant I was concentrating on blocks before the Receiver ("DSP") in Gqrx; osmosdr soapy module, soapy itself or the SDRPlay supplied binary driver.
Well, after a few days experimentation (can take quite a while for the drift to happen for me), I can say that stopping and starting the DSP does reliably reset my latency to nominal! The latency I'm particularly concerned about was between the RSP and the FFT/waterfall/S-meter.
Tim was noting total delay between the signal (SDR) and the audio out of GQRX. This is total delay SDR->FFT->Audio. I have seen the 2nd part increase too, independently, as I've noted below, so perhaps Tim's delay was mostly due to the 2nd part?  I haven't evaluated whether the 2nd part of total latency is affected by restarting the DSP, because that's not part of my test-jig (yet).

However, this is probably good news. It explains why other SDR programs seem unaffected by this issue, it should be easier to fix, and temporary workaround #5 (below) is much easier to implement. it's a GQRX issue only it seems, in the GnuRadio RX blocks. In theory we should be able to see a buffer filling up somewhere in there.
I've already implemented a restart in my GQRX that not only restarts the DSP, but reconnects to the SDR as well - you can simulate this in an unmodified GQRX by connecting to another input device (eg. a small IQ file), then back to your SDR again. This may be a good enough fix for my purposes.

-- 
Cheers,
Bruce