strange problem with rx888-web display

[Tom McDermott]

Have setup rx888 in spectrum display mode.  Having a problem where most of the

frames sent to the http webpage are all black.  A few frames per second have

good spectrum data.  This results in the webpage flickering badly.  The waterfall

shows some valid lines, but mostly black lines.

When tuning a station and enabling audio, the audio sounds fine with no choppiness.

Have tried many experiments trying to isolate the problem:

1. The source is a 6-core/12-thread Core i5 processor that RX888 is plugged into.

HTOP reports about 29% utilization.

ka9q-radio running radiod@rx888-web is running on this (using systemctl start ...).

The logs indicate proper operation.. It says it found all needed fftw3 files, does not

suggest any additional fft3w settings.

TTL=0 is set in the rad...@rx888-web.conf file to avoid any multicast traffic.

2. ka9q-web is also running on this same device. 

3. There are almost no ADC overload events.

4. Using Firefox or Chrome on the i5 to localhost:8081 yields the flickering display.

5. Using Firefox or Chrome on a different core-i7 desktop device to the core-i5

http:<core-i5-ip address>:8081 yields the same flickering behavior.

6. This was tried both with Franco's original ka9q-web.c file and with Newels' port.

behavior the same for both.

At this point I am not sure how to debug where the problem is.

Is ka9q-radio emitting sample lines that are occasionally correct but mostly all zeros?

Is ka9q-web incorrectly converting correct samples from ka9q-radio most but not all the time?

Can anyone provide hints as to what I might do to isolate the problem?

-- Tom, N5EG

 

[Glenn Elmore]

Probably no help at all but I think I've seen something close to this symptom from ka9q-web when there were two sessions open on the same Firefox browser and host. 

I think I simply shut down the offending tab and (possibly) restarted.

I had the impression that this was likely a ka9q-web issue but this really has no good basis.

Glenn n6gn

--
You received this message because you are subscribed to the Google Groups "ka9q-radio" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ka9q-radio+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/ka9q-radio/CACO3nRQ-%2B9Fb_p0PVWa_%2Btut67Za_6q-iaXFd7-%3DPs-0C2%3DeYw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Rob Robinett]

HI Tom,

Are you seeing error reports from the radiod.service in the syslog?

Each client session consumes 600 Kbps, but that shouldn't burden your network

Rob

--

You received this message because you are subscribed to the Google Groups "ka9q-radio" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ka9q-radio+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/ka9q-radio/CACO3nRQ-%2B9Fb_p0PVWa_%2Btut67Za_6q-iaXFd7-%3DPs-0C2%3DeYw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Rob Robinett

AI6VN

r...@robinett.us

mobile: +1 650 218 8896

[Tom McDermott]

Hi Rob - this is what is displayed by $ cat /var/log/syslog | grep ka9q    (truncated to today's runs):

2024-12-30T09:04:29.411303-08:00 tom-ThinkCentre-M740q-N000 radiod[3536]: found rx888 vendor 04b4, device 00f3, manufacturer 'Cypress', product 'WestBridge ', serial '0000000004BE', loading rx888 firmware file /usr/local/share/ka9q-radio/SDDC_FX3.img, done
2024-12-30T09:04:31.584148-08:00 tom-ThinkCentre-M740q-N000 radiod[3536]: fftwf_import_wisdom_from_filename(/var/lib/ka9q-radio/wisdom) succeeded
2024-12-30T09:10:01.954267-08:00 tom-ThinkCentre-M740q-N000 CRON[3699]: (root) CMD (   cd /var/lib/ka9q-radio; find ft4 ft8 wspr -name '*.wav' -mmin +30 -delete)
2024-12-30T09:28:11.100191-08:00 tom-ThinkCentre-M740q-N000 radiod[4029]: fftwf_import_wisdom_from_filename(/var/lib/ka9q-radio/wisdom) succeeded

The connection is hardwired GbE. I don't see any errors...

Per Glenn's comments - have only one web browser page open.   It seems to have a behavior that might suggest a timing or drop

problem between ka9q-web and the browser.  Is there a way to slow down the frame rate to the browser to test this hypothesis?

-- Tom, N5EG

[Rob Robinett]

look for log lines with 'radio'

instead of a grep, I  run:

sudo journalctl -u radiod@rx888-wsprdaemon -f -n 40

[Scott Newell]

At 02:37 PM 12/30/2024, Tom McDermott wrote:

Per Glenn's comments - have only one web browser page open.   It seems to have a behavior that might suggest a timing or drop
problem between ka9q-web and the browser.  Is there a way to slow down the frame rate to the browser to test this hypothesis?

Yes. In ka9q-web.c, look for the spectrum_thread() function. That's the polling loop for the spec demod. Change the usleep() call to change the poll rate. It's in us, so the default of 100,000 is 10 Hz.

Another way to greatly reduce the data transfer rate is to zoom way in, so that there are far less than 1620 bins in the spectrum. As Phil has mentioned before, the worst case is at startup when it's showing 1620 bins of the full spectrum. Zooming in reduces the CPU usage in the radiod spectrum thread, and zooming to <1620 bins will also reduce the data rate between ka9q-web and the browser.

1. The source is a 6-core/12-thread Core i5 processor that RX888 is plugged into.

HTOP reports about 29% utilization.

ka9q-radio running radiod@rx888-web is running on this (using systemctl start ...).

The logs indicate proper operation.. It says it found all needed fftw3 files, does not

suggest any additional fft3w settings.

Does systemctl status or the syslog show that radiod is crashing and restarting? (I got in a situation where ka9q-web was segfaulting radiod, and I'd see the webpage load, empty spectrum for 4 or 5 seconds, then some data, then 4-5 seconds of empty spectrum, and so on. The 4-5 seconds was the crash and systemd timeout for auto restart. That ka9q-web patch that was crashing radiod isn't public yet, and ka9q-radio has been fixed, so I doubt this is your problem, but ?)

4. Using Firefox or Chrome on the i5 to localhost:8081 yields the flickering display.

5. Using Firefox or Chrome on a different core-i7 desktop device to the core-i5

http:<core-i5-ip address>:8081 yields the same flickering behavior.

Wow, that is interesting.

How is ka9q-web running? Is wsprdaemon managing it, or did you create a systemd service for it, or are you running it in a shell? I'm curious if it's in a crash/restart loop, and I'm curious if there could be multiple copies of ka9q-web running.

Are you on a recent version of ka9q-radio?


I'm sure you've run through most of this already, but off the top of my head I'd:

Try a big download, speedtest, or rsync transfer to verify that the LAN isn't flaky due to a wonky ethernet cable/connector.

Close any open ka9q-web browser tabs, then kill ka9q-web and radiod, then restart radiod. Leave a terminal up to watch the logs for any crashes/restarts (tail -F | grep radiod, or journalctl..., or whatever you like).

Run control hf.local and verify that SSRC 1000 and 1001 aren't in use (those are the defaults that ka9q-web will create when the first webpage is loaded).

Verify no ka9q-web instance is running, then start it in a terminal and leave it up, watching for errors.

Open up a browser tab to the ka9q-web webpage (by IP address to rule out DNS and/or IPv6 problems) and see what happens. Does radiod or ka9q-web crash?

Once the spectrum is displayed, maybe run control hf.local again and see that SSRC 1000 and 1001 have been created.


My normal setup is a headless old i7-4790 for radiod (ttl=0) and ka9q-web, with another PC on the gigE LAN running the browser, so not very different from your config. I can even run the webpage (+ audio!) on a $100 ChromeOS tablet over Wifi without drama. My gut tells me this one is going to end up being something simple that we're just overlooking.


--
newell  N5TNL

[Tom McDermott]

Back from my wife's dentist appointment....

The version of ka9q-radio is:

-------------------------

commit 6c116e9eb8cc8277aa538e389c5d99551efae2af (HEAD -> main, origin/main, origin/HEAD)
Author: Phil Karn <ka...@ka9q.net>
Date:   Mon Dec 23 19:52:19 2024 -0800

    run hfdl under user 'radio'

The version of ka9q-web is:

----------------------------

commit 79ce0240eb1f3b689ee3311ed31cd9f8259e4992 (HEAD -> main, tag: v2.34, origin/main, origin/HEAD)
Author: Scott Newell <git...@n5tnl.com>
Date:   Wed Dec 18 06:10:57 2024 +0000

    Tune spectrum demod stream to 0 Hz when webpage closes.
   
    Name the control, audio, and spectrum threads. Remove some unused code.
---------------------------

Here is the output of $ sudo journalctl radiod@rx888-web trimmed to the running time period a few hours ago:

Dec 30 12:39:12 tom-ThinkCentre-M740q-N000 systemd[1]: Started rad...@rx888-web.service - rx888-web radio receiver.
Dec 30 12:39:12 tom-ThinkCentre-M740q-N000 radiod[4520]: KA9Q Multichannel SDR main.c last modified Tue Dec 24 12:00:02 2024
Dec 30 12:39:12 tom-ThinkCentre-M740q-N000 radiod[4520]: Copyright 2023, Phil Karn, KA9Q. May be used under the terms of the GNU Public License
Dec 30 12:39:12 tom-ThinkCentre-M740q-N000 radiod[4520]: Loading config file /etc/radio/rad...@rx888-web.conf...
Dec 30 12:39:12 tom-ThinkCentre-M740q-N000 radiod[4520]: found rx888 vendor 04b4, device 00f3, manufacturer 'Cypress', product 'WestBridge ', serial '0000000004>
Dec 30 12:39:13 tom-ThinkCentre-M740q-N000 radiod[4521]: Established under name 'rx888-web'
Dec 30 12:39:13 tom-ThinkCentre-M740q-N000 radiod[4523]: Established under name 'rx888-web'
Dec 30 12:39:13 tom-ThinkCentre-M740q-N000 radiod[4522]: Established under name 'web.local'
Dec 30 12:39:13 tom-ThinkCentre-M740q-N000 radiod[4524]: Established under name 'web.local'
Dec 30 12:39:13 tom-ThinkCentre-M740q-N000 radiod[4520]: found rx888 vendor 04b4, device 00f1, manufacturer 'sdr prototypes', product 'RX888mk2', serial '000900>
Dec 30 12:39:13 tom-ThinkCentre-M740q-N000 radiod[4520]: rx888 USB speed: Super (5 Gb/s)
Dec 30 12:39:14 tom-ThinkCentre-M740q-N000 radiod[4520]: Nominal samprate 64,800,000, reference 27,000,000.000000, feedback divisor 28 + 4/5, VCO 777,600,000.00>
Dec 30 12:39:14 tom-ThinkCentre-M740q-N000 radiod[4520]: Output divider 12 + 0/1, rdiv 1, actual samprate = 64,800,000.000000
Dec 30 12:39:14 tom-ThinkCentre-M740q-N000 radiod[4520]: rx888 generic: rx888 reference 27,000,000.0 Hz, nominal sample rate 64,800,000 Hz, actual 64,800,000.00>
Dec 30 12:39:15 tom-ThinkCentre-M740q-N000 radiod[4520]: fftwf_import_system_wisdom() succeeded
Dec 30 12:39:15 tom-ThinkCentre-M740q-N000 radiod[4520]: fftwf_import_wisdom_from_filename(/var/lib/ka9q-radio/wisdom) succeeded
Dec 30 12:39:15 tom-ThinkCentre-M740q-N000 radiod[4520]: rx888 running
Dec 30 12:39:15 tom-ThinkCentre-M740q-N000 radiod[4520]: 0 total demodulators started
Dec 30 12:39:16 tom-ThinkCentre-M740q-N000 radiod[4542]: Established under name 'rx888 generic'
Dec 30 12:39:16 tom-ThinkCentre-M740q-N000 radiod[4543]: Established under name 'web.local'

The instance of ka9q-web was run in a shell:  it produced no error messages, just periodic keep-alive messages.

tom@tom-ThinkCentre-M740q-N000:~/ka9q-web$ ./ka9q-web -m web.local
ka9q-web version: v2.34
[87E006C0] [2024-12-30 15:34:56] [INFO response.c:192] [::ffff:192.168.2.217] "GET /" 200 169 (Keep-Alive)
[87E006C0] [2024-12-30 15:34:56] [INFO response.c:192] [::ffff:192.168.2.217] "GET /radio.html" 200 9320 (Keep-Alive)
[87E006C0] [2024-12-30 15:34:56] [INFO response.c:192] [::ffff:192.168.2.217] "GET /style.css" 200 179 (Keep-Alive)
[91A18240] [2024-12-30 15:34:56] [INFO response.c:192] [::ffff:192.168.2.217] "GET /colormap.js" 304 0 (Keep-Alive)
[91A18240] [2024-12-30 15:34:56] [INFO response.c:192] [::ffff:192.168.2.217] "GET /spectrum.js" 304 0 (Keep-Alive)
[87E006C0] [2024-12-30 15:34:56] [INFO response.c:192] [::ffff:192.168.2.217] "GET /pcm-player.js" 304 0 (Keep-Alive)
[87E006C0] [2024-12-30 15:34:56] [INFO response.c:192] [::ffff:192.168.2.217] "GET /radio.js" 304 0 (Keep-Alive)
[91A18240] [2024-12-30 15:34:56] [INFO response.c:192] [::ffff:192.168.2.217] "GET /favicon.ico" 200 15406 (Keep-Alive)
[8CA006C0] [2024-12-30 15:34:56] [INFO response.c:192] [::ffff:192.168.2.217] "GET /version.json" 200 18 (Keep-Alive)
[91A18240] [2024-12-30 15:35:06] [INFO response.c:192] [::ffff:192.168.2.217] "GET /" 101 0 (Keep-Alive)

  [ad nauseum]

--------------------------

Ran iperf3 tests both directions:

tom@tom-ThinkCentre-M740q-N000:~$ iperf3 -s
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 192.168.2.217, port 60262
[  5] local 192.168.2.93 port 5201 connected to 192.168.2.217 port 60274
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   110 MBytes   925 Mbits/sec                  
[  5]   1.00-2.00   sec   111 MBytes   928 Mbits/sec                  
[  5]   2.00-3.00   sec   110 MBytes   927 Mbits/sec                  
[  5]   3.00-4.00   sec   111 MBytes   928 Mbits/sec                  
[  5]   4.00-5.00   sec   111 MBytes   928 Mbits/sec                  
[  5]   5.00-6.00   sec   111 MBytes   928 Mbits/sec                  
[  5]   6.00-7.00   sec   111 MBytes   928 Mbits/sec                  
[  5]   7.00-8.00   sec   110 MBytes   927 Mbits/sec                  
[  5]   8.00-9.00   sec   110 MBytes   925 Mbits/sec                  
[  5]   8.00-9.00   sec   110 MBytes   925 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-9.00   sec  1.03 GBytes   985 Mbits/sec                  receiver
iperf3: the client has terminated

^Ciperf3: interrupt - the server has terminated
tom@tom-ThinkCentre-M740q-N000:~$ iperf3 -c 192.168.2.217
Connecting to host 192.168.2.217, port 5201
[  5] local 192.168.2.93 port 55680 connected to 192.168.2.217 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   113 MBytes   949 Mbits/sec    0    392 KBytes      
[  5]   1.00-2.00   sec   112 MBytes   935 Mbits/sec    0    392 KBytes      
[  5]   2.00-3.00   sec   111 MBytes   930 Mbits/sec    0    392 KBytes      
[  5]   3.00-4.00   sec   112 MBytes   936 Mbits/sec    0    392 KBytes      
[  5]   4.00-5.00   sec   111 MBytes   934 Mbits/sec    0    392 KBytes      
[  5]   5.00-6.00   sec   112 MBytes   937 Mbits/sec    0    410 KBytes      
[  5]   6.00-7.00   sec   111 MBytes   934 Mbits/sec    0    410 KBytes      
[  5]   7.00-8.00   sec   111 MBytes   933 Mbits/sec    0    410 KBytes      
^C[  5]   8.00-8.34   sec  37.5 MBytes   933 Mbits/sec    0    410 KBytes      
- - - - - - - - - - - - - - - - - - - - - - - - -

Killed all ka9q on the server, started radiod@rx888-web in a SSH 1 shell window.  This creates web.local

In SSH 2 shell started ka9q-web -m web.local

in SSH 3 shell ran control web.local

---------------

tom@tom-ThinkCentre-M740q-N000:~/ka9q-radio$ control web.local
         SSRC    preset      freq, Hz   SNR output channel
0 channels; choose SSRC, create new SSRC, or hit return to look for more:
         SSRC    preset      freq, Hz   SNR output channel
0 channels; choose SSRC, create new SSRC, or hit return to look for more: ^C

------------------

Started web instance on the client using the ip of the server:   http://192.168.2.93:8081

tom@tom-ThinkCentre-M740q-N000:~/ka9q-radio$ control web.local
         SSRC    preset      freq, Hz   SNR output channel
         1000       usb    10,000,000  19.9 239.255.59.70:5004
         1001              16,200,000   nan (null)
2 channels; choose SSRC, create new SSRC, or hit return to look for more: ^C

-------------------

** radiod shell shows no errors on radiod@rx888-web

** ka9q-web shows no errors (just keep-alives)

---------------------------

I have not yet started to change the refresh rate......

-- Tom, N5EG

[Scott Newell]

At 05:54 PM 12/30/2024, Tom McDermott wrote:

Dec 30 12:39:12 tom-ThinkCentre-M740q-N000 radiod[4520]: Loading config file /etc/radio/rad...@rx888-web.conf...

Dec 30 12:39:15 tom-ThinkCentre-M740q-N000 radiod[4520]: 0 total demodulators started

0 demodulators? Must be a really bare config. Can you attach a copy of /etc/radio/rad...@rx888-web.conf so I can test?

The instance of ka9q-web was run in a shell:  it produced no error messages, just periodic keep-alive messages.

I don't see periodic keep-alives on my server, but I wouldn't expect it to matter.

tom@tom-ThinkCentre-M740q-N000:~/ka9q-radio$ control web.local
         SSRC    preset      freq, Hz   SNR output channel
         1000       usb    10,000,000  19.9 239.255.59.70:5004
         1001              16,200,000   nan (null)

2 channels; choose SSRC, create new SSRC, or hit return to look for more: ^C
-------------------
** radiod shell shows no errors on radiod@rx888-web
** ka9q-web shows no errors (just keep-alives)

That all looks right. Crap. Running out of ideas here.

[Tom McDermott]

Hi Scott - thanks for all your help!

I hooked up a monitor, keybopard, and mouse to the server and ran everything locally on the server, no networking used.

Ran http://localhost:8081 on Firefox on the server itself.  I zoomed in the spectra to about 120 KHz, same flickering.

I am going to try and record what the flickering looks like and upload it to my YouTube channel. May take awhile.

Here is the contents of rad...@rx888-web.conf - it is very similar to ka9q-web/config/rad...@rx888-web.conf

Meld does not show any important differences (just comments and a today change on the Rx888 attenuator setting).

$ cat rad...@rx888-web.conf
# Generic RX888 config

[global]
# these next two establish defaults for all demod sections
hardware = rx888 # use built-in rx888 driver, configured in [rx888]
status = web.local       # DNS name for receiver status and commands
samprate = 12k        # default output sample rate
mode = usb # default receive mode
# rest are defaults
# ttl = 1                # multicast data leaves source host
ttl = 0                # multicast data doesn't leave source host!

#fft-threads = 2
#blocktime = 20 # allowable Opus block times: 2.5, 5, 10, 20, 40, 60, 80, 100, 120
#overlap = 5 # don't change unless you know what you're doing
#data = ft8-pcm.local
#data = spec.local
data = web.local
#pacing = yes

[rx888]
device = "rx888" # required so it won't be seen as a demod section
description = "rx888 generic" # good to put callsign and antenna description in here
gain = 20 # dB
# rest are defaults
#description = "rx888"
#number = 0
#samprate = 64m8          # Hz
#calibrate = 0  # 1e-6 is +1 ppm
#firmware = SDDC_FX3.img
#queuedepth = 16          # buffers in USB queue
#reqsize = 32             # size of each USB buffer in 16KB units
dither = yes              # built-in A/D dither
rand = yes                # Randomize A/D output bits to spread digital->analog crosstalk

# Attempt to reduce overloads 12/30/2024
#att = 0  # PE4312 digital attenuator, 0-31.5 dB in 0.5 dB steps
att = 10  # PE4312 digital attenuator, 0-31.5 dB in 0.5 dB steps

#gainmode = high          # AD8370 VGA gain mode
#gain = 1.5  # AD8370 VGA gain, -25 to +17 dB (low gain mode) or -8 to +34 dB (high gain mode)
#pacing = yes

[Tom McDermott]

Here is a poor quality video of the flicker problem. The first few seconds are at full range scan,

while last few seconds are zoomed in to about 120 KHz span. The video is in real-time.

https://youtu.be/fXQEXDBAiaI

-- Tom, N5EG

[Scott Newell]

At 06:55 PM 12/30/2024, Tom McDermott wrote:
>Hi Scott - thanks for all your help!

Considering the results so far, I'm not sure I've helped...

>Here is the contents of rad...@rx888-web.conf - it is very similar
>to ka9q-web/config/rad...@rx888-web.conf

Problem duplicated. What a weird one! Please try renaming the status
or data channel. I think they can't both be "web.local".

Yet another bug in ka9q-web. Thanks a bunch for finding it!


--
newell N5TNL

[Tom McDermott]

Hi Newell - thank you!  That was indeed the problem.   The control name was edited to be

webctrl.local  while leaving the data name as web.local

Invoking  ./ka9q-web -m web.local resulted in the display never starting,

invoking  ./ka9q-web -m webctrl.local  makes everything work correctly.  

I am not sure how ka9q-web finds the datastream web.local without knowing its name !!!

-- Tom, N5EG

[Glenn Elmore]

Maybe I'm having a different issue.

Restarted after computer crash,
restored a tab in Firefox and it flickered.
 I think display was alternating between upper and lower 30 MHz spans.
I have 129 MHz sampling.

Then I brought up a second tab. That one did not flicker.
So two tabs on Firefox on a  different host, one flickered and one didn't.