Optimizing FT8 Decode

[Steve Haynal]

Hi Group,

As part of my experiments with synchronized coherent receivers, I want to reduce FT8 decode times so that I can collect data in real time. I'd like to send spots to PSK reporter as kf7o_rx1, kf7o_rx2, kf7o_22p5, etc, for both HL2 receivers plus combinations of the the IQ streams from both radios. I created a gateware variant with larger CIC so that the RX is decimated to 4kHz IQ in the HL2. I wrote Python scripts which save the streams from RX1 and RX2 as .c2 files, which is a WSJT-X internal format of raw IQ values. The .c2 files are accepted by this modified version of ft8d. This differs from what SparkSDR or WSJT-X do where the FT8 decoder accepts 12kHz audio. The idea is that native processing of IQ data at a lower sample rate 2*4kHz will be faster than converting a higher rate audio of 12kHz to complex and then processing. ft8d also takes 1 minute worth of data and decodes the entire minute, which may reduce setup/teardown time overheads. I've collected some data:

ft8d (Optimized, loops unrolled and using processor AVX2, -O3 -funroll-loops -march=znver2 -mtune=znver2 -fopt-info)

0.2 seconds for 41 spots at depth 1

2.2 seconds for 61 spots at depth 3

0.9 seconds for 53 spots at depth 5

jt9 (Unoptimized, built straight from the source with default settings)

0.6 seconds for 18 spots at depth 1

6.8 seconds for 30 spots at depth 3

7.8 seconds for 30 spots at depth 5

jt9 (Optimized, loops unrolled and using processor AVX2, -O3 -funroll-loops -march=znver2 -mtune=znver2 -fopt-info)

0.2 seconds for 18 spots at depth 1

2.0 seconds for 30 spots at depth 3

2.1 seconds for 30 spots at depth 5

What I've learned from this data:

* ft8d is faster than jt9, and simpler as we never have to demodulate to USB. The times are for one file decode and use only one CPU. On my machine with 4 CPUs, I can dedicate 2 or 3 CPUs to decoding. The ft8d times are for a full minute of capture, so I should be able to decode 40 to 60 streams in real time.

* jt9 performance can be significantly improved if built to use processor extensions such as SSE or AVX. I plan to rebuild jt9 on the low-end system at my sister's house which uses SparkSDR and jt9 but will often get bogged down and drop spots during high activity times. Rebuilding jt9 with SSE or AVX enabled should help anyone using SparkSDR with the 6 or 10 RX gateware to skim FT8.

* ft8 and jt9 exhibit different behavior for depth 5 which I don't understand.

73,

Steve

kf7o

[PA3GSB]

Hi Steve,

Real nice!

I have started the same experiment using the radioberry.... bringing the sample rate back to 4K... which is a must for RB because the limited bandwith of the data stream from fpga to RPI.

In the past i did this for the WSPR mode (https://github.com/pa3gsb/Radioberry-2.x/tree/master/SBC/rpi-3/special/WSPRBerry). Making a standalone wspr reporting system.

I have used the prune utility made by John Seamons. (https://github.com/jks-prv/cic_prune)

I am still in development fase.... but found gnuradio; which i liked to use for testing the 4K implementation ( in the past i did use your simulation code... but using gnuradio is also a challenge)

Tnx for sharing your results!

Ans as always have fun!

73 Johan

PA3GSB

Op maandag 26 oktober 2020 om 04:08:32 UTC+1 schreef softerh...@gmail.com:

[Alan Hopper]

Hi Steve,

brilliant, I've been meaning to look at this for ages, spark already has code to save .c2 files as it uses them for WSPR. 

The  .c2 files also have the advantage of using floats so levels might be less critical.  jt9 does seem to be sensitive to levels so it might be worth trying different gains, I never looked hard enough to know if this was due to internal thresholds or the truncation to 16bit.

All x64 processors have SSE2 so I'm surprised the standard builds don't at least target that.

My plan was to turn it into a server that did not need restarting, this could be as simple as listening on stdin for the next filename.

I was also going to see if I could build for all platforms using Github actions.

73 Alan M0NNB

[ron.ni...@gmail.com]

Hi Steve,

Can you describe the naming convention and data format of the .c2 files that ft8d requires?

I can't find any descriptions online.

Thanks & 73,

Ron

n6ywu

[Steve Haynal]

Hi All,

Regarding the optimizations, the make file for ft8d had optimizations in place for the Zynq A9 processor and I just tried similar for my ryzen-based system. It turns out that even if I leave off the optimizations, ft8d will still compile and use AVX by default. I checked for exactly what was used by following this:

https://stackoverflow.com/questions/47878352/how-to-check-if-compiled-code-uses-sse-and-avx-instructions

But for jt9 and wsjt-x, the default compile did not make use of SSE or AVX. I had to edit these lines:

#

# Fortran setup

#

set (General_FFLAGS "-Wall -Wno-conversion -fno-second-underscore -march=znver1 -mtune=znver1 -fopt-info")

in this file:

wsjtx-2.2.2/build/wsjtx-prefix/src/wsjtx/CMakeLists.txt

and then "make jt9" in the wsjtx-build directory to rebuild jt9.

There is probably an easier way, but the WSJT-X make setup is pretty complicated and I don't fully understand it.

Regarding the .c2 format, I think the particular .c2 format used by ft8d may be unique. There is a double (8-bytes) dial frequency at the beginning, then the rest of the data is 32-bit float pairs of complex IQ data. The wsprd .c2 data may be slightly different. See:

https://github.com/pavel-demin/red-pitaya-notes/blob/master/projects/sdr_transceiver_ft8/app/write-c2-files.c

https://wsjt-devel.narkive.com/qLSvuxGL/c2-files

73,

Steve

kf7o

[Steve Haynal]

Hi All,

I went to optimize the jt9 running on a J2900 "silvermont" CPU but found out it was already using SSE4. I'm not sure why it picked it up on this machine but not my ryzen machine.

73,

Steve

kf7o

[Alan Hopper]

Hi Steve and all,

I just found a makefile I used to build a version of jt9.exe on windows without many of the dependentcies and much simplified build setup. I can't find my modified jt9 source, I think I had to tweak it to remove qt stuff and shared memory.  Might be of use for anyone who whats to play with this.

73 Alan M0NNB

[Alan Hopper]

I think I used Pavel's crc14.c as well to avoid boost so the only lib was fftw. 

[Ward Cunningham]

You folks have me thinking that I need my own custom version of jt9. Here is my story.

In the year that I was active on ft8 I wrote a lot of software that exploited the udp decode payloads that wsjt-x would direct to any chosen address. I've abandoned that setup and now have been operating an hl2 via sparksdr which I consider a very fine setup except for access to the decodes.

Reading this thread I'm thinking I could hack the jt9 decoder to leave files for me on the disk while it is interacting with sparksdr. I would probably "stream" this by reusing file names with my downstream software running every 15 seconds. Does this sound like a good place to hook in? Is there a better approach?

Thanks and best regards -- Ward

[Alan Hopper]

Hi Ward and all,

I can always tweak spark if you want a feed of the decodes, or I could just add an option not to delete the files.

I think I have recreated my lost files and updated it for wsjtx 2.2.2, if you put the contents of the zip into the wsjtx 2.2.2 source lib directory, delete crc14.cpp and run 'make all' under mingw64 it should allow a much simpler build on windows (no hamlib,qt or boost), it builds all the encoders and jt9 renamed to ftdecode. My plan when I first did this was to include Pavel's tweaks but try and keep all the other modes.

73 Alan M0NNB

[Ward Cunningham]

Thank you Alan. My son Pat and I will be exploring this over the weekend.