Gateware 20200516_71p2 with 9 Receivers!

[Steve Haynal]

Hi Group,

This is a receive only release with 9 receivers meant for all the skimmers and SparkSDR users out there. See here. Use the hl2b5up_cicrx.rbf file found in the hl2b5up_9rx for the recommended ethernet gateware upgrade using SparkSDR or Quisk.

The final FIR filters are removed. Only the 384kHz bandwidth setting will return meaningful data. 192, 96 and 48kHz bandwidth settings will return garbage. Since only CIC filters are present, you will see droop at either extreme of the receiver bandwidth as seen in the picture below. Also, some aliasing at the extreme ends may occur. But there is still almost 200kHz of usable bandwidth in the center. This is enough to cover most data frequencies within a band. In fact, some of the newer SDR RTL I've looked at only applies a decimate by 2 FIR as the final step after CIC filtering, so would consider 192kHz of this usable. 

Big thanks go to John (JKS) KF6VO the principal designer of the KiwiSDR for his CIC filter register pruning code. I think we discussed CIC register pruning on this list 5 years ago, and it has been on my todo list since, but I am just now trying it out. Without the pruning, I can fit 7 receivers, almost 8. With the pruning I can fit 9 receivers and am only 11 LABs short for 10 receivers. The Cyclone IV we use has 1395 LABs, so 11 is just 0.7%! I think I should be able to fit 10 receivers with just a little more work. 10 receivers would allow us to skim all ham bands 160M-10M with just a single HL2.

I've been running SparkSDR with 9 receivers and see no issues with my midgrade AMD Ryzen 3 3200G system. There are 4 CPUs in my system and SparkSDR is taking just 75% of 1 CPU, or ~17% system total. There are some CPU spikes at the end of every 15 second FT-8 decode cycle.

It turns out that 9 receivers pack perfectly in the protocol 1 scheme with no wasted bytes. If we go to 10 receivers, then there will be 16 wasted bytes in every UDP packet. Still, at 9 or 10 receivers I think we should be able to get up to 16MHz total bandwidth with the HL2's 1Gbps ethernet even with the current UDP scheme and the wasted mic bytes. At 9 384kHz receivers, we are sending 3.456MHz bandwidth, less than 25% of what I expect possible with our 1Gbps connection. I'd like to push this limit to see what is the most bandwidth we can reliably send. I'd like to add an extension to the protocol to select higher bandwidth receivers, which comes basically for free with the current setup. This will be especially helpful for CIC-only gateware variants where the edges droop. The easiest bandwidths to generate are 76.8MHz/10/N where N can range from 1 to 20. The current 384kHz is with N at 20. N of 10 would give 768kHz, N of 5 1.536MHz. Any interest from Alan or other software developers? Let me know a handful of wider bandwidths that you'd prefer.

I would appreciate testing and feedback of this release. Does this work for skimming? How annoying is the droop? Is any aliasing at the edges seen? The beauty of the latest HL2 gateware is that you can easily reprogram over ethernet with SparkSDR or Quisk, and now we have 2 images stored in the EEPROM so you have a rescue image if anything goes wrong with a gateware upgrade. So you can give it a try and then go back to 71_p1 when you want to transmit again.

73,

Steve

kf7o

In this picture, the blue arrow is pointing to the FT-8 frequencies near the center of the bandwidth. You can see the 15 second cycles in the waterfall. I live in a very noisy environment. The green arrow shows time when my monitor was off. You can see that removes some spurs in the waterfall. The other spurs are from various power supplies in my neighborhood and are external to the HL2 as they are not seen without an antenna.

 

[Alan Hopper]

Hi Steve,

brilliant!  Flattening out the droop and hiding the edges should not be too hard on the pc, I have some hooks in Spark for this left over from my hl1 no fir experiments. I think we need something in the discovery to indicate no fir and the bandwidths available.  My code is happy with 768 and 1536kHz as protocol 2 radios have this. I know dropping to 192kHz is tricky and uses more fpga resources but is probably the ideal for skimming.

73 Alan M0NNB

[Roger Critchlow]

Skimming just fine here in Boston, decoding FT8 on 80-10, showing 220% CPU and 2.4% memory usage with top, running on an i5-8265U Lenovo laptop.  The fan is intermittently engaged, more off than on.  The jt9 decoders are hardly evident, but there isn't a lot to decode this morning.  Very nice job.

-- rec -- ad5dz --

--
You received this message because you are subscribed to the Google Groups "Hermes-Lite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hermes-lite...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hermes-lite/b42622f8-7045-4ddb-b0aa-3be8dde55d74%40googlegroups.com.

[neil whiting]

Not so successful here. Starts OK and I can run a single band and get decodes. However adding more recievers it gets to a point where the FT8 traces look a little smeared and decodes stop, even on those which were originally good. I have to restart SparkSDR to get it to start decoding again.

The maximum appears to be 3 RXs before decoding stops (not an extensive investigation).

I wonder if I am running out of ethernet bandwidth with the 384ksps sample rate? My network is 1Gbps but the switch up the garden used for the RXs may be 100Mbps?

73,  Neil  G4BRK

To unsubscribe from this group and stop receiving emails from it, send an email to herme...@googlegroups.com.

[Alan Hopper]

Hi Group,

whilst this firmware should work fine with Spark most of the time there are some situations where you could end up using the bad part of the spectrum near the edges of the receiver, selecting favorites should be fine but tuning across a wide band with multiple virtual receivers  could result in some of them using the edges.  I'll support it fully in future releases.

Neil,

that does sound like a network issue, are you seeing ep6 errors?

73 Alan M0NNB

[neil whiting]

Hi Alan,

I just checked and it is in fact a Gbit switch down the garden.

I am seeing ep6 errors, lots of them! I stopped one of the HL2s and the other stopped getting ep6 errors.

I then tried starting 1 FT8 RX at a time (using favourites) and on the 4th the ep6 errors started again.

I have to restart Spark at this point - closing the RXs does not let it recover.

73,  Neil  G4BRK

[Alan Hopper]

Hi Neil,

Spark currently never releases a firmware receiver once it is opened, this is a hang over from earlier firmware when adding and removing receivers needed the radio to stop and start. Hence the bandwidth not dropping when closing rxs.  It does sound very much like a total network bandwidth problem. Do you have some busy 100mbit device on the network?

73 Alan M0NNB

[neil whiting]

Hi Alan,

Thanks very much for your assistance and suggestions.

I had another HL2 which was running 4m FT8 - the only other real demand on the network.

Turning this off I managed 4 RXs decoding FT8. The 5th started the failure mode.

Looking at the Ethernet usage on this PC with 4 RXs working it is about 90Mbps, so it does look like

exceeding 100Mbps may be the problem. I will research how to tell if my network is really running at 1Gbps

like it is supposed to.

73,  Neil  G4BRK

[neil whiting]

I just checked my internet access speed from the 2 cables going to the HL2s using a

Gbit-capable laptop. I get:

download = 360Mbps

upload     = 685Mbps

So looks like there isn't a problem with the LAN speed for other traffic or the cables to the HL2s.

Looking at the HL2 status LEDs in idle mode, link speed is on, not flashing.

According to the documentation this means link speed is 1000Mbps.

Not sure what to look at next. Could latency be a problem? The cable to the

garden is about 30m long.

73,  Neil  G4BRK

[Steve Haynal]

Hi Neil,

Just to double check, do you have SparkSDR set to 384kHz? With this gateware, other bandwidths produce garbage and don't follow the expected ethernet packet sequencing and may throw SparkSDR off.

73,

Steve

kf7o

[Steve Haynal]

Hi Group,

I have also added a regular hl2b5up_main gateware variant to this release. Please see the release page for details. As always, I'd appreciate and feedback and testing.

https://github.com/softerhardware/Hermes-Lite2/tree/master/gateware/bitfiles/testing/20200516_71p2

73,

Steve

kf7o

[neil whiting]

Hi Steve,

Yes I have set 384kHz. I do see garbage with other sample rates.

Thanks for your work on this - your goal of skimming on 10 HF bands in one HL2

is a good one, if a little challenging :-)

73,  Neil  G4BRK

[Alan Hopper]

Hi Neil,

as a sanity check you could see if you can run lots of 384kHz receivers with previous firmware. Sven had an odd problem that only seemed to appear at high bandwidths with two radios and I think it turned out to be mac or ip settings but as you have a problem with just 1 radio I doubt this is the case here. The cable length really should not matter (I have similar to my shed/pub/shack).

73 Alan M0NNB

[neil whiting]

OK, panic over.

More experiments this morning.

Loaded one HL2 with the 6RX_69p4 release and ran 6 RX at 384kHz with no problems with 130mbps data rate on the LAN.

Loaded another with main_71p2 just to keep up to date.

The 3rd one was running cicrx_71p2 all night on my slower Linux machine with 4 RXs with no problems (a few ep6).

I started adding RXs to the this one again and this time no problem. LAN rate 190Mbps.

Not sure why it didn't work for before (on Linux and Windows 10 machines).

Possibly conflict with the 2 HL2s both on the cicrx_71p2 release?

I'm busy today but will do more testing later.

73,  Neil  G4BRK

[neil whiting]

Hi Alan,

I just did that test before seeing your reply this morning!

See previous message from me. Yes I was running with 2 HL2s in the shed.

Maybe the same problem as Sven had.

Thanks/73,  Neil  G4BRK

On Monday, 18 May 2020 09:23:11 UTC+1, Alan Hopper wrote:

Hi Neil,

[Joe LB1HI]

Hi,

There are problems on Raspber Pi 4 and it stutters from 7 RX already. But it works fine (decodes but I have no visual control) even with 9 RX when the SparkSDR window is minimized to the taskbar.

The CPU consumption peak reaches 99 percent when decoding at the end of the 12 second FT8 cycle.

  Alan, is there a chance to reduce the CPU load during decoding? For example, by increasing the buffer delay for half of the bands?

73, Jozef 

For comparison, the CPU usage for the 6 rx gaterware version below.

[Alan Hopper]

Hi Joe,

you are asking the rpi to do quite a lot here :) Spark already queues wsjtx mode decodes, it only launches as many as the number of cpu cores -1 simultaneously. It would be interesting to see how much cpu is spent in wsjtx decoding and how much is spark by trying just usb. I still have much to do to optimize Spark for arm and the rpi and there are probably optimizations for this new no fir mode. I think there are also optimizations for the ft8 code. Spark uses the gpu on the rpi for display and may need special settings on some installs, does the cpu drop as you make the window smaller (best to test with just 1 rx and no ft8)?.  This new rx mode is more greedy of cpu and network, that is the tradeoff for squeezing more receivers in the HL2. I'll fire up the rpi and do some profiling here later this week.

73 Alan M0NNB

[Sven Palmersjö]

Hi Steve!

The new Gateware 20200516_71p2 with 9 Receivers works great! Thanks for bringing them up for us monitor stations!

With my HL2b9 and the new firmware, I work easily 45 virtual receivers with Allan's SparkSDR on 9 different band.

My AMD Ryzen 9 3900 with 12 cores works with CPU usage, just now,  lower than total of 35 % and then is Skimmer Server(4% 8 band), RTTY Server(15% 8 band) and SparkSDR (6% 9 band)  plus many more  ham, air and weather program up and running. During RTTY contest never more than 50-55 %.

When you feel that you have nothing to do please complete with receiver no 10  so all band between 1.8 - 28 MHz  is covered!

73 de Sven SM6FMB

[PA3GSB]

Alan

I think brute force is not always the answer....

http://www.kiwisdr.com/

Look for review document.... in the document a beter approach is described.

But than we need to change the gateware

Iam looking into it also for radioberry.

73 Johan

PA3GSB

[Joe LB1HI]

The first half with the window open and the second with the window minimized

Hi Alan, :-)

According to your suggestion. One SSB band not FT8. Spark SDR window visible. And the second part on the processor's life chart show the level when the Spark window is reduced to the taskbar. When the SparkSDR window is minimized, CPU usage is reduced from 37 percent to 10 percent
Gateware 9 RX 384 kHz b.w. screenshot The first half with the window open and the second with the window minimized.

It would be nice if some small PC or SBC could be left in the camping-site cabin  . (Or in a shack, because the remote is also another floor of the same building, another room or terrace in the garden.)

Tests of the first candidate, which is PI4, look promising.

(Jetson nano would make sense if CUDA cores were used for decoding and not a basic main processor that does not give an advantage compared to PI4)

Low power SBC 24/7 can also perform control functions for anything that cannot be contained in gateware. 

     At the moment, we have accepted by Steve pull request from Matheew for GP 7 and we do not know yet how much the second MCP23008 will be used in the future).
    For twenty years I have done an external solution for Icom for toggle between antenna`s during RX. This solution is very practical. But with SDR and remote SDR  it is eventually another story.  I

For example : In connection with the idea of ​​remote if the HL2 and control software could contain a button on the software front panel it would be perfect. 

But if for what ever reasons it is not possible to apply all control elements to the second MCP23008 (HPSDR protocol compliance) then a small SBC together with HL2  whether in a shack or in a remote location could play such a role. And at the same time be a skimmer decoder.

For this purpose there are many solutions like Domoticz, Home Assistant, OpenHab I use Domoticz and I am satisfied. And the last one I found and tried is very easy to configure and use applications controlling GPIO pins from the mobile phone "RaspControll".

Basically my writes is answer to Steve's question about behavior of the new gateware version  with 9 RX.

Operation on computers or laptops with medium performance is rather obvious assured of  so I decided to test it on PI4 which is recently the subject of interest and discussion lately.

Everything else came about by the way.

So having HL2 remote (cabin, another room, terrace, another floor of the house) for work ssb and CW controlled from a PC that is with us using software such as SparkSDR, Consloe V3, Quisk. OpenHPSDR, and additionally skimming for PSK Reporter with the help of Spark SDR 24/7 by SBC installed in close proximity to HL2. 

73, Jozef
lb1hi

[Alan Hopper]

Hi Jozef,

it looks like there is an issue with gpu not being used by spark on your pi 4 causing some of the high cpu load, what linux distro are you using?  The real problem is that decoding wsjtx modes is cpu intensive as they are designed to be used one at a time on a reasonable computer, trying to get affordable sbcs to decode lots of modes over lots of bands is tricky even if Spark were perfect and  used 0% cpu.  

73 Alan M0NNB

[Joe LB1HI]

Hi Alan,

One more screenshot with six receivers running on 71p2 cicrx gateware.

visible window = 54% receive; decode peaks 98%

minimized window = 31% receive; Peaks 86%

Small traffic because Monday and I test on a small indoor loop antenna.

I am using OS Raspbian GNU / Linux 10 (buster)

[Joe LB1HI]

New test 7RX in
In  /boot/config.txt force_turbo switched off (commented)
58% CPU usage with visible SparkSDR 
31% CPU usage with minimized SparkSDR window. 

73, Jozef

[Alan Hopper]

Hi Joe,

it might be worth playing with the various GL desktop options in raspi-config advanced.

Steve & all,

just wondered how important it is to maintain compatibility with this mode, we could reduce network load by removing the mic data, reducing bit depth (both the lower decimation and 12 vs 16 bit adc mean 24 bits is over the top) and increasing the packet size.

73 Alan M0NNB

[PA3GSB]

Alan

How much data is necessary for decoding per band?

It is a  waste to send 384Khz and throw away a lot of samples....maybe achievable in a special P-1 version.

I must say i like the kiwi architecture... still need time to go through docs and sources.

73 Johan

PA3GSB

Op dinsdag 19 mei 2020 10:44:22 UTC+2 schreef Alan Hopper:

[Matthew]

This is very interesting stuff and certainly seems like an interesting technical challenge. Regarding deviating from protocol 1, if it had a new device ID I don't see a problem. My reasoning behind this is the HL1 and HL2 both have the same device ID and I recently broke something in linhpsdr for the HL1 but it took me a while to figure this out. In the end I set something to discriminate based on gateware version. Other software won't look at the HL2 extended bits in the discovery packets, one would imagine if a device ID isn't recognised it won't connect to it. So I'm apprehensive about devices that behave differently but appear as the same device.

I have a casual interest in monitoring to Jovian storms. I wrote some software in the past that predicted when a storm was due to start and started recording IQ streams. I've never got around to porting this to the HL2. 10 x 768 kHz receivers would allow nice coverage in the 18-35 MHz region to monitor this.

73 Matthew M5EVT.

[Graeme Jury]

Hello Matthew,

This is what has been applied to piHPSDR

ID=6 version < 40 ==> HL1

ID=6 version >= 40 ==> HL2

I would not be surprised to find those discovery been ported to linHPSDR but would pay to check.

73, Graeme ZL2APV

--
You received this message because you are subscribed to the Google Groups "Hermes-Lite" group.

To unsubscribe from this group and stop receiving emails from it, send an email to hermes-lite...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hermes-lite/36bb1850-153c-4f2b-af62-6454c66a17d2%40googlegroups.com.

[Joe LB1HI]

Hi Alan,

Thank you, so it brought the expected results.

Choosing a G2 driver helped.

With the original non-GL dektop driver, the CPU utilization was 50% even when no receiver was running. SparkSDR window running but not on.

The use of the desktop driver OpenGL desktop driver with fake KMS  the CPU utilization  was only 6%

73, Joe
lb1hi

[Alan Hopper]

Hi Johan,

I've not looked at the kiwi much but got the feeling that it effectively only has 4 receivers. For most modes you only need 3k or so of bandwidth so you could do more decimation in the fpga and return a narrow band for every mode and band you want to decode, and maybe separate data for the display, however decimation and more receivers is expensive in fpga terms so there is an increase in fpga size needed. The spark approach of sharing wider receivers between modes reduces fpga requirements and once you are decoding lots of modes within a band, the network bandwidth cost effectively reduces. For most bands 96 Hz covers the commonly skimmed digi modes and 48kHz is enough on many.  I've not played with Steve's new receivers much and am not sure how much of the bandwidth is useable, It would be nice to use less than 384k but the gains are not that big as the decoders are the real cpu hogs ( when Spark has the GL driver:) ) There have been other approaches suggested like dft, sparse ffts and alternatives to cics or memory based cics. I have wondered about arranging the filters in the fpga in a tree structure so the early high speed ones can be shared between later ones.

I just find it amazing that the little HL2 is now a hair's breadth away from covering all band simultaneously.  Once you cover the bandwidth of all ham bands you can then make a really nice shared sdr with almost unlimited number of virtual receivers, at that point it may make sense to send narrow band audio and display data to the client like the KiwiSDR does.  Having the sdr directly connected to the cpu does let you hide this extra bandwidth and do the next level of decimation on the cpu, this might be a neat feature for the radioberry.

73 Alan M0NNB

[PA3GSB]

Alan

Tnx for you answer; around 3K is sufficent.

Iam aware of additional fpga resource usage for lowering the bandwith.

I did once make a special implementation for wspr.

https://github.com/pa3gsb/Radioberry-2.x/tree/master/SBC/rpi-3/special/WSPRBerry

I think you pointed to this solution:

http://cseweb.ucsd.edu/~schulman/docs/mobisys19-sparsdr.pdf (https://github.com/ucsdsysnet/sparsdr)

Making a simple fpga solution is to send all the sample via the 1G ethernet port and do the processing on a GPU is maybe for max skimming the best solution; iam not aware of hardware prices.

The solution Steve made is working very good! Real nice to to do skimming using my small loop antenna.

Is good to list the alternatives with the advantages and disadvantage.

Tnx again.

73 Johan

PA3GSB

Op woensdag 20 mei 2020 17:19:36 UTC+2 schreef Alan Hopper:

[Steve Haynal]

Hi Joe, Alan and Group,

Joe, I am very impressed you can get that much out of your Raspberry Pi!

For now, I'm thinking of just staying with the current protocol even though it is wasteful. That way any software can at least use this at 384kHz. Maybe in the future we can change... I did do a quick back of the envelope calculation and the processing gains from 76.8MHz decimated down to 384kHz only require an additional 4 bits so 16 bits should be sufficient in the future. (Maybe someone can check this calculation.) I did spend some time the last few days with numpy and scipy better understanding the aliasing that occurs in the CIC filters given both steps of decimation, both filters as well as the external LPF. No surprises but I do have a better understanding of the aliasing. With this current 9 receiver gateware, there are no aliases larger than -100dB down in the middle 60kHz, -80dB down in the middle 110kHz, and -65dB down in the middle 150kHz. I would call the middle 100kHz to 150kHz safe, but you definitely want to center these receivers and avoid using the edges.

73,

Steve

kf7o

[Steve Haynal]

Hi Matthew,

I agree we should use a new device ID for any gateware variant that deviates considerably from protocol1. Your Jovian storm monitoring sounds interesting. Any details or pointers that you can share?

73,

Steve

kf7o

[Steve Haynal]

Hi Johan,

If all you want to do is demodulate the traditional amount of bandwidth like HF radios have been doing for decades, then 3kHz is sufficient. In fact, for waterfall and wide displays that are only for visualization, you don't need to send continuous data, but just enough data periodically so that the display is updated frequently enough to please the eye.

But consider the post from Sven on this thread. He is a true skimmer. He is running 45 virtual receivers in SparkSDR. This averages out to 5 per band. Plus on top of that he is running CW skimmer which is outside of SparkSDR if I understand correctly. When you want the entire CW segment, RTTY, JT9, FT8, WSPR, RTTY and who knows what else, 3kHz is not sufficient. You'll need closer to 100kHz per band.

The advantage of having an inexpensive 1Gbps ethernet port on the HL2 is that we have better opportunities to trade-off where computation can occur: in the FPGA or in the PC. In one extreme, we could send raw 12-bit samples at ~70MHz to the PC for all processing by the PC (I think this will fit with jumbo UDP packets as 12*70e6 is less than 1000Mbs), to spending resources in the FPGA to decimate down to 3kHz so very little work is required by the PC. The gigabit interface allows for the full spectrum of possibilities.

73,

Steve

kf7o

[W. Jozef]

Hi Steve, I think there is some misunderstanding. between 2% and 6%  utilization of the processor was with the Open GL driver when the Spark SDR window was open but the "on" button was not pressed.

However, earlier with the original non-GL driver processor utilization was 50%.
 50% already before pressing the "ON” button. That is, before starting the receiver/s.
it was already when jus only  the SparkSDR GUI window was displayed.

 

I can measure  utilization of the processor depending on the number of receivers. But for FT8 or FT4 and similar emissions, it won't have much meaning. Because the more important moment is when the processor must decode what was received. Then comes the "real test of its possibilities" those decode pick`s hungry for computing power.

But the change of desktop driver from non-GL standard to OpenGL and through this reduce CPU utilization from 50% to ¨6% has already given a significant benefit also to the moment of decoding. Because decoding has much more computing power at its disposal. (much lower starting point)

However, before changing the desktop driver, half of the CPU's computational capabilities were already taken by displaying the SparkSDR GUI

73 Jozef

lb1hi

[Joe LB1HI]

Here are SparkSDR screenshots sequentially from GUI alone and then from one to nine receivers. The screenshots show how many receivers are on and how many percent is  the processor utilization.

This is SSB emission without decoding. As I wrote earlier, this is not meaningful for the digital emission skimmer.

As an add-on, a fairy-tale. Processor utilization by PiHPSDR with two receivers.

73, Jozef

lb1hi

[Steve Haynal]

Hi Joe,

It looks like all the SparkSDR receivers were opened on the 20M band. This means SparkSDR will open 10 virtual or slice receivers since they can share the bandwidth from one hardware receiver. To force SparkSDR to open hardware receivers, you'll need to open receivers on different bands. That would make an interesting experiment. Both the CPU and network demands will go up. You may find difficulty going beyond 4 or 5 receivers.

73,

Steve

kf7o

[Steve Haynal]

Hi Joe,

Sorry, I didn't look closely at the last picture, and you indeed have receivers open on different bands. I think that many hardware receivers at 384kHz is actually quite impressive for a Raspberry Pi.

73,

Steve

kf7o

On Wednesday, May 20, 2020 at 6:19:20 PM UTC-7, Steve Haynal wrote:

Hi Joe,

[Joe LB1HI]

Hi Steve,
Hi Steve,

I compared the skimming of FT8 emissions on Raspberry Pi 4 on five receivers between the new 7p2 and the “old” 6p4 gateware (on a larger number of receivers it didn't make sense because PI4 couldn't decode properly). On the "old" gateware 6p4 processor utilization was 12% lower. But it's probably logical because the sampling rate at 6p4 was 96 kHz.  On the new 9p2 sampling rate was  (forced/ have to be) 384 kHz.

Decoding was also more effective on 6p4 gateware. But it was only 2/3 of the decoding level achieved by SparkSDR on another Windows 10 PC.

The conclusion is that RPI4 is most suitable for cooperation with HL2 with SSB and CW. On the other hand, for now, they are only limited possibilities for skimming digital emissions. Where the limitations is to fewer bands at a time. (Looking at the results, I think no more than a 4 band at once in FT8 )

However, full use of the new capabilities of 9 bands with decoding at once is possible and works perfectly on other more powerful PCs.

73, Joe
LB1HI

[Joe LB1HI]

Yes, exactly it was on different bands.   ( not a problem :-)  ) 

Joe

[Alan Hopper]

Hi Joe,

Spark cpu on rpi  use should drop over the the next few months as I tweak some things and there are oportunities to tweak decoding. Parvel's red pitya decoder reduces the cpu use by removing some of the decimation and filtering that can be done in the radio i.e. it takes .c2 files like the wspr decoder does.  Spark can produce these .c2 files with no significant extra cpu. It would be a fun project for someone to compile this so Spark can use it, it hovers half way down my todo list so I never get to it, my plan was to set it up on github and use CI/github actions to compile for all platforms (winx86 winx64 inux64 linuxarmhf macos). It would also be nice to treat the decoders as servers so they don't need to be repeatedly launched which consumes some cpu. If we did create decode servers these could then be placed on the network so you could use a number of rpis.

All this being said I think multiband multimode decoding on the rpi is asking a bit too much of it for now.

73 Alan M0NNB

[Matthew]

Hi Steve,

I've had to remind myself of the details of this. I wrote it around 7 years ago to use with a FunCube dongle. I wrote it using Python2.x and PyQt. Python libraries made it easy to get the Jupiter ephemeral data, so I just needed to port the code here to make the predictions. I've just found that someone else has done something in a very similar manner to me here. I never got around to writing up anything I did really, but it was nothing clever.

This was the website that started my interest in this:

https://radiojove.gsfc.nasa.gov/observing/sample_data.htm

You want to find nice clear slots away from amateur bands and other HF transmissions in the upper HF bands. Then you can produce some plots like this. I was using matplotlib to do this. I got the impression that the wider frequency coverage you could get the better, see here. So lots of receivers in quiet slots in the upper HF bands is ideal. To be honest, it is probably doable with the current 384 kHz receivers in the gateware discussed in this thread.

There are also theories that storms can be detected in the VLF region, so it would be great to monitor the VLF bands at the same time as the upper HF bands. This would be perfectly possible with the HL2 (acknowledging vlf hardware mod requirements).

My work I've been doing with the application to sit between a cluster of HL2s and SDR software has a lot of code that would be useful for a dedicated HL2 Jovian storm program. Alas, too many projects and ideas and I must focus on one at a time. I imagine diversity rx would very much help with listening to Jovian storms.

73 Matthew M5EVT.

[W. Jozef]

Hi Alan,

 

Yes, you remember how I asked you about Jetson NANO CUDA GPU`s. We talking and considered some of the powerful SBC`s eg.  Odroid  N2 etc.
 Due to various aspects and popularity fell on Raspberry PI4. So, it's his to be examined for kidneys and liver hi.

Notwithstanding the tests with PI 4, Hermes lite is already a fully functional KF transceiver.

 If HL2 in future receives a few additions, in functionality then finality will meet the needs and requirements for contests DXer`s. Needs we know from high-end transceivers and other commercial additional solutions in the form of add. boxes.

The same applies to remote operation. (another room, different floor the same building, garden terrace or camping cabin ). Remote via LAN/WAN, works, and is in the process of modifying, customization and  improvement.
It's nice to see gateware developers, control software developers and users as one team.

73, Jozef
lb1hi

[Steve Haynal]

Hi Alan and Group,

I am trying to squeeze the most out of some older x86 remote machines running Linux. With the new 9 receiver gateware, I am seeing many ep6 errors in SparkSDR. After increasing UDP buffers:

https://medium.com/@CameronSparr/increase-os-udp-buffers-to-improve-performance-51d167bb1360

And playing with nice and ionice:

https://www.tiger-computing.co.uk/linux-tips-nice-and-ionice/

And turning sound and resampling off per receiver, I have the ep6 error rate down to 1 to 3 per minute with 9 384kHz SSB receivers. This takes ~205% of the 4 CPUs (400% max) in this machine. 

But when I switch these 9 receivers to FT-8, I start seeing many ep6 errors (5-10 per second) when JT9 decode kicks in. The SparkSDR usage drops to below 200% and the JT9 take most of everything that is left. Changing the decode depth to 2 didn't seem to help. I found in one overloaded setup with many ep6 errors per second that FT8 was still decoding but very degraded and few spots. On my better machine I never see many ep6 errors.

Any suggestions on how I can improve this situation? Ideally I'd like to dedicate ~250% CPU to SparkSDR and only allow JT9 to use another 100%, spread out over time. Maybe processor affinity? But each JT9 decode is a new process?

73,

Steve

kf7o

[neiljw...@gmail.com]

Hi Steve,

I am trying to do much the same thing, using 2xHL2 with different antennas feeding a Corei3/2 core, 4 thread machine (Linux Mint) and

a CoreI5/4 core, 4 thread machine (Windows 10).

I am not convinced that the 9RX setup is doing any better than the 6RX at maximising the number of FT8 spots I can collect.

I used to typically get into the top 10 for number of reports in the PSKReporter stats running with 2x6RX Gateware, but I'm not managing

to do so with the 2x9RX. Not an exact comparison as conditions are different this time of year, others are changing their setups, etc.

I found I had audio resampling on - it seems to do this by default with a new set of RXs and each one has to be turned off manually.

Turning it off helped a bit with the EP6s, as you found. I must have a look at the UDP buffer and nice settings.

I see a few EP6 errors on the more powerful Windows machine. Minimising the display helps with CPU usage but then can't see the EP6s!

I see a lot more EP6s on the less powerful Linux machine - more than 100/min. Minimising the display doesn't seem to help here.

I'm thinking of reverting to the 6RX setup and trying another few days of running with this.

73,  Neil  G4BRK

[Alan Hopper]

Hi Steve and Neil,

I'm surprised that turning audio resampling off has any effect, when the audio is muted, the default for the built in digimodes, none of the audio code runs and it is the same as selecting none for the audio device, if it is making a difference something is wrong.

I shall add a control to reduce the number of simultaneous decoders, it currently is the number of cores -1 (or 1 for a single core), maybe the jt decoder has started to use more threads.

I have something in the pipeline to speed up the down conversion from 384k.  I'm in the middle of a ui tidy up and making the analog modes more useable, once this is done I'll have another look at the perf here.

73 Alan M0NNB

[neil whiting]

Hi Alan,

OK on the audio resampling - I was just eyeballing the EP6 rates so wasn't very scientific,

maybe something else happened on the machine at the same time.

Nice to have more explanation of when it is used. Does this mean it is just for resampling the

speaker output?

On reflection the best way to test performance would be to run both HL2s from the same antenna

via a hybrid (active antennas so some gain ahead of the RX). With one using 6RX and one 9RX.

I'll try to set this up when I get a moment.

73,  Neil  G4BRK

[Alan Hopper]

Hi Neil,

yep the resampler only affects audio sent to the speaker or virtual audio cable, it is to allow for differencies in the hl2 clock and the soundcard clock, it is also used to control latency by gently adjusting buffer sizes (not sure that is working as well as it once did). The audio sent to the wspr/ft/jt decoders does not pass through it ever with the built in modes.  It is a crude linear interpolating resampler optimised for low cpu, powersdr has a more sophisticated one but uses more cpu and I can't hear any difference on usb.

I've realised Spark is unusual as far as software goes as the virtual receiver feature allows you to put unbounded load on the computer, I think I'll add an indicator to show when you are getting close to capacity and decodes start to be dropped. Some people see ep6 errors at this point, others just loose decodes and some see the display pause or slow down, it seems to vary from computer to computer.

73 Alan  M0NNB

[Steve Haynal]

Hi Alan and Neil,

I confirm that the resampler makes no difference for FT8. It was just one of the variables I had in the mix.

What I did find works well is to create a shield using 1 CPU and then assign what I think is the SparkSDR network thread to this shield:

** Start SparkSDR and configure maximum receivers to start seeing ep6 errors

** Do

ps -efL | grep SparkSDR

and note the process ID for the thread with highest utilization, usually just a little more than half way down the list.

** Setup a shield with one processor. In my case below it is CPU 3

sudo cset shield -c 3

** Assign that thread process ID to the shield

sudo cset shield -s -p <PID>

This clearly gave me the best results. I played around with assigning all threads to the shield, multiple CPUs in the shield, isolcpus and taskset, wrapping JT9 so it is launched with a lower priority, etc., but this was by far the best, and I see only 1 to 2 ep6 errors every 10 minutes on a J2900 processor.

73,

Steve

kf7o

[Alan Hopper]

Hi Steve,

that's interesting. It reminded me that I have a long standing issue on linux with a call to set thread priorities failing, I shall have another look at that.

73 Alan M0NNB

[Alan Hopper]

Hi group,

I've been pondering this a little, there are a few things I can do to tweak the performance of spark but it is getting well up the slope of diminishing returns. I shall add some settings to control the number of simultaneous decoders and look at priorities. Coming at it from another angle reducing the number of udp packets would help, we could go for larger packets, even the 1536 (or whatever the non jumbo max is) would help and getting rid of audio data gains a bit more. Of course this may end up using more fpga and limit the number of receivers which would defeat the object.  I do have some other perf tweaks in the pipeline but need a good chunk of clear time to finish them.

73 Alan M0NNB

[neil whiting]

Some results from FT8/4 skimming with different setups.

I'm happy to try other combinations to get a better idea on what is happening.

73,  Neil  G4BRK

Hermes-Lite 2 with CISC (9RX) Gateware

Running 9 Rxs at 384kHz sampling seems to put quite a strain on the network and computing resources. Does this impact the number of reports when skimming?

Setup

Single G4HUP Eprobe antenna feeding a hybrid which feeds 2x HL2.

SparkSDR 2.0.1.9

RX	gateware	PC	Ant
HL2b6	69p4 _6rx	Core-i3/Linux Mint	G4HUP Eprobe
HL2b8	71p2 _9rx	Core-i5/Windows 10 pro	G4HUP Eprobe

20200621 Run ~2200-0600 = 8h

Both running the same set of 6 bands, 6 virtual RXs skimming FT8 and 4 skimming FT4

Yellow highlight shows the best performer on a band/mode.

Band Mode	6RX spots	9RX spots
80m FT8	18409	18176
40m FT8	53040	54822
30m FT8	27906	24558
20m FT8	31301	27228
17m FT8	2872	3377
10m FT8	327	372
80m FT4	84	82
40m FT4	6693	7044
30m FT4	122	87
20m FT4	11612	9986
PSKReporter reports/countries	7174/111	9159/112
EP6 total/rate	0	852 (1.77/min)

20200622 Run ~0900-0600 = 21h

Both running the same set of 6 bands with 6 virtual RXs skimming FT8 and 4 skimming FT4. In addition the 9RX had 3 additional bands skimming FT8 but on frequencies where there wasn’t any, to give more CPU load but not affect the PSKReporter figures

Band Mode	6RX spots	9RX spots
80m FT8	28796	27797
40m FT8	85224	86155
30m FT8	50719	42148
20m FT8	95277	81956
17m FT8	35470	38793
10m FT8	24235	24355
80m FT4	207	197
40m FT4	9777	9851
30m FT4	678	559
20m FT4	39668	32157
PSKReporter reports/countries	20419/129	23981/125
EP6 total/rate	14 (0.67/hr)	65712 (52/min)

Not sure what to make of this. Report figures from SparkSDR clearly show more overall from 6RX, but PSKReporter gets more from 9RX. This is a similar pattern to the previous case with less EP6.

For next try, swap the gateware so the 9RX stresses the Linux machine.

RX	gateware	PC	Ant
HL2b6	71p2 _9rx	Core-i3/Linux Mint	G4HUP Eprobe
HL2b8	69p4 _6rx	Core-i5/Windows 10 pro	G4HUP Eprobe

20200623 Run 1600-0600 = 14h

Same RX set

Band Mode	6RX spots	9RX spots
80m FT8	27051	21958
40m FT8	89328	63816
30m FT8	50347	40548
20m FT8	60242	42487
17m FT8	28007	18825
10m FT8	11183	8316
80m FT4	133	105
40m FT4	7328	4574
30m FT4	1458	686
20m FT4	31829	17444
PSKReporter reports/countries	19880/126	15134/121
EP6 total/rate	154 (11/hr)	359389 (427/min)

Apply shield to the Linux machine as suggested by Steve in post on 20 June:

> ps -efL | grep SparkSDR

I found the thread PID 31097 in column LWP

> sudo apt install cpuset

> sudo cset shield -c 3

> sudo cset shield -s -p 31097

Try again with Linux machine running 9RX

RX	gateware	PC	Ant
HL2b6	71p2 _9rx	Core-i3/Linux Mint	G4HUP Eprobe
HL2b8	69p4 _6rx	Core-i5/Windows 10 pro	G4HUP Eprobe

20200624 Run 0800-2200 = 14h

Same RX set

Band Mode	6RX spots	9RX spots
80m FT8	8541	8711
40m FT8	38492	37934
30m FT8	25762	26641
20m FT8	79183	90885
17m FT8	43489	41887
10m FT8	31097	30329
80m FT4	124	130
40m FT4	1536	1550
30m FT4	1093	1129
20m FT4	37282	33143
PSKReporter reports/countries	18294/108	15446/109
EP6 total/rate	94 (7/hr)	216 (15/hr)

20200624 Run 0800-0700 = 23h

Band Mode	6RX spots	9RX spots
80m FT8	30054	30329
40m FT8	89948	87836
30m FT8	45483	48704
20m FT8	102913	118440
17m FT8	47757	45874
10m FT8	33577	32762
80m FT4	228	230
40m FT4	4272	4062
30m FT4	1242	1278
20m FT4	45767	40505
PSKReporter reports/countries	27228/132	21518/131
EP6 total/rate	148 (6/hr)	368 (16/hr)

There is inconsistency between spot numbers reported by SparkSDR and PSKReporter.

Band Mode	6RX spots		9RX spots
	SparkSDR	PSKReporter	SparkSDR	PSKReporter
80m FT8 + FT4	30282	1792	30559	1469
40m FT8 + FT4	94220	6628	91898	5035
30m FT8 + FT4	46725	3518	49982	3182
20m FT8 + FT4	148680	10617	158945	8310
17m FT8	47757	3476	45874	2657
10m FT8	33577	2281	32762	1802

In terms of spots reported by SparkSDR the numbers are not too different between the 6RX and 9RX, with either able to provide the best result on a band.

However on PSKReporter the 6RXsetup significantly outperforms the 9RX on all bands.

Conclusions

Not yet quite clear what is going on here.

Firstly I can confirm Steve’s results with shielding the most critical SparkSDR thread. This reduces the EP6 rate.

Generally it seems:

- if the EP6 rate is high then that SparkSDR instance reports significantly less spots than one where the EP6 rate is low.

- if the EP6 rates are low there is some balance between the instances with either able to win on a specific band.

However the 9RX instance seems to send a lot less reports to PSKReporter even with a low EP6 rate.

[neil whiting]

Sorry, looks like it was too big and got truncated.

Now as an attachment.

73,  Neil G4BRK

[Steve Haynal]

Hi Neil,

Thanks for the interesting analysis. Alan once told me that PSKReporter has some set of rules about how spots are counted. Something like a spot for a specific call sign is counted once in a 15 minute interval even if it is reported multiple times. This could explain higher spot numbers in SparkSDR versus PSKReporter. Thanks for confirming that the thread shielding helps with EP6 errors.

It is a mystery why the 9RX has fewer overall spots on PSKReporter than the 6RX. I've noticed that Red Pitaya users near me often have a higher number of spots even though I have spotted more countries. I think there is some subtle tuning to work best with PSKReporter. For example, maybe PSKReporter throttles the data accepted from any one reporter, so the number of unique spots gets reduced because the entire amount of data is not accepted by PSKReporter. Maybe SparkSDR should only send unique spots every few minutes. Or maybe given the higher network load with 9RX, the data to PSKReporter is not going out or going out too late to be accepted by PSKReporter.  

73,

Steve

kf7o

[Alan Hopper]

Hi Group,

thanks for the experimentation with this, I've got a few tweaks done for the next release (not yet released), there is an option to restrict the number of cores used for decoders and an indicator of decodes dropped due to delay to see if this is the issue.  I'm trying to improve the packet reordering incase some of these ep6 errors are out of order packets but I suspect they are simply dropped ones under high cpu peaks. I've had a couple reports of the waterfall freezing with the 9rx gateware, I wonder if this is the watchdog triggering when it all gets too much for the cpu, maybe the watchdog should be longer or defeat-able for extreme skimming. Spark tries to set high thread priority for some of the key tasks  but this fails on linux for a normal user,  It might be worth trying to give it permissions to set priorities.  

73 Alan M0NNB

[Alan Hopper]

Hi Group,

In testing on a number for pcs I found one that was giving me far more ep6 errors than the others, I finally discovered this was due to a cpu heatsink completely blocked with dust and the pc operating in some throttled mode as a result.  I'm sure this is not the case for most but worth checking cooling etc if doing heavy skimming.

73 Alan M0NNB

[Alan Hopper]

Hi Group,

I've spent a bit of time looking at the ep6 errors. I did find one pc had some process that was causing a very short cpu peak every 5s, it was too short to appear high in the resource monitor list but ep6 errors dropped once killed, sadly I don't know what it was as I killed a few with a pause between but I think it took longer to stop than my pause.  Increasing the rx buffers setting in the nic advanced settings seemed to help.

On an old I5, spark uses 15% cpu for 9 384k receivers but the 'NT Kernel & system' and the 'system interrupts' processes increase by 17% so the system is doing more work handling the udp than spark processing them.  The figures are much the same using Boost ASIO and RIO for the udp comms. I think the long term solution is bigger udp packets.

The ep6 error count is a count of whenever a received packet's sequence number is not what it should be relative to the last packet. This means it is not a count of the number of dropped packets, when stress testing I noticed that when errors start there are often block with 10's or even 100's of packets missing so comparing ep6 counts on different systems might not give the full picture i.e. one could be dropping single packets and another could be big blocks.  I've not seen any swapped packets over the last few days.

I can't fully explain Neil's pskreporter results but I'll describe what I belive happens with psk reporter and spark.  The spot numbers for each virtual receiver are a count of every single transmission decoded. Psk Reporter only counts a report submitted by a callsign if there have been no other on that band and mode for the last 20mins ( from memory, may be slightly different). Spark tries to respect the psk reporter bandwidth by only uploading spots it believes psk reporter will accept, the number sent is shown in the spark title bar.  Typically Psk Reporter will reject a few spots and it is possible some packets are lost so the Spark total will be a bit less than the PskReporter total and both will be very much smaller than the receiver totals.

These ep6 errors with lots of packets missing are probably particularly bad as far as decoding goes, I shall look at how best to pad the data to cope better with these.

73 Alan M0NNB

[Alan Hopper]

Hi Group,

I identified the app causing my 5s cpu peaks and ep6 issues, it was mysql notifyer.

73 Alan M0NNB

[Steve Haynal]

Hi Alan,

Thanks for finding this issue. 

I currently have a HL2 setup with 9 receivers skimming FT8 on 80M-10M. If you look at the pskreporter spot stats, I'm just below N6TV, another ham in my region who uses a Red Pitaya. Over the last 24 hours I have spotted 99 countries to his 93. In general, I think there is more DX on my spot maps. He is also spotting on 160M, but not enough spots to make up the difference. I don't understand why his spot count is higher and suspect there is some difference between how Red Pitaya and SparkSDR reports spots.

73,

Steve

kf7o

[Alan Hopper]

Hi Steve,

when I last looked (a long time ago) the rp uploaded all spots and let psk reporter reject most of them, this is not the approach reccomended in the psk reporter docs so I don't really want to do this, I'll ponder an option to allow this for testing . Maybe the psk reporter reject rules have changed slightly, I shall have a look again.

I'm also looking at the linux options to increase the system udp buffer sizes. The real trick seems to be to prevent cpu spikes as the os is doing what it is allowed and meant to do in dropping udp packets when things are maxed out.

Ultimately I'd like more control over the wsjtx decoders and the ability to run them across the network to spread the load.

73 Alan M0NNB

[Steve Haynal]

Hi Alan,

The higher spot counts by RP could very well be an unintentional bug in pskreporter exploited by RP. SparkSDR could very well be the better citizen.

I downloaded logs for the last 24 hours in ADIF format for kf7o (HL2), n6tv (RP) and ve7cc (RP). I then looked at how frequently pskreporter recorded spots of the same strong call on a single band for an extended period of time. There were some interesting patterns.

VE7CC spots of K4YT

053200

054300

055400

067030

062630

062100

070730

072000

N6TV spots of K4YT

053300

054400

055700

061200
062530

063530

065100

070400

071500

KF7O spots of K4YT

053245

055415

061545

063714

065815

071944

Notice that there is always at least 20 minutes between spots from SparkSDR. The RP records spots more frequently, sometimes with just 11 minutes between spots of the same call at the exact same frequency. I looked at more data than just shared above, and in general I never saw more than ~4 spots for a call/band per hour for RP. For SparkSDR it was ~3 for call/band per hour. You can see in the SparkSDR reports that sometimes the time does not align to 15 and 45 seconds, but is 14 or 59. I saw that frequently but never in the RP data. With the RP data I often saw a small change in frequency, 7.075166 or 7.075165, but the frequency seldom drifted with SparkSDR reports. Maybe this small change in frequency causes pskreporter to count more spots?

This was a very active station so was probably actually spotted 1 to 2 times a minute. If the RP is flooding pskreporter with spots as you say, then it looks like pskreporter is pruning this down to 4 call/band allowed per hour. It also looks like SparkSDR is waiting at least 20 minutes between reports for the same call/band. I suspect that if you decrease this to 12-15 minutes (4 or 5 spots call/band per hour) then the SparkSDR numbers would look more like the RP numbers but still not flood pskreporter. 

Here is what I found on the pskreporter dev site

 

The data consists of the calling callsign (at a minimum). Highly desirable extra fields are the frequency, signal to noise ratio and intermodulation distortion. Note that each callsign should be reported no more than once per five minute period. Ideally, a callsign should be reported only once per hour if it has not 'changed'. Precisely what constitutes a change is left to the discretion of the developer, but the goal is to minimize the number of database records! A change might be a move to a different band.

I think 1 hour is too long and will not allow for Apples-to-apples comparisons. Maybe you can make some of these settings user adjustable in SparkSDR?

[Alan Hopper]

Hi Steve,

great, this is really useful, it looks like psk reporter is not as strict as the the docs suggest. It is a huge amount of data it deals with so maybe it takes some short cuts in its measurements.  I'll tweak my rules a bit and restart the conversation I had with Phil (a long time ago) about the optimum approach.

I've had good results limiting the number of cores used by decoders to reduce the peaks and ep6 errors, on my old I5 it seems that one core will just do 1 mode over 9 bands.  I've been waiting for an Avalonia fix for a mac problem but if it does not appear in the next couple of days I'll post a win and linux version.

73 Alan M0NNB

[Steve Haynal]

Hi Alan,

It would be interesting to hear what pskreporter Phil has to say. It just seems wrong that RP should be able to flood pskreporter with reports and be rewarded with artificially higher spot counts. Maybe Phil can/should reject all spots from a reporter if they do this. I'd like to see some sort of standards put in place so that all numbers on pskreporter are independent of particular software, so that fair comparisons can be made.

73,

Steve

kf7o

[Philip Gladstone]

The deal with the Red Pitaya is that that code reports all callsigns that it sees transmitting, whereas the WSJT-X code only reports callsigns that it sees calling CQ (and which have a locator). This latter constraint means that non0standard callsigns will never be reported by WSJT-X. Having said that, I think that WSJT-X is changing to report all CQ callsigns. 

There are (unfortunately) lots of different ways of getting stats out of pskreporter -- and they are often slightly different to each other. Once of the problems is around false decodes -- there are a suprising number of these -- where only a single reporter reports a callsign in a period of time. I try and filter these out as they are often "impossible" callsigns. There are around 1000 of these callsigns reported every hour. A recent sample:

+-------------+

| callsign    |

+-------------+

| MH5ZXG      |

| S68UCE      |

| GL9NIP      |

| JJ5SGG      |

| BH0KQV/R    |

| 0O1ACV      |

| EA2EMO/P    |

| R89HJA/P    |

| R35HIL      |

| GH5OKX      |

| AB0OHC/R    |

| SR7LVH/P    |

| I8CZM       |

| 8N2QA       |

| ML5NJF      |

| VI5LXU/R    |

| 1J6RZB      |

| JT7KIZ/P    |

| AH1WTF      |

| BO2DSW/P    |

| 5J3WOD      |

| TE8AI       |

| U85TUC/R    |

| PY2EDU      |

| GD9WNH/R    |

| 6V2OMF/R    |

| N14VFO      |

| J0FZP       |

| 7K8GMP      |

| IQ9SNQ      |

| K1JDT       |

| 6K0LFE      |

| J08USN/R    |

| D43UII/R    |

| W4BGG       |

| 8L6MGR      |

If you tell me exactly which statistic you are interested in, then I'll figure out how it is calculated.

Philip

[Steve Haynal]

Hi Phillip,

Thanks for the response. I am interested in the "Top Monitors by reports over last 24 hours" statistic on your pskreporter page. If SparkSDR prunes all spots and only reports a call/band/mode spot once every 20 minutes, but the RP is reporting more frequently, will the RP end up with a higher spot count in that category? This appears to be the case from the ADIF logs I downloaded and examined from pskreporter. I can provide more data based on these logs if you like. What is the minimum time between reported spots of the same call/band/mode such that pskreporter will consider it a new spot in that category? I think SparkSDR should report spots as frequently as possible so that they are counted, but not so frequently that they cause unnecessary noise for pskreporter and are not counted.

Thanks again for pskreporter.

73,

Steve

kf7o

[Alan Hopper]

Hi Steve, Phil and group.

if you use the built in decoders in Spark it does report all signal types (not just cq) but if you use wsjtx externally via vac its rules then apply. It does look like my deduping is over cautious. What time limit would guarentee maximum recorded spots without too much wasted traffic?

I need to revisit this code anyway to add a bit more info to the spots.

Thanks Phil for all your work on PSK Reporter, it is a large part of why people use my software.

73 Alan M0NNB

[Jim Ancona]

Is https://github.com/softerhardware/Hermes-Lite2/blob/master/gateware/bitfiles/stable/20200529_71p3/variants/hl2b5up_cicrx/hl2b5up_cicrx.rbf the gateware to use to try the 9 receiver variant?

If I want to use it with SparkSDR, I would configure the radio with a 384k sample rate and up to 9 receivers? Is there anything else I should know?

Thanks!

Jim N1ADJ

--
You received this message because you are subscribed to the Google Groups "Hermes-Lite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hermes-lite...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hermes-lite/228ad7b0-e906-434f-9708-bdec52bc04f1o%40googlegroups.com.

[neil whiting]

Hi Jim,

Yes, that's right. Make sure your network can handle the traffic, I see ~200Mbps with 9 RXs, which needs a Gbit ethernet link.

73,  Neil  G4BRK

To unsubscribe from this group and stop receiving emails from it, send an email to herme...@googlegroups.com.

[Jim Ancona]

I've been running since yesterday afternoon and it's working well. PSKReporter shows 1943 transmitters, 24267 reports in 130 countries over the last 24 hours. That's running 9 receivers with 160M at night and 10M in the daytime (although 10 hasn't been open). I'm running with a direct gigabit ethernet connection. My i7 Linux laptop seems to be keeping up okay, but I see many (more than 250k over 18 hours) ep6 errors. Is there anything I should be doing about that?

Thanks!

Jim N1ADJ

To unsubscribe from this group and stop receiving emails from it, send an email to hermes-lite...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hermes-lite/84688699-88f2-44bb-8593-6a82cac8d06fo%40googlegroups.com.

[Alan Hopper]

Hi Jim,

the next release of spark has a bit more control over the number of cores used for decoding, this can help with ep6 errors. It is worth checking that with nothing running there is not anything creating big cpu spikes, even short ones can upset the udp network traffic.

73 Alan M0NNB

To view this discussion on the web visit https://groups.google.com/d/msgid/hermes-lite/84688699-88f2-44bb-8593-6a82cac8d06fo%40googlegroups.com.

[Jim Ancona]

Thanks Alan! I'll be on the lookout for the new version. Is there a Linux tool you'd recommend to look for short CPU spikes?

Jim N1ADJ

To unsubscribe from this group and stop receiving emails from it, send an email to hermes-lite...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hermes-lite/7166b9fb-7421-45d4-bd2c-8f372e9dcdfbo%40googlegroups.com.

[Alan Hopper]

Hi Group,

I've just released a new version of SparkSDR here http://www.ihopper.org/radio/previews.htm . It allows you to reserve a number of cores, effectively limiting the number used for decoding. You need torestart spark after changing this.  It also reduces the deduping to 15 mins, as I write this I have the top spot count on 40m so I think it is working. Antenna and snr data are also now uploaded.

73 Alan M0NNB

[neil whiting]

Hi Alan,

Thanks for these improvements, I've just moved to the new version.

It sets reserved cores to 1 by default - does this mean 1 core is reserved for SparkSDR and the rest are available for decode?

The Linux machine running 9RX gateware was still showing ep6 errors, using 1 reserved core I got 300/min, using 2 or 3 I got ~60/min.

I could reduce the 1 reserved core run to a low rate by using Steve's shielding technique as before.

The decode rate shows 100% on all 3 machines. What does this actually mean?

I'll leave them running for a while and see how things go.

I am impressed by your 40m results!

Thanks/73,

Neil  G4BRK

[Alan Hopper]

Hi Neil,

yep the default of 1 means all but one core(unless only one) can be used for decode, this is what it did before. Increasing this value should give spark more space but with the trade off that decodes might be dropped, the decode rate will drop to <100 if decodes start to be dropped.  I still need to get to grips with the thread priority stuff on linux.

I don't really understand why I get good results on 40m (new version aside), it is probably the only hf band I've not made an antenna for, it is currently using an inverted L tuned for 80m.

73 Alan M0NNB

[Angel Andres]

Alan

The latest version working very well. I have run it here with the latest 6RX gateware variant for the last 24hrs.

As I type, circa 31K spots in 24hrs with a LZ1AQ loop / HL2 in an urban QTH .

EP6 159 counts ~ I do not have a feel if this is a big number. I am running on a G5400 coffee lake with 16GB of RAM - CPU peaking @ circa 80%. 

73 Angel

M0HDF

[Alan Hopper]

Hi Angel,

good to know it is working. I've lost top spot on 40m to Sven, it was always going to happen but I made the most of the 12hr headstart I gave myself:) Your error count does not look to be a problem.  There is more tweaking to be done on this but I wanted to get a release out that worked with the latest wsjtx.

73 Alan M0NNB

[Angel Andres]

Hi Alan

Thanks for the feedback on the ep6 counter. I cannot compete with your 14K spots on 40m here. I am pleased to have got to the top 10ish [nr of spots] with my limited setup here :-)

I see Sven has a 'bit of kit' in the 'game'.

73 Angel

M0HDF

[Philip Gladstone]

For those of you that don't know, you can mouse-over the callsign in the leaderboard (on https://pskreporter.info/cgi-bin/pskstats.pl ) and it will tell you what hardware/software is in use by that person.

Philip

[Alan Hopper]

Hi Philip,

This feature has tricked me in the past because it appears to come and go, only yesterday I worked out it only shows on a fresh page reload and not if you select a band.  Is the leader board accessible as data? It would be fun to display in Spark. In my latest version I'm now deduping at 15 mins vs 20 and it seems to result in more spots. I'm keen to not send more than needed so please shout if too many are rejected.

73 Alan M0NNB

[Ronald Nicholson]

Hi

What's an ep6 error?  How is it detected?  What does it indicate?

Thanks,

Ron

n6ywu

------

On Sunday, May 17, 2020 at 8:16:47 AM UTC-7, Alan Hopper wrote:

Hi Group,

whilst this firmware should work fine with Spark most of the time there are some situations where you could end up using the bad part of the spectrum near the edges of the receiver, selecting favorites should be fine but tuning across a wide band with multiple virtual receivers  could result in some of them using the edges.  I'll support it fully in future releases.

Neil,

that does sound like a network issue, are you seeing ep6 errors?

73 Alan M0NNB

[Alan Hopper]

Hi Ron,

It is basically a count of breaks in communication between the radio and the pc. I wish I had never put the ep6 name on the screen, its history dates back to the early hpsdr radios that used usb and End Point 6 was the usb end point used for the receiver data (ep4 is the wideband data), the same packet structure is used for the network connection so the names persisted. Each udp network packet is sent with a sequence number and the ep6 error displayed in spark is incremented whenever the sequence number is not 1 greater than the previous one received. Network devices and computers are allowed to drop udp packets or swap them in order, this tends to happen when things get busy or the network is faulty. In normal use on a wired network you should see very few, the 9rx firmware make the task harder as all receivers have to run at the max bandwidth and it consumes nearly 1/4 of a gigabit ethernet bandwidth, at the same time people are often running many 10s of virtual receivers and decoders in spark which does not leave much cpu to deal with the network traffic.  

73 Alan M0NNB 

[Steve Haynal]

Hi Group,

I am trying to understand why there are more unique spots in the .adif logs I download from pskreporter even after filtering to call/band counted once per 20 minute window versus the reported leader board counts. I ran an interesting experiment to show this difference on 12M and posted to the PSK reporter group: https://groups.google.com/d/msg/psk-reporter/JqMh8gN2QO0/ymwlp_PdAgAJ

73,

Steve

kf7o

[Alan Hopper]

Hi Steve,

I'm running a version with the 15 secs added to the report time to psk reporter on 20m, I'd hoped to release it but had updated other stuff and hit an Avalonia issue that I think is now fixed but need to test further.  Another optimisation tool is the decode depth for the ft/jt modes, level 2 uses significantly less cpu than level 3. I noticed M0WID is making HL2 look good on 20m.

73 Alan M0NNB

[Steve Haynal]

Hi Alan,

Thanks for your work on this. I am looking forward to the updated version. I now have a better PC that can keep of with the 9 receivers as well as a better antenna at my sister's house. I'll play around with the decode depth.

73,

Steve

kf7o

[Joe LB1HI]

Hi Steve,
Screenshot of CPU usage (E5-2650V2). The peaks reach 86% when jt9 decodes.

It decodes well 9 bands at a time.

73, Jozef LB1HI

On Sunday, May 17, 2020 at 7:22:14 AM UTC+2 softerh...@gmail.com wrote:

Hi Group,

This is a receive only release with 9 receivers meant for all the skimmers and SparkSDR users out there. See here. Use the hl2b5up_cicrx.rbf file found in the hl2b5up_9rx for the recommended ethernet gateware upgrade using SparkSDR or Quisk.

The final FIR filters are removed. Only the 384kHz bandwidth setting will return meaningful data. 192, 96 and 48kHz bandwidth settings will return garbage. Since only CIC filters are present, you will see droop at either extreme of the receiver bandwidth as seen in the picture below. Also, some aliasing at the extreme ends may occur. But there is still almost 200kHz of usable bandwidth in the center. This is enough to cover most data frequencies within a band. In fact, some of the newer SDR RTL I've looked at only applies a decimate by 2 FIR as the final step after CIC filtering, so would consider 192kHz of this usable. 

Big thanks go to John (JKS) KF6VO the principal designer of the KiwiSDR for his CIC filter register pruning code. I think we discussed CIC register pruning on this list 5 years ago, and it has been on my todo list since, but I am just now trying it out. Without the pruning, I can fit 7 receivers, almost 8. With the pruning I can fit 9 receivers and am only 11 LABs short for 10 receivers. The Cyclone IV we use has 1395 LABs, so 11 is just 0.7%! I think I should be able to fit 10 receivers with just a little more work. 10 receivers would allow us to skim all ham bands 160M-10M with just a single HL2.

I've been running SparkSDR with 9 receivers and see no issues with my midgrade AMD Ryzen 3 3200G system. There are 4 CPUs in my system and SparkSDR is taking just 75% of 1 CPU, or ~17% system total. There are some CPU spikes at the end of every 15 second FT-8 decode cycle.

It turns out that 9 receivers pack perfectly in the protocol 1 scheme with no wasted bytes. If we go to 10 receivers, then there will be 16 wasted bytes in every UDP packet. Still, at 9 or 10 receivers I think we should be able to get up to 16MHz total bandwidth with the HL2's 1Gbps ethernet even with the current UDP scheme and the wasted mic bytes. At 9 384kHz receivers, we are sending 3.456MHz bandwidth, less than 25% of what I expect possible with our 1Gbps connection. I'd like to push this limit to see what is the most bandwidth we can reliably send. I'd like to add an extension to the protocol to select higher bandwidth receivers, which comes basically for free with the current setup. This will be especially helpful for CIC-only gateware variants where the edges droop. The easiest bandwidths to generate are 76.8MHz/10/N where N can range from 1 to 20. The current 384kHz is with N at 20. N of 10 would give 768kHz, N of 5 1.536MHz. Any interest from Alan or other software developers? Let me know a handful of wider bandwidths that you'd prefer.

I would appreciate testing and feedback of this release. Does this work for skimming? How annoying is the droop? Is any aliasing at the edges seen? The beauty of the latest HL2 gateware is that you can easily reprogram over ethernet with SparkSDR or Quisk, and now we have 2 images stored in the EEPROM so you have a rescue image if anything goes wrong with a gateware upgrade. So you can give it a try and then go back to 71_p1 when you want to transmit again.

73,

Steve

kf7o

In this picture, the blue arrow is pointing to the FT-8 frequencies near the center of the bandwidth. You can see the 15 second cycles in the waterfall. I live in a very noisy environment. The green arrow shows time when my monitor was off. You can see that removes some spurs in the waterfall. The other spurs are from various power supplies in my neighborhood and are external to the HL2 as they are not seen without an antenna.

 

[Alan Hopper]

Hi Joe,

you should be able to reduce the cpu spikes by setting 'reserved cores' in settings/logging to a high value. You have to exit Spark and run it again for this to take effect.

73 Alan M0NNB

[Joe LB1HI]

Thanks for the suggestions, Alan. I will test different settings.

Do the "Reserve cores" apply to their reservation for the host operating system? or for the  SparkSDR and jt9 decoder?

73, Joe
PS. Earlier I stated that the value of spikes reaches 86%, but purely for information. I am not worried if the value of spikes for the entire processor and individual cores does not reach 100%.

PS. Having already done a few tests, with " reserve cores" higher setting. I noticed that when I sett to 14 cores than on core 0 (zero that is the first) and on 1 there are no signs of load during decoding, but on cores 3, 7, and 11 the CPU consumption reached 100 for about one to two seconds.

[Alan Hopper]

Hi Joe,

I did not pick a very good name for this.  It reduces the number of simultaneous decoders. The maximum number run at the same time is the number of cores (hyperthreading included) - reserve cores.  By doing this the load will be spread over time (as the decodes are queued) in the hope that there should always be cores free for the time sensitive udp processing.

73 Alan M0NNB 

[Joe LB1HI]

Hi Alan,
Many thanks. Now everything is clear.

It is very good that there is such a function because users use different computers with different numbers of cores.  
73, Joe LB1HI

[Steve Haynal]

Hi Alan,

I've updated to the latest SparkSDR and my pskreporter spot count is now pretty much what I'd expect it to be. I'm leading for my region of North America west coast! The reserved CPUs helped, but I was still seeing EP6 errors. I am still using cpu sets to reserve one cpu for the most active SparkSDR thread, and that pretty much eliminates EP6 errors. I see the message "dropped" many times in the terminal where I am running SparkSDR. What does that mean? When using my Python scripts to examine the .adif logs, I still see a few bogus calls:

Bad Call cm88

Bad Call rr73

Bad Call signal73

Bad Call jh44

Bad Call rr73

Bad Call ft4

Bad Call rr73

Bad Call cm95

Bad Call rr73

Bad Call rr73

The count on pskreporter agrees with my Python scripts now to within 1% difference. This is much better than the ~30% difference I was seeing before. Thanks for all your work!

73,

Steve

kf7o