HL2b9 stops after a while

[IV3DXW Claudio]

Hello group,

I'm testing HL2b9 mostly with SparkSDR.

I encountered an issue that prevents me mainly the continuos skimming. 

After a random time, HL2b9 stops to send data to PC.

Just turning off and on SparkSDR "power" button, HL2b9 starts again well for a while.

Here my observations after many trials and changes:

- no relation with gateware build

- no relation with SDR PC software (SPARKSDR, quisk, sdrconsole)

- no relation with gain and overload

- no relation with watchdog disable

- no relation with temperature (board temp around 18C)

- no relation with data transfer on LAN (I use cable, 94Mbits 192KHz of 9RX behaves like 1 receiver 48KHz)

I notice that after 1 days HL2b9 stay switched off, when powered on it can stay skimming for 15/20hours. But after one stop occurs, next is within 20/60 minutes.

Any advise of debug?

I would investigate my hardware since I didn't find any similar problem described here.

I would try some workaround to remote SparkSDR "power" button periodically, if I can find the way.

Thanks

73

Claudio IV3DXW

[Steve Haynal]

Hi Claudio,

My first thought was that the watchdog timer was stopping the radio, but you disabled it. How did you disable it?

My second thought is that there might be something happening with your network. Can you ping the HL2 at first, and then still after it stops with software? What is the state of all 4 HL2 LEDs before the problem and then after the problem?

Are there any clues in the SparkSDR logs? Assuming you are using Windows, they are at:

c:\users\## your name ##\appdata\roaming\m0nnb\sparksdr2\errorlogs

On my Linux machine:

~/.config/m0nbb

It is possible to run SparkSDR with a HL2 for weeks on end. My remote station has been skimming uninterrupted now for 3+ weeks. I find that after weeks on, something seems to bog up the software or system and I see fewer spots. My spot count always seems higher for the few days after restarting everything.

73,

Steve

kf7o

[Alan Hopper]

Hi Steve,

that is the first report I've had that suggests reports drop over time, do the total spot counts per receiver appear to drop or is it just the upload count? I'll try and add some better stats to make it easier to check for this.

Claudio,

sorry no instant thoughts other than the watchdog, I have never actually tested that the disable watchdog feature in recent versions of Spark actually works, as Steve says the logs might just help.

73 Alan M0NNB

[IV3DXW Claudio]

Hi Steve, Hi Alan,

thanks for feedbacks.

I forgot in the problem description one important detail, the stop is equivalent to power outage (network UDP packets drop and all LEDs start blinking), so I checked that my HL2b9 is properly 12V supplied.

Another detail: I'm using static IP to overcome to DHCP issue in my LAN. My LAN is based on Mikrotik hardware.

I found this hint in my research on the groups and I tried watchdog control in SparkSDR Version 2.0.4.8

I gave up the research... so I posted this issue.

Here attached logs of the latest event:

Here my present configuration. Actually 192.168.0.8 is not HL2 but Afedri which is normally idle.

I also observed EP6 counter but I didn't find a clear relation with the problem. Here an example of problem with EP6 raised to only 185.

Thanks

BR

Claudio iv3dxw

[IV3DXW Claudio]

here some details of logs

[IV3DXW Claudio]

here some details of logs

[IV3DXW Claudio]

I made some test to stress watchdog.

I made short disconnection (less than 1sec) of LAN cable at router port.

Disconnection of PC port seems not to trigger a HL2&SparkSDR stop.

Disconnection of HL2 port seems to be the clue: I see in the first cycles the connection recovered but after 5/7 HL2 stops sending permanently.

Now I made direct eth connection between PC and HL2.

I will post results.

73

Claudio iv3dxw

[philip.j.s...@gmail.com]

My Hl2 also stops after a while when talking to sparksdr on windows. I have cleaned up the network, but that didn't help. I'm hoping to figure out how to run a packet trace for a few hours to see if I can spot what goes wrong.... The fact that it can be brought back to life from the PC end indicates that the network stack is still (mostly at least) functional on the HL2.

Philip

[Steve Haynal]

Hi Claudio, Philip and Group,

Claudio, can you please attach the logs you pointed to. I'd like to see the contents. Also, do you see the same problem if your other SDR is disconnected from your LAN?

Philip, have you also tried to disable the watchdog, only available in recent SparkSDR 2.0.3.9 or later? 

My remote station does use a fixed IP and has a direct connection to the PC. I will start SparkSDR on my local network with a hl2b9 and see how long it stays up for me. I usually run Quisk+WSJT-X at home and that stays up for many days. I don't see the problem you both describe.

73,

Steve

kf7o

[Claudio Deltin]

Hi Steve,

logs are attached in one replay 12hours ago (zip file).

I will test again configuration with mikrotik router after validation of direct connection to the PC.

73

Claudio iv3dxw

--
You received this message because you are subscribed to a topic in the Google Groups "Hermes-Lite" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/hermes-lite/vbdxNFsGtLY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to hermes-lite...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hermes-lite/421f0e88-c8f1-4cd7-ad59-b4d989328949n%40googlegroups.com.

[IV3DXW Claudio]

Hi Steve,

no interrupts after 48 hours with direct ethernet connection to PC and static IP address.

So I consider solved my issue with FT8 skimmer.

Hi Philip,

consider to try this configuration.

Best 73

[Steve Haynal]

Hi Claudio,

I'm glad to hear you have a working configuration. The HL2 should not require a direct connection to stay alive on a network for many days. I will still try on my home network to see if there are any problems after several days.

73,

Steve

kf7o

[philip.j.s...@gmail.com]

I did a packet capture and my hl2 stopped. The packet capture showed this at the end. The .135 is the HL2 and the .124 is my windows box. The .68 machine is the dhcp server. I *wonder* if that ARP request actually caused the packet stream to be redirected to the .68 machine. I'll take a look at the gateware later on today. But this is a bit of a smoking gun!  [There were no more packets seen by .124 after this]

[Philip Gladstone]

For some reason the image didn't work: 

To view this discussion on the web visit https://groups.google.com/d/msgid/hermes-lite/c8577a36-3780-4b82-a20c-f0eef9d23825n%40googlegroups.com.

[Steve Haynal]

Hi Philip,

Thanks for the capture. This may well be the issue. I will have to take a close look at the ARP code. Right now all of my radios are setup for synchronous receiver experiments so it may be a while until I get to it.

PSKreporter shows N1DQ-HL as using SparkSDR 2.0.0.9. Is there any reason you are using this very old SparkSDR release? The old release has problems with reporting FT8 spots with the correct time.

73,

Steve

kf7o

[Steve Haynal]

Hi Alan,

I restarted my setup but saw no improvement. I used to be in the top 20 reporters but now I am in the bottom 10 out of 50. It is probably all in my head that spot counts drop over time. Maybe it is just the Christmas lights at my sister's place causing more interference... I'll let you know if I have anything concrete.

73,

Steve

kf7o

[Steve Haynal]

Hi Group,

Just a short update that Philip N1DQ has solved the problem of his HL2 stopping. For some reason there was a bogus/malformed ICMP sent to the HL2 on his network. This caused the HL2 to hang. He has fixed the issue:

https://github.com/softerhardware/Hermes-Lite2/commit/45717535818fd5df674fa686442b0bc5bdb08d27

This fix is in github and will show up in future gateware releases. I'm not sure if Claudio (or others) had the same issue.

73,

Steve

kf7o

[Alan Hopper]

Hi Steve & Philip,

you maybe suffering from your own success:) There are a growing number of HL2s doing multiband/multimode skimming and western Europe seems to be the place to be, I wonder if  there could be other measures/leader boards that give other parts of the world a chance, maybe extra points for distance.

Good to know the stopping has been fixed, I did spend some time looking for issues in spark. Doing all this network stuff in gateware is an amazing achievement, I have massive respect for the OHPSDR people that started it and I guess it is now getting down to the  last few fractional percent of corner cases.

73 Alan M0NNB 

[Philip Gladstone]

Do you have any thoughts on what would be a good metric? I can think of "total miles per day" but that would be a bit of a stress to actually generate... 

It is interesting that the Red Pitaya boards are still at the top of the leaderboard. In my case, I have a RP StemLab (which is the 16-bit board) connected to the same antenna as my HL2 and even though the HL2 is decoding FT8, FT4 and WSPR, it still reports around the same number of spots as the RP. The decoder on the RP does report more spots as it will report messages that it decodes that are not CQ messages. I suspect that that makes the difference. Sparksdr could certainly do the same -- would be interesting to see if that improved the situation. 

I did the query for the last 24 hours, and these are the total distances reported (in miles). You get credit for each distinct callsign reported in each 20 minute period:

DP0GVN,215682320.961995
EA8BFK,130699034.512877
K1HTV-4,117578405.389507
WZ7I,102987643.85112
DL0PF,98471934.4859318
TA4/G8SCU,95884468.3785028
DL0PF-RP,91013388.173169
HB9CQK,89570185.1807091
OE9GHV,81763082.513495
W2PKY,79082037.9102216
K1HTV-RP,77305544.3927467
KM3T,76810282.4422412
OH6BG,71210256.6634875
K9IMM,65528804.1318081
DK0VH,65421769.239522
ZL4YL,62871101.1782432
KK1D,59963565.8130656
VK2AMA,59100314.4288492
OH2BUA,58848253.4087518
EA4GOY,57742661.0228921
VK3KTS,54938072.648828
WE9V,53727766.6004763
ES5Q,52961764.5840893
IV3DXW,51618300.8572481
DL9GTB,50311095.5647974
VK3BDX,48729427.8042865
DJ9PC,47952485.4408376
VR2BG,47815581.7223826
G0KTN,47476877.5070742
CT7ANO,46516043.8319331
SM6FMB,46360764.3934941
AF7KR,46197547.9265617
G4BRK,46194858.1438907
HB9TMC,45227563.6220456
N6TV,44535319.446206
ON3URE,43720087.4925648
AI4FR,42940990.184306
DO2PHS,42318738.9726628
G8DJF,41884447.5894671
G4HRM,41737282.5232227
DD9LH,41593516.675458
NK1I,41494425.4397857
SV8RV,41179004.7846295
NF3R,40550322.1888995
N1DQ-HL,40189494.1864361
KC9KKO,39905343.654993
W1HS,39904622.6908443
4X4MF,39847182.8065355
K6JO,39689020.3798333
W9PDS,39379674.174119
SM7IUN,39059920.5412495

Interestingly, my HL does better than the RP. I suspect that this is due to the WSPR spots...

Europe still seems to be where the action is. I guess that they get sufficiently more reports to compensate for (maybe) a reduced average distance....

Philip

--
You received this message because you are subscribed to the Google Groups "Hermes-Lite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hermes-lite...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hermes-lite/e4bbde1f-10ac-42ac-8cee-5a96b464a401n%40googlegroups.com.

[Alan Hopper]

Hi Phillip,

I'm not well up on contesting, maybe there are some ideas there. Off the top of my head I wonder if there could be a dx measure that either ignored spots closer than a carefully chosen threshold or applied a non linear scale to distance.  Maybe an opposite hemisphere list, but I guess this might have the opposite problem,  could still be fun.  

73 Alan M0NNB

[Matthew]

I don't know too much about skimming apart from picking up some of the semi-competitive conversations on this group. So perhaps my thoughts are flawed by something I don't understand. However...

That is quite a challenge to come up with a metric that removes geographic bias. I have seen numerous complaints about how DXCC is easier to get in Europe on the LF bands (not sure I agree). But then within Europe contesters seem to have their own theories about the best geographic location. However, for DXCC and contests it is a 2 way process, some popular locations will be self limiting by their relatively low "most wanted" status and people not working them. In this case there is no bias, it is purely about density of transmitters. Adding score weighting for distance to stations heard then encourages a DX focus and changes behaviors? You could perhaps adopt the CQWW approach of spots multiplied by DXCC? You could even go a step further and weight each DXCC by the most wanted status for your given country?

I think the interesting technical challenge to rise to would be to figure out where on the globe there was the largest density, figure out the band that gave the most time coverage of this region in a day and target it with a directional antenna on a dedicated (additional) skimmer? Steve's approach of listening in all directions at the same time covers this.

73 Matthew M5EVT.

[IV3DXW Claudio]

I think that a very nice metric would be % how many station I decoded in the last 24h of those have been active transmitters. This would not remove geographic bias but could another interesting KPI.

Maybe something related to S/N of reports could be just an idea to try geographic bias removal. 

BTW. Would very useful to connect SparkSDR to JTalert. I use JTalert in "singleband mode" with JTDX to compare different antennas or radios. JTalert generates a good (local) database of decoded stations.

I tried Gridtracked  with K1RA's SparkSDR2WSJTX-UDP,  but I didn't find how to do the same job of JTalert.

73 Claudio iv3dxw