a new point on the whish list

[Thomas Beiderwieden]

Hi Alan,
I have just completed an update to a script, which connects SparkSDR to Cloudlog (https://github.com/DL3EL/CloudlogBashCat), so that Cloudlog has the correct frequency and mode for a new log n entry.
I access SparkSDR via rigcrtld, unfortunately, not all modes are sent (FT8, WSPR, etc). So I did some guessing with the bandwitdh, that is far from perfect.
Would it be possible to include an information of the used mode in what you sent to rigctld?

Thomas, DL3EL

[ahop...@googlemail.com]

Hi Thomas,

That looks neat, for a while now Spark sends the mode name as shown in the drop down to rigcrtld with the exception of DigiU (sent as PKTUSB) and DigiL (PKTLSB) so I suspect something else is filtering out the unknown modes, I seem to remember there is a quite small hardcoded list of modes in hamlib.  I'm guessing you are connecting via another program, if so connecting direct to spark should give the proper mode names.  If you are connecting direct then I have a bug, but I can't see one.

73 Alan M0NNB

[Thomas Beiderwieden]

Hi Alan,

to get the information, I use rigctl:

rigctl -r $host:$port -m2 f

rigctl -r $host:$port -m2 m

What I get in return for "m" is

"

4900"
for FT8 and 

"CW

500"

for CW, as an example. PKTUSB and PKTLSB is working, however, I have converted them to "HL2-Data" :-)

vy 73 de Thomas, DL3EL

[ahop...@googlemail.com]

I believe that is rigctl filtering the mode out, you should be able to send the same commands direct to spark eg  yourcomputer:51111and get the unfiltered version..

73 Alan M0NNB

[Thomas Beiderwieden]

Hi Alan,

very good point. I have just changed my script, it is a lot easier that way and works like charm. Will do some more tests and then publish it on github.

vy 73 de Thomas, DL3EL

[Thomas Beiderwieden]

Hi Alan, All,

I have now converted the script to Perl and added FLDigi support (to the existing SparkSDR). With this script the current frequency and mode is send to Cloudlog from FLDIgi or from SparkSDR).

vy 73 de Thomas, DL3EL

[ahop...@googlemail.com]

Hi Thomas,

very neat, I should add some links to projects like this to the website.

73 Alan M0NNB

[Thomas Beiderwieden]

Hi Alan,

good idea. Its all on github: https://github.com/DL3EL/CloudlogConnect 

If there would be any chance to get access to the log data, without manual intervention, I could extend the script i have set up for FLDigi to SparkSDR as well. Meaning, that the last log entry would be automatically pushed into cloudlog.

vy 73 de Thomas, DL3EL

[ahop...@googlemail.com]

Hi Thomas,

the next release will send log messages over the websocket feed, I'm just testing some other stuff before I release.

This is an example json log message

{"cmd":"logResponse","qso":{"Id":3,"Mode":"FT8","SubMode":"","Callsign":"M6NNB","Date":"2023-02-09T16:56:01.3368647Z","Band":"20m","Comment":"test comment"}}

The websockets page on the website has links to some documentation and sample apps.

73 Alan M0NNB

[Thomas Beiderwieden]

Hi Alan,

thanks, that looks good. Take your time, I am currently deep in debugging my  LoraAPRS software, which has some problems between different tasks.

vy 73 de Thomas, DL3EL

[Thomas Beiderwieden]

one thing, please include RST

[ahop...@googlemail.com]

I need to read back and collect together all the other requests for logging, I'm not a log user and will be lead by users on what is needed. I assume RST for digimodes is SNR, is there a standard format?

73 Alan M0NNB

[Thomas Beiderwieden]

I do not know, if there is an explicit standard. A line in WSJT-X looks like:

<call:5>PY5JO <gridsquare:4>GG54 <mode:3>FT8 <rst_sent:3>-05 <rst_rcvd:3>-07 <qso_date:8>20230118 <time_on:6>092700 <qso_date_off:8>20230118 <time_off:6>092830 <band:3>15m <freq:9>21.075403 <station_callsign:5>DL3EL <my_gridsquare:6>JO40HD <tx_pwr:3>100 <eor>
I would take it from there

vy 73 de Thomas, DL3EL

[ahop...@googlemail.com]

Hi Thomas, Mike and all,

thanks for the logging help and suggestions. I think I now have a plan, Mike suggested that the import should preserve all fields which makes sense, once that is accepted I realised I might as well make all adif fields available, so the dialog that pops up when you click log will have a small default set of fields but you will be able customize it and add any adif field .  I shall probably change the field names in the websocket feed to match adif.

Thomas, great to hear you have websockets working, I should get you something to test with in the next few days with the changed field names.

73 Alan M0NNB

[ahop...@googlemail.com]

Hi Thomas,

https://www.sparksdr.com/download/SparkSDR.2.0.944.linux-arm.deb should publish log entries to the webservice, they now use the adif field names.  If you want to preserve your old log you need to export it from a previous version of spark and then import into this version.  It should now merge imports and preserve all fields. I still have to add the customization option to the log form.

73 Alan M0NNB

[Thomas Beiderwieden]

Hi Alan,

thanks. Quick question, do I have to subscribe something or is the the log being entry pushed, when I hit the save button? My first WebSocket Script does receive the answer to a command (like get version)  or the constant flow after subscribe spots. Nothing happens, after I pressed "save". I just entered a dummy qso into the log fields...

One thing: the buttons import / export adif are gey and cannot be used.

vy 73 de Thomas, DL3EL

[ahop...@googlemail.com]

Hi Thomas,

that's annoying, It appears something to do with the log database works on windows but not rpi. I'm just trying an updated version of litedb.  Once I fix that it will just send the log entry, no need to subscribe. The buttons are grey as the db has failed.

73 Alan M0NNB

[ahop...@googlemail.com]

Hi Thomas,

I've so far failed to get the database working properly on rpi and it appears litedb development is a little stalled at the moment so I'm going to do it a different way, this should have other benefits but will take a little longer.

73 Alan M0NNB

[Thomas Beiderwieden]

Hi Alan,

thanks für the Info. I will use the time to enhance my websocket script. I have just discovered, that I can include the frequency and mode update to cloudlog easily into that script.

Quick question are you aware of someone who has got sound via websocket running and is *not* using RUST?

vy 73 de Thomas, DL3EL 

[ahop...@googlemail.com]

Hi Thomas,

I did have sound working in javascript but the test/demo on the website does not seem to work for audio so I need to find where I put the working version ( or maybe web sound has changed since it worked).  Nearly there on the database replacement, just need to do some stress testing.

73 Alan M0NNB

[ahop...@googlemail.com]

Hi Thomas,

this is the last test javascript version that I can find https://www.sparksdr.com/sparkservetest2.htm it draws a waterfall and at some point the audio worked but it seems a bit sporadic now, it might have a few pointers, hopefully I can find time to make more of the webservice stuff.

73 Alan M0NNB

[ahop...@googlemail.com]

Hi Thomas,

Actually the audio seems to work as long as you are not on FM as I added stereo for that but never considered the websocket, another thing for the list.

73 Alan M0NNB

[Thomas Beiderwieden]

Hi Alan,

perfect, that was exactly what I was looking for. FM is not so important, I am mostly listening to cw anyway. My problem with remote operations was the sound. I am logging into my RPI with VNC, that has no sohnd. I have tried with mumble, but that introduces a delay, whch makes cw difficult. With this script, the delay is minimal and it works perfect in the background of my tablet. Thanks.

I have updated my websocket script so that it will send the data from all radios configured to cloudlog, which then will automatically put qrg and mode into a log entry (when you log manually).

vy 73 de Thomas, DL3EL

[ahop...@googlemail.com]

Hi Thomas,

that looks great. There is a version here https://www.sparksdr.com/download/SparkSDR.2.0.946.linux-arm.deb that should allow you to test sending spots over the websocket. Still lots to do to polish logging but something for you to test with. I'm very open to adding any thing to make this work well.

On the web audio front it is good to hear it is working for you but I have a feeling there was a better way to do the audio. The stereo fm problem was caused by two things, first it was stereo data and secondly the packet size was smaller, I now convert to mono and have updated the htm to cope with variable packet size, it does not work well on fm (but no browser lockup) and in a future release I probably need to increase the packet size for fm or make it part of the subscribe request.

73 Alan M0NNB

[ahop...@googlemail.com]

Hi Thomas,

I'm wondering if I should include the receiver Id in the log packet I send, would that be useful?

73 Alan M0NNB

[Thomas Beiderwieden]

Hi Alan,

currently I can not think of a use. I take your data, convert them to adif and send them to cloudlog. So there is no need for the receiver id. However, it would be nice to see the real frequency in the data. And if it is not to much work, it would be cool, if you could extend this logging to other modes.

The data are sent ok with the new version, I am just finalising my script.

vy 73 de Thomas, DL3EL

[Thomas Beiderwieden]

Hi Alan,

one thing we should agree on is the name of the fields. At a first glance I was of the opinion, that you use the adif names, but then i discovered, that for the received rst you use rst_received instead of rst_recvd. I can set up a translation, but if you would directly use the adif names, as described here: http://www.adif.org.uk/314/index.htm, I would not need to care for the tags, just put them in to the propper format (already done) and send them cloudlog.

vy 73 de Thomas

[Thomas Beiderwieden]

Hi Alan,

would it be possible to send the QSO Status per receiver via Websocket?

vy 73 de Thomas, DL3EL

[ahop...@googlemail.com]

Hi Thomas,

that is a typo on my behalf, I have changed it to the adif name so it will be correct in the next release. I shall also add the frequency.

I'm not quite sure what you mean by qso status.

I intend to put the log button on all modes that transmit, I wonder if SWL's use logs if so I could just put it on all modes.

73 Alan M0NNB

[Thomas Beiderwieden]

Hi Alan,

thanks.

By qso status I mean the number, right from the time (no qso -1, calling cq 0, etc.). Reason behind, maybe you remember, that I check the propagation every 30min to adapt the 4 slices to the best bands. Normally the HL2 runs 24x7, 95% unattended. Works perfect. Just now, I am doing a lot of qsos, to test the logging and the cloudlog interface. Just a couple of minutes ago, I have discovered an abormality: I had a qso on 40m on receiver 1, then the cronjob fired the band check and it was decided to change receiver 1 to 12m. The qso status was 4, so they were just exchanging 73s. At the next slot for tx (frequency was now 12m, qso started on 40m), sparksdr just silentyl crashed (errorlog is attached). My idea is now, just to check, if there is a qso ongoing, before I qsy.

Regarding the swl, I receive qsl cards frequently from swls, so I assume, they do logging.

Another thing: when you log a qso and the status is ! -1, you have the qso fields completely filled (except the grid, which could be there, ist sent out by the other staion). However, when you miss that and log the qso, just afer the disconnect, the fields are still filled, but the call field is cleared. Is there any specific reason for that? Screenshot is attached

vy 73 de Thomas, DL3EL

[ahop...@googlemail.com]

Hi Thomas,

thanks for the crash report, I shall fix that, it looks like a result of using bleeding edge versions of Avalonia.  I shall ponder qso status reporting, maybe a tx start stop message could be useful, I use this internally to only connect to the PSKR mqtt feed once you tx and time out after a while so as not to hog the service, I guess other uses of the websocket might want to do similar, it gives you a clue that the radio is being used and works on all modes.

I'll put the log button everywhere, I have a few things to tweak like using the proper adif mode and submode and importing frequency to band if it is not there as I use band for the auto respond filter.

I'm excited about this, lack of a proper logging interface has nagged me for ages but I still cannot bring myself to add serial cat:)

73 Alan

[Thomas Beiderwieden]

Hi Alan,

it looks like you are having a busy weekend. My script is running fine and I am testing while having qsos :-)

Logging is essential, I have serial logging on another project, a LoRa APRS iGate & Tracker. With this tiny devices, you have very little to display, so without serial logging it took ages to find an error.

vy 73 de Thomas, DL3EL

[Thomas Beiderwieden]

Alan,

another thing, which I have noticed later, regarding the log fields: when you log the qso, while it is runnng, before the final "73", everything is ok. If you miss that point and log it later, not only the call field is empty, but the rst sent and received is switched., see screenshot.

vy 73 de Thomas, DL3EL

[ahop...@googlemail.com]

Hi Thomas,

thanks for finding that, I think I need to cache the previous qso until it is logged or until another one has completed. I think this will cover both logging before or after completion.  Or I could maintain a list of recent possible non logged qso's so you can click back and log missed ones.

Once it is working robustly I could add auto logging for the modes where it can be determined by sw.

I think I'm going to move the log entry form to popup over the spectrum/waterfall display, this prevents it covering details that you might want to add to the log and makes it much easier to add it to all modes.  Later I might make it a bit smarter and if used on a big screen it could sit in its own space.

73 Alan M0NNB

[ahop...@googlemail.com]

Hi Thomas,

when you had the crash what did your websocket script change, was it just the frequency?  Would you have had the history view visible?

73 Alan M0NNB

[Thomas Beiderwieden]

Hi Alan,

this crash was not related to websocket, as the frequency changing script is older (prior to my research on websockt). It is done via rigctl and nc. I do a setmode together with the set frequency, however, I normally never change the "standard" scheme (receiver 1-4: FT8, 6-9: WSPR, 10-13:FT4). Reason is that I have seen, when you change from WSPR to something different and then back to WSPR the tx ratio is set to 0. This may be ok, but as it cannot be changed automatically, so I try to avoid changing the mode.

Regarding the history view, sorry, I cannot remember.

vy 73 de Thomas, DL3EL 

[ahop...@googlemail.com]

Hi Thomas,

this https://www.sparksdr.com/download/SparkSDR.2.0.947.linux-arm.deb might fix some of the oddities when logging early and late and adds logging to most modes. The proper adif field names should be used and frequency is saved.  The Avalonia version is updated, that might prevent the crash.  I still need to allow customization of the logged fields.

73 Alan M0NNB 

[Thomas Beiderwieden]

Hi Alan,

looks good on first glance. Will do more tests later tonight. One small thing, it was my typo earlier. It has to be "rst_rcvd" and not "rst_recvd" as I wrote.

vy 73 de Thomas, DL3EL

[Thomas Beiderwieden]

Hi Alan,
some more remarks:
o Logentry
- when you are in an FT8 qso and have exchanged RST, are in status 4, but have not sent out RR73, the log fields are empty in the mask (that could be ok)
- after the first RR73, they are filled
- if you do not save the qso, but cancel the mask and wait until the status is -1 and then reopen the mask, the fields are still filled and saved/sent via WS works correctly
- if you have saved a qso and are in the next one and in same status as above (4 and no RR73 sent), the the old data are still in the fields. Imho, that is not ok. At that point, the fields should be empty or should already have the reveicved data
- save as above works
- if you do nothing of the above and wait until the qso is completely finished (status -1 again), than the log fields are completely empty, which is not ok. I think, the fields should be cleared when you start a new qso, not before.

o Logbook
- I have followed your advice to export / import the data to convert them to the new structure. Basically that works. However, my expectation was, that the import would clear the internal database and then import the data. What happens is a kind of a duplicate check, which mixes the data.
- It works, if you export the data, shutdown spark, delete the old db on OS level, start again and import the data.
- during this tests I have seen, that during export the following could happen:
- you select a directory where to export
- click on an exiting file, because you are lazy and only want to extend the filename
- after doing that, the file save will always happen with the original filename, regardless, what you type into the field. Even if you change the directory the file will be saved in the new directory under the "old" name.

Overall, top job, this are all minor issues.

 
vy 73 de Thomas, DL3EL

[ahop...@googlemail.com]

Hi Thomas,

thanks for the feedback, I have fixed the field name for the next release ( I had it right everywhere except for the ws export!).

Some notes on the import export, the previous versions of spark use a different database so old and new versions can coexist but won't share data except by importing. Only the new version does merging and duplicate filtering, The old version would just add everything on import so to do a clean import back into the old system deleting the .db file would work.  

I shall tweak the ft8 logic further, I now have a test mode that emulates conversations so I can test stuff like incomplete qso's without upsetting real people.  Maybe if the previous qso has been logged I could populate the fields as the data appears so only a single click is required, if you are slow on logging the previous one, once logged then populate the fields received so far.

The psk and cw modes will copy from the callsign box that is filled if you click on something that looks like a callsign.

73 Alan M0NNB

[Thomas Beiderwieden]

Hi Alan,

sounds cool to have this test mode. Does it run completely independent from hf or do you just inject your fake qso into the data comming from hf? I have seen a problem, for which I have not found a scenario to reproduce it. What happens is, when you call cq or are in a qso, sometimes SparkSDR misses the slot for tx and keeps quiet. See attachment "missing_tx 1". In that case I was calling cq.

My gut feeling is, that happens when there is a lot to decode and the decoding process has not been finished or has not reached a certain threshold wihtin the first 5s of the tx slot.

Also interesting are the other two screenshot. We are not only missing the tx slot, but sending out RR73 without having even send an RST. I do not understand, why G3PXT send his 73 back, without having got his RST.

This happens some times per day.

Cool with copying the call with a double click. This works in PSK, but I have not managed to copy something in CW.

vy 73 de Thomas, DL3EL

[ahop...@googlemail.com]

Hi Thomas,

the test mode injects spots from a list into whatever else is decoded but if I run using the HL2 emulator I can test with no rf or sometimes I just set the drive to 0.

I have had another report of tx sometimes not happening but never got to the bottom of it, I think I can use the testmode to catch it by setting up a scenario where it should tx every second frame and detecting a fail.  There is some logic to prevent more than one virtual transceiver transmitting at once that may be failing or as you say it might be slow decoding.

I'll check CW.

73 Alan M0NNB

[Thomas Beiderwieden]

Hi Alan,

thanks for the feedback. Small addition: I just saw that the"clear spot" button in PSK has no obvious effect.

vy 73 de Thomas, DL3EL

[ahop...@googlemail.com]

All it should do is clear the spot count in the mini representation of the virtual receiver in the toolbar at the top of the window, if this is not happening it has become broken.  There are a group of users that like being able to reset these counts, what I keep meaning to do is add some better stats to achieve the same thing e.g. spots in the last hour, 24hours, countries in the last hour etc, This could be done as an add on over the websocket interface.

73 Alan M0NNB

[Thomas Beiderwieden]

Sorry, my fault. It works as designed. I had understood this button same as the "clear" button in cw, as I had never noticed / used the "clear spots" button in psk or in other digi modes.

vy 73 de Thomas, DL3EL

[Thomas Beiderwieden]

update on the cw issue (does not copy call into log field). It works, as long as the call is in the first row (!). In other rows, it does not copy. In PSK it works in other rows as well.

vy 73 de Thomas, DL3EL

ahop...@googlemail.com schrieb am Mittwoch, 1. März 2023 um 09:32:58 UTC+1:

[ahop...@googlemail.com]

Hi Thomas,

There may be a case to make the clear work the same on cw and psk, they are features that users have requested and I've never looked at the bigger picture.  Interesting that clicking only works on the first row (btw only a single click is needed) I shall investigate.

73 Alan M0NNB

[ahop...@googlemail.com]

Hi Thomas,

working on the missed transmissions, not fully got to the bottom of it as I so far have failed to recreate it, but realized I had never implemented priority decoding for the foreground/operating receiver. I have now done that so operation should be better if running lots of receivers.  A transmission triggered more than 40% into the cycle is delayed to the next but one slot,  would that fit with your missed transmissions?

73 Alan M0NNB

[Thomas Beiderwieden]

Hi Alan,

I am not 100% sure, when the transmission is missed. I just looked at it for some time today and could not see anything which was suspisious. The traffic was more or less the same,, tx started in most cases at at slot begin +5s, somtimes at slot begin +0s, sometimes there was no tx. So, if the version with the priority to the foreground receiver is ready, I will be happy to test it and see, if there is a change.

vy 73 de Thomas, DL3EL 

[Thomas Beiderwieden]

Hi Alan,

small update re the logbook fields. If you do not manage to hit the save button during QSO and press later, sometimes the logbook fields are still filled, sometimes not. I have tried several times yesterday and was not able to find some reason. That is a difference to what I wrote before (logbook fields are never filled, when you save later).

Another small thing, what I have seen: you may remember, that my "SparkSDR Controller" sends qsy commands to SparkSDR, when hf conditions change. So when you have a qso eg on 40m, do not save, the congtroller switches 40m to 12m and then you save, you will see 12m as the band in the logbook mask. When you save, the wrong band (here 12m) but the correct frequency (here 7.07xxx) are sent via websocket. 

vy 73 de Thomas, DL3EL

[ahop...@googlemail.com]

Hi Thomas,

I'm rewriting the logging and now have a queue of unlogged QSO's so last and current can't get mixed up, this should also fix the cleared/not cleared issue. The current qso log now updates live so you only need to press save. I'm just working my way through all possible causes of tx failure.

73 Alan M0NNB

[Thomas Beiderwieden]

Hi Alan,
that sounds really good.

Quick question: when I do qsy at SparkSDR triggered through my script, would I run into timing issues, when I Do it to quickly?
Worst case currently is changing 4 receivers at once.

I have seen situations, when after that one or two receivers do not receive anymore. They need a manual qsy of 1 khz or so to be reenabled.

vy 73 de Thomas, DL3EL

[ahop...@googlemail.com]

Hi Thomas,

can you point me to your script that causes the problem so I can test, I wonder if it is a case of not finding the optimal set of real receiver  frequencies when many change.

73 Alan M0NNB