Txt Msg decoding - can more error protection/correction be used?

[Brian Morrison]

I've been using FreeDV a fair bit lately, mostly for receiving due to a
combination of poor propagation and station modifications, and one
thing I notice is that a lot of the signals heard here on 80m have lots
of fast fading and often float in and out of intelligibility with SNRs
in the 0-2dB region in 700D mode.

At the same time the encoded text message is often next to unreadable,
it is garbled a lot and bad decodes overwrite good decodes very
quickly.

I wondered if a better method could be used, if intelligible voice can
be recovered at BERs of 10% or more then it should be possible to use
more robust coding for the data portion of the frame even if the
overall bit rate has to reduce.

Comments?

--

Brian G8SEZ

[Mooneer Salem]

Hi Brian,

If you turn PSK Reporter support on (Tools->Options), it changes the encoding of the text field to use Golay(24,12) plus a hidden CRC8 to ensure we don't send incorrect data to the service. In my experience this seems to help a little bit but there is definitely room to investigate improvements. The challenge is that the text field is a very low bitrate data stream (on the order of low single digit bytes per second), so switching to a different FEC system could very well add significant latency. I'd definitely appreciate suggestions if anyone has some.

Thanks,

-Mooneer K6AQ

--
You received this message because you are subscribed to the Google Groups "digitalvoice" group.
To unsubscribe from this group and stop receiving emails from it, send an email to digitalvoice...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/digitalvoice/63e0c2e3fe67a64dad8d40cbe3b32ae50fe1aa73.camel%40fenrir.org.uk.

[Brian Morrison]

On Wed, 2021-07-21 at 12:37 -0700, Mooneer Salem wrote:

Hi Brian,

If you turn PSK Reporter support on (Tools->Options), it changes the encoding of the text field to use Golay(24,12) plus a hidden CRC8 to ensure we don't send incorrect data to the service. In my experience this seems to help a little bit but there is definitely room to investigate improvements. The challenge is that the text field is a very low bitrate data stream (on the order of low single digit bytes per second), so switching to a different FEC system could very well add significant latency. I'd definitely appreciate suggestions if anyone has some.

OK, I will try your suggestion and see how it goes.

On the wider point, I suspect that it might be worth having a fast and slow mode available and allow people to experiment. I see 2 or 3 regular stations on 80m, I can now recognise them by frequency and clock offset, but rarely is there a clear decode of the text message.

Quite fun playing with this, and the latest merge into master of freedv-gui is excellent. A definite improvement.

-- 

Brian  G8SEZ

[Brian Morrison]

On Wed, 2021-07-21 at 12:37 -0700, Mooneer Salem wrote:

If you turn PSK Reporter support on (Tools->Options), it changes the encoding of the text field to use Golay(24,12) plus a hidden CRC8 to ensure we don't send incorrect data to the service.

I have tried this now, but the end result is that there is no text message shown at all with the sort of stations I hear on air with fairly low SNR and quite a few modem resyncs.

I will see if the saved files show any text with the PSK reporter option enabled.

-- 

Brian  G8SEZ

[Mooneer Salem]

Hi Brian,

The PSK Reporter functionality requires both sides of the contact to have it enabled. In addition, it only transmits the callsign and nothing else, so I wouldn't expect anything in the free text field to appear. Hope that helps!

Thanks,

-Mooneer K6AQ

--
You received this message because you are subscribed to the Google Groups "digitalvoice" group.
To unsubscribe from this group and stop receiving emails from it, send an email to digitalvoice...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/digitalvoice/4fc74c1db8bcf5defd9c68eced3f167e5f7054cd.camel%40fenrir.org.uk.

[Brian Morrison]

On Thu, 2021-07-22 at 11:08 -0700, Mooneer Salem wrote:

The PSK Reporter functionality requires both sides of the contact to have it enabled. In addition, it only transmits the callsign and nothing else, so I wouldn't expect anything in the free text field to appear. Hope that helps!

That explains why the txt msg box is greyed out when I enable the reporting. And yes, someone pointed out the need for both sides to use it, so it seems I am stuck with guessing who people are when they give callsigns that are unintelligible.

I think this is something that should be thought about, but I don't have any great ideas beyond maybe having two fields, one for callsign and the other for free text. The callsign could be sent with extra FEC while the free text could be sent with less FEC or perhaps have everything split into a sequence of 3, with call, free text 1 then free text 2, and stitch the last 2 together in software. Yes, slow and with extra latency, but the important bit is good protection of the callsign I'd say.

On a different note I tried the current freedv-gui master version and the ms-optimization build, the latter is much faster to start up and enable the modem so it certainly gets my vote for being merged into master, it claims to be 40 commits ahead I think.

-- 

Brian G8SEZ

[Mooneer Salem]

Hi Brian,

I'll think some more about the PSK Reporter encoding as well and see if there's a way to encode both (or at least make the callsign bit more reliable).

As for ms-optimization, do you see similar results to master if you enable single threaded mode in Tools->Options under the Modem tab?

Thanks,

-Mooneer K6AQ

--

You received this message because you are subscribed to the Google Groups "digitalvoice" group.
To unsubscribe from this group and stop receiving emails from it, send an email to digitalvoice...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/digitalvoice/81a638fe57bea0e571154fb1ecc5daa8753d8e4c.camel%40fenrir.org.uk.

[Brian Morrison]

On Thu, 2021-07-22 at 23:38 -0700, Mooneer Salem wrote:

As for ms-optimization, do you see similar results to master if you enable single threaded mode in Tools->Options under the Modem tab?

There is no question that there is a significant speed-up with multi-threaded mode, at the cost of a slight CPU use increase. I imagine that it makes the start-up faster because it's running more threads/CPUs at once but in any case it avoids the "did I click the icon?" question after a few seconds without the program opening which the current master branch exhibits.

I've just built and installed another rpm based on ms-optimization with the latest git updates from today, I will keep you informed how it goes. HF conditions have been fairly poor in recent days so it takes time to assess changes.

-- 

Brian  G8SEZ

[Mooneer Salem]

Hi Brian,

I may have missed it in a previous email but can you let me know what specs your machine has (RAM, CPU, etc.)? I was actually noticing the Raspberry Pi 4 that I have here taking longer in multi-threaded mode vs. master, so I'm curious to know under what circumstances that mode works better.

Thanks,

-Mooneer K6AQ

--
You received this message because you are subscribed to the Google Groups "digitalvoice" group.
To unsubscribe from this group and stop receiving emails from it, send an email to digitalvoice...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/digitalvoice/246b403b30a42554fe31449806162f6325a8f914.camel%40fenrir.org.uk.

[Brian Morrison]

On Fri, 2021-07-23 at 12:38 -0700, Mooneer Salem wrote:

Hi Brian,

I may have missed it in a previous email but can you let me know what specs your machine has (RAM, CPU, etc.)? I was actually noticing the Raspberry Pi 4 that I have here taking longer in multi-threaded mode vs. master, so I'm curious to know under what circumstances that mode works better.

I have an i7-9700 3GHz 8-core CPU with 64GB of RAM, bought in April 2020 as a do-it-all machine. It has quite a lot of grunt, which is why I went for something that relatively expensive as somewhat future-proofed. It has 1.5TB of SSD/NVMe storage plus more spinning rust. M/board is an ASUS H370Mplus.

I may not be the best person to test FreeDV function on more modest systems.

-- 

Brian  G8SEZ

[Brian Morrison]

On Thu, 2021-07-22 at 20:12 +0100, Brian Morrison wrote:

Yes, slow and with extra latency, but the important bit is good protection of the callsign I'd say.

The GB2RS news is read fairly locally to me on Sunday mornings using FreeDV 700D mode on 80m, and the reader G6WPJ has the PSK Reporter option enabled. I tried enabling at my end, the result was not what I expected with quite a lot of corruption of the displayed callsign despite an SNR of up to 7dB at times with long periods with 4dB or higher. It didn't appear to be reported by me so maybe the extra hidden CRC prevented that, one station in South Wales significantly further from the newsreader did report his callsign once on PSKreporter this morning.

-- 

Brian  G8SEZ

[Mooneer Salem]

Hi Brian,

I was thinking a bit more about this and it seems LDPC(112,56) might be equivalent to Golay(23,12) in terms of latency. The PSK Reporter logic currently encodes the callsign and CRC8 using a six-bit character set; this means that LDPC and Golay would be able to encode 56/6 = ~9.33 characters and 12/6 = 2 characters * 4-5 blocks = 8-10 characters, respectively. Likewise, encoding the characters + parity would require 112 bits for LDPC and 23*5 = 115 bits for Golay. At ~25 bits/sec transfer speeds per https://github.com/drowe67/codec2/blob/master/README_freedv.md, that would mean approximately 5 seconds either way. 

Of course, the challenge is that the existing LDPC functions in Codec2 seem to return raw bits for encode but require floats for decode. While we could construct floats from the received raw bits just prior to decode, I'm not sure if or how much it'll worsen LDPC's performance on typical HF paths. I'll have to think about this more for sure.

Thanks,

-Mooneer K6AQ

--
You received this message because you are subscribed to the Google Groups "digitalvoice" group.
To unsubscribe from this group and stop receiving emails from it, send an email to digitalvoice...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/digitalvoice/7b23553884684d3b63bc13ad19cb4f7ce2d2c815.camel%40fenrir.org.uk.

[Brian Morrison]

On Fri, 2021-08-13 at 02:26 -0700, Mooneer Salem wrote:

Hi Brian,

I was thinking a bit more about this and it seems LDPC(112,56) might be equivalent to Golay(23,12) in terms of latency. The PSK Reporter logic currently encodes the callsign and CRC8 using a six-bit character set; this means that LDPC and Golay would be able to encode 56/6 = ~9.33 characters and 12/6 = 2 characters * 4-5 blocks = 8-10 characters, respectively. Likewise, encoding the characters + parity would require 112 bits for LDPC and 23*5 = 115 bits for Golay. At ~25 bits/sec transfer speeds per https://github.com/drowe67/codec2/blob/master/README_freedv.md, that would mean approximately 5 seconds either way. 

I think most of us can wait 5 seconds for a correctly decoded callsign :) Maybe even in a contest...

Of course, the challenge is that the existing LDPC functions in Codec2 seem to return raw bits for encode but require floats for decode. While we could construct floats from the received raw bits just prior to decode, I'm not sure if or how much it'll worsen LDPC's performance on typical HF paths. I'll have to think about this more for sure.

Well, there's no mad rush for a solution, it would be bad to mess up what is already working. Thank you for considering the suggestion.

-- 

Brian  G8SEZ

[Mooneer Salem]

Hi Brian,

No worries. It's definitely a good idea to consider it carefully as the Golay implementation is already out in the wild and there needs to be a good reason to break compatibility. As 1600 (which uses Golay) can work down to 4dB SNR, a good first step would probably be to see if the current PSK Reporter implementation can work down that low on other modes and if not, figure out if there are any bugs in the decoding process. No use switching encoding schemes if LDPC would ultimately have the same bugs, right?

Thanks,

-Mooneer K6AQ

--
You received this message because you are subscribed to the Google Groups "digitalvoice" group.
To unsubscribe from this group and stop receiving emails from it, send an email to digitalvoice...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/digitalvoice/ceee92ce72b9929a158ca194a97813c079f0655e.camel%40fenrir.org.uk.