Is an extra CRC check unnecessary if using TLS?

[bilm...@gmail.com]

When I originally built my IOT system I added a CRC to all messages from remotes to make sure bad messages were not processed  by database manager.

Since then I have updated messaging to use TLS and client certificates

Am I correct in thinking my CRC is not needed any more since error checking is built-in to TLS  - or is there still some benefit of an additional check?

[Greg Troxel]

That sounds correct to me.

If you are handling data signficantly after you affix a CRC but before
it is sent via mqtts, or after receipt before it is stored, then this
could make sense still, but if it's merely in memory in a computer that
could corrupt it before CRC, you are probably not getting benefit.

Note that while the mqtts hop from sensor to broker has integrity
protection, and the mqtts hop from broker to data consumer does too, the
broker itself has a window of the data stored in RAM.

But if your computers are corrupting data they are handling, you have
much bigger problems.

My own view is that that mqtt should be over TLS, and then there isn't
any need to worry.

[Andy Stanford-Clark]

TCP/IP ensures that all messages arrive intact, and in the order they were sent.

So you don’t need to add your own CRCs.

It’s TCP/IP that ensures message integrity, not TLS - that’s “just” for encrypted links (and with certificates being confident you’re talking to the right endpoints).

Andy

--
To learn more about MQTT please visit http://mqtt.org
---
You received this message because you are subscribed to the Google Groups "MQTT" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mqtt+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mqtt/2f3d3c27-cffa-44ba-aca4-8cba13a639a9n%40googlegroups.com.

[Greg Troxel]

Andy Stanford-Clark <an...@stanford-clark.com> writes:

> TCP/IP ensures that all messages arrive intact, and in the order they
> were sent. So you don’t need to add your own CRCs.

TCP has a 16-bit checksum. That does not provide robust protection
against errors (1 in 2^16 errors will be accepted). Some data links
used for IP have CRCs, and some don't.

> It’s TCP/IP that ensures message integrity, not TLS - that’s “just”
> for encrypted links (and with certificates being confident you’re
> talking to the right endpoints).

TLS provides confidentiality and integrity both, with keyed hash
functions, which are far stronger integrity protection than a checksum
or a CRC.

[Bill Marriott]

Thanks for the feedback. I will remove the CRC in my next version.

Actually it was a pain to implement it in Python because Dicts have no native order so I had to be really careful to pack and unpack theJSON in the identical order so the CRC did not fail

I will be glad to be rid of that code!.

Associated question:

My message processor (engine) is still subscribing on 1883 (NOT TLS) to  broker on localhost,  though it could connect to a remote broker if scaling required it. I rationalized this because it was easier, local and would be less of a load on CPU.

So, for a subscriber to ALL topics,  should the additional overhead of TLS be a concern ( thus need load testing) or is it likely to be trivial so just do it?

--
To learn more about MQTT please visit http://mqtt.org
---

You received this message because you are subscribed to a topic in the Google Groups "MQTT" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mqtt/AysJJcyqX4M/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mqtt+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mqtt/rmih7jptgan.fsf%40s1.lexort.com.

[Greg Troxel]

Bill Marriott <bilm...@gmail.com> writes:

> My message processor (engine) is still subscribing on 1883 (NOT TLS) to
> broker on localhost, though it could connect to a remote broker if scaling
> required it. I rationalized this because it was easier, local and would be
> less of a load on CPU.
>
> So, for a subscriber to ALL topics, should the additional overhead of TLS
> be a concern ( thus need load testing) or is it likely to be trivial so
> just do it?

Opening a TLS connection requires public-key operations. Once opened,
it's all symmetric crypto. These days most web accessa and almost all
email is TLS. I hear no complaints about CPU time, only about how much
RAM firefox uses.

You should try it but I suspect you will not notice the CPU time. I
strongly advise against turning down security because of a fear of CPU
usage, without having actually measured it.

My own perference is to *not even configure* prot 1883 on the broker.
Even if there is data that you don't consider sensitive, typically one
uses password auth, and that should have protection. Yes, I see you
point about localhost, but I find it simpler to reason about systems
that simply have no cleartext access, period.

[Roger Light]

On Thu, 29 Apr 2021 at 22:40, Greg Troxel <g...@lexort.com> wrote:
>
> Opening a TLS connection requires public-key operations. Once opened,
> it's all symmetric crypto.

This is the really important point. Opening the TLS connection is
relatively expensive, maintaining the connection is much less so. If
you open a fresh connection for every publish that you send, you'll
spend a significant amount of CPU on handshakes. I have seen this
behaviour in the wild, please don't do it!

Regards,

Roger

[Bill Marriott]

---------- Forwarded message ---------
From: Bill Marriott <bilm...@gmail.com>
Date: Fri, Apr 30, 2021 at 11:16 AM
Subject: Re: [mqtt] Is an extra CRC check unnecessary if using TLS?
To: Greg Troxel <g...@lexort.com>

Ok here is a test run.  Graph data is captured every 15 seconds

Python Engine code  and Mosquitto broker are running on same server which is DigitalOcean Ubuntu 20 single CPU droplet

Apache web server is also running but no connections were made during test period.

All  other  MQTT devices are connecting via  TLS 1.2  on 8883, QOS 1

At 10 am I restarted my Engine which subscribes to all topics - causing it to run on port 1883

There is an expected spike in CPU at 10am since my code is restarting and a few messages have stacked up and are then delivered to Engine and processed upon connecting

At 10:30 am I change Engine port to 8883 and restart my code causing it to reconnect using TLS

I see the same spike as the engine restarts and a few messages are queued up and then processed at once

Otherwise,  I see no difference in CPU between 10am to 10:30 (TCP)  and 10:30 to 11am (TLS)

So I am going all-in on TLS and will remove the 1883 listener.

Bill

On Thu, Apr 29, 2021 at 8:21 PM Greg Troxel <g...@lexort.com> wrote:

It would be great to have actual data!  Please post when you believe your measurements