Custom types / Lossy serialization

[Joscha Feth]

Hi,

this might be an odd question, I searched the forum a little but couldn't find an answer.

I am trying to use nanopb for the message payload over LoRa and seem to have problems to get my payload small enough.

I previously used https://github.com/thesolarnomad/lora-serialization which was written for exactly that.

However protobuf and especially nanopb seems to be an awesome fit when it comes to me wanting to read that same data in a different environment again.

A data set of the following:

• unixtime
• latitude
• longitude
• uint8
• 2x uint16
• temperature
• humidity

is 21 bytes with the lora-serialization lib and about 38 (variably up to 45, but minimum seems to be 38) with nanopb:

message LogEvent {
    uint32 unixtime = 1;
    Location coords = 2;
    SolarData solar = 3;
    float temperature = 4;
    float humidity = 5;
}

message SolarData {
    AdsGain gain = 1;
    uint32 solar1 = 2 [(nanopb).int_size = IS_16];
    uint32 solar2 = 3 [(nanopb).int_size = IS_16];
}

enum AdsGain {
  GAIN_TWOTHIRDS = 0;
  GAIN_ONE = 2;
  GAIN_TWO = 4;
  GAIN_FOUR = 6;
  GAIN_EIGHT = 8;
  GAIN_SIXTEEN = 10;
}

message Location {
    float latitude = 1;
    float longitude = 2;
}

A big part of that is due to the fact that the library is catered to produce lossy/boxed variants of the sensor data by making assumptions about the sensor ranges, etc.

So I am wondering whether there is a way to combine the two approaches by using custom data types with nanopb, but I can't find any markers if this is a good idea.

Thoughts?

[Petteri Aimonen]

Hi,

> So I am wondering whether there is a way to combine the two approaches by
> using custom data types with nanopb, but I can't find any markers if this
> is a good idea.

Well, nanopb has to obey the protobuf encoding format, so that limits
the space savings quite a bit. It seems that lora-serialization does not
label the fields, so it is not possible to extend the messages without
separate code written to handle backwards-compatibility. In protobuf,
each field is prefixed by the tag number which makes extending easy but
costs 1 byte per field.

But there are still some things that can be done:

- Submessages are especially costly, because the submessage header takes
atleast 2 bytes.
- Varints close to 0 are very small, down to 1 byte size.
- If your varint values are commonly more than 2097151, it is better to
just use fixed32.

Your example could be rewritten as:

message Event {
uint32 time_of_day = 1; /* Seconds since midnight */
sint32 latitude_mdeg = 2; /* Millidegrees, 100 meter accuracy */
sint32 longitude_mdeg = 3;
AdsGain gain = 4;
uint32 solar1 = 5;
uint32 solar2 = 6;
sint32 temp_mdeg = 7; /* Millidegrees celsius */
uint32 humidity = 8; /* Percents */
}

Protobuf encoding is variable length, but let's check the message size
for typical maximum values. Varints up to 127 take 1 byte, up to 16383
take 2 bytes and up to 2097151 take 3 bytes.

time_of_day = 86399 # 1 + 3 bytes
latitude_mdeg = 90000 # 1 + 3 bytes
longitude_mdeg = 180000 # 1 + 3 bytes
gain = (any) # 1 + 1 bytes
solar1 = 999999 # 1 + 3 bytes
solar2 = 999999 # 1 + 3 bytes
temp_mdeg = 20000 # 1 + 3 bytes
humidity = 50 # 1 + 1 bytes

Giving a total of 20 data bytes + 8 tag bytes = 28 bytes, which is 7
bytes more that the lora-serialization lib.

I used a few assumptions there, like that you wouldn't buffer packets
for very long so time of day might be enough to resolve their
timestamps. And that there aren't any cheap humidity meters that would
give better than 1% accuracy.

If you transmit multiple of these events at once (instead of
one-by-one), there is even more that can be done. For example, location
probably doesn't change much often, so you could make it an optional
field and leave it out when same as previous messages.

--
Petteri

[Joscha Feth]

Hi Petteri,

Thank you very much, this is very helpful, I will try restructuring my event and see if that works out. Regarding the field labeling, you are right, so I am happy to spend those extra 7 bytes to make the message forward compatible.
I saw that you removed the INT_16 size constraints for nanopb - is the hinting generally not needed or detrimental?

Cheers,
Joscha

[Petteri Aimonen]

Hi,

> I saw that you removed the INT_16 size constraints for nanopb - is the
> hinting generally not needed or detrimental?

The size hinting only affects the size of the C struct, encoding format
stays the same (varint).

--
Petteri

[Joscha Feth]

Hi Petteri,

I just transformed my message in the way you suggested (except for the gps coords, 100m resolution is not high enough) - I got it down to 28 bytes which was still too much however - I removed the unixtime for now, will see what I can do about that. 

Thank you very much!

Cheers,

Joscha