Confusion about ember+ real value encoding vs. asn.1

[jorge...@gmail.com]

Hi all!

First of all, I've been using Ember+ for years now and I am a big fan of the protocol. It does a lot of things right that modern waves of json over http seem to have forgotten.

I am currently working in a JVM environment, so we do not have the luxury of using libember without creating some foreign language bindings for it or running some protocol server. However, given that you are based on S101/ASN.1/X.690 this isn't necessary since great tools exist that perform this encoding/decoding native on the JVM with excellent performance (at least as good as you need if you're on the JVM in the first place).

This has worked great for many years, however, currently we're working on providing support for real value encoding. And this is where things become a big confusing. 

From the documentation:

Real values must be encoded in binary form [X.690 8.5.6], base 2 [X.690 8.5.6.2]. Maximum precision shall be double (64 bit).

This makes perfect sense, and it also very nicely ensures that existing X.690 frameworks can both decode and encode Ember+ reals. In practice, base 8, 10, and 16 are very uncommon anyways. 

As an example I've found a good example here:

https://www.strozhevsky.com/free_docs/asn1_by_simple_words.pdf

(page 4, examples of encoding real values in ASN.1). 

The value of 0.15625 is encoded as 80 FB 05. This is also in agreement with other

ASN.1 tools I have tried, such as wireshark's S101 protocol. And it's an easy to follow example.

However, by using libember the same value is encoded as 80 FD 05. 

So there's a difference here. FD is -3 in two's complement, which seems like a strange exponent given that the mantissa is 5. By using EmberViewer and wireshark it seems like there's a pattern where the exponent has a different value than what other ASN.1 tools suggest. By inspecting the libember code it seems like the IEEE754 exponent is just shifted by 1023 without taking into account that the mantissa is represented differently in the normal domain of IEEE754 and ASN.1. But I am not an expert in libember, so please correct me if I am wrong...

Anyways, I am a bit lost in my tracks right now given that apparently no ASN.1 implementations except for libember-based ones are able to work with e.g. ember-viewer for real values.

I am not sure what I am looking for in terms of a response here, but if somewhere in my analysis I've just misunderstood something, or if I am making false assumptions please let me know. I am just looking to clarify this discrepancy, and would appreciate all help I can get doing so.

Cheers! 

[Hoffmann, Kimon (LAWO)]

Hello Jorge,

> This has worked great for many years, however, currently we're working on providing support for real value encoding. And this is where things become a big confusing.
> From the documentation:
>
> Real values must be encoded in binary form [X.690 8.5.6], base 2 [X.690 8.5.6.2]. Maximum precision shall be double (64 bit).
>
> This makes perfect sense, and it also very nicely ensures that existing X.690 frameworks can both decode and encode Ember+ reals. In practice, base 8, 10, and 16 are very uncommon anyways.
>
> As an example I've found a good example here:
> https://www.strozhevsky.com/free_docs/asn1_by_simple_words.pdf
> (page 4, examples of encoding real values in ASN.1).
> The value of 0.15625 is encoded as 80 FB 05. This is also in agreement with other
> ASN.1 tools I have tried, such as wireshark's S101 protocol. And it's an easy to follow example.
>
> However, by using libember the same value is encoded as 80 FD 05.
> So there's a difference here. FD is -3 in two's complement, which seems like a strange exponent given that the mantissa is 5. By using EmberViewer and wireshark it seems like there's a pattern where the exponent has a different value than what other ASN.1 tools suggest. By inspecting the libember code it seems like the IEEE754 exponent is just shifted by 1023 without taking into account that the mantissa is represented differently in the normal domain of IEEE754 and ASN.1. But I am not an expert in libember, so please correct me if I am wrong...
>
> Anyways, I am a bit lost in my tracks right now given that apparently no ASN.1 implementations except for libember-based ones are able to work with e.g. ember-viewer for real values.
>
> I am not sure what I am looking for in terms of a response here, but if somewhere in my analysis I've just misunderstood something, or if I am making false assumptions please let me know. I am just looking to clarify this discrepancy, and would appreciate all help I can get doing so.

Unfortunately I myself am a bit out of the loop regarding the intricacies and peculiarities of the Ember+ implementation, but to me it looks like your assessment is correct and you have discovered a bug in the real value encoding of the Ember+ library. Or rather *libraries* since they all seem to agree and make the same error.

The unfortunate thing about all of this is that I see no means of fixing the Error, without introducing a major incompatibility across the entire portfolio of Ember+ enabled devices and applications. Any suggestions in this regard are wholeheartedly welcomed. :)

Regarding your situation specifically the question would be, if the tools at your disposal allow post-processing of parsed values? If yes then you might be able to introduce a post-processing on floating point values that fixes the misinterpretation of the mantissa bits?!


Best regards,
Kimon



Kimon Hoffmann
Lead Software Developer & Senior Software Systems Architect

--
Lawo AG | Am Oberwald 8 | 76437 Rastatt | Germany
Phone +49 7222 1002 5810 | www.lawo.com
--
Lawo AG | Registered Office: Rastatt | Amtsgericht Mannheim HRB 707330
Vorstand: Philipp Lawo, Claudia Nowak | Chairman of the Supervisory Board: Prof. Dr. Uwe Hack








Unsere Datenschutzerklärung finden Sie hier<https://lawo.com/datenschutzrichtlinie/?lang=de>. | See our Privacy Policy here<https://lawo.com/privacy-policy/>.

[jorge...@gmail.com]

Hi Kimon,

thanks for the quick response.

I agree that it doesn't make sense to fix this on your end (or either end) because of the reasons you provide. 

And that's fine to be honest, these things happen from time to time, especially when the standard is old and doesn't include

any examples of real value encoding/decoding (which baffles me to be honest, given the complexity of this type compared to e.g. integers).

Moreover reals are not very commonly used as integers with a given unit (precision, e.g. milliseconds) often does the job better.

Since you ask, then what I would propose is to maybe add a footnote to the Ember+ documentation section about real value representation, 

where you state that you are compatible with X.690 base 2 etc. and maybe say something about known issues with compatibility with

other ASN.1 libraries. This can save people a lot of time if they are encountering the same problems as me. 

It's really only an issue if you're a client not using libember (or a derivation of it) talking to a device with real value parameters.

As a server you can in most cases transform the real into an int by using some implicit precision.

Regarding your last point about pre/post transformations you're of course right that this is possible.

However, the transformation isn't very linear as the number of shifts done is based on the number of trailing zeros 

in the mantissa after 53 shifts (for double precision). Which of course varies a lot. So for my case the solution would be to use a fork

of the library I am currently developing real value support for (bouncycastle) and then just replace the exponent part of the double encode/decode methods

with the code from libember. This could perhaps also be useful for other JVM-based Ember+ users (I've seen a few mentions over the years).

Thanks again for the quick response, and for helping me clarify the discrepancy :)

Best regards,

Jorgen