encoding/asn1 overly strict?

[peterso...@gmail.com]

I am having an issue where a network device I am communicating with is returning asn1 that the encoding/asn1 package refuses to parse.

The byte stream I am reading is 0x30(sequence) 82(2 bytes for length to follow) 00 12 (length of 18). The asn1 package has a check at https://github.com/golang/go/blob/master/src/encoding/asn1/asn1.go#L472 that does not like the zero byte at the beginning of the length, citing the DER spec as the reason why.

Yes, the length here could be expressed as two, or even one byte, but it seems there are multiple implementations in the wild that do not honor the spec in this regard. It would make sense to me to relax the restriction a bit in encoding/asn1 and only check for length of zero after all numBytes length bytes have been processed. 

Is this something that can be changed? Should I file an issue? I'm not sure how to go about this kind of thing.

[Matt Harden]

If the data you're decoding is not canonical, then it's not DER. It's probably BER. DER is a strict subset of BER, and by definition there is only one way to represent a particular value in DER, which is the whole point. Unfortunately the asn1 package does not include a BER decoder.

See http://go-search.org/search?q=ber

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Paul Borman]

It is probably better, if you have only one, to accept BER and produce DER.  That is pretty much the standard networking mantra.  Be liberal in what you receive and conservative in what you send.

   -Paul

[Matt Harden]

"Package asn1 implements parsing of DER-encoded ASN.1 data structures, as defined in ITU-T Rec X.690."

It's only present in the standard library to support TLS. In the case of TLS it is often *less secure* to be liberal in what you receive. DER is designed have one and only one way to represent data. Validation of conformance to DER is important while parsing this data.

The package is not intended for SNMP or LDAP or other purposes. Other packages exist outside the standard library to parse BER.

On Tue, Jun 30, 2015 at 2:01 PM Craig Peterson <peterso...@gmail.com> wrote:

I agree completely. I made a CR to modify the package, but it wasn't received well: https://go-review.googlesource.com/#/c/11734/

[Matt Harden]

Another possible workaround would be to use the RawValue or RawContent types. These aren't well documented, but if you make the first public field of a struct a RawContent, then the entire raw bytes of the item being decoded should be copied into that field. This might bypass the length check; I don't know. You would still have to decode the field yourself, but you could use the rest of the asn1 package functionality unchanged.

[Craig Peterson]

I can appreciate that reasoning, even if the outcome is not sufficient for my needs. None of the packages I have found for ber are maintained or of very high quality. I will likely fork encoding/asn1 and apply the changes I need.

[Craig Peterson]

I agree it is not DER. It appears to be somewhat non-compliant and lazy. But the severity of the violation does not seem strong enough that rejecting the entire message is necessary, as you can quite easily continue on and get the length from the remaining bytes. 

[Craig Peterson]

I agree completely. I made a CR to modify the package, but it wasn't received well: https://go-review.googlesource.com/#/c/11734/

On Tuesday, June 30, 2015 at 11:34:49 AM UTC-6, Paul Borman wrote:

[Paul Borman]

It seems there is a strong feeling the ability to enforce DER is required.  Fair enough.  That does not mean it cannot accept BER, only that the current API will only accept DER.  The package benefit from having a mechanism to select between BER and strict DER enforcement.  The package is not named der, after all.  The documentation, at a minimum, could probably be improved.

Craig, if you want this in, maybe file an issue to flesh out the asn1 package.

Hopefully fleshing out this package, while retaining all existing semantics, would be acceptable.   My personal feeling is that general use packages should be as complete as reasonably possible, not just sufficient for the initial motivation/use case.  That view comes from the experience of having been a an OS developer for 25 years (BSDi and Cray).

    -Paul

expedient |ikˈspēdēənt|

adjective

Doing something to save time today that you will regret tomorrow.

[Matt Harden]

Just to be clear, SNMP (and LDAP) uses BER, not DER. It's not sloppy or non-compliant that the data you're getting isn't DER. DER is used for more specific purposes than BER. I would support a more extensive version of the asn1 package being created; I might help out with the development. Eventually the new version could be included in the standard library if the language maintainers find it compelling enough.

[cpu...@gmail.com]

Sorry to revive this ancient discussion. 

We've come across https://forum.golangbridge.org/t/x509-certificate-parse-error-with-iot-device/27622/2 where an IOT device from a large vendor uses BER for it's TLS certificate. It's unlikely, that the vendor will fix the certificate any time soon. We've not found an approach for communicating with the device sofar unless using patched Go stdlib.

Are there any options we could use to accept a BER-encoded ASN.1 certificate for TLS connections?

Thanks!

[Andreas Götz]

To be more specific: in order to connect to the IOT server presenting the rogue BER certificate, we have InsecureSkipVerify: true. The description of InsecureSkipVerify reads:

// InsecureSkipVerify controls whether a client verifies the server's

// certificate chain and host name. If InsecureSkipVerify is true, crypto/tls

// accepts any certificate presented by the server and any host name in that

// certificate. In this mode, TLS is susceptible to machine-in-the-middle

// attacks unless custom verification is used. This should be used only for

// testing or in combination with VerifyConnection or VerifyPeerCertificate.

Unfortunately, during connection handshake, that will still call Conn.verifyServerCertificate which in turn calls x509.ParseCertificate before InsecureSkipVerify is evaluated. At this point the handshake is aborted.

Seems there is no way to even plug a custom mechanism for BER decoding into the handshake process due to tls/x509 failing at the certificate decoding step?

[Brian Candler]

On Wednesday, 8 June 2022 at 10:09:26 UTC+1 cpu...@gmail.com wrote:

We've not found an approach for communicating with the device sofar unless using patched Go stdlib.

Connect via a proxy like stunnel?

Out of interest, does raw "openssl s_client" allow communication with the device?

It would seem reasonable to me for InsecureSkipVerify to skip certificates without parsing them at all.  It is, after all, insecure by definition.

[cpu...@gmail.com]

On Wednesday, June 8, 2022 at 5:53:50 PM UTC+2 Brian Candler wrote:

On Wednesday, 8 June 2022 at 10:09:26 UTC+1 andig wrote:

We've not found an approach for communicating with the device sofar unless using patched Go stdlib.

Connect via a proxy like stunnel?

Out of interest, does raw "openssl s_client" allow communication with the device?

We receive an alert 40 (Handshake failure ) when using openssl. So the cert is definitively faulty in some way. 

 :~/wallbox/hack$ openssl s_client  -connect 192.168.1.180:4712 

CONNECTED(00000005)

depth=0 CN = EEBUS, O = EVBox Intelligence, C = NL

verify error:num=18:self signed certificate

verify return:1

depth=0 CN = EEBUS, O = EVBox Intelligence, C = NL

verify return:1

140477570593216:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1528:SSL alert number 40

---

Certificate chain

0 s:CN = EEBUS, O = EVBox Intelligence, C = NL

   i:CN = EEBUS, O = EVBox Intelligence, C = NL

---

Server certificate

-----BEGIN CERTIFICATE-----

MIIBszCCAVmgAwIBAgIBATAKBggqhkjOPQQDAjA6MQ4wDAYDVQQDDAVFRUJVUzEb 

Seems that in this case- if we regard openssl as "the standard" it's obsolete to talk about Go.

It would seem reasonable to me for InsecureSkipVerify to skip certificates without parsing them at all.  It is, after all, insecure by definition.

It doesn't do that as it checks for supported ciphers afterwards, so it needs to decode the cert first.

 

Cheers,

Andi

[Brian Candler]

On Thursday, 9 June 2022 at 14:27:04 UTC+1 cpu...@gmail.com wrote:

We receive an alert 40 (Handshake failure ) when using openssl. So the cert is definitively faulty in some way. 

 :~/wallbox/hack$ openssl s_client  -connect 192.168.1.180:4712 

Certainly it will report a failed verification, but s_client should continue (i.e. allow you to send and receive data), if the TLS connection was established.  That is: it won't drop the connection on the floor unless you've specified "-verify_return_error"

Have you tried adding "-servername EEBUS" and/or "-verify 0" to the command line?  And/or "-CAFile <path_to_file.pem>" where this file contains a copy of the (self-signed) certificate itself.

Seems that in this case- if we regard openssl as "the standard" it's obsolete to talk about Go.

Go doesn't use openssl internally.  I was only trying to establish whether this vendor has *any* claim of their implementation interoperating with any sort of TLS client.  If openssl won't talk to it, then you can pretty much argue they're not talking TLS at all.

However if openssl s_client does talk to it, then you have another workaround: you could exec.Cmd an instance of openssl s_client, and pipe data to and from it.

But really I would question the wisdom of dealing with any vendor who (a) doesn't follow standards for certificates, and (b) will not allow you to replace the device certificate, or won't replace it themselves, when given direct evidence of its brokenness and non-compliance to standards.  How exactly do you think this vendor will respond when you find a more serious security violation, or some other failing in their product?

[Brian Candler]

I forgot to add one thing, and you didn't paste the whole certificate PEM so I can't check this.

Recent versions of Go won't verify the certificate unless it contains a subjectAltName; matching against only the CN is no longer supported.  So if you do get your vendor to re-issue the cert, make sure it also includes a DNS SAN, if it doesn't already.  It doesn't matter if it's just a string like "EEBUS"; you can specify the ServerName at connection time.

[Andreas Götz]

On Thursday, June 9, 2022 at 6:44:58 PM UTC+2 Brian Candler wrote:

I forgot to add one thing, and you didn't paste the whole certificate PEM so I can't check this.

Recent versions of Go won't verify the certificate unless it contains a subjectAltName; matching against only the CN is no longer supported.

I am aware of the rejection of SHA1 hashes, but not this change. Could you kindly share the CL or release note? Much appreciated!

So if you do get your vendor to re-issue the cert, make sure it also includes a DNS SAN, if it doesn't already.  It doesn't matter if it's just a string like "EEBUS"; you can specify the ServerName at connection time.

On Thursday, 9 June 2022 at 17:38:04 UTC+1 Brian Candler wrote:

[Brian Candler]

On Friday, 10 June 2022 at 14:31:10 UTC+1 Andreas Götz wrote:

On Thursday, June 9, 2022 at 6:44:58 PM UTC+2 Brian Candler wrote:

I forgot to add one thing, and you didn't paste the whole certificate PEM so I can't check this.

Recent versions of Go won't verify the certificate unless it contains a subjectAltName; matching against only the CN is no longer supported.

I am aware of the rejection of SHA1 hashes, but not this change. Could you kindly share the CL or release note? Much appreciated!

In the release notes for go 1.15:

https://go.dev/doc/go1.15#commonname

The transitional override (GODBEUG=x509ignoreCN=0) was removed in 1.17.