appspot.com started rejecting SSL without SNI causing U-Blox IoT modems to fail, since May

Bjorn Wesen

unread,

Jul 28, 2020, 8:07:30 AM7/28/20

to Google App Engine

We have a large fleet of U-Blox IoT modem-equipped devices out there, calling in to our backend on AppEngine, using SSL to <ourappname>.appspot.com (we don't have any custom SSL cert or managed cert).

Up until May 26 or so this year this has worked fine, after that some endpoints started failing the SSL handshake, and the problem became worse and worse until the entire fleet was disabled.

U-Blox managed to trace it to a failing SSL negotiation, and that their modems don't support the SSL SNI (Server Name Identification) option, which is required normally to differ between certs on a server that has multiple certs on the same IP.

Sure enough, I checked with openssl s_client -noservername and it fails to appspot.com now - it doesn't provide the normal default cert of *.appspot.com anymore when no SNI is given (just the No SNI provided; please fix your client. message).

I didn't find anything in the AppEngine release notes about this change.

Now, this is a tricky because yes, SNI has been around since 2002 and I think it's even mandatory in TLSv1.3. On the other hand, there must be hundreds of thousands of U-Blox modems out there that doesn't support this, and the companies that have built IoT products around AppEngine and U-Blox (like us) will start to see their fleets fail.

Would it be possible to re-instate a default handling of the SSL endpoints on appspot.com to provide a default cert of *.appspot.com ?

We had to go non-encrypted as a panic work-around, which is obviously not desirable.

Another work-around is to purchase a completely separate IP and custom SSL cert; this would remove the semantical requirement of SNI but it would be helpful to confirm if there would still be a technical requirement for it due to the way the AppEngine SSL endpoints are configured in that case.

Best regards
/Bjorn

Ryan B

unread,

Jul 30, 2020, 8:22:35 PM7/30/20

to Google App Engine

Interesting! And concerning. I'd definitely care about this too.

Just as another data point, I checked one of my apps that serves from both a custom domain (https://brid.gy/) and appspot (https://brid-gy.appspot.com/), and Qualys's SSL Labs test shows that the appspot subdomain is still happily supporting traditional IP-based SSL, without SNI.

Bjorn Wesen

unread,

Jul 31, 2020, 8:10:10 AM7/31/20

to Google App Engine

Hi, thanks for checking through ssllabs on your domains - however if I connect to brid-gy.appspot.com from here with openssl, I do get the same SNI-fail as normal (which makes sense as I guess the *.appspot.com DNS goes to the same set of endpoints regardless of app, maybe localized by appengine country or something):

$ openssl s_client -noservername -connect brid-gy.appspot.com:443

CONNECTED(00000005)

depth=0 OU = "No SNI provided; please fix your client.", CN = invalid2.invalid

verify error:num=18:self signed certificate

verify return:1

depth=0 OU = "No SNI provided; please fix your client.", CN = invalid2.invalid

verify return:1

---

Certificate chain

0 s:OU = "No SNI provided; please fix your client.", CN = invalid2.invalid

i:OU = "No SNI provided; please fix your client.", CN = invalid2.invalid

---

From which part of the SSLLabs report did you conclude that it was delivering certificates without the SNI handshake? I only saw a footnote that it didn't even try with Handshake Simulating IE6 because it doesn't support SNI (last row in that section) so I assume their test has concluded SNI was required after all..

The tricky part is, if you try it directly to the IP number, it still works, as the (modern) clients send the IP number as Server Name during the SNI handshake. Then AppEngine will return the *.appspot-preview.com certificate (this, I assume, could be confusing to some customers as well but is not the same problem).

Like this, to the IP number that brid-gy.appspot.com resolves to (openssl s_client in this case inserts the IP number as Server Name):

$ openssl s_client -connect 172.217.22.180:443

CONNECTED(00000005)

depth=2 OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign

verify return:1

depth=1 C = US, O = Google Trust Services, CN = GTS CA 1O1

verify return:1

depth=0 C = US, ST = California, L = Mountain View, O = Google LLC, CN = *.appspot-preview.com

verify return:1

---

Certificate chain

0 s:C = US, ST = California, L = Mountain View, O = Google LLC, CN = *.appspot-preview.com

i:C = US, O = Google Trust Services, CN = GTS CA 1O1

1 s:C = US, O = Google Trust Services, CN = GTS CA 1O1

i:OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign

---

Regards
/Bjorn

Reply all

Reply to author

Forward