CT V2 log list issue, VERY URGENT

[方一格]

Hi all,  

We recently noticed that our company’s commercial Android devices are facing a serious issue with CA certification checks related to Certificate Transparency (CT) through the third‑party library at https://github.com/appmattus.

Background
In 2021, our commercial android system and applications integrated CT mechanisms via appmattus. Since then, we have not actively tracked updates and unfortunately missed the deprecation notice for CT v2.  

These applications are preloaded in the ROM, making it impossible to update or override through standard installation methods. In addition, many of these devices (purchased by our business partners) have been stored in warehouses, which will make updates solutions even more impossible.  

After evaluation, We believe a significant number of devices (potentially millions) may face a high risk of activation failure and disruption to normal operation due to CT v2 deprecation, with no effective feasible recovery path. This could result in substantial losses for our company.

After reviewing prior discussions, we understand Google’s updated CT policy and we are willingly to complying with it. However, the situation is extremely urgent: the relevant HTTPS certificates are scheduled for renewal in April. Without intervention, millions of our devices in market or stored in warehouse will face a large‑scale disruption or other issues.

Request
To mitigate this risk and give us time to permanently resolve our dependency on CT v2, we kindly request that Google updates the CT v2 log list(https://www.gstatic.com/ct/log_list/v2/log_list.json) to align with the CT v3 log list (https://www.gstatic.com/ct/log_list/v3/log_list.zip).

This would be the most reliable and immediate way to prevent device failures while we complete a full migration away from CT v2 and remove the dependency on the appmattus library.

We look forward to your reply.

Kind regards,  
Ethan  
On behalf of our company

[Ben Cartwright-Cox]

Hey Ethan

I think it would be useful for everybody to understand which company
is impacted here and how many devices (other than just " millions ")
have potentially been bricked

> --
> You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
> To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/ead46425-724c-44c7-8d15-008f624cfbbdn%40chromium.org.

[Joe DeBlasio]

Hi Ethan,

You sent two emails to ct-policy@, though I only approved this one. The other one was a thread which included a bunch of personal information I presume you did not intend to post. It did, however, include an outreach email Google sent to you in March of 2023 (nearly three years ago) that stated clearly that the log list would stop working. We've extended the timeline several times since then to give folks a greater opportunity to adapt. While we're never keen to see app breakage, we're unable to extend our deadlines further.

If you'd like to avoid breakage, you should update your app immediately to remove the appmattus dependency. I acknowledge that users take a long time to update applications, and these users will not naturally update in time. If your app is distributed through the Google Play Store, we recommend using Play Console's mechanism to prompt users to update their installations.

Please note that updating to the latest version of the appmattus library is not sufficient to avoid application breakage. The appmattus library is not authorized to depend on the v2 or v3 log list, and the most recent version of the library will also stop working in the coming months.

Best,

Joe, on behalf of the Chrome CT team.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CAL%3D9YSUTKOojLw8sXecDZevbbX9NChE3Tu_-d28fxanyweRtMw%40mail.gmail.com.

[方一格（Ethan）]

Hi all,

Thanks for your response and your attention to this issue.

To Ben's question, for business reasons I’m not allowed to share my company’s information in this public channel. As of now, some of our customers have encountered this issue (around 2 thousand devices), but it has been temporarily resolved by updating certificates before V2 was completely discontinued. The situation in the future will be unpredictably bad.

To Joe's reply, as I mentioned above, we are a commercial Android device company. Our devices are sold to both business-to-business and business-to-consumer customers. This means that the vast majority of manufactured and sold devices are in a non-active state. Once the V2 log server stoped updates, our device service application, which relies on the third-party library appmattus for verification, will directly fail network requests, which will turn devices into bricks like Ben said. From a practical standpoint, the effeciency of our OTA updates is clearly insufficient.

Our current repair solution for this service application has already abandoned appmattus completely. The key point is that the update solution won't fix the issue after the V2 log list stops updating.

We understand that appmattus does not recommend using it and are aware of Google's latest CT policy. We simply need time to resolve the current issues. At present, the previously mentioned suggestion regarding the V2 log list appears to be the only way to buy us time.

Best,  
Ethan.

[Joe DeBlasio]

Thanks, Ethan.

It is possible for you to further delay breakage of these applications. While the current v2 log list does not contain the CT logs most recently added to Chrome's log list, it does still contain operational logs identified as "usable" that can satisfy appmattus policy requirements.

Assuming the appmattus library fully supports RFC6962, any certificate expiring before 2027 can be used if you configure your server to present SCTs using the TLS extension outlined in RFC6962 section 3.3.1. If the appmattus library does not support this mechanism, you should still be able to work with your CA to receive a certificate with the necessary SCTs embedded.

This technique will already postpone breakage by almost a full year, so it does not appear that adding new logs to the v2 log list is necessary.

Best,

Joe

[方一格（Ethan）]

Hi Joe,

Thank you for your suggestion. However, our technical team has already discussed this approach. We are unable to deliver SCTs via the TLS extension because the intermediate Wangsu CDN terminates the SSL connection and does not propagate the SCTs. 

Therefore, based on our team's analysis, we understand that directly migrating CT log list data from V3 to V2 cannot cover the tiled_logs portion. Is it ok to migrate CT log list data from V3 to V2 while simultaneously unfolding the tiled_logs content? Or we could find a solution that benefits both parties. 

Additionally, what is Google's exact timeline for when the V2 log service will begin returning 404 responses? We have downgrade logic for network requests, in this case most of our devices will still be functional.

Looking forward to replys.

Best,

Ethan

[Joe DeBlasio]

Hi Ethan,

Addressing your questions and comments somewhat out of order:

Additionally, what is Google's exact timeline for when the V2 log service will begin returning 404 responses? We have downgrade logic for network requests, in this case most of our devices will still be functional.

We will not remove the v2 log list (i.e. make its endpoint return HTTP 404) before 2027. At that time, we expect it will be safe to remove the list, as the expiry windows for all logs included in the current list will have passed and it will thus not be possible to procure SCTs from any logs included in the list.

Or we could find a solution that benefits both parties.

While I'm deeply sympathetic to the pain this will cause you and your team, our experience has taught us that extending turndown deadlines does not benefit us, nor the ecosystem broadly. Each time a deadline approaches, a new batch of developers realizes that they will be negatively impacted, and asks for further postponement.

When we fully turn down the list in 2027, about 4.5 years will have passed since the initial announcement of the planned turn down. Notifying impacted developers is extremely difficult, and many developers missed this announcement. However, all impacted applications were broken for several hours in early 2023 when we first attempted to turn the list down. That same day, the most commonly used library was updated to remove reliance on the v2 log list. Developers will have had 4 years to update their applications.

We thus feel strongly that we have provided adequate notice, and that there are diminishing returns to further postponement.

Therefore, based on our team's analysis, we understand that directly migrating CT log list data from V3 to V2 cannot cover the tiled_logs portion. Is it ok to migrate CT log list data from V3 to V2 while simultaneously unfolding the tiled_logs content?

As noted, we do not intend to update the v2 log list with updated contents from the v3 list. However, even if we were to do so, we could not safely add the contents of the tiled_logs arrays to the logs arrays. We do not have confidence that all v2 log list consumers will gracefully handle SCTs that contain an unexpected leaf_index extension, thus combining the arrays would risk creating more unexpected breakage ahead of our announced timelines.

We are unable to deliver SCTs via the TLS extension because the intermediate Wangsu CDN terminates the SSL connection and does not propagate the SCTs.

I assume that migrating away from Wangsu CDN is not an option. That unfortunately suggests that your best option is to work with Wangsu CDN to see if they may be able to provide the necessary SCTs via the TLS extension or by using TLS certificates with the necessary SCTs embedded.

Best,

Joe