CT V2 log list issue, VERY URGENT

[方一格]

Hi all,  

We recently noticed that our company’s commercial Android devices are facing a serious issue with CA certification checks related to Certificate Transparency (CT) through the third‑party library at https://github.com/appmattus.

Background
In 2021, our commercial android system and applications integrated CT mechanisms via appmattus. Since then, we have not actively tracked updates and unfortunately missed the deprecation notice for CT v2.  

These applications are preloaded in the ROM, making it impossible to update or override through standard installation methods. In addition, many of these devices (purchased by our business partners) have been stored in warehouses, which will make updates solutions even more impossible.  

After evaluation, We believe a significant number of devices (potentially millions) may face a high risk of activation failure and disruption to normal operation due to CT v2 deprecation, with no effective feasible recovery path. This could result in substantial losses for our company.

After reviewing prior discussions, we understand Google’s updated CT policy and we are willingly to complying with it. However, the situation is extremely urgent: the relevant HTTPS certificates are scheduled for renewal in April. Without intervention, millions of our devices in market or stored in warehouse will face a large‑scale disruption or other issues.

Request
To mitigate this risk and give us time to permanently resolve our dependency on CT v2, we kindly request that Google updates the CT v2 log list(https://www.gstatic.com/ct/log_list/v2/log_list.json) to align with the CT v3 log list (https://www.gstatic.com/ct/log_list/v3/log_list.zip).

This would be the most reliable and immediate way to prevent device failures while we complete a full migration away from CT v2 and remove the dependency on the appmattus library.

We look forward to your reply.

Kind regards,  
Ethan  
On behalf of our company

[Ben Cartwright-Cox]

Hey Ethan

I think it would be useful for everybody to understand which company
is impacted here and how many devices (other than just " millions ")
have potentially been bricked

> --
> You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
> To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/ead46425-724c-44c7-8d15-008f624cfbbdn%40chromium.org.

[Joe DeBlasio]

Hi Ethan,

You sent two emails to ct-policy@, though I only approved this one. The other one was a thread which included a bunch of personal information I presume you did not intend to post. It did, however, include an outreach email Google sent to you in March of 2023 (nearly three years ago) that stated clearly that the log list would stop working. We've extended the timeline several times since then to give folks a greater opportunity to adapt. While we're never keen to see app breakage, we're unable to extend our deadlines further.

If you'd like to avoid breakage, you should update your app immediately to remove the appmattus dependency. I acknowledge that users take a long time to update applications, and these users will not naturally update in time. If your app is distributed through the Google Play Store, we recommend using Play Console's mechanism to prompt users to update their installations.

Please note that updating to the latest version of the appmattus library is not sufficient to avoid application breakage. The appmattus library is not authorized to depend on the v2 or v3 log list, and the most recent version of the library will also stop working in the coming months.

Best,

Joe, on behalf of the Chrome CT team.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CAL%3D9YSUTKOojLw8sXecDZevbbX9NChE3Tu_-d28fxanyweRtMw%40mail.gmail.com.

[方一格（Ethan）]

Hi all,

Thanks for your response and your attention to this issue.

To Ben's question, for business reasons I’m not allowed to share my company’s information in this public channel. As of now, some of our customers have encountered this issue (around 2 thousand devices), but it has been temporarily resolved by updating certificates before V2 was completely discontinued. The situation in the future will be unpredictably bad.

To Joe's reply, as I mentioned above, we are a commercial Android device company. Our devices are sold to both business-to-business and business-to-consumer customers. This means that the vast majority of manufactured and sold devices are in a non-active state. Once the V2 log server stoped updates, our device service application, which relies on the third-party library appmattus for verification, will directly fail network requests, which will turn devices into bricks like Ben said. From a practical standpoint, the effeciency of our OTA updates is clearly insufficient.

Our current repair solution for this service application has already abandoned appmattus completely. The key point is that the update solution won't fix the issue after the V2 log list stops updating.

We understand that appmattus does not recommend using it and are aware of Google's latest CT policy. We simply need time to resolve the current issues. At present, the previously mentioned suggestion regarding the V2 log list appears to be the only way to buy us time.

Best,  
Ethan.

[Joe DeBlasio]

Thanks, Ethan.

It is possible for you to further delay breakage of these applications. While the current v2 log list does not contain the CT logs most recently added to Chrome's log list, it does still contain operational logs identified as "usable" that can satisfy appmattus policy requirements.

Assuming the appmattus library fully supports RFC6962, any certificate expiring before 2027 can be used if you configure your server to present SCTs using the TLS extension outlined in RFC6962 section 3.3.1. If the appmattus library does not support this mechanism, you should still be able to work with your CA to receive a certificate with the necessary SCTs embedded.

This technique will already postpone breakage by almost a full year, so it does not appear that adding new logs to the v2 log list is necessary.

Best,

Joe

[方一格（Ethan）]

Hi Joe,

Thank you for your suggestion. However, our technical team has already discussed this approach. We are unable to deliver SCTs via the TLS extension because the intermediate Wangsu CDN terminates the SSL connection and does not propagate the SCTs. 

Therefore, based on our team's analysis, we understand that directly migrating CT log list data from V3 to V2 cannot cover the tiled_logs portion. Is it ok to migrate CT log list data from V3 to V2 while simultaneously unfolding the tiled_logs content? Or we could find a solution that benefits both parties. 

Additionally, what is Google's exact timeline for when the V2 log service will begin returning 404 responses? We have downgrade logic for network requests, in this case most of our devices will still be functional.

Looking forward to replys.

Best,

Ethan