CT V2 log list issue, VERY URGENT

[方一格]

Hi all,  

We recently noticed that our company’s commercial Android devices are facing a serious issue with CA certification checks related to Certificate Transparency (CT) through the third‑party library at https://github.com/appmattus.

Background
In 2021, our commercial android system and applications integrated CT mechanisms via appmattus. Since then, we have not actively tracked updates and unfortunately missed the deprecation notice for CT v2.  

These applications are preloaded in the ROM, making it impossible to update or override through standard installation methods. In addition, many of these devices (purchased by our business partners) have been stored in warehouses, which will make updates solutions even more impossible.  

After evaluation, We believe a significant number of devices (potentially millions) may face a high risk of activation failure and disruption to normal operation due to CT v2 deprecation, with no effective feasible recovery path. This could result in substantial losses for our company.

After reviewing prior discussions, we understand Google’s updated CT policy and we are willingly to complying with it. However, the situation is extremely urgent: the relevant HTTPS certificates are scheduled for renewal in April. Without intervention, millions of our devices in market or stored in warehouse will face a large‑scale disruption or other issues.

Request
To mitigate this risk and give us time to permanently resolve our dependency on CT v2, we kindly request that Google updates the CT v2 log list(https://www.gstatic.com/ct/log_list/v2/log_list.json) to align with the CT v3 log list (https://www.gstatic.com/ct/log_list/v3/log_list.zip).

This would be the most reliable and immediate way to prevent device failures while we complete a full migration away from CT v2 and remove the dependency on the appmattus library.

We look forward to your reply.

Kind regards,  
Ethan  
On behalf of our company

[Ben Cartwright-Cox]

Hey Ethan

I think it would be useful for everybody to understand which company
is impacted here and how many devices (other than just " millions ")
have potentially been bricked

> --
> You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
> To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/ead46425-724c-44c7-8d15-008f624cfbbdn%40chromium.org.

[Joe DeBlasio]

Hi Ethan,

You sent two emails to ct-policy@, though I only approved this one. The other one was a thread which included a bunch of personal information I presume you did not intend to post. It did, however, include an outreach email Google sent to you in March of 2023 (nearly three years ago) that stated clearly that the log list would stop working. We've extended the timeline several times since then to give folks a greater opportunity to adapt. While we're never keen to see app breakage, we're unable to extend our deadlines further.

If you'd like to avoid breakage, you should update your app immediately to remove the appmattus dependency. I acknowledge that users take a long time to update applications, and these users will not naturally update in time. If your app is distributed through the Google Play Store, we recommend using Play Console's mechanism to prompt users to update their installations.

Please note that updating to the latest version of the appmattus library is not sufficient to avoid application breakage. The appmattus library is not authorized to depend on the v2 or v3 log list, and the most recent version of the library will also stop working in the coming months.

Best,

Joe, on behalf of the Chrome CT team.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/CAL%3D9YSUTKOojLw8sXecDZevbbX9NChE3Tu_-d28fxanyweRtMw%40mail.gmail.com.

[方一格（Ethan）]

Hi all,

Thanks for your response and your attention to this issue.

To Ben's question, for business reasons I’m not allowed to share my company’s information in this public channel. As of now, some of our customers have encountered this issue (around 2 thousand devices), but it has been temporarily resolved by updating certificates before V2 was completely discontinued. The situation in the future will be unpredictably bad.

To Joe's reply, as I mentioned above, we are a commercial Android device company. Our devices are sold to both business-to-business and business-to-consumer customers. This means that the vast majority of manufactured and sold devices are in a non-active state. Once the V2 log server stoped updates, our device service application, which relies on the third-party library appmattus for verification, will directly fail network requests, which will turn devices into bricks like Ben said. From a practical standpoint, the effeciency of our OTA updates is clearly insufficient.

Our current repair solution for this service application has already abandoned appmattus completely. The key point is that the update solution won't fix the issue after the V2 log list stops updating.

We understand that appmattus does not recommend using it and are aware of Google's latest CT policy. We simply need time to resolve the current issues. At present, the previously mentioned suggestion regarding the V2 log list appears to be the only way to buy us time.

Best,  
Ethan.

[Joe DeBlasio]

Thanks, Ethan.

It is possible for you to further delay breakage of these applications. While the current v2 log list does not contain the CT logs most recently added to Chrome's log list, it does still contain operational logs identified as "usable" that can satisfy appmattus policy requirements.

Assuming the appmattus library fully supports RFC6962, any certificate expiring before 2027 can be used if you configure your server to present SCTs using the TLS extension outlined in RFC6962 section 3.3.1. If the appmattus library does not support this mechanism, you should still be able to work with your CA to receive a certificate with the necessary SCTs embedded.

This technique will already postpone breakage by almost a full year, so it does not appear that adding new logs to the v2 log list is necessary.

Best,

Joe

[方一格（Ethan）]

Hi Joe,

Thank you for your suggestion. However, our technical team has already discussed this approach. We are unable to deliver SCTs via the TLS extension because the intermediate Wangsu CDN terminates the SSL connection and does not propagate the SCTs. 

Therefore, based on our team's analysis, we understand that directly migrating CT log list data from V3 to V2 cannot cover the tiled_logs portion. Is it ok to migrate CT log list data from V3 to V2 while simultaneously unfolding the tiled_logs content? Or we could find a solution that benefits both parties. 

Additionally, what is Google's exact timeline for when the V2 log service will begin returning 404 responses? We have downgrade logic for network requests, in this case most of our devices will still be functional.

Looking forward to replys.

Best,

Ethan

[Joe DeBlasio]

Hi Ethan,

Addressing your questions and comments somewhat out of order:

Additionally, what is Google's exact timeline for when the V2 log service will begin returning 404 responses? We have downgrade logic for network requests, in this case most of our devices will still be functional.

We will not remove the v2 log list (i.e. make its endpoint return HTTP 404) before 2027. At that time, we expect it will be safe to remove the list, as the expiry windows for all logs included in the current list will have passed and it will thus not be possible to procure SCTs from any logs included in the list.

Or we could find a solution that benefits both parties.

While I'm deeply sympathetic to the pain this will cause you and your team, our experience has taught us that extending turndown deadlines does not benefit us, nor the ecosystem broadly. Each time a deadline approaches, a new batch of developers realizes that they will be negatively impacted, and asks for further postponement.

When we fully turn down the list in 2027, about 4.5 years will have passed since the initial announcement of the planned turn down. Notifying impacted developers is extremely difficult, and many developers missed this announcement. However, all impacted applications were broken for several hours in early 2023 when we first attempted to turn the list down. That same day, the most commonly used library was updated to remove reliance on the v2 log list. Developers will have had 4 years to update their applications.

We thus feel strongly that we have provided adequate notice, and that there are diminishing returns to further postponement.

Therefore, based on our team's analysis, we understand that directly migrating CT log list data from V3 to V2 cannot cover the tiled_logs portion. Is it ok to migrate CT log list data from V3 to V2 while simultaneously unfolding the tiled_logs content?

As noted, we do not intend to update the v2 log list with updated contents from the v3 list. However, even if we were to do so, we could not safely add the contents of the tiled_logs arrays to the logs arrays. We do not have confidence that all v2 log list consumers will gracefully handle SCTs that contain an unexpected leaf_index extension, thus combining the arrays would risk creating more unexpected breakage ahead of our announced timelines.

We are unable to deliver SCTs via the TLS extension because the intermediate Wangsu CDN terminates the SSL connection and does not propagate the SCTs.

I assume that migrating away from Wangsu CDN is not an option. That unfortunately suggests that your best option is to work with Wangsu CDN to see if they may be able to provide the necessary SCTs via the TLS extension or by using TLS certificates with the necessary SCTs embedded.

Best,

Joe

[方一格（Ethan）]

Hi Joe,

Thank you for the suggestion, As you mentioned "We will not remove the v2 log list (i.e. make its endpoint return HTTP 404) before 2027", "When we fully turn down the list in 2027" do you mean the official, precise date is the end of 2026? If not, could you provide the specific date, like January 2027 ?

After a long time conclude the risk and current situation, the cooperation with Wangsu CDN is not an possible option. However we have planed many solutions for devices of different level risk. 

For high risk level devices, we plan to update the target application via remote control.

For other risk level devices, we plan to use the downgrade logic, which means when V2 log service request return 404, app will automaticly skipped to ensure network functionality.

We already start sloving our high risk level and online devices. However, devices stored in customer's warhouse remain at risk of failure, thus we would like to know whether Google is willing to cooperate with us in removing the V2 service when we sloved all high risk level devices by the end of January 2027, so the devices will go downgrade logic, it will be a very great help for sloving our issues. 

Best,

Ethan.

[Jasper]

Hi Joe,

We also face the same issue, also the best solution is to make sure the request of V2 log list service returning the 404 code, is there any specific time to share?

Best,

Jasper.

[Joe DeBlasio]

Please see our updated plan, just announced, regarding v2 and v3 log list dependencies from 3p CT enforcement libraries: https://googlechrome.github.io/CertificateTransparency/3p_libraries.html

We will not be 404ing the log lists as past experience has demonstrated that doing so causes some applications to break in a way that can not be mitigated.

Best,

Joe

[方一格（Ethan）]

Hi Joe，

Thank you for the responed and the new announctment! 

Our tech team has begun researching and discussing the mimic logs solution in your announcement. We are attempting to contact the CA agency.

However, Could you provide completion time for adding of mimic logs. So we can have our team make the necessary preparations and plans including device production and public relations efforts.

We really appreciate your long-term attention to this matter. It means a lot.

Best,

Ethan.

On behalf of our company.

[T L]

  Hi Joe,

Great to see we’re hitting the exact same issue! Do you know when the two new CT log “mimics” mentioned in the article will be deployed?  

在2026年1月23日星期五 UTC+8 00:56:01<Joe DeBlasio> 写道：

[T L]

 Hi, Joe  

Thank you for the solution you provided—it is an excellent and elegant one that can fully resolve the issues encountered by the existing devices in our current market.
We have contacted several CA (Certificate Authority) organizations for consultation, and currently, some CAs are willing to customize certificates for us and embed SCTs (Signed Certificate Timestamps).
We would like to know approximately when the two new CT log “mimics” will be updated, as we need to coordinate the development and testing schedules between our internal developers and the CA organizations.

 
Best,
TL