Optimizing RFC 6962 CT Log Data Acquisition Efficiency Based on the Static CT API Format

[Xiaoming Yang]

1. Problem Statement
    For a long time, while operating RFC 6962 CT log servers, we have faced severe performance challenges with the usage patterns of the get-entries endpoint:
        Low Cache Hit Rate: Due to the randomness and uncertainty of the start and end parameters, the cache hit rate at both the CDN and gateway levels is extremely low, preventing the cache from delivering its acceleration benefits.
        Huge Backend Pressure: Every request requires a range query in the backend database and real-time data compression. This not only causes a high database I/O load but also significantly increases the Gzip processing overhead on the gateway.
        Inefficient Bandwidth Utilization: Frequent data requests from certain monitors result in excessively high network transmission bandwidth consumption, exhibiting a long-tail effect.
        Retrieval Latency Degrades with Data Growth: As the scale of CT logs continues to expand, the retrieval latency for monitors keeps climbing, which adversely affects the real-time availability of monitor data.

2. Solution Strategy
    Given that the Static CT API has been recognized within the CT community and mainstream monitor tools have already supported this format, we have decided to deploy a set of data acquisition interfaces that comply with the Static CT API standard .
    Core Logic: Leverage the static, immutable, and pre-cacheable characteristics of Tile data to alleviate backend database query pressure and network bandwidth bottlenecks through CDN caching.

3. Implementation
    Deploy an HTTP service specifically dedicated to serving Tile-formatted data.
    3.1 Architectural Design
        Interface Design:  
            /tile/data/<N>: Provides standard Tile block data.      
            /issuer/<SHA256>: Provides CA issuer information.
            Interfaces and data formats:
                 https://github.com/C2SP/C2SP/blob/main/static-ct-api.md#log-entries
                 https://github.com/C2SP/C2SP/blob/main/static-ct-api.md#issuers
            Among these, the /tile/data interface only supports the format of /tile/data/<N> (i.e., full tiles)  and does not support the [.p/<W>] parameter (partial tiles).

        Data Flow:
            Data Acquisition: The HTTP service communicates with the trillian_log_server via the gRPC protocol to retrieve raw data segments.
            Format Reassembly: The service repackages the JSON data retrieved via gRPC from the RFC 6962 format into the Tile format.
            Caching and Distribution: The packaged Tile data is distributed and cached by CDN nodes.

    3.2 Verification Mechanism
        After a monitor retrieves the Tile data, the processing workflow is as follows:
            Parsing: Parse the Tile data packets according to the Static CT API format.
            Auditing and Verification: Monitors must independently compute the Merkle Tree Leaf Hashes based on the RFC 6962 specification to ensure the consistency proofs and audit.

4. Expected Benefits
    Community Benefits: Monitors can sync full/incremental data much faster, meeting the demands of continuously growing log scales in the future.
    Performance Improvement: With CDN caching, data retrieval latency will be dramatically reduced from "database processing time + transmission time + bandwidth congestion" down to "CDN edge response time".
    Efficiency Gains:
        Lowers database I/O utilization for the operator.
        Enhances overall service stability.

[Xiaoming Yang]

We have deployed this interface at ct2026-a.trustasia.com/log2026a and ct2026-b.trustasia.com/log2026b.
With this interface, each request can retrieve 256 entries without modifying our configuration.

Request Examples:
```bash
curl -v https://ct2026-a.trustasia.com/log2026a/tile/data/000 --output log2026a_000
curl -v https://ct2026-b.trustasia.com/log2026b/tile/data/000 --output log2026b_000
```

[Rob Stradling]

This is great!

Are you intending to implement /checkpoint too?

Or are you envisaging that each monitor will adopt a hybrid RFC6962/StaticCT strategy specifically for TrustAsia's logs - i.e., call /ct/v1/get-sth, then fetch tiles ?

---

From: ct-p...@chromium.org <ct-p...@chromium.org> on behalf of Xiaoming Yang <xiaomi...@trustasia.com>
Sent: 29 May 2026 12:18
To: Certificate Transparency Policy <ct-p...@chromium.org>
Cc: Xiaoming Yang <xiaomi...@trustasia.com>
Subject: [ct-policy] Re: Optimizing RFC 6962 CT Log Data Acquisition Efficiency Based on the Static CT API Format

 

We have deployed this interface at ct2026-a. trustasia. com/log2026a and ct2026-b. trustasia. com/log2026b. With this interface, each request can retrieve 256 entries without modifying our configuration. Request Examples: ```bash curl -v https: //ct2026-a. trustasia. com/log2026a/tile/data/000

ZjQcmQRYFpfptBannerStart

This Message Is From an External Sender

This message came from outside your organization.

Report Suspicious

 

ZjQcmQRYFpfptBannerEnd

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/917c0ada-4415-4b0f-922c-791c1f11fc29n%40chromium.org.

[Xiaoming Yang]

Our original intention in designing the tile/data endpoint was to provide a more efficient data retrieval interface, ensuring that monitors that need faster data access can obtain it in a timely manner.

We believe that for pure data retrieval, one can determine whether the maximum tree_size has been reached simply by checking the HTTP status codes (200 vs. 404) of the tile/data endpoint. Furthermore, we currently do not serve the final, incomplete tile data via tile/data.

The /checkpoint endpoint involves cryptographic signing using a private key, and we are uncertain whether we can (or should) use the current private key to sign the data for this additional endpoint. Note that our implementation does not involve modifying Trillian's codebase; instead, it is achieved via a separate application that retrieves Trillian's results using gRPC.

We would love to hear feedback and suggestions from the community and the monitors on this matter.

[Jonas Meller]

Hi Xiaoming,

I've been struggling to keep up with TrustAsia's log2026a and log2026b so this is a helpful addition.

One heads-up for monitors:

The data-tile timestamped_entry includes the Static CT leaf_index extension, and monitors need to strip it before re-encoding leaf_input for Merkle verification. Otherwise, the computed root will not match the RFC6962 STH.

Best regards,

Jonas Meller

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/e5eff67b-7714-4a81-9647-20c401e984a2n%40chromium.org.

[Filippo Valsorda]

2026-07-09 08:30 GMT+02:00 'Jonas Meller' via Certificate Transparency Policy <ct-p...@chromium.org>:

Hi Xiaoming,

I've been struggling to keep up with TrustAsia's log2026a and log2026b so this is a helpful addition.

One heads-up for monitors:

The data-tile timestamped_entry includes the Static CT leaf_index extension, and monitors need to strip it before re-encoding leaf_input for Merkle verification. Otherwise, the computed root will not match the RFC6962 STH.

This is not ideal: ct-archive already encodes RFC 6962 logs as tiles, but does so without the extension. Not including the extension requires less workaround (because adding it breaks the leaf hash, as you found out) and can rely on existing software: https://github.com/geomys/ct-archive#using-the-archives

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/704a9290-3b66-4957-94f3-97cf4680fb17%40app.fastmail.com.

[Xiaoming Yang]

Hi Jonas,

We have implemented a tile/data endpoint over our RFC 6962-compliant CT service. Our goal is to provide a more efficient data-serving interface without breaking the data format boundaries between the RFC 6962 get-entries endpoint and the Static CT API's tile/data endpoint.

In our previous discussion with Rob, the question of whether we need to implement the checkpoint endpoint was also raised.

Our initial objective was simply to build a more efficient and stable data delivery interface than the current get-entries. We are now internally evaluating whether to provide a full set of Static CT API endpoints for our RFC 6962 CT service. From a technical perspective, we have basically confirmed our capability to do so. However, our main concern is whether the private key used for the RFC 6962 CT service can (or should) be reused to sign data for this newly implemented checkpoint endpoint. We strongly prefer not to create an additional private key solely for checkpoint signatures.

For context, we have not modified the core business logic of our deployed Trillian (https://github.com/google/trillian) or CTFE (https://github.com/google/certificate-transparency-go/tree/master/trillian/ctfe/ct_server) instances. The tile/data endpoint was implemented entirely via auxiliary code.

Additionally, we are planning an improvement to our existing RFC 6962 get-entries endpoint. Currently, our CT service is configured with max_get_entries=32. On top of this, we want to introduce an exceptional rule: if the start and end parameters of an HTTP GET request to get-entries satisfy both end - start == 255* and start % 256 == 0, the endpoint will return all 256 entries within that range.

We have always hoped that client requests to get-entries could align their start and end parameters to fixed multiples (chunk sizes). This would be the simplest way for us to leverage highly efficient CDN caching and data delivery.

We would love to hear your thoughts or concerns regarding this approach, especially on the private key reuse.

Thanks.

[Jonas Meller]

I have added a few comments inline below.

On Mon, Jul 13, 2026, at 05:26, 'Xiaoming Yang' via Certificate Transparency Policy wrote:
> We have implemented a tile/data endpoint over our RFC 6962-compliant CT service. Our goal is to provide a more efficient data-serving interface without breaking the data format boundaries between the RFC 6962 get-entries endpoint and the Static CT API's tile/data endpoint.
>
> In our previous discussion with Rob, the question of whether we need to implement the checkpoint endpoint was also raised.
>
> Our initial objective was simply to build a more efficient and stable data delivery interface than the current get-entries. We are now internally evaluating whether to provide a full set of Static CT API endpoints for our RFC 6962 CT service. From a technical perspective, we have basically confirmed our capability to do so. However, our main concern is whether the private key used for the RFC 6962 CT service can (or should) be reused to sign data for this newly implemented checkpoint endpoint. We strongly prefer not to create an additional private key solely for checkpoint signatures.

I think Filippo and others are better suited to answer this, but one relevant point from https://github.com/C2SP/C2SP/blob/main/static-ct-api.md is: "Note that a RFC 6962 get-sth response can be converted to a checkpoint (with knowledge of the origin line) and vice-versa without access to the private key."

> For context, we have not modified the core business logic of our deployed Trillian (https://github.com/google/trillian) or CTFE (https://github.com/google/certificate-transparency-go/tree/master/trillian/ctfe/ct_server) instances. The tile/data endpoint was implemented entirely via auxiliary code.
>
> Additionally, we are planning an improvement to our existing RFC 6962 get-entries endpoint. Currently, our CT service is configured with max_get_entries=32. On top of this, we want to introduce an exceptional rule: if the start and end parameters of an HTTP GET request to get-entries satisfy both end - start == 255* and start % 256 == 0, the endpoint will return all 256 entries within that range.

One problem with this approach is that it will be hard for monitors to discover that it is actually possible to get 256 entries per request, and not just 32. My monitoring code does align both start and end, but to do that it needs to know the max chunk size. It tries to figure that out dynamically by first requesting something like start=0&end=10000. If I understand you correctly, that would only return 32 entries, so I would instead have to add explicit configuration for your logs to use 256.

> We have always hoped that client requests to get-entries could align their start and end parameters to fixed multiples (chunk sizes). This would be the simplest way for us to leverage highly efficient CDN caching and data delivery.

One approach I think I have seen other logs take is that they try to help the client get aligned. For example, a log with a max chunk size of 32 might return only two entries for the request start=30&end=61, so that the next request will have the aligned start=32. Unfortunately, that does not help with end if the client is using the tree size there.

[Colin Stubbs]

Thanks Xiaoming,

This is great. The rate limits on the TrustAsia logs and hosting approach is quite frustrating.

A thought though... given there doesn't seem to be any formal spec for your specific approach to hybrid log interface... 

Eventually the log will cease accepting entries and it's unlikely the final entry will fall on an exact full tile compatible boundary.

If there's no partial tiles, hence no trailing partial tile with the final entries, are you expecting monitors to be smart enough to detect the rejected state and revert to using RFC6962 API in order to obtain the final entry/entries?

-Colin

[Xiaoming Yang]

Hi Colin,

We are currently working to improve our data transfer efficiency. At present, our entire CT service handles an average of 400 to 500 million requests per day. Our goal is to cache more data on the CDN and increase the CDN cache hit ratio.

Regarding the final tile, once this shard reaches the end of its operational period, we will consider adding the .p/<W> endpoint for it, or alternatively, we may use our monitoring certificates to pad the tree size to a multiple of 256.

[Xiaoming Yang]

hi Jonas,

The JSON-formatted response from the RFC 6962 get-sth endpoint can be converted into text-formatted checkpoint data. However, because the Static CT API uses the CtExtensions field in TimestampedEntry, the TreeHead and leaf hashes between the two are incompatible. We do not want to transmit RFC 6962-format data over the Static CT API transport channel. The reason we are so concerned about the signature issue is that we want to generate checkpoints strictly in accordance with the Static CT API standard.

After deploying these Static CT API endpoints, we observed that crt.sh is already using this interface to fetch data, and it seems to be working quite well.

Regarding the issue of probing the response size for the get-entries endpoint, once deployed, we can consider coding this as a special case into the fetching client.

[Chris Hartwig]

Hi Xiaoming and TrustAsia team,

I just wanted to add my thanks for deploying the Static CT tile endpoints for log2026a and log2026b. This is a huge help to SSLBoard. We’d been struggling to catch up with these logs, and the tile endpoints makes the process way faster. It’s great to see TrustAsia investing in a practical improvement that helps monitors. Thank you for sharing the work with the community.

Best,
Chris
SSLBoard

[Andrew Ayer]

On Tue, 14 Jul 2026 18:10:30 -0700 (PDT)

"'Xiaoming Yang' via Certificate Transparency Policy"

<ct-p...@chromium.org> wrote:

> The JSON-formatted response from the RFC 6962 get-sth endpoint can be
> converted into text-formatted checkpoint data. However, because the
> Static CT API uses the CtExtensions field in TimestampedEntry, the
> TreeHead and leaf hashes between the two are incompatible. We do not
> want to transmit RFC 6962-format data over the Static CT API
> transport channel. The reason we are so concerned about the signature
> issue is that we want to generate checkpoints strictly in accordance
> with the Static CT API standard.

You should omit the leaf_index extension. Although this technically isn't compliant with the static-ct-api spec, these logs will never be fully compliant because the SCTs don't have the extension. It's much more important for the SCTs, leaves, and checkpoints to all match. As Filippo said, there's already precedent for omitting the leaf_index extension when the log is a mirror/archive of an RFC 6962 log.

You MUST NOT use the log's key to sign any checkpoints incorporating leaves with the leaf_index extension. This would create a split view and require the log to be retired.

Regards,
Andrew

[Xiaoming Yang]

Hi all,

Regarding the Static CT API format, it currently appears that monitors face no significant issues using our deployed Static CT API endpoints. The requirement for verifying the tree head can still be met via the get-entries endpoint. Therefore, we will maintain the current Static CT API format and will not implement data format compatibility between the two standards (RFC 6962 and Static CT API).

Additionally, we will not provide checkpoints in the Static CT API format; we will continue to provide the tree head and signature exclusively via the get-sth endpoint. The Static CT API format is intended for more efficient data retrieval. We highly encourage the use of the Static CT API endpoints, as the data format is theoretically more compact and offers better transmission efficiency.

We have also implemented the following rules for our currently deployed get-entries endpoints:

    1. If a request strictly matches start % 256 == 0 AND end - start == 255, it will return exactly 256 entries.
        Format: "https://(ct2026-a|ct2026-b|hetu2027).trustasia.com/(ct2026-a|ct2026-b|hetu2027)/ct/v1/get-entries?start=(N*256)&end=((N+1)*256-1)" where N = [0, 1, 2, 3...]
    2. If a request matches start % 32 == 0 AND end - start >= 31 AND end - start != 255, it will return exactly 32 entries.
    3. Our CDN is configured to only cache responses containing exactly 32 or 256 entries.

The issue we are currently facing:
The network bandwidth between our CDN and our IDC is being heavily saturated by a massive volume of unaligned requests. This has resulted in reduced data transmission rates and longer response times for individual requests. Our solution is to strongly encourage clients to align their start and end parameters and to optimize our CDN configuration to cache data at the edge as effectively as possible.

To encourage aligning the start and end parameters to these block sizes, we will be introducing a global shared rate limit of 400 requests per second (rps) for all get-entries requests that do not fall into either of the two optimized alignment categories mentioned above ([start % 256 == 0 AND end - start == 255] OR [start % 32 == 0 AND end - start >= 31 AND end - start != 255]). Requests that do comply with these block boundaries will not be subject to this global shared limit, but will only be subject to the standard per-IP rate limits.

For reference, our current primary rate-limiting rules are as follows:

    get-entries: 10 rps per IP
    tile/data: 15 rps per IP
    add-chain: 20 rps per IP
    add-pre-chain: 30 rps per IP
    add-chain & add-pre-chain combined: Global shared limit of 400 rps

Request Examples:
```bash
curl -v "https://ct2026-a.trustasia.com/log2026a/ct/v1/get-entries?start=0&end=255"
curl -v "https://ct2026-a.trustasia.com/log2026a/ct/v1/get-entries?start=256&end=511"

curl -v "https://ct2026-b.trustasia.com/log2026b/ct/v1/get-entries?start=0&end=255"
curl -v "https://ct2026-b.trustasia.com/log2026b/ct/v1/get-entries?start=256&end=511"

curl -v "https://hetu2027.trustasia.com/hetu2027/ct/v1/get-entries?start=0&end=255"
curl -v "https://hetu2027.trustasia.com/hetu2027/ct/v1/get-entries?start=256&end=511"
```

Using the test below, when transferring the same number of entries, the amount of data transmitted via the tile/data API is only one-third of that transmitted via the get-entries API.
```bash
curl --compressed "https://ct2026-a.trustasia.com/log2026a/ct/v1/get-entries?start=2239066112&end=2239066367" > 256.json
curl --compressed "https://ct2026-a.trustasia.com/log2026a/tile/data/x008/x746/352" > 001
curl -H "Accept-Encoding: gzip" "https://ct2026-a.trustasia.com/log2026a/ct/v1/get-entries?start=2239066112&end=2239066367" > 256.json.gz
curl -H "Accept-Encoding: gzip" "https://ct2026-a.trustasia.com/log2026a/tile/data/x008/x746/352" > 001.gz

ls -alh
332K Jul 17 07:59 001
196K Jul 17 07:59 001.gz
1.7M Jul 17 07:59 256.json
667K Jul 17 07:59 256.json.gz
```

And also we deployed this for hetu2027.trustasia.com
Request Examples:
```bash
curl -v "https://hetu2027.trustasia.com/hetu2027/tile/data/000"  --output hetu2027_000
```

On Friday, May 29, 2026 at 7:10:14 PM UTC+8 Xiaoming Yang wrote:

[Andrew Ayer]

On Fri, 17 Jul 2026 01:25:24 -0700 (PDT)

"'Xiaoming Yang' via Certificate Transparency Policy"
<ct-p...@chromium.org> wrote:

> Regarding the Static CT API format, it currently appears that
> monitors face no significant issues using our deployed Static CT API
> endpoints. The requirement for verifying the tree head can still be
> met via the get-entries endpoint.

Lest all the monitor feedback appear positive, SSLMate/certspotter cannot use your new endpoints because I view verifying the entries against the STH to be a hard requirement for a monitor, and writing TrustAsia-specific verification logic is not appealing.

> Our solution is to strongly encourage clients to align their
> start and end parameters and to optimize our CDN configuration to cache
> data at the edge as effectively as possible.

I don't think your CDN caching is working. Aligned get-entries requests are still consistently slow:

curl --compressed 'https://ct2026-a.trustasia.com/log2026a/ct/v1/get-entries?start=2240452608&end=2240452863' > /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 679k 0 679k 0 0 49065 0 --:--:-- 0:00:14 --:--:-- 66311

curl --compressed 'https://ct2026-a.trustasia.com/log2026a/ct/v1/get-entries?start=2240452608&end=2240452863' > /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 679k 0 679k 0 0 206k 0 --:--:-- 0:00:03 --:--:-- 206k

curl --compressed 'https://ct2026-a.trustasia.com/log2026a/ct/v1/get-entries?start=2240452608&end=2240452863' > /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 679k 0 679k 0 0 96323 0 --:--:-- 0:00:07 --:--:-- 138k

In contrast, the equivalent tile request is consistently fast:

curl --compressed 'https://ct2026-a.trustasia.com/log2026a/tile/data/x008/x751/768' > /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 332k 0 332k 0 0 15.6M 0 --:--:-- --:--:-- --:--:-- 16.2M

curl --compressed 'https://ct2026-a.trustasia.com/log2026a/tile/data/x008/x751/768' > /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 194k 0 194k 0 0 9661k 0 --:--:-- --:--:-- --:--:-- 9702k

curl --compressed 'https://ct2026-a.trustasia.com/log2026a/tile/data/x008/x751/768' > /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 194k 0 194k 0 0 7875k 0 --:--:-- --:--:-- --:--:-- 8085k

Regards,
Andrew

[Filippo Valsorda]

Another issue of adding the extra extension, besides introducing an unspecified format in the ecosystem, is that the tiles can’t be used to produce an archive for ct-archive without being reencoded.

If the thesis is that monitors don’t check inclusion in the STH (which is a sad state of affairs and not something to encourage), why not stop including the extension? Nothing should break if nothing was doing strict checks.

-- 

You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/20260720090255.0031c5c36fc8472262bb2767%40andrewayer.name.

[Xiaoming Yang]

Hi Andrew,

Based on our previous discussions and recent observations, we would like to share some insights regarding the data provisioning of our CT log services:

    For monitors that only focus on certificate data: We highly recommend retrieving data via our deployed tile/data endpoint. Based on current production metrics, tile/data delivers data faster and with higher efficiency. Additionally, we have configured a higher rate limit (QPS) for this endpoint.
    For monitors that prioritize data consistency and wish to retrieve data faster using the get-entries endpoint: You only need to strictly define the start and end parameters. This will allow you to successfully retrieve data using a batch size of 256.
    For monitors that do not strictly define the start and end range: There are no significant changes in behavior.

We are currently conducting further testing and deployment to optimize data transfer and CDN cache hit rates.

We previously observed that requests with the exact same start and end combinations are often initiated from different IP addresses within a short time window (approx. 5 to 10 minutes). Once these requests are cached by the CDN, it significantly reduces our IDC's egress bandwidth usage, which in turn allows uncached requests to be served faster to their originators.

Please note that the behavior of the CDN provider's edge nodes and intermediate transit nodes is somewhat unpredictable to us, and we cannot directly control or manipulate the caching behavior of specific CDN edge nodes.

Regarding your three get-entries requests to log2026a:

    The first request took 17 seconds: This was because the CDN edge node serving your request did not have this specific query cached, causing the request to go all the way back to our origin IDC (cache miss).
    The second request took 3 seconds: Based on our test, this request successfully hit the CDN cache.
    The third request took 7 seconds: This was likely because DNS resolved your request to a different CDN edge node, which had to fetch and rebuild the cache from our origin.

For the same range of certificates, the payload size of the get-entries endpoint is more than three times larger than that of the tile/data endpoint. This larger payload translates to longer transit times, which is why get-entries appears slower.

Lastly, we observed an traffic pattern recently. Prior to July 6, our CT services received approximately 600 million requests daily. Starting July 7, this number began to decline, dropping to just over 300 million daily requests by July 20. Through our ongoing series of optimizations, we aim to minimize the bandwidth usage between the CDN and our origin IDC. Our ultimate goal is to ensure that every request can be served from our infrastructure to the requester within a stable and predictable timeframe.