Cross-Validating Linters [zlint]

[Wayne]

I've been doing some preliminary research into gaining anything of substance out of AI models. My approach was to consider what their best use-case is: using tools and aggressively self-validating by building unit tests on top of each other. I also just described a linter.

Due to this I put some focus on a from RFCs/BRs only homebrewed linter on what a pedantic Claude would do, and with a very focused 21k certificate corpus. The focus has been primarily zlint to date, and I think enough ground has been covered on that front to present the data for upstream fixes where applicable.

This also independently rediscovered a variety of existing issues, alongside expectations on what zcrypto can do that is misleading the developers on what a lint is actually doing.

Currently zlint has 431 lints, of which in the corpus 333 could fire. Cross-validating the output of the homebrew linter vs zlint and judging every outcome got us to here. This isn't an exhaustive check, just where divergences occurred so there could be shared mistakes as well.

Hopefully someone finds the details in here interesting, as the rest of pkimetal's supported linters are next.

[Adriano Santoni]

Wayne,

I find your work interesting and useful. 

Some of the issues that emerge from it are of relatively low importance, IMO, but others would certainly be worth addressing.

Adriano

Il 19/08/2026 18:01, Wayne ha scritto:

External sender. Verify for authenticity before opening any links or attachments.

--
You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org .
To view this discussion visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/1dd177c5-0075-4d4e-8189-52de16387501n%40mozilla.org .

[Wayne]

Yes there are a lot of low-importance items there, but not including them wouldn't be entirely honest thus the full disclosure and category breakdown. The pedantic date cases are ones I was less than enthused about as well, but if you want to know what you'll get on such a research project then here's the results.

There is a lot of reliance on the current linting tooling, and being aware of their individual pitfalls gives us room to find gaps to fill. Do not be under the impression this is some silly plan to release a vibecoded linter - even if its significantly faster than the existing ones and can do what LE hinted at in #2038351 C21. It's approaching the same set of rules and seeing where our presumptions don't match where a corpus of actual incidents is used, and the friction there is interesting. It's a numbers game of running thousands of tests and finding the outliers, so very slow and tedious work that can be automated in a fashion that brings replicable and addressable issues.

For zlint it's worth being aware that in production it errs on faulting when finding 1 malformed certificate. That means if you're doing any batch processing of post-issuance you won't get full coverage without planning around that. I'd also have to recommend what people have been saying for years: disable the lint_rsa_fermat_factorization by default, or optimise the rounds for your use-case. It's hyper-specific and very very costly as a lint - 44% gains with it disabled. For instance FTFY is considered 'costly' in PKIMetal terms, but is 3% runtime when directly implemented across all fields.

The test corpus in zlint also having unmutated certificates that are clean and need internal mutations in the test cases is also a bizarre choice. Full disclosure the linter built against it is in Rust, but that was security properties and that the x509-parser was practically built for linting compared to any other library being used currently.

I consider this output to be functionally similar to a pure fuzz and crashdump analysis, informative to developers but not to be considered in isolation. While I have personal opinions on individual lint choices and categories, those are decisions better left to developers who know what their intended coverage and use-case are. There is a lot of RFC minutae that is not fully implemented in linters and gets hits, and for CAs who say they want to help on linters consider this a simple starting list to check the box.

As always my inbox is open to queries, however the corpus is only generally sorted.

[Wayne]

Okay finished up checking the other linters, attached an updated version of the zlint findings + pkilint, x509lint, and certlint. I did have to send one of these out-of-band so there's a new version of certlint (1.8.6) as of 9 hours ago for CT-035.

I am a bit concerned over the amount of testing the linters in generally have had historically given the low-hanging fruit that dropped here. Nevertheless I hope this gives some good ground for CAs to try and help improve the linters we use today. There's even a document of feasible new lints at the head of the directory, but I think there should be some focus on cleaning up the existing ones especially where the lints aren't actually functioning.

And MISSING-LINTS.md hints at the approach to get fairly reliable cross-sign and root re-use checks functional in a linter. So here's how it was implemented:
- Pull CCADB data, both AllCerts and PEMs (reconstruct via REST api)
- On linting if it's a root/subordinate then lookup the CCADB data (keeping in mind its base64) and find the oldest SPKI/Subject match
- branch accordingly

With a data/ source under the linter it can do that dynamically, and if you poll CCADB at most once a day it's all self-contained. The data also allows checking for root-program applicability for specific lint checks. Or just dynamically polling for CRL disclosure off of the CT logs, and other per-issuer checks that would otherwise be retrospective and hard to surface.

There's also a lot of experimental features I'm trying out detailed in the notes. If you wondered what firehose-level cert analysis can pull off - it's quite a lot more than you'd think.