PEM of root certs in Mozilla's root store

[Kathleen Wilson]

All,

I've been asked to publish Mozilla's root store in a way that is easy to
consume by downstreams, so I have added the following to
https://wiki.mozilla.org/CA/Included_Certificates

CCADB Data Usage Terms
<https://www.ccadb.org/rootstores/usage#ccadb-data-usage-terms>

PEM of Root Certificates in Mozilla's Root Store with the Websites
(TLS/SSL) Trust Bit Enabled (CSV)
<https://ccadb-public.secure.force.com/mozilla/IncludedRootsPEM?TrustBitsInclude=Websites>

PEM of Root Certificates in Mozilla's Root Store with the Email (S/MIME)
Trust Bit Enabled (CSV)
<https://ccadb-public.secure.force.com/mozilla/IncludedRootsPEM?TrustBitsInclude=Email>


Please let me know if you have feedback or recommendations about this.

Thanks,
Kathleen

[Ryan Sleevi]

It seems like there should be a link to
https://wiki.mozilla.org/CA/FAQ#Can_I_use_Mozilla.27s_set_of_CA_certificates.3F
there

I realize there’s a tension between making this easily consumable, and the
fact that “easily consumed” doesn’t and can’t relieve an organization of
having to be responsible and being aware of the issues and discussions here
about protecting their users.

I do worry this is going to encourage one of the things that can make it
more difficult for Mozilla to protect Mozilla users, which is when vendors
blindly using/build a PEM file and bake it into a device they never update.
We know from countless CA incidents that when vendors do that, and aren’t
using these for “the web”, that it makes it more difficult for site
operators to replace these certificates. It also makes it harder for
Mozilla to fix bugs in implementations or policies and otherwise take
actions that minimize any disruption for users. At the same time, Mozilla
running a public and transparent root program does indeed mean it’s better
for users than these vendors doing nothing at all, which is what would
likely happen if there were too many roadblocks.

While personally, I want to believe it’s “not ideal” to make it so easy, I
realize the reality is plenty of folks already repackage the Mozilla store
for just this reason, totally ignoring the above link, and make it easy for
others to pull in. At least this way, you could reiterate that this list
doesn’t really absolve these vendors of having to keep users up to date and
protected and be able to update their root stores for their products, by
linking to
https://wiki.mozilla.org/CA/FAQ#Can_I_use_Mozilla.27s_set_of_CA_certificates.3F

> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>

[Matthew Hardeman]

Would it be unreasonable to also consider publishing, as an "easy to use"
list, that set of only those anchors which are currently trusted in the
program and for which no exceptional in-product policy enforcement is
imposed? (TLD constraints, provisional distrusts, etc.)

The lazier implementers are going to take the raw set of anchors and none
of the policy associated, and so the default assumption should be that none
of the enhanced policy enforcements from nss or firefox would get copied
along.

[Jakob Bohm]

Please note that at least the first CSV download is not really a CSV
file, as there are line feeds within each "PEM" value, and only one
column. It would probably be more useful as a simple concatenated PEM
file, as used by various software packages as a root store input format.

I have also noted that at least one downstream root store (Debian) takes
all Mozilla-trusted certificates and labels them as simply
"mozilla/cert-public-name", even though more useful naming can be
extracted from the last (most complete) report, after finding a non-gui
tool that can actually parse CSV files with embedded newlines in string
values.




Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

[Kathleen Wilson]

On 10/6/20 7:09 PM, Ryan Sleevi wrote:
> It seems like there should be a link to
> https://wiki.mozilla.org/CA/FAQ#Can_I_use_Mozilla.27s_set_of_CA_certificates.3F
> there
>

I added that link to https://wiki.mozilla.org/CA/Included_Certificates

Thanks,
Kathleen

[Kathleen Wilson]

On 10/7/20 9:30 AM, Matthew Hardeman wrote:
> Would it be unreasonable to also consider publishing, as an "easy to use"
> list, that set of only those anchors which are currently trusted in the
> program and for which no exceptional in-product policy enforcement is
> imposed? (TLD constraints, provisional distrusts, etc.)
>
> The lazier implementers are going to take the raw set of anchors and none
> of the policy associated, and so the default assumption should be that none
> of the enhanced policy enforcements from nss or firefox would get copied
> along.
>

These reports are automatically generated by CCADB (Salesforce), so I
cannot filter out all of the exceptions that may occur or that are
currently listed in https://wiki.mozilla.org/CA/Additional_Trust_Changes

I could add a report that filters out root certificates that are
name-constrained. However, there is currently only one name-constrained
included root cert, and this option ended up not being very popular
among CAs requesting root inclusion.

Also note that in Mozilla's program being name-constrained does not
release the CA from following the same rules that all of the other CAs
have to follow.

Therefore, I'm not currently inclined to add another report to filter
out name-constrained root certs (currently just the one root cert).

Thanks,
Kathleen

[Kathleen Wilson]

On 10/7/20 1:09 PM, Jakob Bohm wrote:
> Please note that at least the first CSV download is not really a CSV
> file, as there are line feeds within each "PEM" value, and only one
> column.  It would probably be more useful as a simple concatenated PEM
> file, as used by various software packages as a root store input format.
>

Here's updated reports...

Text:

https://ccadb-public.secure.force.com/mozilla/IncludedRootsPEMTxt?TrustBitsInclude=Websites

https://ccadb-public.secure.force.com/mozilla/IncludedRootsPEMTxt?TrustBitsInclude=Email

CSV:
https://ccadb-public.secure.force.com/mozilla/IncludedRootsPEMCSV?TrustBitsInclude=Websites

https://ccadb-public.secure.force.com/mozilla/IncludedRootsPEMCSV?TrustBitsInclude=Email


If the Text reports look good, I'll add them to the wiki page,
https://wiki.mozilla.org/CA/Included_Certificates


Thanks,
Kathleen

[Jakob Bohm]

On 2020-10-12 20:50, Kathleen Wilson wrote:
> On 10/7/20 1:09 PM, Jakob Bohm wrote:
>> Please note that at least the first CSV download is not really a CSV
>> file, as there are line feeds within each "PEM" value, and only one
>> column.  It would probably be more useful as a simple concatenated PEM
>> file, as used by various software packages as a root store input format.
>>
>
>
> Here's updated reports...
>
> Text:
>
> https://ccadb-public.secure.force.com/mozilla/IncludedRootsPEMTxt?TrustBitsInclude=Websites
>
>
> https://ccadb-public.secure.force.com/mozilla/IncludedRootsPEMTxt?TrustBitsInclude=Email
>

These two are bad multi-pem files, as each certificate likes the
required line wrapping.

The useful multi-pem format would keep the line wrapping from the
old report, but not insert stray quotes and commas.

>
> CSV:
> https://ccadb-public.secure.force.com/mozilla/IncludedRootsPEMCSV?TrustBitsInclude=Websites
>
>
> https://ccadb-public.secure.force.com/mozilla/IncludedRootsPEMCSV?TrustBitsInclude=Email
>

These two are like the old bad multi-pem reports, with the stray
internal line feeds after/before the ----- marker lines, but without the
internal linefeeds in the Base64 data.

Useful actual-csv reports like IncludedCACertificateWithPEMReport.csv
would have to omit the marker lines and their linefeeds as well as the
internal linefeeds in the Base64 data in order to keep each CSV record
entirely on one line.

>
>
> If the Text reports look good, I'll add them to the wiki page,
> https://wiki.mozilla.org/CA/Included_Certificates
>
>
> Thanks,
> Kathleen
>

[Ryan Sleevi]

Jakob,

I had a little trouble following your mail, despite being quite familiar
with PEM, so hopefully you'll indulge me in making sure I've got your
criticisms/complaints correct.

Your objection to the text report is that RFC 7468 requires generators to
wrap lines (except the last line) at exactly 64 characters, right? That is,
the textual report includes the base-64 data with no embedded newlines, and
this causes your PEM decoder trouble, despite being able to easily inject
them programmatically after you download the file.

I'm not sure I fully understand the CSV objection, despite having inspected
the file, so perhaps you can clarify a bit more.

Perhaps the simplest approach would be that you could attach versions that
look how you'd want.

However, it might also be reasonable that if these concerns aren't easily
addressable, perhaps not offering the feed is another option, since it
seems a lot of work for something that should be and is naturally
discouraged.

On Tue, Oct 13, 2020 at 3:26 AM Jakob Bohm via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> On 2020-10-12 20:50, Kathleen Wilson wrote:

> > On 10/7/20 1:09 PM, Jakob Bohm wrote:
> >> Please note that at least the first CSV download is not really a CSV
> >> file, as there are line feeds within each "PEM" value, and only one
> >> column. It would probably be more useful as a simple concatenated PEM
> >> file, as used by various software packages as a root store input format.
> >>
> >
> >
> > Here's updated reports...
> >
> > Text:
> >
> >
> https://ccadb-public.secure.force.com/mozilla/IncludedRootsPEMTxt?TrustBitsInclude=Websites
> >
> >
> >
> https://ccadb-public.secure.force.com/mozilla/IncludedRootsPEMTxt?TrustBitsInclude=Email
> >

> These two are bad multi-pem files, as each certificate likes the
> required line wrapping.
>
> The useful multi-pem format would keep the line wrapping from the
> old report, but not insert stray quotes and commas.
>
> >

> > CSV:
> >
> https://ccadb-public.secure.force.com/mozilla/IncludedRootsPEMCSV?TrustBitsInclude=Websites
> >
> >
> >
> https://ccadb-public.secure.force.com/mozilla/IncludedRootsPEMCSV?TrustBitsInclude=Email
> >
>

> These two are like the old bad multi-pem reports, with the stray
> internal line feeds after/before the ----- marker lines, but without the
> internal linefeeds in the Base64 data.
>
> Useful actual-csv reports like IncludedCACertificateWithPEMReport.csv
> would have to omit the marker lines and their linefeeds as well as the
> internal linefeeds in the Base64 data in order to keep each CSV record
> entirely on one line.
>
>
>
> >
> >

> > If the Text reports look good, I'll add them to the wiki page,
> > https://wiki.mozilla.org/CA/Included_Certificates
> >
> >
> > Thanks,
> > Kathleen
> >
>
>

> Enjoy
>
> Jakob
> --
> Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com
> Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10
> This public discussion message is non-binding and may contain errors.
> WiseMo - Remote Service Management for PCs, Phones and Embedded

[Jakob Bohm]

On 2020-10-13 16:32, Ryan Sleevi wrote:
> Jakob,
>
> I had a little trouble following your mail, despite being quite familiar
> with PEM, so hopefully you'll indulge me in making sure I've got your
> criticisms/complaints correct.
>
> Your objection to the text report is that RFC 7468 requires generators to
> wrap lines (except the last line) at exactly 64 characters, right? That is,
> the textual report includes the base-64 data with no embedded newlines, and
> this causes your PEM decoder trouble, despite being able to easily inject
> them programmatically after you download the file.
>

I was commenting on the /general/ usability of the reports mentioned in
the first message on this thread, by considering what naive parsers
would do upon reading the files. As I have no direct need to parse
these files, I have no actual parser failing to do so.

My comments fell in two categories:

1. The reports that contain /only/ PEM data. I argue that the
traditional format of concatenated PEM files (as used by e.g. the
openssl command line tool) without CSV embellishments would be
preferable, and that the reports in the latest post by Kathleen
lacked the PEM line wrapping while still containing CSV
artifacts.

2. The reports that contain other data in CSV format. I argue that
those reports would be more useful without in-field line breaks, thus
having the Base64 encoded certificates as long strings without PEM
embellishments. Goal is to make them traditional CSV files with one
record per line, commas only between fields and optional double quotes
around non-numerical field values. A sample parser would be awk, cut or
the perl command line option "-F,".

Simply viewing each report in a basic text viewer should make the
problematic format deviations clear.

> I'm not sure I fully understand the CSV objection, despite having inspected
> the file, so perhaps you can clarify a bit more.
>
> Perhaps the simplest approach would be that you could attach versions that
> look how you'd want.
>

Here are the top 3 lines of IncludedCACertificateWithPEMReport.csv
reformatted (but with extra line breaks every 68 chars for posting
purposes):

"Owner","Certificate Issuer Organization","Certificate Issuer Organ
izational Unit","Common Name or Certificate Name","Certificate Seri
al Number","SHA-256 Fingerprint","Subject + SPKI SHA256","Valid Fro
m [GMT]","Valid To [GMT]","Public Key Algorithm","Signature Hash Al
gorithm","Trust Bits","Distrust for TLS After Date","Distrust for S
/MIME After Date","EV Policy OID(s)","Approval Bug","NSS Release Wh
en First Included","Firefox Release When First Included","Test Webs
ite - Valid","Test Website - Expired","Test Website - Revoked","Moz
illa Applied Constraints","Company Website","Geographic Focus","Cer
tificate Policy (CP)","Certification Practice Statement (CPS)","Sta
ndard Audit","BR Audit","EV Audit","Auditor","Standard Audit Type",
"Standard Audit Statement Dt","PEM Info"
"AC Camerfirma, S.A.","AC Camerfirma SA CIF A82743287","http://www.
chambersign.org","Chambers of Commerce Root","00","0C258A12A5674AEF
25F28BA7DCFAECEEA348E541E6F5CC4EE63B71B361606AC3","BC2FD9EA61581CB2
2BB859690D61430E7D222D1119E8C41649B9B1D556D439A4","2003.09.30","203
7.09.30","RSA 2048 bits","SHA1WithRSA","Email","","","Not EV","http
s://bugzilla.mozilla.org/show_bug.cgi?id=261778","","Firefox 1","",
"","","","http://www.camerfirma.com","Spain","","https://www.camerf
irma.com/publico/DocumentosWeb/politicas/CPS_eidas_EN_1.2.12.pdf","
https://www.csqa.it/getattachment/Sicurezza-ICT/Documenti/Attestazi
one-di-Audit-secondo-i-requisiti-ETSI/2020-03-CSQA-Attestation-CAME
RFIRMA-rev-2-signed.pdf.aspx?lang=it-IT","https://bugzilla.mozilla.
org/attachment.cgi?id=8995930","","CSQA Certificazioni srl","ETSI E
N 319 411","2020.03.05","MIIEvTCCA6WgAwIBAgIBADANBgkqhkiG9w0BAQUFAD
B/MQswCQYDVQQGEwJFVTEnMCUGA1UEChMeQUMgQ2FtZXJmaXJtYSBTQSBDSUYgQTgyN
zQzMjg3MSMwIQYDVQQLExpodHRwOi8vd3d3LmNoYW1iZXJzaWduLm9yZzEiMCAGA1UE
AxMZQ2hhbWJlcnMgb2YgQ29tbWVyY2UgUm9vdDAeFw0wMzA5MzAxNjEzNDNaFw0zNzA
5MzAxNjEzNDRaMH8xCzAJBgNVBAYTAkVVMScwJQYDVQQKEx5BQyBDYW1lcmZpcm1hIF
NBIENJRiBBODI3NDMyODcxIzAhBgNVBAsTGmh0dHA6Ly93d3cuY2hhbWJlcnNpZ24ub
3JnMSIwIAYDVQQDExlDaGFtYmVycyBvZiBDb21tZXJjZSBSb290MIIBIDANBgkqhkiG
9w0BAQEFAAOCAQ0AMIIBCAKCAQEAtzZV5aVdGDDg2olUkfzIx1L4L1DZ77F1c2VHfRt
bunXF/KGIJPov7coISjlUxFF6tdpg6jg8gbLL8bvZkSM/SAFwdakFKq0fcfPJVD0dBm
pAPrMMhe5cG3nCYsS4No41XQEMIwRHNaqbYE6gZj3LJgqcQKH0XZi/caulAGgq7YN6D
6IUtdQis4CwPAxaUWktWBiP7Zme8a7ileb2R6jWDA+wWFjbw2Y3npuRVDM30pQcakjJ
yfKl2qUMI/cjDpwyVV5xnIQFUZot/eZOKjRa3spAN2cMVCFVd9oKDMyXroDclDZK9D7
ONhMeU+SsTjoF7Nuucpw4i9A5O4kKPnf+dQIBA6OCAUQwggFAMBIGA1UdEwEB/wQIMA
YBAf8CAQwwPAYDVR0fBDUwMzAxoC+gLYYraHR0cDovL2NybC5jaGFtYmVyc2lnbi5vc
mcvY2hhbWJlcnNyb290LmNybDAdBgNVHQ4EFgQU45T1sU3p26EpW1eLTXYGduHRooow
DgYDVR0PAQH/BAQDAgEGMBEGCWCGSAGG+EIBAQQEAwIABzAnBgNVHREEIDAegRxjaGF
tYmVyc3Jvb3RAY2hhbWJlcnNpZ24ub3JnMCcGA1UdEgQgMB6BHGNoYW1iZXJzcm9vdE
BjaGFtYmVyc2lnbi5vcmcwWAYDVR0gBFEwTzBNBgsrBgEEAYGHLgoDATA+MDwGCCsGA
QUFBwIBFjBodHRwOi8vY3BzLmNoYW1iZXJzaWduLm9yZy9jcHMvY2hhbWJlcnNyb290
Lmh0bWwwDQYJKoZIhvcNAQEFBQADggEBAAxBl8IahsAifJ/7kPMa0QOx7xP5IV8EnNr
JpY0nbJaHkb5BkAFyk+cefV/2icZdp0AJPaxJRUXcLo0waLIJuvvDL8y6C98/d3tGfT
oSJI6WjzwFCm/SlCgdbQzALogi1djPHRPH8EjX1wWnz8dHnjs8NMiAT9QUu/wNUPf6s
+xCX6ndbcj0dc97wXImsQEcXCz9ek60AcUFV7nnPKoF2YjpB0ZBzu9Bga5Y34OirsrX
dx/nADydb47kMgkdTXg0eDQ8lJsm7U9xxhl6vSAiSFr+S30Dt+dYvsYyTnQeaN2oaFu
zPu5ifdmA6Ap1erfutGWaIZDgqtCYvDi1czyL+Nw="
"AC Camerfirma, S.A.","AC Camerfirma S.A.","","Chambers of Commerce
Root - 2008","00A3DA427EA4B1AEDA","063E4AFAC491DFD332F3089B8542E94
617D893D7FE944E10A7937EE29D9693C0","849AD3279D9B805A288339468C41774
4AC1CE2758A6E283A446685384D5D6CD2","2008.08.01","2038.07.31","RSA 4
096 bits","SHA1WithRSA","Websites;Email","","","1.3.6.1.4.1.17326.1
0.14.2.1.2","https://bugzilla.mozilla.org/show_bug.cgi?id=406968","
NSS 3.12.9","Firefox 4.0","https://server3ok.camerfirma.com","https
://server3.camerfirma.com","https://server3rv.camerfirma.com","","h
ttp://www.camerfirma.com","Spain","","https://www.camerfirma.com/pu
blico/DocumentosWeb/politicas/CPS_eidas_EN_1.2.12.pdf","https://www
..csqa.it/getattachment/Sicurezza-ICT/Documenti/Attestazione-di-Audi
t-secondo-i-requisiti-ETSI/2020-03-CSQA-Attestation-CAMERFIRMA-rev-
2-signed.pdf.aspx?lang=it-IT","https://www.csqa.it/getattachment/Si
curezza-ICT/Documenti/Attestazione-di-Audit-secondo-i-requisiti-ETS
I/2020-03-CSQA-Attestation-CAMERFIRMA-rev-2-signed.pdf.aspx?lang=it
-IT","https://www.csqa.it/getattachment/Sicurezza-ICT/Documenti/Att
estazione-di-Audit-secondo-i-requisiti-ETSI/2020-03-CSQA-Attestatio
n-CAMERFIRMA-rev-2-signed.pdf.aspx?lang=it-IT","CSQA Certificazioni
srl","ETSI EN 319 411","2020.03.05","MIIHTzCCBTegAwIBAgIJAKPaQn6ks
a7aMA0GCSqGSIb3DQEBBQUAMIGuMQswCQYDVQQGEwJFVTFDMEEGA1UEBxM6TWFkcmlk
IChzZWUgY3VycmVudCBhZGRyZXNzIGF0IHd3dy5jYW1lcmZpcm1hLmNvbS9hZGRyZXN
zKTESMBAGA1UEBRMJQTgyNzQzMjg3MRswGQYDVQQKExJBQyBDYW1lcmZpcm1hIFMuQS
4xKTAnBgNVBAMTIENoYW1iZXJzIG9mIENvbW1lcmNlIFJvb3QgLSAyMDA4MB4XDTA4M
DgwMTEyMjk1MFoXDTM4MDczMTEyMjk1MFowga4xCzAJBgNVBAYTAkVVMUMwQQYDVQQH
EzpNYWRyaWQgKHNlZSBjdXJyZW50IGFkZHJlc3MgYXQgd3d3LmNhbWVyZmlybWEuY29
tL2FkZHJlc3MpMRIwEAYDVQQFEwlBODI3NDMyODcxGzAZBgNVBAoTEkFDIENhbWVyZm
lybWEgUy5BLjEpMCcGA1UEAxMgQ2hhbWJlcnMgb2YgQ29tbWVyY2UgUm9vdCAtIDIwM
DgwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQCvAMtwNyuAWko6bHiUfaN/
Gh/2NdW928sNRHI+JrKQUrpjOyhYb6WzbZSm891kDFX29ufyIiKAXuFixrYp4YFs8r/
lfTJqVKAyGVn+H4vXPWCGhSRv4xGzdz4gljUha7MI2XAuZPeEklPWDrCQiorjh40G07
2QDuKZoRuGDtqaCrsLYVAGUvGef3bsyw/QHg3PmTA9HMRFEFis1tPo1+XqxQEHd9ZR5
gN/ikilTWh1uem8nk4ZcfUyS5xtYBkL+8ydddy/Js2Pk3g5eXNeJQ7KXOt3EgfLZEFH
cpOrUMPrCXZkNNI5t3YRCQ12RcSprj1qr7V9ZS+UWBDsXHyvfuK2GNnQm05aSd+pZgv
MPMZ4fKecHePOjlO+Bd5gD2vlGts/4+EhySnB8esHnFIbAURRPHsl18TlUlRdJQfKFi
C4reRB7noI/plvg6aRArBsNlVq5331lubKgdaX8ZSD6e2wsWsSaR6s+12pxZjptFtYe
r49okQ6Y1nUCyXeG0+95QGezdIp1Z8XGQpvvwyQ0wlf2eOKNcx5Wk0ZN5K3xMGtr/R5
JJqyAQuxr1yW84Ay+1w9mPGgP0revq+ULtlVmhduYJ1jbLhjya6BXBg14JC7vjxPNyK
5fuvPnnchpj04gftI2jE9K+OJ9dC1vX7gUMQSibMjmhAxhduub+84Mxh2EQIDAQABo4
IBbDCCAWgwEgYDVR0TAQH/BAgwBgEB/wIBDDAdBgNVHQ4EFgQU+SSsD7K1+HnA+mCIG
8TZTQKeFxkwgeMGA1UdIwSB2zCB2IAU+SSsD7K1+HnA+mCIG8TZTQKeFxmhgbSkgbEw
ga4xCzAJBgNVBAYTAkVVMUMwQQYDVQQHEzpNYWRyaWQgKHNlZSBjdXJyZW50IGFkZHJ
lc3MgYXQgd3d3LmNhbWVyZmlybWEuY29tL2FkZHJlc3MpMRIwEAYDVQQFEwlBODI3ND
MyODcxGzAZBgNVBAoTEkFDIENhbWVyZmlybWEgUy5BLjEpMCcGA1UEAxMgQ2hhbWJlc
nMgb2YgQ29tbWVyY2UgUm9vdCAtIDIwMDiCCQCj2kJ+pLGu2jAOBgNVHQ8BAf8EBAMC
AQYwPQYDVR0gBDYwNDAyBgRVHSAAMCowKAYIKwYBBQUHAgEWHGh0dHA6Ly9wb2xpY3k
uY2FtZXJmaXJtYS5jb20wDQYJKoZIhvcNAQEFBQADggIBAJASryI1wqM58C7e6bXpeH
xIvj99RZJe6dqxGfwWPJ+0W2aeaufDuV2I6A+tzyMP3iU6XsxPpcG1Lawk0lgH3qLPa
YRgM+gQDROpI9CF5Y57pp49chNyM/WqfcZjHwj0/gF/JM8rLFQJ3uIrbZLGOU8W6jx+
ekbURWpGqOt1glanq6B8aBMz9p0w8G8nOSQjKpD9kCk18pPfNKXG9/jvjA9iSnyu0/V
U+I22mlaHFoI6M6taIgj3grrqLuBHmrS1RaMFO9ncLkVAO+rcf+g769HsJtg1pDDFOq
xXnrN2pSB7+R5KBWIBpih1YJeSDW4+TTdDDZIVnBgizVGZoCkaPF+KMjNbMMeJL0eYD
6MDxvbxrN8y8NmBGuScvfaAFPDRLLmF9dijscilIeUcE5fuDr3fKanvNFNb0+RqE4QG
tjICxFKuItLcsiFCGtpA8CnJ7AoMXOLQusxI0zcKzBIKinmwPQN/aUv0NCB9szTqjkt
k9T79syNnFQ0EuPAtwQlRPLJsFfClI9eDdOTlLsn+mCdCxqvGnrDQWzilm1DefhiYtU
U79nm06PcaewaD+9CL2rvHvRirCG88gGtAPxkZumWK5r7VXNM21+9AUiRgOGcEMeyP8
4LG3rlV8zsxkVrctQgVrXYlCg17LofiDKYGvCYQbTed7N14jHyAxfDZd0jQ"

[Kathleen Wilson]

The text version has been updated to have each line limited to 64
characters.

Text:

https://ccadb-public.secure.force.com/mozilla/IncludedRootsPEMTxt?TrustBitsInclude=Websites

https://ccadb-public.secure.force.com/mozilla/IncludedRootsPEMTxt?TrustBitsInclude=Email

CSV:
https://ccadb-public.secure.force.com/mozilla/IncludedRootsPEMCSV?TrustBitsInclude=Websites

https://ccadb-public.secure.force.com/mozilla/IncludedRootsPEMCSV?TrustBitsInclude=Email

> 1. The reports that contain /only/ PEM data. I argue that the
> traditional format of concatenated PEM files (as used by e.g. the
> openssl command line tool) without CSV embellishments would be
> preferable, and that the reports in the latest post by Kathleen
> lacked the PEM line wrapping while still containing CSV
> artifacts.

Jakob, please explain what you mean by "still containing CSV artifacts".
Does the new version of the text report have the problem?


> 2. The reports that contain other data in CSV format. ...

Opening the CSV version of the reports in Excel and Numbers works fine
for me. So I don't understand what the problem is with having this
report be a direct extract of the PEM data that the CCADB has for each
root certificate.

> However, it might also be reasonable that if these concerns aren't easily
> addressable, perhaps not offering the feed is another option, since it
> seems a lot of work for something that should be and is naturally
> discouraged.

It's not a lot of work, just a matter of understanding what is most
useful to folks.

I think it would be good to have downstreams using current data, e.g.
data published directly via the CCADB. I think that providing the data
in an easily consumable format is better than having folks extract the
data from certdata.txt.

Thanks,
Kathleen

[Jakob Bohm]

On 2020-10-15 00:16, Kathleen Wilson wrote:
> The text version has been updated to have each line limited to 64
> characters.
>
> Text:
>
> https://ccadb-public.secure.force.com/mozilla/IncludedRootsPEMTxt?TrustBitsInclude=Websites
>
>
> https://ccadb-public.secure.force.com/mozilla/IncludedRootsPEMTxt?TrustBitsInclude=Email
>
>
> CSV:
> https://ccadb-public.secure.force.com/mozilla/IncludedRootsPEMCSV?TrustBitsInclude=Websites
>
>
> https://ccadb-public.secure.force.com/mozilla/IncludedRootsPEMCSV?TrustBitsInclude=Email
>
>
>> 1. The reports that contain /only/ PEM data.  I argue that the
>> traditional format of concatenated PEM files (as used by e.g. the
>> openssl command line tool) without CSV embellishments would be
>> preferable, and that the reports in the latest post by Kathleen
>> lacked the PEM line wrapping while still containing CSV
>> artifacts.
>
> Jakob, please explain what you mean by "still containing CSV artifacts".
> Does the new version of the text report have the problem?
>

Only the CSV form now contains CSV artifacts. And it isn't really CSV
either (even if Microsoft Excel handles it).

>
>> 2. The reports that contain other data in CSV format.  ...
>
> Opening the CSV version of the reports in Excel and Numbers works fine
> for me. So I don't understand what the problem is with having this
> report be a direct extract of the PEM data that the CCADB has for each
> root certificate.
>
>
>> However, it might also be reasonable that if these concerns aren't easily
>> addressable, perhaps not offering the feed is another option, since it
>> seems a lot of work for something that should be and is naturally
>> discouraged.
>
> It's not a lot of work, just a matter of understanding what is most
> useful to folks.
>
> I think it would be good to have downstreams using current data, e.g.
> data published directly via the CCADB. I think that providing the data
> in an easily consumable format is better than having folks extract the
> data from certdata.txt.
>
> Thanks,
> Kathleen
>

[Ryan Sleevi]

On Wed, Oct 14, 2020 at 7:31 PM Jakob Bohm via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> Only the CSV form now contains CSV artifacts. And it isn't really CSV
> either (even if Microsoft Excel handles it).

Hi Jakob,

Could you be more precise here? Embedded new lines within quoted values is
well-defined for CSV.

Realizing that there are unfortunately many interpretations of what
“well-defined for CSV” here, perhaps you can frame your concerns in terms
set out in
https://tools.ietf.org/html/rfc4180 . This at least helps make sure we can
understand and are in the same understanding.

For example, embedded new lines are discussed in 2.6 and the ABNF therein.

[Jakob Bohm]

The one difference from RFC4180 is that CR and LF are not part of the
alternatives for the inner part of "escaped".

[Ryan Sleevi]

> The one difference from RFC4180 is that CR and LF are not part of the
> alternatives for the inner part of "escaped".

Again, it would do a lot of benefit for everyone if you would be more
precise here.

For example, it seems clear and unambiguous that what you just stated is
factually wrong, because:

escaped = DQUOTE *(TEXTDATA / COMMA / CR / LF / 2DQUOTE) DQUOTE

[Jakob Bohm]

I was stating the *difference* from RFC4180 being precisely that
"simple, traditional CSV" doesn't accept the CR and LF alternatives in
that syntax production.

[Ryan Sleevi]

On Thu, Oct 15, 2020 at 7:44 PM Jakob Bohm via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> On 2020-10-15 11:57, Ryan Sleevi wrote:

> I was stating the *difference* from RFC4180 being precisely that
> "simple, traditional CSV" doesn't accept the CR and LF alternatives in
> that syntax production.

Ah, that would explain my confusion: you’re using “CSV” in a manner
different than what is widely understood and standardized. The complaint
about newlines would be as technically accurate and relevant as a complaint
that “simple, traditional certificates should parse as JSON” or that
“simple, traditional HTTP should be delivered over port 23”; which is to
say, it seems like this concern is not relevant.

As the CSVs comply with RFC 4180, which is widely recognized as what “CSV”
means, I think Jakob’s concern here can be disregarded. Any implementation
having trouble with the CSVs produced is confused about what a CSV is, and
thus not a CSV parser.

[Jakob Bohm]

RFC4180 section 3 explicitly warns that there are other variants and
specifications of the CSV format, and thus the full generalizations in
RFC4180 should not be exploited to their extremes.

Splitting the input at newlines before parsing for quotes and commas is
a pretty common implementation strategy as illustrated by my examples of
common tools that actually do so.

[Ryan Sleevi]

On Fri, Oct 16, 2020 at 5:27 PM Jakob Bohm via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> RFC4180 section 3 explicitly warns that there are other variants and
> specifications of the CSV format, and thus the full generalizations in
> RFC4180 should not be exploited to their extremes.
>

You're referring to this section, correct?

"""
Interoperability considerations:
Due to lack of a single specification, there are considerable
differences among implementations. Implementors should "be
conservative in what you do, be liberal in what you accept from
others" (RFC 793 [8]) when processing CSV files. An attempt at a
common definition can be found in Section 2.

Implementations deciding not to use the optional "header"
parameter must make their own decision as to whether the header is
absent or present.

"""


> Splitting the input at newlines before parsing for quotes and commas is
> a pretty common implementation strategy as illustrated by my examples of
> common tools that actually do so.

This would appear to be at fundamental odds with "be liberal in what you
accept from others" and, more specifically, ignoring the remark that
Section 2 is an admirable effort at a "common" definition, which is so
called as it minimizes such interoperability differences.

As your original statement was the file produced was "not CSV", I believe
that's been thoroughly dispelled by highlighting that, indeed, it does
conform to the grammar set forward in RFC 4180, and is consistent with the
IANA mime registration for CSV.

Although you also raised concern that naive and ill-informed attempts at
CSV parsing, which of course fail to parse the grammar of RFC 4180, there
are thankfully alternatives for each of those concerns. With awk, you have
FPAT. With Perl, you have Text::CSV. Of course, as the cut command is too
primitive to handle a proper grammar, there are plenty of equally
reasonable alternatives to this, and which take far less time than the
concerns and misstatements raised on this thread, which I believe we can,
thankfully, end.

Enjoy

Ryan

[Jakob Bohm]

Please stop trolling, the section you quoted clearly states that section
2 of the RFC was only an *attempt* at a common definition.

Ideally, a CSV parser should be liberal in what it accepts, but a CSV
producer (which is the subject of this thread) should be conservative in
what it provides, thus avoiding the least implemented/tested aspects
of the generalized grammar in that section 2.

Putting line feeds inside CSV fields is like using bang paths in an
RFC822 To: field. Theoretically permitted and implemented by the most
complete e-mail parsing libraries in the world, but not something that
one can expect to work for a global audience.