ct-server fails after submitting first certificate

[m...@flanga.io]

Hello everyone,

I am trying to setup a certificate transparency log.

I installed certificate-transparency following the Quick Build Guide on Github. I have 3 etcd nodes which are working fine, I generated my keys for the log and also prepared the etcd log using prepare-etcd.sh.

Both logservers are running fine and it is also possible to retrieve the STH from both logs. However, if I try to add a key to a logserver, this is not possible. The Certificate is obviously submitted (and I also have something in the sct.out file), but the certificate never shows up in the log itself. The certificates are fetched over openssl s_client (as suggested on Github) and the root certificate is also included in the ca-roots.pem file.

I noticed that one ct-node crashes a few minutes after submitting the certificate:

I0824 00:36:12.639580  1391 fetcher.cc:225] error fetching entries at index 0: UNKNOWN:
I0824 00:36:12.639744  1388 fetcher.cc:225] error fetching entries at index 0: UNKNOWN:
I0824 00:36:12.639902  1385 fetcher.cc:225] error fetching entries at index 0: UNKNOWN:
I0824 00:36:12.640063  1384 fetcher.cc:225] error fetching entries at index 0: UNKNOWN:
I0824 00:36:12.640221  1386 fetcher.cc:225] error fetching entries at index 0: UNKNOWN:
I0824 00:36:12.640380  1387 fetcher.cc:225] error fetching entries at index 0: UNKNOWN:
I0824 00:36:12.640540  1390 fetcher.cc:225] error fetching entries at index 0: UNKNOWN:

The other node continues to run, but both nodes also show some errors for etcd:

I0824 00:36:49.330212  6183 masterelection.cc:546] /root/election/24959eff-6969-4804-ab21-3e0049286393: Became master
I0824 00:36:49.336323  6212 etcd_consistent_store.cc:784] Cleaning old entries up to and including sequence number: -1
W0824 00:36:49.394676  6183 connection_pool.cc:339] Releasing errored connection to etcd1.flanga.io:2379
W0824 00:36:49.394821  6183 connection_pool.cc:364] error flag (0x41): READING TIMEOUT : Success
W0824 00:36:49.394969  6190 etcd.cc:798] Got invalid JSON:
W0824 00:36:49.413753  6183 connection_pool.cc:339] Releasing errored connection to etcd1.flanga.io:2379
W0824 00:36:49.413821  6183 connection_pool.cc:364] error flag (0x41): READING TIMEOUT : Success
W0824 00:36:49.413897  6186 etcd.cc:798] Got invalid JSON:
W0824 00:36:54.951530  6183 connection_pool.cc:339] Releasing errored connection to etcd1.flanga.io:2379
W0824 00:36:54.951694  6183 connection_pool.cc:364] error flag (0x41): READING TIMEOUT : Success
W0824 00:36:54.951851  6191 etcd.cc:798] Got invalid JSON:
W0824 00:36:54.954378  6183 connection_pool.cc:339] Releasing errored connection to etcd1.flanga.io:2379
W0824 00:36:54.954489  6183 connection_pool.cc:364] error flag (0x41): READING TIMEOUT : Success
W0824 00:36:54.954610  6188 etcd.cc:798] Got invalid JSON:

The ct-servers are started using this command:

cd /opt/ct/certificate-transparency && cpp/server/ct-server --key=privkey.pem --trusted_cert_file=ca-roots.pem --etcd_servers=etcd1.flanga.io:2379,etcd2.flanga.io:2379,etcd3.flanga.io:2379 -tree_signing_frequency_seconds=600 --port=6900 --leveldb_db=cert-dbA.ldb --logtostder

Both servers are running on the latest Ubuntu 16.04.03 LTS.

Can someone give me a hint?

Best regards,

Moritz

[Al Cutter]

Hi Moritz,

I suspect that there's something funny with your etcd from the messages - certainly it doesn't seem to be returning JSON of the expected form. You might be able to get more info adding a -v 1 flag to ct-server, or perhaps by making requests to etcd manually and seeing what you get back.

However, I'd strongly suggest investigating Trillian-based CT Logs, which is what we're moving towards (you may have seen we just filed for Chrome inclusion requests for a range of Logs based on this new codebase).  Trillian Logs are designed to remove some of the limits inherent in the C++ code, have a more concise and easily understandable code, and have lower operational overhead for operators with multiple logs.

Code is here:

  Trillian repo (this provides the underlying "General Transparency" services): https://github.com/google/trillian

  CTFE repo (this provides the public facing CT APIs): https://github.com/google/certificate-transparency-go/tree/master/trillian

Documentation is a "bit sparse" at the moment, we'll be addressing that very soon, but in the meantime you should hopefully be able to figure out how to build and run a simple CT log from the commands used in the .travis.yml file in the CTFE repo.

HTH,

Al.

--
You received this message because you are subscribed to the Google Groups "certificate-transparency" group.
To unsubscribe from this group and stop receiving emails from it, send an email to certificate-transparency+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[m...@flanga.io]

Well I guess then I will restart with Trillian - I'm sure I can work myself through it :)

Whatever failes there with the etcd, the key is now in the logfile (I stopped the log yesterday and restarted it again) - at least the tree_size increased to 1.

Thank you for your time!

Best regards,

Moritz