CT updating software - issues with database

[David Fernandez]

Hi all,

We are planning to update our CT Server software from github. We've doing some tests with the latest versions and now we have some problems and doubts with the sqlite database.

• The newest software creates 3 tables (leaves, node, trees)  but we only have 2 in the old version (leaves, trees)
• The actual size of our database is about 7 Gb (mainly because we set the tree signing parameter to 1s. at the beginning)

Our doubts are:

1. ¿Can we start with an empty database?....I presume not....but we want to be sure at this point.
2. If the answer to previous question was "no", then how should we face the migration of the sqlite database?

Thank you!!!!

[Al Cutter]

Hi David,

we're in the process of adding support for migration from the old codebase to the new clustered code.

As you've noticed the DB schema differs so it's not just a case of running the new code with an old DB instance.

Instead we currently envision a process something along the lines of:
- put old server in a read only (not accepting new additions) mode
- wait for all pending certs to be integrated
- start new cluster in a mirroring mode, targeting the old instance as its source
- wait for cluster to catch up
- double check that STH from new cluster matches the one from the old log
- restart cluster in Log mode
- wait for new cluster to create a new STH, ensure that the root hash is identical (because no more certs should've been added)
- point your load balancer at new cluster
- take down old log

Final details may differ slightly but that's the just of it.

Currently we have support in the reference branch for the mirroring target (for including scts) and read only modes. We still need to add a bit of support for migrations to the master branch, test it, and write a HOWTO.

We had planned to make that happen in Q1, but for one reason or another we only partially completed the work, but we'll keep working on it. We'd certainly welcome some help with any of testing/validation, docs, code if you have the inclination - that would likely help to get you moved sooner!

Incidentally, we recommend using the LevelDB storage for the new clustered code.

Saludos,

Al.

> --
> You received this message because you are subscribed to the Google Groups "certificate-transparency" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to certificate-transp...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[David Fernandez]

Hi Al,

thank you for your answer but I still don't see it clear in your answer...

Regardless of the migration process, "old" database schema won't be valid in the new version because in the first tests we did, CT software complained saying one table was missing....¿is this part of the support needed that you are preparing?

And finally the last question, what would happen if the database was completly erased? Which wolud the implications?

Thanks in advance!!!!

> To unsubscribe from this group and stop receiving emails from it, send an email to certificate-transparency+unsub...@googlegroups.com.

[Pierre Phaneuf]

On Fri, Apr 8, 2016 at 4:00 PM, David Fernandez <david...@gmail.com> wrote:

> Regardless of the migration process, "old" database schema won't be valid in
> the new version because in the first tests we did, CT software complained
> saying one table was missing....¿is this part of the support needed that you
> are preparing?

With the migration process that we are working on, the new software
will NOT use the existing database, but rather use its own database
(with its own updated schema), and mirror the data from the old
software (which would need to be still running while this is done).

The new CT software is designed to work as a group of servers (can be
just one, like with the old software, but would leave you exposed to
catastrophic failure, again, just like the old software), replicating
data between each others, without using a single database (they each
have their own database, and come into agreement using etcd). This
means that each server could use a different storage backend (for
example, one using SQLite, and another using LevelDB). The only shared
storage is the etcd service.

The support that is needed but still missing is some extra flags for
the new software, to tell it to get the data from the old software.
The new software servers publish their existence (by writing their IP
and port to an etcd key), but the old software does not, so we need to
add this information manually (with some flags) during the migration.
Once the data is migrated, you would then restart the new software
without those flags to operate them normally. But these flags do not
exist at the moment.

> And finally the last question, what would happen if the database was
> completly erased? Which wolud the implications?

I'm not sure I understand the question properly? My "obvious" answer
would be that you would lose all your data?

[David Fernandez]

Hi,

we are also thinking about  setting up a new CT log....what do you think it would take longer, the migration guide or the inclusion process of a new CT log server?

Regards,