Speed concerns

[Ross Presser]

I'm wondering if my vault is getting too full. It takes many seconds now -- 30 or more -- to add a new identifier, or submit a challenge, or complete a challenge. 

PS C:\temp> (Get-ACMEVault).Identifiers.count

4973

PS C:\temp> (Get-ACMEVault).Certificates.count

1411

Is there any provision for removing unneeded identifiers?

I also have my code creating a new identifier each time. Is that wasteful? Should I be searching the vault for a matching identifier first?

[Ross Presser]

I exaggerated ... it's more like 10-12 seconds per identifier/submit/complete operation.

[Eugene Bekker]

Well it's possible the main meta-data file is getting rather large, it's the file 00-VAULT in the root of the Vault directory (which is C:\ProgramData\ACMESharp\sysVault if you're using the default system Vault Profile).

This file is a JSON file that is a serialization of all the state that's been accumulated over time, including all identifiers and challenge meta data.  It gets serialized and deserialized every time the vault data is read and updated.

Now, you have to keep in mind that the client needs to communicate with the server using a few exchanges every time you need to do all those steps (submit, handle challenge, request and retrieve cert, etc.)  So it could just be the speed of the traffic.

But one way to see if the problem is indeed the size of this file is to create a brand new Vault Profile with a new root Vault path.  Then complete one instance of all those steps to request a new cert.  If you do see a noticeable difference in speed, then that could be the culprit.

Now in my testing I have never noticed a difference, but I typically only have a couple dozen DNS names in a "real" working Vault, and my test/dev vaults frequently get blown away and reset, so they never get very large.  Do you mind sharing -- around how many Identifiers, Challenge Requests and Certificate Requests/Issues do you have?

[Ross Presser]

Like I said in the first message:

PS C:\temp> (Get-ACMEVault).Identifiers.count

4973

PS C:\temp> (Get-ACMEVault).Certificates.count

1411

Not sure how to count the Challenge Requests.

Probably about 50% of the identifiers are waste. The total population of domains right now is something like 2500.

00-VAULT is 62,782,025 bytes.

Would you recommend I reset the vault with a new Initialize and Registration?

[Eugene Bekker]

Doh!  You now how you're sometimes just staring at something and still can't see it?

Well, anyway -- yeah those numbers are definitely a bit high for the typical usage that I'm generally targeting and testing for.  At 62mb, the JSON serialization/deserialization is most likely the culprit.

So one way to tackle the issue is to partition your Vault, say break it up by time, every month (or every week if you're frequency is even greater), just create a new vault, or archive the old one (zip it up, rename it and remove the folder it's at, and then initialize a new one).  Or you can partition by some other dimension that's intrinsic to your data, kinda like sharding a DB.  For example, let's say you're creating SSL certs for customer-specific sites on your service, you can partition it up by the site's name (say A-Z and have one vault for each starting letter), or if you you have a unique customer ID that's all numerical, you can take the first 1 or 2 digits of that ID and partition it by that (10 or 100 vaults).

These are all kinda stop gaps and a bit klunky, but if you wrap it up in a couple of your own PS scripts, then it will happen automagically and consistently.

Now the other thing that would pby work for you is a new development that's begun in the current development version (0.8.2), namely the introduction of alternate Vault providers.  The current version of ACMESharp only has a single vault type, namely a local file-based one, but from the start, the vault system was based on a provider model, and I've begun working on a couple different vault providers that are backed by different storage services.  One of them is an EF Core based provider, which would theoretically work with any data store that has an EF Core provider, such as SQLite or SQL Server, or even some NoSQL stores that should become supported in the future.

However, the alternate providers are not ready yet and pby won't be till the next dev release as those take a while to implement and test properly.

[Ross Presser]

Thanks. I'll work on breaking it up. 

FYI, here's my distribution of domains per basedomain [1]:

https://docs.google.com/spreadsheets/d/1UnnZyUIV8TxE6pXzn8DlBcGnV14Ac3zyMlXozN49II8/edit?usp=sharing

As you can see, the largest basedomain has 650 domains in it right now, with an expectation of a few per month in the future (it's the temporary basedomain we let our customers use until they get their own). The next few have 216, 100, and so on down. I think I will break out the largest 5 into their own vaults, then segregate the rest by the first 4 characters.

[1] By "basedomain" I mean the stem domain, which Lets Encrypt uses to keep track of rate limiting:

foo.bar.baz.com = baz.com

foo.bar.baz.co.uk = baz.co.uk

[Eugene Bekker]

Cool, that sounds like a good strategy.