Release Stability

[Iain Duncan]

Hi!

I'm just looking to pull in the latest version of Clair as I was keen to get the Debian/Ubuntu issue fix into our deployment.  Usually when we pull in Clair we wait for a release of quay/clair and use the versions in that as our update path.  I saw there wasn't a full release of quay/clair that had the fix I wanted in but there was a release candidate (v4.7.0-rc.1 using v1.5.8 of claircore) so I have been testing that one.  Sadly when using that in our test environment we kept on getting lots of timeouts (the get vulnerability query has a 30s timeout) when querying for vulnerabilities under a moderate load.  However, I've since noticed that there have been another 5 releases of claircore in the month since then including one that further altered the query builder for the vulnerability query and although our tests show that it is still a little slower (no doubt because of the join) it isn't hitting the timeout any more.

This leads me to ask what the expected stability of claircore releases is?  Did I just get unlucky with v1.5.8 that it had a bit of a performance degradation in it or should I not pick up random claircore releases but wait for a clair release with a more sanctioned claircore version in it?  It seems there is a bit of a churn in claircore at the moment, should we be looking to pick up the releases as quickly as they come out or is the release frequency likely to slow?

Along the way I noticed that support has been dropped for specifying how many connection Clair can make to postgres and it is now hardcoded as 30.  We'd found it quite useful when the previous hardcoding was removed in favour of getting it from the connection string as we found managing the number of connections an important part of the performance tuning.  Is that something that you would consider re-introducing?

Thanks for any information you can share!

Iain

[Joseph Crosland]

Hi Iain, sorry for the late reply.

As we approach the end of a release cycle and features start to get merged, we start deploying our main branch to the Clair instance backing quay.io. This process helps us uncover anything that hasn’t been caught by our testing framework or in our staging environment — mainly corner-case scale issues that we have trouble reproducing with load-testing (seems like we were both experiencing the same issues here). During this time we will potentially create pre-release versions suffixed with “rc.N”. These pre-releases are intended to allow stakeholders to preview changes in the release, but are not quite production ready.

Some more context around the increased latency:

Previously we had been querying the whole vuln table (with the filtering clauses), this created some issues:

• If a source chose to update, say, their advisory links to a new domain, the updater would see those as new vulnerabilities and ingest them, causing (apparent) duplicates.
• If a source deleted an advisory it would linger in the DB until it was GCd and be potentially still surfaced in results.
• If there was a configuration change to filter down some portion of a source's advisories, the desired result would not be seen until the GC deleted the old update_operation (which, depending on retention, could be a while).

For these reasons and I'm sure others that I'm forgetting it was seen as imperative that we start querying with a little more nuance, hence the join to ensure we're only considering latest_update_operations when querying for results.

We don't really have a stability policy for claircore other than; if it's in a major Clair release it's considered stable. As more and more stakeholders integrate directly with claircore this will probably be something we need to address. The release frequency of claircore does tend to increase around the time of a release as we address any outstanding issues/fix bugs.

Thanks for bringing up the last point, I believe Hank has tried to kill that off but somehow during a refactor it has sneaked back in, here is PR to remove it once and for all.

Thanks,

Crozzy

[Iain Duncan]

Hi Crozzy,

Thanks for taking the time for the detailed reply, this was really useful!  It sounds like we've been taking the right approach by in general waiting for a full quay/clair release before picking up updates.

One follow on question I have is what is the time frame for doing Clair releases, is it just when you are wanting to update the quay.io instance?  The reason I ask is one of the new features we wanted was the new Debian parser which came in on the claircore v1.5.0 release in May but has only just made it into a clair release.  As it happened in this case we've only just got around to doing the update so it hasn't held us up but if we were waiting for a new quay/clair release with the change in it may have done.

Thanks for removing the connection pool size hard coding, I think that should help with any latency increases from the expanded queries!

Cheers,

Iain