Filtering updaters

[David H.]

Hi!

First of all, thanks for your patient and time.

I'm trying to implement the Clair Scan process in a CI/CD pipeline using GitHub Actions. It's great what you can do with that. However, I realized that creating a new full Clair instance up and running each time a build is triggered is such a long process. So, I'm taking advantage of the updaters set configuration to restrict the distribution to load to the only one necessary. The process is too long when the set is RHEL, so I'm trying to filter de distributions load by the RHEL updater set. 

I want to load only version 7 of the distribution, so in the updater configuration section, in the config file, I have this:

updaters:

  filter: 'RHEL7.*'

But when the Clair instance starts, I can still see it loaded other rules for other versions. For instance, this one "RHEL6-amq-clients-2":

"level":"info","component":"libvuln/updates/Manager.driveUpdater","updater":"RHEL6-amq-clients-2","time":"2021-09-06T06:56:21Z","message":"finished update"}

Doing some research in the source code (I'm not a Go programmer, and it's my first time searching in Go code), I found the config load process with a comment (clair/config/updaters.go:37):

// Filter is a regexp that disallows updaters that do not match from

// running.

// TODO(louis): this is only used in clairctl, should we keep this?

// it may offer an escape hatch for a particular updater name

// from running, vs disabling the updater set completely.

Filter string `yaml:"filter" json:"filter"`

 

Does it mean that the filter option is not working as expected and loads the updater anyway, increasing the vulnerability load process?

Regards.

[Hank Donnay]

Hello,

On Mon, Sep 06, 2021 at 03:08:31AM -0700, David H. wrote:
>I'm trying to implement the Clair Scan process in a CI/CD pipeline using
>GitHub Actions.

Awesome! This is one of the use cases Louis and I talked about when
working on v4, but haven't implemented.

>It's great what you can do with that. However, I realized
>that creating a new full Clair instance up and running each time a build is
>triggered is such a long process. So, I'm taking advantage of the updaters
>set configuration to restrict the distribution to load to the only one
>necessary. The process is too long when the set is RHEL, so I'm trying to
>filter de distributions load by the RHEL updater set.

I think a more immediately fruitful approach would be to use the
import/export subcommands of clairctl.

>[...]

>
>Does it mean that the filter option is not working as expected and loads
>the updater anyway, increasing the vulnerability load process?

The updater list is dynamically generated _after_ that filter mechanism
happens. It was added mostly for testing, and prior to dynamic updaters,
so its unsurprising but annoying that it happens as you describe. If
using the export command of clairctl, you could pass it through a jq
filter -- the output is gzipped json, if I remember off the top of my
head.

Ultimately, I think the Correct solution is to write a new wrapper
around the claircore module and add an SQLite backend to claircore.
The new command can make assumptions about the layers (e.g. that they're
local and novel) and the SQLite backend could just be pre-populated by
an Actions cronjob or the like.

I've started poking at an SQLite backend, as it'd be useful to help
build out an acceptance test quite, but don't have a prototype yet. I
can lay out a design document if anyone would like, but it'll be some
time before I can start in on the needed new code.

--
hank

[David H.]

Thanks for your answer!

I think I will try the export/import option using the client. Is there any place to get a fully updated file with all the vulnerabilities (or for a given updater set), or should I create my own exported file from a running and up-to-date Clair instance?

In the second case, if I have to create the export file from an already running instance, why not creating a container image with a populated database to use in the CICD pipeline when starting the Clair container?

Regards.

[Hank Donnay]

On Wed, Sep 08, 2021 at 12:09:53PM -0700, David H. wrote:
>I think I will try the export/import option using the client. Is there any
>place to get a fully updated file with all the vulnerabilities (or for a
>given updater set), or should I create my own exported file from a running
>and up-to-date Clair instance?

You should create your own. You don't need a running instance to do so.

>In the second case, if I have to create the export file from an already
>running instance, why not creating a container image with a populated
>database to use in the CICD pipeline when starting the Clair container?

As a project, I'm hesitant to provide a compiled database without having
someone do the work to check all the licenses of the databases Clair
talks to by default. That, and I think the CI/CD use case is best served
by a dedicated tool using the claircore module.

--
hank