How to set blob expiration?

[Andreas Hasenkopf]

Hi everyone,

half a year ago I set up my own registry with Quay and MinIO as backend. 

I noticed that the images and tags expire and vanish as expected. ✔

But, I also noticed that the uploaded blobs in MinIO do not get deleted. ⚠

I searched the docs and found this note:

»Blob expiration is independent of the time machine. If you push a tag to Project Quay and the time machine is set to 0 seconds, and then you delete a tag immediately, garbage collection deletes the tag and everything related to that tag, but will not delete the blob storage until the blob expiration time is reached.«

But I could not figure out how to configure the blob expiration. Can somebody point me in the right direction (docs, what to look for in STDOUT, ...)?

Thank you very much,

Andi

[bdet...@redhat.com]

Hi Andi- can you confirm that you've got garbage collection enabled? There are some details in our docs on how to do this- https://access.redhat.com/documentation/en-us/red_hat_quay/3.10/html/manage_red_hat_quay/garbage-collection

-Bill

[Andreas Hasenkopf]

Hi Bill,

thanks for the quick response. Yes, I can confirm that GC is enabled:

• The values in my config are all true
• I see similar messages in the output

FEATURE_GARBAGE_COLLECTION: true

gcworker stdout | 2024-02-09 13:56:46,269 [60] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-02-09 13:57:16 UTC)" executed successfully

nginx stdout | 10.42.1.0 (-) - - [09/Feb/2024:13:51:19 +0000] "DELETE /v2/*****/*****/blobs/uploads/4a87ffca-f4fb-42af-bb8d-91b03470a2f8 HTTP/1.1" 204 0 "-" "go-containerregistry/v0.17.0" (0.018 1533 0.020)

I also started MinIO and Quay locally and pushed a single image. I tried both: letting the image expire and permanent deletion. Hours later the files are still in the bucket in MinIO.

[Andreas Hasenkopf]

OK, on Friday I had started instances of MinIO and Quay locally and uploaded an image that expired after an hour. On Tuesday, the files in the MinIO bucket actually got deleted.

Is there a configuration value to change this delay between tag expiration and file deletion?

Thank you for the help/support so far :)

[Andreas Hasenkopf]

OMG, I am so blind. I was always looking at configuration via config.yaml, that I did not bother exploring the configuration options in the web UI.

Reducing the Time Machine setting in the UI to "a few seconds" triggered a nice garbage collection.

Thanks again for all the helpful pointers.

[bdet...@redhat.com]

Thanks for the update and letting us know how it got resolved.

Sounds like we should clarify this in our docs. I'll raise a ticket with our docs team- thanks Andi!

[Andreas Hasenkopf]

Hello again,

so, about a month ago I figured out where the "Time Machine" setting was hidden and confirmed locally, that the files in my bucket in MinIO get deleted.

But my "production" instance ran out of disk space again:

According to MinIO, there are approx. 5k files in the bucket using 48G (i.e. 100% of the partition). 

But according to Quay (v3.10.3), there are only about 10 images left.

Is there another config I might have overlooked?

Would there be a better workaround but removing all the files (MinIO) and images (Quay) to get the setup working again?

Thank you,

Andi