limiting the etcd event history

[riche]

I thought I had seen an option to limit the number of events in etcd's event history, but am not finding it. Did this ever exist?

Thanks,

_Rich

[Brandon Philips]

Hello RIch-

You can find the information in the docs here: https://etcd.io/docs/v3.3.12/op-guide/maintenance/#history-compaction-v3-api-key-value-database

Brandon

--
You received this message because you are subscribed to the Google Groups "CoreOS User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to coreos-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/coreos-user/8566db37-14df-4c25-9262-96ebf01cf727%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[riche]

Thanks Brandon. My question was not clear. I want to permanently limit the size of the history, as in roll events out when the queue is 100 deep, rather than the current 1000.

We have some devs who insist on regularly stuffing 1M blobs into the database, which results in steading increasing memory and disk usage. The minimum value for auto-compaction-retention is one hour, and these requests will add GBs to etcd's memory usage in one hour.

_Rich

[Brandon Philips]

--auto-compaction-mode=revision --auto-compaction-retention=100 should do that.

LMK if there is an issue with that set of flags. Also, cc'ing in the etcd mailing list.

--
You received this message because you are subscribed to the Google Groups "CoreOS User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to coreos-user...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/coreos-user/f4052722-f2eb-4c35-aa87-f82822417bda%40googlegroups.com.

[riche]

Thanks Brandon, that is what I was seeking. We'll experiment to find combinations of those two options that will keep these blobs from causing etcd to use too much memory.

_Rich

On Monday, May 20, 2019 at 9:54:25 AM UTC-7, bphilips wrote:

--auto-compaction-mode=revision --auto-compaction-retention=100 should do that.

LMK if there is an issue with that set of flags. Also, cc'ing in the etcd mailing list.

On Fri, May 17, 2019 at 6:27 PM riche <richard...@gmail.com> wrote:

Thanks Brandon. My question was not clear. I want to permanently limit the size of the history, as in roll events out when the queue is 100 deep, rather than the current 1000.

We have some devs who insist on regularly stuffing 1M blobs into the database, which results in steading increasing memory and disk usage. The minimum value for auto-compaction-retention is one hour, and these requests will add GBs to etcd's memory usage in one hour.

_Rich

On Friday, May 17, 2019 at 4:04:04 PM UTC-7, bphilips wrote:

Hello RIch-

You can find the information in the docs here: https://etcd.io/docs/v3.3.12/op-guide/maintenance/#history-compaction-v3-api-key-value-database

Brandon

--
You received this message because you are subscribed to the Google Groups "CoreOS User" group.

To unsubscribe from this group and stop receiving emails from it, send an email to coreo...@googlegroups.com.

[Brandon Philips]

This is in the v3 data store right?

To unsubscribe from this group and stop receiving emails from it, send an email to coreos-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/coreos-user/3aea0e2f-639e-4b55-b7da-c4f6325156d6%40googlegroups.com.

[riche]

Yes it is. With --auto-compaction-retention=1000 it does not gobble up memory and cpu under the load that drove v2 to the limit (oom).

_Rich

On Monday, May 20, 2019 at 1:51:07 PM UTC-7, bphilips wrote:

This is in the v3 data store right?

On Mon, May 20, 2019 at 10:13 AM riche <richard...@gmail.com> wrote:

Thanks Brandon, that is what I was seeking. We'll experiment to find combinations of those two options that will keep these blobs from causing etcd to use too much memory.

_Rich

On Monday, May 20, 2019 at 9:54:25 AM UTC-7, bphilips wrote:

--auto-compaction-mode=revision --auto-compaction-retention=100 should do that.

LMK if there is an issue with that set of flags. Also, cc'ing in the etcd mailing list.

On Fri, May 17, 2019 at 6:27 PM riche <richard...@gmail.com> wrote:

Thanks Brandon. My question was not clear. I want to permanently limit the size of the history, as in roll events out when the queue is 100 deep, rather than the current 1000.

We have some devs who insist on regularly stuffing 1M blobs into the database, which results in steading increasing memory and disk usage. The minimum value for auto-compaction-retention is one hour, and these requests will add GBs to etcd's memory usage in one hour.

_Rich

On Friday, May 17, 2019 at 4:04:04 PM UTC-7, bphilips wrote:

Hello RIch-

You can find the information in the docs here: https://etcd.io/docs/v3.3.12/op-guide/maintenance/#history-compaction-v3-api-key-value-database

Brandon

--
You received this message because you are subscribed to the Google Groups "CoreOS User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to coreo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/coreos-user/f4052722-f2eb-4c35-aa87-f82822417bda%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "CoreOS User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to coreo...@googlegroups.com.

[Brandon Philips]

great! glad this resolved it.

To unsubscribe from this group and stop receiving emails from it, send an email to coreos-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/coreos-user/47852a48-100c-4c28-b40e-3d82bb4aefcc%40googlegroups.com.