ETCD memory and disk setup

[James Murkin]

Hi there,

I am trying to understand how much disk and memory my etcd machines should have. 

However I am finding it difficult to understand exactly how etcd uses disk and ram.

I'll break it down into a list of questions:

1. Is  --quota-backend-bytes limiting the disk usage or ram usage of etcd?

2. Should the amount of ram available to etcd always be greater than  --quota-backend-bytes (for good performance)?

3. Should a etcd node have more disk than the value of --quota-backend-bytes? If so what does it use this disk for and how much more? (10x?)

4. When you see "mvcc: database space exceeded" has etcd run out of disk or ram? 

  a. Is it recommended to defrag regularly to prevent this on a live instance? Or is regular compaction sufficient

5. What are the downsides to just having a massive amount of ram/disk available and setting  --quota-backend-bytes very high? 

Context: We want to run 1000 kubernetes clusters with extremely high pod churn. I have seen this recommendation https://etcd.io/docs/v3.5/op-guide/hardware/#large-cluster but it doesn't really go into disk and ram recommendations in detail.

Any insight would be much appreciated and sorry this is probably quite a basic question.

Thanks in advanced,

James