Key-value (KV) best practices

[Jason W]

We are interested in using Consul's KV store to help with environment configuration. I haven't seen much discussion on how people are structuring their data inside the KV store. This leaves me with some questions about the best way to approach this. 

My organization processes data for our customers for the purposes of marketing, which means there's a degree of geographic detail we configure for each customer. A simple first task I'd like to work with is storing the zipcodes for each customer in Consul's KV store. Each customer may have anywhere from a few dozen to a few thousand zipcodes. I see two ways of doing this:

1. Store all of a customers zipcodes in a single key i.e.:
1. consul kv put customer/<customer a>/service_area 12345,82848,99329
2. Store a key for each customer zipcode with some kind of junk value i.e.:
1. consul kv put customer/<customer a>/service_area/12345 true
2. consul kv put customer/<customer a>/service_area/82848 true
3. consul kv put customer/<customer a>/service_area/99329 true

I came across some presentation that suggested the former would be preferred, basically stating that larger blobs work a little better than tons of small values. This looked like an older version of Consul and it didn't have a great amount of detail.

Specifically, I would like to know how one would store such data in Consul (either one of the above solutions or some other solution I'm not aware of). More generally, what guidelines would you use to drive decisions as far as key hierarchy and granularity?

Thanks

JW

[James Phillips]

Hi Jason,

Consul's not designed as a general purpose DB - it's not super clear
to me from your example if this is application data or if the zip
codes are somehow used to configure your infrastructure. Application
data is probably better placed in some other database, especially if
you need complex queries over it.

Other than that, values are limited to 512 KB and all data does need
to fit in RAM on your Consul servers. Super generally, Consul does
well with the number of keys roughly under 1 million, especially if
you are doing things like large delete operations, so if you have a
choice a more aggregated approach with fewer keys with larger values
is probably better if your application can work with it.

-- James

> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/consul/issues
> IRC: #consul on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Consul" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to consul-tool...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/consul-tool/9aefc221-2656-480d-aa6d-2522774b21d8%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.