Blocking query on multiple keys with prefix and recieve only updated keys

[Anzor Apshev]

Hi all.
My app manages a lot of small executing objects. Each object of execution stores configuration in consul kv.The configuration is moderate size xml (up to 50kb). In order to be update, my app would like to monitor changes in configuration and react appropriately.

 Currently the app gets a snapshot of configuration at start (several thousands of kv pairs) and monitors changes with ?recurse&Index=<max_modify_index>. When user make any change, blocking query returns with a full snapshot of configuration. I have to compare two snapshots (using ModifyIndex field) and filter out unchanged packets. This works but quite inefficient because users can make many small changes to individual objects, and  each such change leads to downloading full snapshot .

So is there any way to do better here?

It would be good to have blocking query which returns key and modify index pair for each record in consul, something like
http://127.0.0.1:8500/v1/kv/<prefix>/?recurse&pretty&keys&indexes&index=<someIndex>

It would solve my problem.

[James Phillips]

Hi Anzor,

There's a discussion close to this started on https://github.com/hashicorp/consul/issues/1815. Currently in Consul the only way to cut down the noise is to watch multiple prefixes, which may not be possible with every KV layout, which is pretty efficient under the hood in Consul. We could do some relatively small enhancements to add an API like you describe, though, and perform filtering to only return the changed keys.

-- James

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/consul/issues
IRC: #consul on Freenode
---
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/b2ac895a-e630-4145-8d9e-b06e5ea27b98%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Anzor Apshev]

Hi James.
Thanks for replay. Indeed I am very interested for filtering api. Current solution is a big headache for me. Is there any way I can help with implementing this feature? I think I can open pull request if you give me some hints where to start digging.

[James Phillips]

Hi Anzor,

Let's continue this on #1815 - when I get a little time I'll try to scope it out.

Thanks!

To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/5f01df93-0965-4056-a8e2-78448837e7f6%40googlegroups.com.