need session ID to lock Key/Value (and other locking idea)

[Andrew L]

Hi,

I am trying to create lock-protected Key/Value store for a system with many apps. The values in the store are a list. Different apps running on different machines must all be able to write to the list Values in append-mode. For example, say the state of the Key/Value store looks like this:

animals: dogs|cats

fruits: grapes|apples|pears

An app might need to append frogs to the list of animals. To do this, they must query for the current animals, then create a new value with dogs|cats|frogs and then write this new Value to the animals Key. Then the key/value store would look like this: 

animals: dogs|cats|frogs

fruits: grapes|apples|pears

If their is no lock for the animals Key, another app may initiate a similar operation around the same time. Maybe they try to append spiders to the list. For a brief moment, the Key/Value store may have dogs|cats|spiders, but this will be overwritten with dogs|cats|frogs when the second app finishes writing to the store. I need to implement locks for each key to prevent situations like this.

I've been doing lots of reading on this topic... 

https://www.consul.io/api/kv.html#key

https://learn.hashicorp.com/consul/developer-configuration/elections

https://learn.hashicorp.com/consul/developer-configuration/semaphore

https://github.com/hashicorp/consul/tree/master/api

However, from what I see, an app must supply a session ID to use in a lock acquisition/release operation. (https://www.consul.io/api/kv.html#key). However, only the first app that added animals: dogs will know the session ID. How can apps running on different machines starting up at different times know this session ID? I see this link says you can acquire a session (https://learn.hashicorp.com/consul/developer-configuration/elections). But the previous link makes me think you already need to know the session ID.

Do they need to know the session ID ahead of time? If so, any suggestions on how to share it? Or any suggestions on how to achieve same requirements with a different approach? 

Also, based on the "single key for coordination" section of https://learn.hashicorp.com/consul/developer-configuration/semaphore, I was inspired to come up with another solution that doesn't use the Consul acquire/release functionality. Whenever someone wants to append to the animals list, first they must successfully put to a key called animals_lock with cas=0. After they are done updating animals, they then delete animals_lock key. That way, if another app starts up and wants to read and then write to animals, they first must try and Put the animals_lock key with cas=0. They will not successfully be able to do this until that key is removed. When the put succeeds, they now have ownership of the animals Key. 

Does this solution seem like it could work? One potential pitfall I see is if the current owner of animals_lock dies, then nobody else can get it. I figure this can be solved by associating animals_key with a session that that owner creates. Are there any other pitfalls to watch out for?

Thanks,

Andrew

[Freddy Vallenilla]

Hi Andrew,

For your use case it may be enough to use the Check-And-Set functionality without acquiring locks on the keys. 

The flow for all apps could look like this:

1. Get the animals key and store the ModifyIndex from the API output
2. Do a Check-And-Set PUT with the new value, where the cas index is the ModifyIndex from step 1.

If the KV put returns false, animals was modified by a different app between the current app's read and update. 

In that case it should go back to step 1, since it is likely no longer appending frogs and may be deleting other recently-added animals.

Best,

Freddy

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/consul/issues
Community chat: https://gitter.im/hashicorp-consul/Lobby
---
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/9a7c3b6e-c4c2-41be-8310-8a91d8489cea%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.