Replace deployment conflict

2,852 views
Skip to first unread message

lin.for...@gmail.com

unread,
Aug 5, 2016, 2:05:50 AM8/5/16
to kubernetes-dev
Hi,

Sometime in my cluster, the PUT action for a deployment was failed due to conflict. Here is error message that Kubernetes throw

PUT http://localhost:4200/kubernetes/apis/extensions/v1beta1/namespaces/default/deployments/admin-11?_timestamp=1470375890232 409 (Conflict)

Operation cannot be fulfilled on deployment; the object has been modified; please apply your changes to the latest version and try again.

Does anyone knows what happened? How can I resolve this issue?

Thanks

Timo Reimann

unread,
Aug 5, 2016, 3:38:22 AM8/5/16
to lin.for...@gmail.com, kubernetes-dev

I suppose what happened is that you are trying to modify a deployment which was previously modified by a different operation, causing the deployment version to deviate from yours. A common way to deal with concurrent write operations in APIs is to require the specification of the version so that conflicts can be detected and passed back to clients.

When these conflicts happen, can you observe a difference between the resourceVersion field of the deployment's metadata object and yours? Can you fetch the latest version and put that in your PUT request?

See also the ObjectMeta specification http://kubernetes.io/docs/api-reference/extensions/v1beta1/definitions/#_v1_objectmeta, which is contained in the deployment spec: http://kubernetes.io/docs/api-reference/extensions/v1beta1/definitions/#_v1beta1_deployment

HTH,
Timo


--
You received this message because you are subscribed to the Google Groups "kubernetes-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-dev@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/31ae5306-b96a-4060-a7de-70b7ec72ace9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

lin.for...@gmail.com

unread,
Aug 5, 2016, 4:06:31 AM8/5/16
to kubernetes-dev, lin.for...@gmail.com
You mean I need to wait as long as I can until the previous modification finished? Then I can get a new versioned deployment to avoid the conflict.

Thanks


On Friday, August 5, 2016 at 3:38:22 PM UTC+8, Timo Reimann wrote:

I suppose what happened is that you are trying to modify a deployment which was previously modified by a different operation, causing the deployment version to deviate from yours. A common way to deal with concurrent write operations in APIs is to require the specification of the version so that conflicts can be detected and passed back to clients.

When these conflicts happen, can you observe a difference between the resourceVersion field of the deployment's metadata object and yours? Can you fetch the latest version and put that in your PUT request?

See also the ObjectMeta specification http://kubernetes.io/docs/api-reference/extensions/v1beta1/definitions/#_v1_objectmeta, which is contained in the deployment spec: http://kubernetes.io/docs/api-reference/extensions/v1beta1/definitions/#_v1beta1_deployment

HTH,
Timo

Am 05.08.2016 08:05 schrieb <lin.for...@gmail.com>:
Hi,

Sometime in my cluster, the PUT action for a deployment was failed due to conflict. Here is error message that Kubernetes throw

PUT http://localhost:4200/kubernetes/apis/extensions/v1beta1/namespaces/default/deployments/admin-11?_timestamp=1470375890232 409 (Conflict)

Operation cannot be fulfilled on deployment; the object has been modified; please apply your changes to the latest version and try again.

Does anyone knows what happened? How can I resolve this issue?

Thanks

--
You received this message because you are subscribed to the Google Groups "kubernetes-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.
To post to this group, send email to kuberne...@googlegroups.com.

Timo Reimann

unread,
Aug 5, 2016, 5:18:08 AM8/5/16
to lin.for...@gmail.com, kubernetes-dev

That shouldn't be necessary and likely isn't guaranteed to succeed either. I think what you should do is fetch the deployment that you want to update, get the current version from it, put that in your request's metadata object, and try the deployment again.

Of course, this could also fail if someone manages to squeeze in between fetching the latest deployment and retrying your request. So you probably want to do this in a loop until you succeed or x attempts fail, where x is sufficiently high to account for the amount of concurrent updates you're expecting in your environment.

The pseudo code would look something like this:

while true {
  depl = getDeployment()
  version = readVersion()
  if sendUpdate(depl, version, changes) {
    // no conflict, yay
    break
  }
}

Hope I was able to make things somewhat clearer.


To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-dev@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/77ddc2b7-9b7f-4edc-a00a-a5b84b5fe46b%40googlegroups.com.

Clayton Coleman

unread,
Aug 5, 2016, 9:45:37 AM8/5/16
to Timo Reimann, lin.for...@gmail.com, kubernetes-dev
An example of the suggested loop can be found in "pkg/client/unversioned/util.go".  You can also use the "patch" verb to send a patch describing your changes - the server will then perform the loop for you.

--
You received this message because you are subscribed to the Google Groups "kubernetes-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.
To post to this group, send email to kuberne...@googlegroups.com.

Brian Grant

unread,
Aug 5, 2016, 10:07:01 AM8/5/16
to Timo Reimann, lin.for...@gmail.com, kubernetes-dev

--
You received this message because you are subscribed to the Google Groups "kubernetes-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-dev@googlegroups.com.

lin.for...@gmail.com

unread,
Aug 7, 2016, 9:01:54 PM8/7/16
to Kubernetes developer/contributor discussion, ttr...@googlemail.com, lin.for...@gmail.com
Thanks all above.  I will try your solution and take a look what Kubernetes clients do for it.

--
You received this message because you are subscribed to the Google Groups "kubernetes-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.
To post to this group, send email to kuberne...@googlegroups.com.
Reply all
Reply to author
Forward
This conversation is locked
You cannot reply and perform actions on locked conversations.
0 new messages