Garbage Collection of ConfigMaps and Secrets (join kubernetes-sig-apps@googlegroups.com to access) - Invitation to comment

747 views
Skip to first unread message

Janet Kuo (via Google Docs)

unread,
Oct 4, 2017, 11:13:51 PM10/4/17
to kubernete...@googlegroups.com
Janet Kuo has invited you to comment on the following document:
Sender's profile photoI wrote a proposal for ConfigMap/Secret garbage collection. Please take a look.
Google Docs: Create and edit documents online.
Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043, USA
You have received this email because someone shared a document with you from Google Docs.		Logo for Google Docs

kfox11...@gmail.com

unread,
Oct 5, 2017, 2:08:23 PM10/5/17
to kubernetes-sig-apps
(note, I'm going to just talk about configmaps, but take that to be configmap/secret)

I prefer some combination of "Specify ConfigMaps/Secrets garbage collection in PodSpec" and "Automatic ConfigMap/Secret References Updates"

Having configmaps be snapshotted at ReplicaSet creation time plays much better with things like helm I think. config can me managed by a helm chart. ReplicationSets snapshot the specified configmap so they can manage the lifecycle of the snapshot, not the source material. I don't think users will expect so much their uploaded configmaps getting taken over by a controller and getting deleted out from under them.

Having a flag on a volume reference does not seem overly verbose to me. It actually allows the use case where you may want some of the configmaps to be snapshotted and some to not be. For example, for kolla-kubernetes, I'd want config files to be snapshotted for things to be atomic. But the fernet-token configmap should never be snapshotted. it always must be the most recent version or keystone will malfunction and the system relies on updates making it from the ctronjob to the keystone pods. So both modes in one deployment is very important.

The snapshot reference update could be done at the Deployment level, where the snapshotting is done at the ReplicaSet level. The Deployment could look for all volumes labeled snapshot and watch them. When an snapshot labeled config is changed the Deployment just makes a new RS based on its template as normal. The template for the rs doesn't change. the rs just switches out the configmap name to its snapshotted version when it templates out the pod. I think this solves the concern with rewriting things?

It really feels like there could be two separate features here that combined covers the use case.
 1. configmap snapshotting by lower level primitives like ReplicaSets.
 2. configmap watching, new rs rollout triggering when changed.

This would allow a user to do something like kubectl edit configmap myconfig and things would roll automatically. or if the configmap was in a helm chart. tjhe user could helm upgrade foo stable/foo --set foo=bar and it would update just the configmap and it also would work properly.

Thanks,
Kevin


On Wednesday, October 4, 2017 at 8:13:51 PM UTC-7, Janet Kuo wrote:
Janet Kuo has invited you to comment on the following document:
Sender's profile photoI wrote a proposal for ConfigMap/Secret garbage collection. Please take a look.

Anthony Yeh

unread,
Oct 5, 2017, 8:27:54 PM10/5/17
to kfox11...@gmail.com, kubernetes-sig-apps
Regarding the second part, automatic reference updates, I spent some time trying to think of a way to make that work too. Janet found a blocker for my version of that plan though; it's summarized in the doc under Alternatives.

The crux of the issue was that we have two incompatible desires:

(1) A ConfigMap/Secret update should trigger a Deployment rollout.
(2) Each Deployment should be able to use its normal rollback procedure to go back to a previous state, including previous ConfigMap/Secret values.

Granted, (2) is not currently possible since ConfigMaps are mutable, but we would like to make it possible as part of this effort. This seems to require giving up on (1) because they conflict with each other when you try to roll back a Deployment without also rolling back the corresponding ConfigMap update. If anyone can think of a way out of this, I'm personally still interested in seeing if automatic reference updates are feasible.

Regarding your particular version of auto-updates:
> When an snapshot labeled config is changed the Deployment just makes a new RS based on its template as normal

Deployment uses the content of the Pod Template to know whether any existing RS already matches its desired state. If the ConfigMap name stays the same (in the Pod Template), Deployment will see the existing RS and decide no action is needed.

On Thu, Oct 5, 2017 at 11:08 AM <kfox11...@gmail.com> wrote:
(note, I'm going to just talk about configmaps, but take that to be configmap/secret)

I prefer some combination of "Specify ConfigMaps/Secrets garbage collection in PodSpec" and "Automatic ConfigMap/Secret References Updates"

Having configmaps be snapshotted at ReplicaSet creation time plays much better with things like helm I think. config can me managed by a helm chart. ReplicationSets snapshot the specified configmap so they can manage the lifecycle of the snapshot, not the source material. I don't think users will expect so much their uploaded configmaps getting taken over by a controller and getting deleted out from under them.

Having a flag on a volume reference does not seem overly verbose to me. It actually allows the use case where you may want some of the configmaps to be snapshotted and some to not be. For example, for kolla-kubernetes, I'd want config files to be snapshotted for things to be atomic. But the fernet-token configmap should never be snapshotted. it always must be the most recent version or keystone will malfunction and the system relies on updates making it from the ctronjob to the keystone pods. So both modes in one deployment is very important.

The snapshot reference update could be done at the Deployment level, where the snapshotting is done at the ReplicaSet level. The Deployment could look for all volumes labeled snapshot and watch them. When an snapshot labeled config is changed the Deployment just makes a new RS based on its template as normal. The template for the rs doesn't change. the rs just switches out the configmap name to its snapshotted version when it templates out the pod. I think this solves the concern with rewriting things?

It really feels like there could be two separate features here that combined covers the use case.
 1. configmap snapshotting by lower level primitives like ReplicaSets.
 2. configmap watching, new rs rollout triggering when changed.

This would allow a user to do something like kubectl edit configmap myconfig and things would roll automatically. or if the configmap was in a helm chart. tjhe user could helm upgrade foo stable/foo --set foo=bar and it would update just the configmap and it also would work properly.

Thanks,
Kevin

On Wednesday, October 4, 2017 at 8:13:51 PM UTC-7, Janet Kuo wrote:
Janet Kuo has invited you to comment on the following document:
Sender's profile photoI wrote a proposal for ConfigMap/Secret garbage collection. Please take a look.
Google Docs: Create and edit documents online.
Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043, USA
You have received this email because someone shared a document with you from Google Docs.		Logo for Google Docs

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-...@googlegroups.com.
To post to this group, send email to kubernete...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-apps/427d34f8-e347-4bd6-b63a-af5fa26bf4bc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

kfox11...@gmail.com

unread,
Oct 5, 2017, 9:16:47 PM10/5/17
to kubernetes-sig-apps
I think pod volumes should have a new flag, snapshot=true/false.

For the deployment object: the RS template the Deployment creates, I think, it creates it exactly as is, no modifications. The snapshot: True flag on the volumes are intact and the volume name is of the primary specified configmap.
When done this way, The Deployment can compare and still get the same result as works today. All the deployment does, is watch for changes in the primaries that are labeled as snapshot=true, and kick off exactly the same RS template over again to kick off a new snapshot process.


The ReplicaSet behavior would be something like the following: When a RS is first created, it looks for any snapshot=true volumes. It copies the configmap specified to a new configmap named: <rsname>-<somerandom-suffix>, and places in the status section of the new RS a mapping of the primary volume name to corrisponding snapshotted name.

When the ReplicaSet goes to instantiate a Pod, it copies its spec to a new pod document, substitutes any volume name with its snapshot-ed name from status, and removes the snapshot=true flag on the volume. then passes it on to be created. The created pod would work as normal, just pointing at the snapshotted configmap instead of the primary.

On ReplicaSet delete, it deletes all snaphosts listed in the status part of the document before deleting itself.

In this way, configmap lifecycle of the primary is up to the end user. It is only used by pods when snapshot=false or RS's during creation. The life cycle of the snapshotted configmaps follow the RS's life cycle which should work properly with the deployment roll forward/backwards logic.

I believe this covers both use case 1, and 2, while allowing a 3rd. supporting both snapshotted and nonsnapshotted configmaps in the same deployment for the keystone case.

What do you think?

Thanks,
Kevin

On Thursday, October 5, 2017 at 5:27:54 PM UTC-7, Anthony Yeh wrote:
Regarding the second part, automatic reference updates, I spent some time trying to think of a way to make that work too. Janet found a blocker for my version of that plan though; it's summarized in the doc under Alternatives.

The crux of the issue was that we have two incompatible desires:

(1) A ConfigMap/Secret update should trigger a Deployment rollout.
(2) Each Deployment should be able to use its normal rollback procedure to go back to a previous state, including previous ConfigMap/Secret values.

Granted, (2) is not currently possible since ConfigMaps are mutable, but we would like to make it possible as part of this effort. This seems to require giving up on (1) because they conflict with each other when you try to roll back a Deployment without also rolling back the corresponding ConfigMap update. If anyone can think of a way out of this, I'm personally still interested in seeing if automatic reference updates are feasible.

Regarding your particular version of auto-updates:
> When an snapshot labeled config is changed the Deployment just makes a new RS based on its template as normal

Deployment uses the content of the Pod Template to know whether any existing RS already matches its desired state. If the ConfigMap name stays the same (in the Pod Template), Deployment will see the existing RS and decide no action is needed.

On Thu, Oct 5, 2017 at 11:08 AM <kfox11...@gmail.com> wrote:
(note, I'm going to just talk about configmaps, but take that to be configmap/secret)

I prefer some combination of "Specify ConfigMaps/Secrets garbage collection in PodSpec" and "Automatic ConfigMap/Secret References Updates"

Having configmaps be snapshotted at ReplicaSet creation time plays much better with things like helm I think. config can me managed by a helm chart. ReplicationSets snapshot the specified configmap so they can manage the lifecycle of the snapshot, not the source material. I don't think users will expect so much their uploaded configmaps getting taken over by a controller and getting deleted out from under them.

Having a flag on a volume reference does not seem overly verbose to me. It actually allows the use case where you may want some of the configmaps to be snapshotted and some to not be. For example, for kolla-kubernetes, I'd want config files to be snapshotted for things to be atomic. But the fernet-token configmap should never be snapshotted. it always must be the most recent version or keystone will malfunction and the system relies on updates making it from the ctronjob to the keystone pods. So both modes in one deployment is very important.

The snapshot reference update could be done at the Deployment level, where the snapshotting is done at the ReplicaSet level. The Deployment could look for all volumes labeled snapshot and watch them. When an snapshot labeled config is changed the Deployment just makes a new RS based on its template as normal. The template for the rs doesn't change. the rs just switches out the configmap name to its snapshotted version when it templates out the pod. I think this solves the concern with rewriting things?

It really feels like there could be two separate features here that combined covers the use case.
 1. configmap snapshotting by lower level primitives like ReplicaSets.
 2. configmap watching, new rs rollout triggering when changed.

This would allow a user to do something like kubectl edit configmap myconfig and things would roll automatically. or if the configmap was in a helm chart. tjhe user could helm upgrade foo stable/foo --set foo=bar and it would update just the configmap and it also would work properly.

Thanks,
Kevin

On Wednesday, October 4, 2017 at 8:13:51 PM UTC-7, Janet Kuo wrote:
Janet Kuo has invited you to comment on the following document:
Sender's profile photoI wrote a proposal for ConfigMap/Secret garbage collection. Please take a look.
Google Docs: Create and edit documents online.
Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043, USA
You have received this email because someone shared a document with you from Google Docs.		Logo for Google Docs

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-apps+unsub...@googlegroups.com.

Janet Kuo

unread,
Oct 6, 2017, 2:40:08 PM10/6/17
to kfox11...@gmail.com, kubernetes-sig-apps
On Thu, Oct 5, 2017 at 6:16 PM <kfox11...@gmail.com> wrote:
I think pod volumes should have a new flag, snapshot=true/false.

For the deployment object: the RS template the Deployment creates, I think, it creates it exactly as is, no modifications. The snapshot: True flag on the volumes are intact and the volume name is of the primary specified configmap.
When done this way, The Deployment can compare and still get the same result as works today. All the deployment does, is watch for changes in the primaries that are labeled as snapshot=true, and kick off exactly the same RS template over again to kick off a new snapshot process.
 

The ReplicaSet behavior would be something like the following: When a RS is first created, it looks for any snapshot=true volumes. It copies the configmap specified to a new configmap named: <rsname>-<somerandom-suffix>, and places in the status section of the new RS a mapping of the primary volume name to corrisponding snapshotted name.

When the ReplicaSet goes to instantiate a Pod, it copies its spec to a new pod document, substitutes any volume name with its snapshot-ed name from status, and removes the snapshot=true flag on the volume. then passes it on to be created. The created pod would work as normal, just pointing at the snapshotted configmap instead of the primary.

This forces every ConfigMaps update (with snapshot=true) to make a referencing ReplicaSet update its pods. This breaks the Deployment rolling update feature. We separate ReplicaSets and Deployments intentionally. We don't want ReplicaSets to trigger any rollouts -- it should only watch current number of pods and create/delete some to make sure the number match its spec.replicas. 
 
Sender's profile photoI wrote a proposal for ConfigMap/Secret garbage collection. Please take a look.
Google Docs: Create and edit documents online.
Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043, USA
You have received this email because someone shared a document with you from Google Docs.		Logo for Google Docs

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-...@googlegroups.com.

To post to this group, send email to kubernete...@googlegroups.com.

kfox11...@gmail.com

unread,
Oct 6, 2017, 3:43:34 PM10/6/17
to kubernetes-sig-apps

On Friday, October 6, 2017 at 11:40:08 AM UTC-7, Janet Kuo wrote:

On Thu, Oct 5, 2017 at 6:16 PM <kfox11...@gmail.com> wrote:

<SNIP>

When the ReplicaSet goes to instantiate a Pod, it copies its spec to a new pod document, substitutes any volume name with its snapshot-ed name from status, and removes the snapshot=true flag on the volume. then passes it on to be created. The created pod would work as normal, just pointing at the snapshotted configmap instead of the primary.

This forces every ConfigMaps update (with snapshot=true) to make a referencing ReplicaSet update its pods. This breaks the Deployment rolling update feature. We separate ReplicaSets and Deployments intentionally. We don't want ReplicaSets to trigger any rollouts -- it should only watch current number of pods and create/delete some to make sure the number match its spec.replicas. 

No. The only time a RS ever does anything with configmaps labeled snapshoted is when it is created. From then on out, it uses its snapshot of the configmap for any pod it spawns. Only deployments watch snapshots.

I think maybe a more concreate example is in order.... Lets walk through it from the users perspective and what the system does in reaction.

Say user uploads configmaps named config, and fernet.

and then uploads the following document (psudo for brevity):
Kind=Deployment
spec:
  metadata:
      name: foo
  template:
    spec:
      volume:
      - {name: config, configMap: {name: config, snapshot=true}}
      - {name: fernet, configMap: {name: fernet}}
 
The deployment controller takes that, and sets a watch on the configmap named 'config' and then creates:
Kind=ReplicaSet    #This changed
spec:
  metadata:
      name foo-123456    #And this changed. nothing else.
  template:
    spec:
      volume:
      - {name: config, configMap: {name: config, snapshot=true}}
      - {name: fernet, configMap: {name: fernet}}
 
The ReplicaSet controller takes that, and does the following:
1. copy configmap 'config' to foo-123456-123
2. modify the doc to record the name:
Kind=RepilcaSet
spec:
  metadata:
      name foo-123456
  template:
    spec:
      volume:
      - {name: config, configMap: {name: config, snapshot=true}}
      - {name: fernet, configMap: {name: fernet}}
status:                   #Only this changed.
  snapshots:
    config: foo-123456-123

Then when the ReplicaSet Controller tries to create a pod for foo-123456, it does so like:
Kind=Pod
spec:
  metadata:
      name foo-123456-567
  template:
    spec:
      volume:
      - {name: config, configMap: {name: foo-123456-123}}     #This changed to point at the snapshotted configmap rather then the original one. It is read out of status.snapshots out of the RS document, and the snapshot=true is dropped.
      - {name: fernet, configMap: {name: fernet}}

Now, say the user updates the configmap named config.

The Deployment controller see's the change to a snapshotted configmap. All it does is create a new RS as if the user told it to do an update. it has a new name but everything else the same as before:
Kind=ReplicaSet
spec:
  metadata:
      name foo-67890     #New name, nothing else is different from the Old version.
  template:
    spec:
      volume:
      - {name: config, configMap: {name: config, snapshot=true}}
      - {name: fernet, configMap: {name: fernet}}
 
The ReplicaSet controller takes that, and does the same behavior as above:
1. copy configmap 'config' to foo-67890-456
2. modify the doc to include the reference to the snapshot:
Kind=RepilcaSet
spec:
  metadata:
      name foo-67890
  template:
    spec:
      volume:
      - {name: config, configMap: {name: config, snapshot=true}}
      - {name: fernet, configMap: {name: fernet}}
status:
  snapshots:
    config: foo-67890-456

Then when the ReplicaSet Controller tries to create a pod for foo-67890, it does so like:
Kind=pod
spec:
  metadata:
      name foo-67890-789
  template:
    spec:
      volume:
      - {name: config, configMap: {name: foo-67890-456}}
      - {name: fernet, configMap: {name: fernet}}


The roll forward/rollback between either ReplicaSet's would work as expected, as each ReplicaSet only uses its own snapshots for pods, and all the Deployment does is increment/decrement the replica count in each one. The spec of the RS always looks like what the Deployment uploaded so it wont get confused. And when the RS is deleted (triggered by the user completing the deployment upgrade), the old RS deletes its configmap snapshots listed in status.snapshots. Thus garbage collecting the unused snapshots.

Does that help clarify the idea?

Thanks,
Kevin

 
<snip>

Janet Kuo

unread,
Oct 6, 2017, 5:26:34 PM10/6/17
to kfox11...@gmail.com, kubernetes-sig-apps
We usually don't store permanent state in status. Status should be considered soft state that could be reconstructed by observation if it were lost. That's why we don't keep a list of Deployment's history (ReplicaSets) in its status, but use label/selector and observe current state in every sync loop. 
Here we need a way to generate ReplicaSet names that are both unique and idempotent (cannot be random).  We currently use a hash of Deployment template, but we cannot use this anymore because the template of two ReplicaSets will be the same. 

In addition, we need a way to generate unique label/selectors so that these a ReplicaSet won't match the other's pods. Probably similar to how names are generated. 

 
Kind=pod
spec:
  metadata:
      name foo-67890-789
  template:
    spec:
      volume:
      - {name: config, configMap: {name: foo-67890-456}}
      - {name: fernet, configMap: {name: fernet}}


The roll forward/rollback between either ReplicaSet's would work as expected, as each ReplicaSet only uses its own snapshots for pods, and all the Deployment does is increment/decrement the replica count in each one.

How is rollback done? Wouldn't it require the original ConfigMap (config) be updated back to foo-123456-123? 
 
The spec of the RS always looks like what the Deployment uploaded so it wont get confused. And when the RS is deleted (triggered by the user completing the deployment upgrade), the old RS deletes its configmap snapshots listed in status.snapshots.

What if the configmaps are still used by other resources?
 
Thus garbage collecting the unused snapshots.

Does that help clarify the idea?

Thanks,
Kevin

 
<snip>

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-...@googlegroups.com.
To post to this group, send email to kubernete...@googlegroups.com.

kfox11...@gmail.com

unread,
Oct 6, 2017, 8:47:09 PM10/6/17
to kubernetes-sig-apps

Ok. then we can do that then. For snapshottetd configmaps, we store the rs owner and parent configmap name it was originally snapshotted from. I think the algorithm should still work the same with that instead of storing it in status?
 

Ah. That is a problem.... hmm.... so I can think of a few options....
1. make the Deployment somehow add in the hash of the snapshotted configmaps to the hash of the deployment... This feels kind of race prone, as well as makes the Deployment care about things it maybe shouldnt...
2. Somehow let the RS determine the name and pass it back to the Deployment? It could build the hash of its spec and snapshotted configmaps all together. This seems messy though to pass that back to the Deployment...
3. do actually change the Deployments definition slightly... increment a number in it that gets passed through the RS? This is kind of how you have to do it today if you want to trigger a rolling upgrade of a deployment to have it re-read a configmap. Its kind of an established pattern already...
4. same as 3 but maybe use the Deployments resourceVersion? It should be guaranteed to be unique.
 
What other constraints are there on the hashing. Like, ever needing to reverse hash things? Does it ever try and rebuild the Deployments spec from an older RS if you roll back, or does the Deployment's spec only ever get used when launching a new RS?


In addition, we need a way to generate unique label/selectors so that these a ReplicaSet won't match the other's pods. Probably similar to how names are generated. 


Yeah. If we can pin down the consistent hashing/naming thing, I think that can still be the solution to that problem?
 
 
Kind=pod
spec:
  metadata:
      name foo-67890-789
  template:
    spec:
      volume:
      - {name: config, configMap: {name: foo-67890-456}}
      - {name: fernet, configMap: {name: fernet}}


The roll forward/rollback between either ReplicaSet's would work as expected, as each ReplicaSet only uses its own snapshots for pods, and all the Deployment does is increment/decrement the replica count in each one.

How is rollback done? Wouldn't it require the original ConfigMap (config) be updated back to foo-123456-123? 

Rollback is just getting foo-123456-123 have Deployment.spec.replicas again and decreasing RS foo-67890-789 to 0, then eventually deleting it? Is there anything more to it?
 
The lifecycle of the configmap snapshots needs to match 1 to 1 the lifecycle of a RS that owns them I think. I think if its the RS doing the copy/delete, that happens rather naturally?

 
The spec of the RS always looks like what the Deployment uploaded so it wont get confused. And when the RS is deleted (triggered by the user completing the deployment upgrade), the old RS deletes its configmap snapshots listed in status.snapshots.

What if the configmaps are still used by other resources?

The Primary configmap is owned by the user. If they use the configmap with other Pods that don't specify snapshot=true, then they are responsible for ensuring the configmap exists while pods reference it. or suffer the malfunction if they delete it out from under the pod. Thats how it works today.

For things marked snapshot=true, the primary configmap only needs to exist until the RS that references it makes a copy. After that, the RS or the Pods it creates will only ever reference the snapshot configmaps. The user could delete the primary configmap and the RS/Pods will only use their copy.  Nothing outside of the RS or its pods should reference its copy of the configmap as it is the owner. Other things should be able to reference the primary, or their own snapshots of the primary.

This may not be as space efficient as aggressively sharing the configmaps and reference counting and things. but I think it makes a pretty good tradeoff between saving space, ensuring the needed immutable features are implemented, providing usability by users expecting to own things they create themselves, and not having a complex garbage collection system that could be buggy?
 
Thus garbage collecting the unused snapshots.

Does that help clarify the idea?

Thanks,
Kevin

 
<snip>

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-apps+unsub...@googlegroups.com.

Janet Kuo

unread,
Oct 7, 2017, 2:07:24 AM10/7/17
to kfox11...@gmail.com, kubernetes-sig-apps
This needs to be done declaratively. How does a Deployment controller know that it needs to scale up and down those ReplicaSets? If it's not by updating the original ConfigMap back to foo-123456-123 version, how?
 
 
The lifecycle of the configmap snapshots needs to match 1 to 1 the lifecycle of a RS that owns them I think. I think if its the RS doing the copy/delete, that happens rather naturally?

 
The spec of the RS always looks like what the Deployment uploaded so it wont get confused. And when the RS is deleted (triggered by the user completing the deployment upgrade), the old RS deletes its configmap snapshots listed in status.snapshots.

What if the configmaps are still used by other resources?

The Primary configmap is owned by the user. If they use the configmap with other Pods that don't specify snapshot=true, then they are responsible for ensuring the configmap exists while pods reference it. or suffer the malfunction if they delete it out from under the pod. Thats how it works today.

For things marked snapshot=true, the primary configmap only needs to exist until the RS that references it makes a copy. After that, the RS or the Pods it creates will only ever reference the snapshot configmaps. The user could delete the primary configmap and the RS/Pods will only use their copy.  Nothing outside of the RS or its pods should reference its copy of the configmap as it is the owner. Other things should be able to reference the primary, or their own snapshots of the primary.

This may not be as space efficient as aggressively sharing the configmaps and reference counting and things. but I think it makes a pretty good tradeoff between saving space, ensuring the needed immutable features are implemented, providing usability by users expecting to own things they create themselves, and not having a complex garbage collection system that could be buggy?
 
Thus garbage collecting the unused snapshots.

Does that help clarify the idea?

Thanks,
Kevin

 
<snip>

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-...@googlegroups.com.

To post to this group, send email to kubernete...@googlegroups.com.

kfox11...@gmail.com

unread,
Oct 9, 2017, 1:18:33 PM10/9/17
to kubernetes-sig-apps

Thats what the Deployment does today. The deployemnt's job is rolling Old RS's replica's down and New RS's replica's up until in the desired state, and on roll back doing the opposite. I guess I'm saying, I don't think
this use case needs to behave any differently then it does today, unless you can think of some behavior it has that I'm not aware of?
 
 
 
The lifecycle of the configmap snapshots needs to match 1 to 1 the lifecycle of a RS that owns them I think. I think if its the RS doing the copy/delete, that happens rather naturally?

 
The spec of the RS always looks like what the Deployment uploaded so it wont get confused. And when the RS is deleted (triggered by the user completing the deployment upgrade), the old RS deletes its configmap snapshots listed in status.snapshots.

What if the configmaps are still used by other resources?

The Primary configmap is owned by the user. If they use the configmap with other Pods that don't specify snapshot=true, then they are responsible for ensuring the configmap exists while pods reference it. or suffer the malfunction if they delete it out from under the pod. Thats how it works today.

For things marked snapshot=true, the primary configmap only needs to exist until the RS that references it makes a copy. After that, the RS or the Pods it creates will only ever reference the snapshot configmaps. The user could delete the primary configmap and the RS/Pods will only use their copy.  Nothing outside of the RS or its pods should reference its copy of the configmap as it is the owner. Other things should be able to reference the primary, or their own snapshots of the primary.

This may not be as space efficient as aggressively sharing the configmaps and reference counting and things. but I think it makes a pretty good tradeoff between saving space, ensuring the needed immutable features are implemented, providing usability by users expecting to own things they create themselves, and not having a complex garbage collection system that could be buggy?
 
Thus garbage collecting the unused snapshots.

Does that help clarify the idea?

Thanks,
Kevin

 
<snip>

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-apps+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-apps+unsub...@googlegroups.com.

Janet Kuo

unread,
Oct 9, 2017, 2:11:46 PM10/9/17
to kfox11...@gmail.com, kubernetes-sig-apps
Deployment does this by observing desired state and current state. The desired state is specified in Deployment spec -- Deployment controller checks Deployment's current .spec.template, try to create and scale up new ReplicaSet (the one with the same .spec.template) and scale down all the other old ReplciaSets. 

If this does not need to behave differently then it does today, rollback should change the Deployment's spec by copying pod template from a target ReplicaSet. However, both ReplicaSets in this case would have exactly the same spec.template. How should Deployment spec be updated? If a CM update triggers a rollout, in this case, rollback can be done in the same way, by updating a CM. However, this will force other referencing resources be rolled back with it, as pointed out in the alternatives section in the proposal. 
 
 
 
 
The lifecycle of the configmap snapshots needs to match 1 to 1 the lifecycle of a RS that owns them I think. I think if its the RS doing the copy/delete, that happens rather naturally?

 
The spec of the RS always looks like what the Deployment uploaded so it wont get confused. And when the RS is deleted (triggered by the user completing the deployment upgrade), the old RS deletes its configmap snapshots listed in status.snapshots.

What if the configmaps are still used by other resources?

The Primary configmap is owned by the user. If they use the configmap with other Pods that don't specify snapshot=true, then they are responsible for ensuring the configmap exists while pods reference it. or suffer the malfunction if they delete it out from under the pod. Thats how it works today.

For things marked snapshot=true, the primary configmap only needs to exist until the RS that references it makes a copy. After that, the RS or the Pods it creates will only ever reference the snapshot configmaps. The user could delete the primary configmap and the RS/Pods will only use their copy.  Nothing outside of the RS or its pods should reference its copy of the configmap as it is the owner. Other things should be able to reference the primary, or their own snapshots of the primary.

This may not be as space efficient as aggressively sharing the configmaps and reference counting and things. but I think it makes a pretty good tradeoff between saving space, ensuring the needed immutable features are implemented, providing usability by users expecting to own things they create themselves, and not having a complex garbage collection system that could be buggy?
 
Thus garbage collecting the unused snapshots.

Does that help clarify the idea?

Thanks,
Kevin

 
<snip>

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-...@googlegroups.com.

To post to this group, send email to kubernete...@googlegroups.com.

kfox11...@gmail.com

unread,
Oct 9, 2017, 2:21:19 PM10/9/17
to kubernetes-sig-apps

Maybe I'm misunderstanding a bit how deployments work.... I have a Deployment running, that if I do a kubectl rollout history deplyment xxxx I see many versions listed. looking at the docs, it mentions 'kubectl rollout undo daemonset/abc --to-revision=3', like it can roll back/forward between the many versions you may have defined. I though (could be wrong) that the deplyment created RS's from the template spec but never looked at the RS other then for RS.replica's during roll forward/back? That it kept track of a pointer to the 'currnet' and "previous' RS's and just tweaked them.


 
 
 
 
The lifecycle of the configmap snapshots needs to match 1 to 1 the lifecycle of a RS that owns them I think. I think if its the RS doing the copy/delete, that happens rather naturally?

 
The spec of the RS always looks like what the Deployment uploaded so it wont get confused. And when the RS is deleted (triggered by the user completing the deployment upgrade), the old RS deletes its configmap snapshots listed in status.snapshots.

What if the configmaps are still used by other resources?

The Primary configmap is owned by the user. If they use the configmap with other Pods that don't specify snapshot=true, then they are responsible for ensuring the configmap exists while pods reference it. or suffer the malfunction if they delete it out from under the pod. Thats how it works today.

For things marked snapshot=true, the primary configmap only needs to exist until the RS that references it makes a copy. After that, the RS or the Pods it creates will only ever reference the snapshot configmaps. The user could delete the primary configmap and the RS/Pods will only use their copy.  Nothing outside of the RS or its pods should reference its copy of the configmap as it is the owner. Other things should be able to reference the primary, or their own snapshots of the primary.

This may not be as space efficient as aggressively sharing the configmaps and reference counting and things. but I think it makes a pretty good tradeoff between saving space, ensuring the needed immutable features are implemented, providing usability by users expecting to own things they create themselves, and not having a complex garbage collection system that could be buggy?
 
Thus garbage collecting the unused snapshots.

Does that help clarify the idea?

Thanks,
Kevin

 
<snip>

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-apps+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-apps+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-apps+unsub...@googlegroups.com.

Janet Kuo

unread,
Oct 9, 2017, 3:35:25 PM10/9/17
to kfox11...@gmail.com, kubernetes-sig-apps
`kubectl rollout history` is done by looking at existing ReplicaSets and find the ones owned by the Deployment.

`kubectl rollout undo` is done by finding the ReplicaSet of the specified revision and then copying the ReplicaSet's .spec.template to Deployment's .spec.template. To the Deployment controller, it is treated the same as a rollout (when the user or kubectl updates its template). That said, Deployment controller will then scale up a ReplicaSet that matches the Deployment's current template and scale down all the rest ones. The only "pointer" between Deployment and ReplicaSets is label/selector. Their relationship is observed dynamically by the system. 
 


 
 
 
 
The lifecycle of the configmap snapshots needs to match 1 to 1 the lifecycle of a RS that owns them I think. I think if its the RS doing the copy/delete, that happens rather naturally?

 
The spec of the RS always looks like what the Deployment uploaded so it wont get confused. And when the RS is deleted (triggered by the user completing the deployment upgrade), the old RS deletes its configmap snapshots listed in status.snapshots.

What if the configmaps are still used by other resources?

The Primary configmap is owned by the user. If they use the configmap with other Pods that don't specify snapshot=true, then they are responsible for ensuring the configmap exists while pods reference it. or suffer the malfunction if they delete it out from under the pod. Thats how it works today.

For things marked snapshot=true, the primary configmap only needs to exist until the RS that references it makes a copy. After that, the RS or the Pods it creates will only ever reference the snapshot configmaps. The user could delete the primary configmap and the RS/Pods will only use their copy.  Nothing outside of the RS or its pods should reference its copy of the configmap as it is the owner. Other things should be able to reference the primary, or their own snapshots of the primary.

This may not be as space efficient as aggressively sharing the configmaps and reference counting and things. but I think it makes a pretty good tradeoff between saving space, ensuring the needed immutable features are implemented, providing usability by users expecting to own things they create themselves, and not having a complex garbage collection system that could be buggy?
 
Thus garbage collecting the unused snapshots.

Does that help clarify the idea?

Thanks,
Kevin

 
<snip>

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-...@googlegroups.com.

To post to this group, send email to kubernete...@googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-...@googlegroups.com.

To post to this group, send email to kubernete...@googlegroups.com.

kfox11...@gmail.com

unread,
Oct 9, 2017, 3:59:54 PM10/9/17
to kubernetes-sig-apps

Ok. Thanks for the explanation. So, then to support roll forward/roll back when snapshots are involved, something must change in the Deployment.Spec for it to kick off a new RS rather then reusing an existing one and it getting confused....

Hmm.... That leads to an interesting ui question.... Is there any difference between rolling back a deployment and uploading the same deployment spec used to create a previous version? Kind of sounds like not.

In the case of snapshots, would a user ever want there to be a difference?

Ie, rolling back getting old snapshots, and uploading earlier version getting fresh snapshots?
 
 


 
 
 
 
The lifecycle of the configmap snapshots needs to match 1 to 1 the lifecycle of a RS that owns them I think. I think if its the RS doing the copy/delete, that happens rather naturally?

 
The spec of the RS always looks like what the Deployment uploaded so it wont get confused. And when the RS is deleted (triggered by the user completing the deployment upgrade), the old RS deletes its configmap snapshots listed in status.snapshots.

What if the configmaps are still used by other resources?

The Primary configmap is owned by the user. If they use the configmap with other Pods that don't specify snapshot=true, then they are responsible for ensuring the configmap exists while pods reference it. or suffer the malfunction if they delete it out from under the pod. Thats how it works today.

For things marked snapshot=true, the primary configmap only needs to exist until the RS that references it makes a copy. After that, the RS or the Pods it creates will only ever reference the snapshot configmaps. The user could delete the primary configmap and the RS/Pods will only use their copy.  Nothing outside of the RS or its pods should reference its copy of the configmap as it is the owner. Other things should be able to reference the primary, or their own snapshots of the primary.

This may not be as space efficient as aggressively sharing the configmaps and reference counting and things. but I think it makes a pretty good tradeoff between saving space, ensuring the needed immutable features are implemented, providing usability by users expecting to own things they create themselves, and not having a complex garbage collection system that could be buggy?
 
Thus garbage collecting the unused snapshots.

Does that help clarify the idea?

Thanks,
Kevin

 
<snip>

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-apps+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-apps+unsub...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-apps+unsub...@googlegroups.com.

To post to this group, send email to kubernete...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-apps" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-apps+unsub...@googlegroups.com.

abejide...@getbraintree.com

unread,
Aug 10, 2018, 6:26:46 PM8/10/18
to kubernetes-sig-apps
Ping!
Reply all
Reply to author
Forward
0 new messages