Events question (using Fabric8 Java client)

[Laird Nelson]

I had a question about Kubernetes Events.

I am accessing my cluster (minikube for the time being) using the (mostly undocumented) Fabric8 Java Kubernetes client (https://github.com/fabric8io/kubernetes-client/).  This is, however, a question about events in general, despite the Fabric8 slant below.

I am able to use its events API to connect to my cluster and observe various events occurring in the cluster.  I have connected using the following code fragment:

client.events().inAnyNamespace().watch(myWatcher);

So I am watching everything, everywhere.  And indeed, I see output so everything is hooked up properly.

But I am a rookie when it comes to understanding what I'm seeing.

Here is a (random) example (I'm also standing up the service-catalog, but that's another story—I mention it here only for context):

+++ action: DELETED

Event(apiVersion=v1, count=1, firstTimestamp=2017-04-25T23:41:54Z, involvedObject=ObjectReference(apiVersion=v1, fieldPath=spec.containers{controller-manager}, kind=Pod, name=catalog-catalog-controller-manager-1242994143-ddl0l, namespace=catalog, resourceVersion=462865, uid=11fc24bf-2a05-11e7-a27a-080027117396, additionalProperties={}), kind=Event, lastTimestamp=2017-04-25T23:41:54Z, message=Started container with id 7b51c389f153832e7719a99738706c2ff38aa28b298b80741f439b712f166262, metadata=ObjectMeta(annotations=null, clusterName=null, creationTimestamp=2017-04-25T23:41:54Z, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=null, labels=null, name=catalog-catalog-controller-manager-1242994143-ddl0l.14b8c87cc177fb77, namespace=catalog, ownerReferences=[], resourceVersion=472706, selfLink=/api/v1/namespaces/catalog/events/catalog-catalog-controller-manager-1242994143-ddl0l.14b8c87cc177fb77, uid=c3851fae-2a10-11e7-a27a-080027117396, additionalProperties={}), reason=Started, source=EventSource(component=kubelet, host=minikube, additionalProperties={}), type=Normal, additionalProperties={})

So the "+++ action: DELETED" part is something I'm printing out myself, but the "DELETED" string is the name of a Fabric8 Kubernetes client Watcher.Action (http://static.javadoc.io/io.fabric8/kubernetes-client/2.2.14/io/fabric8/kubernetes/client/Watcher.Action.html).

I am not sure what to make of this.  What has been DELETED?  To my eyes, the event describes the starting of a container whose Pod is catalog-catalog-controller-manager-1242994142-ddl01.  On the other hand, the kind of the actual event is Event, so perhaps this is describing the deletion of the Event resource itself?

Thanks for any pointers,

Best,

Laird

--

http://about.me/lairdnelson

[Laird Nelson]

On Wednesday, April 26, 2017 at 12:09:07 PM UTC-7, Laird Nelson wrote:

So the "+++ action: DELETED" part is something I'm printing out myself, but the "DELETED" string is the name of a Fabric8 Kubernetes client Watcher.Action (http://static.javadoc.io/io.fabric8/kubernetes-client/2.2.14/io/fabric8/kubernetes/client/Watcher.Action.html).

Talking to myself :-) I can see that the Watcher.Action enum is made up of valid values for the type field of the underlying Kubernetes WatchEvent.  And the Event in the output I quoted in my previous post is the Kubernetes Event that is the value of the WatchEvent's object field.  So it really does look like this semantic event that I'm receiving here is informing me of the deletion of a Kubernetes Event resource.  That is…kind of strange, yes?

(You can see how fabric8 processes WatchEvents here: https://github.com/fabric8io/kubernetes-client/blob/master/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/dsl/internal/WatchConnectionManager.java#L203-L217)

[james.s...@gmail.com]

You can see which resource the event is related to the `involvedObject` property. i.e. its a pod deletion event:

involvedObject=ObjectReference(apiVersion=v1, fieldPath=spec.containers{controller-manager}, kind=Pod, name=catalog-catalog-controller-manager-1242994143-ddl0l, namespace=catalog, resourceVersion=462865, uid=11fc24bf-2a05-11e7-a27a-080027117396, additionalProperties={}

[Laird Nelson]

On Thursday, April 27, 2017 at 12:10:10 AM UTC-7, james.s...@gmail.com wrote:

You can see which resource the event is related to the `involvedObject` property. i.e. its a pod deletion event:

involvedObject=ObjectReference(apiVersion=v1, fieldPath=spec.containers{controller-manager}, kind=Pod, name=catalog-catalog-controller-manager-1242994143-ddl0l, namespace=catalog, resourceVersion=462865, uid=11fc24bf-2a05-11e7-a27a-080027117396, additionalProperties={}

Thanks for your reply.  Yeah, I got that already, but note that that is a property of the (Kubernetes, not fabric8) Event this WatchEvent (as represented by fabric8) is describing, not a direct property of the WatchEvent itself.  So in the WatchEvent below (broken into its type (what fabric8 calls "action") and object (the Kubernetes Event the WatchEvent is describing))…

+++ action: DELETED

Event(apiVersion=v1, count=1, firstTimestamp=2017-04-25T23:41:54Z, involvedObject=ObjectReference(apiVersion=v1, fieldPath=spec.containers{controller-manager}, kind=Pod, name=catalog-catalog-controller-manager-1242994143-ddl0l, namespace=catalog, resourceVersion=462865, uid=11fc24bf-2a05-11e7-a27a-080027117396, additionalProperties={}), kind=Event, lastTimestamp=2017-04-25T23:41:54Z, message=Started container with id 7b51c389f153832e7719a99738706c2ff38aa28b298b80741f439b712f166262, metadata=ObjectMeta(annotations=null, clusterName=null, creationTimestamp=2017-04-25T23:41:54Z, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=null, labels=null, name=catalog-catalog-controller-manager-1242994143-ddl0l.14b8c87cc177fb77, namespace=catalog, ownerReferences=[], resourceVersion=472706, selfLink=/api/v1/namespaces/catalog/events/catalog-catalog-controller-manager-1242994143-ddl0l.14b8c87cc177fb77, uid=c3851fae-2a10-11e7-a27a-080027117396, additionalProperties={}), reason=Started, source=EventSource(component=kubelet, host=minikube, additionalProperties={}), type=Normal, additionalProperties={})

…the involvedObject (which I highlighted in purple) is a Pod, but the thing housing that involvedObject is a (Kubernetes, not fabric8) Event (see the selfLink, for example) which means that the thing that was actually Deleted was the Kubernetes Event, not the Pod, and the reason, pertaining to the Kubernetes Event, not the involvedObject, is Started—how do I read this?  What started, exactly?  Why is deletionTimestamp null?  What does it mean for me to receive what amounts to a WatchEvent whose type is Deleted and whose object is a Kubernetes Event?  What does it mean for a Kubernetes Event that describes the starting of a Pod's container to be Deleted? What is this describing, exactly?  Clearly it's more than just "Pod x was deleted".  

The best I can do is: this is trying, perhaps, to tell me that a Kubernetes Event describing the starting of a particular Pod's container was Deleted (probably because another container (from presumably another Pod) was started).  Does that seem right?

Thanks again for your help,

[James Strachan]

I think its a Pod deleted event (not a deleted Event event)

--

James
-------
Red Hat

Twitter: @jstrachan
Email: james.s...@gmail.com

Blog: https://medium.com/@jstrachan/

fabric8: https://fabric8.io/

open source development platform

funktion: https://funktion.fabric8.io/

open source event based lambda programming

[Laird Nelson]

On Thursday, April 27, 2017 at 8:27:28 AM UTC-7, James Strachan wrote:

I think its a Pod deleted event (not a deleted Event event)

Thanks; I (respectfully!) disagree.  Let's suppose that's all it was.  Why would the reason be Started?  Why would the kind be Event?  Why would the selfLink not reference a Pod?  No, I think this is the deletion of an Event resource that described the starting of a Pod's container (which makes sense, as to trigger this I had deleted a Deployment and created a new one).

[James Strachan]

Try run "kubectl get event -w" and see if you ever see an event for kind "Event" being created/deleted. I don't ;). 

Maybe watching events with `kubectl` while also using the REST API / fabric8 kubernetes-client might help you grok how the JSON/YAML is structured

[Laird Nelson]

On Thursday, April 27, 2017 at 8:47:16 AM UTC-7, James Strachan wrote:

Try run "kubectl get event -w" and see if you ever see an event for kind "Event" being created/deleted. I don't ;). 

That's because kubectl get event -w asks for an EventList.  (Add -v=8 (I'm sure you know this! this is for others reading this) to kubectl to see which REST endpoint it is hitting; I see "GET https://192.168.99.100:8443/api/v1/namespaces/default/events".  Note this does not have "/watch/" in it.)  And I have no doubt that if I use the fabric8 APIs to set up watches on, say, particular resource kinds, like Pods, that similar stuff would come back.  On the other hand, when you do:

client.events().inAnyNamespace().watch(myWatcher);

...in fabric8, the client is hitting a different sort of resource that starts with /api/v1/watch/, involves Kubernetes Event resources, and is getting WatchEvents, which are different, describing those Event resources.  This explains why the JSON object that is unpacked and handled by WatchConnectionManager is a WatchEvent, and why its object becomes the io.fabric8.kubernetes.client.Event that myWatcher is handed, and why its type becomes the io.fabric8.kubernetes.client.Watcher.Action that myWatcher is handed.

[Laird Nelson]

On Thursday, April 27, 2017 at 9:11:13 AM UTC-7, Laird Nelson wrote:

On the other hand, when you do:

client.events().inAnyNamespace().watch(myWatcher);

...in fabric8, the client is hitting a different sort of resource that starts with /api/v1/watch/,

Sorry, I was only partially correct here.  It is hitting the same sort of resource (in the same way that kubectl get events -w does), but under the covers it is receiving a "stream" (via WebSocket) of Kubernetes WatchEvent resources.  My guess is that kubectl get events -w is probably unpacking those WatchEvents so that their underlying Event payloads are exposed and interpreted.  In any event, with the Java listed above, you definitely get back WatchEvents that (from my earlier posting) fundamentally describe Events.

I captured some kubectl get events -w output and my printout of the same stuff received by my fabric8 watcher so you can see the difference.  Here's kubectl get events -w's output:

LASTSEEN                        FIRSTSEEN                       COUNT     NAME                       KIND      SUBOBJECT                  TYPE      REASON    SOURCE              MESSAGE

2017-04-27 09:29:35 -0700 PDT   2017-04-27 09:29:35 -0700 PDT   1         busybox-2844454261-j348g   Pod       spec.containers{busybox}   Normal    Killing   kubelet, minikube   Killing container with id docker://1d677a6b864b574f2f938fe6de437b31315a6ee933a7312e52abc49a8521db6d:Need to kill Pod

The same event received by my watcher constructed with the Java above and printed (note that what is printed is an io.fabric8.kubernetes.client.Event):

Event(apiVersion=v1, count=1, firstTimestamp=2017-04-27T16:29:35Z, involvedObject=ObjectReference(apiVersion=v1, fieldPath=spec.containers{busybox}, kind=Pod, name=busybox-2844454261-j348g, namespace=default, resourceVersion=552840, uid=702f8a3d-2b65-11e7-a27a-080027117396, additionalProperties={}), kind=Event, lastTimestamp=2017-04-27T16:29:35Z, message=Killing container with id docker://1d677a6b864b574f2f938fe6de437b31315a6ee933a7312e52abc49a8521db6d:Need to kill Pod, metadata=ObjectMeta(annotations=null, clusterName=null, creationTimestamp=2017-04-27T16:29:35Z, deletionGracePeriodSeconds=null, deletionTimestamp=null, finalizers=[], generateName=null, generation=null, labels=null, name=busybox-2844454261-j348g.14b94e0e94f764c1, namespace=default, ownerReferences=[], resourceVersion=553456, selfLink=/api/v1/namespaces/default/events/busybox-2844454261-j348g.14b94e0e94f764c1, uid=b3b7043d-2b66-11e7-a27a-080027117396, additionalProperties={}), reason=Killing, source=EventSource(component=kubelet, host=minikube, additionalProperties={}), type=Normal, additionalProperties={})

I've highlighted the properties of the actual Kubernetes Event contained by the received WatchEvent in bold/black, and the properties of its involvedObject (the Pod) in purple.

Mysteries solved!  Thanks for your help.

[Laird Nelson]

On Thursday, April 27, 2017 at 9:50:01 AM UTC-7, Laird Nelson wrote:

Mysteries solved!

I wrote up everything here: https://lairdnelson.wordpress.com/2017/04/27/kubernetes-events-can-be-complicated/