[Incubator] Auger (Direct access to data objects stored in Etcd by Kubernetes)

232 views
Skip to first unread message

Joe Betz

unread,
Sep 15, 2017, 2:32:40 PM9/15/17
to kuberne...@googlegroups.com
Hi,

We are proposing a new Kubernetes incubator project `Auger`. This project provides direct access to data objects stored in Etcd by Kubernetes, which are stored in a binary format in Kubernetes 1.6+.  This goal is to provide Kubernetes developers and cluster operators with simple way to decode the binary storage data back to YAML and JSON, and to support access to data in a variety of situations, including when etcd is not running and the data in a db file needs to be accessed directly. See the below README for motivation, use cases and project status.

  Sponsor: Brian Grant (@bgrant0607)
  Champion: Daniel Smith (@lavalamp)
  Initial Maintainers: Joe Betz (@jpbetz), Matt Liggett (@mml), Walter Fender (@cheftako)
  Owner: Joe Betz (@jpbetz)
  SIG: sig-apimachinery

Thanks!
-Joe

README.md

Auger

-----


Direct access to data objects stored in Etcd by Kubernetes.


Encodes and decodes Kubernetes objects from the binary storage

encoding used to store data to etcd. Supports data conversion

to YAML, JSON and Protobuf.


Why?

----


In earlier versions of Kubernetes, data written to Etcd was stored as

JSON and could easily be inspected or manipulated using standard tools such as

`etcdctl`. In Kubernetes 1.6+, for efficiency reasons, much of the data is now

stored in a binary storage representation, and is non-trivial to decode.


This tool provides Kubernetes developers and cluster operators with simple way

to decode the binary storage data back to YAML and JSON.


Interest in such a tool has been discussed before (e.g. https://github.com/kubernetes/kubernetes/issues/45974),

and we believe there is general need for it in the community.


Use cases

---------


### Access data via etcdctl


A kubernetes developer or cluster operator needs to inspect the data actually

stored to etcd for a particular kubernetes object.


E.g., decode a pod from etcd v3, where `<pod-name>` is the name of one of your pods:


``` sh

ETCDCTL_API=3 etcdctl get /registry/pods/default/<pod-name> | auger decode

> apiVersion: v1

> kind: Pod

> metadata:

>   annotations: ...

>   creationTimestamp: 2017-06-27T16:35:34Z

> ...

```


### Modify data via etcdctl


A kubernetes developer or etcd developer needs to modify state of an object stored in etcd.


E.g. Write an updated pod to etcd v3:


``` sh

cat updated-pod.yaml | auger encode | ETCDCTL_API=3 etcdctl put /registry/pods/default/<pod-name>

```


### Access data directly from db file


A cluster operator, kubernetes developer or etcd developer is needs to inspect

etcd data without starting etcd. In extreme cases, it may not be possible to

start etcd and inspecting the data may help a etcd developer understand what

state it is in.


E.g. find an etcd value by it's key and extract it from a boltdb file:


``` sh

auger extract -f <boltdb-file> -k /registry/pods/default/<pod-name>

> apiVersion: v1

> kind: Pod

> metadata:

>   annotations: ...

>   creationTimestamp: 2017-06-27T16:35:34Z

> ...

```


Status

------


Implementation leverages existing apimachinery code to keep footprint

under 1k of go code including tests.


Tool has proven useful during multiple production issue investigations

and has been updated based off feedback from those investigations. We

believe sharing this with a broader audience as a Kubernetes incubator

project will benefit the community.


Clayton Coleman

unread,
Sep 15, 2017, 4:04:19 PM9/15/17
to Joe Betz, kubernetes-dev
+1 we have analogues of this that I would prefer be folded into one community shared project.

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-dev@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CAP-qnZ-1eak6OGFhMGqc5Lg8NBxYq7X04hdQZHV%3DFPc1xDRdpw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Klaus Ma

unread,
Sep 15, 2017, 10:46:35 PM9/15/17
to Clayton Coleman, Joe Betz, kubernetes-dev
+1, great feature :).

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.
To post to this group, send email to kuberne...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CAP-qnZ-1eak6OGFhMGqc5Lg8NBxYq7X04hdQZHV%3DFPc1xDRdpw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.
To post to this group, send email to kuberne...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CAH16ShJstbRom9%2BmrWfEyekbi62nEwFKsS_WFbOwB%3D9GH0uRHg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.
--

Regards,
----
Da (Klaus), Ma (马达), PMP® | Software Architect
IBM Platform Development & Support, STG, IBM GCG
+86-10-8245 4084 | mad...@cn.ibm.com | http://k82.me

Brian Grant

unread,
Sep 16, 2017, 2:33:57 AM9/16/17
to Joe Betz, kuberne...@googlegroups.com
From experience I can say that it's better to have such a tool before you really need it.

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-dev@googlegroups.com.

Bobby Salamat

unread,
Sep 18, 2017, 7:04:50 PM9/18/17
to Brian Grant, Joe Betz, kuberne...@googlegroups.com
+1. This tool will be very useful.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.
To post to this group, send email to kuberne...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.
To post to this group, send email to kuberne...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CAKCBhs6dSjkcQ-heET%3Dd7yhC5ykdLjKm7RqWbF%2BLtUrQqLoBWA%40mail.gmail.com.

Dawn Chen

unread,
Sep 18, 2017, 7:13:45 PM9/18/17
to Bobby Salamat, Brian Grant, Joe Betz, kuberne...@googlegroups.com
+1 to have this useful tool to improve the production introspection and debuggability.

On Mon, Sep 18, 2017 at 4:04 PM, 'Bobby Salamat' via Kubernetes developer/contributor discussion <kuberne...@googlegroups.com> wrote:
+1. This tool will be very useful.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-dev@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-dev@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-dev@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/CADexmvFL2qY2EhY1KbFHGc7dKxZW898_tZLwWZ62ahsDYttz2w%40mail.gmail.com.

Marek Grabowski

unread,
Sep 19, 2017, 6:01:48 AM9/19/17
to Kubernetes developer/contributor discussion
+1


On Tuesday, September 19, 2017 at 12:13:45 AM UTC+1, Dawn Chen wrote:
+1 to have this useful tool to improve the production introspection and debuggability.
On Mon, Sep 18, 2017 at 4:04 PM, 'Bobby Salamat' via Kubernetes developer/contributor discussion <kuberne...@googlegroups.com> wrote:
+1. This tool will be very useful.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.
To post to this group, send email to kuberne...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.
To post to this group, send email to kuberne...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.
To post to this group, send email to kuberne...@googlegroups.com.

Joe Betz

unread,
Sep 19, 2017, 2:13:48 PM9/19/17
to Dawn Chen, Bobby Salamat, Brian Grant, kuberne...@googlegroups.com
Glad to hear there is interest! When the incubator project freeze ends in a few weeks, after elections, we'll finalize this request.

Project code available at https://github.com/jpbetz/auger. Reviews/comments/concerns welcome.

On Mon, Sep 18, 2017 at 4:13 PM Dawn Chen <dawn...@google.com> wrote:
+1 to have this useful tool to improve the production introspection and debuggability.

On Mon, Sep 18, 2017 at 4:04 PM, 'Bobby Salamat' via Kubernetes developer/contributor discussion <kuberne...@googlegroups.com> wrote:
+1. This tool will be very useful.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.
To post to this group, send email to kuberne...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.
To post to this group, send email to kuberne...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.
To post to this group, send email to kuberne...@googlegroups.com.
Reply all
Reply to author
Forward
This conversation is locked
You cannot reply and perform actions on locked conversations.
0 new messages