etcd data format

[Justin Garrison]

I'm trying to understand how data gets encoded and stored in etcd (I'm assuming via http or grpc is the same data format)

In minikube (k8s version 1.6.0-beta3) I created a deployment and used minikube ssh and etcdctl to look at some of the keys.

eg

etcdctl get /registry/deployments/default/microbot

azhzAAogChJleHRlbnNpb25zL3YxYmV0YTESCkRlcGxveW1lbnQSzQQKygEKCG1pY3JvYm90EgAaB2RlZmF1bHQiQC9hcGlzL2V4dGVuc2lvbnMvdjFiZXRhMS9uYW1lc3BhY2VzL2RlZmF1bHQvZGVwbG95bWVudHMvbWljcm9ib3QqJDU0NWYxMzJlLTBhY2MtMTFlNy1iYjFmLTA4MDAyN2Y5M2ZhOTIAOAFCDAiM063GBRCA+peZAVoPCgNydW4SCG1pY3JvYm90YiYKIWRlcGxveW1lbnQua3ViZXJuZXRlcy5pby9yZXZpc2lvbhIBMXoAEoACCAESEQoPCgNydW4SCG1pY3JvYm90GsEBCiMKABIAGgAiACoAMgA4AEIAWg8KA3J1bhIIbWljcm9ib3R6ABKZARJXCghtaWNyb2JvdBITcm90aGdhci9taWNyb2JvdDp2MSoAQgBqFC9kZXYvdGVybWluYXRpb24tbG9ncgxJZk5vdFByZXNlbnSAAQCIAQCQAQCiAQRGaWxlGgZBbHdheXMgHjIMQ2x1c3RlckZpcnN0QgBKAFIAWABgAGgAcgCCAQCKAQCaARFkZWZhdWx0LXNjaGVkdWxlciIhCg1Sb2xsaW5nVXBkYXRlEhAKBggAEAEaABIGCAAQARoAKAA4ABp7CAEQARgBIAEoADJtCglBdmFpbGFibGUSBFRydWUiGE1pbmltdW1SZXBsaWNhc0F2YWlsYWJsZSokRGVwbG95bWVudCBoYXMgbWluaW11bSBhdmFpbGFiaWxpdHkuMgwIjNOtxgUQ7/zAnwE6DAiM063GBRDd/cCfATgBGgAiAA==

That looks base64 encoded when I pipe it to base64 -d but I'm obviously missing something (compression?)

k8s

 

extensions/v1beta1

Deployment

microbotdefault"@/apis/extensions/v1beta1/namespaces/default/deployments/microbot*$545f132e-0acc-11e7-bb1f-080027f93fa928B

                                                                                                                         ӭZ

rumicrobotb&

!deployment.kubernetes.io/revision1z

rumicrobot

#

"*28BZ

rumicrobotzW

microbotrothgar/microbot:v1*Bj/dev/termination-logr

                                                   IfNotPresentFileAlways 2

                                                                           ClusterFirstBJRX`hrdefault-scheduler"!

RollingUpdate

(8 (2m

        AvailableTrue"MinimumReplicasAvailable*$Deployment has minimum availability.2

                                                                                    ӭ:

                

I tried looking at https://github.com/kubernetes/kubernetes/tree/master/staging/src/k8s.io/apiserver/pkg/storage/ to see if I could figure out what it's doing but sadly my go understanding is still very basic and that's more code than I could reverse engineer.

Can someone help me understand

1. How the date is stored in etcd (steps to manually reversing the data would be great)
2. Why the data is stored this way (I'm assuming performance and small data size)
3. Can someone point me to the code so I can try to understand it better or maybe provide manual steps to encode the data using linux tools?

I'm just trying to understand some of the data formats better. Don't worry, I won't mess with this in a production cluster.

Thank you

[Clayton Coleman]

You have the unusual data we caught and fixed in Prevent protobuf storage with etcd2, which is base64 encoded protobuf (etcd2 does not support storing binary values).  In 1.6 we default to etcd3 mode, which supports binary values, and defaults to storing in "application/vnd.kubernetes.protobuf" which is the following form for the values

4 bytes - `k8s\x00`

protobuf encoding of runtime.Unknown (pkg/runtime/types.go#Unknown)

* the "typeMeta" field set to the same values of kind and apiVersion that would be returned from the API

* the "raw" field set to the protobuf encoded bytes for the golang struct identified by typeMeta - no magic number

You can decode this with:

head -4 (strip the magic number) | protoc --raw (decodes the proto)

You can then decode the raw body using protoc as wel.

--
You received this message because you are subscribed to the Google Groups "Kubernetes developer/contributor discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-dev+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-dev@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-dev/1af7a30f-457a-49e6-97ff-1bf8e24c01e6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Clayton Coleman]

Design is https://github.com/kubernetes/community/blob/master/contributors/design-proposals/protobuf.md

On Fri, Mar 17, 2017 at 1:46 AM, Clayton Coleman <ccol...@redhat.com> wrote:

You have the unusual data we caught and fixed in Prevent protobuf storage with etcd2, which is base64 encoded protobuf (etcd2 does not support storing binary values).  In 1.6 we default to etcd3 mode, which supports binary values, and defaults to storing in "application/vnd.kubernetes.protobuf" which is the following form for the values

4 bytes - `k8s\x00`

protobuf encoding of runtime.Unknown (pkg/runtime/types.go#Unknown)

* the "typeMeta" field set to the same values of kind and apiVersion that would be returned from the API

* the "raw" field set to the protobuf encoded bytes for the golang struct identified by typeMeta - no magic number

You can decode this with:

head -4 (strip the magic number) | protoc --raw (decodes the proto)

You can then decode the raw body using protoc as wel.

To post to this group, send email to kuberne...@googlegroups.com.

[Justin Garrison]

Thanks Clayton! The design reasoning is especially helpful for what I was looking for.

liggitt in slack also pointed me to the code.

pasted here for anyone else interested

https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/extensions/v1beta1/generated.proto#L191-L204

https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/extensions/v1beta1/generated.pb.go#L768-L776

https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/extensions/v1beta1/generated.pb.go#L5408

and let me know I can manually set the storage type with --storage-media-type flag for the api server.

Design is https://github.com/kubernetes/community/blob/master/contributors/design-proposals/protobuf.md

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-de...@googlegroups.com.