--etcd_config file format / configure kube-apiserver to use etcd client certificates?

[Stephen Ayotte]

My goal is to configure kube-apiserver to use a client certificate when communicating with etcd.

I found this Issue relevant to the topic: https://github.com/kubernetes/kubernetes/issues/1338

That issue links to this PR (https://github.com/kubernetes/kubernetes/issues/1464) which allows an etcd config file to be specified.

Since the one links to the other I'm guessing it's possible within that file to tell the kube-apiserver builtin client to use a client-cert when talking to etcd, but I haven't found any documentation on the syntax or the names of the config knobs that can be specified within that file.

Is there a document or an example config file which includes configuration for an etcd-client certificate?

Is there some other straightforward way to do this (e.g. CLI options)?

[David Oppenheimer]

From the PR you mentioned, it looks like you can pass the -etcd_config=xxx command-line argument to point the API server to an etcd config file.

The client code that reads the file appears to be here

https://github.com/coreos/go-etcd/blob/master/etcd/client.go

(see function NewClientFromFile())

It appears to just use the standard Go mechanism for converting JSON to a Go struct (in this case the Client struct defined in that same file)

So, put a JSON representation of the Client struct into a file, use the -etcd_config flag, and I think it should work.

It looks like the fields that specify the cert information are in the Config struct which is in the Client struct (e.g. config.CertFile)

I hope this helps.

--
You received this message because you are subscribed to the Google Groups "Containers at Google" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-contain...@googlegroups.com.
To post to this group, send email to google-c...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-containers.
For more options, visit https://groups.google.com/d/optout.

[Stephen Ayotte]

Look reasonable?

{

  "cluster": [https://kube-master.internal:2379,https://kube-minion1.internal:2379,https://kube-minion2.internal:2379],

  "config": {

    "certFile": "/etc/etcd/kube-master.internal-client.pem",

    "keyFile": "/etc/etcd/kube-master.internal-client.pem",

    "caCertFiles": [ "/etc/etcd/kubecluster-ca.pem" ],

    "timeout": 1,

    "consistency": "STRONG_CONSISTENCY"

   }

}

If so, how can we/I get this example or suitable prose into the docs?

[Jay Vyas]

Sure, The ha examples/ vagrant recipe would benefit from clustering etcd.  

I didnt setup etcd on all nodes when I first pass implemented it, I didn't want to add any unnecessary feature other than podmaster ha.

SO, since We have a vagrant recipe in examples/high-availability.

I'd love to test or help on a patch that implements etcd clustering via mounted data dirs, especially The important thing to test is that the etcd containers restart and successfully rejoin the cluster .  If mounts aren't setup perfectly, they will fail to join, iirc

[Stephen Ayotte]

Jay are you suggesting improving that Vagrant recipe as a way to document this file's format/contents? 

[Jay Vyas]

Yes.  And of course in general it's a worthwhile improvement .

[Stephen Ayotte]

On Saturday, September 12, 2015 at 3:45:51 PM UTC-4, jay vyas wrote:

Yes.  And of course in general it's a worthwhile improvement .

I'm surprised! In a week of playing with Kubernetes, running into issues, and searching for resolutions I have not managed to stumble onto Vagrant recipes as a source of examples. Perhaps we can do something to improve the discoverability of those recipes.

Separately, I think I'm not understanding what you're suggesting technically. Achieving etcd clustering by sharing data dirs b/w instances sounds weird-- why would we bother with Raft if we can synchronize on a local filestore-- did you mean something else?

Finally, is there a generally best place to add documentation for the CLI flags for kube-apiserver? The online help is very terse; e.g. the --basic_auth_file flag says nothing about the format of THAT file, which as an administrator I'd guess would be in .htpasswd format, but having inspected the source appears to be a CSV (and I'm still not sure what hash formats it supports or where the UIDs are drawn from).

[Stephen Ayotte]

Update: I had the schema wrong, it should be more like this:

{

  "cluster": {

    "machines": ["https://kube-master.internal:2379","https://kube-minion1.internal:2379","https://kube-minion2.internal:2379"]

  },

  "config": {

    "certFile": "/etc/etcd/kube-master.internal.pem",

    "keyFile": "/etc/etcd/kube-master.internal.key",

    "caCertFiles": [ "/etc/etcd/kubecluster-ca.pem" ],

    "consistency": "STRONG_CONSISTENCY",

[Jay Vyas]

Re apiserver auth:  

I'm working on a PR to improve auth docs which will have examples for token basic auth files , stay tuned https://github.com/kubernetes/kubernetes/pull/13848

If you want to leave suggestions in that thread on other things you'd like to see we can incorporate them as well.

On Sep 12, 2015, at 3:57 PM, Stephen Ayotte <stephen...@gmail.com> wrote:

On Saturday, September 12, 2015 at 3:45:51 PM UTC-4, jay vyas wrote:

Yes.  And of course in general it's a worthwhile improvement .

I'm surprised! In a week of playing with Kubernetes, running into issues, and searching for resolutions I have not managed to stumble onto Vagrant recipes as a source of examples. Perhaps we can do something to improve the discoverability of those recipes.

Separately, I think I'm not understanding what you're suggesting technically. Achieving etcd clustering by sharing data dirs b/w instances sounds weird-- why would we bother with Raft if we can synchronize on a local filestore-- did you mean something else?

Finally, is there a generally best place to add documentation for the CLI flags for kube-apiserver? The online help is very terse; e.g. the --basic_auth_file flag says nothing about the format of THAT file, which as an administrator I'd guess would be in .htpasswd format, but having inspected the source appears to be a CSV (and I'm still not sure what hash formats it supports or where the UIDs are drawn Sent from my iPhone

 

On Sep 12, 2015, at 1:35 PM, Stephen Ayotte <stephen...@gmail.com> wrote:

Jay are you suggesting improving that Vagrant recipe as a way to document this file's format/contents? 

[Jay Vyas]

let me clarify: the sharing of the data dir is for cluster membership, NOT for sharing etcd data.... I may have not been clear when I explained .... Of course one would never share data dir between etcd nodes :)!

You see, The high-availability recipe uses etcd, but as I recall, when a container restarts, it has to re use the same data-dir, NOT because it needs the data BUT because that directory I think has some identifiers for the nodes ip... Of course you're right raft will allow nodes to update to consistency, but in etcd implementation unfortunately there was a issue with some part of datadir  being used for cluster Id and membership, which meant you can't just create a brand new one every time you relaunch a container,... If an IP has been used before, it's best to keep the data dir around .  

Tldr rejoining an. Etcd cluster from same ip can be a little tricky so it's good to keep the data dir .