etcdctl stuck on localhost

[Jason Price]

Running fresh 2.0.0.  On centos7 with selinux disabled and firewalling disabled.  (yuck)

Within Vagrant, I have 4 servers: master1, master2, master3 and client1.  masters1-3 are running etcd and seem to be communicating just fine.  Started with the following command:

./etcd -name master1 -initial-advertise-peer-urls http://10.0.3.21:2380 \
    -listen-peer-urls http://10.0.3.21:2380 \
    -initial-cluster-token etcd-vagrant-1 \
    -initial-cluster master1=http://10.0.3.21:2380,master2=http://10.0.3.22:2380,master3=http://10.0.3.23:2380 \
    -initial-cluster-state new -listen-client-urls http://10.0.3.21:4001

The other masters increment the master numbers and their local IP addresses.

On client1 however, when I try to run etcdctl commands, I'm stuck.  Here are several examples:

1) Set single peer on CLI:

$ ./etcdctl --peers http://10.0.3.21:4001 --debug set foo 42
Cluster-Endpoints: http://localhost:2379, http://localhost:2379, http://localhost:2379, http://localhost:4001, http://localhost:4001, http://localhost:4001
Curl-Example: curl -X PUT http://localhost:2379/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:2379/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:2379/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:2379/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:4001/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:4001/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:4001/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:2379/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:2379/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:2379/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:4001/v2/keys/foo -d value=42
Error:  501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]

2) Set multiple peers on the CLI:

$ ./etcdctl --peers http://10.0.3.21:4001,http://10.0.3.22:4001,http://10.0.3.23:4001 --debug set foo 42
Cluster-Endpoints: http://localhost:2379, http://localhost:2379, http://localhost:2379, http://localhost:4001, http://localhost:4001, http://localhost:4001
Curl-Example: curl -X PUT http://localhost:2379/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:2379/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:2379/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:2379/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:4001/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:4001/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:4001/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:2379/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:2379/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:2379/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:4001/v2/keys/foo -d value=42
Error:  501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]

3) use env var to set peers:

$ ETCDCTL_PEERS="http://10.0.3.21:4001,http://10.0.3.22:4001,http://10.0.3.23:4001"
$ export ETCDCTL_PEERS
$ ./etcdctl --debug set foo 42Cluster-Endpoints: http://localhost:2379, http://localhost:2379, http://localhost:2379, http://localhost:4001, http://localhost:4001, http://localhost:4001
Curl-Example: curl -X PUT http://localhost:2379/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:2379/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:2379/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:2379/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:4001/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:4001/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:4001/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:2379/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:2379/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:2379/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:4001/v2/keys/foo -d value=42
Error:  501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]

4) Remove http from --peers command:

$ ./etcdctl --peers 10.0.3.21:4001,10.0.3.22:4001,10.0.3.23:4001 --debug set foo 42
Cluster-Endpoints: http://localhost:2379, http://localhost:2379, http://localhost:2379, http://localhost:4001, http://localhost:4001, http://localhost:4001
Curl-Example: curl -X PUT http://localhost:2379/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:2379/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:2379/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:2379/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:4001/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:4001/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:4001/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:2379/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:2379/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:2379/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:4001/v2/keys/foo -d value=42
Error:  501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]

5) Override localhost to be actually a remote host in /etc/hosts (TERRIBLE HACK! DO NOT USE)

$ ./etcdctl --debug set foo 42
Cluster-Endpoints: http://localhost:2379, http://localhost:2379, http://localhost:2379, http://localhost:4001, http://localhost:4001, http://localhost:4001
Curl-Example: curl -X PUT http://localhost:2379/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:2379/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:2379/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:2379/v2/keys/foo -d value=42
Curl-Example: curl -X PUT http://localhost:4001/v2/keys/foo -d value=42
42

What the heck am I doing wrong?

[Jason Price]

ok... so I've continued chewing on this issue, since it's a big problem for me.  It seems like etcdctl is overwriting the peers somewhere in ./etcdctl/command/handle.go, specifically in the rawhandle function.

I've added some debugging to my local copy as follows:

// rawhandle wraps the command function handlers and sets up the
// environment but performs no output formatting.
func rawhandle(c *cli.Context, fn handlerFunc) (*etcd.Response, error) {
    endpoints, err := getEndpoints(c)
    if err != nil {
        return nil, err
    }

    tr, err := getTransport(c)
    if err != nil {
        return nil, err
    }

    client := etcd.NewClient(endpoints)
    client.SetTransport(tr)

    if c.GlobalBool("debug") {
        fmt.Fprintf(os.Stderr, "one Cluster-Endpoints: %s\n", strings.Join(client.GetCluster(), ", "))
    }

    if c.GlobalBool("debug") {
        go dumpCURL(client)
    }

    if c.GlobalBool("debug") {
        fmt.Fprintf(os.Stderr, "two Cluster-Endpoints: %s\n", strings.Join(client.GetCluster(), ", "))
    }

    // Sync cluster.
    if !c.GlobalBool("no-sync") {
        fmt.Println("zDEBUG doing the no-sync")
        if ok := client.SyncCluster(); !ok {
            fmt.Println("zDEBUG no-sync !ok")
            handleError(FailedToConnectToHost, errors.New("cannot sync with the cluster using endpoints "+strings.Join(endpoints, ", ")))
        }
    }

    if c.GlobalBool("debug") {
        fmt.Fprintf(os.Stderr, "three Cluster-Endpoints: %s\n", strings.Join(client.GetCluster(), ", "))
    }

    // Execute handler function.
    return fn(c, client)
}




The relevant output when I attempt to connect is as follows:

$ ./bin/etcdctl --peers 10.0.3.21:4001,10.0.3.22:4001,10.0.3.23:4001 --debug ls / -recursive
ZDEBUGpeer string is %s 10.0.3.21:4001,10.0.3.22:4001,10.0.3.23:4001
ZDEBUG u and err 10.0.3.21:4001 <nil>
ZDEBUG u and err 10.0.3.22:4001 <nil>
ZDEBUG u and err 10.0.3.23:4001 <nil>
one Cluster-Endpoints: http://10.0.3.21:4001, http://10.0.3.22:4001, http://10.0.3.23:4001
two Cluster-Endpoints: http://10.0.3.21:4001, http://10.0.3.22:4001, http://10.0.3.23:4001
zDEBUG doing the no-sync
three Cluster-Endpoints: http://localhost:2379, http://localhost:2379, http://localhost:2379, http://localhost:4001, http://localhost:4001, http://localhost:4001



so somehow this block of code is overwriting the client list:

        if ok := client.SyncCluster(); !ok {
            fmt.Println("zDEBUG no-sync !ok")
            handleError(FailedToConnectToHost, errors.New("cannot sync with the cluster using endpoints "+strings.Join(endpoints, ", ")))
        }

Thoughts?  I'm currently picking apart InternalSyncCluster.  The line "c.cluster.updateFromStr(string(b))"  Seems to be the smoking gun... but something's not wired up together here...

[Yicheng Qin]

Sorry for late response.

1. You should set `-listen-client-urls` and `-advertise-client-urls` each time you start etcd. It will listen the client requests on the given urls, and publish the advertised urls to the cluster

2. etcdctl greps the advertised urls from the cluster and sends requests to them. Because you set incorrect advertised client urls(default value is localhost:2379, localhost:4001), etcdctl cannot connect to the cluster then.

3. you could use --no-sync to forbid the grep-advertised-urls process.

--
You received this message because you are subscribed to the Google Groups "etcd-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to etcd-dev+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.