vitess cluster startup failure

[bent...@taboola.com]

Hi

The vttablet-up.sh script failed to start mysql

I have done the following steps:

1. install GO

2. install Percona 5.7

rpm -qa | grep Percona 
Percona-Server-server-57-5.7.21-20.1.el7.x86_64 
Percona-Server-shared-compat-57-5.7.21-20.1.el7.x86_64 
Percona-Server-client-57-5.7.21-20.1.el7.x86_64 
Percona-Server-devel-57-5.7.21-20.1.el7.x86_64 
Percona-Server-shared-57-5.7.21-20.1.el7.x86_64

mysqld --version
mysqld  Ver 5.7.21-20 for Linux on x86_64 (Percona Server (GPL), Release 20, Revision ed217b06ca3)

service mysqld status
Redirecting to /bin/systemctl status mysqld.service
● mysqld.service - MySQL Server
  Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
  Active: inactive (dead)...

ls -l /bin/mysql 
-rwxr-xr-x. 1 root root 3905088 Feb 16 10:12 /bin/mysql 

3. install jdk

4. define all variables:

env | grep -i vt 

VTDATAROOT=/vtdataroot 
VTTOP=/opt/vitess/src/vitess.io/vitess 
VT_TEST_FLAGS=--topo-server-flavor=consul 
VTROOT=/opt/vitess 
VT_MYSQL_ROOT=/ 
VTPORTSTART=15000 
PYTHONPATH=/opt/vitess/py-vtdb:/opt/vitess/dist/selenium/lib/python2.7/site-packages:/opt/vitess/dist/py-mock-1.0.1/lib/python2.7/site-packages:/opt/vitess/dist/grpc/usr/local/lib/python2.7/site-packages: 
[root@vitess703 bin]# env | grep -i mysql 
MYSQL_FLAVOR=MySQL56 
VT_MYSQL_ROOT=/

5. built vitess

6. started ZK

root     30451     1  0 10:40 ?        00:00:00 /bin/bash /opt/vitess/bin/zksrv.sh /vtdataroot/zk_001/logs /vtdataroot/zk_001/zoo.cfg /vtdataroot/zk_001/zk.pid
root     30455 30451  0 10:40 ?        00:00:04 /usr/bin/java -server -DZOO_LOG_DIR=/vtdataroot/zk_001/logs -cp /opt/vitess/dist/vt-zookeeper-3.4.10/lib/zookeeper-3.4.10-fatjar.jar:/usr/local/lib/zookeeper-3.4.10-fatjar.jar:/usr/share/jav
a/zookeeper-3.4.10.jar org.apache.zookeeper.server.quorum.QuorumPeerMain /vtdataroot/zk_001/zoo.cfg

7. starting vtctld

./vtctld-up.sh     
Starting vtctld...
Access vtctld web UI at http://vitess701.gus.taboolasyndication.com:15000
Send commands with: vtctlclient -server vitess701.gus.taboolasyndication.com:15999 ...

cat $VTDATAROOT/tmp/vtctld.out
F0313 11:44:59.831021    8359 servenv.go:87] servenv.Init: running this as root makes no sense

the vtctld is not running, what could be the problem?

1. what am I missing in the mysql configuration?

./vttablet-up.sh
Starting mysqld for all shards
Starting mysqld for shard -80 ...
Running mysqld-up-instance-100.sh
Starting MySQL for tablet cell1-0000000100...
W0313 10:46:18.241393    6049 mysqld.go:231] mysqld_safe not found in any of //{sbin,bin}: trying to launch mysqld instead
E0313 10:51:12.678403    6049 mysqld.go:532] failed starting mysqld in time (check mysyql error log /vtdataroot/vt_0000000100/error.log for more info): deadline exceeded waiting for mysqld socket file to appear: /vtdataroot/vt_0000000100/
mysql.sock
E0313 10:51:12.679289    6049 mysqlctl.go:260] failed init mysql: deadline exceeded waiting for mysqld socket file to appear: /vtdataroot/vt_0000000100/mysql.sock

cd /vtdataroot/vt_0000000101

cat error.log
2018-03-13T10:51:14.388860Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2018-03-13T10:51:14.388926Z 0 [Warning] 'NO_ZERO_DATE', 'NO_ZERO_IN_DATE' and 'ERROR_FOR_DIVISION_BY_ZERO' sql modes should be used with strict mode. They will be merged with strict mode in a future release.
2018-03-13T10:51:14.388930Z 0 [Warning] 'NO_AUTO_CREATE_USER' sql mode was not set.
2018-03-13T10:51:14.388991Z 0 [ERROR] Can't find error-message file '/share/percona-server/errmsg.sys'. Check error-message file location and 'lc-messages-dir' configuration directive.
2018-03-13T10:51:14.395652Z 0 [Warning] InnoDB: Using innodb_support_xa is deprecated and the parameter may be removed in future releases. Only innodb_support_xa=ON is allowed.
2018-03-13T10:51:15.568170Z 0 [Warning] InnoDB: New log files created, LSN=45790
2018-03-13T10:51:16.278937Z 0 [Warning] InnoDB: Creating foreign key constraint system tables.
2018-03-13T10:51:16.390264Z 0 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 746675e2-26ac-11e8-91a3-42010a800022.
2018-03-13T10:51:16.392188Z 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
2018-03-13T10:51:16.701119Z 0 [Warning] CA certificate ca.pem is self signed.
2018-03-13T10:51:16.901375Z 1 [Warning] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
2018-03-13T10:51:17.603072Z 1 [Warning] 'user' entry 'root@localhost' ignored in --skip-name-resolve mode.
2018-03-13T10:51:17.603096Z 1 [Warning] 'user' entry 'mysql.session@localhost' ignored in --skip-name-resolve mode.
2018-03-13T10:51:17.603103Z 1 [Warning] 'user' entry 'mysql.sys@localhost' ignored in --skip-name-resolve mode.
2018-03-13T10:51:17.603113Z 1 [Warning] 'db' entry 'performance_schema mysql.session@localhost' ignored in --skip-name-resolve mode.
2018-03-13T10:51:17.603116Z 1 [Warning] 'db' entry 'sys mysql.sys@localhost' ignored in --skip-name-resolve mode.
2018-03-13T10:51:17.603123Z 1 [Warning] 'proxies_priv' entry '@ root@localhost' ignored in --skip-name-resolve mode.
2018-03-13T10:51:17.603157Z 1 [Warning] 'tables_priv' entry 'user mysql.session@localhost' ignored in --skip-name-resolve mode.
2018-03-13T10:51:17.603163Z 1 [Warning] 'tables_priv' entry 'sys_config mysql.sys@localhost' ignored in --skip-name-resolve mode.

2. each tablet will create mysql instance on different port? or tablets will reside on same mysql instance?

[bent...@taboola.com]

I removed Percona 5.7 ans installed mysql community 5.6

running:

./vttablet-up.sh
Starting MySQL for tablet test-0000000100...
Resuming from existing vttablet dir:
   /vtdataroot/vt_0000000100
Starting MySQL for tablet test-0000000101...
Resuming from existing vttablet dir:
   /vtdataroot/vt_0000000101
Starting MySQL for tablet test-0000000102...
Resuming from existing vttablet dir:
   /vtdataroot/vt_0000000102
Starting MySQL for tablet test-0000000103...
Resuming from existing vttablet dir:
   /vtdataroot/vt_0000000103
Starting MySQL for tablet test-0000000104...
Resuming from existing vttablet dir:
   /vtdataroot/vt_0000000104

E0313 12:22:20.111479   13130 mysqlctl.go:260] failed start mysql: deadline exceeded waiting for mysqld socket file to appear: /vtdataroot/vt_0000000103/mysql.sock
E0313 12:22:20.111485   13131 mysqlctl.go:260] failed start mysql: deadline exceeded waiting for mysqld socket file to appear: /vtdataroot/vt_0000000104/mysql.sock
E0313 12:22:20.111727   13128 mysqlctl.go:260] failed start mysql: deadline exceeded waiting for mysqld socket file to appear: /vtdataroot/vt_0000000101/mysql.sock
E0313 12:22:20.111863   13127 mysqlctl.go:260] failed start mysql: deadline exceeded waiting for mysqld socket file to appear: /vtdataroot/vt_0000000100/mysql.sock
E0313 12:22:20.112168   13129 mysqlctl.go:260] failed start mysql: deadline exceeded waiting for mysqld socket file to appear: /vtdataroot/vt_0000000102/mysql.sock
Starting vttablet for test-0000000100...
Access tablet test-0000000100 at http://SERVERNAME:15100/debug/status
Starting vttablet for test-0000000101...
Access tablet test-0000000101 at http://SERVERNAME:15101/debug/status 

Starting vttablet for test-0000000102...
Access tablet test-0000000102 at http://SERVERNAME:15102/debug/status 

Starting vttablet for test-0000000103...
Access tablet test-0000000103 at http://SERVERNAME:15103/debug/status 

Starting vttablet for test-0000000104...
Access tablet test-0000000104 at http://SERVERNAME/debug/status

on second terminal i see:

ps -ef | grep mysql
root     13127 13123  0 12:17 pts/0    00:00:00 /opt/vitess/bin/mysqlctl -log_dir /vtdataroot/tmp -tablet_uid 100 -db-config-dba-uname vt_dba -db-config-dba-charset utf8 -mysql_port 17100 start
root     13128 13123  0 12:17 pts/0    00:00:00 /opt/vitess/bin/mysqlctl -log_dir /vtdataroot/tmp -tablet_uid 101 -db-config-dba-uname vt_dba -db-config-dba-charset utf8 -mysql_port 17101 start
root     13129 13123  0 12:17 pts/0    00:00:00 /opt/vitess/bin/mysqlctl -log_dir /vtdataroot/tmp -tablet_uid 102 -db-config-dba-uname vt_dba -db-config-dba-charset utf8 -mysql_port 17102 start
root     13130 13123  0 12:17 pts/0    00:00:00 /opt/vitess/bin/mysqlctl -log_dir /vtdataroot/tmp -tablet_uid 103 -db-config-dba-uname vt_dba -db-config-dba-charset utf8 -mysql_port 17103 start
root     13131 13123  0 12:17 pts/0    00:00:00 /opt/vitess/bin/mysqlctl -log_dir /vtdataroot/tmp -tablet_uid 104 -db-config-dba-uname vt_dba -db-config-dba-charset utf8 -mysql_port 17104 start


after installing mysql rpm "yum install mysql-community-server" do I need to do manual steps before running the vtctld-up.sh ?

[bent...@taboola.com]

what scripts need to run as root user and what as different user?

after creating vitess group and user i succeed to start vtctld

but the section to create mysql instances still fail.

On Tuesday, March 13, 2018 at 1:50:52 PM UTC+2, bent...@taboola.com wrote:

[Sugu Sougoumarane]

Just a quick sanity check: can you see if apparmor is running on your system? These failures may be caused by that.

--
You received this message because you are subscribed to the Google Groups "vitess" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vitess+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[bent...@taboola.com]

we are running on centos 7, no apparmor and the selinux configure to:

getenforce

Permissive

also firewall is disabled

also what components must be run as root user and what as different user? (the vtctld started only when running not as root user)

To unsubscribe from this group and stop receiving emails from it, send an email to vitess+un...@googlegroups.com.

[Sugu Sougoumarane]

https://github.com/vitessio/vitess/issues/3175 may be of interest to you.

To unsubscribe from this group and stop receiving emails from it, send an email to vitess+unsubscribe@googlegroups.com.

[bent...@taboola.com]

still getting failure when configure percona 5.7.

I manually created vitess user and group

running the zk-up.sh from the examples/local give error: "mysql_safe not found in path"

I want to try working with the base docker image on 3 physical nodes.

I run:

docker run -h `hostname` -ti vitess/base bash

each docker can ping other hosts.

what is the best approach for setting vitess cluster on the 3 dockers? (the examples/local brings all on one node, should I need manually edit it?)

[Sugu Sougoumarane]

If you want to bring them up in separate nodes, might as well use kubernetes. There are helm charts under the helm directory for you to start with.

If so, you can also use the #kubernetes sub-channel to ask questions.

To unsubscribe from this group and stop receiving emails from it, send an email to vitess+unsubscribe@googlegroups.com.

[Simon Mudd]

> On 19 Mar 2018, at 10:43, bentzi.m via vitess <vit...@googlegroups.com> wrote:
>
> still getting failure when configure percona 5.7.
> I manually created vitess user and group

> running the zk-up.sh from the examples/local give error: "mysql_safe not found in path”

Are you sure this isn’t an error but a warning? I’ve seen this several time with mysql-community 5.7 rpms and it looks “worrying”
but actually mysqld does get started fine. It’s confusing but should be ok to ignore this.

Simon

[bent...@taboola.com]

rpm -qa | grep -i percona
Percona-Server-server-57-5.7.21-20.1.el7.x86_64
Percona-Server-shared-compat-57-5.7.21-20.1.el7.x86_64
Percona-Server-client-57-5.7.21-20.1.el7.x86_64
Percona-Server-devel-57-5.7.21-20.1.el7.x86_64
percona-release-0.1-4.noarch
Percona-Server-shared-57-5.7.21-20.1.el7.x86_64

[vitess@vitess702 vitess]$ cd src/vitess.io/vitess/examples/local/

./zk-up.sh
which: no mysqld_safe in (/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/vitess/bin:/usr/local/go/bin:/bin/etcd:/sbin/mysqld)

cat env.sh  | grep safe
# Try to find mysqld_safe on PATH.
 mysql_path=`which mysqld_safe`
   echo "Can't guess location of mysqld_safe. Please set VT_MYSQL_ROOT so it can be found at \$VT_MYSQL_ROOT/bin/mysqld_safe."

[bent...@taboola.com]

changed the which to search for mysqld instead and the zk is up

[bent...@taboola.com]

now i ran:

[vitess@vitess702 local]$ ./vtctld-up.sh
Starting vtctld...
Access vtctld web UI at http://vitess702.gus.taboolasyndication.com:15000
Send commands with: vtctlclient -server vitess702.gus.taboolasyndication.com:15999 ...

[vitess@vitess702 local]$ netstat -na | grep 15000
tcp6       0      0 :::15000                :::*                    LISTEN      
tcp6       0      0 10.128.0.35:15000       172.25.13.24:54810      CLOSE_WAIT  
tcp6       0      0 10.128.0.35:15000       172.25.13.24:54890      ESTABLISHED
tcp6       0      0 10.128.0.35:15000       172.25.13.24:54892      ESTABLISHED
tcp6       0      0 10.128.0.35:15000       172.25.13.24:54888      ESTABLISHED
tcp6       0      0 10.128.0.35:15000       172.25.13.24:54808      CLOSE_WAIT  
[vitess@vitess702 local]$ netstat -na | grep 15999
tcp6       0      0 :::15999                :::*                    LISTEN      
[vitess@vitess702 local]$ ./lvtctl.sh InitShardMaster -force test_keyspace/0 test-100
./lvtctl.sh: line 19: exec: vtctlclient: not found

any idea what it on tcp6?

i only can access to 15000

also on other server get this error:

./zk-up.sh
Starting zk servers...
Resuming from existing ZK data dir:
   /vtdataroot/zk_001
Resuming from existing ZK data dir:
   /vtdataroot/zk_002
Resuming from existing ZK data dir:
   /vtdataroot/zk_003
Waiting for zk servers to be ready...
Started zk servers.
E0320 12:48:38.647217    8788 vtctl.go:103] action failed: AddCellInfo zk: could not connect to a server
Configured zk servers.

[Sugu Sougoumarane]

Looks like 15999 is the grpc port. If you look at vtctlclient, it should be using that port to send commands to vtctld.

--
You received this message because you are subscribed to the Google Groups "vitess" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vitess+unsubscribe@googlegroups.com.

[bent...@taboola.com]

I succeed to brink zk, vtctld, vttablets from examples/local

now I am stuck on:

./lvtctl.sh InitShardMaster -force test_keyspace/0 test-100

+ exec vtctlclient -server localhost:15999 InitShardMaster -force test_keyspace/0 test-100
W0321 12:51:10.273940   20438 main.go:58] W0321 12:51:10.273512 reparent.go:181] master-elect tablet test-0000000100 is not the shard master, proceeding anyway as -force was used
W0321 12:51:10.275960   20438 main.go:58] W0321 12:51:10.273583 reparent.go:187] master-elect tablet test-0000000100 is not a master in the shard, proceeding anyway as -force was used

how to debug?

To unsubscribe from this group and stop receiving emails from it, send an email to vitess+un...@googlegroups.com.

[bent...@taboola.com]

log shows:

I0321 13:46:20.716778   18165 replication_reporter.go:67] Slave is stopped. Trying to reconnect to master...
I0321 13:46:20.717213   18165 replication_reporter.go:70] Failed to reconnect to master: no master tablet for shard test_keyspace/0
I0321 13:46:25.718059   18165 replication_reporter.go:67] Slave is stopped. Trying to reconnect to master...
I0321 13:46:25.718476   18165 replication_reporter.go:70] Failed to reconnect to master: no master tablet for shard test_keyspace/0
I0321 13:46:30.719666   18165 replication_reporter.go:67] Slave is stopped. Trying to reconnect to master...
I0321 13:46:30.720109   18165 replication_reporter.go:70] Failed to reconnect to master: no master tablet for shard test_keyspace/0
I0321 13:46:35.721026   18165 replication_reporter.go:67] Slave is stopped. Trying to reconnect to master...
I0321 13:46:35.721552   18165 replication_reporter.go:70] Failed to reconnect to master: no master tablet for shard test_keyspace/0
I0321 13:46:40.722514   18165 replication_reporter.go:67] Slave is stopped. Trying to reconnect to master...
I0321 13:46:40.723000   18165 replication_reporter.go:70] Failed to reconnect to master: no master tablet for shard test_keyspace/0


ps -ef | grep mysqld
vitess   18007     1  0 12:30 pts/1    00:00:02 /usr/sbin/mysqld --defaults-file=/vtdataroot/vt_0000000104/my.cnf
vitess   18008     1  0 12:30 pts/1    00:00:03 /usr/sbin/mysqld --defaults-file=/vtdataroot/vt_0000000100/my.cnf
vitess   18009     1  0 12:30 pts/1    00:00:02 /usr/sbin/mysqld --defaults-file=/vtdataroot/vt_0000000102/my.cnf
vitess   18010     1  0 12:30 pts/1    00:00:04 /usr/sbin/mysqld --defaults-file=/vtdataroot/vt_0000000101/my.cnf
vitess   18011     1  0 12:30 pts/1    00:00:02 /usr/sbin/mysqld --defaults-file=/vtdataroot/vt_0000000103/my.cnf

cat /vtdataroot/vt_0000000100/my.cnf | grep port
innodb_support_xa = 0
port = 17100
[vitess@vitess702 tmp]$ netstat -na | grep 17100
tcp6       0      0 :::17100                :::*                    LISTEN

[Sugu Sougoumarane]

The warnings are harmless. The command most likely worked. You should be able to proceed to the next step.

To unsubscribe from this group and stop receiving emails from it, send an email to vitess+unsubscribe@googlegroups.com.

[Sugu Sougoumarane]

+enisoc/derek (in case they know more)

This looks like a problem with the etcd operator itself. There must be some logs somewhere about why it's not coming up.

On 21 March 2018 at 03:01, 조수연 <ace...@gmail.com> wrote:

Hi~

The etcd-up.sh script failed.

vitess/examples/kubernetes$ ./etcd-up.sh

msg is

No resources found.

Downloading and installing etcd-operator for first time use...

Cloning into 'etcd-operator'...

remote: Counting objects: 13162, done.

remote: Compressing objects: 100% (50/50), done.

remote: Total 13162 (delta 13), reused 25 (delta 6), pack-reused 13102

Receiving objects: 100% (13162/13162), 3.04 MiB | 1.30 MiB/s, done.

Resolving deltas: 100% (7911/7911), done.

deployment "etcd-operator" created

No resources found.

etcd-operator installation failed.

installed at google cloud kubernetes Engine.

I had a problem with the steps below.

Running Vitess on Kubernetes( https://vitess.io/getting-started/#start-a-vitess-cluster )

 Start a Vitess cluster

  3. Start an etcd cluster

What is the problem and how will it work?

Thank you.

- Aceinme

[Anthony Yeh]

I think this is https://github.com/vitessio/vitess/issues/3439

We really should not be trying to install etcd-operator as part of etcd-up.sh since etcd-operator is a shared resource. I'll work on a change to the docs and script to clarify this.

To unsubscribe from this group and stop receiving emails from it, send an email to vitess+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "vitess" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vitess+un...@googlegroups.com.

[Sugu Sougoumarane]

Sorry about replying late on this.

It feels like some http proxy is trying to authenticate the request, and that part is failing (Squid software?).

Your non-http requests seem to be working fine.

On 28 March 2018 at 02:57, bentzi.m via vitess <vit...@googlegroups.com> wrote:

we set vitess to with zk not etcd.

we are not using kubernetes yet.

we are installing on 3 physical hosts.

when starting vtctld  port 15000 and 15999 listen on tcp6 even after I disable ipv6 by setting in /etc/sysctl.conf:

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

[vitess@vitess701 bin]$ netstat -na | grep 15

tcp6       0      0 :::15000                :::*                    LISTEN      

tcp6       0      0 :::15999                :::*                    LISTEN 

I can connect to http://vitess701.gus.taboolasyndication.com:15000

now when running:
orig_shards=$(python -c "import json; print ' '.join(json.loads(open('/opt/vitess/vitess-deployment/config/vttablet.json').read())['shard_sets'][0])")

echo $orig_shards
-80 80-

vtctlclient -server vitess701.gus.taboolasyndication.com:15999 ListShardTablets cell1_keyspace/-80

i get communication error:

W0328 09:37:41.238172    5569 clientconn.go:934] grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing failed to do connect handshake, response: \"HTTP/1.1 403 Forbidden\
\r\\nContent-Length: 3587\\r\\nConnection: keep-alive\\r\\nContent-Language: en\\r\\nContent-Type: text/html;charset=utf-8\\r\\nDate: Wed, 28 Mar 2018 09:37:41 GMT\\r\\nMime-Version: 1.0\\r\\nServer: squid/3.5.20\\r\\nVary: Accept-Languag
e\\r\\nVia: 1.1 proxy701.gus.taboolasyndication.com (squid/3.5.20)\\r\\nX-Cache: MISS from proxy701.gus.taboolasyndication.com\\r\\nX-Cache-Lookup: NONE from proxy701.gus.taboolasyndication.com:3128\\r\\nX-Squid-Error: ERR_ACCESS_DENIED 0
\\r\\n\\r\\n<!DOCTYPE html PUBLIC \\\"-//W3C//DTD HTML 4.01//EN\\\" \\\"http://www.w3.org/TR/html4/strict.dtd\\\">\\n<html><head>\\n<meta type=\\\"copyright\\\" content=\\\"Copyright (C) 1996-2016 The Squid Software Foundation and contrib
utors\\\">\\n<meta http-equiv=\\\"Content-Type\\\" content=\\\"text/html; charset=utf-8\\\">\\n<title>ERROR: The requested URL could not be retrieved</title>\\n<style type=\\\"text/css\\\"><!--\\n /*\\n * Copyright (C) 1996-2016 The Squid
Software Foundation and contributors\\n *\\n * Squid software is distributed under GPLv2+ license and includes\\n * contributions from numerous individuals and organizations.\\n * Please see the COPYING and CONTRIBUTORS files for details
.\\n */\\n\\n/*\\n Stylesheet for Squid Error pages\\n Adapted from design by Free CSS Templates\\n http://www.freecsstemplates.org\\n Released for free under a Creative Commons Attribution 2.5 License\\n*/\\n\\n/* Page basics */\\n* {\\n
\\tfont-family: verdana, sans-serif;\\n}\\n\\nhtml body {\\n\\tmargin: 0;\\n\\tpadding: 0;\\n\\tbackground: #efefef;\\n\\tfont-size: 12px;\\n\\tcolor: #1e1e1e;\\n}\\n\\n/* Page displayed title area */\\n#titles {\\n\\tmargin-left: 15px;\\
n\\tpadding: 10px;\\n\\tpadding-left: 100px;\\n\\tbackground: url('/squid-internal-static/icons/SN.png') no-repeat left;\\n}\\n\\n/* initial title */\\n#titles h1 {\\n\\tcolor: #000000;\\n}\\n#titles h2 {\\n\\tcolor: #000000;\\n}\\n\\n/*
special event: FTP success page titles */\\n#titles ftpsuccess {\\n\\tbackground-color:#00ff00;\\n\\twidth:100%;\\n}\\n\\n/* Page displayed body content area */\\n#content {\\n\\tpadding: 10px;\\n\\tbackground: #ffffff;\\n}\\n\\n/* Genera
l text */\\np {\\n}\\n\\n/* error brief description */\\n#error p {\\n}\\n\\n/* some data which may have caused the problem */\\n#data {\\n}\\n\\n/* the error message received from the system or other software */\\n#sysmsg {\\n}\\n\\npre
{\\n    font-family:sans-serif;\\n}\\n\\n/* special event: FTP / Gopher directory listing */\\n#dirmsg {\\n    font-family: courier;\\n    color: black;\\n    font-size: 10pt;\\n}\\n#dirlisting {\\n    margin-left: 2%;\\n    margin-right:
2%;\\n}\\n#dirlisting tr.entry td.icon,td.filename,td.size,td.date {\\n    border-bottom: groove;\\n}\\n#dirlisting td.size {\\n    width: 50px;\\n    text-align: right;\\n    padding-right: 5px;\\n}\\n\\n/* horizontal lines */\\nhr {\\n
\\tmargin: 0;\\n}\\n\\n/* page displayed footer area */\\n#footer {\\n\\tfont-size: 9px;\\n\\tpadding-left: 10px;\\n}\\n\\n\\nbody\\n:lang(fa) { direction: rtl; font-size: 100%; font-family: Tahoma, Roya, sans-serif; float: right; }\\n:la
ng(he) { direction: rtl; }\\n --></style>\\n</head><body id=ERR_ACCESS_DENIED>\\n<div id=\\\"titles\\\">\\n<h1>ERROR</h1>\\n<h2>The requested URL could not be retrieved</h2>\\n</div>\\n<hr>\\n\\n<div id=\\\"content\\\">\\n<p>The following
error was encountered while trying to retrieve the URL: <a href=\\\"vitess701.gus.taboolasyndication.com:15999\\\">vitess701.gus.taboolasyndication.com:15999</a></p>\\n\\n<blockquote id=\\\"error\\\">\\n<p><b>Access Denied.</b></p>\\n</b
lockquote>\\n\\n<p>Access control configuration prevents your request from being allowed at this time. Please contact your service provider if you feel this is incorrect.</p>\\n\\n<p>Your cache administrator is <a href=\\\"mailto:root?sub
ject=CacheErrorInfo%20-%20ERR_ACCESS_DENIED&body=CacheHost%3A%20proxy701.gus.taboolasyndication.com%0D%0AErrPage%3A%20ERR_ACCESS_DENIED%0D%0AErr%3A%20%5Bnone%5D%0D%0ATimeStamp%3A%20Wed,%2028%20Mar%202018%2009%3A37%3A41%20GMT%0D%0A%0D%
0AClientIP%3A%2010.128.0.34%0D%0A%0D%0AHTTP%20Request%3A%0D%0ACONNECT%20%2F%20HTTP%2F1.1%0AUser-Agent%3A%20grpc-go%2F1.7.1%0D%0AHost%3A%20vitess701.gus.taboolasyndication.com%3A15999%0D%0A%0D%0A%0D%0A\\\">root</a>.</p>\\n<br>\\n</div>\\n\
\n<hr>\\n<div id=\\\"footer\\\">\\n<p>Generated Wed, 28 Mar 2018 09:37:41 GMT by proxy701.gus.taboolasyndication.com (squid/3.5.20)</p>\\n<!-- ERR_ACCESS_DENIED -->\\n</div>\\n</body></html>\\n\""; Reconnecting to {vitess701.gus.taboolasy
ndication.com:15999 0  <nil>}

so something is wrong with connection to 15999 port

when running:

./lvtctl.sh InitShardMaster -force cell1_keyspace/-80 cell1-100
W0328 09:46:49.662960    6828 main.go:58] W0328 09:46:49.662336 reparent.go:181] master-elect tablet cell1-0000000100 is not the shard master, proceeding anyway as -force was used
W0328 09:46:49.663979    6828 main.go:58] W0328 09:46:49.663189 reparent.go:187] master-elect tablet cell1-0000000100 is not a master in the shard, proceeding anyway as -force was used

and no progress and no replica become a master

log files shows:

tailf vttablet.vitess701.vitess.log.INFO.20180328-092745.3396
I0328 09:48:51.290930    3396 replication_reporter.go:67] Slave is stopped. Trying to reconnect to master...
I0328 09:48:51.291505    3396 replication_reporter.go:70] Failed to reconnect to master: no master tablet for shard cell1_keyspace/80-
I0328 09:48:56.292724    3396 replication_reporter.go:67] Slave is stopped. Trying to reconnect to master...
I0328 09:48:56.293376    3396 replication_reporter.go:70] Failed to reconnect to master: no master tablet for shard cell1_keyspace/80-

Any idea?

see attached file

To unsubscribe from this group and stop receiving emails from it, send an email to vitess+unsubscribe@googlegroups.com.

[bent...@taboola.com]

The issue present only on our Google servers, in house physical servers are working fine.

Will update in case we find the root cause.

Thanks.

[bent...@taboola.com]

we found the problem, http_proxy was configured, removing that solve the problem