tiDb docker crashing

[Wilfred Chau]

Hi,

    I have a docker cluster setup.   Have an issue.  I cannot log in to TiDB (port 4000) and TiKVs keep restarting.

    I restarted the entire cluster but to no avail.  Where is the log file located?

    Help!!

CONTAINER ID        IMAGE                          COMMAND                  CREATED             STATUS                         PORTS                                              NAMES
4d2f06b8cb50        pingcap/tidb:latest            "/tidb-server --st..."   2 weeks ago         Up 4 minutes                   0.0.0.0:4000->4000/tcp, 0.0.0.0:10080->10080/tcp   tidbdockercompose_tidb_1
dbbd7e4267ca        pingcap/tikv:latest            "/tikv-server --ad..."   2 weeks ago         Restarting (1) 2 seconds ago                                                      tidbdockercompose_tikv5_1
b158d4bc15de        pingcap/tikv:latest            "/tikv-server --ad..."   2 weeks ago         Up 5 seconds                   20160/tcp                                          tidbdockercompose_tikv4_1
15db79d5b02b        pingcap/tikv:latest            "/tikv-server --ad..."   2 weeks ago         Up 3 seconds                   20160/tcp                                          tidbdockercompose_tikv3_1
d7bd68e2e399        pingcap/tikv:latest            "/tikv-server --ad..."   2 weeks ago         Up Less than a second          20160/tcp                                          tidbdockercompose_tikv2_1
4c7a689e23b2        pingcap/tikv:latest            "/tikv-server --ad..."   2 weeks ago         Up 4 seconds                   20160/tcp                                          tidbdockercompose_tikv1_1
f45eac7b8691        pingcap/tikv:latest            "/tikv-server --ad..."   2 weeks ago         Restarting (1) 2 seconds ago                                                      tidbdockercompose_tikv6_1
898f70c5d36e        pingcap/tikv:latest            "/tikv-server --ad..."   2 weeks ago         Up 1 second                    20160/tcp                                          tidbdockercompose_tikv7_1
3c63b7251bf4        pingcap/tikv:latest            "/tikv-server --ad..."   2 weeks ago         Up 7 seconds                   20160/tcp                                          tidbdockercompose_tikv0_1
ca2cec2658a7        pingcap/pd:latest              "/pd-server --name..."   2 weeks ago         Up 6 minutes                   2380/tcp, 0.0.0.0:32835->2379/tcp                  tidbdockercompose_pd4_1
d7341b1d00cf        pingcap/pd:latest              "/pd-server --name..."   2 weeks ago         Up 6 minutes                   2380/tcp, 0.0.0.0:32834->2379/tcp                  tidbdockercompose_pd2_1
38418e0ec7b8        prom/prometheus:v2.0.0         "/bin/prometheus -..."   2 weeks ago         Up 4 minutes                   0.0.0.0:9090->9090/tcp                             tidbdockercompose_prometheus_1
d5afbae5b466        pingcap/pd:latest              "/pd-server --name..."   2 weeks ago         Up 6 minutes                   2380/tcp, 0.0.0.0:32836->2379/tcp                  tidbdockercompose_pd6_1
6c24a91d45ce        grafana/grafana:4.6.3          "/run.sh"                2 weeks ago         Up 3 minutes                   0.0.0.0:3000->3000/tcp                             tidbdockercompose_grafana_1
5331d3b0eec8        pingcap/tidb-vision:latest     "npm start"              2 weeks ago         Up 4 minutes                   0.0.0.0:8010->8010/tcp                             tidbdockercompose_tidb-vision_1
e53d0da74111        pingcap/pd:latest              "/pd-server --name..."   2 weeks ago         Up 6 minutes                   2380/tcp, 0.0.0.0:32833->2379/tcp                  tidbdockercompose_pd0_1
c1766e73d297        prom/pushgateway:v0.3.1        "/bin/pushgateway"       2 weeks ago         Up 4 minutes                   9091/tcp                                           tidbdockercompose_pushgateway_1

[shenli]

Seems that you've started at least 7 PDs, but some of them not started. You can check service logs by `docker-compose logs pd0` and `docker-compose logs tikv0` where `pd0` and `tikv0` can be replaced by other pd or tikv you've started. Also you can check if pd cluster is working correctly by `curl http://localhost:32833/pd/api/v1/members` where `32833` is your pd0 port

[Wilfred Chau]

Thanks Shenli,  exactly what you said.  Need to restart docker to fix it. 

[Wilfred Chau]

Hi Shenli,

     I used loader to import the mydumper backup but it failed with these errors;

2018/05/06 06:50:07.166 log.go:80: [error] rafthttp: [failed to heartbeat f739d1cd5bfb41ba on stream Message (write tcp 172.21.0.3:2380->172.21.0.7:58706: write: connection reset by peer)]
2018/05/06 06:50:12.641 etcd_kv.go:117: [error] load from etcd error: context deadline exceeded
2018/05/06 06:50:12.641 leader.go:129: [error] failed to load leader priority: context deadline exceeded
2018/05/06 06:50:12.847 etcd_kv.go:117: [error] load from etcd error: context deadline exceeded
2018/05/06 06:50:12.847 leader.go:81: [error] get leader err context deadline exceeded
2018/05/06 06:50:13.310 etcd_kv.go:117: [error] load from etcd error: context deadline exceeded


     any idea, how to fix and resume the data load?  thanks.