Not all cluster nodes are getting up when running the script to bring all cluster nodes up

[Sai Bhargavi Aireni]

Hi All

I have a script runAllNodes.sh with the below content:

./redis-server 7000/redis7000.conf &

./redis-server 7001/redis7001.conf &

./redis-server 7002/redis7002.conf &

./redis-server 7003/redis7003.conf &

./redis-server 7004/redis7004.conf &

./redis-server 7005/redis7005.conf &

But when I run it in the jenkins job, sometimes only few of them come up as stated in the below logs :

+ ./runAllNodes.sh
4706:C 28 Feb 2020 17:46:24.657 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
4706:C 28 Feb 2020 17:46:24.657 # Redis version=5.0.7, bits=64, commit=00000000, modified=0, pid=4706, just started
4706:C 28 Feb 2020 17:46:24.657 # Configuration loaded
4706:M 28 Feb 2020 17:46:24.658 * Increased maximum number of open files to 10032 (it was originally set to 4096).
4706:M 28 Feb 2020 17:46:24.658 # Could not create server TCP listening socket 0.0.0.0:7000: bind: Address already in use
4707:C 28 Feb 2020 17:46:24.662 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
4707:C 28 Feb 2020 17:46:24.662 # Redis version=5.0.7, bits=64, commit=00000000, modified=0, pid=4707, just started
4707:C 28 Feb 2020 17:46:24.662 # Configuration loaded
4709:C 28 Feb 2020 17:46:24.662 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
4709:C 28 Feb 2020 17:46:24.662 # Redis version=5.0.7, bits=64, commit=00000000, modified=0, pid=4709, just started
4709:C 28 Feb 2020 17:46:24.662 # Configuration loaded
4707:M 28 Feb 2020 17:46:24.663 * Increased maximum number of open files to 10032 (it was originally set to 4096).
4707:M 28 Feb 2020 17:46:24.663 # Could not create server TCP listening socket 0.0.0.0:7001: bind: Address already in use
4709:M 28 Feb 2020 17:46:24.663 * Increased maximum number of open files to 10032 (it was originally set to 4096).
4709:M 28 Feb 2020 17:46:24.663 # Could not create server TCP listening socket 0.0.0.0:7003: bind: Address already in use
4710:C 28 Feb 2020 17:46:24.665 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
4710:C 28 Feb 2020 17:46:24.665 # Redis version=5.0.7, bits=64, commit=00000000, modified=0, pid=4710, just started
4710:C 28 Feb 2020 17:46:24.665 # Configuration loaded
4710:M 28 Feb 2020 17:46:24.666 * Increased maximum number of open files to 10032 (it was originally set to 4096).
4708:C 28 Feb 2020 17:46:24.666 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
4708:C 28 Feb 2020 17:46:24.666 # Redis version=5.0.7, bits=64, commit=00000000, modified=0, pid=4708, just started
4708:C 28 Feb 2020 17:46:24.666 # Configuration loaded
4710:M 28 Feb 2020 17:46:24.667 * No cluster configuration found, I'm 2a8f09eccc351a62e01fdb4b28d1b8cd7bbbe264
4708:M 28 Feb 2020 17:46:24.667 * Increased maximum number of open files to 10032 (it was originally set to 4096).

Sometimes it work fine. Can anyone help me on this?

Thanks

Bhargavi

[Famous Blog]

I was waiting for 5 seconds after script to bring all the nodes up runs. But even after that sometimes am facing this issue. Any pointer on this?

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/8d2dd8f9-61bc-416c-ab15-8669d80076d7%40googlegroups.com.

[Tuco]

the logs say that address already in use, that means that something(mostly your redis server) is already running on those ports.

To unsubscribe from this group and stop receiving emails from it, send an email to redi...@googlegroups.com.

[Famous Blog]

These are the logs when running the runAllNodes.sh(attached below logs): 

+ fuser -k 7000/tcp
+ true
+ fuser -k 7001/tcp
+ true
+ fuser -k 7002/tcp
+ true
+ fuser -k 7003/tcp
+ true
+ fuser -k 7004/tcp
+ true
+ fuser -k 7005/tcp
+ true
[StartRedisCluster] $ /bin/sh -xe /tmp/jenkins4449940367398141783.sh
+ cd /home/redis-stable/src
+ ./runAllNodes.sh
21595:C 09 Mar 2020 11:24:00.211 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
21595:C 09 Mar 2020 11:24:00.211 # Redis version=5.0.7, bits=64, commit=00000000, modified=0, pid=21595, just started
21595:C 09 Mar 2020 11:24:00.211 # Configuration loaded
21599:C 09 Mar 2020 11:24:00.211 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
21599:C 09 Mar 2020 11:24:00.211 # Redis version=5.0.7, bits=64, commit=00000000, modified=0, pid=21599, just started
21599:C 09 Mar 2020 11:24:00.211 # Configuration loaded
21595:M 09 Mar 2020 11:24:00.212 * Increased maximum number of open files to 10032 (it was originally set to 4096).
21599:M 09 Mar 2020 11:24:00.212 * Increased maximum number of open files to 10032 (it was originally set to 4096).
21598:C 09 Mar 2020 11:24:00.213 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
21598:C 09 Mar 2020 11:24:00.213 # Redis version=5.0.7, bits=64, commit=00000000, modified=0, pid=21598, just started
21598:C 09 Mar 2020 11:24:00.213 # Configuration loaded
21595:M 09 Mar 2020 11:24:00.213 * No cluster configuration found, I'm f6baaa56d72fa60c8a862f1dbd98249644f47c3f
21599:M 09 Mar 2020 11:24:00.213 * No cluster configuration found, I'm 195d87265c3a935e1dcd6f12ed83f02162564276
21597:C 09 Mar 2020 11:24:00.214 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
21597:C 09 Mar 2020 11:24:00.214 # Redis version=5.0.7, bits=64, commit=00000000, modified=0, pid=21597, just started
21597:C 09 Mar 2020 11:24:00.214 # Configuration loaded
21598:M 09 Mar 2020 11:24:00.214 * Increased maximum number of open files to 10032 (it was originally set to 4096).
21597:M 09 Mar 2020 11:24:00.215 * Increased maximum number of open files to 10032 (it was originally set to 4096).
21598:M 09 Mar 2020 11:24:00.216 * No cluster configuration found, I'm dfbca0850df6edba6c90fdb8558d8965b51e2247
21597:M 09 Mar 2020 11:24:00.216 * No cluster configuration found, I'm 9b0697c077d710710faaae9592d4a9b2b3cea691
21600:C 09 Mar 2020 11:24:00.217 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
21600:C 09 Mar 2020 11:24:00.217 # Redis version=5.0.7, bits=64, commit=00000000, modified=0, pid=21600, just started
21600:C 09 Mar 2020 11:24:00.217 # Configuration loaded
[StartRedisCluster] $ /bin/sh -xe /tmp/jenkins3357839519215680844.sh
+ lsof -i
+ grep redis
redis-ser 21595   root    6u  IPv4 5423241      0t0  TCP *:afs3-fileserver (LISTEN)
redis-ser 21595   root    8u  IPv4 5423246      0t0  TCP *:17000 (LISTEN)
redis-ser 21597   root    6u  IPv4 5428348      0t0  TCP *:afs3-prserver (LISTEN)
redis-ser 21597   root    8u  IPv4 5423070      0t0  TCP *:17002 (LISTEN)
redis-ser 21598   root    6u  IPv4 5422620      0t0  TCP *:afs3-vlserver (LISTEN)
redis-ser 21598   root    8u  IPv4 5423249      0t0  TCP *:17003 (LISTEN)
redis-ser 21599   root    6u  IPv4 5429432      0t0  TCP *:afs3-kaserver (LISTEN)
redis-ser 21599   root    8u  IPv4 5429435      0t0  TCP *:17004 (LISTEN)
redis-ser 21600   root    6u  IPv4 5429434      0t0  TCP *:afs3-volser (LISTEN)
redis-ser 21600   root    8u  IPv4 5429438      0t0  TCP *:17005 (LISTEN)
[StartRedisCluster] $ /bin/sh -xe /tmp/jenkins6671663477957992193.sh
+ sleep 5s
+ cd /home/redis-stable/src
+ echo yes
+ ./redis-cli --cluster create 10.225.68.157:7000 10.225.68.157:7001 10.225.68.157:7002 10.225.68.157:7003 10.225.68.157:7004 10.225.68.157:7005 --cluster-replicas 1
Could not connect to Redis at 10.225.68.157:7001: Connection refused

runAllNodes.sh

./redis-server 7000/redis7000.conf &
./redis-server 7001/redis7001.conf &
./redis-server 7002/redis7002.conf &
./redis-server 7003/redis7003.conf &
./redis-server 7004/redis7004.conf &
./redis-server 7005/redis7005.conf &

The issue here is unable to bring the redis server at 7001 port. And I could not find any logs regarding not able to bring up the redis server on 7001.

Thanks 

Bhargavi

To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/53d636b7-bbab-4c06-8c7d-f66bf20271a5%40googlegroups.com.