DRBD failing after upgrade

[Chris Friedmann]

I have been running v2.0.0 for a while now with no issues.

I upgraded to 2.1.5 and corosync/pacemaker would not work.

I found that there were issues with the later 2.x releases so upgraded to 3.0.0z
now corosync/pacemaker works, but dbrd does not. I get the following errors
Nothing else has changed with the configuration.

[root@tjaxp0101san ~]# /etc/rc.d/rc.drbd start

Setting DRBD parameters...

r0: Invalid argument

Command 'drbdsetup-84 new-resource r0 --on-no-data-accessible=io-error' terminated with exit code 20

r1: Invalid argument

Command 'drbdsetup-84 new-resource r1 --on-no-data-accessible=io-error' terminated with exit code 20

r2: Invalid argument

Command 'drbdsetup-84 new-resource r2 --on-no-data-accessible=io-error' terminated with exit code 20

Waiting for device creation...

Waiting for connection...

error determining which timeout to use: minor not available

error determining which timeout to use: minor not available

error determining which timeout to use: minor not available

[root@tjaxp0101san ~]#

[Marc Smith]

I'd check the kernel log to see what it's complaining about.

--Marc

> --
> You received this message because you are subscribed to the Google Groups "esos-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to esos-users+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/esos-users/ba94c312-a3af-4fc1-a2d1-60eb5d98a5d5n%40googlegroups.com.

[Chris Friedmann]

There are no messages being logged in any log file /var/log when trying to start DRBD.

When trying to run the above command manually, I get the following

drbdsetup-84 new-resource r0 --on-no-data-accessible=io-error

bash: drbdsetup-84: command not found

[W. Heiss]

maybe try running dmesg?

also, running with bash -x may help, e.g. bash -x /etc/rc.d/rc.drbd start

[Chris Friedmann]

I think it might be a bug post 5.x kernel based on my research

Nothing is logged via dmesg either...

[root@tjaxp0101san ~]# bash -x /etc/rc.d/rc.drbd start

+ source /etc/rc.d/common

++ RC_D_PREFIX=/etc/rc.d

++ STOP_TIMEOUT=300

++ EARLY_SERVICES='rc.openibd rc.opensm'

++ LATE_SERVICES='rc.lldpad rc.sshd rc.iscsid rc.multipathd rc.mdraid rc.lvm2 rc.zfs rc.eio rc.dmcache rc.btier rc.drbd rc.rbdmap rc.sbd rc.corosync rc.dlm rc.clvmd rc.lvmlockd rc.pacemaker rc.fsmount rc.mhvtl rc.scst rc.nvmet rc.nfsd rc.nrpe rc.munin rc.snmpd rc.snmptrapd rc.nut rc.smartd rc.rpcagent rc.stunnel rc.nginx rc.telegraf rc.vmtoolsd rc.acpid rc.serial rc.watchdog rc.rasdaemon'

++ RUNNING=0

++ STOPPED=1

++ UNKNOWN=2

+ DRBDADM=/sbin/drbdadm

+ DEV_TIMEOUT=10

+ PROC_DRBD=/proc/drbd

+ check_args start

+ '[' 1 -ne 1 ']'

+ '[' start '!=' start ']'

+ export DRBD_DONT_WARN_ON_VERSION_MISMATCH=1

+ DRBD_DONT_WARN_ON_VERSION_MISMATCH=1

+ start

+ /bin/echo 'Setting DRBD parameters...'

Setting DRBD parameters...

+ /sbin/drbdadm adjust all

r0: Invalid argument

Command 'drbdsetup-84 new-resource r0 --on-no-data-accessible=io-error' terminated with exit code 20

r1: Invalid argument

Command 'drbdsetup-84 new-resource r1 --on-no-data-accessible=io-error' terminated with exit code 20

r2: Invalid argument

Command 'drbdsetup-84 new-resource r2 --on-no-data-accessible=io-error' terminated with exit code 20

+ /bin/echo 'Waiting for device creation...'

Waiting for device creation...

++ /sbin/drbdadm sh-resources

+ for i in `${DRBDADM} sh-resources`

++ /sbin/drbdadm sh-dev r0

+ for j in `${DRBDADM} sh-dev ${i}`

+ dev_timeout_local=10

+ '[' '!' -e /dev/drbd0 ']'

+ '[' 10 -gt 0 ']'

+ /bin/sleep 1

+ dev_timeout_local=9

+ '[' '!' -e /dev/drbd0 ']'

+ '[' 9 -gt 0 ']'

+ /bin/sleep 1

+ dev_timeout_local=8

+ '[' '!' -e /dev/drbd0 ']'

+ '[' 8 -gt 0 ']'

+ /bin/sleep 1

+ dev_timeout_local=7

+ '[' '!' -e /dev/drbd0 ']'

+ '[' 7 -gt 0 ']'

+ /bin/sleep 1

+ dev_timeout_local=6

+ '[' '!' -e /dev/drbd0 ']'

+ '[' 6 -gt 0 ']'

+ /bin/sleep 1

+ dev_timeout_local=5

+ '[' '!' -e /dev/drbd0 ']'

+ '[' 5 -gt 0 ']'

+ /bin/sleep 1

+ dev_timeout_local=4

+ '[' '!' -e /dev/drbd0 ']'

+ '[' 4 -gt 0 ']'

+ /bin/sleep 1

+ dev_timeout_local=3

+ '[' '!' -e /dev/drbd0 ']'

+ '[' 3 -gt 0 ']'

+ /bin/sleep 1

+ dev_timeout_local=2

+ '[' '!' -e /dev/drbd0 ']'

+ '[' 2 -gt 0 ']'

+ /bin/sleep 1

+ dev_timeout_local=1

+ '[' '!' -e /dev/drbd0 ']'

+ '[' 1 -gt 0 ']'

+ /bin/sleep 1

+ dev_timeout_local=0

+ '[' '!' -e /dev/drbd0 ']'

+ '[' 0 -gt 0 ']'

+ for i in `${DRBDADM} sh-resources`

++ /sbin/drbdadm sh-dev r1

+ for j in `${DRBDADM} sh-dev ${i}`

+ dev_timeout_local=10

+ '[' '!' -e /dev/drbd1 ']'

+ '[' 10 -gt 0 ']'

+ /bin/sleep 1

+ dev_timeout_local=9

+ '[' '!' -e /dev/drbd1 ']'

+ '[' 9 -gt 0 ']'

+ /bin/sleep 1

+ dev_timeout_local=8

+ '[' '!' -e /dev/drbd1 ']'

+ '[' 8 -gt 0 ']'

+ /bin/sleep 1

+ dev_timeout_local=7

+ '[' '!' -e /dev/drbd1 ']'

+ '[' 7 -gt 0 ']'

+ /bin/sleep 1

+ dev_timeout_local=6

+ '[' '!' -e /dev/drbd1 ']'

+ '[' 6 -gt 0 ']'

+ /bin/sleep 1

+ dev_timeout_local=5

+ '[' '!' -e /dev/drbd1 ']'

+ '[' 5 -gt 0 ']'

+ /bin/sleep 1

+ dev_timeout_local=4

+ '[' '!' -e /dev/drbd1 ']'

+ '[' 4 -gt 0 ']'

+ /bin/sleep 1

+ dev_timeout_local=3

+ '[' '!' -e /dev/drbd1 ']'

+ '[' 3 -gt 0 ']'

+ /bin/sleep 1

+ dev_timeout_local=2

+ '[' '!' -e /dev/drbd1 ']'

+ '[' 2 -gt 0 ']'

+ /bin/sleep 1

+ dev_timeout_local=1

+ '[' '!' -e /dev/drbd1 ']'

+ '[' 1 -gt 0 ']'

+ /bin/sleep 1

+ dev_timeout_local=0

+ '[' '!' -e /dev/drbd1 ']'

+ '[' 0 -gt 0 ']'

+ for i in `${DRBDADM} sh-resources`

++ /sbin/drbdadm sh-dev r2

+ for j in `${DRBDADM} sh-dev ${i}`

+ dev_timeout_local=10

+ '[' '!' -e /dev/drbd2 ']'

+ '[' 10 -gt 0 ']'

+ /bin/sleep 1

+ dev_timeout_local=9

+ '[' '!' -e /dev/drbd2 ']'

+ '[' 9 -gt 0 ']'

+ /bin/sleep 1

+ dev_timeout_local=8

+ '[' '!' -e /dev/drbd2 ']'

+ '[' 8 -gt 0 ']'

+ /bin/sleep 1

+ dev_timeout_local=7

+ '[' '!' -e /dev/drbd2 ']'

+ '[' 7 -gt 0 ']'

+ /bin/sleep 1

+ dev_timeout_local=6

+ '[' '!' -e /dev/drbd2 ']'

+ '[' 6 -gt 0 ']'

+ /bin/sleep 1

+ dev_timeout_local=5

+ '[' '!' -e /dev/drbd2 ']'

+ '[' 5 -gt 0 ']'

+ /bin/sleep 1

+ dev_timeout_local=4

+ '[' '!' -e /dev/drbd2 ']'

+ '[' 4 -gt 0 ']'

+ /bin/sleep 1

+ dev_timeout_local=3

+ '[' '!' -e /dev/drbd2 ']'

+ '[' 3 -gt 0 ']'

+ /bin/sleep 1

+ dev_timeout_local=2

+ '[' '!' -e /dev/drbd2 ']'

+ '[' 2 -gt 0 ']'

+ /bin/sleep 1

+ dev_timeout_local=1

+ '[' '!' -e /dev/drbd2 ']'

+ '[' 1 -gt 0 ']'

+ /bin/sleep 1

+ dev_timeout_local=0

+ '[' '!' -e /dev/drbd2 ']'

+ '[' 0 -gt 0 ']'

+ /bin/echo 'Waiting for connection...'

Waiting for connection...

+ /sbin/drbdadm wait-con-int

error determining which timeout to use: minor not available

error determining which timeout to use: minor not available

error determining which timeout to use: minor not available

[Marc Smith]

Try using the 'drbdadm' and 'drbdsetup' tools manually instead of via
rc/init script. Like you said, perhaps something has changed. Follow
the DRBD user guide and see which part is failing. Perhaps we need to
update the DRBD userland tools for the new kernel...

--Marc

> To view this discussion on the web visit https://groups.google.com/d/msgid/esos-users/73b90099-7f91-4efb-bf5e-d1c0c3c918f8n%40googlegroups.com.

[Chris Friedmann]

Here is the output from drbdadm and drbdsetup

[root@tjaxp0101san ~]# drbdadm up all

r0: Invalid argument

Command 'drbdsetup-84 new-resource r0 --on-no-data-accessible=io-error' terminated with exit code 20

drbdadm: new-minor r0: skipped due to earlier error

r1: Invalid argument

Command 'drbdsetup-84 new-resource r1 --on-no-data-accessible=io-error' terminated with exit code 20

drbdadm: new-minor r1: skipped due to earlier error

r2: Invalid argument

Command 'drbdsetup-84 new-resource r2 --on-no-data-accessible=io-error' terminated with exit code 20

drbdadm: new-minor r2: skipped due to earlier error

and from drbdsetup

[root@tjaxp0101san ~]# drbdsetup new-resource r0 --on-no-data-accessible=io-error

r0: Invalid argument

This started with the 3.0 release If I go back to 2.1.5, it works just fine.

[Marc Smith]

I'd look at those DRBD scripts and see where it returns exit status
"20" and that may give a clue. Usually DRBD logs something in the
kernel log when it's complaining, we don't get much to go on with the
output of those commands you tried.

Probably worth posting on the drbd-users mailing list as well.

--Marc

> To view this discussion on the web visit https://groups.google.com/d/msgid/esos-users/d7ad294e-b6e5-4c0b-8fca-0259bfc36a3en%40googlegroups.com.

[Chris Friedmann]

I have 3.0.0 installed as Primary, 2.1.5 installed as Secondary. If I boot the Primary, drbdadm executable (not the script) gives the error. If I boot to Secondary, drbadm executable works fine
based on my searches, it is definitely a bug with the 5.x kernel (3.0.0 is 5.4.69, 2.1.5 is 4.14.120)
This is the discussion I found
https://bugzilla.kernel.org/show_bug.cgi?id=204623
It does appear that the drbd-utils need to be updated.

[Chris Friedmann]

I have confirmed it is the drbd-utils. 
I downloaded drbd84-utils-9.12.2-1.el7.elrepo.x86_64.rpm and scp to my esos host

deleted the following files

----------------------------------------------------------

rm -rf /usr/lib/drbd/*

rm /usr/share/man/ja/man5/drbd.conf.5.gz

rm /usr/share/man/ja/man8/drbd.8.gz

rm /usr/share/man/ja/man8/drbdadm.8.gz

rm /usr/share/man/ja/man8/drbdmeta.8.gz

rm /usr/share/man/ja/man8/drbdmon.8.gz

rm /usr/share/man/ja/man8/drbdsetup.8.gz

rm /usr/share/man/man5/drbd.conf.5.gz

rm /usr/share/man/man8/drbd.8.gz

rm /usr/share/man/man8/drbdadm.8.gz

rm /usr/share/man/man8/drbdmeta.8.gz

rm /usr/share/man/man8/drbdmon.8.gz

rm /usr/share/man/man8/drbdsetup.8.gz

----------------------------------------------------------

rpm -i drbd84-utils-9.12.2-1.el7.elrepo.x86_64.rpm

Now /etc/rc.d/rc.drbd start works as expected.

[Chris Friedmann]

Even though this works, it is not persistent. I have to do it each time I reboot.

[Marc Smith]

Thanks Chris for researching this. I updated drbd-utils in master and
3.0.7 to version 9.16.0. When those builds post, if you can confirm
it's resolved, I'd greatly appreciate it.

--Marc

> To view this discussion on the web visit https://groups.google.com/d/msgid/esos-users/063c3532-ff31-4d1f-9f1d-1c6deb38ba76n%40googlegroups.com.

[Chris Friedmann]

I can confirm the latest master release does work without modification...

Thank you!

[Fabien CRUZ]

Hi, thank you i have search long time for this problem.

On download page http://download.esos-project.com/?prefix=packages/3.x.x/ i find only 3.0.6 and lower.

I need building ESOS from source or where find already built package ?

[Marc Smith]

Hmm... not sure why the 3.x.x versions aren't building automatically anymore. I just kicked one off let's see if it posts in a couple hours.

--Marc

To view this discussion on the web visit https://groups.google.com/d/msgid/esos-users/041797b5-ccf2-4dd7-989a-548f512ecf0fn%40googlegroups.com.

[Fabien CRUZ]

It's work, thank you.

I have succesfull build imself after few days of try.

I have make fresh install ubuntu 19.04 with 32GB disk.

After installation disable auto update and run this following commands :

1. sudo sed -i -re 's/([a-z]{2}\.)?archive.ubuntu.com|security.ubuntu.com/old-releases.ubuntu.com/g' /etc/apt/sources.list
2. sudo apt-get update && sudo apt-get dist-upgrade
3. sudo apt-get install build-essential
4. sudo apt-get install git autoconf gawk flex bison gcc-multilib unzip libtinfo-dev libtool lsscsi kpartx libxslt1-dev groff gettext xsltproc pkg-config autopoint zip dosfstools
5. git clone https://github.com/quantum/esos.git
6. cd esos
7. git checkout 3.x.x
8. autoconf
9. ./configure
10. sudo make symlink
11. make -j4 -w -Orecurse > /tmp/bootstrap.out 2>&1 ( -j [number of cpu core] )
12. Normally the previous command fail, run a second time without "-j4" make -w -Orecurse > /tmp/bootstrap.out 2>&1
13. sudo ./chroot_build.sh -j4 -w -Orecurse > /tmp/chroot.out 2>&1 ( -j [number of cpu core] )
14. sudo rm /tools
15. sudo make image
16. make pkg_dist

After the step 12 you have new file un ESOS directory "./chroot_build.sh" if this file doesn't exist 😢 and retry.