CentOS7 and systemd ordering when shutting down results in unclean unmount

[awidde...@hotmail.com]

Setup iSCSI on CentOS7. Mounted a iSCSI disk and am running a small MySQL instance on the disk. The iSCSI disk and MySQL instance all come online fine with booting but when shutting down things seem to get very upset and the drive does not get unmounted cleanly.

Does not look like I'm the only one having the issue. Another report that is very similar was posted here:

https://www.centos.org/forums/viewtopic.php?f=47&t=49337

This might be a systemd issue but figured I'd post here first to see if anyone else has had this issue and has any suggestions.

Relevant version info:

CentOS Linux release 7.0.1406 (Core)

systemd-208-11.el7_0.4.x86_64

iscsi-initiator-utils-6.2.0.873-21.el7.x86_64

Systemd unit file for MySQL server:

[Unit]

Description=MySQL Server

After=nss-lookup.target network.target remote-fs.target time-sync.target

Wants=nss-lookup.target network.target remote-fs.target time-sync.target

Logs from journalctl:

Dec 09 09:27:03 example.server.com kernel: EXT4-fs error (device sdb1): __ext4_get_inode_loc:4039: inode #58989720: block 235930089: comm mysqld: unable to read itable block

Dec 09 09:27:03 example.server.com kernel: EXT4-fs error (device sdb1) in ext4_reserve_inode_write:4962: IO failure

Dec 09 09:27:03 example.server.com iscsiadm[3906]: Logging out of session [sid: 1, target: example.target, portal: 192.168.1.30,3260]

Dec 09 09:27:03 example.server.com iscsiadm[3906]: Logout of [sid: 1, target: example.target, portal: 192.168.1.30,3260] successful.

Dec 09 09:27:03 example.server.com systemd[1]: Stopped Login and scanning of iSCSI devices.

Dec 09 09:27:03 example.server.com systemd[1]: Stopping Open-iSCSI...

Dec 09 09:27:03 example.server.com systemd[1]: Stopped Open-iSCSI.

Dec 09 09:27:03 example.server.com systemd[1]: Stopped MySQL database.

Dec 09 09:27:03 example.server.com systemd[1]: Stopping System Time Synchronized.

Dec 09 09:27:03 example.server.com systemd[1]: Stopped target System Time Synchronized.

Dec 09 09:27:03 example.server.com systemd[1]: Stopping Remote File Systems.

Dec 09 09:27:03 example.server.com systemd[1]: Stopped target Remote File Systems.

Dec 09 09:27:03 example.server.com systemd[1]: Unmounting /iscsi-disk...

Dec 09 09:27:03 example.server.com systemd[1]: Stopping Host and Network Name Lookups.

Dec 09 09:27:03 example.server.com systemd[1]: Stopped target Host and Network Name Lookups.

Dec 09 09:27:03 example.server.com kernel: Buffer I/O error on device sdb1, logical block 40960

Dec 09 09:27:03 example.server.com iscsid[853]: Connection1:0 to [target: example.target, portal: 192.168.1.30,3260] through [iface: default] 

Dec 09 09:27:03 example.server.com iscsid[853]: iscsid shutting down.

Dec 09 09:27:03 example.server.com mysqld_safe[1694]: 141209 09:27:03 mysqld_safe mysqld from pid file /backup/mysql-bacula/data/mysqld.pid ended

Dec 09 09:27:03 example.server.com kernel: EXT4-fs warning (device sdb1): ext4_end_bio:287: I/O error writing to inode 58989720 (offset 0 size 4096 starting block 40960)

Dec 09 09:27:03 example.server.com kernel: EXT4-fs error (device sdb1): ext4_find_entry:1309: inode #58982448: comm mysqld_safe: reading directory lblock 0

Dec 09 09:27:03 example.server.com kernel: Aborting journal on device sdb1-8.

Dec 09 09:27:03 example.server.com kernel: Buffer I/O error on device sdb1, logical block 133726208

Dec 09 09:27:03 example.server.com kernel: lost page write due to I/O error on sdb1

Dec 09 09:27:03 example.server.com kernel: JBD2: Error -5 detected when updating journal superblock for sdb1-8.

Dec 09 09:27:03 example.server.com kernel: EXT4-fs error (device sdb1): ext4_put_super:789: Couldn't clean up the journal

Dec 09 09:27:03 example.server.com kernel: EXT4-fs (sdb1): Remounting filesystem read-only

Dec 09 09:27:03 example.server.com systemd[1]: Unmounted /iscsi-disk.

Dec 09 09:27:03 example.server.com systemd[1]: Stopping Network is Online.

Dec 09 09:27:03 example.server.com systemd[1]: Stopped target Network is Online.

Dec 09 09:27:03 example.server.com systemd[1]: Stopping Network.

Dec 09 09:27:03 example.server.com systemd[1]: Stopped target Network.

Any help would be greatly appreciated.

[The Lee-Man]

I'm not familiar with CentOS. Does you iscsi-utils package have any systemd unit files, i.e. is the version of open-iscsi you are using integrated with systemd?

[awidde...@hotmail.com]

Yes, it does. These are the two main unit files provided for open-iscsi. The first is to log in/out of all "automatic" targets on startup/shutdown:

[Unit]

Description=Login and scanning of iSCSI devices

Documentation=man:iscsid(8) man:iscsiadm(8)

DefaultDependencies=no

Conflicts=shutdown.target

After=systemd-remount-fs.service network.target iscsid.service iscsiuio.service

Before=remote-fs-pre.target

ConditionDirectoryNotEmpty=|/var/lib/iscsi/nodes

ConditionDirectoryNotEmpty=|/sys/class/iscsi_session

[Service]

Type=oneshot

RemainAfterExit=true

ExecStart=/usr/libexec/iscsi-mark-root-nodes

SuccessExitStatus=21

ExecStart=/sbin/iscsiadm -m node --loginall=automatic

ExecStop=/sbin/iscsiadm -m node --logoutall=automatic

ExecReload=/sbin/iscsiadm -m node --loginall=automatic

[Install]

WantedBy=sysinit.target

The second is for iscsid:

[Unit]

Description=Open-iSCSI

Documentation=man:iscsid(8) man:iscsiadm(8)

DefaultDependencies=no

Conflicts=shutdown.target

After=network.target iscsiuio.service

Before=remote-fs-pre.target

[Service]

Type=forking

PIDFile=/var/run/iscsid.pid

ExecStart=/usr/sbin/iscsid

ExecStop=/sbin/iscsiadm -k 0 2

[Install]

WantedBy=multi-user.target

Here is the systemd generated mount until file for the iscsi-disk found in "/run/systemd/generator/iscsi-disk.mount":

# Automatically generated by systemd-fstab-generator

[Unit]

SourcePath=/etc/fstab

Before=remote-fs.target

[Mount]

What=/dev/disk/by-label/\x2fiscsi-disk

Where=/iscsi-disk

Type=ext4

Options=_netdev

Here is the fstab entry:

LABEL=/iscsi-disk           /iscsi-disk                ext4    _netdev         0 0

[Andy Grover]

On 12/09/2014 07:28 AM, awidde...@hotmail.com wrote:
> Setup iSCSI on CentOS7. Mounted a iSCSI disk and am running a small
> MySQL instance on the disk. The iSCSI disk and MySQL instance all come
> online fine with booting but when shutting down things seem to get very
> upset and the drive does not get unmounted cleanly.
>
> Does not look like I'm the only one having the issue. Another report
> that is very similar was posted here:
>
> https://www.centos.org/forums/viewtopic.php?f=47&t=49337
>
> This might be a systemd issue but figured I'd post here first to see if
> anyone else has had this issue and has any suggestions.

Does adding the "_netdev" mount option to the fstab entry for the
filesystem on the iscsi device help?

Regards -- Andy

[awidde...@hotmail.com]

Sorry, I didn't post the fstab entry in my original post when I should have. The "_netdev" entry is being applied to the disk and it seems like systemd generator is seeing that option properly and creating the mount point correctly (or so I think). Here is the fstab entry:

LABEL=/iscsi-disk           /iscsi-disk                ext4    _netdev         0 0

Long story short, no it doesn't seem to be helping unfortunately.

[The Lee-Man]

It seems clear that the I/O error is because iscsi is stopped before the device is unmounted:

On Tuesday, December 9, 2014 7:28:52 AM UTC-8, awidde...@hotmail.com wrote:

Setup iSCSI on CentOS7. Mounted a iSCSI disk and am running a small MySQL instance on the disk. The iSCSI disk and MySQL instance all come online fine with booting but when shutting down things seem to get very upset and the drive does not get unmounted cleanly.

Does not look like I'm the only one having the issue. Another report that is very similar was posted here:

https://www.centos.org/forums/viewtopic.php?f=47&t=49337

This might be a systemd issue but figured I'd post here first to see if anyone else has had this issue and has any suggestions.

Relevant version info:

CentOS Linux release 7.0.1406 (Core)

systemd-208-11.el7_0.4.x86_64

iscsi-initiator-utils-6.2.0.873-21.el7.x86_64

Systemd unit file for MySQL server:

[Unit]

Description=MySQL Server

After=nss-lookup.target network.target remote-fs.target time-sync.target

Wants=nss-lookup.target network.target remote-fs.target time-sync.target

Logs from journalctl:

Dec 09 09:27:03 example.server.com kernel: EXT4-fs error (device sdb1): __ext4_get_inode_loc:4039: inode #58989720: block 235930089: comm mysqld: unable to read itable block

Dec 09 09:27:03 example.server.com kernel: EXT4-fs error (device sdb1) in ext4_reserve_inode_write:4962: IO failure

Dec 09 09:27:03 example.server.com iscsiadm[3906]: Logging out of session [sid: 1, target: example.target, portal: 192.168.1.30,3260]

Dec 09 09:27:03 example.server.com iscsiadm[3906]: Logout of [sid: 1, target: example.target, portal: 192.168.1.30,3260] successful.

Dec 09 09:27:03 example.server.com systemd[1]: Stopped Login and scanning of iSCSI devices.

This looks like a race condition between the SQL shutdown and stopping the "iscsi.service" iSCSI login service.

 

Dec 09 09:27:03 example.server.com systemd[1]: Stopping Open-iSCSI...

Dec 09 09:27:03 example.server.com systemd[1]: Stopped Open-iSCSI.

There should be no more iSCSI from here on, right? But it turns out that "stopping the iscsid service" doesn't actually wait for the service to stop before it continues on to the next systemd task. (See below)

 

Dec 09 09:27:03 example.server.com systemd[1]: Stopped MySQL database.

But isn't your MySQL database using an iSCSI target?

 

Dec 09 09:27:03 example.server.com systemd[1]: Stopping System Time Synchronized.

Dec 09 09:27:03 example.server.com systemd[1]: Stopped target System Time Synchronized.

Dec 09 09:27:03 example.server.com systemd[1]: Stopping Remote File Systems.

Dec 09 09:27:03 example.server.com systemd[1]: Stopped target Remote File Systems.

Dec 09 09:27:03 example.server.com systemd[1]: Unmounting /iscsi-disk...

And now systemd tried to unmount the iscsi-backed device, but iscsid is already trying to stop. That's not going to work.

 

Dec 09 09:27:03 example.server.com systemd[1]: Stopping Host and Network Name Lookups.

Dec 09 09:27:03 example.server.com systemd[1]: Stopped target Host and Network Name Lookups.

Dec 09 09:27:03 example.server.com kernel: Buffer I/O error on device sdb1, logical block 40960

And, sure enough, I/O to/from the iSCSI device is failing.

 

Dec 09 09:27:03 example.server.com iscsid[853]: Connection1:0 to [target: example.target, portal: 192.168.1.30,3260] through [iface: default] 

Dec 09 09:27:03 example.server.com iscsid[853]: iscsid shutting down.

And iscsid finally finishes. It actually took less than a second ...

 

Dec 09 09:27:03 example.server.com mysqld_safe[1694]: 141209 09:27:03 mysqld_safe mysqld from pid file /backup/mysql-bacula/data/mysqld.pid ended

Dec 09 09:27:03 example.server.com kernel: EXT4-fs warning (device sdb1): ext4_end_bio:287: I/O error writing to inode 58989720 (offset 0 size 4096 starting block 40960)

Dec 09 09:27:03 example.server.com kernel: EXT4-fs error (device sdb1): ext4_find_entry:1309: inode #58982448: comm mysqld_safe: reading directory lblock 0

Dec 09 09:27:03 example.server.com kernel: Aborting journal on device sdb1-8.

Dec 09 09:27:03 example.server.com kernel: Buffer I/O error on device sdb1, logical block 133726208

Dec 09 09:27:03 example.server.com kernel: lost page write due to I/O error on sdb1

Dec 09 09:27:03 example.server.com kernel: JBD2: Error -5 detected when updating journal superblock for sdb1-8.

Dec 09 09:27:03 example.server.com kernel: EXT4-fs error (device sdb1): ext4_put_super:789: Couldn't clean up the journal

Dec 09 09:27:03 example.server.com kernel: EXT4-fs (sdb1): Remounting filesystem read-only

Dec 09 09:27:03 example.server.com systemd[1]: Unmounted /iscsi-disk.

This is all the filesystem trying and retrying to finish it's I/O before it gives up and lets you unmount the now-unreachable device.

 

Dec 09 09:27:03 example.server.com systemd[1]: Stopping Network is Online.

Dec 09 09:27:03 example.server.com systemd[1]: Stopped target Network is Online.

Dec 09 09:27:03 example.server.com systemd[1]: Stopping Network.

Dec 09 09:27:03 example.server.com systemd[1]: Stopped target Network.

Any help would be greatly appreciated.

One more question: does all of this just work if you do not use MySQL? 

[awidde...@hotmail.com]

I have tried this same process without any services running on the iSCSI drive and it seems to unmount without issue. Probably because nothing is trying to write to it so no I/O issues would arise. 

I agree with your log analysis that it doesn't seem like the order requested by the iscsi, iscsid, MySQL and iscsi-disk.mount files is being enforced or things are not stopping when they say they are. The ordering requested in those unit files seems sane to me though.

The order should be as follows:

Stop MySQL

Unmount remote-fs (/iscsi-disk)

Stop iscsi (logout of targets)

Stop iscsid

Again, I'm not sure why it seems like iscsi, iscsid services are stopping before MySQL is stopped or more importantly, before /iscsi-disk is unmounted.

I find debugging systemd and ordering particularly difficult because it starts/stops multiple things at once so when looking at the log files I'm not sure if the order in which things are logged is the order in which things actually happened.

[The Lee-Man]

Okay, I'm no systemd expert, though I play one at work. :)

You might try adding this to your mysql unit file:

 [Unit]

 Description=MySQL Server

-After=nss-lookup.target network.target remote-fs.target time-sync.target

-Wants=nss-lookup.target network.target remote-fs.target time-sync.target

+After=nss-lookup.target network.target remote-fs.target time-sync.target iscsi-disk.mount

+Wants=nss-lookup.target network.target remote-fs.target time-sync.target iscsi-disk.mount

And, of course, run "systemctl daemon-reload" after the change.

This is probably an issue for the mysql package maintainers, but this might get you working.

[awidde...@hotmail.com]

Thanks for the feedback and suggestion. I'm fairly certain (haven't actually tried though) that adding iscsi-disk.mount or even iscsi.target to the "After=" of the MySQL service would probably solve this problem I don't think it's a good solution.

Just to start, I don't think the MySQL package includes anything about the remote-fs.target at all. This particular MySQL instance/unit file was home grown to include that since it's known it will be running on the iSCSI disk.

The reason I don't feel like adding iscsi-disk.mount is a good solution is because that is a dynamically generated file made by systemd when it looks at fstab. That means if I choose to change my mount point from /iscsi-disk to /foo-bar for example having iscsi.mount as an "After=" will no longer work.

As for adding iscsi.service as an "After=", I also feel like that is inadequate. What if the underlying storage becomes NFS instead of iSCSI? The whole point of the special systemd remote-fs.target is to handle this and ordering properly yet clearly something isn't right here.

While adding these new orderings to the MySQL unit file will help make MySQL less upset it doesn't change the fact that it appears that /iscsi-disk is being unmounted after iscsi.service is stopped which can cause all sorts of file system problems.

Again, I appreciate the attempt at a solution but there is something else wrong here that needs a more concrete fix.

[Lee Duncan]

If you find that you think something needs to change in the open-iscsi unit files, please post it here.

I think it would be great if the open-iscsi tree contained a set of systemd unit files that worked for all distributions. Right now I think each distro has their own slightly-modified flavor.

-- 

The Lee-Man

Prepare to be unprepared. -- Me

[awidde...@hotmail.com]

I'm not sure if it is the unit files, iscsi or even systemd itself that is the problem. It all seems very strange to me right now. I certainly will post back if it is an open-iscsi issue. As I said, doesn't really seem like the unit files and their requirements are misconfigured. My hunch is something else is in play here, I am just not sure what that something else is.

[Chris Leech]

That was going to be my first guess as well, as I've been working with
the systemd developers to try and sort out the remote-fs/_netdev
behavior - it currently only works if there's an entry in fstab with the
_netdev option at the time systemd-fstab-generator is run.
It sounds like you've got that though.

My next guess is that the remote-fs-pre.target is not actually active,
so ordering on shutdown isn't being enforced properly.

What does 'systemctl status remote-fs.target remote-fs-pre.target' show?

Could you try adding a Wants=remote-fs-pre.target to the [Unit] section
of iscsi.service ?

Note that the special remote-fs-pre.target block manual activation, so
after editing iscsi.service and doing a daemon-reload you'll probably
need to fully stop and start the iscsi.service to get it to pull in
remote-fs-pre.target (or reboot, or some ugly hack to override
remote-fs-pre.target and remove the RefuseManualStart setting).

- Chris

[awidde...@hotmail.com]

Yes, I've found your bug reports and thread on the mailing lists about the issue you have been having. Very similar but unfortunately, different as you've noticed. Here is the output of the relevant services:

[root@some-server ~]# systemctl status iscsi iscsid remote-fs.target remote-fs-pre.target 

iscsi.service - Login and scanning of iSCSI devices

   Loaded: loaded (/usr/lib/systemd/system/iscsi.service; enabled)

   Active: active (exited) since Tue 2014-12-09 09:37:01 EST; 1 weeks 2 days ago

     Docs: man:iscsid(8)

           man:iscsiadm(8)

 Main PID: 881 (code=exited, status=0/SUCCESS)

Dec 09 09:37:00 some-server systemd[1]: Starting Login and scanning of iSCSI devices...

Dec 09 09:37:00 some-server iscsi-mark-root-nodes[869]: iscsiadm: No active sessions.

Dec 09 09:37:01 some-server iscsiadm[881]: Logging in to [iface: default, target: some-target, portal: 10.174.1.38,3260] (multiple)

Dec 09 09:37:01 some-server iscsiadm[881]: Login to [iface: default, target: some-target, portal: 10.174.1.38,3260] successful.

Dec 09 09:37:01 some-server systemd[1]: Started Login and scanning of iSCSI devices.

iscsid.service - Open-iSCSI

   Loaded: loaded (/usr/lib/systemd/system/iscsid.service; enabled)

   Active: active (running) since Tue 2014-12-09 09:37:00 EST; 1 weeks 2 days ago

     Docs: man:iscsid(8)

           man:iscsiadm(8)

 Main PID: 857 (iscsid)

   CGroup: /system.slice/iscsid.service

           +-855 /usr/sbin/iscsid

           +-857 /usr/sbin/iscsid

Dec 09 09:37:00 some-server systemd[1]: Starting Open-iSCSI...

Dec 09 09:37:00 some-server iscsid[850]: iSCSI logger with pid=855 started!

Dec 09 09:37:00 some-server systemd[1]: Failed to read PID from file /var/run/iscsid.pid: Invalid argument

Dec 09 09:37:00 some-server systemd[1]: Started Open-iSCSI.

Dec 09 09:37:01 some-server iscsid[855]: iSCSI daemon with pid=857 started!

Dec 09 09:37:01 some-server iscsid[855]: Connection1:0 to [target: some-target, portal: 10.174.1.38,3260] through [ifac...ational now

remote-fs.target - Remote File Systems

   Loaded: loaded (/usr/lib/systemd/system/remote-fs.target; enabled)

   Active: active since Tue 2014-12-09 09:37:03 EST; 1 weeks 2 days ago

     Docs: man:systemd.special(7)

Dec 09 09:37:03 some-server systemd[1]: Starting Remote File Systems.

Dec 09 09:37:03 some-server systemd[1]: Reached target Remote File Systems.

remote-fs-pre.target - Remote File Systems (Pre)

   Loaded: loaded (/usr/lib/systemd/system/remote-fs-pre.target; static)

   Active: inactive (dead)

     Docs: man:systemd.special(7)

I also noticed "remote-fs-pre.target" is not active and guessed that it might be an issue but haven't been able to figure out what needs to be changed to correct it from not being active. local-fs-pre.target is active and I would think these two would be have the same way for the most part.

I just updated the iscsi.service unit file to include "Wants=remote-fs-pre.target" and it seems to have solved the issue! I'm not sure how I didn't find this myself. I thought I tried just about everything. Also, tried this same thing on RHEL7.1 beta and it seems to be fixed there out of the box (no changes necessary). Just looked at the iscsi.service file on RHEL7.1 and it has "Wants=remote-fs-pre.target".

I guess, according to the changelog, https://bugzilla.redhat.com/show_bug.cgi?id=1161417 may have fixed this issue in 6.2.0.873-22? I looked through the patch you uploaded and didn't see this particular change and was for the fstab issue, which again seems pretty different than this. Nothing else in the changelog for iscsi-initiator-utils on RHEL7.1 seems to indicate where this fix might have come from.

Thanks for your help. Will this fix get applied to RHEL7.0 eventually? Will that happen after 7.1?

[br...@brentrjones.com]

Posting here since I am running into the same exact issue, however I am running the MariaDB included with RHEL 7.

We have a slightly different configuration, we use a bonded interface from a pair of 10Gb NICs, using LACP.
On that bonded interface, have three sub interfaces:

bond0 - parent, max MTU 9000, has these children:

  bond0.50 ("public", MTU 1500)

  bond0.100 (iSCSI VLAN 100, 10.1.1.10/32, MTU 9000)

  bond0.101 (iSCSI VLAN 101, 10.1.2.10/32, MTU 9000)

Firstly, NetworkManager simply wouldn't bring up those bonded sub-interfaces at all. Redhat support eventually said to give up on NetworkManager (good riddance... RH needs to dump it, it still isn't production quality).
Switching to just the network service brings up all the bonded interfaces just fine. However, on to the next problem.

Second problem is the same as stated above, at shutdown, system would tear down network interfaces and try to stop iSCSI before unmounting the iSCSI volumes - this lead to data loss and fun times with MariaDB.

Third problem is startup, RHEL7 MariaDB does not wait for remote volumes to mount before trying to start:

/etc/systemd/system/multi-user.target.wants/mariadb.service:

[Unit]

Description=MariaDB database server

After=syslog.target

After=network.target

[Service]

Type=simple

User=mysql

Group=mysql

$ systemctl status mariadb

mariadb.service - MariaDB database server

   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled)

   Active: failed (Result: exit-code) since Fri 2015-01-16 17:26:08 EST; 18min ago

  Process: 6132 ExecStartPre=/usr/libexec/mariadb-prepare-db-dir %n (code=exited, status=1/FAILURE)

Jan 16 17:26:08 mysqldb-01 mariadb-prepare-db-dir[6132]: mkdir: cannot create directory ‘/mnt/db_data_01/mysql/’: Permission Denies

Jan 16 17:26:08 mysqldb-01 systemd[1]: mariadb.service: control process exited, code=exited status=1

Jan 16 17:26:08 mysqldb-01 systemd[1]: Failed to start MariaDB database server.

Jan 16 17:26:08 mysqldb-01 systemd[1]: Unit mariadb.service entered failed state.

I've added:

"After=remote-fs.target" to mariadb.service as a workaround, but yah..

I don't mind systemd, and in fact I like the structure of the unit files better than init for sure... but this stuff should've been caught pretty early on in QA!

Hope this helps other people if they encounter it, and hope some additional data and use case can help Redhat in the future.

[Chris Leech]

On Fri, Jan 16, 2015 at 04:10:19PM -0800, br...@brentrjones.com wrote:
> Posting here since I am running into the same exact issue, however I am
> running the MariaDB included with RHEL 7.

Thanks for sharing this. I'll respond briefly, as much of what you
bring up isn't really on topic for Open-iSCSI. But if you want to copy
me on any other discussions, point me at Red Hat bugzillas, etc. please
go ahead.

<skipping network manager issue>

> Second problem is the same as stated above, at shutdown, system would tear
> down network interfaces and try to stop iSCSI before unmounting the iSCSI
> volumes - this lead to data loss and fun times with MariaDB.

I'm sorry you had to deal with that, it's been a difficult issue to get
right.

> Third problem is startup, RHEL7 MariaDB does not wait for remote volumes to
> mount before trying to start:

...

> Jan 16 17:26:08 mysqldb-01 mariadb-prepare-db-dir[6132]: mkdir: cannot
> create directory ‘/mnt/db_data_01/mysql/’: Permission Denies

..

> I've added:
> "After=remote-fs.target" to mariadb.service as a workaround, but yah..

From what I see the issue here is that you've configured mariadb to use
a not-default datadir on an iSCSI disk, but systemd has no idea that the
path /mnt/db_data_01/mysql is needed before starting the database
server. Rather than add a general remote-fs ordering, it's probably
better to have a Requires/After on the mount point (or create a path
unit, but I'm not fully up on their use).

Should changing a service configuration also require making
modifications to the init configuration? Ideally not, but as this case
shows it can.

- Chris

[garp...@gmail.com]

> Second problem is the same as stated above, at shutdown, system would tear
> down network interfaces and try to stop iSCSI before unmounting the iSCSI

> volumes - this lead to data loss and (not) fun times

Hi All

I've just lost 1,5 TB (thousansd of very important files) of data due the same raason: closing iscsi before unmountig volumes
file system totally corrupted

has anyone knows in witch version of Centos7/Open-Iscsi this dangeours bug was fixed ?
 

[garp...@gmail.com]

> Second problem is the same as stated above, at shutdown, system would tear
> down network interfaces and try to stop iSCSI before unmounting the iSCSI
> volumes - this lead to data loss

Hi All,

I've just  lost 1,5 TB (many thousand if very important files) due the same reason
shutdown iscsi before unmounting volume
file system totally corrupted, nothing  can help

has anyone know in witch version of Centos7/Open-iscsi this very dangerous bug was fixed ?
 

[Mike Christie]

Chris, it looks like you fixed this in systemd-208-20.el7 and that is
RHEL 7.2, right?

[Chris Leech]

That's the right systemd package, but I think it was released with RHEL
7.1 along with iscsi-initiator-utils-6.2.0.873-29.

- Chris

[David]

Hi Guys,

Chris Leech <cleech@...> writes:

>
> On Tue, Jan 19, 2016 at 02:12:39PM -0600, Mike Christie wrote:

> > On 01/11/2016 08:52 AM, garpen01@... wrote:
> > >
> > > > Second problem is the same as stated above, at shutdown, system
> > > would tear
> > > > down network interfaces and try to stop iSCSI before unmounting
> > > the iSCSI
> > > > volumes - this lead to data loss and (not) fun times
> > >
> > >
> > > Hi All
> > >
> > > I've just lost 1,5 TB (thousansd of very important files) of data due
> > > the same raason: closing iscsi before unmountig volumes
> > > file system totally corrupted
> > >
> > > has anyone knows in witch version of Centos7/Open-Iscsi this dangeours
> > > bug was fixed ?
> > >
> >
> > Chris, it looks like you fixed this in systemd-208-20.el7 and that is
> > RHEL 7.2, right?
>
> That's the right systemd package, but I think it was released with RHEL
> 7.1 along with iscsi-initiator-utils-6.2.0.873-29.
>
> - Chris
>

I've just installed a Centos7 box, fully updated to:

CentOS Linux release 7.2.1511 (Core)

I have installed and set up an iSCSI volume. Done a mkfs on it. Mounted
it. No problem.

I am having the same problem when shutting down. The iscsi stuff seems to
stop before the volume is un-mounted. I've looked at the systemd files in
the rest of these posts, and the "fixes" are in place out of the box.

I have one other, possibly related, issue upon start up. The iscsi
connection is not automatically "logged" into. I have to do this manually.
Not sure why, as the iscsi systemd files have lines in them saying to auto
login.

[Chris Leech]

On Tue, Jun 28, 2016 at 04:01:37AM +0000, David wrote:
> I've just installed a Centos7 box, fully updated to:
>
> CentOS Linux release 7.2.1511 (Core)
>
> I have installed and set up an iSCSI volume. Done a mkfs on it. Mounted
> it. No problem.
>
> I am having the same problem when shutting down. The iscsi stuff seems to
> stop before the volume is un-mounted. I've looked at the systemd files in
> the rest of these posts, and the "fixes" are in place out of the box.
>
> I have one other, possibly related, issue upon start up. The iscsi
> connection is not automatically "logged" into. I have to do this manually.
> Not sure why, as the iscsi systemd files have lines in them saying to auto
> login.

Hello David,

Could you share the output of 'iscsiadm -m node -o show' with me? That
will dump the full details of the node record. In particular I'd like
to check the setting of node.startup.

If node.startup is set to manual, then the systemd units for
managing automatic session startup and shutdown will ignore this target.

The default node.startup setting that will be applied to newly added
targets can be found in /etc/iscsi/iscsid.conf

Chris Leech

[Chris Leech]

On Tue, Jun 28, 2016 at 04:01:37AM +0000, David wrote:
>

> I've just installed a Centos7 box, fully updated to:
>
> CentOS Linux release 7.2.1511 (Core)
>
> I have installed and set up an iSCSI volume. Done a mkfs on it. Mounted
> it. No problem.
>
> I am having the same problem when shutting down. The iscsi stuff seems to
> stop before the volume is un-mounted. I've looked at the systemd files in
> the rest of these posts, and the "fixes" are in place out of the box.
>
> I have one other, possibly related, issue upon start up. The iscsi
> connection is not automatically "logged" into. I have to do this manually.
> Not sure why, as the iscsi systemd files have lines in them saying to auto
> login.

David and I sorted his issue out, the target node record had
node.startup set to "onboot" so it was being treated as an initramfs
managed session needed for the root filesystem. Resetting it to
"automatic" got things working as expected.

I'm guessing it's from installing the iscsi-initiator-utils package,
creating and bringing up the session manually, and then starting the
iscsi.service unit manually as well. It's supposed to handle session
startup at boot, so any sessions active when it's started are assumed to
be from the initramfs and switched to "onboot."

Chris

[bamblew...@gmail.com]

I just want to thank David and Chris for sorting this out. I had exactly the same problem: CentOS 7 system "hanging" on shutdown when iSCSI volume is mounted, but not when unmounted and logged out of iscsi session first; iSCSI session not logging in automatically on startup.

As Chris described, I located the iSCSI node definition files at /var/lib/iscsi/nodes/.../default and changed "node.startup = onboot" to "node.startup = automatic". Initially, I had created the node definitions by running "iscsiadm -m discovery -t st -p <target_ip_address>" manually, then started the iscsid and iscsi services manually as well. Two node files are present, as I the target is reachable via multipath.

Additionally, I have a service (Code42 CrashPlanPROe) that relies on the iSCSI volume for it's archive storage. I was able to smooth system startup and shutdown by adding the following lines to the Systemd unit file for this service:

After=network.target iscsi.service iscsid.service

Wants=network.target iscsi.service iscsid.service

These lines ensure that the proserver service starts after iscsi and iscsid on system startup, and stops before them on system shutdown.

Here are some details on the system, FYI:

OS: CentOS 7.2.1511

Linux kernel: 3.10.0-327.22.2.el7.x86_64

iscsi-initiator-utils.x86_64: 6.2.0.873-33.el7_2.1

device-mapper-multipath.x86_64: 0.4.9-85.el7_2.5

Thanks for the solution!

-jm

[cin.s...@gmail.com]

I know this is the old thread,but anyway this problem still persists in many distros. I've found sane workaround to it. Using x-systemd.after=iscsi.service mount option adds necessary dependency to mount and prevents iscsi to log out before the file system is clearly unmounted.

here is the example from my fstab

/dev/mapper/mpathm         /storage          ext4      acl,user_xattr,_netdev,x-systemd.after=iscsi.service          0          0

вторник, 9 декабря 2014 г., 18:28:52 UTC+3 пользователь awidde...@hotmail.com написал: