Tutorial: Force ESXi to Rescan iSCSI HBA upon QuadStor Boot

[Paul Reid]

Hi, everyone,

During my adventures with hyper-converged VMware vSphere setups with QuadStor (ie: storage provided to the host by a VM on the host), I ran into an issue with the ESXi hosts not seeing the QuadStor iSCSI targets after it had fully started up, leaving all my VMs in an "unknown" state, which of course won't auto-start. I always had to connect to the host and manually rescan the HBAs to get the disks to come online, which is a problem if the host restarts for some reason and I don't know about it until someone complains that VMs are down.

So, after some fiddling around, this is what I came up with, which is a variation of what I'd done on other hosts that were using Windows Storage Spaces to present iSCSI targets (though I used VMware PowerCLI for this on Windows):

Note: My OS of choice for QuadStor is CentOS 7, so the below instructions are intended for that distribution. It should work on other distributions in a similar manner, with some modifications, I'd expect.

1. Install the EPEL mirror by typing:
   yum install epel-release
2. Install SSHPass by typing:
   yum install sshpass
3. Create the HBA rescan script by typing:
   nano /root/esxihbarescan.script
4. Add the following into the esxihbarescan.script file (note the \ to escape the $'s in the password):
   #!/bin/bash
   sshpass -p Password11\$ ssh -oStrictHostKeyChecking=no ro...@192.168.0.10 esxcli storage core adapter rescan --all
   exit
5. Close nano with CTRL+X and save the file
6. Make the script executable by typing:
   chmod 777 /root/esxihbarescan.script
7. Create a new service by typing:
   nano /usr/lib/systemd/system/esxihbarescan.service
8. Add the following into the esxihbarescan.service file:
   [Unit]
   Description=ESXi HBA Rescan
   After=quadstor.service
   
   [Service]
   ExecStart=/root/esxihbarescan.script
   
   [Install]
   WantedBy=muliti-user.target
9. Close nano with CTRL+X and save the file
10. Enable the service by typing:
    systemctl enable esxihbarescan

The long command we put into the esxihbarescan.script tells the system to SSH into the host at 192.168.0.10 as root, using the password Password11$, then execute the command esxcli storage core adapter rescan --all, which makes ESXi rescan all of it's HBAs.

The esxihbarescan.service file tells the system to wait for the QuadStor service to be alive, then start the script. If we don't do this, we'll end up telling ESXi to rescan before QuadStor has the iSCSI targets online.

Now, with the vSphere C# Client or the vSphere Web Interface, tell the host to automatically start your QuadStor VM on boot, and everything will come online shortly after the host finishes it's boot process, QuadStor is booted, and everything settles down. You can even configure the ESXi host to auto-start your other VMs that live on the QuadStor presented storage, and they will probably start for you - but you'll have to fiddle with the delay the host waits before continuing to start VMs after QuadStor is up (you can configure that in the ESXi auto start stuff). I set mine to wait a couple of minutes to let QuadStor get started and the host to rescan the HBA and bring the targets online before the rest of the VMs attempt to auto-start.

I imagine some of the above can be adjusted a little, just as the chmod 777 I used . I'm not too concerned about exact permissions on the script - I just wanted it to be executable by anyone, so I gave it all permissions.

Any of you that have set up hyper-converged configurations with iSCSI will already know that ESXi boots slowly, due to the iSCSI driver loading and then waiting for a timeout as it searches for the iSCSI targets you had configured that are (of course) not yet online during boot, since the VM with the iSCSI targets isn't alive yet. I've yet to find a solution to this, but it's a minor issue. If you build a cluster and multipath the iSCSI initiator within ESXi, then your first host will have this delay but the second host should find the first host's iSCSI targets and boot faster.

If you run into issues, let me know - but it works fine for me on my dual host cluster.