Failed to kick off vmware-iso builds

[Dan Linder]

I'm using Packer with VMWare Workstation 15.5 to build VM templates for our VMware environment.

Problem: The same process that works to build on a Windows 10 system fails with `VMware error: Error: The operation was cnceled` when performed on a Red Hat Linux 7 system.

Working environment:

• Windows 10
• Packer 1.4.4
• VMWare Workstation 15.5.0 build-14665864

Failing environment:

• Red Hat Enterprise Linux 7
• Packer 1.4.4
• VMWare Workstation 15.5.0 build-14665864

The only difference is the OS build environment - packer and VMworkstation are the same release versions.

When I execute it from the Windows environment, the execution starts normally:

==> vmware-iso: Building and writing VMX file
==> vmware-iso: Starting HTTP server on port 8802
==> vmware-iso: Starting virtual machine...
    vmware-iso: The VM will be run headless, without a GUI. If you want to
    vmware-iso: view the screen of the VM, connect via VNC with the password "hUOf31vh" to
    vmware-iso: vnc://127.0.0.1:5964
==> vmware-iso: Waiting 8s for boot...
==> vmware-iso: Connecting to VM via VNC (127.0.0.1:5964)
==> vmware-iso: Typing the boot command over VNC...
==> vmware-iso: Using ssh communicator to connect: 192.168.141.134
==> vmware-iso: Waiting for SSH to become available...

[...the build proceeds normally...]

When I execute it from the Red Hat environment it fails when it executes the VMware `/usr/bin/vmrun` command:

(with `PACKER_LOG=1` set)

==> vmware-iso: Creating required virtual machine disks
2019/10/21 14:34:14 packer: 2019/10/21 14:34:14 [INFO] Creating disk with Path: RHEL-8-novra-20191021_1433/disk.vmdk and Size: 61440M
2019/10/21 14:34:14 packer: 2019/10/21 14:34:14 Executing: /usr/bin/vmware-vdiskmanager -c -s 61440M -a lsilogic -t 0 RHEL-8-novra-20191021_1433/disk.vmdk
2019/10/21 14:34:14 packer: 2019/10/21 14:34:14 stdout: Creating disk 'RHEL-8-novra-20191021_1433/disk.vmdk'
2019/10/21 14:34:14 packer: Virtual disk creation successful.
==> vmware-iso: Building and writing VMX file
2019/10/21 14:34:14 packer: 2019/10/21 14:34:14 stderr:
2019/10/21 14:34:14 packer: 2019/10/21 14:34:14 Located networkmapper configuration file using Player: /etc/vmware/netmap.conf
2019/10/21 14:34:14 packer: 2019/10/21 14:34:14 Writing VMX to: RHEL-8-novra-20191021_1433/RHEL-8-novra-20191021_1433.vmx
2019/10/21 14:34:14 packer: 2019/10/21 14:34:14 Configuring VMX...
2019/10/21 14:34:14 packer: 2019/10/21 14:34:14 Setting VMX: 'cpuid.coresPerSocket' = '1'
2019/10/21 14:34:14 packer: 2019/10/21 14:34:14 Setting VMX: 'memsize' = '8192'
2019/10/21 14:34:14 packer: 2019/10/21 14:34:14 Setting VMX: 'numvcpus' = '2'
2019/10/21 14:34:14 packer: 2019/10/21 14:34:14 Setting VMX: 'ethernet0.virtualDev' = 'vmxnet3'
2019/10/21 14:34:14 packer: 2019/10/21 14:34:14 Setting VMX: 'virtualHW.version' = '8'
2019/10/21 14:34:14 packer: 2019/10/21 14:34:14 Setting VMX: 'ethernet0.connectionType' = 'nat'
2019/10/21 14:34:14 packer: 2019/10/21 14:34:14 Setting VMX: 'applianceView.coverPage.author' = 'GTS-Converged...@west.com'
2019/10/21 14:34:14 packer: 2019/10/21 14:34:14 Setting VMX: 'applianceView.coverPage.version' = '20191021_1433'
2019/10/21 14:34:14 packer: 2019/10/21 14:34:14 Floppy path present, setting in VMX
2019/10/21 14:34:14 packer: 2019/10/21 14:34:14 Writing VMX to: RHEL-8-novra-20191021_1433/RHEL-8-novra-20191021_1433.vmx
2019/10/21 14:34:14 packer: 2019/10/21 14:34:14 Suppressing messages in VMX
2019/10/21 14:34:14 packer: 2019/10/21 14:34:14 Found available port: 8214 on IP: 0.0.0.0
==> vmware-iso: Starting HTTP server on port 8214
2019/10/21 14:34:14 packer: 2019/10/21 14:34:14 Looking for available port between 5900 and 6000
2019/10/21 14:34:14 packer: 2019/10/21 14:34:14 Found available port: 5980 on IP: 127.0.0.1
2019/10/21 14:34:14 packer: 2019/10/21 14:34:14 Found available VNC port: 127.0.0.1:5980
2019/10/21 14:34:14 packer: 2019/10/21 14:34:14 Writing VMX to: RHEL-8-novra-20191021_1433/RHEL-8-novra-20191021_1433.vmx
==> vmware-iso: Starting virtual machine...
    vmware-iso: view the screen of the VM, connect via VNC with the password "qlKPz2Bm" to
    vmware-iso: vnc://127.0.0.1:5980
2019/10/21 14:34:14 packer: 2019/10/21 14:34:14 Executing: /usr/bin/vmrun -T player start RHEL-8-novra-20191021_1433/RHEL-8-novra-20191021_1433.vmx nogui
    vmware-iso: The VM will be run headless, without a GUI. If you want to
    vmware-iso: view the screen of the VM, connect via VNC with the password "qlKPz2Bm" to
    vmware-iso: vnc://127.0.0.1:5980
2019/10/21 14:34:14 packer: 2019/10/21 14:34:14 stdout: Error: The operation was canceled
2019/10/21 14:34:14 packer: 2019/10/21 14:34:14 stderr:
==> vmware-iso: Error starting VM: VMware error: Error: The operation was canceled
==> vmware-iso: Error starting VM: VMware error: Error: The operation was canceled
==> vmware-iso: Step "StepRun" failed

==> vmware-iso: [c] Clean up and exit, [a] abort without cleanup, or [r] retry step (build may fail even if retry succeeds)?

When I try to manually run the "/usr/bin/vmrun -T player start RHEL-8-novra-20191021_1433/RHEL-8-novra-20191021_1433.vmx nogui" it exits with the same "VMware error: Error: The operation was canceled", nothing more:

[u@h Oct-21 14:35:53 W ]$ /usr/bin/vmrun -T player start RHEL-8-novra-20191021_1433/RHEL-8-novra-20191021_1433.vmx nogui ; echo Exit code: $?
Error: The operation was canceled
Exit code: 255
[u@h Oct-21 14:35:59 W ]$

[Dan Linder]

No-one has any ideas?  Where should I look for additional VMware Workstation debug logs/messages?

I've tried turning on the firewall, setting SELinux to Permissive, and validating that the CPU supports virtualization.

My next step is to see if I can confirm VMWare workstation will run under X11/NoMachine to see if it is an issue with headless still expecting some X11 stuff.

Dan

2019/10/21 14:34:14 packer: 2019/10/21 14:34:14 Setting VMX: 'applianceView.coverPage.author' = 'vmt...@company.com'

[Dan Linder]

So I've made a bit more progress today.

My build server (a physical system with 64GB RAM, 8 cores) is running Red Hat 7, and it is only setup for SSH, not X11 (which was the root of part of my problem).

Here's what I've done so far:

1. Install some X11 and Kernel packages VMware Workstation needs:

   * sudo yum install xorg-x11-xauth libXtst xorg-x11-fonts-* xorg-x11-font-utils xorg-x11-fonts-Type1 kernel-devel kernel-headers

2. Download and install VMware Workstation ".bundle":

   * sudo ./VMware-Workstation-Full-15.5.0-14665864.x86_64.bundle --console --set-setting vmware-workstation serialNumber MY-SER-IAL-NUM-RED-ACT-ED --eulas-agreed

   * Accept the default configuration options it provides

3. Ensure the physical server has 4GiB swap (mine had 2GiB and VMware complained).

4. For good measure I ran "vmware" and confirmed that I could build a simple VM manually

   * Since the VMWare Workstation uses X11 and I don't use that much, I installed the free "MobaXterm" application on my Windows workstation so the VMware Workstation GUI would work.

If I run it with in headless mode I get an "Unexpected signal: 7" error:

==> vmware-iso: Building and writing VMX file

2019/10/28 16:20:13 packer: 2019/10/28 16:20:13 stderr:
2019/10/28 16:20:13 packer: 2019/10/28 16:20:13 Located networkmapper configuration file using Workstation: /etc/vmware/netmap.conf
2019/10/28 16:20:13 packer: 2019/10/28 16:20:13 Writing VMX to: RHEL-7-novra-20191028_1619/RHEL-7-novra-20191028_1619.vmx
2019/10/28 16:20:13 packer: 2019/10/28 16:20:13 Configuring VMX...
2019/10/28 16:20:13 packer: 2019/10/28 16:20:13 Setting VMX: 'cpuid.coresPerSocket' = '1'
2019/10/28 16:20:13 packer: 2019/10/28 16:20:13 Setting VMX: 'memsize' = '8192'
2019/10/28 16:20:13 packer: 2019/10/28 16:20:13 Setting VMX: 'numvcpus' = '2'
2019/10/28 16:20:13 packer: 2019/10/28 16:20:13 Setting VMX: 'ethernet0.virtualDev' = 'vmxnet3'
2019/10/28 16:20:13 packer: 2019/10/28 16:20:13 Setting VMX: 'virtualHW.version' = '8'
2019/10/28 16:20:13 packer: 2019/10/28 16:20:13 Setting VMX: 'ethernet0.connectionType' = 'nat'
2019/10/28 16:20:13 packer: 2019/10/28 16:20:13 Setting VMX: 'applianceView.coverPage.author' = 'GTS-Converged...@west.com'
2019/10/28 16:20:13 packer: 2019/10/28 16:20:13 Setting VMX: 'applianceView.coverPage.version' = '20191028_1619'
2019/10/28 16:20:13 packer: 2019/10/28 16:20:13 Floppy path present, setting in VMX
2019/10/28 16:20:13 packer: 2019/10/28 16:20:13 Writing VMX to: RHEL-7-novra-20191028_1619/RHEL-7-novra-20191028_1619.vmx
2019/10/28 16:20:13 packer: 2019/10/28 16:20:13 Suppressing messages in VMX
2019/10/28 16:20:13 packer: 2019/10/28 16:20:13 Found available port: 8015 on IP: 0.0.0.0
==> vmware-iso: Starting HTTP server on port 8015
2019/10/28 16:20:13 packer: 2019/10/28 16:20:13 Looking for available port between 5900 and 6000
2019/10/28 16:20:13 packer: 2019/10/28 16:20:13 Found available port: 5990 on IP: 127.0.0.1
2019/10/28 16:20:13 packer: 2019/10/28 16:20:13 Found available VNC port: 127.0.0.1:5990
2019/10/28 16:20:13 packer: 2019/10/28 16:20:13 Writing VMX to: RHEL-7-novra-20191028_1619/RHEL-7-novra-20191028_1619.vmx

==> vmware-iso: Starting virtual machine...

    vmware-iso: view the screen of the VM, connect via VNC with the password "DobAhVuR" to
    vmware-iso: vnc://127.0.0.1:5990
2019/10/28 16:20:13 packer: 2019/10/28 16:20:13 Executing: /usr/bin/vmrun -T ws start RHEL-7-novra-20191028_1619/RHEL-7-novra-20191028_1619.vmx nogui

    vmware-iso: The VM will be run headless, without a GUI. If you want to

    vmware-iso: view the screen of the VM, connect via VNC with the password "DobAhVuR" to
    vmware-iso: vnc://127.0.0.1:5990
2019/10/28 16:20:13 packer: 2019/10/28 16:20:13 stdout: Error: Unknown error
2019/10/28 16:20:13 packer: 2019/10/28 16:20:13 stderr: Unexpected signal: 7.
==> vmware-iso: Error starting VM: VMware error: Unexpected signal: 7.
==> vmware-iso: Error starting VM: VMware error: Unexpected signal: 7.

==> vmware-iso: Step "StepRun" failed

==> vmware-iso: [c] Clean up and exit, [a] abort without cleanup, or [r] retry step (build may fail even if retry succeeds)?

If I run in "headless: false" mode the VMware Workstation GUI pops up for a split second, then goes away and eventually the execution of "vmrun" times out.

2019/10/28 16:22:14 packer: Virtual disk creation successful.

==> vmware-iso: Building and writing VMX file

2019/10/28 16:22:14 packer: 2019/10/28 16:22:14 stderr:
2019/10/28 16:22:14 packer: 2019/10/28 16:22:14 Located networkmapper configuration file using Workstation: /etc/vmware/netmap.conf
2019/10/28 16:22:14 packer: 2019/10/28 16:22:14 Writing VMX to: RHEL-7-novra-20191028_1621/RHEL-7-novra-20191028_1621.vmx
2019/10/28 16:22:14 packer: 2019/10/28 16:22:14 Configuring VMX...
2019/10/28 16:22:14 packer: 2019/10/28 16:22:14 Setting VMX: 'ethernet0.connectionType' = 'nat'
2019/10/28 16:22:14 packer: 2019/10/28 16:22:14 Setting VMX: 'applianceView.coverPage.author' = 'GTS-Converged...@west.com'
2019/10/28 16:22:14 packer: 2019/10/28 16:22:14 Setting VMX: 'applianceView.coverPage.version' = '20191028_1621'
2019/10/28 16:22:14 packer: 2019/10/28 16:22:14 Setting VMX: 'cpuid.coresPerSocket' = '1'
2019/10/28 16:22:14 packer: 2019/10/28 16:22:14 Setting VMX: 'memsize' = '8192'
2019/10/28 16:22:14 packer: 2019/10/28 16:22:14 Setting VMX: 'numvcpus' = '2'
2019/10/28 16:22:14 packer: 2019/10/28 16:22:14 Setting VMX: 'ethernet0.virtualDev' = 'vmxnet3'
2019/10/28 16:22:14 packer: 2019/10/28 16:22:14 Setting VMX: 'virtualHW.version' = '8'
2019/10/28 16:22:14 packer: 2019/10/28 16:22:14 Floppy path present, setting in VMX
2019/10/28 16:22:14 packer: 2019/10/28 16:22:14 Writing VMX to: RHEL-7-novra-20191028_1621/RHEL-7-novra-20191028_1621.vmx
2019/10/28 16:22:14 packer: 2019/10/28 16:22:14 Suppressing messages in VMX
2019/10/28 16:22:14 packer: 2019/10/28 16:22:14 Found available port: 8413 on IP: 0.0.0.0
==> vmware-iso: Starting HTTP server on port 8413
2019/10/28 16:22:14 packer: 2019/10/28 16:22:14 Looking for available port between 5900 and 6000
2019/10/28 16:22:14 packer: 2019/10/28 16:22:14 Found available port: 5983 on IP: 127.0.0.1
2019/10/28 16:22:14 packer: 2019/10/28 16:22:14 Found available VNC port: 127.0.0.1:5983
2019/10/28 16:22:14 packer: 2019/10/28 16:22:14 Writing VMX to: RHEL-7-novra-20191028_1621/RHEL-7-novra-20191028_1621.vmx

==> vmware-iso: Starting virtual machine...

2019/10/28 16:22:14 packer: 2019/10/28 16:22:14 Executing: /usr/bin/vmrun -T ws start RHEL-7-novra-20191028_1621/RHEL-7-novra-20191028_1621.vmx gui

2019/10/28 16:23:44 packer: 2019/10/28 16:23:44 stdout: Error: Cannot connect to the virtual machine
2019/10/28 16:23:44 packer: 2019/10/28 16:23:44 stderr:
==> vmware-iso: Error starting VM: VMware error: Error: Cannot connect to the virtual machine
==> vmware-iso: Error starting VM: VMware error: Error: Cannot connect to the virtual machine

==> vmware-iso: Step "StepRun" failed

[Dan Linder]

And a bit more progress - apparently the initial "4GiB" swap wasn't sufficient, so I had to bump it up to "8GiB" swap.

It's now starting the VM in GUI mode, so it's probably going to work now.

I'll update this thread with a better/concise list of changes done when I have them hammered out.

On Monday, October 28, 2019 at 3:24:52 PM UTC-5, Dan Linder wrote:

So I've made a bit more progress today.

My build server (a physical system with 64GB RAM, 8 cores) is running Red Hat 7, and it is only setup for SSH, not X11 (which was the root of part of my problem).

Here's what I've done so far:

1. Install some X11 and Kernel packages VMware Workstation needs:

   * sudo yum install xorg-x11-xauth libXtst xorg-x11-fonts-* xorg-x11-font-utils xorg-x11-fonts-Type1 kernel-devel kernel-headers

2. Download and install VMware Workstation ".bundle":

   * sudo ./VMware-Workstation-Full-15.5.0-14665864.x86_64.bundle --console --set-setting vmware-workstation serialNumber MY-SER-IAL-NUM-RED-ACT-ED --eulas-agreed

   * Accept the default configuration options it provides

3. Ensure the physical server has 4GiB swap (mine had 2GiB and VMware complained).

4. For good measure I ran "vmware" and confirmed that I could build a simple VM manually

   * Since the VMWare Workstation uses X11 and I don't use that much, I installed the free "MobaXterm" application on my Windows workstation so the VMware Workstation GUI would work.

If I run it with in headless mode I get an "Unexpected signal: 7" error:

==> vmware-iso: Building and writing VMX file
2019/10/28 16:20:13 packer: 2019/10/28 16:20:13 stderr:
2019/10/28 16:20:13 packer: 2019/10/28 16:20:13 Located networkmapper configuration file using Workstation: /etc/vmware/netmap.conf
2019/10/28 16:20:13 packer: 2019/10/28 16:20:13 Writing VMX to: RHEL-7-novra-20191028_1619/RHEL-7-novra-20191028_1619.vmx
2019/10/28 16:20:13 packer: 2019/10/28 16:20:13 Configuring VMX...
2019/10/28 16:20:13 packer: 2019/10/28 16:20:13 Setting VMX: 'cpuid.coresPerSocket' = '1'
2019/10/28 16:20:13 packer: 2019/10/28 16:20:13 Setting VMX: 'memsize' = '8192'
2019/10/28 16:20:13 packer: 2019/10/28 16:20:13 Setting VMX: 'numvcpus' = '2'
2019/10/28 16:20:13 packer: 2019/10/28 16:20:13 Setting VMX: 'ethernet0.virtualDev' = 'vmxnet3'
2019/10/28 16:20:13 packer: 2019/10/28 16:20:13 Setting VMX: 'virtualHW.version' = '8'
2019/10/28 16:20:13 packer: 2019/10/28 16:20:13 Setting VMX: 'ethernet0.connectionType' = 'nat'

2019/10/28 16:20:13 packer: 2019/10/28 16:20:13 Setting VMX: 'applianceView.coverPage.author' = 'vmt...@company.com'

2019/10/28 16:22:14 packer: 2019/10/28 16:22:14 Setting VMX: 'applianceView.coverPage.author' = 'vmt...@company.com'