Troubleshooting ovftool, unable to get specific errors

[jackson....@willowglensystems.com]

Running Packer version 1.3.1

I have a template that uses the vmware-iso builder to give me a VM built on a local copy of VMware Fusion. This works great.

I've tried to add in the vsphere post-processor but I'm running into an error at that step of the process. I wasn't getting any info from the normal output so I enabled PACKER_DEBUG=1. This also isn't giving me any output to determine the cause of the error (relevant portion included below):

2018/10/22 14:21:32 [INFO] (telemetry) Starting post-processor vsphere

2018/10/22 14:21:32 ui: ESC[0;32m    vmware-iso (vsphere): Uploading centos7-tmpl-33b99e6/centos7-tmpl-33b99e6.vmx to vSphereESC[0m

2018/10/22 14:21:32 packer: 2018/10/22 14:21:32 Starting ovftool with parameters: --acceptAllEulas --name=centos7-tmpl-33b99e6 --datastore=datastore --diskMode=thick --network="Network Name" centos7-tmpl-33b99e6/centos7-tmpl-33b99e6.vmx vi://<username>:<password>@vcenter-host/datacenter/host/cluster/?dns=esxi-host

2018/10/22 14:21:32 [INFO] (telemetry) ending vsphere

2018/10/22 14:21:32 Deleting original artifact for build 'vmware-iso'

2018/10/22 14:21:32 ui error: ESC[1;31mBuild 'vmware-iso' errored: 1 error(s) occurred:

* Post-processor failed: Error uploading virtual machine: exit status 1

ESC[0m

2018/10/22 14:21:32 Builds completed. Waiting on interrupt barrier...

2018/10/22 14:21:32 machine readable: error-count []string{"1"}

2018/10/22 14:21:32 ui error:

==> Some builds didn't complete successfully and had errors:

2018/10/22 14:21:32 machine readable: vmware-iso,error []string{"1 error(s) occurred:\n\n* Post-processor failed: Error uploading virtual machine: exit status 1\n\n"}

2018/10/22 14:21:32 ui error: --> vmware-iso: 1 error(s) occurred:

* Post-processor failed: Error uploading virtual machine: exit status 1

Because the build fails, Packer removes all the intermediate artifacts so I'm unable to perform a manual test with the output from a failed build. I've tried adding "keep_input_artifact" as true to my post-processor but this has no effect. The documentation implies that this is only checked after the successful execution of a post-processor. I've also tried -on-error=ask or -on-error=abort but neither of these ask me or abort when the post-processor runs into an error and keeps my VM artifact.

So far the only thing that has worked has been to run "packer build -debug" and babysit the process as it runs, moving through each step. Once it finishes the builder, but before it runs my post-processor, manually make a separate copy of the artifact and continue on to let the build fail. A manual test will let me see the ovftool error.

If anyone has any help for me to solve any of the above issues or other directions I might not have considered please let me know. I'll provide more info as necessary 

1. Lack of ovftool output

2. Unable to retain output from vmware-iso builder when post-processor fails

Thanks!

[Rickard von Essen]

Could you attach the full PACKER_LOG ? 

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/mitchellh/packer/issues
IRC: #packer-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Packer" group.
To unsubscribe from this group and stop receiving emails from it, send an email to packer-tool...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/a6b1f4a9-61c3-48e8-b4db-5ad12b7a0194%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jackson]

I've attached the log file. The only changes I made were to anonymize anything site specific. Thank you for taking the time to look.

[Rickard von Essen]

Unfortunately the output from the ovftool command is the empty line after "exit status 1"

* Post-processor failed: Error uploading virtual machine: exit status 1

 [0m

-on-error doesn't affect post-provisioners, I filed https://github.com/hashicorp/packer/issues/6901to fix or clarify this.

I would advice that you remove the post-processor and run a build so you get the artifact and then try to run:

ovftool --acceptAllEulas --name=centos7-tmpl-33b99e6 --datastore=datastore --diskMode=thick --network="Network Name" centos7-tmpl-33b99e6/centos7-tmpl-33b99e6.vmx vi://<username>:<password>@vcenter-host/datacenter/host/cluster/?dns=esxi-host

Manually to troubleshoot (change the user password to the correct, unmasked values).

To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/a7cafd2e-c816-490a-a406-5a6560765662%40googlegroups.com.

[Jackson]

My apologies, I included my manual testing as a short sentence near the end of my initial post and it is easy to miss. I've been working with this as the norm, created this post during one of my runs of "packer build" and kept going.

Output from ovftool isn't getting captured in the log or in the stdout of "packer build", forcing me to manually run the command to get useful debugging info to modify my packer template file.

For example, I took my template, changed my "esxi_host" value in the vsphere post-processor to something invalid. If I run the ovftool command manually, I still get "exit status 1" but I do not get the full error.

Manual run:

Error: Locator does not refer to an object: vi://packer%40vsphere.local@vcenter:443/Datacenter/host/Cluster/?dns=incorrect-esxi-host-dns

packer debug log gives me:

2018/10/23 11:56:32 [INFO] (telemetry) ending vsphere

2018/10/23 11:56:32 Deleting original artifact for build 'vmware-iso'

2018/10/23 11:56:32 ui error: ESC[1;31mBuild 'vmware-iso' errored: 1 error(s) occurred:

* Post-processor failed: Error uploading virtual machine: exit status 1

ESC[0m

2018/10/23 11:56:32 Builds completed. Waiting on interrupt barrier...

2018/10/23 11:56:32 machine readable: error-count []string{"1"}

2018/10/23 11:56:32 ui error:

==> Some builds didn't complete successfully and had errors:

2018/10/23 11:56:32 machine readable: vmware-iso,error []string{"1 error(s) occurred:\n\n* Post-processor failed: Error uploading virtual machine: exit status 1\n\n"}

2018/10/23 11:56:32 ui error: --> vmware-iso: 1 error(s) occurred:

* Post-processor failed: Error uploading virtual machine: exit status 1

2018/10/23 11:56:32 ui:

==> Builds finished but no artifacts were created.

Where does "Locator does not refer to an object" go? Additionally, I'm at the point now where a manual run of ovftool with the same command will succeed while packer is reporting the "exit status 1" failure.

It looks like I've got the following issues. I think 2 is either an error or misunderstanding on my part, it seems like 3 and 4 are bugs. I can open issues but I wanted to make sure I wasn't missing something

1. -on-error= behaviour is not clear (issue created by Rickard, thanks!)

2. "keep_input_artifact: true" in a post-processor does not work if the post-processor fails.

3. packer swallows legitimate errors from ovftool requiring a manual run with a copy of the artifacts from previous post-processor or build steps grabbed during pauses using -debug switch

4. vsphere post-processor is reporting a failure with no specific error from ovftool. Manual runs with the identical command work.

Thanks for all your help so far.

[Rickard von Essen]

2. "keep_input_artifact: true" in a post-processor does not work if the post-processor fails.

This is intended, if something in the build - provision - post-processor chain fails the whole build is removed. 

 

3. packer swallows legitimate errors from ovftool requiring a manual run with a copy of the artifacts from previous post-processor or build steps grabbed during pauses using -debug switch

It seems so, but I taught this was fixed in https://github.com/hashicorp/packer/pull/5499 which was released in 1.1.2.

But reading a bit more carefully I see it only prints the STDOUT not STDERR. Can you run the ovftool command that you run to get the error in your previous post but append:

2> /dev/null

on the end of the line (this will strip out stderr) and post the output here. What version of ovftool do you use?

4. vsphere post-processor is reporting a failure with no specific error from ovftool. Manual runs with the identical command work.

This is most likely a discrepancy in what you command packer is running and the one you run. This should hopefully be clarified by seeing the output when packer fails.

To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/778be8ac-9ce7-4fb3-995e-85efbb84462e%40googlegroups.com.

[Jackson]

2. "keep_input_artifact: true" in a post-processor does not work if the post-processor fails.

 This is intended, if something in the build - provision - post-processor chain fails the whole build is removed. 

Okay, thanks for clarifying, though this makes troubleshooting the below stuff more frustrating since I have to babysit a run with -debug in order to copy the artifact out at the right time

 2> /dev/null
on the end of the line (this will strip out stderr) and post the output here. What version of ovftool do you use?

A manual run dumping stderr to /dev/null does not show any different in output. This is on ovftool version 4.3.0, which is bundled with VMware Fusion 11.0.

 

4. vsphere post-processor is reporting a failure with no specific error from ovftool. Manual runs with the identical command work.

This is most likely a discrepancy in what you command packer is running and the one you run. This should hopefully be clarified by seeing the output when packer fails.

The command I run manually is identical to what Packer is using. I am copy/pasting this command from the packer.log file and replacing the masked password with the actually password but urlencoded. The credentials are the only changes I'm making to this line.

Based on what I'm getting here, is that 2 is a "me problem" but 3 and 4 are bugs.

[Rickard von Essen]

Okay, thanks for clarifying, though this makes troubleshooting the below stuff more frustrating since I have to babysit a run with -debug in order to copy the artifact out at the right time

This should be improved by when -on-error works for post-processors too.

 

A manual run dumping stderr to /dev/null does not show any different in output. This is on ovftool version 4.3.0, which is bundled with VMware Fusion 11.0.

Oh, now everything makes sense. Packer only outputs stderr (where the error should be shown). But it seems ovftool only sends it to stdout.

(The relevant packer code for reference)

         if err := cmd.Run(); err != nil {

err := fmt.Errorf("Error uploading virtual machine: %s\n%s\n", err, p.filterLog(errOut.String()))

return nil, false, err

}

Can you please try this (darwin) build where I just changed so STDOUT is printed on error instead and see if you get any output.

 packer

To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/536e527e-ba53-4e59-a8fc-9300661c5004%40googlegroups.com.

[Jackson]

Thanks for the custom build. I ran and was able to get the errors from the execution.

My issue ended up being the --network argument to ovftool. Because it has spaces, I had it quoted in my build template. Pasting the arguments from the packer log into the shell worked great, as my network name was already quoted, but when executed directly from packer, the quotes were causing the error.

Thanks very much for all your help.

[Rickard von Essen]

I filed https://github.com/hashicorp/packer/issues/6923 for fixing the output of STDOUT.

To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/19c7dd50-6874-4eeb-b427-48103242bdee%40googlegroups.com.