Unprivileged and non-interactive

291 views
Skip to first unread message

andr...@gmail.com

unread,
Aug 9, 2017, 4:20:24 AM8/9/17
to Packer
I'm trying to run packer from a Teamcity agent, which means that the packer is spawned under unprivileged service account.

If I log in to that service account interactively and run packer build it works.

This is the build I'm running: https://github.com/boxcutter/ubuntu

It also works from the service account, if I give the service account admin rights on the box. But if it's not admin, and it's from service I'm getting errors.

Here are errors I'm getting with headless: false:

==> virtualbox-iso: Error configuring VirtualBox to suppress messages: VBoxManage error: VBoxManage.exe: error: Failed to create the VirtualBox object!
==> virtualbox-iso: VBoxManage.exe: error: The object is not ready
==> virtualbox-iso: VBoxManage.exe: error: Details: code E_ACCESSDENIED (0x80070005), component VirtualBoxClientWrap, interface IVirtualBoxClient
==> virtualbox-iso: Deleting output directory...
Build 'virtualbox-iso' errored: Error configuring VirtualBox to suppress messages: VBoxManage error: VBoxManage.exe: error: Failed to create the VirtualBox object!
VBoxManage.exe: error: The object is not ready
VBoxManage.exe: error: Details: code E_ACCESSDENIED (0x80070005), component VirtualBoxClientWrap, interface IVirtualBoxClient
==> vmware-iso: Error starting VM: VMware error: Error: There was an error in communication
==> vmware-iso: Deleting output directory...
Build 'vmware-iso' errored: Error starting VM: VMware error: Error: There was an error in communication

==> Some builds didn't complete successfully and had errors:
--> virtualbox-iso: Error configuring VirtualBox to suppress messages: VBoxManage error: VBoxManage.exe: error: Failed to create the VirtualBox object!
VBoxManage.exe: error: The object is not ready
VBoxManage.exe: error: Details: code E_ACCESSDENIED (0x80070005), component VirtualBoxClientWrap, interface IVirtualBoxClient
--> vmware-iso: Error starting VM: VMware error: Error: There was an error in communication

Here are errors I'm getting with headless true:

==> virtualbox-iso: Error configuring VirtualBox to suppress messages: VBoxManage error: VBoxManage.exe: error: Failed to create the VirtualBox object!
==> virtualbox-iso: VBoxManage.exe: error: The object is not ready
==> virtualbox-iso: VBoxManage.exe: error: Details: code E_ACCESSDENIED (0x80070005), component VirtualBoxClientWrap, interface IVirtualBoxClient
==> virtualbox-iso: Deleting output directory...
Build 'virtualbox-iso' errored: Error configuring VirtualBox to suppress messages: VBoxManage error: VBoxManage.exe: error: Failed to create the VirtualBox object!
VBoxManage.exe: error: The object is not ready
VBoxManage.exe: error: Details: code E_ACCESSDENIED (0x80070005), component VirtualBoxClientWrap, interface IVirtualBoxClient
==> vmware-iso: Error starting VM: VMware error: Error: Unknown error
==> vmware-iso: 
==> vmware-iso: Packer detected a VMware 'Unknown Error'. Unfortunately VMware
==> vmware-iso: often has extremely vague error messages such as this and Packer
==> vmware-iso: itself can't do much about that. Please check the vmware.log files
==> vmware-iso: created by VMware when a VM is started (in the directory of the
==> vmware-iso: vmx file), which often contains more detailed error information.
==> vmware-iso: Deleting output directory...
Build 'vmware-iso' errored: Error starting VM: VMware error: Error: Unknown error

Packer detected a VMware 'Unknown Error'. Unfortunately VMware
often has extremely vague error messages such as this and Packer
itself can't do much about that. Please check the vmware.log files
created by VMware when a VM is started (in the directory of the
vmx file), which often contains more detailed error information.

==> Some builds didn't complete successfully and had errors:
--> virtualbox-iso: Error configuring VirtualBox to suppress messages: VBoxManage error: VBoxManage.exe: error: Failed to create the VirtualBox object!
VBoxManage.exe: error: The object is not ready
VBoxManage.exe: error: Details: code E_ACCESSDENIED (0x80070005), component VirtualBoxClientWrap, interface IVirtualBoxClient
--> vmware-iso: Error starting VM: VMware error: Error: Unknown error

Packer detected a VMware 'Unknown Error'. Unfortunately VMware
often has extremely vague error messages such as this and Packer
itself can't do much about that. Please check the vmware.log files
created by VMware when a VM is started (in the directory of the
vmx file), which often contains more detailed error information.


How can I configure the service account to run, so that the build does not fail, without giving full admin rights?

Thank you in advance,
Andrew


Alvaro Miranda Aguilera

unread,
Aug 9, 2017, 5:15:49 AM8/9/17
to packe...@googlegroups.com
Hello

is this windows?

you can use runas to manually run the tools and see where it fails. perhaps to a user without admin first.


you can set a variable PACKER_LOG=1 to get more verbose messages.



Virusvrij. www.avast.com

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/mitchellh/packer/issues
IRC: #packer-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Packer" group.
To unsubscribe from this group and stop receiving emails from it, send an email to packer-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/0ab765e5-b65a-4a4d-9516-bef9fa5293c9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
Alvaro

andr...@gmail.com

unread,
Aug 9, 2017, 6:03:52 AM8/9/17
to Packer
Thank you for your reply! Yes, it's windows.

runas works without error - as long as it's run interactively, that is not from a windows service it works.

I set the variable and I examined the more verbose log. No additional clues found. :(

Andrew
To unsubscribe from this group and stop receiving emails from it, send an email to packer-tool...@googlegroups.com.



--
Alvaro

andr...@gmail.com

unread,
Aug 9, 2017, 6:29:21 AM8/9/17
to Packer
Here is the verbose log https://gist.github.com/AndrewSav/670622133d5b24e00b6d85308254fd34

I used this https://nssm.cc/ to create a windows service that points to a command file that starts packer (with the same command line that works interactively, and also as runas). The logs produced are at the link above.

Any insight how to make that work will be greatly appreciated.

Cheers,
Andrew

On Wednesday, 9 August 2017 21:15:49 UTC+12, Alvaro Miranda Aguilera wrote:
To unsubscribe from this group and stop receiving emails from it, send an email to packer-tool...@googlegroups.com.



--
Alvaro

Alvaro Miranda Aguilera

unread,
Aug 9, 2017, 7:01:06 AM8/9/17
to packe...@googlegroups.com
Hello

So is there any Antivirus? can you disable it?

It seems realted to the processes in this case virtualbox and vmware not being able to create COM objects.

I have TeamCity at home, and I use the agent to run under a normal user, so I can login, open VMWARE/Virtualbox and ofcourse packer runs fine.

is anything that impedes you to run into a nomal user, or directly from TC agent run packer?


Alvaro.


Virusvrij. www.avast.com

To unsubscribe from this group and stop receiving emails from it, send an email to packer-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/17e86fd0-f110-4447-9a2a-20735ee72c85%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Alvaro

andr...@gmail.com

unread,
Aug 9, 2017, 3:24:25 PM8/9/17
to Packer
No there is no Antivirus.

>> I can login, open VMWARE/Virtualbox and ofcourse packer runs fine.

Well If I login it also runs fine. It does not run fine from within service.

>>Is anything that impedes you to run into a nomal user, or directly from TC agent run packer?

I'm not sure what you mean.
As I explained before whenever I run it interactively it works. Whenever it runs from a non admin  user from a windows service it does not.
Since I realize that not everyone has Teamcity setup handy to reproduce, I reproduced this also without teamcity using nssm.

Note that both VMWare and VirtualBox provisioners fail, they just fail differently.

Thanks,
Andrew



--
Alvaro

andr...@gmail.com

unread,
Aug 9, 2017, 4:17:03 PM8/9/17
to Packer
Here is how one can reproduce this behaviour.

Make sure you are running Windows 10 or Windows Server 2016 with:
- Virtualbox
- Vmware Workstation
- Windows Management Framework 5.1
- nssm

All commands are run from elevated powershell prompt.

Run:
mkdir c:\packertest
cd c:\packertest
cd ubuntu
notepad ubuntu1604.json


Change following line in ubuntu1604.json:
 "iso_checksum": "f3532991e031cae75bcf5e695afb844dd278fff9",
 "iso_name": "ubuntu-16.04.3-server-amd64.iso",

Run packer from command line and observe that it works:
packer build -var-file="ubuntu1604.json" -only=virtualbox-iso -only=vmware-iso ubuntu.json

Now install it as a service:
nssm install aaaPackerTest packer build -var-file="c:\packertest\ubuntu\ubuntu1604.json" -only=virtualbox-iso -only=vmware-iso c:\packertest\ubuntu\ubuntu.json
nssm set aaaPackerTest AppDirectory c:\packertest\ubuntu
nssm set aaaPackerTest Start SERVICE_DEMAND_START

$pwd = ConvertTo-SecureString "P@ssW0rD!" -AsPlainText -Force
New-LocalUser "packerTest" -Password $pwd

nssm set aaaPackerTest ObjectName .\packerTest "P@ssW0rD!"
nssm set aaaPackerTest AppExit Default Exit
nssm set aaaPackerTest AppStdout c:\packertest\log-out.txt
nssm set aaaPackerTest AppStderr c:\packertest\log-err.txt
nssm set aaaPackerTest AppStdoutCreationDisposition 2
nssm set aaaPackerTest AppStderrCreationDisposition 2
nssm set aaaPackerTest AppEnvironmentExtra PACKER_LOG=1 PACKER_LOG_PATH=c:\packertest\packer.log

At this point Logs you can run the service with:
Start-Service aaaPackerTest

And observe the errors I described. The logs and output will be in c:\packertest\packer.log and c:\packertest\log-out.txt respectively.

Cheers,
Andrew



--
Alvaro

andr...@gmail.com

unread,
Aug 9, 2017, 4:32:41 PM8/9/17
to Packer
Also note that as soon as you add `packerTest` user to the admin group, suddenly it works.

Alvaro Miranda Aguilera

unread,
Aug 10, 2017, 3:13:29 AM8/10/17
to packe...@googlegroups.com
can you create VMs using Virtualbox or vmware?

because per logs packer runs fines, is vmware/virtualbox that fails to the their job

you can use VBOXMange to do lots of things

try create an empty VM and boot

or deploy a VM from OVA and boot



Virusvrij. www.avast.com

To unsubscribe from this group and stop receiving emails from it, send an email to packer-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/310426c8-f070-43bd-bfcc-446d8a0c9f52%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Alvaro

andr...@gmail.com

unread,
Aug 11, 2017, 5:18:33 AM8/11/17
to Packer
>>because per logs packer runs fines, is vmware/virtualbox that fails to the their job

Well, yes, but since the intimate relationships between packer and vmware/virtualbox are not in the plain sight it's not immediately obvious.

So I had to run sysinternals procmon to be able to capture command lines packers uses to interact with vmware/virtualbox. Here are my findings:

Virtualbox 

Indeed, running VBoxManage that packer runs from the service results in the same error about COM object.
After a few hours of investigation I concluded, that the "Access Denied" error is result of different privileges held by the process. However the only difference between the service and the interactive process is that the service is in SERVICE built-in security group and interactive in the INTERACTIVE one. This gave me a clue. I opened the Component Services applet, right clicked My Computer:


I edited the Launch permissions and added the Account to the list. That allowed the Virtualbox part to work.


I was not that fortunate with VMWare. I do not believe it's possible in principle with it. I was able to find some VIX log files that identified that vmrun was not able to connect to vmware-auth daemon. I'm suspecting that in headless mode it might not be possible at all (the unprivileged account works in headed mode if not in a service but that, does not help much with headed mode, that is connected differently).


The last straw was somewhat unrelated happening: when I ran `vagrant up` for a vmware machine I got this nice message:


"When using Vagrant with VMware Workstation on Windows, Vagrant must

run with administrative privileges. I realize this is not ideal,

but this is the only way to make network modifications with VMware.

Please run Vagrant in a console with Windows administrative privileges."


That convinced me me that vmware internal workings in entirely FUBAR, and I was exhausted after 2 days of investigation and experiments, so I gave up.


I accept, that this is something that can be solved only by someone who developed VmWare suite, which is not very likely.


But hey, I at least figured out the virtualbox part!



--
Alvaro

Reply all
Reply to author
Forward
0 new messages