Packer fails to connect after domain join, windows-restart stuck in 401 invalid content

[Dina Muscanell]

Hi All,

configuration:

Packer v 1.4.0

Azure Builder

I have worked with Packer a bit in the past but not specifically a domain joined packer run.  Has anyone been able to get a Packer run to work while joining a domain?  

I have all of my logic written in Chef and in the Packer run, I just execute the Chef code. My test kitchen in Chef works fine and can keep going after I've joined the domain but Packer can't seem to continue.  

Both Chef and Packer use winrm to communicate but there has to be some difference if Chef is able to make sense of the domain join but Packer cannot.  Unfortunately, I haven't been able to figure out that difference.

After the domain join occurs, we have to run a windows_restart for the changes to take effect but the Packer run never recovers.  Even though the machine is up and running (I can log into as the domain account) Packer doesn't pick up that it is available.  Here is the debug log showing some output from the powershell script that runs to join the domain, the windows restart is initiated properly but never recovers and throws 401 invalid content until the windows-restart times out.

019/05/13 10:36:05 ui:     azure-arm: VERBOSE: Performing the operation "Join in domain 'mydomain.com'" on target
2019/05/13 10:36:05 ui:     azure-arm:  "pkrvm4hxw4n4f89".
2019/05/13 10:36:08 packer.exe: 2019/05/13 10:36:08 [INFO] command 'powershell -executionpolicy bypass -file "C:/Windows/Temp/packer-elevated-shell-5cd980cd-a187-c086-26f2-8d764c3484ff.ps1"' exited with code: 0
2019/05/13 10:36:08 packer.exe: 2019/05/13 10:36:08 [INFO] RPC endpoint: Communicator ended with: 0
2019/05/13 10:36:08 [INFO] 313 bytes written for 'stdout'
2019/05/13 10:36:08 [INFO] 0 bytes written for 'stderr'
2019/05/13 10:36:08 ui:     azure-arm: HasSucceeded ComputerName
2019/05/13 10:36:08 [INFO] RPC client: Communicator ended with: 0
2019/05/13 10:36:08 [INFO] RPC endpoint: Communicator ended with: 0
2019/05/13 10:36:08 packer.exe: 2019/05/13 10:36:08 [INFO] 313 bytes written for 'stdout'
2019/05/13 10:36:08 packer.exe: 2019/05/13 10:36:08 [INFO] 0 bytes written for 'stderr'
2019/05/13 10:36:08 packer.exe: 2019/05/13 10:36:08 [INFO] RPC client: Communicator ended with: 0
2019/05/13 10:36:08 ui:     azure-arm: ------------ ------------
2019/05/13 10:36:08 packer.exe: 2019/05/13 10:36:08 Error in Ui.Message RPC call: connection is shut down
2019/05/13 10:36:08 packer.exe: 2019/05/13 10:36:08 Error in Ui.Message RPC call: connection is shut down
2019/05/13 10:36:08 packer.exe: 2019/05/13 10:36:08 Error in Ui.Message RPC call: connection is shut down
2019/05/13 10:36:08 packer.exe: 2019/05/13 10:36:08 Error in Ui.Message RPC call: connection is shut down
2019/05/13 10:36:08 [INFO] (telemetry) ending powershell
2019/05/13 10:36:08 [INFO] (telemetry) Starting provisioner windows-restart
2019/05/13 10:36:08 ui: ==> azure-arm: Pausing 2m0s before the next provisioner...
2019/05/13 10:38:08 ui: ==> azure-arm: Restarting Machine
2019/05/13 10:38:09 [INFO] 0 bytes written for 'stderr'
2019/05/13 10:38:09 [INFO] 0 bytes written for 'stdout'
2019/05/13 10:38:09 packer.exe: 2019/05/13 10:38:09 [INFO] 0 bytes written for 'stderr'
2019/05/13 10:38:09 packer.exe: 2019/05/13 10:38:09 [INFO] 0 bytes written for 'stdout'
2019/05/13 10:38:09 packer.exe: 2019/05/13 10:38:09 Retryable error: http response error: 401 - invalid content type
2019/05/13 10:38:12 [INFO] 0 bytes written for 'stderr'
2019/05/13 10:38:12 [INFO] 0 bytes written for 'stdout'

windows-restart logic:

{

"type": "windows-restart",

"restart_command": "shutdown -r -f -t 10 && net stop winrm",

"restart_check_command": "powershell -command \"& {Write-Output 'restarted.'}\"",

"restart_timeout": "15m",

"pause_before": "2m"

},

My packer build has the following configured for winrm:

"communicator": "winrm",

"winrm_use_ssl": "true",

"winrm_insecure": "true",

"winrm_timeout": "5m",

"winrm_username": "packer"

I've tried to increase the timeout and updating winrm settings such as (but no luck): 

winrm set winrm/config/service @{AllowUnencrypted="true"}

Any ideas or help would be appreciated, thanks for taking the time to read this.

[Megan Marsh]

I'm a novice at windows-specific sysadmining -- when you do a domain join, does the original username/password you connected with still work or does the domain join render it no longer valid?

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/mitchellh/packer/issues
IRC: #packer-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Packer" group.
To unsubscribe from this group and stop receiving emails from it, send an email to packer-tool...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/4a7024b1-6f7f-469f-89aa-e2122189d1cb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dina Muscanell]

The packer user should still work.  Since we don't know the initial randomized packer password, to validate this I used the Azure portal to update the password for the packer user (still a local admin) and logged in with .\packer and the password I set in Azure.  I was able to log into the machine with both the domain account and the local packer admin account.  My packer run is not valid after changing the password for that packer user though, since Packer itself doesn't know the new password.

To unsubscribe from this group and stop receiving emails from it, send an email to packe...@googlegroups.com.

[Megan Marsh]

I'd like to validate this for absolute sure.  You can use the shell-local provisioner to find out the packer password and make sure that you can still connect via WinRM using those passwords.

  "provisioners": [

    {

      "type": "shell-local",

      "inline": ["echo \"Packer WinRMPassword is {{.WinRMPassword}}\""]

    }

  ]

Does the domain join initiate a restart itself? Packer does a pretty poor job of handling restarts that aren't initiated from inside of the restart provisioner.

[Dina Muscanell]

I used your snippet below (thank you!) to grab the packer password and login as the packer user during the point where the build gets stuck.  I've attached a side-by-side screen grab that shows:

• The output where the packer build gets stuck (the debug logs show 401 invalid content, logs in the original post)
• Top has a successful login as the Packer user
• Bottom has a successful login as the domain user

From what I can tell in the in the logs and screen grab, the domain join does not initiate a restart.  It seems the powershell that joins the domain returns a lastexitcode of 0. Then gets stuck on the next provisioner which is the windows-restart (orchestrated via packer).

[Dina Muscanell]

Screen grab described in prior post attached here..

[Megan Marsh]

I just took a look at the golang winrm library that we're using -- the README says it doesn't support domain users, which would explain why this stops working for you once you do a join. I saw that you found and commented on an issue open on the Packer repo that had to do with domain joins -- I'll share this information there, and label that issue as a feature request rather than a question. 

[Dina Muscanell]

That would be AWESOME!! Thanks so much for looking into this.  I'd be more than happy assist in any beta testing :)

[Jordan Borean]

The winrm library should support domain users, just not with basic auth (default). Set 'winrm_use_ntlm: true' in your packer.json to use NTLM auth which supports both local and domain accounts.

[Jordan Borean]

Further to the above, if using NTLM auth still doesn't work I would be curious what the output of 'winrm get winrm/config/service' looks like after the host comes back from the reboot after joining the domain.

[Dina Muscanell]

I tried to update the winrm_use_ntlm to true and see if that worked.  Unfortunately, it did not.  

There is also a screen grab attached of the winrm settings on both the local packer user and the domain user, these are identical on both machines

Here is what my winrm configuration looks like in my packer build file.

"communicator": "winrm",

"winrm_use_ssl": "true",

"winrm_insecure": "true",

"winrm_timeout": "5m",

"winrm_username": "packer",

"winrm_use_ntlm": true,

[Jordan Borean]

Darn I was thinking there was a GPO policy that was forcing the CbtHardeningLevel to Strict which I don't believe the Go winrm library supports but relaxed seems to be fine.

There's potentially another setting that a GPO policy may be overriding but it's a long shot. The LocalAccountTokenFilterPolicy [1] registry value needs to be set to 1 for local admin accounts to work. Because you said Chef is still working I doubt that this will be it but it's worth looking at the registry key to see if it gets changed.

I'm not sure what release of Packer included the dependency change of the ntlm library but there is some background info on this [2]. Going by the GPO idea above, it could be setting the minimum NTLM protocol version required and if you aren't using the newer ntlm Go library then it will fail. Probably worth looking at.

Thanks

Jordan

[1] - https://support.microsoft.com/en-us/help/942817/how-to-change-the-remote-uac-localaccounttokenfilterpolicy-registry-se

[2] - https://github.com/hashicorp/packer/issues/6205

[Shubham Bansal]

Hey I was using packer and facing issues.

Can you help me with how you took screen grab of the packer built machine?

[Dylan Wilson]

Hi Dina, were you able to resolve this? Coworkers and I are running into similar issues.

[Andrew Webber]

Hi all. I am coming across the same issue. I successfully domain join, but after the restart Packer just times out. I assume it is because the "packer" account is being tried against the AD domain rather than local machine accounts. Is there a way to force packer to try the local machine i.e ".\"

[dragon788]

There are a number of potential issues after joining the domain, it could be group policies that block/disable WinRM or enable the firewall or a number of other things. Your best bet is to do a run that keeps the input artifact and doesn't clean up so you can log in yourself and check the state of the system. you can also do this during the course of Iran by putting a long sleep and enabling remote desktop or a VNC connection and connecting to that before the domain join and immediately after the restart to see what happened.

[Andrew Webber]

Thanks for the response :). I will double check the gpos and see if anything is stopping winrm or enabling the FW. Many thanks again

[Andrew Webber]

So I double checked GPOs and nothing was blocking FW or WINRM. I have logged on to the build (prior to the failure as i have a 10m timeout) and I can see though in the event log that the packer account is trying to logon to the domain, rather than a local account. Is there a way to tell packer to do a local logon?

[trey perrone]

I don't know packer specific, but I recall years ago using `.\` as in `.\username` at the UI prompt to override that kind of issue.

[Andrew Webber]

I did think that might be an idea, but the JSON file i was using with AzureDevOps didn't like the syntax. I will take another look as might be worth a shot :)

[Andrew Webber]

Tried using .\packer in the winrm_username section, but it still tried to do a domain logon.

[trey perrone]

I'm not super up on azure, but if you domain join and create an image, won't it be baked into that image and cause dupe machines in AD when multiples are launched? I've not done a ton with Windows in recent years so maybe that's changed?

Doesn't any Windows image need sysprep run before creation to avoid dupe ids; IIRC from doing this back in the day on regular old laptops...when you domain join it generates a new SID for the domain at join time.

[Drew]

I am testing building an image for a Citrix environment, so want some things to be baked into the image :)

On Thu, 16 Apr 2020 at 12:35, trey perrone <trey.p...@gmail.com> wrote:

I'm not super up on azure, but if you domain join and create an image, won't it be baked into that image and cause dupe machines in AD when multiples are launched? I've not done a ton with Windows in recent years so maybe that's changed?

Doesn't any Windows image need sysprep run before creation to avoid dupe ids; IIRC from doing this back in the day on regular old laptops...when you domain join it generates a new SID for the domain at join time.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.

GitHub Issues: https://github.com/mitchellh/packer/issues
IRC: #packer-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Packer" group.

To unsubscribe from this group and stop receiving emails from it, send an email to packer-tool...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/packer-tool/5ccb244a-3741-4e4a-bdd0-6eb0c0f4d6bd%40googlegroups.com.