win_template or win_copy Ansible 2.3 Windows 2008R2 - possible bug.

[Kent Collins]

Ansible version - 2.3 

Python version - 2.7.12

My playbook runs through until it stumbles on copying some SSL certs across to a remote Windows 2008R2 box. The same error using both win_template and win_copy when trying to debug the issue. Also happens trying to copy any other file type. When running the same playbook with Ansible verison 2.2.1, it executes the win_template or win_copy successfully. When trying to debug the error message, the error doesn't accurately represent the problem as you will see below, "i dont believe this is a authorisation issue". 

Does anyone have any suggestions with a way around this or a fix? I have thought about copying the module that is used for both the modules stated from 2.2 to our new Ansible 2.3 controller. However, in my opinion, that is not best practice, i'd rather find a solution.

Apologies if this has already been reported, I did search for keywords around the error and didn't find anything relevant.

Any help would be appreciated!  - Kent

Non-verbose Ansible error:

fatal: [10.47.254.119]: UNREACHABLE! => {"changed": false, "msg": "Authentication or permission failure. In some cases, you may have been able to authenticate and did not have permissions on the remote directory. Consider changing the remote temp path in ansible.cfg to a path rooted in \"/tmp\". Failed command was: PowerShell -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -EncodedCommand UwBlAHQALQBTAHQAcgBpAGMAdABNAG8AZABlACAALQBWAGUAcgBzAGkAbwBuACAATABhAHQAZQBzAHQACgAoAE4AZQB3AC0ASQB0AGUAbQAgAC0AVAB5AHAAZQAgAEQAaQByAGUAYwB0AG8AcgB5ACAALQBQAGEAdABoACAAJABlAG4AdgA6AHQAZQBtAHAAIAAtAE4AYQBtAGUAIAAiAGEAbgBzAGkAYgBsAGUALQB0AG0AcAAtADEANAA5ADYAMgAyADUANQAyADcALgAxADUALQAxADAANwA4ADQAMwAwADcAOQAyADcANwA1ADgAMQAiACkALgBGAHUAbABsAE4AYQBtAGUAIAB8ACAAVwByAGkAdABlAC0ASABvAHMAdAAgAC0AUwBlAHAAYQByAGEAdABvAHIAIAAnACcAOwAKAEkAZgAgACgALQBuAG8AdAAgACQAPwApACAAewAgAEkAZgAgACgARwBlAHQALQBWAGEAcgBpAGEAYgBsAGUAIABMAEEAUwBUAEUAWABJAFQAQwBPAEQARQAgAC0ARQByAHIAbwByAEEAYwB0AGkAbwBuACAAUwBpAGwAZQBuAHQAbAB5AEMAbwBuAHQAaQBuAHUAZQApACAAewAgAGUAeABpAHQAIAAkAEwAQQBTAFQARQBYAEkAVABDAE8ARABFACAAfQAgAEUAbABzAGUAIAB7ACAAZQB4AGkAdAAgADEAIAB9ACAAfQA=, exited with result 1", "unreachable": true}

The powershell temp module that is run for windows_template:

#Requires -Version 3.0

begin {

$DebugPreference = "Continue"

$ErrorActionPreference = "Stop"

Set-StrictMode -Version 2

function ConvertTo-HashtableFromPsCustomObject ($myPsObject){

$output = @{};

$myPsObject | Get-Member -MemberType *Property | % {

$val = $myPsObject.($_.name);

If ($val -is [psobject]) {

$val = ConvertTo-HashtableFromPsCustomObject $val

}

$output.($_.name) = $val

}

return $output;

}

# stream JSON including become_pw, ps_module_payload, bin_module_payload, become_payload, write_payload_path, preserve directives

# exec runspace, capture output, cleanup, return module output

$json_raw = ""

}

process {

$input_as_string = [string]$input

$json_raw += $input_as_string

}

end {

If (-not $json_raw) {

Write-Error "no input given" -Category InvalidArgument

}

$payload = ConvertTo-HashtableFromPsCustomObject (ConvertFrom-Json $json_raw)

# TODO: handle binary modules

# TODO: handle persistence

$actions = $payload.actions

# pop 0th action as entrypoint

$entrypoint = $payload.($actions[0])

$payload.actions = $payload.actions[1..99]

$entrypoint = [System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($entrypoint))

# load the current action entrypoint as a module custom object with a Run method

$entrypoint = New-Module -ScriptBlock ([scriptblock]::Create($entrypoint)) -AsCustomObject

Set-Variable -Scope global -Name complex_args -Value $payload["module_args"] | Out-Null

# dynamically create/load modules

ForEach ($mod in $payload.powershell_modules.GetEnumerator()) {

$decoded_module = [System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String($mod.Value))

New-Module -ScriptBlock ([scriptblock]::Create($decoded_module)) -Name $mod.Key | Import-Module -WarningAction SilentlyContinue | Out-Null

}

$output = $entrypoint.Run($payload)

Write-Output $output

}

[jhawkesworth]

Please can you share your playbook, or better still a cut down version of it that reproduces the problem?

Also, since you are running Server 2008 R2, please ensure you have this hotfix installed http://support.microsoft.com/kb/2842230 
as mentioned at the bottom of this section of the documentation here: http://docs.ansible.com/ansible/intro_windows.html#windows-system-prep

There is a similar-sounding issue reported here: https://github.com/ansible/ansible/issues/24057

If this sounds like the same issue, please can you re-run and check the event log for any error messages.  I'm keen to resolve the above issue but cannot reproduce it myself.

Many thanks,

Jon

[Kent Collins]

Thanks, Yes the 2008R2 does have that hotfix. I will have a read, so far looking at the issue is pointing towards pipelining from what I can tell. I will have a further read of the link you posted, and post further updates to this post if I feel its relevant (as in if its a different bug). If not, il remove the post.

Thanks

[Kent Collins]

Having believed that the Microsoft hotfix @hhawkesworth posted was already installed, I stand corrected. On further investigation the hotfix was not applied. 

Thinking out loud, this is to do with pipelining and powershell tmp module not being able to handle the threads??  

[jhawkesworth]

With the hotfix applied, is the problem now resolved?

If I recall the reason for the hotfix being required was a bug in the WMF 3.0 framework which caused only a tiny amount of memory to be allocated to winrm processes, causing frequent OutOfMemory errors.  Its pretty much unusable without - I haven't tried it for a long time but I think the only thing I could get to work without it was win_ping.

I'm not using s2008 releases any more but when I was I settled on upgrading to WMF 4.0 so I was using the same winrm version on the 2008s and 2012 boxes.

Upgrading to WMF 4.0 (which also gets you later powershell) might be worth trying if you are still having problems.

Hope this helps,

Jon

[Kent Collins]

The Microsoft hotfix as posted by @jhawkesworth did fix the memory quota with Powershell on Windows 2008R2. However, i'm still leaning that this is a Ansible 2.3 bug, as the only thing that has changed is upgrading from Ansible 2.2. 

However, I am happy with this work around for the moment. We dont have a large number of 2008R2 boxes so happy to apply this fix. 

I wont be replying anymore in this post as I am happy with the result. But will leave it open for others to find and be of some use. 

Kent