Suggestions for win_copy and win_ansible

[Trond Hindenes]

There's been a lot of talk about these, and its time to get them into source.

As several (I can't remember the names right now, sorry!) in this group have tested, the copy module will work (albeit with limited functionality) as long as there's a copy.ps1 module in the windows directory, and that file is able to lay down the input file onto disk.

My suggestion is that we build win_copy and win_template modules, in order to reflect the differences around permissions on NTFS vs linux filesystems.

From what I can see, Ansible uses a 1:1 mapping between action plugins and modules, which would mean that we would need a win_copy.py and win_template.py action plugin as well.

The win_copy.py action plugin would be 100% identical to its copy.py sibling, and the win_template.py plugin would be identical to the template.py plugin with the only difference being that it calls win_copy instead of copy when executing the file copy operation(s) underneath.

my suggestion is that we build out simple support for copying/templating files using the matching modules, since the lack of these functions are limiting the usefullness of Ansible on the Windows platform. Once we have those in source we can start building out support for permissions etc.

I am more than willing to do this tweaking, but I'd like Michael (or someone's) take on this path before i mash together (or steal) the code.

What say ye?

-Trond

[Chris Church]

Given the differences in specifying permissions and no need for some of the other file/copy options, I'm +1 for having separate win_file/win_copy/win_template modules.

So, to add to your notes, I think the following changes are needed to get all of this working.  I'm currently tackling some kerberos-related things, but would gladly offer my feedback if you wanted to submit a PR or point me to a branch of work in progress.

In ansible:

• lib/ansible/runner/action_plugins/win_copy.py would be mostly based on copy.py. 
  
• lib/ansible/runner/action_plugins/win_template.py would be mostly based on template.py. Somebody with initials CC should probably fix this todo while we're at it: https://github.com/ansible/ansible/blob/devel/lib/ansible/runner/action_plugins/template.py#L79
• lib/ansible/module_utils/powershell.ps1 - Add common functions (MD5, etc.) needed for win_file and win_copy modules.
• test/integration/... - Integration tests for win_file/win_copy/win_template.

In ansible-modules-core:

• windows/win_file.ps1 could be based on https://gist.github.com/tkinz27/8a0b49f3c8b0ea4575c6 without the permissions code.  I'd think only the path and state parameters are needed; state only needs to support file, directory, touch and absent.
  
• windows/win_file.py would define module docs.
  
• windows/win_copy.ps1 could be based on https://gist.github.com/tkinz27/fd92ba9af0e0309614ee but use PowerShell 3 compatible MD5 code from https://github.com/ansible/ansible/blob/devel/lib/ansible/runner/shell_plugins/powershell.py#L87 or use a common function for MD5.
  
• windows/win_copy.py would define module docs.
  
• windows/win_template.py would define module docs.
  

The permissions code from https://gist.github.com/tkinz27/8a0b49f3c8b0ea4575c6 could probably become the basis for a future win_acl module or be added to win_file/win_copy/win_template in the future.

--
You received this message because you are subscribed to the Google Groups "Ansible Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-deve...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Trond Hindenes]

Great stuff! I think we're on the same page here.

There are some problems I see with my win_template (copied from template and changed the referenced command from copy to win_copy). It seems to work nicely if the dest file doesn't already exist, but if it the dest file exists it throws some weird error saying it's missing some core modules. I'll try and branch out my changed files so that someone in the know can have a look.

In the meantime, I'll get going with win_copy and win_template powershell scripts, making sure to test them on PS v3.

[Trond Hindenes]

I've started working on this at:

https://github.com/trondhindenes/ansible/tree/win_copy_win_template

and

https://github.com/trondhindenes/ansible-modules-core/tree/win_copy_win_template

Appreciating any help!

-Trond

[jhawkesworth]

Thanks for this I hope to get some testing time this week.  If I do I will give these a try - completely agree on the need for these.

Best 

Jon

[John Jelinek]

Has there been any progress on this?

[jhawkesworth]

A little.  I tried out Trond's changes and sent him some revisions last week.  I had a functioning but incomplete win_copy module, but no tests yet and haven't actually tried the win_template functionality.

I'm hoping to do a bit more on this as I get time as I have a functioning win_unzip module which I want to contribute.  It works well but cannot yet copy files from the controller to the host.

Jon

[Joey]

Trying out win_copy and it looks like Get-FileHash requires Powershell 4. Do the configure_remoting and upgrade_powershell scripts need to be updated to require Powershell 4? Or is there another way of doing the file hash that doesn't require updating?

[Joey]

Answering my own question, this seems to work.

function md5hash($path)
{
    $fullPath = Resolve-Path $path
    $md5 = new-object -TypeName System.Security.Cryptography.MD5CryptoServiceProvider
    $file = [System.IO.File]::Open($fullPath,[System.IO.Filemode]::Open, [System.IO.FileAccess]::Read)
    [System.BitConverter]::ToString($md5.ComputeHash($file))
    $file.Dispose()
}

[Joey]

After switching to the new md5 hash function, win_copy is partially working for me. If the md5 match fails and it has to write a new file, it works, but breaks most whitespace in the file (specifically it seems to remove linebreaks and tabs).

If the md5 match succeeds and it doesn't have to write the file, there is a WINRM error and it fails. The error looks like the WINRM PUT command are writing out to ansible-tmp-****\\file, and the WINRM EXEC command is looking for ansible-tmp-****\\file.ps1 (with the extension), so it errors out.

[jhawkesworth]

This is probably because behind the scenes it is attempting to use the file module to get info about the file if the md5 match succeeds.

I have been making progress on implementing win_copy, win_template and win_file and hope to have a PR to share very shortly - I will also include integration tests which I have passing for win_copy and win_file right now.

Jon

[Chris Church]

Very nice, thanks for your work on this!

Regarding the winrm connection adding a .ps1 extension, I may be able to remove that part entirely if we officially require windows modules to have the .ps1 extension.  At the moment, it still searches for module.ps1 then module (no extension).

--

[jhawkesworth]

Sorry this is taking longer than I hoped. I am debugging my through the consequences of the changes around md5 and checksum logic at the moment. With a bit of luck and a couple of clear hours I should have a PR to share. Just hope I can get something in before 1.8 sails.

Jon

[jhawkesworth]

Just to say at long last I have created PRs to add win_copy, win_file and win_template modules.

Both PRs would be needed (the modules all need action_plugins, which are in the main ansible project).

If anyone has the chance to review these and try them out I would be very grateful.  Right now I don't have a windows server 2008 /2008 r2 machine I can run against so if any one could try the integration tests against that, that would be really helpful.

The PRs are:

https://github.com/ansible/ansible/pull/9611

https://github.com/ansible/ansible-modules-core/pull/384

All the best,

Jon

[Joey]

How do I run the integration tests? I'm trying out win_template but it does the checksum and decides it doesn't need to do anything even though it hasn't updated the file. I'm trying to find the flag that keeps debug files on the client so I can look at the temp ps1 files and figure out why.

[Joey]

Trying to run the script in PowerShell ISE directly on the server gives this error:

PowerShell : Cannot find an overload for "CompareTo" and the argument count: "1".

At line:1 char:1

+ PowerShell -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -File "C:\Us ...

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : NotSpecified: (Cannot find an ...ent count: "1".:String) [], RemoteException

    + FullyQualifiedErrorId : NativeCommandError

 

At C:\Users\localadmin\AppData\Local\Temp\ansible-tmp-1416867885.79-1510893868

84966\win_copy.ps1:223 char:9

+     If (! $src_md5.CompareTo($dest_md5))

+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : NotSpecified: (:) [], MethodException

    + FullyQualifiedErrorId : MethodCountCouldNotFindBest

 

Cannot find an overload for "CompareTo" and the argument count: "1".

At C:\Users\localadmin\AppData\Local\Temp\ansible-tmp-1416867885.79-1510893868

84966\win_copy.ps1:230 char:10

+     If ( $src_md5.CompareTo($dest_md5))

+          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : NotSpecified: (:) [], MethodException

    + FullyQualifiedErrorId : MethodCountCouldNotFindBest

 

{

    "changed":  false,

    "checksum":  [

                     "3688374325b992def12793500307566d",

                     ""

                 ]

}

[Chris Church]

Assuming you're working from an ansible checkout, create test/integration/inventory.winrm to point to one or more windows hosts (they must be in the "windows" group).

From within that same directory, I'd run something like this: 

ANSIBLE_KEEP_REMOTE_FILES=1 TEST_FLAGS="-t test_win_template -vvv" make test_winrm

I use --start-at-task or temporarily add my own tags if I'm trying to work out an issue with one particular task.

make test_win already adds -v, so you can specify -vvvv and you'll be able to see the full stdout/stderr returned from trying to run the module.

You probably won't have much luck running the module standalone through PowerShell ISE unless you paste all of https://raw.githubusercontent.com/ansible/ansible/devel/lib/ansible/module_utils/powershell.ps1 in place of the # POWERSHELL_COMMON line in the module.

I haven't yet tried the new win_template module, will let you know if I'm seeing the same results.

--

[jhawkesworth]

Thanks for this, I'd add that in this case I wouldn't recommend running win_copy win_file or win_template without the corresponding action plugins as the plugins are needed to 

That said, I do debug powershell by setting ANSIBLE_KEEP_REMOTE_FILES=1 and then running the module.ps1 from inside Powershell ISE after a playbook run has completed or failed.  One little issue to be aware of is you need to pass the name of the arguments file in without the .\ that powershell likes to put in place if you use tab completion.

So run

.\win_copy.ps1 arguments

not

.\win_copy.ps1 .\arguments

Jon

[Janaka Abeywardhana]

Hi,

What's the latest with the win_copy module?

Thanks

[Chris Church]

There are 2 pull requests (one needs a rebase): https://github.com/ansible/ansible/pull/9611 and https://github.com/ansible/ansible-modules-core/pull/384 and some ongoing discussion on implementation details: https://groups.google.com/d/topic/ansible-devel/VRXSKQgEV70/discussion

Once those few minor issues are resolved and we have a chance to review and test, should be merged into devel.

--

[jhawkesworth]

Just to say I rebased https://github.com/ansible/ansible/pull/9611

Sorry its taken a while to get to it.

Jon

[jhawkesworth]

Just in case anyone is following this thread, win_copy, win_file and win_template have now been merged into dev.  

Thanks everyone.

Jon

[Michael DeHaan]

Yeah we just had a discussion about this.

I think we're going to (temporarily) un-merge, and look into the copy-paste code de-duplication first.

(They were also merged into -core, and really should be -extras first, which mostly was because the PRs were made to the wrong spots)

[Guillaume Belrose]

Hi all, 

I apologize in advance for the dumb question. I've been tasked at work to look at supporting Windows servers with Ansible. I started simple by trying to configure and install CouchDB on Windows. So far I like the agent-less model.

I now need to be able to upload files and tweak configurations via the template module. Which Ansible branch can I look at to try the win_copy, win_file and win_template modules? 

I've tried the devel branch (at commit  1698b17b1475e833b3b2930d723e047d7ff36b9e ) but it does not seem to contain the modules that I am looking for. 

I am getting an error like: ERROR: win_copy is not a legal parameter in an Ansible task or handler

I am sure this is a stupid configuration error from my part, but I can't work out how to make it work ( my Python and PowerShell scripts are not very good I am afraid). 

Thanks in advance, 

Guillaume.

[Phil Schwartz]

Hi Guillaume,

The win_copy module was reverted and pull out of the repos.  So you most likely cannot find it now.  I had tested it at some point and could not get it to work.

However, I am still very curious about this module because it would be so useful.  Is there any word on the latest of this module development?  I'd like to see the source, maybe even pull it into devel at ansible-modules-extras so that others could contribute to it and test.

It was only in the repo for a couple days before being merged into core and then reverted like the next day.


Thanks,

Phil

[Janaka Abeywardhana]

Hi,

+1

I'm also interested in this and might get some time to help test so I +1 for having it on a branch.

Thanks

Janaka Abeywardhana

Chief Technology Officer

 

(w):  www.axonn.co.uk |  (blog): glipho.com/axonn-technology

(in): uk.linkedin.com/in/janakaabeywardhana  |  (tw): @janaka_a

Metropolitan Wharf | 70 Wapping Wall | London | E1W 3SS

Agoraa - Software for marketers. Power your content hub with Agoraa and simplify effective content marketing.

[Michael DeHaan]

We're not going to likely hit this until the 'v2' tree is complete, as we do think it needs to inherit from other modules, but we need to take time to look at the architecture of how this is implemented in the new tree.

[Guillaume Belrose]

Thanks for the update Michael. 

For the time being, I am applying the template locally (on the Linux 'master' node) and I serve the template output via a web server using the win_get_url module. It feels a bit like a hack, but it actually works quite well. 

Is there a time frame for the completion of the v2 tree?

Cheers, 

Guillaume.

[Michael DeHaan]

Yep!

v2 is going to be done for 1.9 (undecided on whether we call this a "2.0", but our goal is to have it 99.X% compatible) .... we hope to have it in a state where people can help test and contribute to it in a month or so (approximately).

--Michael

[Martin Etmajer]

Hi!

1.9 rc1 has just been released on GitHub, but I don't see any extra windows modules shipping with it. Is there any update on your "schedule"?

Thanks, Martin

[Janaka Abeywardhana]

I would also like to know when this stuff will be out please.

-- Sent from a mobile device  --

[jhawkesworth]

My understanding is this.

The plan was to tackle the common logic issue (specifically so that template and win_template can share as much code as it makes sense to share and any other action plugins that we may well want to have different implementations of for windows) in the v2 codebase.  1.9 is still using the older codebase.

So when 1.9 is released and v2 becomes the current development version this can be tackled.

That said, the win_copy and win_template action plugins have been merged into 1.9 rc so that the action plugins can be worked on once v2 is there to support the necessary changes.

This means that if you really want to try win_copy / win_file / win_template you can grab them from https://github.com/ansible/ansible-modules-core/pull/384 into your /etc/ansible/library.  Obviously these may well change and you may or may not have to fix your playbooks to work with the supported versions.

If you do try this, be aware of the following.

If you need windows line endings  (\r \n)

1/  put the following on the first line of your template:

#jinja2: newline_sequence:'\r\n'

2/ If your template file was created on a windows machine, remove the byte order mark if present (will be the first hidden character of the file) as this trips up Ansible.  You can see if this is present using od -cx 

Jon

[John Jelinek]

v2 is now in active development, but u don't see mention of win_copy/file/template in the CHANGELOG. Any updates whether this will get integrated?

[Miguel Cantu]

+1. It would be great seeing these modules pulled in.

[jhawkesworth]

In case anyone is still watching this thread, win_copy, win_file and win_template have now been merged into latest devel.

Jon

[Janaka Abeywardhana]

Nice!

Janaka Abeywardhana

Chief Technology Officer

 

(w):  www.axonn.mediae |  (blog): glipho.com/axonn-technology

(in): uk.linkedin.com/in/janakaabeywardhana  |  (tw): @janaka_a

Metropolitan Wharf | 70 Wapping Wall | London | E1W 3SS

agoraa - Software for marketers. Power your content hub with Agoraa and simplify effective content marketing.

--
You received this message because you are subscribed to a topic in the Google Groups "Ansible Development" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ansible-devel/Jl5qP73CiKo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ansible-deve...@googlegroups.com.

[Bob Tanner]

On Thursday, June 11, 2015 at 3:24:23 PM UTC-5, jhawkesworth wrote:

In case anyone is still watching this thread, win_copy, win_file and win_template have now been merged into latest devel.

Did it get back ported to v1?

$ find . -name "win_copy*"
./lib/ansible/modules/core/windows/win_copy.ps1
./lib/ansible/modules/core/windows/win_copy.py
./v1/ansible/modules/core/windows/win_copy.ps1
./v1/ansible/modules/core/windows/win_copy.py
./v1/ansible/runner/action_plugins/win_copy.py