Salt-cp corrupting the file after transfer to minion

[Ramakanta Sahoo]

Hello all,

I am evaluating salt for one of my projects and was testing the salt-cp to send a small exe file to one of the minions.

Though the files sent was successful from master to minion but it seems the file is getting corrupted. If I just do a binary copy of the file to the minion system and run it it works very fine.

Is there something wrong I am doing or some thing needs to be added to command to stop it corrupting?

Example command which I used:

salt-cp -t 200 '*' "/opt/NDP451-KB2858728-x86-x64-AllOS-ENU.exe" "C:\TEMP\NDP451-KB2858728-x86-x64-AllOS-ENU.exe" -l debug

[Ramakanta Sahoo]

Any help much appreciated!!

[Todd Stansell]

Digging through the code, it looks like salt-cp does this to read
the file (at least in 2014.7.5):

with salt.utils.fopen(fn_, 'r') as fp_:
data = fp_.read()

salt.utils.fopen() is simply a wrapper for python's open() and passes all the
args directly to it without modification, so it's doing an open(fn_, 'r').
Since this doesn't include the 'b' flag, it's potentially not opening the file
in binary mode.

The open() docs on docs.python.com say this:

The default is to use text mode, which may convert '\n' characters to a
platform-specific representation on writing and back on reading. Thus,
when opening a binary file, you should append 'b' to the mode value to
open the file in binary mode, which will improve portability.

So, I'm guessing it's corrupting the file because it's not enforcing binary
mode while reading the file.

You could test this by manually editing the salt/cli/cp.py file and making the
following change in the _file_dict() function:

- with salt.utils.fopen(fn_, 'r') as fp_:
+ with salt.utils.fopen(fn_, 'rb') as fp_:

Though, you probably also have to modify the code when writing the file too.
That would be in salt/modules/cp.py in the recv() function:

- salt.utils.fopen(final, 'w+').write(data)
+ salt.utils.fopen(final, 'w+b').write(data)

Todd

> --
> You received this message because you are subscribed to the Google Groups "Salt-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Todd Stansell]

If this works for you, we should open an issue on github and track this change
to get it fixed permanently!

Todd

[Ramakanta Sahoo]

Thanks Todd for responding. but it seems not to be working.

I changed in all the files mentioned below and restarted the master and did a transfer again but still I see the md5sum is different between source and destination and the file is corrupt at minion.

/usr/lib/python2.6/site-packages/salt/cli/cp.py

/usr/lib/python2.6/site-packages/salt/modules/cp.py

Not sure what to do next now... really confused. 

Thanks,

Ramakanta

[Todd Stansell]

I think you'll have to make the same changes on the minion and restart it too, if you haven't tried that.

Todd

[themalkolm]

Oh, that explains why salt corrupts msi files whenever I try to transfer them using salt-cp.

[Ramakanta Sahoo]

I dont find any cp.py in C:\salt folder itself. is there any other place I shall look for the same in minion?

Thanks

[C. R. Oldham]

It sounds like more than one of you have encountered this, so would you mind creating an issue in GitHub?  This would be a high severity one, so we'd want to get someone looking at it ASAP.

--cro

C. R. Oldham, Platform Engineer, SaltStack

c...@saltstack.com / 801-564-4673

[Ramakanta Sahoo]

Issue created: https://github.com/saltstack/salt/issues/23566

Thanks