Clean example of using S3 to retrieve objects for SaltStack

[ErikJ]

Hi salt user community,

I've been going around and around trying to find a simple way to download a file from an S3 bucket.  If I've gone down the wrong path, please let me know.

The basics:

1) I have a company's proprietary executable/bits stored in a private S3 bucket.   I can see it and download it just fine using aws-cli with my keys.

2) I want to download it, and run it on a masterless salt minion for now. 

I've read the documentation[1] and tried a number of simple examples, as well as read other reports here [2] and other github issues [3], and am admittedly baffled at what I'd assumed would be a straightforward workflow.  The conclusion I've come to is that the only way to get this is to install the aws-cli and download it (this is for an environment where less is more.)

Something like the example on s3 modules, which I would turn into a state looks like:

   salt-call --local s3.get eriktestsalt my-test-script.sh return_bin=True local_file=/tmp/my-test-script.sh keyid=[my-keyid] key=[my-secret-key] verify_ssl=False location=us-east-1

only fills the /tmp/my-test-script.sh with "SignatureDoesNotMatch" XML output ( AWS v4 signature error ?)   I've managed variations that produce 404's and 403's and 'Key Not Found' (always written to my intended output file - ouch, not a nice thing to do) and read my debug-level logging output.

Turning this into an s3.put or s3.head - no such luck either, and usually less output.

 

Going back to "The basics" - what would be an example workflow for this use case?  I see illusions to "s3fs" and "just use aws-cli" and if that's preferred, than a google search of "saltstack s3" should probably go somewhere besides the s3 modules page?

 

Thanks in advance!

Erik

[1] https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.s3.html

[2] https://groups.google.com/forum/#!searchin/salt-users/s3$20get%7Csort:date/salt-users/_YodxI2THVM/HLSrbT3dyFkJ

[3] https://github.com/saltstack/salt/issues/23007

[ErikJ]

I think I've found part of the problem but not all of it - every call returns SignatureDoesNotMatch still, but I'm getting to see where things might be getting muddled when the AWS v4 signature is built.

Stripping down the request to this (which should return a directory listing in the salt docs): 

salt-call s3.get ${_MY_BUCKET} keyid=${MY_AWS_KEY} key=${MY_SECRET_KEY} location=us-east-1

And reading the debugging output - the URL is formed as:

2016-06-01 16:53:21,119 [salt.utils.s3    ][DEBUG   ][16964] S3 Request: https://${_MY_BUCKET}.amazonaws.com/None?

(emphasis mine)

Looking at the code - the debug line comes from in git:

log.debug('S3 Request: {0}'.format(requesturl))

Tracing back requesturl is made up of endpoint and path - and it's clearly the "path" variable we care about.

Now things get weird - unlike every other s3 util query entry, path isn't handled the same, and I don't know why - 

def query(key, keyid, method='GET', params=None, headers=None,
         requesturl=None, return_url=False, bucket=None, service_url=None,
          path='', return_bin=False, action=None, local_file=None,
          verify_ssl=True, full_headers=False, kms_keyid=None,
          location=None, role_arn=None, chunk_size=16384):

Either way, method.s3 sets "path=None" if there isn't one, which becomes a string None, which becomes URL ending in None

Overloading with path=foo

2016-06-01 17:18:42,732 [salt.utils.s3    ][DEBUG   ][17230] S3 Request: https://${_MY_BUCKET}.amazonaws.com/foo?

This isn't everything - I created a valid path and put a file in it:

2016-06-01 17:42:38,347 [salt.utils.s3    ][DEBUG   ][17413] S3 Request: https://{_MY_BUCKET}.s3.amazonaws.com/saltMystery?

and it fixed the path part right, but still got the SignatureDoesNotMatch error.

So there seems to be some less-than-precise checking going on in the variables being provided to build the aws sig4 generator call - but I'm out of time looking at this for today.

Anyone else got a working example of modules.s3 - doing at least a directory listing from "get" - in the meantime?

Thanks,

Erik

[ErikJ]

FYI for community that's hoping for an answer - I gave up and downloaded the AWS cli module.

I recommend against using the modules.s3; I'd fight a little harder for trying to figure out what's going on in the failed AWS v4 encryption calls from salt.modules.s3  but it doesn't seem that important to the community to acknowledge.

Erik

On Tuesday, May 31, 2016 at 7:03:56 PM UTC-6, ErikJ wrote:

[Florian Ermisch]

Hi Erik,

sorry to hear, I who know how frustrating
broken modules can be.
I guess you've already checked if your
underlying python modules are up-to-date
and tried to put the latest s3 module in
your fileserver's _modules?

Regards, Florian

> <https://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.s3.html>[1]

> and
> > tried a number of simple examples, as well as read other reports
> here
> >

> <https://groups.google.com/forum/#!searchin/salt-users/s3$20get%7Csort:date/salt-users/_YodxI2THVM/HLSrbT3dyFkJ>
> [2] and
> > other github issues <https://github.com/saltstack/salt/issues/23007>

> [3],
> > and am admittedly baffled at what I'd assumed would be a
> straightforward
> > workflow. The conclusion I've come to is that the only way to get
> this is
> > to install the aws-cli and download it (this is for an environment
> where
> > less is more.)
> >
> > Something like the example on s3 modules, which I would turn into a
> state
> > looks like:
> >

> > *salt-call --local s3.get eriktestsalt my-test-script.sh
> > return_bin=True local_file=/tmp/my-test-script.sh keyid=[my-keyid]
> > key=[my-secret-key] verify_ssl=False location=us-east-1*

> >
> > only fills the /tmp/my-test-script.sh with "SignatureDoesNotMatch"
> XML
> > output ( AWS v4 signature error ?) I've managed variations that
> produce
> > 404's and 403's and 'Key Not Found' (always written to my intended
> output
> > file - ouch, not a nice thing to do) and read my debug-level logging
> output.
> >

> > Turning this into an *s3.put *or* s3.head* - no such luck either,

[tim.h...@gmail.com]

Hi Erik,

Sorry no-one gave you a timely response. I've been using the s3.get command as in your original post with instance profile creds and (at least with the latest 2015.8.10 release) haven't had any issues. I know that between 2015.8.4->2015.8.7 s3.get was broken with instance creds, but you didn't tell us the version you were using. Note that I haven't tested 2016.3.0 so I can't comment on that version.

cheers,
Tim