Using a .pex as an AWS Lambda?

[John Ioannidis]

Before I go down a rathole... has anyone here experimented (even better, succeeded) building a .pex and using it as the "executable" in an AWS Lambda function? For obvious reasons, I would much rather have a workflow that builds a pex with pants and moves it to the appropriate S3 bucket, than manually do pretty much the same work with virtualenv, installing local libraries, and zipping everything.

Good idea? Bad idea? Should I attempt it anyway?

Thanks!

/ji

PS: 

[Yi Cheng]

It's a thing. As suggested by Kris in slack before, you can use https://github.com/wickman/lambdex.

I tried in a simplest use case, and it worked for me.

[Björn Andersson]

Can confirm that Lambdex works great. I have ~12 Lambdas that are deployed from PEX files. Amazon has some weirdness when it comes to what it accepts as a zip-file, though. So I ended up adding a post-build step for my Lambda files that unzips the pex and then rezips it, so it won't work to ./file.pex anymore, but since I'm running on Lambda it doesn't matter much to me.

This is the bash function I use to rezip my pex file in case it's helpful. I also realized that it contains a strip step to shrink the file size on your compiled dependencies. It mostly just works and the few cases it doesn't I suggest you add the ignore like I have with psycopg2. :)

function rezip-pex() {

    local artifact_prefix="$1"

    local tempdir=$(mktemp -d)

    local dir=$(pwd)

    unzip -qq "dist/${artifact_prefix}.pex" -d ${tempdir}

    exit_code=$?

    if [ "${exit_code}" -gt 2 ] ; then

        echo-failure "Failed to unzip pex file"

        return ${exit_code}

    fi

    strip_bin=strip

    [ "$(uname)" = 'Darwin' ] && strip_bin=gstrip

    (cd "${tempdir}" &&

        find .deps -name '*.so'| grep -v psycopg2 | xargs ${strip_bin} ;  # Shrink dependency size by stripping them first

        zip -rqq "${dir}/dist/${artifact_prefix}.pex.rezip" .) &&

        mv "dist/${artifact_prefix}.pex.rezip" "dist/${artifact_prefix}.pex" &&

        rm -rf "${tempdir}"

        echo-success "Rezipped ${artifact_prefix}"

}

/Björn

--

Björn Andersson

Senior Software Engineer

Spire Global Singapore Pte Limited

19 China Street #02-01, Far East Square Singapore 049561

[Brian Wickman]

Interesting. I've definitely had the experience of lambda doing odd things like hiding files w/o u+r permissions.  Do you know what zip is doing differently than zipfile?  Happy to take lambdex PRs and issues.  Glad that it's being used by someone!

[John Sirois]

On Thu, Aug 16, 2018 at 8:57 AM 'Brian Wickman' via Pants Developers <pants...@googlegroups.com> wrote:

Interesting. I've definitely had the experience of lambda doing odd things like hiding files w/o u+r permissions.  Do you know what zip is doing differently than zipfile?  Happy to take lambdex PRs and issues.  Glad that it's being used by someone!

I assume its just the shebang header. Once he unzips/rezips that is nuked.

[Björn Andersson]

On Thu, Aug 16, 2018 at 11:03 PM, John Sirois <john....@gmail.com> wrote:

On Thu, Aug 16, 2018 at 8:57 AM 'Brian Wickman' via Pants Developers <pants...@googlegroups.com> wrote:

Interesting. I've definitely had the experience of lambda doing odd things like hiding files w/o u+r permissions.  Do you know what zip is doing differently than zipfile?  Happy to take lambdex PRs and issues.  Glad that it's being used by someone!

I assume its just the shebang header. Once he unzips/rezips that is nuked.

That was my expectation as well so my very first fix to this problem was to use sed to lop off the first line, but alas it also stopped working. 

To be honest I don't know the intricasies of zip files so I can't elaborate on it more than that I tried that. AWS support wasn't very helpful when I raised a ticket because they felt that if it worked after rezipping then I should just do that.

One thing I did notice is that the unzip utility do exit with a non-zeo exit code when unpacking a pex file despite being able to unpack it properly (which is why my bash function checks for exit code greater than 2, because that's the level of error when it's not expected for me).

[Mateo Rodriguez]

A new hire at Foursquare just integrated Lambdex into our codebase as well @OniOni