Bazel and pycrypto (how to add an extension rule that supplies both .py and .so files?)

[Wade]

TL;DR

I'd like to build a python library against pycrypto using Bazel.  I've been reading up on workspace rules and external dependencies, and have surmounted a variety of issues, but have one last snag I have not yet resolved.  I'd appreciate any insights.  

For those preferring to see and play with actual code rather than just read about it, you can see the issue on your own machine in 60 seconds by download the attached tar bundle, saving it to crypt.tgz, and executing the following commands: 

% tar xzvf crypt.tgz          # crypt.tgz is attached in this message

% cd crypt

% bazel build :crypt > log    # ignore warnings ... they don't indicate failure

% bazel run :crypt            # fails to import _AES  (because .so files aren't being linked)

The final step fails to import Crypto.Cipher._AES (which is provided by a .so file) when loading Crypto.Cipher.AES (a .py file).  I've identified the reason (the relevant runfiles directory does not contain symlinks for .so files, but does for .py files), but I do not know how to address the issue.

Motivating Questions:

1. Is there an example WORKSPACE and associated BUILD file for linking a python library against pycrypto?  Google searches turn up nothing relevant.  The solution presented here may be Just Plain Wrong.
   
   
2. If an external source is not BUILD aware, but does have a setup.py (and 'python setup.py build' and 'python setup.py install' work as expected), isn't there some relatively automated way of providing the relevant output files?  I'm imagining a 'new_http_python_archive' rule that does the right thing for such archives.
   
   
3. Suppose a py_library is defined as:
   
      py_library(
        name = "main",
        srcs = [':setup'],
        visibility = ["//visibility:public"],
        imports = ['pycrypto_2_6_1/lib/python'])
   
   and the :setup target is a genrule that outputs a collection of .py and .so files.  How to ensure that the .so files are properly symlinked in the "main" target? (in the example below, .py files are properly symlinked, but .so files are not).
   
   
4. Is there a better way to specify output files than an explicit enumeration? Using glob() doesn't work when the files to be outputted do not yet exist.  See my example BUILD.pycypto below for an example.

Example Code

For others experiencing similar issues, or those inclined to help diagnose the issue, the following contains a small example.  Suppose I have the following code:

from Crypto.Cipher import AES

from Crypto.Cipher import ARC2

def CryptoEx():

  print 'Here with ' + AES.__name__

  print 'Here with ' + ARC2.__name__

if __name__ == '__main__':

  CryptoEx()

A BUILD file for the above code:

py_library(

  name = "cryptlib",

  srcs = ['crypt.py'],

  deps = ['@pycrypto//:main'],

  visibility = ['//visibility:public'],

)

py_binary(

  name = "crypt",

  srcs = ['crypt.py'],

  deps = [':cryptlib'],

  visibility = ['//visibility:public'],

)

Note that in the above I've specified that cryptlib depends on @pycrpto//:main.  I've added the following to the WORKSPACE file:

new_http_archive(

  name = 'pycrypto',

  url = 'https://pypi.python.org/packages/60/db/645aa9af249f059cc3a368b118de33889219e0362141e75d4eaf6f80f163/pycrypto-2.6.1.tar.gz',

  sha256 = 'f2ce1e989b272cfcb677616763e0a2e7ec659effa67a88aa92b3a65528f60a3c',

  build_file = 'BUILD.pycrypto',

)

which references BUILD.pycrypto for specifying the pycrypto targets. Here's my BUILD.pycrypto.

py_library(

  name = "main",

  srcs = [':setup'],

  visibility = ["//visibility:public"],

  imports = ['pycrypto_2_6_1/lib/python'],

)

genrule(

  name = "setup",

  outs = [

    'pycrypto_2_6_1/lib/python/Crypto/__init__.py',

    'pycrypto_2_6_1/lib/python/Crypto/Cipher/__init__.py', 

    'pycrypto_2_6_1/lib/python/Crypto/Cipher/_AES.so',

    'pycrypto_2_6_1/lib/python/Crypto/Cipher/AES.py',

    'pycrypto_2_6_1/lib/python/Crypto/Cipher/_ARC2.so',

    'pycrypto_2_6_1/lib/python/Crypto/Cipher/ARC2.py',

    ... lots of .py and .so files snipped for readability ...

    'pycrypto_2_6_1/lib/python/Crypto/Util/__init__.py',

    'pycrypto_2_6_1/lib/python/Crypto/Util/_counter.so',

    'pycrypto_2_6_1/lib/python/Crypto/Util/_number_new.py',

    ... more files snipped for readability

  ],

  cmd = """

    destdir="$$(pwd)/$(@D)";

    cd external/pycrypto/pycrypto-2.6.1;

    python setup.py build;

    python setup.py install --home="$${destdir}/pycrypto_2_6_1";

  """,

)  

From the directory containing all of these files (WORKSPACE, BUILD, BUILD.pycrypto), we can build the binary ... it sucessfully downloads and sets up pycrypto:

% bazel build :crypt > log

bazel build :crypt > log

____Loading package: 

____Loading package: @local_jdk//

____Loading package: @local_config_cc//

____Loading complete.  Analyzing...

____Found 1 target...

blaze: Entering directory `/private/var/tmp/_bazel_wmh/259bfb29b85d73dae7230bfefd9cc335/execroot/crypt/'

____[0 / 1] BazelWorkspaceStatusAction stable-status.txt

____From Executing genrule @pycrypto//:setup:

warning: GMP or MPIR library not found; Not building Crypto.PublicKey._fastmath.

warning: GMP or MPIR library not found; Not building Crypto.PublicKey._fastmath.

blaze: Leaving directory `/private/var/tmp/_bazel_wmh/259bfb29b85d73dae7230bfefd9cc335/execroot/crypt/'

Target //:crypt up-to-date:

  bazel-bin/crypt

____Elapsed time: 1.886s, Critical Path: 0.31s

We can truly verify that the downloaded and compiled pycrypto code is working by doing the following:

% pushd $(bazel info output_path)/local-fastbuild/genfiles/external/pycrypto/pycrypto_2_6_1/lib/python

% python -c 'import Crypto.Cipher.AES; print Crypto.Cipher.AES'

<module 'Crypto.Cipher.AES' from 'Crypto/Cipher/AES.pyc'>

% popd 

However, when I attempt to run the binary, I encounter an error:

% bazel run :crypt

____Loading...

____Found 1 target...

blaze: Entering directory `/private/var/tmp/_bazel_wmh/259bfb29b85d73dae7230bfefd9cc335/execroot/crypt/'

blaze: Leaving directory `/private/var/tmp/_bazel_wmh/259bfb29b85d73dae7230bfefd9cc335/execroot/crypt/'

Target //:crypt up-to-date:

  bazel-bin/crypt

____Elapsed time: 0.070s, Critical Path: 0.00s

____Running command line: bazel-bin/crypt

Traceback (most recent call last):

  File "/private/var/tmp/_bazel_wmh/259bfb29b85d73dae7230bfefd9cc335/execroot/crypt/bazel-out/local-fastbuild/bin/crypt.runfiles/__main__/crypt.py", line 1, in <module>

    from Crypto.Cipher import AES

  File "/private/var/tmp/_bazel_wmh/259bfb29b85d73dae7230bfefd9cc335/execroot/crypt/bazel-out/local-fastbuild/bin/crypt.runfiles/pycrypto/pycrypto_2_6_1/lib/python/Crypto/Cipher/AES.py", line 50, in <module>

    from Crypto.Cipher import _AES

ImportError: cannot import name _AES

ERROR: Non-zero return code '1' from command: Process exited with status 1.

Note that the above is referencing 

     $(bazel info output_path)/local-fastbuild/bin/crypt.runfiles/pycrypto/pycrypto_2_6_1/lib/python

whereas we had verified things were working in

     $(bazel info output_path)/local-fastbuild/genfiles/external/pycrypto/pycrypto_2_6_1/lib/python

These two directories diff in that the former does not have symlinks for .so files, while the latter does:

% ls -F $(bazel info output_path)/local-fastbuild/genfiles/external/pycrypto/pycrypto_2_6_1/lib/python/Crypto/Cipher

AES.py* CAST.py* PKCS1_v1_5.py* _CAST.so*

AES.pyc CAST.pyc PKCS1_v1_5.pyc _DES.so*

ARC2.py* DES.py* XOR.py* _DES3.so*

ARC2.pyc DES.pyc XOR.pyc _XOR.so*

ARC4.py* DES3.py* _AES.so* __init__.py*

ARC4.pyc DES3.pyc _ARC2.so* __init__.pyc

Blowfish.py* PKCS1_OAEP.py* _ARC4.so* blockalgo.py*

Blowfish.pyc PKCS1_OAEP.pyc _Blowfish.so* blockalgo.pyc

 

% ls $(bazel info output_path)/local-fastbuild/bin/crypt.runfiles/pycrypto/pycrypto_2_6_1/lib/python/Crypto/Cipher

AES.py@ Blowfish.py@ PKCS1_OAEP.py@ __init__.pyc

AES.pyc CAST.py@ PKCS1_v1_5.py@ blockalgo.py@

ARC2.py@ DES.py@ XOR.py@ blockalgo.pyc

ARC4.py@ DES3.py@ __init__.py@

Am I correct that when a genrule is used in the 'srcs' attribute of a 'py_library' rule, only '.py' files are heeded? How do we get .so files linked in?

[mbab...@gmail.com]

AFAIK, the only way to bundle .so files into py_library targets is to include them in the 'data' attribute. You'll want to break out the .so files and .py files into separate lists.

That said, you lose a lot of the power of bazel by compiling your .so's externally through setup.py -- if you run into trouble (mismatched compiler options, etc.), consider writing proper bazel rules for the shared libraries. Use 'cc_binary', with 'linkshared' and 'linkstatic' both turned on.

[robe...@gmail.com]

I'm also trying to build python extensions with bazel.

The problem with using cc_binary is that it doesn't correctly set up the include paths, compiler options, etc. If setup.py is discouraged, is there a better way to get all this information? Can you somehow "depend" on the CPython environment? (It'd be really handy if there was a py_extension rule that took .c files and produced the correct .so files, but I can't find anything like that.