setup.py depending on another wheel

[Rudi Chiarito]

Hi,

I'm using a variant of https://github.com/bazelbuild/rules_python/issues/96#issuecomment-395818751 to load dependencies on demand. 

I'm currently seeing problems with a specific wheel. Let's call it bar. Not only it depends on foo (which gets downloaded first), its setup.py actually calls code from foo, so the wheel will fail during `python setup.py egg_info`, with an error similar to

Collecting bar==X (from -r /blablabla/requirements.bazel.txt (line 23))

  Using cached https://files.pythonhosted.org/packages/BLA/BLA/blabla.tar.gz

    Complete output from command python setup.py egg_info:

    Traceback (most recent call last):

      File "<string>", line 1, in <module>

      File "/tmp/XYZ/bar/setup.py", line N, in <module>

        import foo

    ImportError: No module named foo

I changed the configuration temporarily to call pip_import() and I still get the same error, so it seems like it's more like a fundamental issue with how dependencies are brought in.

Is this an use case that has been considered before? I know it's ugly, but that's what you have to deal with in the mystical and magic world of Python packages.

One way of dealing with this is to bake foo into my base Docker image for Bazel, but of course that has a number of tricky ramifications. Another way, whose details I haven't been able to figure yet, would be to do the work in two stages: the first installs foo, while the second installs bar (and has foo in PYTHON_PATH).

Thanks,

Rudi

[Jon Brandvein]

Hi Rudy,

Pip's model mixes fetching and building, which kind of clashes with Bazel's model of having a separate WORKSPACE loading stage. I don't think we have a standard answer for all variants of this use case yet.

But in this case, IIUC you're saying that bar would've been able to find foo if only its setup had succeeded, and the problem is foo just isn't made available as early as the setup phase. While I only have a high-level view of this ruleset, I'm not sure I see any fundamental reason why that can't be made to work. The two-stage approach you mention sounds plausible, though maybe pip_import would have to be modified to take in requirement() deps.

Out of curiosity, what does pip do outside of Bazel when you try to install foo and bar on the same command line?

In any case, could you file a github issue for this?

[Rudi Chiarito]

Hi Jon,

Yes, PyPI is the new CPAN, the Wild West where people get to do the craziest things. :-)

If I run `pip install foo bar` outside of Bazel, it fails the same way.

If I run `pip install foo`, then run `pip install bar`, everything works.

If I bake a Docker image for Bazel with foo pre-installed system-wide, Bazel has no problems, although you can obviously argue that that's a bad solution. Given that the Docker image is locked down, though, it's not as bad as other approaches.

On a hunch, I tried `pip install bar` in a clean virtualenv. It doesn't declare any dependencies on foo... I had never considered that the package could be broken at such a level (it's relatively popular, but looking at it more closely, it seems semi-abandoned). :-( This is not really Bazel's fault. I'll file an issue, but I also understand if it won't be considered high priority.

Thanks a lot!

--
You received this message because you are subscribed to the Google Groups "Bazel/Python Special Interest Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-sig-pyth...@googlegroups.com.
To post to this group, send email to bazel-si...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-sig-python/2f3e5499-a136-42ed-97d3-0ae1f4649d18%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Rudi Chiarito — Infrastructure — Clarifai, Inc.

"Trust me, I know what I'm doing." (Sledge Hammer!)

[Jon Brandvein]

Thanks for experimenting with bar's dependencies and discovering it's an underlying issue.

If it turns out there's some packages that by design can only ever be installed using separate pip install commands, then maybe we could consider that a future improvement to rules_python. Hopefully pip's more flexible than that.