can't build pex that has python version != 2.7 with pants 1.8.0rc0

[Jesse Hersch]

Hi again,

I am trying out 1.8.0rc0 and am hitting a problem with the wrong python version being used in the pyprep.sources.cache step. Here's my environment:

1. target python version is 3.6.3
2. ubuntu 17.04

Here's the pants.ini:

[GLOBAL]

pants_version: 1.8.0rc0

pythonpath: ['%(buildroot)s/plugins']

backend_packages: +['pycharm', 'concourse', 'pants.backend.docgen', 'cython']

plugins: [

    'pantsbuild.pants.contrib.node==%(pants_version)s'

  ]

[python-setup]

interpreter_constraints: CPython>=3.6.1

[pytest]

requirements: pytest==3.0.6

timeout_requirements: pytest-timeout==1.2.0

cov_requirements: pytest-cov==2.4.0

[test.pytest]

fast: false

[cache]

read_from: ['/tmp/pantscache']

write_to: ['/tmp/pantscache']

Here's the BUILD file:

python_library(

  name='impl',

  sources=rglobs('*.py', exclude=[rglobs('tests/*.py')]),

  dependencies=[

    '3rdparty:requests',

    '3rdparty:pendulum',

  ],

)

python_binary(

  name='tf-variable-updater',

  source='main.py',

  dependencies=[':impl'],

)

python_tests(

  name="test",

  sources=globs('tests/*.py'),

  dependencies=[':impl'],

)

And here's the result:

$ ./pants test tools/tf-variable-updater:

14:47:12 00:00 [main]

               (To run a reporting server: ./pants server)

14:47:12 00:00   [setup]

14:47:12 00:00     [parse]

               Executing tasks in goals: bootstrap -> imports -> unpack-jars -> deferred-sources -> gen -> jvm-platform-validate -> pyprep -> resolve -> compile -> resources -> test

14:47:12 00:00   [bootstrap]

14:47:12 00:00     [substitute-aliased-targets]

14:47:12 00:00     [jar-dependency-management]

14:47:12 00:00     [bootstrap-jvm-tools]

14:47:12 00:00     [provide-tools-jar]

14:47:12 00:00   [imports]

14:47:13 00:01     [ivy-imports]

14:47:13 00:01   [unpack-jars]

14:47:13 00:01     [unpack-jars]

14:47:13 00:01   [deferred-sources]

14:47:13 00:01     [deferred-sources]

14:47:13 00:01   [gen]

14:47:13 00:01     [antlr-java]

14:47:13 00:01     [antlr-py]

14:47:13 00:01     [jaxb]

14:47:13 00:01     [protoc]

14:47:13 00:01     [ragel]

14:47:13 00:01     [thrift-java]

14:47:13 00:01     [thrift-py]

14:47:13 00:01     [wire]

14:47:13 00:01   [jvm-platform-validate]

14:47:13 00:01     [jvm-platform-validate]

14:47:13 00:01   [pyprep]

14:47:13 00:01     [interpreter]

14:47:13 00:01     [build-local-dists]

14:47:13 00:01     [requirements]

14:47:13 00:01     [sources]

14:47:13 00:01       [cache]    

                   No cached artifacts for 4 targets.

                   Invalidated 4 targets.

               Waiting for background workers to finish.

14:47:13 00:01   [complete]

               FAILURE

Exception caught: (<class 'pex.compiler.CompilationFailure'>)

Exception message: encountered NonZeroExit("received exit code 1 during execution of `['/usr/bin/python2.7', '/tmp/tmp_VVqtO']` while trying to execute `['/usr/bin/python2.7', '/tmp/tmp_VVqtO']`",) during bytecode compilation.

stderr was:

Encountered 5 errors compiling 59 files:

  copy-env-versions.py:   File "copy-env-versions.py", line 13

    "ops-test": f"{TERRAFORM_BASE_PATH}/ops-test/variables.tf",

                                                             ^

SyntaxError: invalid syntax

  tests/test_hclupdater.py:   File "tests/test_hclupdater.py", line 21

    HCL_STRING = f"""

{STANZA1}

{STANZA2}

"""

                     

         

         

  ^

SyntaxError: invalid syntax

  main.py:   File "main.py", line 94

    logging.info(f"Updating services: {', '.join(sorted(services))}")

                                                                   ^

SyntaxError: invalid syntax

  hclupdater.py:   File "hclupdater.py", line 42

    logger.info(f"Reading {self.filename}")

                                         ^

SyntaxError: invalid syntax

  version_fetcher.py:   File "version_fetcher.py", line 30

    repository = f"gemini/{service}"

                                   ^

SyntaxError: invalid syntax

Obviously the wrong python version is being used. It should not be using python2.7 for bytecode compilation since interpreter_constraints: CPython>=3.6.1. 

I get same fail for ./pants binary as the test verb above.

Jesse

[Jesse Hersch]

looks like this fixes it, in the pants.ini file:

interpreter_constraints: ['CPython>=3.6.1']

current behavior violates the "priniple of least surprise" in my book :)

Jesse

[Benjy Weinberger]

In the pants options system, setting a list-valued option to a scalar appends that scalar to the existing value of the list, which in this case is the default value of ['CPython>=2.7,<3'].

So in this case you ended up with ['CPython>=2.7,<3', 'CPython>=3.6.1'], and the lowest version that satisfies these ORed constraints is indeed 2.7...

There were various reasons why this appending behavior was desirable. Changing it would break... a lot of things.  But since  this was surprising, we should better communicate how list-valued options work. 

[Jesse Hersch]

fyi I tried this while debugging it and it returns nothing: ./pants python-setup --help

Isn't that how to get help on a config option?  If so maybe add some help for python-setup and put the answer in there?

[John Sirois]

On Wed, Jun 20, 2018 at 4:29 PM Jesse Hersch <jesse...@fastmail.fm> wrote:

fyi I tried this while debugging it and it returns nothing: ./pants python-setup --help

Isn't that how to get help on a config option?  If so maybe add some help for python-setup and put the answer in there?

It is, but we split options into two classes, normal - which --help addresses, and advanced, which --help-advanced addresses.

[Jesse Hersch]

lol ok. never heard of --help-advanced. clearly I need advanced help. :)  I got on the --help train from this page https://www.pantsbuild.org/options.html under the "troubleshooting config files" section.

I have 1.8.0 working except for one thing, which is a plugin we wrote for compiling cython. Is there support for compiling cython in 1.8? maybe we don't need that plugin anymore.  

Anyway the plugin fails when calling create_sources_field() in target.py:

Exception caught: (<class 'pants.base.exceptions.TargetDefinitionException'>)

Exception message: Invalid target CompileCython(address not yet set): Expected 'sources' to be a glob, an address or a list, but was <class 'pants.engine.legacy.structs.RGlobs'>

Any idea how to turn an RGlobs instance into a FilesetWithSpec? it used to be one of those in 1.3.

[Jesse Hersch]

We figured this one out in the meantime btw.