Question: how to extend Pants without modifying the sources

[Sebastian Kreft]

Hi Pants team, I'm evaluating the migration from BUCK to Pants, as your Python support is much better. However, there are a couple of things I still don't know how to do.

I'm trying to define a set of helper functions and to define custom target types. The latter is the most important feature we need as we would need for example to compile C extensions, cython, package an app with a different format (like for appengine), compile/minify javascript among other tasks.

In BUCK there are two concepts that make extending it quite easy. These are genrule which allows you to define a new target rule that calls an executable and include_defs which allows to easily include those macro definitions. Additionally, in the configuration one can specify which macro files will be included before parsing each BUILD file.

Ideally I'm looking for a solution which does not require to modify Pants' sources.

[Patrick Lawson]

Interesting, it looks like maybe this hasn't made it to the documentation yet--yes, we do have a notion of plugins in pants, and while I'm not sure if we have the same terminology here, I'm pretty sure it's easy to do what you're talking about.

Namely, in your pants.ini you can add an entry like this:

```

[backends]

packages: ['yourpackage.foo.bar']

```

At startup, pants will attempt to import `'yourpackage.foo.bar.register` and from there, call a couple of entry points that allow you to do custom target and task registration.  Here is an example of a registration hook used by the backend:  https://github.com/pantsbuild/pants/blob/master/src/python/pants/backend/python/register.py

Note that the module pants tries to import needs to be importable with that name from sys.path, meaning you might need to modify PYTHONPATH in a wrapper script around pants itself.  In the near future we will have a formal plugin system that allows you to distribute these plugins as regular python packages rather than hacking PYTHONPATH.

Let me know if anything was unclear or if you have further questions!

[Joe Smith]

Any work you have that makes it easier to compile C extensions will be much appreciated by the broader python-using pants community :)

[Patrick Lawson]

Indeed!  We're actually about to start an internal project that's going to involve integrating a couple C/C++/Go extensions into pants itself (or maybe just a plugin).  There will probably be a lot of opportunity for collaboration, so please feel free to ping the list a lot and publish branches/CRs.

[Sebastian Kreft]

@Patrick, thanks for the backend reference. However, I think that there are still missing many pieces (particularly documentation and examples).

Writing a backend feels pretty much like changing the sources of Pants, as one needs to know all the internal in order to successfully achieve something. For example: writing a simple function (not context aware) requires that one wraps that function in a wrapper that discard such context. Furthermore, from within a backend one needs to call the original pants 'function' definition and not the exposed ones.

In addition to that, after digging into the code it looks like implementing a new target is not an easy task and there are many concepts and boilerplate involved. An example of how to convert:

$executable $SRCS > $OUTFILE

into a target would be much appreciated.

--
Sebastian Kreft

[Patrick Lawson]

Yes, pants doesn't really have a notion of an extremely light weight (and ad hoc) way to run a shell command.  Also note that targets specified in BUILD files are encapsulating input (e.g. sources and configuration) and dependencies: they do not say anything at all about actions to take.  This is defined by task/goal registration and how the task(s) choose to operate on the build graph (where targets are nodes and their dependencies are directed edges).  It would be feasible to make a `shell_command` target that's operated on by a thin task that just does what the target says.

It sounds like this is what you want, and that John is also in favor of supporting this?  If so, we could walk you through adding support for this (we love contributions), or one of us could knock it out otherwise (assuming everyone is on board with the design).

Benjy, John, Eric: what are your thoughts on this?

[John Sirois]

Here's an internal feature request we've had for a while and not gotten to yet:

> Syntax suggestions are welcome, we'll miss the mark without them.

I think the first step is to get everyone on the same page by answering some basic questions:

How do you run multiple lines of shell script?

How do you handle errors?

How do you interpolate Python data?

How do you make sure the interpolated Python data is safe given it might contain things like "; rm -rf /"?

Is it possible to use shells other than bash?

Is there a way to configure dependencies so that the shell commands don't have to get re-run in cases where it's unnecessary? For instance, should the shell command create a file that can be stat-ed and compared to other files? Do you need to configure the source files used by the shell command?

Given the above, what's the syntax?

Here's my first pass at some answers:

> How do you run multiple lines of shell script?

You pass it in a triple quoted string in the BUILD file. The triple quoted string gets written to a temporary shell script and executed.

> How do you handle errors?

The shell script should be run with -e by default.

> How do you interpolate Python data?

Via the normal Python mechanism.

> How do you make sure the interpolated Python data is safe given it might contain things like "; rm -rf /"?

There should be a helper function exposed by default and well documented. See pipes.quote or shlex.quote (http://docs.python.org/2/library/pipes.html).

> Is it possible to use shells other than bash?

Yes. One of the keyword arguments should be shebang="#!/usr/bin/env bash -e". This lets the user configure how the script should be run.

> Is there a way to configure dependencies so that the shell commands don't have to get re-run in cases where it's unnecessary? For instance, should the shell command create a file that can be stat-ed and compared to other files? Do you need to configure the source files used by the shell command?

If the user configures sources=some_list_of_files and target=some_file, the system can use stat to avoid re-executing the shell script in some cases (the same way make would). If the user doesn't configure those things, the shell script will get re-evaluated every time the pants target is used as a dependency (the way make works if you don't specify a target file).

> Given the above, what's the syntax?

How about:

shell_script(name = 'some_name',
  script = """
    for i in $( find . -name \*.nnn ); do
      nnn_processor < $i > $i.mmm
    done
    mmm_processor $( find . -name \*.mmm ) -o some_file
  """,

  sources = rglobs('*.nnn'),

  # If this is left to the default of None, perhaps the system can touch
  # a temporary file to know whether that file is newer than any *.nnn file.
  target = "some_file",

  # shell_scripts can depend on each other.
  # dependencies = [],

  # This is the default.
  # shebang="#!/usr/bin/env bash -e"
)

--
John Sirois
303-512-3301

[Patrick Lawson]

What John is proposing seems reasonable to me.

[Eric Ayers]

+1