pylint and Bazel

1,067 views
Skip to first unread message

Hyrum Wright

unread,
Nov 15, 2017, 9:51:03 PM11/15/17
to bazel-discuss
I'd like to integrate pylint with my Bazel workflow.

First, I'm interested to hear if anybody has already done so, and if so, how they did it.

If I'm the pioneer in this space, I'm interested in knowing how best to run pylint, either as an analysis during the normal build, or in some separate step.  The core problem, as I see it, is that because Bazel constructs the output python tree in a sandbox, and pylint needs all of those dependencies and paths in place to function correctly, it needs Bazel to tell it where the sandbox is and what the correct PYTHONPATH to use is.  Where's the right place to get that information from Bazel?

-Hyrum

Marcel Hlopko

unread,
Nov 17, 2017, 3:14:00 AM11/17/17
to Hyrum Wright, lbe...@google.com, dsl...@google.com, bazel-discuss
Hmm I'd say bazel aspect, but let's see what +Lukács T. Berki and +Dmitry Lomov have to say.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CALmYJEUvUJ5HbH0nBoiwJLFPEiCEQOgj8WM%3DAJDg2gPW7K%3Dq4Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
--
-- 
Marcel Hlopko | Software Engineer | hlo...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891

Lukács T. Berki

unread,
Nov 17, 2017, 4:23:38 AM11/17/17
to Marcel Hlopko, Hyrum Wright, dsl...@google.com, bazel-discuss
Agreed. I'm not sure if we expose enough information about Python to Skylark to be able to express "give me all the Python source files, the interpreter and tell me what PYTHONPATH is", but if we don't, we should.

On 17 November 2017 at 09:13, 'Marcel Hlopko' via bazel-discuss <bazel-...@googlegroups.com> wrote:
Hmm I'd say bazel aspect, but let's see what +Lukács T. Berki and +Dmitry Lomov have to say.
On Thu, Nov 16, 2017 at 3:51 AM Hyrum Wright <hwr...@duolingo.com> wrote:
I'd like to integrate pylint with my Bazel workflow.

First, I'm interested to hear if anybody has already done so, and if so, how they did it.

If I'm the pioneer in this space, I'm interested in knowing how best to run pylint, either as an analysis during the normal build, or in some separate step.  The core problem, as I see it, is that because Bazel constructs the output python tree in a sandbox, and pylint needs all of those dependencies and paths in place to function correctly, it needs Bazel to tell it where the sandbox is and what the correct PYTHONPATH to use is.  Where's the right place to get that information from Bazel?

-Hyrum

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.
--
-- 
Marcel Hlopko | Software Engineer | hlo...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.
 To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAFuL9G%3DuuuU8088z2BOVuTvAK8%2B29VbC%3D-2rG93nB3bG2vy10g%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
Lukács T. Berki | Software Engineer | lbe...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891

Hyrum Wright

unread,
Nov 17, 2017, 9:37:18 AM11/17/17
to Lukács T. Berki, Marcel Hlopko, dsl...@google.com, bazel-discuss
I agree that information would be useful.  Anything I can do to get us there?

On Fri, Nov 17, 2017 at 4:23 AM, Lukács T. Berki <lbe...@google.com> wrote:
Agreed. I'm not sure if we expose enough information about Python to Skylark to be able to express "give me all the Python source files, the interpreter and tell me what PYTHONPATH is", but if we don't, we should.

Marcel Hlopko

unread,
Nov 20, 2017, 3:21:47 AM11/20/17
to Hyrum Wright, yile...@google.com, Lukács T. Berki, dsl...@google.com, bazel-discuss
I guess just trying it out and letting us know what's missing. Python rules especially are a bit neglected, and we don't have many free cycles to improve them. Maybe +Yilei Yang has more experience?

On Fri, Nov 17, 2017 at 3:37 PM Hyrum Wright <hwr...@duolingo.com> wrote:
I agree that information would be useful.  Anything I can do to get us there?
On Fri, Nov 17, 2017 at 4:23 AM, Lukács T. Berki <lbe...@google.com> wrote:
Agreed. I'm not sure if we expose enough information about Python to Skylark to be able to express "give me all the Python source files, the interpreter and tell me what PYTHONPATH is", but if we don't, we should.
On 17 November 2017 at 09:13, 'Marcel Hlopko' via bazel-discuss <bazel-...@googlegroups.com> wrote:
Hmm I'd say bazel aspect, but let's see what +Lukács T. Berki and +Dmitry Lomov have to say.
On Thu, Nov 16, 2017 at 3:51 AM Hyrum Wright <hwr...@duolingo.com> wrote:
I'd like to integrate pylint with my Bazel workflow.

First, I'm interested to hear if anybody has already done so, and if so, how they did it.

If I'm the pioneer in this space, I'm interested in knowing how best to run pylint, either as an analysis during the normal build, or in some separate step.  The core problem, as I see it, is that because Bazel constructs the output python tree in a sandbox, and pylint needs all of those dependencies and paths in place to function correctly, it needs Bazel to tell it where the sandbox is and what the correct PYTHONPATH to use is.  Where's the right place to get that information from Bazel?

-Hyrum

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
--
-- 
Marcel Hlopko | Software Engineer | hlo...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAFuL9G%3DuuuU8088z2BOVuTvAK8%2B29VbC%3D-2rG93nB3bG2vy10g%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
Lukács T. Berki | Software Engineer | lbe...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891

Lukács T. Berki

unread,
Nov 20, 2017, 6:53:27 AM11/20/17
to Marcel Hlopko, Hyrum Wright, yile...@google.com, dsl...@google.com, bazel-discuss
Yep -- if there is something you need, we can make that happen (except if proves to be too much work, but I doubt that's the case -- our Python rules are simple enough)

On 20 November 2017 at 09:21, Marcel Hlopko <hlo...@google.com> wrote:
I guess just trying it out and letting us know what's missing. Python rules especially are a bit neglected, and we don't have many free cycles to improve them. Maybe +Yilei Yang has more experience?
On Fri, Nov 17, 2017 at 3:37 PM Hyrum Wright <hwr...@duolingo.com> wrote:
I agree that information would be useful.  Anything I can do to get us there?
On Fri, Nov 17, 2017 at 4:23 AM, Lukács T. Berki <lbe...@google.com> wrote:
Agreed. I'm not sure if we expose enough information about Python to Skylark to be able to express "give me all the Python source files, the interpreter and tell me what PYTHONPATH is", but if we don't, we should.
On 17 November 2017 at 09:13, 'Marcel Hlopko' via bazel-discuss <bazel-discuss@googlegroups.com> wrote:
Hmm I'd say bazel aspect, but let's see what +Lukács T. Berki and +Dmitry Lomov have to say.
On Thu, Nov 16, 2017 at 3:51 AM Hyrum Wright <hwr...@duolingo.com> wrote:
I'd like to integrate pylint with my Bazel workflow.

First, I'm interested to hear if anybody has already done so, and if so, how they did it.

If I'm the pioneer in this space, I'm interested in knowing how best to run pylint, either as an analysis during the normal build, or in some separate step.  The core problem, as I see it, is that because Bazel constructs the output python tree in a sandbox, and pylint needs all of those dependencies and paths in place to function correctly, it needs Bazel to tell it where the sandbox is and what the correct PYTHONPATH to use is.  Where's the right place to get that information from Bazel?

-Hyrum

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.
--
-- 
Marcel Hlopko | Software Engineer | hlo...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.



--
Lukács T. Berki | Software Engineer | lbe...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891
--
-- 
Marcel Hlopko | Software Engineer | hlo...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891

Yilei Yang

unread,
Nov 20, 2017, 2:17:52 PM11/20/17
to Lukács T. Berki, Marcel Hlopko, Hyrum Wright, dsl...@google.com, bazel-discuss
I believe if you run pylint on the python file in the runfiles tree, that should be enough?

I'm not sure if you can get the path in skylark though. Another approach is to put the integration point outside bazel, since you should be able to calculate the path of the .py file in the runfiles tree, based on which rule the .py file belongs to.

On Mon, Nov 20, 2017 at 3:53 AM Lukács T. Berki <lbe...@google.com> wrote:
Yep -- if there is something you need, we can make that happen (except if proves to be too much work, but I doubt that's the case -- our Python rules are simple enough)
On 20 November 2017 at 09:21, Marcel Hlopko <hlo...@google.com> wrote:
I guess just trying it out and letting us know what's missing. Python rules especially are a bit neglected, and we don't have many free cycles to improve them. Maybe +Yilei Yang has more experience?
On Fri, Nov 17, 2017 at 3:37 PM Hyrum Wright <hwr...@duolingo.com> wrote:
I agree that information would be useful.  Anything I can do to get us there?
On Fri, Nov 17, 2017 at 4:23 AM, Lukács T. Berki <lbe...@google.com> wrote:
Agreed. I'm not sure if we expose enough information about Python to Skylark to be able to express "give me all the Python source files, the interpreter and tell me what PYTHONPATH is", but if we don't, we should.
On 17 November 2017 at 09:13, 'Marcel Hlopko' via bazel-discuss <bazel-...@googlegroups.com> wrote:
Hmm I'd say bazel aspect, but let's see what +Lukács T. Berki and +Dmitry Lomov have to say.
On Thu, Nov 16, 2017 at 3:51 AM Hyrum Wright <hwr...@duolingo.com> wrote:
I'd like to integrate pylint with my Bazel workflow.

First, I'm interested to hear if anybody has already done so, and if so, how they did it.

If I'm the pioneer in this space, I'm interested in knowing how best to run pylint, either as an analysis during the normal build, or in some separate step.  The core problem, as I see it, is that because Bazel constructs the output python tree in a sandbox, and pylint needs all of those dependencies and paths in place to function correctly, it needs Bazel to tell it where the sandbox is and what the correct PYTHONPATH to use is.  Where's the right place to get that information from Bazel?

-Hyrum

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
--
-- 
Marcel Hlopko | Software Engineer | hlo...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.



--
Lukács T. Berki | Software Engineer | lbe...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891
--
-- 
Marcel Hlopko | Software Engineer | hlo...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891

Hyrum Wright

unread,
Nov 20, 2017, 2:26:10 PM11/20/17
to Yilei Yang, Lukács T. Berki, Marcel Hlopko, dsl...@google.com, bazel-discuss
I've hacked together something like the following:

import concurrent.futures
import multiprocessing
import os
import sys

from pylint import epylint


def get_externals_paths():
    project = os.path.basename(os.getcwd())
    bazel_root = os.path.join(os.getcwd(), 'bazel-%s' % project, 'external')

    if not os.path.exists(bazel_root):
        print("Please run `bazel build ...` to resolve external dependencies")
        sys.exit(1)

    return [os.path.join(bazel_root, x) for x in os.listdir(bazel_root)]


def lint_file(filename, options):
    stdout, stderr = epylint.py_run('%s %s' % (filename, options), return_std=True)
    output = stdout.read()
    return output


def main(argv=None):
    argv = argv if argv is not None else sys.argv[1:]

    sys.path.append(os.path.join(os.getcwd(),
        'bazel-%s' % os.path.basename(os.getcwd())))
    sys.path.extend(get_externals_paths())

    rc = 0
    options = '--rcfile=%s/.pylintrc' % os.getcwd()

    with concurrent.futures.ThreadPoolExecutor(max_workers=multiprocessing.cpu_count()) as executor:
        future_to_pylint = {executor.submit(lint_file, filename, options):
                filename for filename in argv}
        for future in concurrent.futures.as_completed(future_to_pylint):
            output = future.result()
            if len(output) > 0:
                sys.stdout.write(output)
                rc += 1
    return rc


if __name__ == '__main__':
    sys.exit(main())


The intention is that this script is used with the pre-commit infrastructure to lint outside of Bazel.  It tries and calculates the location of the external dependencies using the generated bazel symlinks, but I'm open to other suggestions.

The runfiles tree might be a better location, but it isn't easy to "guess" it's location given current tooling.

On Mon, Nov 20, 2017 at 2:17 PM, Yilei Yang <yile...@google.com> wrote:
I believe if you run pylint on the python file in the runfiles tree, that should be enough?

I'm not sure if you can get the path in skylark though. Another approach is to put the integration point outside bazel, since you should be able to calculate the path of the .py file in the runfiles tree, based on which rule the .py file belongs to.
On Mon, Nov 20, 2017 at 3:53 AM Lukács T. Berki <lbe...@google.com> wrote:
Yep -- if there is something you need, we can make that happen (except if proves to be too much work, but I doubt that's the case -- our Python rules are simple enough)
On 20 November 2017 at 09:21, Marcel Hlopko <hlo...@google.com> wrote:
I guess just trying it out and letting us know what's missing. Python rules especially are a bit neglected, and we don't have many free cycles to improve them. Maybe +Yilei Yang has more experience?
On Fri, Nov 17, 2017 at 3:37 PM Hyrum Wright <hwr...@duolingo.com> wrote:
I agree that information would be useful.  Anything I can do to get us there?
On Fri, Nov 17, 2017 at 4:23 AM, Lukács T. Berki <lbe...@google.com> wrote:
Agreed. I'm not sure if we expose enough information about Python to Skylark to be able to express "give me all the Python source files, the interpreter and tell me what PYTHONPATH is", but if we don't, we should.
On 17 November 2017 at 09:13, 'Marcel Hlopko' via bazel-discuss <bazel-discuss@googlegroups.com> wrote:
Hmm I'd say bazel aspect, but let's see what +Lukács T. Berki and +Dmitry Lomov have to say.
On Thu, Nov 16, 2017 at 3:51 AM Hyrum Wright <hwr...@duolingo.com> wrote:
I'd like to integrate pylint with my Bazel workflow.

First, I'm interested to hear if anybody has already done so, and if so, how they did it.

If I'm the pioneer in this space, I'm interested in knowing how best to run pylint, either as an analysis during the normal build, or in some separate step.  The core problem, as I see it, is that because Bazel constructs the output python tree in a sandbox, and pylint needs all of those dependencies and paths in place to function correctly, it needs Bazel to tell it where the sandbox is and what the correct PYTHONPATH to use is.  Where's the right place to get that information from Bazel?

-Hyrum

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.
--
-- 
Marcel Hlopko | Software Engineer | hlo...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.



--
Lukács T. Berki | Software Engineer | lbe...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891
--
-- 
Marcel Hlopko | Software Engineer | hlo...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891

Gregory P. Smith

unread,
Nov 27, 2017, 6:52:18 PM11/27/17
to bazel-discuss
Within Google our pylint infrastructure runs it most often with a "single file" configuration (which we call "style mode") that tries avoid needing a runfiles tree and just works as a single file analyzer.  This is obviously not as powerful as pylint _can_ be, but still provides a lot of value.

The pylint rc for that looks something similar to this:

...........................
[PATHS]
standalone=y

[MESSAGES CONTROL]

# List of checkers and msgs to disable in single file / style mode.
# These are frequent false positives that are almost all due to type inference not being possible to
# due missing imports.
disable=import-error,relative-import,unused-wildcard-import,no-name-in-module,method-hidden,access-member-before-definition,attribute-defined-outside-init,interface-is-not-class,missing-interface-method,arguments-differ,signature-differs,abstract-method,unresolved-interface,no-init,no-member,maybe-no-member

confidence=HIGH,INFERENCE,UNDEFINED
...........................

People running pylint manually within their source tree can run in nicer mode that doesn't disable so many checks.  Our internal pylint wrapper does some "magic" to try to get information from Blaze (our internal superset of Bazel) build system so that pylint run by a user in their source tree can figure out what paths to map for import resolution. But looking that over, it doesn't appear to be anything we could meaningfully release for use with Bazel (too many internal assumptions that don't match how Bazel works).

The approach I would recommend with Bazel is similar to what Yilei said or what you've done: require a build to be done to resolve dependencies and find the tree that creates.

-gps
Reply all
Reply to author
Forward
0 new messages