MLIR Python bindings

[Peteris Erins]

Dear All,

What is the status for the Python bindings in terms of feature coverage and availability? Are there any instructions how to use the Python bindings/reproduce the tests? If not available what would be their current priority on the roadmap?

Best regards

-- 

Peteris Erins

[Alex Zinenko]

Hi Peteris,

we have Python bindings that let you construct MLIR modules (functions, operations, types), which were mostly developed for Python-based tools that want to lower their representation (e.g., AST) into MLIR.  Further transformations on MLIR are expected to be run using `mlir-opt` to invoke MLIR->MLIR passes.  Currently, we prefer passes and analyses to be written in C++ for the sake of reusability.  The bindings require the "pybind11"  library to compile, but they are localized in a single file https://github.com/tensorflow/mlir/blob/master/bindings/python/pybind.cpp.  It should be possible to compile using a single command of the style "c++ -O3 -shared -fPIC -I/path/to/mlir/includes -I/path/to/llvm/includes -I/path/to/pybind11 -L/path/to/mlir/lib -L/path/to/llvm/lib -lMLIRIR -lMLIREDSC -lMLIRSupport -lLLVM -o mlir_bindings", or using cmake https://pybind11.readthedocs.io/en/stable/compiling.html#building-with-cmake after setting up the include/link locations so that MLIR and LLVM libraries can be found.  It should give you a Python binary module that can be loaded as usual in Python.

Could you please describe a specific use case where you need Python bindings?

Alex

[Peteris Erins]

Thanks Alex - that's very helpful.

I'm developing a program analysis tool, which is currently Python-based. My expectation was to use Python in:

- converting AST->MLIR (seems like that's supported by the bindings)

- converting back from final MLIR form into AST (would this be supported by the bindings?). This is required because my back-end is an SMT solver that I'm using with Python bindings, not LLVM

- ideally I wanted to express MLIR operations for custom dialects using a high-level DSL in Python or Table gen

- ideally wanted to build compiler passes in Python (as I already have existing IR transformation code), but understand the motivation for reusability

- I have a Python back-end that currently processes custom IRs into a UI for debugging purposes

Based on your description, I can use the Python bindings to get into MLIR form, but would have to build additional bindings to 1) invoke mlir-opt 2) convert back intermediate dialects from MLIR into my Python AST for custom debugging/rewriting to final back-end?

Best regards

Peteris Erins

[Alex Zinenko]

On Wed, May 8, 2019 at 9:38 AM Peteris Erins <peteri...@gmail.com> wrote:

Thanks Alex - that's very helpful.

I'm developing a program analysis tool, which is currently Python-based. My expectation was to use Python in:

- converting AST->MLIR (seems like that's supported by the bindings)

Indeed.

 

- converting back from final MLIR form into AST (would this be supported by the bindings?). This is required because my back-end is an SMT solver that I'm using with Python bindings, not LLVM

I don't think there are enough bindings to inspect the IR.

 

- ideally I wanted to express MLIR operations for custom dialects using a high-level DSL in Python or Table gen

See below.

 

- ideally wanted to build compiler passes in Python (as I already have existing IR transformation code), but understand the motivation for reusability

- I have a Python back-end that currently processes custom IRs into a UI for debugging purposes

Based on your description, I can use the Python bindings to get into MLIR form, but would have to build additional bindings to 1) invoke mlir-opt 2) convert back intermediate dialects from MLIR into my Python AST for custom debugging/rewriting to final back-end?

MLIR has an extensible list of almost everything (operations/instructions, types, attributes).  This is great for use cases, but tricky for APIs.  An mlir::Operation internally is just a structure with string names and lists of values.  In C++, we discourage the use of mlir::Operation outside of very generic IR manipulations such as cloning or use-def chain construction.  Instead, we use Tablegen to define C++ wrappers around mlir::Operation that provide, among others, "semantic" accessors such as `getUpperBound` instead of `getOperand(2)`.  To get those in Python, we would need to also produce Python wrappers using Tablegen, and to manually wrap the operations that are not defined in Tablegen for some reason.

As a short term solution, you can look into exposing `mlir::Operation` in Python bindings (`Block` and `Function` are already exposed but may need a bit more functionality).  This will let you to inspect the IR which, in combination with IR construction capabilities, can also let you write simple IR transformations.  However, unless you define Python wrappers for the operations in your dialect, the interface will look quite unintuitive, e.g. with string comparisons and integer constants everywhere.  MLIR also works with unknown operations, but does so conservatively.  This means you can instantiate an operation "mydialect.myop", insert it into functions/blocks and get it back, but MLIR will not do anything further than trivial type-checking of the operands.

Alex

 

Best regards

Peteris Erins

On Tuesday, May 7, 2019 at 10:04:16 PM UTC+1, Alex Zinenko wrote:

Hi Peteris,

we have Python bindings that let you construct MLIR modules (functions, operations, types), which were mostly developed for Python-based tools that want to lower their representation (e.g., AST) into MLIR.  Further transformations on MLIR are expected to be run using `mlir-opt` to invoke MLIR->MLIR passes.  Currently, we prefer passes and analyses to be written in C++ for the sake of reusability.  The bindings require the "pybind11"  library to compile, but they are localized in a single file https://github.com/tensorflow/mlir/blob/master/bindings/python/pybind.cpp.  It should be possible to compile using a single command of the style "c++ -O3 -shared -fPIC -I/path/to/mlir/includes -I/path/to/llvm/includes -I/path/to/pybind11 -L/path/to/mlir/lib -L/path/to/llvm/lib -lMLIRIR -lMLIREDSC -lMLIRSupport -lLLVM -o mlir_bindings", or using cmake https://pybind11.readthedocs.io/en/stable/compiling.html#building-with-cmake after setting up the include/link locations so that MLIR and LLVM libraries can be found.  It should give you a Python binary module that can be loaded as usual in Python.

Could you please describe a specific use case where you need Python bindings?

Alex

 
On Tuesday, May 7, 2019 at 10:21:15 PM UTC+2, Peteris Erins wrote:

Dear All,

What is the status for the Python bindings in terms of feature coverage and availability? Are there any instructions how to use the Python bindings/reproduce the tests? If not available what would be their current priority on the roadmap?

Best regards

-- 

Peteris Erins

--
You received this message because you are subscribed to the Google Groups "MLIR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mlir+uns...@tensorflow.org.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/mlir/100b6dc3-2bb7-4bbf-8dc9-d9d9410e42d3%40tensorflow.org.

--

-- Alex

[Peteris Erins]

Thanks Alex, that's super clear and sensible.

You've managed shift my appetite towards relying on C++ a bit more :), which is not necessarily a bad thing.

Best regards

Peteris Erins

To unsubscribe from this group and stop receiving emails from it, send an email to ml...@tensorflow.org.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/mlir/100b6dc3-2bb7-4bbf-8dc9-d9d9410e42d3%40tensorflow.org.

--

-- Alex