[RFC] ONNX Dialect

[Tian Jin]

I hope to share with you our plan to contribute an ONNX (Open Neural Network Exchange) dialect and a reference lowering process converting ONNX dialect to existing MLIR dialects (e.g., StandardOps, AffineOps, Linalg/StructuredOps).

Introduction

ONNX is an open format to represent any neural network models; ONNX format contains a list of standard, framework-agnostic definition of operations and their semantic specifications (https://github.com/onnx/onnx/blob/master/docs/Operators.md). Naturally, we can express ONNX operations as a dialect within MLIR and connect it to the rest of the MLIR infrastructure. To this end, we propose to contribute the following:

• ONNX Dialect: ONNX dialect reflects the standard definition of ONNX operations in an MLIR dialect.
• Reference Lowering: Reference lowering reflects the semantic specification of ONNX operations by lowering them to existing transformation dialects such as Affine and Linalg/StructuredOps dialects in straightforward ways. Reference lowering helps provide a set of low-level, testable, and self-contained implementation of ONNX standard.

After discussing with stakeholders in both MLIR and ONNX community (prior Github discussion at https://github.com/onnx/onnx/issues/2499), we believe it would be beneficial to both MLIR and ONNX communities to upstream these contributions under the MLIR/LLVM umbrella. In doing so, we hope to facilitate the usage and adoption of the ONNX standard/dialect in the MLIR community.

Rationale

In this section, we present rationales/key aspects of considerations for including ONNX dialect as a core dialect in MLIR following the developer guidelines.

• What is the overall goal of the dialect? What is the first implementation milestone?

  The goal is to provide an MLIR dialect implementation and a reference lowering of ONNX standard operations. The implication includes the following, to name just a few:

  • ONNX dialect enables ONNX converters to make use of MLIR infrastructures, which can help tremendously with model conversions to and from ONNX formats in areas such as verification/graph rewriting.

  • Reference lowering provides a set of IR definitions for ONNX operations. These IR definitions are low-level, testable, and self-contained by construction. Such IR definitions are instrumental for hardware vendors working to support ONNX.

  Drawing on my experience working with the ONNX community on some of the converter efforts, the first meaningful milestone will be reached when we can support expressing and lowering the set of model tests embedded in the ONNX standard package. Reaching this mile stone will demonstrate some end-to-end functionalities/correctness of our contribution.

• How does it fit into the MLIR dialect ecosystem?

  Our contribution lowers the ONNX dialect to other built-in dialects such as Affine and Linalg/StructuredOps dialects. Moreover, we also leverage the efforts of the MLIR community to lower these intermediate representations further down to LLVM IR in order to generate test programs.

• What is the community of users that it is serving?

  We believe this dialect primarily serves the ONNX community. The benefits to the MLIR community are also substantial, given that hardware vendors will have access to a set of IR definitions of commonly used NN/ML operations as specified by an open, vendor-neutral, widely used standard.

• Who are the future contributors/maintainers beyond those who propose the dialect?

  The initial support will come from IBM Research. We expect members of the ONNX communities to join our development and maintenance effort imminently.

Contributing Peripheral Components

We are interested in your opinions on how to contribute some supporting components of our contribution. Some of them may not fit well under MLIR directly, and it may be a good idea to upstream them as an LLVM Project, to provide driver code for essentially an ONNX frontend for MLIR. These include:

• Model Ingestion Util: model ingestion from ONNX protobuf into MLIR. This utility has a dependency on protobuf.
• End-to-end Unit Tests: testing reference lowering end-to-end using unit tests embedded in the ONNX python package consisting of reference inputs/outputs is hugely desirable. The Python unittest package drives these tests, and it makes use of several ONNX python API. Naturally, the ONNX unit tests have a dependency on ONNX python package.
• Driver: to enable end-to-end unit tests, a minimum ONNX backend has to be built, along with some driver/glue code to connect with Python API; this introduces dependencies on pybind, numpy, etc.

Thus, ideally, to merge our contribution into MLIR/LLVM, we are considering the following:

Incubate our contribution as an LLVM project, then upstream to MLIR as much as possible. Specifically, we may proceed with three steps:

• Initial development happens under a project named “ONNF” (Open Neural Network Frontend) in llvm-project monorepo.
• Work with ONNX and MLIR community members to get consensus on project structure and rules of contribution, implement reference lowering towards the first milestone.
• Upstream dialect + reference lowering to upstream MLIR, leaving only driver code, python unit tests, model importing tools in ONNF under llvm-project mono repo.

We will try to upstream ONNX dialect and reference lowering as early as possible so as not to make upstreaming to MLIR a significant burden for MLIR reviewers. Such burden can be further mitigated by involving MLIR reviewers early into our review process as an llvm-project.

An alternative path is to upstream our contribution (minus the three components mentioned above) to MLIR as soon as possible and to iterate with community feedback entirely within upstream MLIR. Driver, python unit tests, and model ingestion code will still be hosted elsewhere (again, preferably under llvm-project).

We would like to hear your opinions on how to best proceed with the contribution to MLIR/LLVM community.

Example

A preliminary version of our code repository has been pushed to https://github.com/clang-ykt/ONNF . In this section, we demonstrate the usage and utility of this ONNF (Open Neural Network Frontend) project.

To ingest an ONNX model protobuf add.onnx which contains an add operation, run the following:

./onnf --EmitONNXIR add.onnx

The output is:

module {
  func @main_graph(%arg0: tensor<10x10x10xf32>, %arg1: tensor<10x10x10xf32>) -> tensor<10x10x10xf32> {
    %0 = "onnx.Add"(%arg0, %arg1) : (tensor<10x10x10xf32>, tensor<10x10x10xf32>) -> tensor<10x10x10xf32>
    return %0 : tensor<10x10x10xf32>
  }
}

To see the built-in dialect representation of ONNX model protobuf, run the following:

./onnf --EmitMLIR add.onnx

The output is:

#map0 = () -> (0)
#map1 = () -> (10)

module {
  func @main_graph(%arg0: memref<10x10x10xf32>, %arg1: memref<10x10x10xf32>) -> memref<10x10x10xf32> {
    %0 = alloc() : memref<10x10x10xf32>
    affine.for %arg2 = 0 to 10 {
      affine.for %arg3 = 0 to 10 {
        affine.for %arg4 = 0 to 10 {
          %1 = load %arg0[%arg2, %arg3, %arg4] : memref<10x10x10xf32>
          %2 = load %arg1[%arg2, %arg3, %arg4] : memref<10x10x10xf32>
          %3 = addf %1, %2 : f32
          store %3, %0[%arg2, %arg3, %arg4] : memref<10x10x10xf32>
        }
      }
    }
    return %0 : memref<10x10x10xf32>
  }
}

Acknowledgement

This version of Open Neural Network Frontend is contributed by Tian Jin, Doru Bercea, Tung D. Le, Tong Chen, Haruki Imai, and Alex Eichenberger from IBM Research.

[Alex Zinenko]

Hi Tian,

this proposal sounds exciting! Since MLIR code is a part of LLVM monorepo as of today, I would suggest involving the broader LLVM community in the discussion of project layering and dependencies. Personally, I think a case can be made for an ONNX dialect in MLIR and it would make sense to develop the dialect upstream. I am also interested in the choice when to use Affine or Linalg and the idea of reference lowering (which implies there are other possible lowerings), but don't want to side-track this higher-level discussion.

--
You received this message because you are subscribed to the Google Groups "MLIR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mlir+uns...@tensorflow.org.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/mlir/CAEmcUquxidogLWi5y2YiFwjCsRztvCzWeTsD2AnEMQqmjhZSRg%40mail.gmail.com.

--

-- Alex

[Nicolas Vasilache]

Hello Tian,

Thank you for sharing this, this looks great!

A few comments on the pieces that would connect to work I am doing (mostly Linalg).

1. I am planning to add support for tensors in Linalg ops. After thinking a bit more about the concept of StructuredOp on tensors and implementing a prototype internally, I find that it does not make a big difference to just let Linalg ops take buffers or tensors as arguments (and return tensors where appropriate). What this will give is a unified way to specify ops that can partially lower by performing buffer allocation. I anticipate significant code reuse benefits, better mixing of transformations and more progressive transition from the tensor to the buffer world.

2. I have started using Linalg more as a frontend with the same objectives and benefits you mention for building small models. I pointed Alex to some of the code privately but the code in Linalg/EDSC is generic enough to build full models with a C++ EDSL style. There are still opportunities to evolve the C++ sugar and serve as a common layer for frameworks/programming APIs that would want to hook into MLIR at this level. In particular, I am interested in evolving our HLOS -> LHLO -> Linalg lowering to something that much more closely resemble custom ops based on Linalg. Notice how at that level of abstraction, ONNX and HLO look quite similar (https://github.com/tensorflow/tensorflow/blob/master/tensorflow/compiler/mlir/xla/tests/hlo-legalize-to-lhlo.mlir). Let's discuss :)

3. Linalg itself will start evolving in the non-dense world (I anticipate sometime end of Q1, beginning of Q2), TACO-style compilation is a pretty clear direction in which to go and this may also interplay nicely with what you guys are doing.

In any case I am glad to see this effort and I look forward to talking more in 2020!  

N

--
You received this message because you are subscribed to the Google Groups "MLIR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mlir+uns...@tensorflow.org.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/mlir/CAEmcUquxidogLWi5y2YiFwjCsRztvCzWeTsD2AnEMQqmjhZSRg%40mail.gmail.com.

--

N

[Mehdi AMINI]

On Tue, Dec 24, 2019 at 2:22 AM 'Alex Zinenko' via MLIR <ml...@tensorflow.org> wrote:

Hi Tian,

this proposal sounds exciting! Since MLIR code is a part of LLVM monorepo as of today, I would suggest involving the broader LLVM community in the discussion of project layering and dependencies.

Moving forward we'll use https://llvm.discourse.group/c/llvm-project/mlir to discuss Core MLIR proposals. This proposal does not seem directly a contribution to MLIR Core right now though, it seems more like positioned as proposal for the LLVM foundation to adopt a new subproject, the llvm-dev@ mailing-list seems like the current best place to start such a discussion.

 

Personally, I think a case can be made for an ONNX dialect in MLIR and it would make sense to develop the dialect upstream.

It isn't that clear to me: why wouldn't this be developed in ONNX itself? It seems like one possible producer of MLIR like many others, similarly to the TensorFlow dialect, are we gonna import upstream every possible framework-specific dialects?

If ONNF becomes an LLVM subproject independently of MLIR Core, then I could see the ONNX dialect lives in this subproject just like FIR lives with flang: this is parallel and independent of MLIR in my opinion.

Best,

--

Mehdi

 

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/mlir/CAPL655iPFoPW%2BS18z4aS3sLuYE%2B3QZtKTOjaZCXHXB4x85vUYA%40mail.gmail.com.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/mlir/CAPL655iPFoPW%2BS18z4aS3sLuYE%2B3QZtKTOjaZCXHXB4x85vUYA%40mail.gmail.com.

[Jacques Pienaar]

Hi Tian,

This looks interesting and could be a good topic for a future open design discussion. So feel free to suggest it there for discussion and we could incorporate it into the schedule.

I share the same concerns as Mehdi here though. We've been focussing on frontend dialects living with their frontends. A big part of the reason is that there is normally a large amount of infrastructure and ecosystem that goes along with it (e.g., the peripheral components you mention are new & largish dependencies) and separating them has an engineering, dependency and maintenance cost. One of the goals of making MLIR's core extensible is to avoid needing to make changes to core to support new dialects, and please let us know if you hit roadblocks there.

As others have said, proposing a new frontend to LLVM project should be discussed on the main mailing list (just as flang was proposed) as that is more a question for the LLVM community.

I think developing it with ONNX itself also makes a lot of sense. That would seem very natural place for it to evolve.

Best,

Jacques

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/mlir/CANF-O%3DZ-EAneB7B%3D3%3Djpm%2BS%2BLBnSWZNpaSd19euXucSKURzD8w%40mail.gmail.com.

[Chris Lattner]

On Dec 24, 2019, at 1:15 AM, Tian Jin <tjin...@gmail.com> wrote:

I hope to share with you our plan to contribute an ONNX (Open Neural Network Exchange) dialect and a reference lowering process converting ONNX dialect to existing MLIR dialects (e.g., StandardOps, AffineOps, Linalg/StructuredOps).

This is very exciting - I’m very happy to see your work on this!

Happy holidays,

-Chris

[Alex Zinenko]

On Tue, Dec 24, 2019 at 9:51 PM Mehdi AMINI <joke...@gmail.com> wrote:

On Tue, Dec 24, 2019 at 2:22 AM 'Alex Zinenko' via MLIR <ml...@tensorflow.org> wrote:

Hi Tian,

this proposal sounds exciting! Since MLIR code is a part of LLVM monorepo as of today, I would suggest involving the broader LLVM community in the discussion of project layering and dependencies.

Moving forward we'll use https://llvm.discourse.group/c/llvm-project/mlir to discuss Core MLIR proposals. This proposal does not seem directly a contribution to MLIR Core right now though, it seems more like positioned as proposal for the LLVM foundation to adopt a new subproject, the llvm-dev@ mailing-list seems like the current best place to start such a discussion.

 

Personally, I think a case can be made for an ONNX dialect in MLIR and it would make sense to develop the dialect upstream.

It isn't that clear to me: why wouldn't this be developed in ONNX itself? It seems like one possible producer of MLIR like many others, similarly to the TensorFlow dialect, are we gonna import upstream every possible framework-specific dialects?

That's why I wrote "can be made". It's not for me to make this case.

IMO, it depends on the positioning and evolution goals of ONNX. If we see it as an independent, open specification (rather than a specific implementation of that specification), it may make sense as a core dialect. If we see it as yet another framework, it does not.

--

-- Alex

[Stephen Neuendorffer]

I think that having core dialect support on some form for video/ml computation would be a good idea.  I think having a discussion about whether onnx is the best design basis or something else would be worthwhile.

Steve

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/mlir/CAPL655iG%2BDNjSC%2B4ms__XtW_Kr9jRuZVAzTG_yka--iSZYLU4Q%40mail.gmail.com.

[Ke Zhang]

+1.

Supporting Video/ML is definitely one goal of ONNX (ONNX has had couple operators supporting traditional ML, btw). Comments/Proposals/Discussions about it are welcome :-)

-Ke

[Jun Yang]

In addition to ONNX, do we have any other alternative candidates for the dialect to start from? 

Or do you suggest to build a new video/ml dialect from the scratch?

In my opinion, it is not quite difficult to design such a dialect. The most difficult thing is the underlying engineering cost and overhead to ensure the dialect to be accepted by others. 

Let me take ONNX as an example. During daily deployment, it is not unusual for us to meet with functionality or performance or even accuracy issues with importing another model format into ONNX form. The devils lie in the details. 

So if we want to design a new dialect for video/ml(just like to re-invent a new wheel) we may take the potential engineering cost to support it into consideration.  

Regards 

Jun

在 2019年12月27日星期五 UTC+8上午4:55:55，Stephen Neuendorffer写道：

To unsubscribe from this group and stop receiving emails from it, send an email to ml...@tensorflow.org.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/mlir/CAEmcUquxidogLWi5y2YiFwjCsRztvCzWeTsD2AnEMQqmjhZSRg%40mail.gmail.com.

--

-- Alex

--
You received this message because you are subscribed to the Google Groups "MLIR" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ml...@tensorflow.org.

To unsubscribe from this group and stop receiving emails from it, send an email to ml...@tensorflow.org.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/mlir/CAEmcUquxidogLWi5y2YiFwjCsRztvCzWeTsD2AnEMQqmjhZSRg%40mail.gmail.com.

--

-- Alex

--
You received this message because you are subscribed to the Google Groups "MLIR" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ml...@tensorflow.org.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/mlir/CAPL655iPFoPW%2BS18z4aS3sLuYE%2B3QZtKTOjaZCXHXB4x85vUYA%40mail.gmail.com.

--

-- Alex

--
You received this message because you are subscribed to the Google Groups "MLIR" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ml...@tensorflow.org.

[Stephen Neuendorffer]

I'm pretty familiar with OpenCV and OpenVX.  OpenVX was designed as a standard with portability, standardized behavior and conformance test suites.   For ML, I think the landscape is more muddled, with many alternatives at different layers, each if which has made slightly different design decisions.   Is there interest/value in trying to find the commonality between different frameworks, or just let each one figure out it's own lowering into stencils/Linalg?  I agree that the 'devil is in the details', particularly for ML where there is more of a premium on performance in most cases, rather than bit-for-bit compatibility.  It is difficult to be on the receiving end of trying to sort out these differences.  It is certainly important to consider semantics and behavior in addition to simply syntaxes and file formats.  

As an aside, this is an area where I'd like to see MLIR grow: right now the 'definition' of dialect semantics is solely through its syntactic lowering to/from other dialects.  In this sense, the lowering to LLVM is pretty much the only think in core MLIR which defines semantics.  I think the challenge is that this lowering through a sequence of other dialects can be hard to understand/verify/debug as a whole when it fails, even if each individual step in the lowering is relatively simple.

Steve

To unsubscribe from this group and stop receiving emails from it, send an email to mlir+uns...@tensorflow.org.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/mlir/af1dd066-6522-4620-976c-29f28e73393c%40tensorflow.org.

[Nicolas Vasilache]

The thread "Roadmap about adding dynamic shape support in MLIR HLO dialect" should be relevant to this discussion since HLO is known to be Turing-complete and that Tensorflow, JAX and PyTorch -> TPU use it.

Re Stephen's comment on Linalg, one big missing piece of infrastructure today is a way to automatically specify and generate the "named" ops from their generic counterpart. 

The current thinking is to create a new Tablegen component where one would specify a "name" for an op + attributes and how they map to linalg.generic/linalg.indexed_generic attributes and regions.

Then we could automatically generate pretty-printers and parsers for the sugared form (i.e. use the "named" op instead of the generic op) as well as matchers to raise from a generic op to the "named" op.

Of course some things won't fit magically but a large part does fit quite naturally (much more than people acutally realize).

For the parts that indeed don't fit, the same discussion as in XLA applies: does a new, non-generic op in the dialect pay for itself?

Details here a super important and there is a tradeoff between (1) trying hard to fit within the same dialect (to benefit from composition of transformations) and (2) just implement at a more expressive, lower-level dialect and lose the high-level transformation compositions. 

Once we have that piece of infrastructure, every framework can declare their own set of ops matching their preferred programming model and they will be guaranteed to transform nicely in Linalg and below (affine, scf, cfg, llvm).

To unsubscribe from this group and stop receiving emails from it, send an email to mlir+unsubscribe@tensorflow.org.

[Stephen Neuendorffer]

I don't understand your comments about named ops vs generic ops.  Can you elaborate?

Steve

To unsubscribe from this group and stop receiving emails from it, send an email to mlir+uns...@tensorflow.org.

[Nicolas Vasilache]

See the Payload Ops section in https://mlir.llvm.org/docs/Dialects/LinalgDoc/.

Basically, I added `copy`, `fill`, `dot`, `matmul` and `conv` for illustrative purposes.

But there are really just configurations of `generic` and should be auto-generated along with matchers.

One could always use `generic` everywhere but it quickly becomes much more verbose than it needs to be.

See for example https://github.com/llvm/llvm-project/blob/0f0d0ed1c78f1a80139a1f2133fad5284691a121/mlir/test/Dialect/Linalg/llvm.mlir#L156

```

#matmul_accesses = [
  (m, n, k) -> (m, k),
  (m, n, k) -> (k, n),
  (m, n, k) -> (m, n)
]
#matmul_trait = {
  args_in = 2,
  args_out = 1,
  iterator_types = ["parallel", "parallel", "reduction"],
  indexing_maps = #matmul_accesses,
  library_call = "external_outerproduct_matmul"
}

!vector_type_A = type vector<4xf32>
!vector_type_B = type vector<4xf32>
!vector_type_C = type vector<4x4xf32>

!matrix_type_A = type memref<?x?x!vector_type_A>
!matrix_type_B = type memref<?x?x!vector_type_B>
!matrix_type_C = type memref<?x?x!vector_type_C>

func @matmul_vec_impl(%A: !matrix_type_A, %B: !matrix_type_B, %C: !matrix_type_C) {
  linalg.generic #matmul_trait %A, %B, %C {
    ^bb0(%a: !vector_type_A, %b: !vector_type_B, %c: !vector_type_C):
      %d = vector.outerproduct %a, %b, %c: !vector_type_A, !vector_type_B
      linalg.yield %d: !vector_type_C
  } : !matrix_type_A, !matrix_type_B, !matrix_type_C

  return
}

```

--

N

[Tian Jin]

Thanks Alex for your thoughtful comments!

I will certainly post an RFC on llvm-dev mailing list to gather thoughts from the broader LLVM community as soon as I get more clarity on the scope and direction of this proposal!

I agree that there will be choices to be made. My preliminary thought is that we want to prioritize clarity & simplicity (to provide interpretable low-level operation semantic definition), ease of instrumentation & customization (to facilitate downstream usage of such IR definitions) when making these decisions.

Tian.

To unsubscribe from this group and stop receiving emails from it, send an email to ml...@tensorflow.org.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/mlir/CAEmcUquxidogLWi5y2YiFwjCsRztvCzWeTsD2AnEMQqmjhZSRg%40mail.gmail.com.

--

-- Alex

[Tian Jin]

Hi Mehdi:

Thank you for your insightful questions and comments, sharing with you my thoughts/responses below:

There are many possibilities to take this proposal further, one of which we'd like to see happen is to propose ONNX as one of the core dialects.

Getting opinions from llvm community is essential; however, given the relevance of this RFC to MLIR, we'd love to start a discussion with MLIR and ONNX community as a first step so that some decisions can be made early (e.g., do we see ONNX dialect going into core or into another llvm-project?), and when we present the broader community with another followup RFC, it will bear more clarity regarding our goals and methods to achieve them.

More specific responses below:

> It isn't that clear to me: why wouldn't this be developed in ONNX itself? It seems like one possible producer of MLIR like many others, similarly to the TensorFlow dialect, are we gonna import upstream every possible framework-specific dialects?

It makes sense to develop this project under LLVM/MLIR umbrella because the bulk of this contribution (reference lowering) concerns the frontend aspect of the MLIR compiler infrastructure. That being said, providing infrastructural support for ONNX model converters is also an important objective of this contribution, which concerns the exchange aspect of ONNX.

The case for making ONNX dialect a core dialect within MLIR is two-fold:

- MLIR infrastructure will benefit from a high-level core dialect: with MLIR gaining traction, there will likely be many more projects looking to utilize it, quite possibly with many having a focus on supporting AI workloads. Thus providing built-in access to definitions of a set of core operations commonly used in the NN/ML context seems to be a natural response to the anticipated demand. It relieves developers from the burden of defining such operations all by themselves, which can be a huge burden. As Nicolas pointed out earlier, there are opportunities to have a "common layer for frameworks/programming APIs that would want to hook into MLIR at this level". I absolutely agree with his view.

- Evolving such a core dialect from ONNX is the most reasonable way forward: positioned as an IR for exchanging model between frameworks, ONNX is unlike Tensorflow dialect in that it is designed to represent NN/ML workloads under the constraints of framework and vendor-neutrality. These two qualities are best demonstrated with the variety of frontend frameworks  (TF, Pytorch, scikit-learn to name a few) and backend hardware/software combinations currently supported by ONNX. Putting the ONNX dialect in the MLIR core is thus natural and desirable because the operations and their definitions are reusable across multiple frameworks and backend infrastructures by design. Moreover, with a variety of tools available today within the ONNX ecosystem and the many people actively using them, ONNX is likely the most battle-tested re-usable high-level IR available today and evolving a high-level core dialect based on ONNX seems to be the most reasonable path forward.

Tian.

To unsubscribe from this group and stop receiving emails from it, send an email to ml...@tensorflow.org.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/mlir/CAEmcUquxidogLWi5y2YiFwjCsRztvCzWeTsD2AnEMQqmjhZSRg%40mail.gmail.com.

--

-- Alex

--
You received this message because you are subscribed to the Google Groups "MLIR" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ml...@tensorflow.org.

To unsubscribe from this group and stop receiving emails from it, send an email to ml...@tensorflow.org.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/mlir/CAEmcUquxidogLWi5y2YiFwjCsRztvCzWeTsD2AnEMQqmjhZSRg%40mail.gmail.com.

--

-- Alex

--
You received this message because you are subscribed to the Google Groups "MLIR" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ml...@tensorflow.org.

[Tian Jin]

Hi Jacques:

Thanks for your instructive comments!

I've proposed the discussion. I assume TBD means to be determined by the MLIR team? Or do people usually propose the discussion with a specific date in mind?

I share your concern regarding the peripheral infrastructure around ONNX. However, the core merits of an ONNX dialect does not depend on these peripheral infrastructures. And as per my response to Mehdi, including ONNX as a high-level core dialect brings about substantial benefits in and of itself and the pros may very well outweigh the cons we may experience in order to prevent burdening MLIR core with extra dependencies.

We will certainly post a more refined version of this RFC to llvm mailing list in the near future!

Can't wait to chat with you during the upcoming open design discussion!

Tian.

To unsubscribe from this group and stop receiving emails from it, send an email to ml...@tensorflow.org.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/mlir/CAEmcUquxidogLWi5y2YiFwjCsRztvCzWeTsD2AnEMQqmjhZSRg%40mail.gmail.com.

--

-- Alex

--
You received this message because you are subscribed to the Google Groups "MLIR" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ml...@tensorflow.org.

To unsubscribe from this group and stop receiving emails from it, send an email to ml...@tensorflow.org.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/mlir/CAEmcUquxidogLWi5y2YiFwjCsRztvCzWeTsD2AnEMQqmjhZSRg%40mail.gmail.com.

--

-- Alex

--
You received this message because you are subscribed to the Google Groups "MLIR" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ml...@tensorflow.org.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/mlir/CAPL655iPFoPW%2BS18z4aS3sLuYE%2B3QZtKTOjaZCXHXB4x85vUYA%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "MLIR" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ml...@tensorflow.org.

[River Riddle]

Hi Tian,

On Thu, Jan 2, 2020 at 8:17 PM Tian Jin <tjin...@gmail.com> wrote:

Hi Jacques:

Thanks for your instructive comments!

I've proposed the discussion. I assume TBD means to be determined by the MLIR team? Or do people usually propose the discussion with a specific date in mind?

I share your concern regarding the peripheral infrastructure around ONNX. However, the core merits of an ONNX dialect does not depend on these peripheral infrastructures. And as per my response to Mehdi, including ONNX as a high-level core dialect brings about substantial benefits in and of itself and the pros may very well outweigh the cons we may experience in order to prevent burdening MLIR core with extra dependencies.

I actually have a pretty conflicting view on this. I find starting with "well established" frameworks to be very bad for building these types of *core* dialects. They generally bring unnecessary technical debt, political baggage, compatibility guarantees, external dependencies, etc. that actively inhibit developing the "best" solution. I can understand why adding an ONNX frontend would be beneficial from the perspective of the ONNX community, but unless LLVM/MLIR had full ability to add/change/remove operations at will, we will likely end up adding a similar-but-different dialect that we can control. What are the benefits from developing in /mlir vs in the ONNX project itself? That location seems like a much more natural fit IMO.

-- River

 

 

To unsubscribe from this group and stop receiving emails from it, send an email to mlir+uns...@tensorflow.org.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/mlir/5ba14c3b-4848-4bbf-bbf4-b5fd8de312d9%40tensorflow.org.

[Stella Laurenzo]

On Thu, Jan 2, 2020 at 8:24 PM 'River Riddle' via MLIR <ml...@tensorflow.org> wrote:

Hi Tian,

On Thu, Jan 2, 2020 at 8:17 PM Tian Jin <tjin...@gmail.com> wrote:

Hi Jacques:

Thanks for your instructive comments!

I've proposed the discussion. I assume TBD means to be determined by the MLIR team? Or do people usually propose the discussion with a specific date in mind?

I share your concern regarding the peripheral infrastructure around ONNX. However, the core merits of an ONNX dialect does not depend on these peripheral infrastructures. And as per my response to Mehdi, including ONNX as a high-level core dialect brings about substantial benefits in and of itself and the pros may very well outweigh the cons we may experience in order to prevent burdening MLIR core with extra dependencies.

I actually have a pretty conflicting view on this. I find starting with "well established" frameworks to be very bad for building these types of *core* dialects. They generally bring unnecessary technical debt, political baggage, compatibility guarantees, external dependencies, etc. that actively inhibit developing the "best" solution. I can understand why adding an ONNX frontend would be beneficial from the perspective of the ONNX community, but unless LLVM/MLIR had full ability to add/change/remove operations at will, we will likely end up adding a similar-but-different dialect that we can control. What are the benefits from developing in /mlir vs in the ONNX project itself? That location seems like a much more natural fit IMO.

I'm +1 on River's viewpoint on this wrt "core". For the benefit of the community, this has been a very active debate that we've been having on the Google side for the last year and we've been in a mode of operation where we are trying to chart a course where we keep some distance between the core dialects and the diaspora of current ML op sets. You'll notice, for example, that neither the TensorFlow nor XLA/HLO dialects are part of the MLIR project -- despite it many times seeming expedient to make them so. As River says, the baggage that comes from the existing op sets can inhibit the search for a "best" common set of capabilities defined at the right level of granularity. I agree that the ONNX op set is by some (many?) measures a more principled set of common ML ops than some of the others, but I'm -1 on taking such a high level thing into MLIR itself in a single step and without some distillation that involves merging the perspectives that other op sets have. In general as well, we need to provide time for the MLIR eco-system to evolve the right granularity of features, and having such a large fixed-point so early in the process seems like it may be skipping steps.

With that said, I would be *very* supportive of what you propose existing as a high-quality implementation, with minimal dependencies built to complement MLIR and the tools built around it. We haven't started any kind of MLIR-commons federation of projects yet, but if that existed, this seems like the perfect thing to seed it with. I would love to see a place where high quality, opinionated, implementations like yours could thrive with some common build/dep machinery and a reasonable expectation of being able to use it and interop with other adjacent projects. This would also help us have a path to both introduce things and look to graduating projects that have achieved traction towards the core.

For the record, I lead the IREE project, where we are also trying to seed something in this commons space that I hope we can build up around MLIR.

My 2 cents...

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/mlir/CAPXRNp%2BsCDoiZ5z-Mu6ZhZxsbon4pE0ta7ZyMYFL-EvDqta5Eg%40mail.gmail.com.

[Tian Jin]

Hi River:

I think your comments raise important discussion points for the design meeting and let's make sure to cover it with the broader community!

My responses below:

Let's try grounding our discussion on specific points - for one, ONNX versioning policy is designed to minimize the presence of technical debt - semantics of operations are versioned and therefore new (often more reasonable) operation schema/semantics can be introduced without any constraints. Often full support for every version of an operation is unnecessary, which further mitigates concerns about technical debt.

I also do not see political baggage going forward. Linux Foundation currently hosts the ONNX project and ONNX spec is managed with an explicit open model of governance steered by a few elected vendors (currently including Intel and NVidia who are likely to be equally interested in MLIR). The political principle underpinning the ONNX community is a democratic one.

In general, I'm afraid that sometimes the "best" solution for a developer may not the "best" solution for a user. To avoid such pitfalls, I'd like to examine the specifics rather than relying on general impressions.

> but unless LLVM/MLIR had full ability to add/change/remove operations at will, we will likely end up adding a similar-but-different dialect that we can control

I'm actually more optimistic about the prospect. If the objective and model of governance of ONNX align well with the vision of MLIR/LLVM community in developing such a high-level core dialect, then re-using the ONNX spec is not only natural but immensely desirable. 

> What are the benefits from developing in /mlir vs in the ONNX project itself?

The reason for the preference of the MLIR/LLVM umbrella is that reference lowering is more relevant to the MLIR compiler infrastructure community. Recall that ONNX concerns the exchange of models across frameworks/backend infrastructures.

Tian.

Hi Tian,

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/mlir/5ba14c3b-4848-4bbf-bbf4-b5fd8de312d9%40tensorflow.org.

[Jonny Shipton]

The thread "Roadmap about adding dynamic shape support in MLIR HLO dialect" should be relevant to this discussion since HLO is known to be Turing-complete and that Tensorflow, JAX and PyTorch -> TPU use it.

Hi Nicolas, that thread you link isn't accessible to me. Is it a private group?

Apologies for the massive reply prior; I have deleted it on the web interface.

[Nicolas Vasilache]

Hi Jonny,

We seem to have a new interface internally for google groups and the links seem to not be compatible with the outside world!

So, to get to the discussion, go to https://groups.google.com/a/tensorflow.org/g/mlir and search for "Roadmap about adding dynamic shape support in MLIR HLO dialect".

Sorry for the confusion..

--

You received this message because you are subscribed to the Google Groups "MLIR" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mlir+uns...@tensorflow.org.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/mlir/e442cf9e-7014-4c85-91c0-eab2aeb58b72%40tensorflow.org.

--

N

[Tian Jin]

Thanks for your thoughtful comments Stella!

Your comments really helped me understand the broader context of discussions around having a high-level core dialect. I fully agree that some iterations are necessary to ensure the "fitness for purpose". 

I also concur that further investigations are needed for understanding the optimal granularity of features and would really appreciate having a deeper dive into this subject some time into the future.

> With that said, I would be *very* supportive of what you propose existing as a high-quality implementation, with minimal dependencies built to complement MLIR and the tools built around it. We haven't started any kind of MLIR-commons federation of projects yet, but if that existed, this seems like the perfect thing to seed it with.

That sounds like an exciting direction and a reasonable step for us to explore, are there thoughts around such a federation of projects within the MLIR team today?

Do you see such a federation of projects to be scattered and distributed around different places or do you seek to have a more organized eco-system?

> For the record, I lead the IREE project, where we are also trying to seed something in this commons space that I hope we can build up around MLIR.

I would really like to know how far you and your team have ventured towards establishing this commons space.

> As River says, the baggage that comes from the existing op sets can inhibit the search for a "best" common set of capabilities defined at the right level of granularity

Sounds like both you and River have some strong and concrete concerns, I would really appreciate having a deeper understanding into these reservations.

Do you see a way to open up this discussion of design constraints around a high-level core dialect to the broader community?

Really appreciate your detailed explanations and constructive suggestions!

Tian.

-- River

 

 

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/mlir/5ba14c3b-4848-4bbf-bbf4-b5fd8de312d9%40tensorflow.org.

--
You received this message because you are subscribed to the Google Groups "MLIR" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ml...@tensorflow.org.

To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/mlir/CAPXRNp%2BsCDoiZ5z-Mu6ZhZxsbon4pE0ta7ZyMYFL-EvDqta5Eg%40mail.gmail.com.

[Ke Zhang]

as I'm reading this thread carefully, having a core dialect for ML may be widely agreed and accepted :), as for a specific domain (say, ML), it's both hardware and software vendors' interest to have the common one as the interface between software and hardware, so that software guys can focus on frontend and compilation (till to the common IR layer) and hardware guys can focus on design customized and high performant chips based on the common IR.

Whether reusing ONNX as the core dialect or creating a new one?

For ONNX, from the very first day when it's born, it targets to be the common IR between hardware and software (for ML). This leads to its full openness, and finally its fairly nice eco-system (including all software and hardware following ONNX). However, I echo the concern of " the baggage that comes from the existing op sets can inhibit the search for a "best" common set of capabilities defined at the right level of granularity", more than 100 operators looks not a proper set ops for a common IR/dialect. ONNX community also realized this issue so that a composed operator concept (Function) was designed to fix the operator explosion issue. A function is a composed op which builds its semantics based on primitive ops. I'd suggest folks to join the efforts to design this "common" IR/dialect and come out of a common primitive op set for ONNX (all other ONNX ops should be generic function calls designed in the dialect).

btw, ONNX community is fully open (under LFAI foundation), the concern of "control" should not be an issue, I believe :).

[Mehdi AMINI]

(Catching up on this thread with some delay)

To me there is a question of goals: as you mention ONNX seems like first an exchange format more than a compiler IR. There is likely a place for such an exchange format, but it this what we'd want to base our compiler IR and our transformation on, or is this something we would import and then map onto a compiler IR?

I don't know the answer, but I invite you to participate in the thinking about the compiler IR part we proposed here: https://llvm.discourse.group/t/development-of-high-level-tensor-compute-primitives-dialect-s-and-transformations/388/22 

I also invite you to open a discussion about the possible role of ONNX in LLVM/MLIR on Discourse, as this mailing-list is oriented towards TensorFlow and the use of MLIR in TensorFlow.

Cheers,

-- 

Mehdi