RFC: Building out shark_turbine core as iree.turbine

[ste...@laurenzo.org]

Hey folks,

We are in the process of figuring out the 1.0 plan for the shark_turbine/core project. This is presently released as the shark_turbine pip package with a 0.9.x version. We'd like to align its naming and distribution with the broader project as we move forward.

In the original conception of the project, this "Turbine" component augmented IREE to make it interop with PyTorch Dynamo (get it? Turbines are used to build a Dynamo, ba-dum-bum). Since then, it has built out the baseline support for this:

• AOT graph capture, including exposing direct coding of IREE native features
• Eager execution via torch.compile
• IREE based custom kernel registration and dispatch in both eager and AOT modes
• Kernel languages for targeting IREE's different levels of abstraction
• Bridging and transformations for certain categories of model adaptation

We have broader plans for the SHARK Turbine project in general, but we are looking to align this core infrastructure with IREE itself. Note that this core component is already documented as the approach for interopping IREE with PyTorch: https://iree.dev/guides/ml-frameworks/pytorch/#simple-api

Proposal:

We (the SHARK team at AMD) would like to claim the `iree.turbine` Python namespace and distribute these core components as an `iree-turbine` wheel on PyPi. While the project has no policies regarding such borrowing or co-distribution of the namespaces in such a way, it is generally polite to ask and explain. If and when the IREE project becomes neutrally managed by a foundation, we would be open to considering this a sub-project of the whole from a GitHub organization perspective, but for the moment, we would continue to build it out in an AMD managed repository, as it is today.

Alternatives considered:

• Contribute this project directly to the IREE repo: SHARK-Turbine core is intentionally pure Python development and should remain as such, with a canonical Python-oriented organization. It has been working well for us to have it as a standalone, Python-based repository and we would like to prioritize development a Python-native organization and approach for this piece.
• Maintain as-is: Creates confusion on why this piece of critical infrastructure is managed outside of the aegis of the overall project organization.
• Organize it within IREE as `iree.torch`: This would create both an unfortunate naming situation (i.e. the dreaded "which 'import torch' is that?"). Further, we've found that historically, PyTorch progresses through "epochs" with respect to its interop story and eagerly claiming at any point in time that something is "the way" to interface to it has cost a lot of confusion. Layering this with a name that makes it dynamo adjacent but so that there can be another in the future if the situation evolves seems prudent.

Invariants:

We have been using the SHARK Turbine core project to develop core infra for months, and a number of these have already been upstreamed to IREE or torch-mlir (i.e. the fx_importer, the onnx_importer, and various additions to make Python interop work across the projects). We are open to upstreaming further work where there is demand, regardless of what the project is named or where it lives. This proposal is primarily about providing IREE with a unified set of capabilities for interop with PyTorch, but explicitly does not preclude further developments of this type.

Cheers.

- Stella

[benvanik]

+1 to iree.turbine, and keeping it separate to allow for differing integration timelines/easier issue triage/contributor groups and avoiding the dreaded monorepo syndrome

[Jacques Pienaar]

This sounds like a good layering/naming (checked other turbine python packages and wouldn't seem to cause confusion with any of them). Given the use as connector while filling in some programming model aspects, this seems a useful component as part of toolkit.

-- Jacques

--
You received this message because you are subscribed to the Google Groups "iree-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to iree-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/iree-discuss/15ceb6be-c370-4fd1-99d4-5118eabd89e0n%40googlegroups.com.

[ste...@laurenzo.org]

This has now been completed and the new repo is up: https://github.com/iree-org/iree-turbine

We have some secondary CI and connections that are still lingering on the old repo, so that has been left where it was. I'll be mirroring any patches as we move things. Once we only have one, I'll rename the packages in iree-turbine from shark_turbine -> iree.turbine (currently, the latter is just an alias to the former). There's some other cleanup and such that I will hold off on doing until we have one source of truth.

This is an exciting milestone for me personally, as I started this project about 9 months ago, and along the way it birthed multiple new developments and upstreaming activities (both the fx_importer and onnx_importer started in this project and have since been upstreamed to torch-mlir independently). In addition, this gave us a chance to break out and promote one of the sub-projects on the SHARK side (https://github.com/nod-ai/sharktank). Thanks, everyone, for being supportive of the journey. New development at this scale can be a little bit non linear.

I'll be aiming to push a clean version to PyPI in conjunction with the PyTorch 2.3 release.

On Monday, February 12, 2024 at 11:15:58 AM UTC-8 ste...@laurenzo.org wrote: