Creating runfiles tree is very slow - because of hermetic Python

[Alexander Pivovarov]

Hello Everyone

Couple month ago one commit was added to XLA main

Use hermetic Python in TSL and XLA

https://github.com/openxla/xla/commit/90a69e9c16ac949f3d93734cdaca5349ad14b652

The issue is:

- It affects mlir/hlo tests time by several minutes (for "Creating runfiles trees" step)

- It uses gigabytes of disk space for duplicates of pypi_lit and python_x86_64-unknown-linux-gnu folders. Total 161,343 additional files

Command I used: bazel test //xla/mlir_hlo/tests/...

Before / After stats:

- Size of bazel-out/k8-opt/bin/xla/mlir_hlo/tests/Dialect/mhlo
before: 40M
after: 1.8G !!!

- Creating runfiles tree
before: 1 second
after: 110 seconds !!!
Content of typical xxx.mlir.test.runfiles folder:Before (size 476K):
MANIFEST __init__.py llvm-project xla

After (size 22M):
MANIFEST  __init__.py  llvm-project  
pypi_lit  python_x86_64-unknown-linux-gnu  xla

MANIFEST number of lines
before: 101 linesafter: 4815 lines
bazel test //xla/mlir_hlo/tests/creates 63 mlir.test.runfiles folders containing the same copy of pypi_lit  and python_x86_64-unknown-linux-gnu  subfolders python_x86_64-unknown-linux-gnu folder contains 2490 filespypi_lit contains 71 files

Total 161,343 additional files !!! (for 63 tests)

Is it possible to give particular options/flags to bazel to use old fast approach?

[Mitko (Dimitar) Asenov]

+Vadym Matsishevskyi, is this run-time increase an expected side-effect of using hermetic Python? Is there something that can be done to speed up the build and make it more similar to how it was before?

--
You received this message because you are subscribed to the Google Groups "OpenXLA Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openxla-discu...@openxla.org.
To view this discussion on the web visit https://groups.google.com/a/openxla.org/d/msgid/openxla-discuss/CAKKt98Q%3D-jiv6o6VJPheULC9oPzwaxCVUH37srJtnPA3GeyPdw%40mail.gmail.com.
For more options, visit https://groups.google.com/a/openxla.org/d/optout.