Turning retention policy on github actions pre&post submit GCS buckets

[Nancy Yuen]

The GCS (Google Cloud Storage) buckets currently keep all items indefinitely.  It's not necessary and so we are going to turn on a retention policy of 60 days on the two largest buckets:

• iree-github-actions-postsubmit-artifacts
• iree-github-actions-presubmit-artifacts

The retention policy will turn on end of next week, 2/16.  Please reply if you can any concerns about the retention policy.

-Nancy

[Scott Todd]

SGTM. I'd also like to switch some workflows from using GCS storage to using GitHub's workflow artifacts if possible (for files under 1-5GB). Workflow artifacts are easier to use from GitHub-hosted runners as they don't require additional authentication. GitHub workflow artifacts can also apply custom retention policies, but I'm not sure yet if there would be billing costs associated with our usage that would prompt us to customize them.

I expect that large benchmark artifacts (.vmfb files with real model weights embedded in them) will continue to use GCS or equivalent cloud storage. We might be able to optimize that down to a scale where workflow artifacts could be an option... depending on what data is being stored today. For example,  we could use fake / splat weights or use parameters with external weights (store frequently changing .vmfb compiled program files in workflow artifacts and store infrequently changing .irpa parameter files in GCS).

Any data that we want to archive for historical analysis (e.g. benchmark results) should definitely stay in a bucket that we have direct access to.

[Jacques Pienaar]

SGTM too.

I wonder for some of the models if we can't take advantage of the parameter files (to be discussed tomorrow): I saw some are different sized variants but could be sharing some of the same data (which could then be extracted and serialized only once). I mean that may require more work (maybe no more than tool that enables specifying a name of variable to extract that only extracts those to parameter file and mutates the module?), but could end reducing size a bit.

-- Jacques

--
You received this message because you are subscribed to the Google Groups "iree-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to iree-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/iree-discuss/3af4cc2b-50d7-468f-9673-d815fecfdf2en%40googlegroups.com.

[Scott Todd]

Parameter files could help, but benchmarks will need to be careful about the current caveats associated with using parameters. Many critical compiler optimizations like const-eval, packing, and load -> transpose rely on the compiler having access to the model weights ahead of time (today).

I took a look at some of the GCS bucket contents and there is lower hanging fruit.

For this postsubmit run: https://github.com/openxla/iree/actions/runs/7878012964 here are the file sizes in the e2e-test-artifacts/ folder: https://gist.github.com/ScottTodd/db7daa932e50ea9c10d3621d586d0e6a . Note the total of ~130GB for one benchmark run.

• The "compile-stats_/module.vmfb" files don't need to be persisted. Dropping those would save half of the storage space.
• The "dt-uk" and "dt-only" variant files are nearly the same size. Those could possibly be compiled into the same module
• The ".mlir" and ".mlirbc" input files are likely shared between jobs. Are those copies without deduplication? Maybe they could use symlinks with checksums or version information to somehow point to some shared storage

With parameters, I noticed a 400MB .mlirbc file could be split into a 300KB .mlir file and a 1.1GB .irpa file (I'd expect the .irpa file to be around 400MB also, need to investigate there). That .irpa file could be stored outside of the bucket if the performance caveats mentioned above were addressed.

[Nancy Yuen]

Reminder, retention policy will be turned end of day today.