WILDS v1.2 Release + ICML Announcement

[Shiori Sagawa]

Hi everyone,

We’ve released WILDS v1.2! Please update the package by running pip install -U wilds. Here is a summary of the changes.

New benchmark datasets. We have added two new benchmark datasets: GlobalWheat-WILDS and RxRx1-WILDS. 

The Global Wheat Head detection dataset comprises images of wheat fields collected from 12 countries around the world. The task is to draw bounding boxes around instances of wheat heads in each image, and the distribution shift is over images taken in different locations.This dataset is adapted from the Global Wheat Head Dataset 2021 from David et al., 2021. 

The RxRx1 dataset comprises images of genetically-perturbed cells taken with fluorescent microscopy and collected across 51 experimental batches. The task is to classify the identity of the genetic perturbation applied to each cell, and the distribution shift is over different experimental batches. This dataset is adapted from the RxRx1 dataset released by Recursion.

Paper. We have updated our arXiv paper to include results on GlobalWheat-WILDS and RxRx1-WILDS. In addition, we have added an analysis of distribution shifts in a genomic dataset based on the ENCODE-DREAM in vivo Transcription Factor Binding Site Prediction Challenge. The task is to classify if a given genomic location will be bound by a particular transcription factor, and the distribution shift is over different cell types. 

We have also rewritten and expanded Section 5, which discusses various approaches to measuring the performance drop due to a distribution shift.

Leaderboard. We have updated the leaderboard submission guidelines and added evaluation scripts and other infrastructure to support submission. Our first submission is from Yuge Shi and colleagues at Oxford, Edinburgh, and Facebook, on a gradient matching method called Fish. We’ve been very excited to see all the work that’s being done on WILDS. Thank you to everyone who’s given us feedback and suggestions on the leaderboard!

For more details on the v1.2 update, please see our release notes.

In addition, we will be presenting WILDS at ICML this week as a long talk! The talk is at 6pm Pacific Time on Thursday, July 22, 2021, and the poster session is from 9pm to midnight Pacific Time on the same day. If you’d like to find out more, please drop by https://icml.cc/virtual/2021/poster/10117! (The link requires ICML registration.)

Finally, we are actively developing WILDS and would love to hear how we can make it better for you:

- If you encounter any issues, please report them as Github issues.

- For questions, feedback, and discussions, please head over to Github discussions.

Thank you!

WILDS Team