I mainly use IntelliJ to develop Error Prone, and I have exclusions defined for the 'testdata' directories to prevent it from trying to compile them:
$ find . -name '*.iml' | xargs grep testdata
./core/error_prone_core.iml: <excludeFolder url="file://$MODULE_DIR$/src/test/java/com/google/errorprone/bugpatterns/android/testdata" />
./core/error_prone_core.iml: <excludeFolder url="file://$MODULE_DIR$/src/test/java/com/google/errorprone/bugpatterns/flogger/testdata" />
./core/error_prone_core.iml: <excludeFolder url="file://$MODULE_DIR$/src/test/java/com/google/errorprone/bugpatterns/inject/testdata" />
./core/error_prone_core.iml: <excludeFolder url="file://$MODULE_DIR$/src/test/java/com/google/errorprone/bugpatterns/testdata" />
./core/error_prone_core.iml: <excludeFolder url="file://$MODULE_DIR$/src/test/java/com/google/errorprone/refaster/testdata" />
./core/error_prone_core.iml: <excludeFolder url="file://$MODULE_DIR$/src/test/java/com/google/errorprone/suppress/testdata" />
Possibly we should be checking those .iml files into version control so they're shared.