Can Android Lint can be run on the unit tests source code (src/test)?

[Samuel Guirado Navarro]

Hi!

I'm currently working on some Custom Lint Rules and I noticed that Lint doesn't get run on unit tests source code (src/test). Though it does on android tests source code (src/androidTest). What's the reason for that? Is there any way to make it run on both? Does it make even sense? I'm making those questions because one of the Custom Lint Rules I was trying to implement aimed to accomplish the following:

We use a Dependency Injection framework on our project to inject (@Inject) dependencies into Activities and Fragments. In order to unit tests those Activities and Fragments, we use mocks (@Mocks) for those dependencies. Example:

public class MyActivity extends Activity {

@Inject Foo foo;

// ...
// Activity lifecycle methods
// ...
}

public class MyActivityTest {

@Mock Foo foo;

@Inject MyActivity myActivityUnderTest;

// ...
// Unit tests for Activity lifecycle methods
// ...
}

Our Dependency Injection framework will read the @Mock annotation and when instantiating the MyActivity under test it will use a mock instance of Foo instead of a real one. However, it's pretty common that over time, a new dependency gets added to my Activity, but the developer forgets about adding the mock to the test.

public class MyActivity extends Activity {

@Inject Foo foo;
@Inject Bar bar;

// ...
// Activity lifecycle methods
// ...
}

If we run MyActivityTest again, our dependency injection framework will inject Foo as a mock, but Bar will be a real instance because it was not added as a @Mock in the test.

My idea is to write a Custom Lint Rule that is able to analyze the test class and report an error if the list of @Inject fields in the class under test and the list of @Mock fields in the test class don't match.

Does it make sense? Is Lint the proper tool for something like that or should I try to do it with other static analysis tools? Thanks in advance.

Cheers,
Samuel


P.S: This is how the Custom Lint Rule would look like in code:

public class MissingMockDetector extends Detector implements JavaPsiScanner {

private static final Implementation IMPLEMENTATION = new Implementation(
MissingMockDetector.class,
EnumSet.of(Scope.TEST_SOURCES,
Scope.JAVA_FILE));

public static final Issue ISSUE = Issue.create("MissingMock",
"All injected fields should be mocked",

"In your test class, if you do not mock (@Mock) " +
"all the injected fields (@Inject) from your class " +
"under test, you could get unexpected (flaky) test results.",

Category.CORRECTNESS,
6,
Severity.ERROR,
IMPLEMENTATION);

public MissingMockDetector() {
}

@Override
public List<Class<? extends PsiElement>> getApplicablePsiTypes() {
final List<Class<? extends PsiElement>> types = new ArrayList<>(1);
types.add(PsiClass.class);
return types;
}

@Override
public JavaElementVisitor createPsiVisitor(JavaContext context) {
return new MissingMockChecker(context);
}

private static class MissingMockChecker extends JavaElementVisitor {

private final JavaContext context;

private MissingMockChecker(JavaContext context) {
this.context = context;
}

@Override
public void visitClass(PsiClass aClass) {
PsiClass subjectUnderTestClass = null;

final Set<String> injectedMocks = new HashSet<>();

for (PsiField field : aClass.getAllFields()) {
final String fieldTypeClassName = ((PsiClassType) field.getType()).getClassName();
// I'm assuming that the class under test is named as the test class without the "Test" prefix. Ideally this would be easier with a custom annotation. E.g: @SubjectUnderTest
if (aClass.getName().equals(fieldTypeClassName + "Test")) {
subjectUnderTestClass = ((PsiClassType) field.getType()).resolve();
} else {
for (PsiAnnotation annotation : field.getModifierList().getAnnotations()) {
final String annotationName = annotation.getQualifiedName();
if ("Mock".equals(annotationName)) {
injectedMocks.add(fieldTypeClassName);
break;
}
}
}
}

if (subjectUnderTestClass == null) {
return;
}

for (PsiField field : subjectUnderTestClass.getAllFields()) {
for (PsiAnnotation annotation : field.getModifierList().getAnnotations()) {
final String annotationName = annotation.getQualifiedName();
if ("Inject".equals(annotationName) && !injectedMocks.remove(((PsiClassType) field.getType()).getClassName())) {
context.report(ISSUE, aClass, context.getLocation(field), String.format("The following injected field is not being mocked in %s", aClass.getQualifiedName()));
break;
}
}
}
}
}
}

[Tor Norbye]

This is actually a new behavior/feature in 2.4.

First, lint only runs some lint checks on test sources -- specifically, those that indicate that they apply to tests.

There's only one such check in the builtin set of checks: The unused resource check. If that one didn't look at test code at all, then lint would tell you a resource was unused, and you'd delete it, and then your build would fail.

Similarly, if you write a specific check that apply to tests (such as your scenario), obviously you'd want that check to look at the test files!

The way you do this is by adding the TEST scope to your issue:

new Implementation(

            UnusedResourceDetector.class,

            EnumSet.of(Scope.MANIFEST, Scope.ALL_RESOURCE_FILES, Scope.ALL_JAVA_FILES,

                    Scope.BINARY_RESOURCE_FILE, Scope.TEST_SOURCES));

However, it's possible that some users actually want to have all the checks apply to their tests too. If they want that, they can opt into it with this lint option in build.gradle:

android {

    lintOptions {

        checkTestSources false

    }

}

With that set, lint will behave as in previous versions -- all checks will be run through all source files.  

(By the way, there's a similar change for generated sources; lint no longer looks at those -- there's no way to opt an individual detector in (is there a use case for this?), but if you want to run lint checks on all generated source you can do that with lint option "checkGeneratedSources true".)

HOWEVER: The discrepancy between test/ and androidTest/ you discovered was not intentional -- that was a bug. I've just fixed that (as well as another problem where the wrong detectors were passed to the UAST visitor for test sources).

I've fixed both problems, and they'll be included in preview 8. Thanks for bringing this up!

-- Tor

[Samuel Guirado Navarro]

Hi Tor!

Thank you very much for your detailed response. As you can see in my message I'm already adding the Scope.TEST_SOURCES to my Lint Rule's Issue implementation:

private static final Implementation IMPLEMENTATION = new Implementation(
MissingMockDetector.class,
EnumSet.of(Scope.TEST_SOURCES,
Scope.JAVA_FILE));

That's how I found that the Lint was running on the androidTest source code and not unitTest source code. Anyway, I appreciate the time that you took to look into it and I'm glad to hear that the bug has been fixed and will be included in preview 8. Looking forward to use my Custom Lint Rule.

Regards,
Samuel

[Tor Norbye]

I just noticed that I had a typo in my post; 

On Friday, April 28, 2017 at 5:10:27 AM UTC+2, Tor Norbye wrote:

android {

    lintOptions {

        checkTestSources false

    }

}

Obviously this should be "checkTestSources true" if you want the old behavior with all warnings running on test sources too.

-- Tor

[Anthony F.]

Hi,

  As not many people build custom lint rules for unit test class, I just arrived here.

  I have the same issue, that my custom lint rule doesn't parse test files (I'm in kotlin)

  I added the test scope to my rule, and even tried checkTestSources..

Any idea ?

[Anthony F.]

I Tor,

  There is no much discussion about lint rules for testcase... so I arrived here.

  I'm in the same context and issue, the lint rule I developed do not want to run through my junit unit test.

  

  I enabled checkTestSources

  And I setup my rule with  Scope.TEST_SOURCES
  but nothing made it working.

  Any idea ?

Anthony

Le vendredi 8 septembre 2017 09:00:42 UTC+2, Tor Norbye a écrit :

[Tor Norbye]

It's supposed to work, and is working for my unit tests.

Can you share more details about exactly what you're doing in case I can spot the problem?

-- Tor

[Thomas Chen]

I want to post some of my findings here. The root cause could be related to gradlePluginVersion in your project. I was using `gradlePluginVersion = '7.0.0-alpha10'` and with that I can consistently repro this issue. However, after roll it back to v4.1.0 the issue is gone. 

[Tor Norbye]

I can reproduce this with 7.0.0-alpha10, but not with a more recent build -- such as 7.0.0-beta05 (or 7.1.0-alpha03). Can you try updating?

-- Tor