Android testing musings Part 2

[Rakesh]

Hi,

I wrote earlier this year about my move to an Android team and how
surprised (shocked? disappointed?) I was at the woeful state of
android testing.

I embarked on a strategy of automated, black box testing using Appium
and Genymotion hoping that would solve the problem.

That turned out to be a partial success but now I know thats not
really a solution for 3 reasons:

1. The tests are really flaky. Every few runs the Genymotion emulator
would crash. Fixing this is not within my power. Also adb is
unreliable at times. I asked around (there are a few other teams doing
the same thing) and it turns out they use real devices and run the
tests against them manually. The teams are QA teams as opposed to
developers.
2. Turns out mocking the dependencies is really hard. External web
services are not too bad but my app communicates with a UPnP device on
the local wan. Its too difficult to mock it properly. It doesn't help
that documentation for the device is limited.
3. "Don't test your app through the UI". A lesson hard learned from my
web days. And thats what I was doing. If I had overcome the flakiness
issues I would then be faced with many long running tests that would
be hard to maintain. I'm reminded of JB Rainsberger's talk about
Integration tests are a scam(http://vimeo.com/80533536). He's right
you know, I won't be able to cover all the tests needed to get full
coverage this way.

So back to basics I thought. We had some time so I convinced the team
to move to Android Studio. Took a bit of effort but we have it
working. Gradle builds. Nice and modern. IntelliJ as an IDE. I love
that IDE from my server side days. I got my key bindings back. I felt
good.

Right, unit testing. Thats the way to go. Make me feel better about
not having written any in months.

Crap, Google still only support Junit3. Tests look terrible,
unreadable. I have to teach my fellow devs how to write unit tests,
this was not going to fly.

Robolectric right? Thats what everyone is saying is the way to go on
Android. Been around for ages.

I started looking into using Robolectric with my project. Well now
that I'm using Android Studio, it should be easy. I Google'd around.
Then irony hit me in the face like a sucker punch.

I have to do WHAT to make it work? Hack the iml file! Thats rubbish
but maybe its a one-time deal. Nope, it turns out that I have to hack
it each time I introduce a new dependency. I did it anyway. The test
still didn't run in the IDE. It couldn't find the test class. Weird,
let me research some more.

So the fix is (and I'm not joking), run the test, copy and paste the
runtime classpath that was used by the IDE into a text editor. extract
the -classpath parameter (only a few thousand characters long), make
sure junit4 is first and add to the end the location of the test
classes directory. Then take this and pass as a parameter to the VM
during a test run.

WTF!

If something is this convoluted, you know it stinks. I can't do it. I
feel dirty. Its hard enough to convince my team that unit testing is
the professional thing to do and then show them this work around to
get it working? I would lose all credibility.

I remember watching the Google IO talk with the Android team. You
could see them squirming when asked about testing on Android. Why such
poor support for Robolectric considering the out of the box tools were
so poor? Why don't Google buy Genymotion as its a MUCH better emulator
than the stock Android one?

So my current plan? Create a pure Java module, move as much logic as I
can into it and write unit tests there (using Spock, my preferred unit
testing tool). I'll have to keep Android specific code in the main app
(untested) but hopefully it will be things like downloading
functionality that I can make generic and hope it works through manual
testing.

All the new libraries like Picasso, Retrofit, Rx Java, Otto, Dagger
are great but they need the Android SDK which means I won't be able to
write tests with them. Maybe I can get away with it with my approach.
Not sure.

Imagine a man, who has just experienced some tragedy, on his knees (in
the rain) looking up to the sky and saying "Why me God?". Of course he
gets no reply.

In the same vain, I ask "Tor, if my years of devotion to the
JavaPosse, since episode one (I'm a true believer) mean anything to
you, can you please tell me if there is a plan to make unit testing
easier in the future? If anyone knows its you. You work on Android
Studio!"
Of course I expect no reply.

But just maybe?

Rakesh

[Cédric Beust ♔]

On Wed, Oct 1, 2014 at 1:56 PM, Rakesh <rakesh.m...@gmail.com> wrote:

Crap, Google still only support Junit3.

Sadly, I have to bear the blame for that.

In the early days of Android (I should say the first days of Android), the director of engineering came to me and asked me "Do you think we should use TestNG for Android?".

I was not prepared for this question so I told him I'd get back to him. I spent the next few days going over the TestNG code base (that was circa 2007 so TestNG was just three years old). Obviously, I knew that code base very well but I had written it with server testing in mind, certainly not with the intent of running on a mobile device, especially on an OS that we were still in the process of writing.

After a couple of days, I got back to him and I told him "No, I don't think we should make TestNG the standard on Android, I just didn't write it with this in mind and I'm afraid that it just won't perform well".

So we went with JUnit 3.

I'm sorry.

-- 

Cédric

[Rakesh]

you know, I actually had an inkling this is what happened but I chose
to say nothing because I had no proof.

However, I don't blame you because its 2014 and no one moved to JUnit
4 in the intervening years! Thats what I'm really pissed about. We all
start projects with the best tools available at the time. That does
not mean we stay with them forever!

Rakesh

> --
> You received this message because you are subscribed to the Google Groups
> "Java Posse" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to javaposse+...@googlegroups.com.
> To post to this group, send email to java...@googlegroups.com.
> Visit this group at http://groups.google.com/group/javaposse.
> For more options, visit https://groups.google.com/d/optout.

[pforhan]

IMO, it's not so much that it's not TestNG, it's that it's Junit 3.  4 would have been fine. 

Android's still got an incredibly obtuse approach to testing with the stub exceptions and pushing you toward running on-device.  So, we work around by pulling all our logic out into non-android classes and test those using standard junit 4 and mockito.  Add in robolectric with no config [@RunWith(RobolectricTestRunner.class) @Config(manifest = Config.NONE)] when you have to.  Picasso, Retrofit, Rx Java, Otto, Dagger -- yep, use them all.

Everything you've said about robolectric and android studio is true, from what I understand.  I always build on the command line with maven (yeah, I know I'm behind).  I have a friend who has claimed he's cracked the Android Studio+robolectric barrier, but he hasn't told me how yet.

Pat.

[Ricky Clarkson]

This looks promising: https://github.com/dthommes/JUnit4Android/wiki

I'd quit worrying about what you can or can't do in the IDE fairly quickly though.

[Rakesh]

Ricky,

its not been updated in 2 years plus the docs are all about Eclipse not AS.

My IDE is VERY important to me because its the tool of my trade and
helps me work faster and more efficiently.

Please don't turn this thread into a debate about to IDE or not to IDE...

[Cédric Beust ♔]

The approach I've adopted regarding IDE's is to be fluent with both IDEA and Eclipse.

Best tool, best job, yaddi, yadda.

-- 

Cédric

[Fabrizio Giudici]

On Wed, 01 Oct 2014 23:13:25 +0200, Cédric Beust ♔ <ced...@beust.com>
wrote:

> So we went with JUnit 3.
>
> I'm sorry.

It was a very prudential decision of you at the time, man. The real
question is why do they stick with JUnit 3 after so many years!! I mean,
why they at least didn't upgrade to JUnit 4?

For the rest, Rakesh, I do second the idea of keeping most of the code
Android independent, so you can test it as you like. It's what I did since
the beginning for my first project. If you also do a good separation of
concerns between the presentation and its controllers, possibly using a
pattern that minimizes/zeroes the logic in the presentation, you gain some
points. My suggested approach is also to keep presentation controller as
clean from Android code as possible, even though this requires some more
effort and it needs to be evaluated on a per-project basis. Of course this
won't solve the problem, because in the end you also need to test the
whole system... and yes, adb is not reliable, and there are the other
problems you mentioned, and you need to do some manual testing. But at
least you minimize the chances of finding bugs at this level and reduce
the number of test rounds.

--
Fabrizio Giudici - Java Architect @ Tidalwave s.a.s.
"We make Java work. Everywhere."
http://tidalwave.it/fabrizio/blog - fabrizio...@tidalwave.it

[Rakesh]

an architectural pattern that I was more and more drawn to in my
server side days was Hexagonal/Ports and Adapters.

I'm a big fan of BDD and the way it would allow me to test the
business logic without the 'real' infrastructure really appealed.

I wonder if we could apply it to Android? Google turned up absolutely
nothing specific to Android.

It did, however, uncover a wealth of videos from the Ruby on Rails
community using Hexagonal architecture to help system scale out.

I was struck by the parallels between the issues on RoR and Android.
In fact, you could swap out Rails for Android in the talks and it
would still apply.

Anyone tried this approach on Android?

Rakesh