Support for Randomness in TestLab RoboTests

[salesfor...@gmail.com]

Hi, 

Currently the Robotest simulates the same user activities in the same order when you test an app with a same maximum depth, though this helps in verifying the fixes or finding regression. It kind of takes away the randomness from the test since the max depth currently supported is only from 2 to 50. So there is only 48 unique paths the RoboTest might take within the App and it is going to remain the same for all the runs. For Apps with lot of complexity this may not be enough, when we chose RoboTest, we were thinking that it could explore different user activities that we normally wouldn't cover in manual or automated testing. So it could find issues that we might have missed. It would be really helpful if you could add additional parameter for Randomness or increase the Depth Setting Limit. 

Thanks,

Mahendran

[Doug Stevenson]

Mahendran,

I believe the current algorithm for Robo is intentionally deterministic to help facilitate predictable test results.  For many cases, it would be confusing and even surprising to interpret completely different results on a test run between two APKs that are exactly the same.  Similarly, it would be confusing to add randomness in a written unit test or instrumentation test.  When a test passes, you should be confident that running it any number of times would also only result in passes.

The easiest way to get Robo to explore more behavior, without writing any lines of code, is to simply increase the duration of the Robo test (unless you are finding that it just repeats the same paths somehow).  You can adjust that along with the maximum depth using advanced settings.  The advanced settings are initially hidden in the web UI.  Please read the section "Configuring Robo Test" on this page: https://firebase.google.com/docs/test-lab/robo-ux-test

When an app gets complex and you need to test more paths that Robo isn't finding, that's a good time to start writing instrumentation tests to specifically invoke those paths.  If you find that Robo discovers a lot of your app but misses a few key paths, you could write instrumentation tests for just those paths, and run both Robo and the instrumentation tests to get better coverage.  Eventually, the goal would be to cover more of your app in instrumentation and depend less on Robo, as you find time to write more of those tests yourself.

Doug

[mahendran shanmugavelu]

Hi Doug, 

Thanks for your response.Currently it only goes deep on certain elements of the screen and kind of miss the others. Like the depth setting, we could have a new breadth setting.This would definitely help the app to be explored more, this additional  setting would  increase the unique paths it takes substantially and still be deterministic. 

Thanks,

Mahendran

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-talk+unsubscribe@googlegroups.com.
To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/05db9837-4213-4413-8005-f9f81c440e7c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.