Test Lab Infrastructure Failure: Exhausted attempts

[Emily Kowalczyk]

I've been getting some Infrastructure Errors on Test Lab, and was hoping someone might be able to help figure out the cause.

I'm running a single instrumentation test on a hammerhead device running API 21 & 23.  During the test I write a couple files and take some screenshots to external storage, and I want to be able to pull these files into a bucket once the test has finished.

I've been doing this with the below command:

gcloud alpha test android run --timeout 2m --app br.com.threeloops.android.paint.apk --device-ids hammerhead --directories-to-pull /sdcard --locales en_US --os-version-ids 21,23 --test app-debug-androidTest.apk

Below is an example of the terminal output with the error, when I run for a single os-version:  

$ gcloud alpha test android run --timeout 2m --app br.com.threeloops.android.paint.apk --device-ids hammerhead --directories-to-pull /sdcard --locales en_US --os-version-ids 23 --test app-debug-androidTest.apk

Have questions, feedback, or issues? Get support by visiting:

  https://firebase.google.com/support/

Uploading [paint/apps-new/br.com.threeloops.android.paint-signed.apk] to Firebase Test Lab...

Uploading [paint/apps-new/app-debug-androidTest-signed.apk] to Firebase Test Lab...

Raw results will be stored in your GCS bucket at [https://console.developers.google.com/storage/browser/test-lab-XXXXX/2017-01-19_23:15:10.753350_IXwO/]

Test [matrix-1nkfvavbi3pbr] has been created in the Google Cloud.

Firebase Test Lab will execute your instrumentation test on 1 device(s).

Creating individual test executions...done.

Test results will be streamed to [https://console.developers.google.com/project/XXXXXXX/testlab/mobile/histories/XXXXXX/executions/XXXXXXX].

23:15:30 Test is Pending

23:15:48 Starting attempt 1

23:15:48 Test is Running

23:16:20 Installing APK: br.com.threeloops.android.paint

23:16:32 Installing APK: br.com.threeloops.android.paint.test

23:16:38 Logging into the device

23:16:51 Running instrumentation test. Package: br.com.threeloops.android.paint.test testrunner: android.support.test.runner.AndroidJUnitRunner options: []

23:18:18 Instrumentation test has finished

23:18:24 Generating video

23:18:24 Retrieving test artifacts

23:18:24 Test is Pending

23:18:43 Starting attempt 2

23:18:43 Test is Running

23:19:20 Installing APK: br.com.threeloops.android.paint

23:19:32 Installing APK: br.com.threeloops.android.paint.test

23:19:39 Logging into the device

23:19:51 Running instrumentation test. Package: br.com.threeloops.android.paint.test testrunner: android.support.test.runner.AndroidJUnitRunner options: []

23:20:23 Instrumentation test has finished

23:20:23 Generating video

23:20:29 Retrieving test artifacts

23:20:35 Test is Pending

23:20:42 Starting attempt 3

23:20:42 Test is Running

23:21:07 Installing APK: br.com.threeloops.android.paint

23:21:26 Installing APK: br.com.threeloops.android.paint.test

23:21:32 Logging into the device

23:21:45 Running instrumentation test. Package: br.com.threeloops.android.paint.test testrunner: android.support.test.runner.AndroidJUnitRunner options: []

23:23:24 Instrumentation test has finished

23:23:24 Generating video

23:23:30 Retrieving test artifacts

23:23:30 Test is Pending

ERROR: (gcloud.alpha.test.android.run) Firebase Test Lab infrastructure failure: Exhausted attempts

Does any one know what's causing this or how best to work around?  I've been able to run this successfully when I run it on virtual devices and it has occasionally succeeded on the physical device running 23 (although not in the last couple days). 

[Jordan Carlson]

Hi Emily,

The problem is on our end, and has to do specifying "--directories-to-pull /sdcard". We made a change this week that accidentally makes it impossible to pull all of /sdcard. If you can live with only pulling a subdirectory of /sdcard (e.g. /sdcard/Pictures), then that should get things working for you immediately. Or if you can wait a few hours, we'll be rolling back the offending change this afternoon.

Thanks for reaching out. I'll update this thread when the rollback is complete.

--Jordan--

[Jordan Carlson]

Just to close the loop on this, the rollback was successful on Friday evening. As of that time, it should be safe once again to specify "--directories-to-pull /sdcard".

--Jordan--