Using Amazon Device Farm for integration testing

[Nathan Samson]

Hi

Currently we are using (android and iOS) emulatators on our Mac CI machine to do some basic testing.

This is very limited (with them being emulators)

Ideally we would expand to some other system live Amazon Device Farm (https://aws.amazon.com/device-farm/)

Sadly enough there is no documentation around this (if it works, or how it works)

I could find 2 references online

* https://forums.aws.amazon.com/message.jspa?messageID=851217 Which claims there is no support yet, but a feature request ticket was made. Not sure what the authority of this account is

* The other is https://github.com/flutter/flutter/issues/18879, which mentions "is the only test we have that makes sure that we are able to run tests on off-the-shelf infrastructures, such as Firebase Test Lab, Saucelabs, AWS Device Farm, etc."

But I can't find any instructions anywhere how to run it on these off-the shelf infrastructures

Regards,

Nathan

[Daniel]

Nathan,

I have the same interest in cloud device testing as you. Did you end up finding any resources or solutions?

Daniel

[Maurice McCabe]

I was able to get an integration test running on an android device on AWS Device Farm. Still working on getting a test to run on an iOS device. I will try to post on example on https://github.com/mmcc007/fledge or someplace similar if there is an interest.

I followed the pointers provided in https://stackoverflow.com/a/53328272/1420881

Involved writing a script to call the app and handle the port number.

--Maurice

[eseidel]

I (and I suspect several members of the flutter teams) would be very interested to hear your results when you have them. :)

[Daniel]

The stackoverflow post is actually from me. I was able to get the app up and running, then get the ip and port number from a script (using adb logcat) but I was never able to connect using "flutter drive --use-existing-app". It always resulted in a timeout (even when run locally on a pretty minimal app)

I'd love to see your example if you've got one.

On Tuesday, February 19, 2019 at 9:48:15 AM UTC-8, Maurice McCabe wrote:

[Maurice McCabe]

Totally missed these most recent posts (... my bad!)... since my last post here I have been able to automate running an integration test on android from the command line (using the 'aws' interface to the Device Farm API). 

I posted the above-mentioned script I wrote to a related issue I filed about flutter_driver:
https://github.com/flutter/flutter/issues/28662

I hope the issue is accurate!

I will post all code shortly to github... then on to getting it to work for iOS!

On Friday, September 7, 2018 at 1:37:57 PM UTC-7, Nathan Samson wrote:

[Maurice McCabe]

fyi: here is the test spec log and video (attached) from Device Farm (probably only need to install dart):

[DEVICEFARM] Setting up your device. This usually takes 2-3 minutes.

[DEVICEFARM] ########### Start executing testspec ###########

 

 

[DEVICEFARM] ########### Entering phase install ###########

 

 

[DEVICEFARM] ########### Entering phase pre_test ###########

 

[DeviceFarm] echo "Install flutter"

Install flutter

[DeviceFarm] if [ $DEVICEFARM_DEVICE_PLATFORM_NAME = "Android" ]; then

    # Run EC2 setup code here

    curl https://storage.googleapis.com/flutter_infra/releases/stable/linux/flutter_linux_v1.0.0-stable.tar.xz -o flutter_linux.tar.xz

    tar xf flutter_linux.tar.xz

fi

if [ $DEVICEFARM_DEVICE_PLATFORM_NAME = "iOS" ]; then

    # Run Mac setup code here

    curl https://storage.googleapis.com/flutter_infra/releases/stable/macos/flutter_macos_v1.0.0-stable.zip -o flutter_macos.zip

    unzip -qq flutter_macos.zip

fi

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

 12  231M   12 28.5M    0     0  26.8M      0  0:00:08  0:00:01  0:00:07 26.8M

 35  231M   35 82.3M    0     0  39.8M      0  0:00:05  0:00:02  0:00:03 39.8M

 60  231M   60  139M    0     0  45.6M      0  0:00:05  0:00:03  0:00:02 45.6M

 81  231M   81  188M    0     0  46.3M      0  0:00:04  0:00:04 --:--:-- 46.3M

100  231M  100  231M    0     0  47.6M      0  0:00:04  0:00:04 --:--:-- 48.3M

[DeviceFarm] mv flutter ~/flutter

[DeviceFarm] echo "export PATH=$PATH:$HOME/flutter/bin:$HOME/flutter/bin/cache/dart-sdk/bin">> ~/.bash_profile

[DeviceFarm] export PATH=$PATH:$HOME/flutter/bin:$HOME/flutter/bin/cache/dart-sdk/bin

[DeviceFarm] flutter doctor

  ╔════════════════════════════════════════════════════════════════════════════╗

  ║ A new version of Flutter is available!                                     ║

  ║                                                                            ║

  ║ To update to the latest version, run "flutter upgrade".                    ║

  ╚════════════════════════════════════════════════════════════════════════════╝

  ╔════════════════════════════════════════════════════════════════════════════╗

  ║                 Welcome to Flutter! - https://flutter.io                   ║

  ║                                                                            ║

  ║ The Flutter tool anonymously reports feature usage statistics and crash    ║

  ║ reports to Google in order to help Google contribute improvements to       ║

  ║ Flutter over time.                                                         ║

  ║                                                                            ║

  ║ Read about data we send with crash reports:                                ║

  ║ https://github.com/flutter/flutter/wiki/Flutter-CLI-crash-reporting        ║

  ║                                                                            ║

  ║ See Google's privacy policy:                                               ║

  ║ https://www.google.com/intl/en/policies/privacy/                           ║

  ║                                                                            ║

  ║ Use "flutter config --no-analytics" to disable analytics and crash         ║

  ║ reporting.                                                                 ║

  ╚════════════════════════════════════════════════════════════════════════════╝

Doctor summary (to see all details, run flutter doctor -v):

[✓] Flutter (Channel stable, v1.0.0, on Linux, locale en_US.UTF-8)

Android sdkmanager tool not found (/opt/dev/android-sdk-linux/tools/bin/sdkmanager).

Try re-installing or updating your Android SDK,

visit https://flutter.io/setup/#android-setup for detailed instructions.

[DeviceFarm] flutter devices

1 connected device:

SM G960U1 • 4852393932573398 • android-arm64 • Android 8.0.0 (API 26)

 

[DEVICEFARM] ########### Entering phase test ###########

 

[DeviceFarm] echo "Navigate to test package directory"

Navigate to test package directory

[DeviceFarm] cd $DEVICEFARM_TEST_PACKAGE_PATH

[DeviceFarm] echo "Start Flutter integration test"

Start Flutter integration test

[DeviceFarm] cd flutter_app

[DeviceFarm] if [ $DEVICEFARM_DEVICE_PLATFORM_NAME = "Android" ]; then

  ./script/test_android.sh com.example.flutterapp test_driver/main_test.dart

fi

if [ $DEVICEFARM_DEVICE_PLATFORM_NAME = "iOS" ]; then

  # ./script/test_ios.sh --test com.example.flutterApp

  flutter -v run

fi

Starting Flutter app com.example.flutterapp in debug mode...

failed to clear the 'main' log

Starting: Intent { act=android.intent.action.RUN flg=0x20000000 cmp=com.example.flutterapp/.MainActivity (has extras) }

Observatory on port 46245

Local port 4723 forwarded to observatory port 46245

Running integration test test_driver/main_test.dart on app com.example.flutterapp ...

Running "flutter packages get" in flutter_app...                14.1s

00:00 [32m+0 [0m: end-to-end test (setUpAll) [0m

[info ] FlutterDriver: Connecting to Flutter application at http://127.0.0.1:4723

[trace] FlutterDriver: Isolate found with number: 55156156

[trace] FlutterDriver: Isolate is not paused. Assuming application is ready.

[info ] FlutterDriver: Connected to Flutter application.

00:01 [32m+0 [0m: end-to-end test tap on the floating action button; verify counter [0m

00:02 [32m+1 [0m: end-to-end test (tearDownAll) [0m

00:02 [32m+1 [0m: All tests passed! [0m

[DeviceFarm] echo done.

done.

 

[DEVICEFARM] ########### Entering phase post_test ###########

 

 

[DEVICEFARM] ########### Finish executing testspec ###########

 

[DEVICEFARM] ########### Setting upload permissions ###########

 

chmod: cannot access ‘/tmp/customer_log_directoryJpS0PM/*’: No such file or directory

 

[DEVICEFARM] Tearing down your device. Your tests report will come shortly.

[Maurice McCabe]

BTW: depending on your needs, instead of using real devices, it's possible to use the Genymotion emulator to run integration tests (fast) in CI/CD using an ssh tunnel to the emulator. Though they charge for each usage, it's more convenient and maybe cheaper alternative to running on real devices in the cloud. Though it only works for android. The iOS simulator is much easier to use in the cloud (though slower). 

Here is an example of running the Flutter Gallery integration test on Genymotion emulator in the cloud (with link to source code):

https://travis-ci.com/mmcc007/test_flutter_saucelabs/jobs/180159349

[Maurice McCabe]

I beefed-up the prototype of running a flutter integration test on a real android device running on device farm. It is implemented as a dart package.

Just released the source code for the first version. It includes a live demo which ends with publishing the artifacts generated on device farms during the test run (including a video). 

You can find the source code here:

https://github.com/mmcc007/sylph

The log of the live run here:

https://travis-ci.com/mmcc007/sylph

The resulting artifacts (including video) here:

https://github.com/mmcc007/sylph/releases

Still have to resolve the problem with running on iOS devices.

At their invitation, I am scheduling a second phone call with device farm... probably early this week. This time with engineers and business dev. So there is some ongoing interest there.

There is a disconnect between Flutter Tooling and the Device Farm API. There is a workaround for android (as implemented in code). There is so far no way found for running a Flutter integration test on iOS via the device farm API (have not spent a lot of time on it). 

Hopefully the upcoming meeting will assist.

Eric,

if a Flutter Team member wants to be included in upcoming call with device farm engineer(s), please let me know ASAP, my email is mmc...@gmail.com. I suspect the device farm team engineer would be delighted to talk to a flutter team member. That would certainly help to focus on a fix for the iOS issue.

--
You received this message because you are subscribed to the Google Groups "Flutter Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flutter-dev...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alex Fallenstedt]

Maurice,

How did the conversation with AWS Device Farm engineers go? Was there some progress made on this?

Also thank you for your work with sylph!

To unsubscribe from this group and stop receiving emails from it, send an email to flutt...@googlegroups.com.

[Maurice McCabe]

Hi Alex,

I had a one hour phone meeting with 2 of the AWS Device Farm engineering team and a biz dev back in March. They showed great interest in understanding the problem and learning more about the details of the Flutter build chain for iOS and how it might integrate with Appium (using a workaround similar to the workaround that worked with the Flutter build chain for Android). They expressed confidence that they could fix the problem. 

I had filed an issue with the Flutter team about this problem back in Feb which was resolved in April (and which I had since forgotten about). Time permitting, I will take another look at it.

https://github.com/flutter/flutter/issues/28662

This issue is being tracked in Sylph with

https://github.com/mmcc007/sylph/issues/7

[Maurice McCabe]

After a lot of experiments, I found a way to run integration tests on iOS devices. 

So it is now possible to run Flutter integration tests on Device Farm for both iOS and Android devices. 

I released a new version of Sylph with this new feature:

https://pub.dev/packages/sylph

Sylph runs from the command line (mac is required if testing on iOS devices).

Sylph also supports building in a CI environment (no local mac required) for doing CI/CD:

https://travis-ci.com/mmcc007/sylph/builds/116507297

For the Device Farm artifacts (including video) from the CI run see:

https://github.com/mmcc007/sylph/releases/tag/0.0.9

Thanks to the Flutter and Device Farm teams for their support! 

Let me know if you want new features or find bugs.

--Maurice

[aditya mvs]

Hi Maurice

Any idea on how we can have have integration tests built using flutter_driver run on Firebase Test Lab (Device Cloud)? As Flutter and Firebase are both from Google, I am looking for an integration piece. ANy help is highly appreciated. 

Regards,

Aditya

[Maurice McCabe]

I looked into this a few months ago and wrote something up:

https://github.com/mmcc007/sylph/issues/90