Running Sandstorm's browser tests

[Troy Farrell]

Since I've been working on https://github.com/sandstorm-io/sandstorm/pull/3463, I've

been trying to get the in-browser tests working.  I haven't succeeded yet.  Perhaps you can help.  Here's what I've done, using a vagrant-spk VM:

vagrant@localhost:~$ lsb_release -a
No LSB modules are available.
Distributor ID:    Debian
Description:    Debian GNU/Linux 9.13 (stretch)
Release:    9.13
Codename:    stretch
vagrant@localhost:~$ sudo apt install build-essential libcap-dev xz-utils zip unzip strace curl discount git python zlib1g-dev golang-go cmake strace

vagrant@localhost:~$ curl https://install.meteor.com/?release=1.8.2 | sh
vagrant@localhost:~$ git clone https://github.com/sandstorm-io/sandstorm.git
vagrant@localhost:~$ cd sandstorm/
vagrant@localhost:~/sandstorm$ make bundle

vagrant@localhost:~/sandstorm$ make test
tests/run-local.sh sandstorm-0-fast.tar.xz test-app.spk
Couldn't find executable 'firefox' - try installing the firefox package?
Makefile:167: recipe for target 'test' failed
make: *** [test] Error 1

vagrant@localhost:~/sandstorm$ sudo apt install firefox-esr default-jre-headless xvfb

vagrant@localhost:~/sandstorm$ make test

At this point, the test fails with a generic error.  So I ran the command I found in Makefile:

vagrant@localhost:~/sandstorm$ bash tests/run-local.sh sandstorm-0-fast.tar.xz test-app.spk

This also fails.  Reading tests/run-local.sh, I figure that I need this:

vagrant@localhost:~/sandstorm$ cd tests
vagrant@localhost:~/sandstorm/tests$ xvfb-run --server-args="-screen 0, 1280x1024x24" node_modules/.bin/selenium-standalone start &
vagrant@localhost:~/sandstorm/tests$ tmp-sandstorm/sandstorm start
vagrant@localhost:~/sandstorm/tests$ bash ../find-meteor-dev-bundle.sh

This gives me the path /home/vagrant/.meteor/packages/meteor-tool/.1.11.1.pnhw2i.xi5lh++os.linux.x86_64+web.browser+web.browser.legacy+web.cordova/mt-os.linux.x86_64/dev_bundle.  Next, I can run the tests with npm test.

vagrant@localhost:~/sandstorm/tests$ /home/vagrant/.meteor/packages/meteor-tool/.1.11.1.pnhw2i.xi5lh++os.linux.x86_64+web.browser+web.browser.legacy+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/npm test

This fails because it cannot find Chrome.  Why did we install Firefox?

vagrant@localhost:~/sandstorm/tests$ sudo apt install chromium chromiumdriver
vagrant@localhost:~/sandstorm/tests$ /home/vagrant/.meteor/packages/meteor-tool/.1.11.1.pnhw2i.xi5lh++os.linux.x86_64+web.browser+web.browser.legacy+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/npm test

Fails.  "This version of ChromeDriver only supports Chrome version 87".  Where is my eye roll emoji?  Let's replace the chromedriver binary with the one apt installed.

vagrant@localhost:~/sandstorm/tests$ find . -name chromedriver -type d
./node_modules/selenium-standalone/.selenium/chromedriver
vagrant@localhost:~/sandstorm/tests$ cd node_modules/selenium-standalone/.selenium/chromedriver
vagrant@localhost:~/sandstorm/tests/node_modules/selenium-standalone/.selenium/chromedriver$ ls
latest-x64-chromedriver  latest-x64-chromedriver.zip
vagrant@localhost:~/sandstorm/tests/node_modules/selenium-standalone/.selenium/chromedriver$ mv latest-x64-chromedriver latest-x64-chromedriver.new
vagrant@localhost:~/sandstorm/tests/node_modules/selenium-standalone/.selenium/chromedriver$ which chromedriver
/usr/bin/chromedriver
vagrant@localhost:~/sandstorm/tests/node_modules/selenium-standalone/.selenium/chromedriver$ ln -s /usr/bin/chromedriver latest-x64-chromedriver
vagrant@localhost:~/sandstorm/tests/node_modules/selenium-standalone/.selenium/chromedriver$ cd ~/sandstorm/tests
vagrant@localhost:~/sandstorm/tests$ /home/vagrant/.meteor/packages/meteor-tool/.1.11.1.pnhw2i.xi5lh++os.linux.x86_64+web.browser+web.browser.legacy+web.cordova/mt-os.linux.x86_64/dev_bundle/bin/npm test

Success!  "TEST FAILURE:  22 assertions failed, 0 passed. (1m 59s)"  At least they ran!  I've attached the test output as sandstorm-tests-failing-1.txt.

I haven't solved the test failures yet.  Let me know if I'm doing something wrong.  It should not be this hard to run the test suite.

Thanks.

Troy

[Troy Farrell]

I think that I've solved the riddle.  It occurred to me that the loginDevAccountFast function would only be present when Sandstorm is in development mode, so if it is missing, the tests are not connecting to a development server.  I looked at the test code to see how the tests find the server and realized that I needed to export LAUNCH_URL=http://local.sandstorm.io:9000.  Now most of the tests pass.  I've attached my current test results.

[Ian Denhardt]

Glad you figured it out. The log you attached looks similar to the
failures on master:

https://github.com/sandstorm-io/sandstorm/runs/1444169254#step:11:1970

...Unfortunately some time between shutting the old Jenkins setup run
by box that Sandstorm INC and (years later) setting up CI with GitHub
actions, some failures snuk in, and we haven't managed to fix all of
them yet (it looks like we've actually backslid a bit; we should
probably try to find a way to mark tests as "expected to fail" so we
don't have to remember to check whether the tests failing in CI are the
ones we expect).

But in any case, looks like you're all set up.

Happy hacking,

-Ian

Quoting Troy Farrell (2020-12-15 13:11:27)

> --
> You received this message because you are subscribed to the Google
> Groups "Sandstorm Development" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to [1]sandstorm-de...@googlegroups.com.
> To view this discussion on the web visit
> [2]https://groups.google.com/d/msgid/sandstorm-dev/b2284c9f-0ffb-45fe-9
> a1a-0fe9f7cc5e6en%40googlegroups.com.
>
> Verweise
>
> 1. mailto:sandstorm-de...@googlegroups.com
> 2. https://groups.google.com/d/msgid/sandstorm-dev/b2284c9f-0ffb-45fe-9a1a-0fe9f7cc5e6en%40googlegroups.com?utm_medium=email&utm_source=footer