[dart-announce] All the world's a test, And all the men and women merely testers

[Kevin Moore]

Natalie has been cranking on the test package since the beginning of the year. You should start taking a look. - Kevin

They have their tearDowns and their setUps, and one man in his time tests many units.

Hello Dartisans!

After several months of work, we’re nearing the release of a stable version of the test package, formerly known as unittest. In fact, all the features we wanted for the stable release are in place and all the bugs we know about are squashed. But since this is a pretty important piece of infrastructure for a lot of Dart users, though, we want to make absolutely sure that it’s solid before we put a stamp on it and call it truly stable. So we’re putting out a release candidate: version 0.12.0-rc.1.

The central feature of version 0.12.0 is the test runner. Rather than running each test file individually, they’re run via pub run test:test. This command on its own will run every test in your package, but a single test can also be run with pub run test:test path/to/test.dart, and all the tests in a directory can be run with pub run test:test path/to/dir.

The test runner also makes it way easier to run browser tests. Just pass the --platform flag: pub run test:test --platform chrome. It’ll handle compiling the tests to JS, starting the browser, and loading the tests. All the results will be available right there on the command line (or wherever you ran the test command), side by side with results from the VM or other browsers if you run on multiple platforms at once. Supported browsers include Chrome, Firefox, Safari, Internet Explorer, Dartium, Dartium Content Shell, and PhantomJS.

One of the major design goals of test 0.12.0 is making test configuration declarative. There are several types of declarative metadata that can be attached to tests, groups, and test suites, but the most important is @TestOn, which specifies which platforms a test is supposed to run on. It takes a string which describes the supported platforms. You can check out the full syntax for platform selectors, but for now the most important values are vm and browser. Here’s how you’d write a VM-only test:

@TestOn("vm")

import "dart:io";
import "package:test/test.dart";

void main() {
 // ...
}

To learn more about these and other exciting new features, check out the README.

So what’s next after 0.12.0? For a full list of stuff we have planned, you can check out the 1.0.0 milestone on GitHub, but there are a couple big things that we’re particularly interested in. The biggest of these is debugging support, which we plan to provide in one form or another across all supported browsers and the VM. We’re also going to add a package-wide configuration file for the test runner, and there are a number of packages that we want to help make compatible with 0.12.0 such as scheduled_test, guinness, and mock.

So, try it out! Just change your dependency on unittest to test and bump up the version range:

dev_dependencies:

 test: "^0.12.0-rc.1"

- Natalie

--
For more news and information, visit https://plus.google.com/+dartlang
 
To join the conversation, visit https://groups.google.com/a/dartlang.org/

[Cristian Garcia]

Nice!

BTW: How do you test the "test" library?!?!?!?

--
For other discussions, see https://groups.google.com/a/dartlang.org/
 
For HOWTO questions, visit http://stackoverflow.com/tags/dart
 
To file a bug report or feature request, go to http://www.dartbug.com/new

To unsubscribe from this group and stop receiving emails from it, send an email to misc+uns...@dartlang.org.

[Kevin Moore]

"pub run test" :-)

You need to have every browser supported on your OS to run the tests, though.

...I think

We won't try to run Safari tests on Windows, Linux

Same for IE on Mac, Linux, etc

If you have Chrome on your machine, you could try things out by syncying down the repo and doing 'pub run test -p chrome'

[Günter Zöchbauer]

At last proper HTML testing!
I'm using it since a while and it's great to work with.
Note: Debugging works for most of the tests if they are run as single script already.

[Lasse Damgaard]

Can't wait to use this but upgrading from 0.11 is not an option until the mock package compatibility is fixed. Do you guys have any ETA on this? I would imagine I'm not the only one with a large testsuite that depends on both (unit)test and mock.

 Lasse

[Kevin Moore]

I'm looking into that right now, Lasse.

[Anders Holmgren]

Anyone know if there is any impact to mockito?

[Günter Zöchbauer]

Seems to work fine. I created a PR a while ago to update it to use `test` instead of `unittest` but it wasn't applied yet.

I don't use mockito myself yet, I updated it because there was some dependency with a shelf package (probably one of yours ;-) ) I guess (can't remember exactly)

[Anders Holmgren]

Cool. Thanks Günter

--

[Kevin Moore]

mock 0.12.0-beta.0 is out now - https://pub.dartlang.org/packages/mock

You'll have to specify the beta version explicitly, but it should work.

[DoHyung Kim]

Did anybody try to run tests in browsers on Windows?
I'm on Windows 8.1 and having the following stacktrace:

c:\msys64\home\dynaxis\work\seed>pub run test -p firefox test
Failed to create server socket (OS Error: Failed to start accept), address = loc
alhost, port = 0
dart:isolate  _RawReceivePortImpl._handleMessage

This is an unexpected error. Please file an issue at http://github.com/dart-lang
/test
with the stack trace and instructions for reproducing the error.
Unhandled exception:
Uncaught Error: SocketException: Failed to create server socket (OS Error: Faile
d to start accept), address = localhost, port = 0
Stack Trace:
#0      _NativeSocket.bind.<anonymous closure> (dart:io-patch/socket_patch.dart:
502)
#1      _RootZone.runUnary (dart:async/zone.dart:1155)
#2      _Future._propagateToListeners.handleValueCallback (dart:async/future_imp
l.dart:494)
#3      _Future._propagateToListeners (dart:async/future_impl.dart:577)
#4      _Future._completeWithValue (dart:async/future_impl.dart:368)
#5      _Future._asyncComplete.<anonymous closure> (dart:async/future_impl.dart:
422)
#6      _asyncRunCallbackLoop (dart:async/schedule_microtask.dart:41)
#7      _asyncRunCallback (dart:async/schedule_microtask.dart:48)
#8      _runPendingImmediateCallback (dart:isolate-patch/isolate_patch.dart:96)
#9      _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dar
t:143)

#0      _rootHandleUncaughtError.<anonymous closure> (dart:async/zone.dart:886)
#1      _asyncRunCallbackLoop (dart:async/schedule_microtask.dart:41)
#2      _asyncRunCallback (dart:async/schedule_microtask.dart:48)
#3      _runPendingImmediateCallback (dart:isolate-patch/isolate_patch.dart:96)
#4      _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dar
t:143)

I know that the message says I need to file a bug. But wanted to check with others before doing so.

DH

2015년 4월 25일 토요일 오전 7시 0분 43초 UTC+9, Kevin Moore 님의 말:

[Kevin Moore]

Please do! Include the stack trace and your system config

--

[Burak Emir]

Thanks for all the improvements ... : ) ... but...

  test("documentation up to date", https://www.dartlang.org/articles/dart-unit-tests/) ==> FAIL :D

To unsubscribe from this group and stop receiving emails from it, send an email to misc+uns...@dartlang.org.

[Natalie Weizenbaum]

Can you file an issue for that on http://github.com/dart-lang/dartlang.org/issues ?

[Kevin Moore]

Filed: https://github.com/dart-lang/www.dartlang.org/issues/1338

[Lex Berezhny]

Can you elaborate on how you're managing to debug your browser unit tests? Everything I can find in dart test github repo suggests this is impossible.

 - lex

[Günter Zöchbauer]

I think currently the only way is to use a custom html file and add the Dart test file as script tag.

[Patrice Chalin]

Does the new test package provide/suggest a way to determine if tests are being executed in check vs. production mode? I can write my own boolean isCheckedMode() function and use it, e.g., as a value for the "skip" named parameter, but it might be useful to provide such a functionality as part of the test package, if it isn't there already.

Thanks,

Patrice

[Günter Zöchbauer]

I haven't seen support for this but I also run into a situation where I needed it.
You could just create an issue in the test GitHub repo and see what happens.

[Kevin Moore]

Related: https://github.com/dart-lang/test/issues/50

On Tue, May 12, 2015 at 10:12 PM, Günter Zöchbauer <gzo...@gmail.com> wrote:

I haven't seen support for this but I also run into a situation where I needed it.
You could just create an issue in the test GitHub repo and see what happens.

[Patrice Chalin]

FYI: https://github.com/dart-lang/test/issues/111 - activate or skip tests based on execution mode (checked vs. production)

[Don Olmstead]

@Kevin any chance a new version of stagehand can be pushed that uses the new test library? See the commit to enable it but the pub package is from April.

To unsubscribe from this group and stop receiving emails from it, send an email to misc+uns...@dartlang.org.

[Kevin Moore]

Don: Devon is cranking on it! 

https://github.com/google/stagehand/pull/251

[Don Olmstead]

Sounds good. Will keep a look out for it landing.

[Vadim Tsushko]

I've switched to test on a couple of my projects and generally happy with it.

But I miss old `solo_test` functionality of unit_test. We have all sorts of mean to emulate old skip_test behaviour but what with `solo_test` ?

What was wrong with it?

When I explore some specific breaking test, and in process of modifying related code I really want to run only that breaking test (and generally just run dart on current test file), and remove that solo_ prefix only when that test not break.

Is there some other way to achieve that or something like that in new test package?  

[Jim Trainor]

solo_test == must have IMO

--

[Kevin Moore]

A couple of things.

1) You can run an individual test file (or set of files) by specifying a list of test files/directories in the command line.

pub run test test/integration/

2) You can filter tests at the 'test(...)' level by providing a filter string as either a straight string match or a regex

-n, --name                     A substring of the name of the test to run.

                               Regular expression syntax is supported.

-N, --plain-name               A plain-text substring of the name of the test to run.

Even w/ these, do you still want solo_test?

We've had to fight A LOT to eliminate solo_test when they were unintentionally committed to source repos. We'd don't want to add it back if we can provide other means to get the same behavior.

[Jim Trainor]

I haven't used the new test package yet. I'll have to try it the new way and see...

When running tests from a browser does "#testFilter=my test" still work?

Does the new test package have an equivalent of solo_group?  That is/was useful too.

I can see how the accidental checkin is a problem since I do it all the time... but I'm a team of one at the moment so it doesn't matter for me.

[Kevin Moore]

We haven't done work to support browser tests directly. Only via the pub run command.

There is no support for #testFilter etc.

If you want to filter by a group, you can specify a filter containing the group name.

[Kevin Moore]

Debugging is on our list. We know there are limitations w/ the 'pub run' model.

On Mon, May 25, 2015 at 11:06 AM, Günter Zöchbauer <gzo...@gmail.com> wrote:

The `--name` and `--plain-name` are better replacements. Currently the problem is that debugging doesn't work when tests are run with `pub run test ...`. You just modify you launch configuration to specify which test you want to run. 

--

[Jim Trainor]

No browser test directly.... that puzzles me.  I run my tests regularly on chrome and safari mobile. How do I do that with no direct browser testing - i.e. load via url in browser?

To unsubscribe from this group and stop receiving emails from it, send an email to misc+uns...@dartlang.org.

[Kevin Moore]

If you create an HTML page for your test, that'll work okay – I think.

We have no done work to support manual test control in a browser, though. Filtering by test, etc.

[Jim Trainor]

thank-you, I'll try it

[Natalie Weizenbaum]

On Mon, May 25, 2015 at 11:42 AM, 'Kevin Moore' via Dart Misc <mi...@dartlang.org> wrote:

If you create an HTML page for your test, that'll work okay – I think.

We have no done work to support manual test control in a browser, though. Filtering by test, etc.

This is not officially supported. It works coincidentally now as long as you have a dev inspector, but it'll probably stop working in the future.

The best thing to do if you want to test on a currently-unsupported browser is to add support for it. It shouldn't be too hard; you can copy and modify one of the existing launchers in this directory. I've also filed an issue to add support for running against unknown browsers.

[Jim Trainor]

I'm a bit confused, and maybe I just have to try it. 

When I run my tests they access data on my dev server.  If I launch from DartEditor or Web Storm I disable web security (Dartium --disable-web-security option) so that cross-site requests work.  For everything else I server the test directly from my dev server (which happens to be appengine) so that there is no cross site access.  It sounds like that may not work anymore.

[Günter Zöchbauer]

I'm not sure what you try to accomplish but you could try to pass `--pub-serve=port_of_your_appengine_server`

[Jim Trainor]

I never run pub serve.  Some of my testing habits pre-date the existence of pub serve and I've had no pressing reason to change. Time has come to change my ways perhaps.

--

[Günter Zöchbauer]

I didn't suggest to run pub serve, just to point the testrunner to your actual server using the `--pub-serve` parameter. But I also might totally misunderstand your problem...

[Jim Trainor]

Thanks for the suggestions. 

I just point my browser at  my dev server and select a link that runs my test suite, or parts of it. The test code is all dart2js output driven by the old unittest package. I run dart2js itself from build scripts not from pub serve. That's it, quite simple.  The exception is when I'm running out of the DartEditor or WebStorm in which case I turn off pub serve and host the dart files using whatever server is built into the editor, and I do cross-site access to fetch data from my server in that case.

I'm guess I doing my testing a bit old-school at the moment. But it does have an advantage in that I can point any browser, running on any device/platform/emulator/whatever on my network, at my server and run my full test suite with one click.

[Vadim Tsushko]

понедельник, 25 мая 2015 г., 22:02:27 UTC+5 пользователь Kevin Moore написал:

Even w/ these, do you still want solo_test?

We've had to fight A LOT to eliminate solo_test when they were unintentionally committed to source repos. We'd don't want to add it back if we can provide other means to get the same behavior.

Hi, Kevin.

I wonder what was the main consideration when you eliminated solo_test. Was it costly in some way to support that piece of functionality? Was it conceptually wrong approach? 

As I understand it new recommended scheme is a separation of test code base from configuration artefacts, so we may use filters (thank you for directing in that way)

Maybe it is a good approach in general but I think for some scenarios old way worked better still.

So speaking for myself - I used unit_test exclusively with server-based, mostly async apps. Most of that experience was year and more ago.

Given state of debugging support (especially in async test codebase) - mostly it was a print-debug scheme.

But I believe even now with Sublime Text, vi, emacs and so on - I would have the same.

 

Let me describe my old process: 

Suppose at the usual running of all my tests I see that one test is broken. By clicking at stack trace line, I open concrete test file at the concrete broken test. 

Now I'm looking at offending `expect` method. More often than not at that moment I wand either set a breakpoint (if I'm in IDE and its current state permit me to do so) or print some local object to see it inner state.

Then what? With old `solo_test` method available I just mark current test for solo execution and run current file (for example pressing F7 in Sublime Text)

I'm already just in that location of my code that I wand to investigate further, I marked it for solo execution in that same location, do some iterations (maybe adjusting or adding expectations) and finally remove `solo_` mark and run all tests in a current file again.

What should I do, for example in Sublime Text now, when `solo_test` is eliminated? Probably I should edit Build system, to make it use `filter` parameter with name of current test. It is not especially cool. 

Actually I found that I cut and paste current test on the top of `main` function and insert `return` while I working with that test, effectively achiving same `solo_test` result. Alternatevely I can comment out all tests in file except for current test. It is not cool at all.

I think, maybe test management through configuration is better way when we are testing something, and management through code is better when we are actually coding? On the other way - why `skip` functionality present of code level. Surely we can slip tests through some sort of command line parameters? (Actually I do not mean to propose that, god forbid)

[Günter Zöchbauer]

I think the new scheme is a big advantage if you have tool support. If you have a view that shows all the tests in a file or a package and you click it for execution, it just passes the test name to the test runner and you have selected a single test to run without changing the code.

If you don't have such a kind of support in your IDE or development environment then the old style might be easier (this isn't available anywhere yet).

Skip is for a different purpose. You can skip tests on criteria (don't run this test in IE, don't run this test until bug Xxxx is fixed, ...).

[Vadim Tsushko]

вторник, 26 мая 2015 г., 17:46:00 UTC+5 пользователь Günter Zöchbauer написал:

I think the new scheme is a big advantage if you have tool support. If you have a view that shows all the tests in a file or a package and you click it for execution, it just passes the test name to the test runner and you have selected a single test to run without changing the code.

I think we may agree that tool support is a long shot for that functionality.

And even if I have IDE with such a support, I would probably prefer old way - no need to switch to the other view just to click there to run my test (And BTW - would be that view synchronized with my primary editor view, or I would need to find current test from the long list of tests?)

I mean, at that moment I do not mind to change my code, I would change it anyway now - to add print statement on so on 

 

[Günter Zöchbauer]

I also wouldn't mind changing the `test` to `solo_test` but I'm prone to forget to change back, therefore the current way is to my advantage.

[Günter Zöchbauer]

You could also create a reusable "solo" run configuration in your IDE to run tests with "solo" in their name. This should result in a quite similar experience like you have with the old unittest package. 

[Vadim Tsushko]

Great idea. I'll try it.

It probably would be perfect when `pub test` would support debugging. 

 

вторник, 26 мая 2015 г., 19:55:39 UTC+5 пользователь Günter Zöchbauer написал:

[Natalie Weizenbaum]

Jim: Running browser tests in the context of a custom server is tricky. In order for the test runner to handle them, they need a special wrapper which a server isn't going to know how to provide. Not only that the test runner need a way to figure out which URLs to even request, which is hard to make work in general. Are you sure you can't use CORS to make the cross-origin requests work?

Günter: --pub-serve isn't going to work with any old server. It specifically requires that the use of the test runner's transformer to ensure that tests have the appropriate wrappers.

Vadim: I explain in this issue why we decided not to support a notion of "solo". In the long term, it'll be much easier to get good tool support for the command-line argument—this is how most other test frameworks work, so most IDEs have good support for it.

--

[Vadim Tsushko]

среда, 27 мая 2015 г., 2:15:52 UTC+5 пользователь Natalie Weizenbaum написал:

Vadim: I explain in this issue why we decided not to support a notion of "solo". In the long term, it'll be much easier to get good tool support for the command-line argument—this is how most other test frameworks work, so most IDEs have good support for it.

Natalie, thank you for clarification. Read the issue and I think I understood now why `solo` was eliminated. 

I should have thought about it myself. Accidentally committing `solo` could make all build-bot process irrelevant.

So `skip` and `solo` are totally different in that way.

Good to know that.