Stacktraces in Mocha output?

1,702 views
Skip to first unread message

Eric Tschetter

unread,
Dec 9, 2013, 3:34:37 PM12/9/13
to moc...@googlegroups.com
Hello all,

I'm using Mocha+Chai.expect to write some unit tests.  Some of my tests look something like

```
doX();
testA();

doY();
testA(); // Y shouldn't affect A

doZ();
testA(); // Z also shouldn't affect A

doW();
testB(); // W actually should affect A, so verify that it became B
```

I run into a problem though in that when my expect() statements in testA() fail, they don't actually show me which call to testA() caused the failure.  I then have to go in and setup log lines old-school style to figure out which one failed.  Test frameworks in other languages have solved this problem by showing the stack trace of a failure along with the "expected/actual" display.  Is there some option to get mocha to show me the same thing?  Is it an easy change to make (that would get incorporated and pushed out to NPM relatively quickly) were I to choose to make the change myself?

--Eric

Miroslav Bajtoš

unread,
Dec 10, 2013, 4:06:30 AM12/10/13
to moc...@googlegroups.com
I'm using Mocha+Chai.expect to write some unit tests.  Some of my tests look something like

```
doX();
testA();

doY();
testA(); // Y shouldn't affect A

doZ();
testA(); // Z also shouldn't affect A

doW();
testB(); // W actually should affect A, so verify that it became B
```
 
I run into a problem though in that when my expect() statements in testA() fail, they don't actually show me which call to testA() caused the failure.  I then have to go in and setup log lines old-school style to figure out which one failed. 
 
It is considered a good practice to have a single assert per test. When each of the pair `doSomething(); testSomething();` lives in it's own test, you don't need the stack trace to identify which pair failed:

it('should not be affected by X', function() {
  doX();
  testA();
});

it('should not be affected by Y', function() {
  doY();
  testA();
});
// etc.

Test frameworks in other languages have solved this problem by showing the stack trace of a failure along with the "expected/actual" display.  Is there some option to get mocha to show me the same thing?  Is it an easy change to make (that would get incorporated and pushed out to NPM relatively quickly) were I to choose to make the change myself?

I believe mocha already has access to stack traces. Check out lib/reporters/base.js [1] and descendants like spec [2] for details on why the stack trace is being discarded.

I can't answer what are the chances of getting your change accepted into mocha. I'd suggest you to discuss your proposed change in a github issue first, that way you'll avoid wasting time on a pull request that may get rejected at the end.

Don't expect quick reactions from Mocha maintainers, they seem to be overloaded with issues and pull requests from my experience [3]. At the moment, there are 166 opened issues & pull requests.


Eric Tschetter

unread,
Dec 10, 2013, 12:34:21 PM12/10/13
to moc...@googlegroups.com
I run into a problem though in that when my expect() statements in testA() fail, they don't actually show me which call to testA() caused the failure.  I then have to go in and setup log lines old-school style to figure out which one failed. 
 
It is considered a good practice to have a single assert per test. When each of the pair `doSomething(); testSomething();` lives in it's own test, you don't need the stack trace to identify which pair failed:

it('should not be affected by X', function() {
  doX();
  testA();
});

it('should not be affected by Y', function() {
  doY();
  testA();
});
// etc.

Hrm, test frameworks I've used in the past frown on this behavior because it means that the tests are not parallelizable (i.e. each "it" block cannot be run on a separate thread/process/server).  That is, doY() is depending on the doX() stuff to have occurred in order to adjust the state of the object.  In order to make it parallelizable, I have to have the full sequence of events inside the same test.

Is there some other mechanism for parallelizing tests that lets you define safe parallelization "chunks" and still work with the model you've suggested?

 
Test frameworks in other languages have solved this problem by showing the stack trace of a failure along with the "expected/actual" display.  Is there some option to get mocha to show me the same thing?  Is it an easy change to make (that would get incorporated and pushed out to NPM relatively quickly) were I to choose to make the change myself?

I believe mocha already has access to stack traces. Check out lib/reporters/base.js [1] and descendants like spec [2] for details on why the stack trace is being discarded.

I can't answer what are the chances of getting your change accepted into mocha. I'd suggest you to discuss your proposed change in a github issue first, that way you'll avoid wasting time on a pull request that may get rejected at the end.

Don't expect quick reactions from Mocha maintainers, they seem to be overloaded with issues and pull requests from my experience [3]. At the moment, there are 166 opened issues & pull requests.


Oh, is there another test library that is supported that people are moving to then?  Or even with the lack of support is mocha still the best out there?

--Eric


Miroslav Bajtoš

unread,
Dec 11, 2013, 4:05:14 AM12/11/13
to moc...@googlegroups.com
On Tuesday, December 10, 2013 6:34:21 PM UTC+1, Eric Tschetter wrote:
Hrm, test frameworks I've used in the past frown on this behavior because it means that the tests are not parallelizable (i.e. each "it" block cannot be run on a separate thread/process/server).  That is, doY() is depending on the doX() stuff to have occurred in order to adjust the state of the object.  In order to make it parallelizable, I have to have the full sequence of events inside the same test.

Is there some other mechanism for parallelizing tests that lets you define safe parallelization "chunks" and still work with the model you've suggested?

I see. I was not sure if doY() depends on doX(). I don't know if there is a solution for defining parallelization "chunks", the usual advice is to redesign your tests so that they are both small (single assert) and independent. It means each test has to recreate the environment (possibly run doX() again). Considering the effort required, it may be better to stay with your original code (one test with multiple asserts) for now.
  
Oh, is there another test library that is supported that people are moving to then?  Or even with the lack of support is mocha still the best out there?
 
Mocha is still the most popular framework. It is extensible, you can write your own test reporter that will include stack trace in the output.

What you can do:
 - Start your own test reporter by copying the code of the mocha reporter you like to use/fix. See [1] for an example of a third-party reporter and how to use it in your project.
 - Once you have a working version, back-port your changes to the original mocha reporter and submit a pull request.

To answer your question - another popular test library is tap [2].

===

I wrote a simple test case to reproduce your problem. It seems the problem is in the algorithm for printing nice diff output when the AssertionError includes information about the expected and the actual values.

Variant A:

var expect = require('chai').expect;
describe('foo', function() {
  it('should bar', function() {
    expect(false).to.equal(true);
  });
});

Output:

  1) foo should bar:

      + expected - actual

      +true
      -false

Variant B:

var assert = require('assert');

describe('foo', function() {
  it('should bar', function() {
    assert.equal(true, false);
  });
});

Output:

  1) foo should bar:
     AssertionError: true == false
      at Context.<anonymous> (/private/tmp/asd/test.js:5:12)
      at Test.Runnable.run (/usr/local/lib/node_modules/mocha/lib/runnable.js:211:32)
      [cut]

This is related to an existing issue [2] that I have reported recently, I'll update the description to mention your use case too.

Reply all
Reply to author
Forward
0 new messages