Re: testing framework and package.el
Masatake YAMATO
> On 28/09/10 8:57, Stefan Monnier wrote:
>>> is there a chance to get a testing framework on elpa.gnu.org on the
>>> short run?
>>> Ert [1] is used in nxhtml, simple to use and documented.
>>
>> IIRC we've agreed to try and install ERT, either on elpa or in Emacs
>> itself, and then try and move the few tests we already have
>> scattered about to use ERT. But I haven't heard much about it
>> recently.
>
> Incidentally, I finished a major overhaul of ERT's manual a few days
> ago. Over the next few weeks, I should be able to find some time to
> put together a patch to integrate it into Emacs. If anyone is
> interested, you can already review the code at
> http://github.com/ohler/ert and send me comments to speed up the
> integration process.
I read the manual and I've had three small comments.
1. When explaining about the name of test, the read may want
to know how the name is used in ert. My understanding is
it can be used to select the tests to run.
2. I would like to know the difference between `Failed' and `unexpected'.
3. I would like to see the example of `should-error'.
Following the patch tells the place where I got above comments.
diff --git a/ert.texinfo b/ert.texinfo
index 3543738..1f65426 100644
--- a/ert.texinfo
+++ b/ert.texinfo
@@ -125,7 +125,8 @@ better error reporting. @xref{The @code{should} Macro}.
The names of tests can be chosen arbitrarily --- they are in a
namespace separate from functions and variables --- but should follow
the usual Emacs Lisp convention of having a prefix that indicates
-which package they belong to.
+which package they belong to. The names can be used to select the
+tests to run. @xref{Test Selectors}.
The empty parentheses @code{()} in the first line don't currently have
any meaning and are reserved for future extension. They also make
@@ -214,6 +215,7 @@ shows details about each test that had an unexpected result. In the
example above, there are two failures, both due to failed
@code{should} forms. @xref{Understanding Explanations}, for more
details.
+@c ??? Difference between `Failed' and `unexpected'.
In the ERT results buffer, @kbd{TAB} and @kbd{S-TAB} cycle between
buttons. Each name of a function or macro in this buffer is a button;
@@ -357,6 +359,8 @@ checks that the form called within it signals an error. There is no
@code{should-not-error} macro since tests that signal an error fail
anyway, so @code{should-not-error} is effectively the default.
+@c An example of should-error
+
@xref{Understanding Explanations}, for more details on what
@code{should} reports.
Masatake YAMATO
Christian Ohler
> A structure like that could be enforced later
> if needed. We will start with a very small project. I want to make it
> easy to jump to the test-file for a given source-file and execute tests
> for current defun automatically. This could be done based on such a
> structure (i.e. conventions).
>
> I guess reading more of ERT's code I'll find other ways to find the file
> a certain test (or tests matching a selector) is defined.
I just started experimenting with a directory layout where ERT's code is
in lisp/emacs-lisp/*.el and the tests are in test/lisp/emacs-lisp/*.el.
It is indeed very clumsy to jump back and forth between the tests and
the code under test. (Putting the tests in test/emacs-lisp/*.el would
make no difference.)
I suspect it would be easier if the convention was that a file
/some/path/foo.el had its tests in /some/path/test/foo.el. The
convention of having a top-level src/ and test/ split makes sense in
Java because of its package file layout and visibility rules, but Emacs
Lisp has no such constraints. Has anyone else thought about the pros
and cons of these different conventions? Is there a deep reason why
Emacs currently has test/ as a separate top-level directory?
>> Lennart's code is already integrated in ERT, obsoleting ert2. I remember that I
>> wasn't quite happy with his design, but didn't have time to fix it, and since it
>> solved an important problem, I included it as an experimental addition in
>> ert-exp.el.
> OK, good. Found that, thanks. From skimming over that file I guess
> that's what I was looking for and maybe even a bit more.
I just rewrote the test buffer functionality. The design is now more
orthogonal and flexible, but reading input from files will require an
explicit call to `insert-file-contents' or similar.
> Org's exports to XHTML, LaTeX etc. We could compare them to control
> files. The publishing mechanism might create quite big files and even
> entire file/directory trees. Currently we publish our stuff and detect
> regressions when clicking a link that points nowhere or validating a
> XHTML page (spot checking).
I see. Comparing directory trees may be important to org-mode, but
overall, it seems like a feature that would be used by the tests of only
very few packages. I think you're better off writing some utility
functions on top of ERT, as you have already started doing for other
things. Not every test-related utility function belongs in ERT itself;
it's fine to build stuff on top.
The `ert-info' feature that I just implemented allows you to add custom
context information that ERT will display on test failures. It's not
documented in the manual yet, but it makes it easier to build the kind
of test utilities that you may want, so please check it out.
>> One limitations of my approach is that we really need some kind of
>> "with-test-buffer" rather than "with-temp-buffer" that keeps the buffer around
>> for inspection if the test fails, but this is item 2.) on your list already.
>> Another limitation is that the explainer for `equal' is not as pretty as ediff,
>> but OTOH, as soon as you want to compare text properties, ediff can't help
>> anymore (AFAIK), while there is an explainer for
>> ert-equal-including-properties'.
> Also, I believe ediff is too interactive for a simple test run (diff's
> exit status is 0 if no difference is found - don't know yet if and to
> how ediff could be used non-interactively).
I have a rough idea now how to extend ERT's "explanation" feature to
allow interactive invocation of ediff to visualize differences; this
seems like a good way to solve this. Tests would simply use (should
(equal string1 string2)), and the explainer for `equal' would take care
of the rest, putting a button in the results buffer that allows the user
to invoke ediff. string1 and string2 could be the result of calling
`buffer-string' on different buffers, of course.
>> Text properties (e.g., font lock properties) are important to many kinds of
>> tests. Both ediff and using files are problematic in such cases because they
>> don't support text properties. Do text properties never come up in your
>> context?
> Some so some so I guess. I didn't start to write actual tests yet.
> This will be done by many people (I hope) and there will be many
> different types of tests (I hope). Some will involve text properties,
> too.
OK, that makes sense. It is important to write actual tests, though;
ideas for improvement that come from experience with real-world problems
carry much more weight than thought experiments.
Christian.
Eric Schulte
> On 4/10/10 6:22, Sebastian Rose wrote:
>> A structure like that could be enforced later
>> if needed. We will start with a very small project. I want to make it
>> easy to jump to the test-file for a given source-file and execute tests
>> for current defun automatically. This could be done based on such a
>> structure (i.e. conventions).
>>
>> I guess reading more of ERT's code I'll find other ways to find the file
>> a certain test (or tests matching a selector) is defined.
> I just started experimenting with a directory layout where ERT's code
> is in lisp/emacs-lisp/*.el and the tests are in
> test/lisp/emacs-lisp/*.el. It is indeed very clumsy to jump back and
> forth between the tests and the code under test. (Putting the tests
> in test/emacs-lisp/*.el would make no difference.)
>
> I suspect it would be easier if the convention was that a file
> /some/path/foo.el had its tests in /some/path/test/foo.el. The
> convention of having a top-level src/ and test/ split makes sense in
> Java because of its package file layout and visibility rules, but
> Emacs Lisp has no such constraints. Has anyone else thought about the
> pros and cons of these different conventions? Is there a deep reason
> why Emacs currently has test/ as a separate top-level directory?
>
I've been working with Sebastian on a branch of the Org-mode repository
which includes a test suite [1]. Included in this setup is some
functionality for jumping between e-lisp files and their related test
files. This uses an old utility I wrote long ago [2] which allows the
definition of "jumping" functions defined using a simple schema. In the
org-mode repository the tests and elisp are organized as
elisp file in
/lisp/org-foo.el
related tests can be in either of the following
/testing/lisp/test-org-foo.el
/testing/lisp/org-foo.el/test.el
/testing/lisp/org-foo.el/tests.el
the following is used to define our jumping function in this setup [3].
Maybe this would be helpful for navigating the Emacs tests as well.
Cheers -- Eric
Footnotes:
[1] http://repo.or.cz/w/org-mode.git/shortlog/refs/heads/combined-testing
[2] http://github.com/eschulte/jump.el
I was just learning elisp when I wrote this,
so please don't judge me by the code :)
[3] example jump schema
--8<---------------cut here---------------start------------->8---
;;; Navigation Functions
(defjump 'org-test-jump
'(("lisp/\\1.el" . "testing/lisp/test-\\1.el")
("lisp/\\1.el" . "testing/lisp/\\1.el/test.*.el")
("contrib/lisp/\\1.el" . "testing/contrib/lisp/test-\\1.el")
("contrib/lisp/\\1.el" . "testing/contrib/lisp/\\1.el/test.*.el")
("testing/lisp/test-\\1.el" . "lisp/\\1.el")
("testing/lisp/\\1.el" . "lisp/\\1.el/test.*.el")
("testing/contrib/lisp/test-\\1.el" . "contrib/lisp/\\1.el")
("testing/contrib/lisp/test-\\1.el" . "contrib/lisp/\\1.el/test.*.el"))
(concat org-base-dir "/")
"Jump between org-mode files and their tests.")
--8<---------------cut here---------------end--------------->8---
Sebastian Rose
> On 4/10/10 6:22, Sebastian Rose wrote:
> I suspect it would be easier if the convention was that a file /some/path/foo.el
> had its tests in /some/path/test/foo.el. The convention of having a top-level
> src/ and test/ split makes sense in Java because of its package file layout and
> visibility rules, but Emacs Lisp has no such constraints. Has anyone else
> thought about the pros and cons of these different conventions? Is there a deep
> reason why Emacs currently has test/ as a separate top-level directory?
This would place many "test" directoies inside the source tree. They
might get in the way of builds.
Emacs conventions are indeed naming conventions: Start function names
whith the name of the "package", name the file like it, (provide
'PACKAGE) etc.
> I just rewrote the test buffer functionality. The design is now more orthogonal
> and flexible, but reading input from files will require an explicit call to
> insert-file-contents' or similar.
That's not a big hurdle I guess.
> I see. Comparing directory trees may be important to org-mode, but overall, it
> seems like a feature that would be used by the tests of only very few packages.
> I think you're better off writing some utility functions on top of ERT, as you
> have already started doing for other things. Not every test-related utility
> function belongs in ERT itself; it's fine to build stuff on top.
OK.
> The `ert-info' feature that I just implemented allows you to add custom context
> information that ERT will display on test failures. It's not documented in the
> manual yet, but it makes it easier to build the kind of test utilities that you
> may want, so please check it out.
>
>
>>> One limitations of my approach is that we really need some kind of
>>> "with-test-buffer" rather than "with-temp-buffer" that keeps the buffer around
>>> for inspection if the test fails, but this is item 2.) on your list already.
>>> Another limitation is that the explainer for `equal' is not as pretty as ediff,
>>> but OTOH, as soon as you want to compare text properties, ediff can't help
>>> anymore (AFAIK), while there is an explainer for
>>> ert-equal-including-properties'.
>> Also, I believe ediff is too interactive for a simple test run (diff's
>> exit status is 0 if no difference is found - don't know yet if and to
>> how ediff could be used non-interactively).
> I have a rough idea now how to extend ERT's "explanation" feature to allow
> interactive invocation of ediff to visualize differences; this seems like a good
> way to solve this. Tests would simply use (should (equal string1 string2)), and
> the explainer for `equal' would take care of the rest, putting a button in the
> results buffer that allows the user to invoke ediff. string1 and string2 could
> be the result of calling `buffer-string' on different buffers, of course.
+1
> OK, that makes sense. It is important to write actual tests, though; ideas for
> improvement that come from experience with real-world problems carry much more
> weight than thought experiments.
>
> Christian.
Yes, true.
Sebastian
Christian Ohler
> Christian Ohler writes:
>
> > IIUC, this design requires developers who want to add features and write
> > tests for them to check out two different bzr branches and submit two
> > different patches.
>
> Currently, yes. With nested branches, the checkout will be implicit.
Let's find a solution that works well now rather than speculating on a
future bzr feature that may be delayed, may be slow in its initial
implementation, or may turn out not to work the way we think it will.
Once nested branches are implemented, we can learn how they work, and
then split the repositories if it still makes sense.
> > Isn't that a rather high bar that will be inconvenient for
> > developers and discourage them from writing tests?
>
> In my experience, people who actually write tests (unless there's a
> firm requirement that tests be submitted before committing) are hard
> to discourage.
We want more than this; we want those who don't write tests to start
doing so. Putting significant barriers in their way makes it much less
likely that Emacs will gain a useful test suite.
> The real problem is getting those tests backported.
You mean, running a test suite in an older Emacs version, and adding
fboundp checks to the tests that test missing features, is harder than
writing the test suite in the first place? I don't expect that will be
true.
> > As an alternative that avoids this problem, we could maintain the tests
> > in Emacs trunk, but our makefiles could have a test_src variable that
> > defaults to tests, and that we can set to ../trunk/tests when running
> > tests in the emacs-23 directory in Stephen's layout.
>
> Which still requires serious developers to have multiple checkouts.
You are right that developers who want to run tests in multiple versions
of Emacs will need the code for all those versions, and the code for the
tests. There is no way around that. However, including the tests in
Emacs' trunk saves them one checkout, and means less hassle submitting
Emacs bugfixes together with additional tests. Also, casual
contributors often won't have multiple Emacs versions checked out, and
they are worth catering for. It seems more likely that they would
contribute tests if they can do so without the overhead of multiple
workspaces.
The solution of adding a makefile variable like test_src that allows
running the test suite from a different workspace seems to solve the
problem you described, and is much simpler than splitting the
repository. Do you think it has significant drawbacks?
Christian.
Christian Ohler
can be useful to run the current version of the test suite on an older
version of Emacs. If it's as simple as adding a makefile variable, we
should implement it.
Christian.