[cursive] run-all-tests run configuration

152 views
Skip to first unread message

Matt Bossenbroek

unread,
May 23, 2017, 12:31:58 PM5/23/17
to cur...@googlegroups.com
Is there a way to setup a run configuration to run a set of tests in Cursive? Something similar to how I can set up JUnit to point at a directory or package & have it run all of the tests run it.

Having to run each ns separately is kind of a pain, especially for large projects. I saw a couple old threads on this, but nothing recent so wasn’t sure if it was added when we weren’t looking :)

Thanks,
Matt

Colin Fleming

unread,
May 25, 2017, 10:11:55 PM5/25/17
to cur...@googlegroups.com

So, there isn’t at the moment, no, sorry. It’s something I’ve been planning for a while. Generally I assume people use the testing features in lein, boot, gradle or whatever for project-wide tests. That said, this would clearly be very nice to have.

I guess you’re using Gradle, right? Would you want Cursive to just get all test namespaces from a module and run the tests from it?

--
You received this message because you are subscribed to the Google Groups "Cursive" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cursive+u...@googlegroups.com.
To post to this group, send email to cur...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/cursive/CAHj4K5py_S0F%2BKMR-V7S4MfAbLnGPWDUd15Q8AFZ2-Gwz5d39w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Stephen Nelson

unread,
May 25, 2017, 10:43:30 PM5/25/17
to cur...@googlegroups.com
I have been wishing for this feature for a long time. IntelliJ has a lot of context-aware suggestions that show up when you're writing code in other languages, things like "run this test", "run all tests in this file", "run all tests in this module".  Have a look at the Golang plugin for a non-Java example if you're looking for inspiration.

Along with running tests, I'd also like to see context-aware suggestions like "navigate to tests", or "define a test for this function".

Speaking specifically about our project, we use lein to run tests for a particular module, as well as a script that uses lein to run tests for all modules concurrently in dependency order.


For more options, visit https://groups.google.com/d/optout.
--
Stephen Nelson | Platform Lead Developer | Montoux
e.      ste...@montoux.com
t.      @montoux
w.      montoux.com

Luke Burton

unread,
May 26, 2017, 11:14:58 AM5/26/17
to cur...@googlegroups.com

+100 for “create test for this function”. I know it’s a non-trivial feature, but lowering the bar for test creation has a big impact. 

Additionally, being able to see test coverage in the source file would be magical. Just having reflection warnings show up in the editor really impels me to clean them up, I’m sure I’d improve my test coverage drastically with a similar inspection. 

Matt Bossenbroek

unread,
May 26, 2017, 12:35:34 PM5/26/17
to cur...@googlegroups.com, Luke Burton
> I guess you’re using Gradle, right? Would you want Cursive to just get all test namespaces from a module and run the tests from it?

We have some gradle, some lein. It would be nice to be able to select a namespace, set of namespaces, or a namespace regex (which would encompass the other options). Also, all tests in a module would be useful too.

> Generally I assume people use the testing features in lein, boot, gradle or whatever for project-wide tests.

You’ve made the test setup in Cursive so good that I don’t want to leave anymore. The diffs of large data structures in test results are phenomenal!


And +100 for automated test creation & navigation. Would be extremely helpful.

-Matt

Beau Fabry

unread,
May 26, 2017, 1:14:32 PM5/26/17
to Cursive, luke_...@icloud.com
fwiw, if anyone wants a quick fix, I usually have something like this in user.clj

(defn run-unit-tests []
  (repl/refresh)
  (test/run-all-tests #"^(project-ns).*$"))

Wilker

unread,
May 26, 2017, 1:24:43 PM5/26/17
to Cursive, luke_...@icloud.com
Or you can get the code from Beau and make a REPL Command out of it, so you can run anywhere from command pallet or assign a keyboard shortcut if you want.

To view this discussion on the web visit https://groups.google.com/d/msgid/cursive/336c99ed-ed3d-4c10-8c0d-a089dbe3543f%40googlegroups.com.

Colin Fleming

unread,
May 29, 2017, 11:17:16 PM5/29/17
to cur...@googlegroups.com

Ok, so I looked into this briefly. I’ll need to try it to see how it will work since the test finding infrastructure in IntelliJ is fairly class-based. Hopefully I can make that translate nicely to namespaces. This also might affect test creation - “create test for this class” would normally create a test class, but I’m assuming in Clojure you’d want to create a test var inside an existing test namespace (creating it if required). I’m not sure how that mapping to vars should work - see below.

It seems that there are a couple of things here.

  1. How to map from test namespaces to non-test and vice versa. This is pretty straightforward I think - removing -test and test from namespace segments and then trying all combinations should be sufficient.
  2. I guess “create test for this function” would create a test var in the corresponding test namespace with called <my-fn>-test. Would you want mapping from other things to test vars, i.e. records or protocols? If anyone has ideas about how this mapping might work or be configured, let me know. One thing that this would help with is navigation - if the test searching is namespace based, when you navigate to test or navigate to source, you’ll end up with the caret at the namespace decl, i.e. at the top of the file.
  3. Finding test namespaces should be pretty straightforward as well - as a first cut I’ll just find all namespaces which depend on clojure.test - if that needs refining later we can do that.
  4. One other question - would you want the tests to run in a separate fresh process (like, say, Java) or would you want the tests to execute in your open REPL? Would you want to be able to choose?

Matt Bossenbroek

unread,
May 30, 2017, 12:30:26 PM5/30/17
to cur...@googlegroups.com, Colin Fleming
Awesome!

> Would you want mapping from other things to test vars, i.e. records or protocols?

I’m fine with just functions for now (would be a great start), but I don’t know what other people would want. Usually testing a record or protocol is more involved anyway.

> Finding test namespaces should be pretty straightforward as well - as a first cut I’ll just find all namespaces which depend on clojure.test - if that needs refining later we can do that.

Doesn’t IntelliJ mark source code vs test code automatically? At least in the gradle world, each show up as their own module. And I’m pretty sure I’ve seen the equivalent in lein too.

> One other question - would you want the tests to run in a separate fresh process (like, say, Java) or would you want the tests to execute in your open REPL? Would you want to be able to choose?

I just assumed they would run in-REPL, but starting a new process seems like it would be valuable to make sure there’s no crud in my existing REPL that’s not persistent. Having a choice there would be great.

Thanks!

-Matt

Daniel Compton

unread,
May 30, 2017, 4:32:34 PM5/30/17
to cur...@googlegroups.com, Colin Fleming
> 2. I guess “create test for this function” would create a test var in the corresponding test namespace with called <my-fn>-test.

Seems sensible

> One thing that this would help with is navigation - if the test searching is namespace based, when you navigate to test or navigate to source, you’ll end up with the caret at the namespace decl, i.e. at the top of the file.

Could navigating to tests also look for tests that contain that function inside an assertion?

> 4. One other question - would you want the tests to run in a separate fresh process (like, say, Java) or would you want the tests to execute in your open REPL? Would you want to be able to choose?

I assumed in-REPL too. That makes things easier for Boot users. I can see the benefit for running in a separate REPL, but suspect that I would most often want to run it in-process. I'm using a tools.namespace based workflow so I don't get too much cruft accumulated in the REPL, although I do occasionally wind up in tricky spots. If the tests were executing in another process, I can see it being occasionally useful to get a REPL into that one to check some state.

To view this discussion on the web visit https://groups.google.com/d/msgid/cursive/CAHj4K5o5H%2B4VsC%3D3NyKxtpcB2Emaz1rRiNr2635uuybDa5D07w%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.
--

Daniel

Stephen Nelson

unread,
May 30, 2017, 10:54:40 PM5/30/17
to cur...@googlegroups.com
1&3: mapping between main and test namespaces could be quite project-specific. Would it be possible to have a configurable pattern in project settings? Eg a regex to transform from namespace to test, and maybe one in the other direction too. I would expect that you would identify test namespaces by looking at their root (marked source or test)

4. Our 100k-loc-ish project takes about 20 seconds just to build a leiningen class path, using the existing REPL if available would be nice, but being firing up a separate JVM where no REPL is available would be nice too. 

For more options, visit https://groups.google.com/d/optout.

Colin Fleming

unread,
Jun 5, 2017, 9:57:20 PM6/5/17
to cur...@googlegroups.com

One other question - do you all like the UI with the icons painted in the gutter, or for a “Run all project tests” type thing would you prefer the standard IntelliJ test UI? The current one is more designed for interactive work, e.g. it doesn’t have an easy way to navigate to test failures in files that aren’t open in an editor. I could work something up for that if the current UI is preferable, though.

One potential issue with the built-in one is that it’s very much oriented to a process-based runner model, I’m not sure I can make it work in the REPL - I’ll have to investigate that.

Luke Burton

unread,
Jun 5, 2017, 10:05:35 PM6/5/17
to cur...@googlegroups.com

Is there something that exemplifies the "icons painted in the gutter" that would be similar to what you're thinking? For instance, would this be like the VCS annotations (right click gutter / annotate) or code inspections in the gutter? 


Colin Fleming

unread,
Jun 5, 2017, 10:17:03 PM6/5/17
to cur...@googlegroups.com

You can see what I mean here. Those screenshots are a little out of date, but you get the idea.

Pros/cons:

  1. The feedback is very interactive, things are painted in the UI when done.
  2. Easy to integrate with the REPL.
  3. Good feedback per assertion rather than per test var.
  4. Currently no UI as such for global results (ran x tests, y passed sort of thing), and in particular no easy way to navigate to failures in other files.
  5. Currently no handling or intelligent display of process output.

For reference, the standard test runner looks like this, i.e. it opens in its own tool window with a tree of results. For Clojure, I guess the top level of the tree would be Clojure namespaces, and the next level would be the vars.

Pros/cons:

  1. It’s what anyone used to IntelliJ will expect.
  2. It’s easier to integrate with IntelliJ’s coverage integration etc, although that is another ball of trickiness.
  3. The UI assumes that a test fails on the first assertion failure, which is not true of clojure.test - you can get multiple failures per test.
  4. It’s very tied to running tests in an external process rather than an existing REPL process - I’m not sure if I can work around that or not.
  5. Automatically provides timings for tests and other niceties, and a slick UI.
  6. Better handling of stdout etc from tests.

Luke Burton

unread,
Jun 5, 2017, 10:34:15 PM6/5/17
to cur...@googlegroups.com

Thanks, that makes things a lot more clear.

I far prefer the gutter/interactive model, but I could also see the test runner being useful for different conditions.

I think coupling with the current REPL session promotes iteration on tests. Having to work on a test, then burn a new JVM before you get feedback would make me (again just me personally) less likely to use it.


Matt Bossenbroek

unread,
Jun 5, 2017, 11:38:35 PM6/5/17
to cur...@googlegroups.com, Luke Burton
For the project I’m currently working on, I’d lean towards the standard test runner. My tests are in lots of files & I wouldn’t want to have to open all of them to see pass/fail rates + summary, and each test produces a LOT of log output (see other thread), so it would be really nice to have that chopped up.

Would it be possible to have two modes? The first one is what we have today, which runs in the current real & annotates the current file. The second could be the traditional IntelliJ test runner, with a new process, results in the bottom tab, etc. Basically a lightweight & heavyweight version.


-Matt

Stephen Nelson

unread,
Jun 7, 2017, 5:17:05 PM6/7/17
to cur...@googlegroups.com
I think both are useful. I would certainly prefer to keep the gutter annotations on is forms, but the visual summary in the standard test runner is very useful for finding failures, something the gutter annotations are bad at showing.


For more options, visit https://groups.google.com/d/optout.

Colin Fleming

unread,
Jun 7, 2017, 6:39:27 PM6/7/17
to cur...@googlegroups.com, Stephen Nelson

I spent some time looking into this yesterday. It looks like I can hopefully reuse the IntelliJ test UI even when running in a REPL, so that’s good. I’ll still have to work out which of the actions should create a run configuration - probably “Run all tests in project/module” should, but “Run test under caret” or “Run tests in file” should not - basically anything that can be easily run with a simple command is probably not worth it.

I think I can probably make it work so that it works as it does currently, but also allows the test UI to be shown - that seems like the best of both worlds. It seems like hiding the IntelliJ test UI by default is probably the best option, but allowing it to be opened to investigate if tests fail. That means that I can probably use the test UI even for the existing commands, and hopefully that will mean that I can capture the output per-test and show it there as well.

Colin Fleming

unread,
Jun 21, 2017, 12:56:02 AM6/21/17
to cur...@googlegroups.com

The first part of this (test navigation/creation) has been released today in 1.6.0-eap1. I’m very interested in feedback on how well it works. In subsequent 1.6.x releases I’m going to add the test runner/UI and REPL improvements.

Beau Fabry

unread,
Jun 21, 2017, 2:25:19 PM6/21/17
to Cursive, cur...@cursive-ide.com
fwiw, for us, navigate between code and test (high frequency, non automatable action) is a great addition. Thank you so much. Having the tests runnable via the intellij hooks (low frequency, automatable action) probably won't be used until we start a new project where we don't already have a test runner set up.

Colin Fleming

unread,
Jun 29, 2017, 7:23:44 PM6/29/17
to Beau Fabry, Cursive

I haven’t heard anyone saying that this is awful, so I’m going to release it as 1.6.0 today so that everyone can get a build with the locals clearing enabled (i.e. fixing up the 1.5.2 problem). There are a few minor issues with this still, they’ll be fixed in 1.6.1-*.

Cheers,
Colin

Reply all
Reply to author
Forward
0 new messages