Test plugin, combining data files in parallel mode?

[Arve Knudsen]

Hi

When using the built-in test plugin in version 3.4 from Nose, I noticed that data files created in parallel mode were not combined before the report was produced. Therefore I modified CoverageTestWrapper.finish to also combine the data files after stopping coverage analysis, iff coverage is in parallel mode. I have verified that this solves my problem. Is it a sensible change though? See the patch below.

diff -r 38a5f871de88 coverage/runners/plugin.py

--- a/coverage/runners/plugin.py        Tue Jun 08 20:41:07 2010 -0400

+++ b/coverage/runners/plugin.py        Thu Jun 10 16:22:36 2010 +0200

@@ -53,6 +53,9 @@

         """Finish coverage after the test suite."""

         self.coverage.stop()

         self.coverage.save()

+        if self.coverage.config.parallel:

+            self.coverage.combine()

+            self.coverage.save()

Thanks,

Arve

[Ned Batchelder]

Arve, first of all, thanks so much for diving in to make the plugin
great. I really appreciate it.

I don't understand how the plugin would work in parallel mode, though?
Wouldn't you need to have a number of nose invocations running at once?
Each would produce its own data file, right? But with this code, each
would also try to combine the data files? I must be missing something.
My model of parallel mode is that there are two phases: phase 1 is
collecting the data in many processes, each producing one data file.
Phase 2 is combining the files and reporting, and phase 2 can only
happen once all of the phase 1 work is done.

Where have I gone wrong?

--Ned.

[Arve Knudsen]

On Fri, Jun 11, 2010 at 4:09 AM, Ned Batchelder <n...@nedbatchelder.com> wrote:

Arve, first of all, thanks so much for diving in to make the plugin great.  I really appreciate it.

I'm the one who should be thanking you for making such a useful tool :)

 

I don't understand how the plugin would work in parallel mode, though?  Wouldn't you need to have a number of nose invocations running at once?  Each would produce its own data file, right?  But with this code, each would also try to combine the data files?  I must be missing something.  My model of parallel mode is that there are two phases: phase 1 is collecting the data in many processes, each producing one data file.  Phase 2 is combining the files and reporting, and phase 2 can only happen once all of the phase 1 work is done.

I think you misunderstand me. It is Coverage.py that is in parallel mode (as controlled by the --cover-parallel-mode option to the plugin), not Nose. In tests, I am spawning child processes that also perform coverage analysis (in parallel mode). It just seems to me the parallel mode of Coverage's test plugin doesn't go all the way, in that it doesn't combine files in phase 2.

 

Arve

[Ned Batchelder]

On 6/11/2010 4:09 AM, Arve Knudsen wrote:

On Fri, Jun 11, 2010 at 4:09 AM, Ned Batchelder <n...@nedbatchelder.com> wrote:

Arve, first of all, thanks so much for diving in to make the plugin great.  I really appreciate it.

I'm the one who should be thanking you for making such a useful tool :)

 

I don't understand how the plugin would work in parallel mode, though?  Wouldn't you need to have a number of nose invocations running at once?  Each would produce its own data file, right?  But with this code, each would also try to combine the data files?  I must be missing something.  My model of parallel mode is that there are two phases: phase 1 is collecting the data in many processes, each producing one data file.  Phase 2 is combining the files and reporting, and phase 2 can only happen once all of the phase 1 work is done.

I think you misunderstand me. It is Coverage.py that is in parallel mode (as controlled by the --cover-parallel-mode option to the plugin), not Nose. In tests, I am spawning child processes that also perform coverage analysis (in parallel mode). It just seems to me the parallel mode of Coverage's test plugin doesn't go all the way, in that it doesn't combine files in phase 2.

OK, now I understand.  So one nose run encapsulates all of the parallel test runs.  I think you are right, the plugin must combine the data files.  Thanks for finding this.

--Ned.

[Ned Batchelder]

Actually, as I was thinking about this, why not have the nose plugin always operate in parallel mode?  There's no harm in using it even if you aren't multi-process, it just means the data will get written to an oddly-named file before it is "combined" to .coverage.  And it will mean that you don't have to learn about parallel mode if your test suite uses processes, it will just work.  There will be a small time penalty when combining the one data file, but that doesn't take long and could even be optimized away.

Am I missing something?

--Ned.

[Arve Knudsen]

On Fri, Jun 11, 2010 at 3:18 PM, Ned Batchelder <n...@nedbatchelder.com> wrote:

Actually, as I was thinking about this, why not have the nose plugin always operate in parallel mode?  There's no harm in using it even if you aren't multi-process, it just means the data will get written to an oddly-named file before it is "combined" to .coverage.  And it will mean that you don't have to learn about parallel mode if your test suite uses processes, it will just work.  There will be a small time penalty when combining the one data file, but that doesn't take long and could even be optimized away.

Am I missing something?

I can see one possible problem, although it might not matter in practice: the user has previously generated .coverage.* files and expects them not to be overwritten, since he knows that coverage (the standalone program) by default doesn't operate in parallel mode. Also, doesn't coverage combine to .coverage.something, as opposed to just .coverage? At least it does that for me.

Arve