nose2 config usage

[Aaron Cohen]

 

I am confused about nose2 configuration expected behavior:

1. Why doesn't nose2 tell the user when they try to pass a non-existent config file with the --config option?

2. Please explain the concept of top-level directory as opposed to start directory. It seems like you can go down into  the package and specify the path to the tests you want to run, but the --config option specified is relative to what nose2 considers the top level directory. For example, given this project structure:

• dir
  • tests
    • scenarios
      • default.py
    • another_cfg.cfg
    • unittest.cfg

...If I am in the tests directory, how come if I do this:
$ tests> nose2 -c ./another_cfg.cfg scenarios.default

nose2 doesn't read from the config file, but if I go up a level and do:

$ dir> nose2 -c tests/another_cfg.cfg tests.scenarios.default

nose2 DOES read from the named config file?

Or if I do:

$ tests> nose2 --t ../ -c ./another_cfg.cfg tests.scenarios.default

it loads the config file.

The real confusing part is that when nose2 isn't able to load the config file nose2 doesn't complain, so the user has no idea what's going on.

Thanks. I'd really like to use nose2 for our organization's testing, but some of the behavior seems a bit strange. We are willing to contribute but I'd like to know if I am just missing something with the design of the framework.

[Aaron Cohen]

anyone?

Also, why doesn't nose2 complain about config files not found...? I was told to post my questions here, but nobody's answering. Wrong forum? Low community involvement with nose2?

I could create a pull request, but I assume nose2 doesn't complain about a non-existent config file for a reason....

[jason pellerin]

I think it would make sense to complain about a missing config file as long as it was one specified by the user. Right now I think those are falling down the same path as the default ones, which (intentionally) don't get a warning when missing. I'd be happy to see a PR that fixes that and handles user-specified config files more appropriately.

Sorry to take a while to reply. If you haven't seen the TIP list or nose-devs, the reason is that I'm trying to retire from leadership of nose2 because I no longer have time to give it the attention it needs.

JP

--
You received this message because you are subscribed to the Google Groups "nose-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nose-users+...@googlegroups.com.
To post to this group, send email to nose-...@googlegroups.com.
Visit this group at http://groups.google.com/group/nose-users.
For more options, visit https://groups.google.com/groups/opt_out.

[Aaron Cohen]

Thanks Jason, I think we will work on a pull request for this. Failing silently is probably not what one wants when supplying a config.

Who is taking over as the lead?

[Aaron Cohen]

Hi Jason, if you are retiring is someone else stepping up to lead nose2?

Here are some nose2 behaviors which we think is confusing and could be improved. We are mentioning these because nose2 seems very promising, and we are trying to get a feel for the design decisions and use-cases that may have gone into this behavior. We may want to contribute to the project:

1. It was not apparent from the docs, but the default config (at the current directory) interacts with  a user-supplied config with the --config option. It seems that the default config overrides the user supplied config. This seems to be the opposite of what most people would want. Seems to us that there's more of a use-case to supply a "base" config, which users can override.
   
2. The fact that nose2 takes from a default config at the current directory is a bit confusing. That means that test behavior depends on where I am running the test from. This seems like it would lead to results that are inconsistent and hard to reproduce.
3. Some options seem to be exposed to the command line while others don't. For example, with the mp plugin i don't see that the --process-timeout option can be set on the cmdline, while --processes can. Why is this?

In sum, I think nose2 could be improved with the following behaviors:

1. All options one can set with config files should be exposed for command-line usage.
2. Command line options always override config files
3. By default no config files are looked at. --config must be set explicitly. This will avoid confusion and allow quick development of tests using command line yet still allow easily reproducible test runs using configs.

Do you think our observations are reasonable, or are we missing real-world use-cases that drove these decisions?

Thanks again for your time,

Aaron

[jason pellerin]

I'm hoping someone will step up to take over as lead, but as of yet no one has expressed an interest. I know Omer Katz does a lot of work on the project, but I think he's also very time-limited, and for the most part all of the early contributors have gone on to other things. So I am not particularly optimistic.

For your points of confusion or disagreement with how nose2 handles configuration, those are basically all intentional. Running nose in complex configurations tends to be a nightmare of commandline flags. The reason is basically as summarized here: http://nose2.readthedocs.org/en/latest/differences.html#configuration -- if every option is exposed as a command line option, then nose2 --help is useless, unreadably long, with even a few plugins installed. So nose2 is intentionally pushing away from that. As far as using the config file in the directory making things confusing... I guess I don't see that? Since nose2 loads tests from the working directory too (unless you tell it not to), it always is going to do different things when run in different places.

That said I'm sure we can find some way to at least document all of this more clearly so that it might continue to be disagreeable, but will at least be less confusing. :) Or if you have some ideas that will make things better from your point of view without sacrificing the things I think nose2 needs to preserve, I would be happy to hear them.

Thanks,

JP

--

[Aaron Cohen]

Hi Jason, thanks for your response as well as your time that you've given to nose. Again, our organization , National Center for Biotechnology Information, may be interested in contributing to the project since we are basing one of our tools on nose. I wanted to follow up with a few of the "usability" issues I mentioned and get some more insight on the health of nose2.

>  and for the most part all of the early contributors have gone on to other things. So I am not particularly optimistic.

Can you tell me why this is? Have the developers just done what they wanted and moved on and/or there's not much for them to do given the low adoption of nose2? Regarding the low adoption rate, do you you attribute this to the fact that nose1 just works for most people, or do you think there's some other reason?

> Running nose in complex configurations tends to be a nightmare of commandline flags. ... -- if every option is exposed as a command line option, then nose2 --help is useless, unreadably long, with even a few plugins installed. So nose2 is intentionally pushing away from that. As far as using the config file in the directory making things confusing... I guess I don't see that? Since nose2 loads tests from the working directory too (unless you tell it not to), it always is going to do different things when run in different places.

Yeah, I've read that bit about configuration before, and frankly this is one reason why I've steered clear of nose2 before:

1. We would want at least the option of using all command line flags. Again seems to me using config files is a nice fix for the problem of unwieldy runs with lots of options, but in most cases the run doesn't have that many options and/or a dev simply wants to quickly set some options via the command line. This is how things work in the Unix environment in general, and so it seems to us that nose2 kind of breaks this standard. 
2. A better solution, again, might be to have config(s) set options, and have the ability for command-line options to override any config options. That way developers have the option to keep their runs clean, but they always have the ability to override any option set in a  config file. There might be a flag that would let nose2 report what option is coming from what source. The other problem--probably the main problem--with the current system is that it is inconsistent. For example, apparently, you can't set the path option in junitxml plugin with the command-line, but you can set the processes option on the command-line with the mp plugin. This confusing behavior makes us wary of basing our tools on nose2. It's one thing to have behavior we don't 100% agree with, but if it's seemingly inconsistent, then it's a deal-breaker.

>As far as using the config file in the directory making things confusing... I guess I don't see that? Since nose2 loads tests from the working directory too (unless you tell it not to), it always is going to do different things when run in different places.

This may be true if the main use-case is simply running nose2 and letting it discover everything under the current directory, but many times we specify a directory/package or even module to test. In those cases it would be nice to not have to think about what directory we are starting from. I'm a fairly experienced developer and tester, and while playing around with nose2 I am finding this behavior something I have to think about. I'd rather it be explicit than implicit.

We don't mean to complain. We see some great potential in nose2 but obviously disagree with a few of the details of the design decisions. We are really trying to get a sense if our ideas might be accepted. Of course, even if our changes would be accepted and pull requests accepted, that would break the API of nose2, so we are not sure if that's realistic, and we don't see the use of forking the project. In that case we may as well use nose1 since it basically works and has our desired behavior.

I then wonder if some of these nose2 design decisions could have something to do with the low adoption rate of the product, or if that's attributable to something else.

Thanks again for all your contributions,

Aaron

[Olivier Delalleau]

That's only one datapoint, but as a nose1 user, I just thought you might be interested in hearing my point of view:

I'm happy using nose1 and never really bothered trying to figure out why I should switch to nose2. I've been many times on the nose1 webpage to read the doc on how to achieve specific things, and I've never seen any hint that it would be a good idea to use nose2. In my uninformed mind nose2 is some kind of work-in-progress that might eventually be ready to replace nose1... some day in the not-so-near future.

My 2 cts,

-=- Olivier

2014-01-31 Aaron Cohen <kahun...@gmail.com>: