Make Pyramid 1.7 scaffolds ready for naïve usage of py.test and tox in new projects?

[Vincent Férotin]

Hi, Pyramid developers!

Here are some thoughts about what I consider a small problem for new users:
while usage of py.test is now promoted in documentation
(http://docs.pylonsproject.org/projects/pyramid/en/1.7-branch/whatsnew-1.7.html#documentation-enhancements)
scaffolds are not ready for naive usage of py.test by new comers to Pyramid,
i.e. running ``py.test [--cov]`` at new project's root collects no tests.

As I don't know if this could be considered a bug, I'd prefer opening discussion here, before opening a bug in the tracker or submitting a pull request.

What I, as new Pyramid user, attempt just after creating project via ``pcreate -s [scaffold] [myproject]``,
and seeing the "Welcome to Pyramid.  Sorry for the convenience." message, is that
project is ready to be tested out of the box by py.test or tox at project's root.
Currently, this seems to not be the case: py.test collects no item (outputs "collected 0 items").

To change that, I tried to modify `alchemy` scaffold so that:
- py.test runs out-of-the-box,
- little effort is required for tox to also work.

To do that, I:
- add a `tox.ini` configuration file for py.test;
- add a `.coveragerc` configuration file for pytest-cov plugin;
- move all tests material in its own directory (``{{pacakge}}/tests/``);
- split test file in `tests/base.py` and `tests/test_myview.py`.
Full change could be found here:
https://github.com/vincent-ferotin/pyramid/commit/7240f2214509bfb3b9b1aa0705f2021808d4dd5e

Discuss-able choices I'm aware of are:
* usage of tox.ini in place of setup.cfg:
  as setup.cfg generation was removed from 1.7, and usage of tox could be possible,
  I've preferred putting py.test configuration in tox.ini.
  This allows small change (i.e. adding ``[tox] envlist=py35``) for also having tox running on the new project;
* moving tests on their own sub-directory:
  This avoids py.test scanning all project's files in order to collect tests, which could be perceived as slow,
  by explicitly telling it where tests stand.
  Also, this allow splitting tests into material for testing (`tests/base.py`) to tests them selves (`tests/test_*.py`),
  in same manner than models is splitted between `models/meta.py` and `models/mymodel.py`.

So, what do you guys think about that?

-- Vincent

[Michael Merickel]

Hey Vincent, thanks for working on this.

It would be very helpful if you could open an issue or a PR on github as it's much easier to discuss things there. I will say up front that a couple of these decisions are made already.

1) no tox. This is an app, not a library and we've already had a couple discussions about this where it was decided we would not promote tox in the scaffolds.

2) no explicit location where tests must live, I'd opt for scanning the project for particular conventions such as files starting with ``test_``.

As far as adding a setup.cfg or a pytest.ini, I'm all for these things if they make running the tests simpler but avoid certain opinions like package layout. I'm not sure what coverage settings you'd need so I'd have to see the PR.

- Michael

--
You received this message because you are subscribed to the Google Groups "pylons-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-devel...@googlegroups.com.
To post to this group, send email to pylons...@googlegroups.com.
Visit this group at https://groups.google.com/group/pylons-devel.
For more options, visit https://groups.google.com/d/optout.

[Steve Piercy]

On 5/5/16 at 9:44 AM, vincent...@gmail.com (Vincent
Férotin) pronounced:

>scaffolds are not ready for naive usage of py.test by new comers to Pyramid,
>i.e. running ``py.test [--cov]`` at new project's root collects no tests.

You are correct in that one cannot just naïvely run:

$ py.test [--cov]

py.test discovers tests in the following manner:
http://pytest.org/latest/goodpractices.html#python-test-discovery

By following the instructions for the SQLAlchemy + URL dispatch
wiki tutorial to get started:
http://docs.pylonsproject.org/projects/pyramid/en/1.7-branch/tutorials/wiki2/installation.html

I can run the tests:

stevepi-mbp:tutorial stevepiercy$ $VENV/bin/py.test
tutorial/tests.py -q
..
2 passed in 0.54 seconds

stevepi-mbp:tutorial stevepiercy$ $VENV/bin/py.test
--cov=tutorial --cov-report=term-missing tutorial/tests.py
=======================================================================
test session starts ========================================================================
platform darwin -- Python 3.5.0, pytest-2.9.1, py-1.4.31, pluggy-0.3.1
rootdir: /Users/stevepiercy/pyramidtut/tutorial, inifile:
plugins: cov-2.2.1
collected 2 items

tutorial/tests.py ..
---------------------------------------------------------
coverage: platform darwin, python 3.5.0-final-0 ----------------------------------------------------------
Name Stmts Miss Cover Missing
----------------------------------------------------------------
tutorial/__init__.py 8 6 25% 7-12
tutorial/models/__init__.py 22 0 100%
tutorial/models/meta.py 5 0 100%
tutorial/models/mymodel.py 8 0 100%
tutorial/routes.py 3 3 0% 1-3
tutorial/scripts/__init__.py 0 0 100%
tutorial/scripts/initializedb.py 26 26 0% 1-45
tutorial/tests.py 39 0 100%
tutorial/views/__init__.py 0 0 100%
tutorial/views/default.py 12 0 100%
tutorial/views/notfound.py 4 4 0% 1-7
----------------------------------------------------------------
TOTAL 127 39 69%

=====================================================================
2 passed in 0.57 seconds =====================================================================
stevepi-mbp:tutorial stevepiercy$

--steve

------------------------
Steve Piercy, Soquel, CA

[Vincent Férotin]

Thank you, Michael and Steve, for the quick responses.

I will open a PR on github with changes to take in consideration what you said.

@Steve:
Sure you can explicitly tell py.test to run a specific test module,
but the whole point of my suggestion is to not pass test module path
to py.test. It's the point of view of a beginner, not familiar with
Pyramid scaffolds nor py.test: isn't it easier/more convenient to only
run ``py.test`` at project's root directory, just after project
creation by scaffold, and get all tests collected by default?

-- Vincent

[Steve Piercy]

In this case, py.test was totally new to me. I had always used
``nose`` or ``tox`` that someone else wrote or setup.py'ed for
me. I was a beginner to py.test and test discoverability.

When I tried running tests out of the box following that pattern
via ``py.test``, it didn't work. Hmm, I thought, there *must*
be some way to make it work. So I Google Fu'd, visited the
link[1] to read about test discovery, and it became clear in my
mind how to do so. I then wrote the documentation for how to
run tests for the scaffold. As a beginner, I managed to work it out.

Because I learned something from this process, I think it is
good for beginners to realize that there is more to py.test than
its mere invocation without options. There's a lot to explore.
After having this discussion, I think I should add this
reference to the tutorial installation step.

For what it's worth, when we go deeper into tests, we use the
simple invocation with a single option to suppress output,
``py.test -q``.[2]

[1] http://pytest.org/latest/goodpractices.html#python-test-discovery
[2] http://docs.pylonsproject.org/projects/pyramid/en/1.7-branch/tutorials/wiki2/tests.html#running-the-tests

--steve


On 5/6/16 at 2:51 PM, vincent...@gmail.com (Vincent
Férotin) pronounced:

[Vincent Férotin]

I totally agree, Steve, that there is much more to explore with
py.test than simple invocation we describe here. And I better
understand how you come to the pattern ``py.test mymodule/mytests.py
-q``. My experience took an other path, where I was sure that using
``py.test`` alone should work, googled about it, and found
configuration i purposed here ;-)

@all:
Anyway, I rework previous patch to meet Michael preferences, and it is
now simpler: just adding tow new file to each scaffold (one for
`pytest`, the other for `coverage`). I also update narrative doc. to
first use simple ``py.test -q`` call (letting a reference to the more
explicit ``py.test mymodule/mytests.py -q`` pattern).

The bad news is that I simply failed to create a PR, in that pushing
new commits to 1.7-branch of my forked repo. aborts (with a
[french/here translated] message like:
" ! [rejected] 1.7-branch -> 1.7-branch (non-fast-forward)
error: unable to push references to
'https://github.com/vincent-ferotin/pyramid.git'
astuce: Update failed due to branch head is behind
astuce: its distant homologue.
astuce: Pull distant changes (e.g. 'git pull ...')
astuce: before pushing. [...]").

I'm not familiar enough with git to resolve this issue.
So I push patches here: https://github.com/vincent-ferotin/pyramid-patches
Patches could be imported by StackedGit (`stg`) through ``stg import
--series pyramid-patches/series``).

I'm sorry about that, and hope this will not abort insert of these
changes in Pyramid's repo.

-- Vincent

[Steve Piercy]

We try to follow a good git workflow.[1]

The error provides instructions of what to do.

>astuce: Pull distant changes (e.g. 'git pull ...')
>astuce: before pushing. [...]").

Which implies you did not follow the helpful instructions in
contributing.md under "Building documentation for a Pylons
Project project", steps 6-10[1] which I wrote to prevent such
occurrences. I haz a sad. :-(

[1] https://help.github.com/articles/what-is-a-good-git-workflow/
[2] https://github.com/Pylons/pyramid/blob/master/contributing.md#building-documentation-for-a-pylons-project-project

--steve



On 5/6/16 at 6:41 PM, vincent...@gmail.com (Vincent
Férotin) pronounced:

[Vincent Férotin]

Ok, here's an attempt for the PR: https://github.com/Pylons/pyramid/pull/2545

Thank you for your patience, and thank you Steve for pointing me the
right document! I wasn't aware of its existence, as it seems not
integrated to HTML documentation (?). (I've previously found the
http://www.pylonsproject.org/community/how-to-contribute which is not
so helpful.)
As an aside, following steps GitHub Flow, I create new branch directly
from master, but was confused by the paragraph on other branches
(https://github.com/Pylons/pyramid/blob/master/contributing.md#git-branches):
should I have taken this in consideration?

Again, thank you Steve!

-- Vincent

[Steve Piercy]

It's hard to miss contributing.md. When you create a new issue
or PR, a friendly yellow bar appears with the message:

Please review the [guidelines for
contributing](https://github.com/Pylons/pyramid/issues/new) to
this repository.

www.pylonsproject.org is limping along. It's on the list of
to-dos, behind the [new marketing site for
Pyramid](https://trypyramid.com) and a new Sphinx theme for documentation.

https://github.com/Pylons/trypyramid.com
https://github.com/Pylons/pyramid-sphinx-themes
https://github.com/Pylons/pylonsproject.org

The branch to which you submit a PR should always be taken into
consideration. 'master' is usually default. We usually
backport changes from master to a specific branch.

--steve


On 5/6/16 at 8:08 PM, vincent...@gmail.com (Vincent
Férotin) pronounced:

[Vincent Férotin]

Much clear. Thanks!