RFC: Pyramid showcase of RealWorld.io

[Nejc Zupan]

Hey all!

TL;DR: I’m working on a RealWorld.io backend implementation in Pyramid. I’d
like to ask fellow pharaohs and mummies to review my code so that I don’t teach
people bad patterns: https://github.com/niteoweb/pyramid-realworld-example-app

RealWorld.io is a community effort of implementing a Medium.com clone app in a
number of interchangeable frontend and backend frameworks. People can then use
these implementations to compare different stacks and to learn about best
practices for a chosen framework.

There already exist several Python backend implementations: Django, Flask,
Serverless Python on AWS, etc. But not one for Pyramid!

So I've built it! I chose to use SQLAlchemy as my ORM and pyramid_openapi3 to
handle the API things. The example has 100% tests & types coverage, a bunch
of linters, etc. It's basically a dump of best practices that we have developed
internally in Niteo in recent years.

And there's the catch: our best practices might be outright wrong. I don't want
to teach people bad patterns. So I'm asking for code review before I submit
the repository to the "official" list of RealWorld implementations.

Please review the code in https://github.com/niteoweb/pyramid-realworld-example-app
and let me know what could be improved. Either as a reply to this email, as
issues on the repo, or even better, as pull requests.


Thanks a ton!
z.

[Theron Luhn]

Very nice!  Several years ago I asked on this mailing list for some example Pyramid applications, this would have been a great one to look at.

You’ve organized the modules by sections of the website, and the relevant views and models in each section.  I usually do the inverse, split my app into views, models, templates, etc.  I’d be interested to see what “best practice” is considered, I’ve always wondered how everybody else organizes their apps.

What’s your experience with type hints/mypy?  I was looking into using them in my applications, but decided it wouldn’t be worth it because Pyramid does not yet have support (there’s an open issue being discussed) and SQLAlchemy’s support is nascent (but very exciting!).

— Theron

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.
To post to this group, send email to pylons-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/B152285B-BAFC-48A0-9DCE-0AA11939E635%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Mike Orr]

On Sat, May 25, 2019 at 9:56 AM Theron Luhn <the...@luhn.com> wrote:
>
> Very nice! Several years ago I asked on this mailing list for some example Pyramid applications, this would have been a great one to look at.
>
> You’ve organized the modules by sections of the website, and the relevant views and models in each section. I usually do the inverse, split my app into views, models, templates, etc. I’d be interested to see what “best practice” is considered, I’ve always wondered how everybody else organizes their apps.

I organize mine by code type:

appname/__init__.py (top-level main, with imports only in functions)
app.py : application-wide things: auth policy, subscribers, layout, etc.
models/
core.py : column type subclasses, full-text search functions, etc
section1.py: models for section 1.
section2.py: models for section 2.
routes.py: Routes. (includeme.)
views:
__init__.py: View configurations. (Includeme. I avoid
@view_config because it's magical.)
section1/: Views for section 1.
__init__.py: Main views/small views for section 1.
alargeview.py: A large view with several private utility functions.
templates: Templates.
site: Site template, section templates, shared template functions, etc.
index.html: Main page template.
section1:
index.html: Section 1 main page.

I use Mako, template inheritance, PostgreSQL, SQLAlchemy,
SQLAlchemy-FullText-Search, and pyramid_redis_sessions.

Frontends I have only moderate experience with, but our frontend
developer organized things by code:
main.js: Top-level ("ready") functions, Backbone models and collections.
section1.js: Backbone views.

I made my own section, an Admin section, and for that I put most
things in admin.js, and if one page had a lot of things I put them all
in a separate file.

The front-end templates are Underscore templates embedded in the HTML
templates. I'm not quite satisfied with this because sometimes there's
a conflict between the Mako, Python, Javascript, and Underscore
syntaxes. Mako uses "<% ... %>" for Python escapes, while Underscore
uses "<% ... %>" for placeholders containing Javascript expressions.
We had to do some elaborate things to get these to coexist, and in one
case I had to pull some Javascript expressions out of the template and
put them into a function in the Underscore dict because they contained
'\:' (a Javascript regex pattern, but an invalid Python string
escape). This was deprecated in Python 3.6 and will be a syntax error
in 3.7, as I discovered when I upgraded PyTest and it showed me the
warnings that had been hidden. I'd like to move the Underscore
templates into different files somewhere, but since the rest of them
are working I've left them alone.

> To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/12FDFC49-945A-40DD-BC0C-5F68AE547730%40luhn.com.

> For more options, visit https://groups.google.com/d/optout.

--
Mike Orr <slugg...@gmail.com>

[Nejc Zupan]

> On 25 May 2019, at 18:56, Theron Luhn <the...@luhn.com> wrote:
>
> You’ve organized the modules by sections of the website, and the relevant views and models in each section. I usually do the inverse, split my app into views, models, templates, etc. I’d be interested to see what “best practice” is considered, I’ve always wondered how everybody else organizes their apps.

We have one 5-year old project with 70k and one fairly recent project with 15k lines of Python code. Both Pyramid based.
The old project uses the structure you described. And it is the structure I have used exclusively since last summer,
when we started working on the new project. We had Domen Kozar helping us out and he suggested to try splitting up
the codebase into sections. He claimed it worked well on his last Pyramid project.

Initially, I was repulsed. But I imagined I could always refactor back to “my way” if I wanted, so it was a rather
cheap experiment. And I’m glad I did it. The codebase of the new project (while ~4x smaller than the old) is
vastly more readable and navigate-able. The only drawback that I found so far is that it is a bit harder to
ask my editor, for example, “I forgot how to build an SQLAlchemy query to do X in tests, find me all tests that do it”
because I I’ll get non-tests code in my “find in all files” results. This is a super minor drawback. Benefits far outweigh it.

> What’s your experience with type hints/mypy? I was looking into using them in my applications, but decided it wouldn’t be worth it because Pyramid does not yet have support (there’s an open issue being discussed) and SQLAlchemy’s support is nascent (but very exciting!).

I am absolutely loving it. Having deep integration between mypy and my editor prevents me from introducing a whole
class of bugs that I would previously only find much later, maybe in integration tests or even in production.

The feeling is quite similar to January 2014 when I enforced 100% unit tests coverage in our big project that
I talked about above. I remember suddenly feeling *much* more at ease when faced with a refactor. Before,
there would always be that lingering “what if I break things” feeling when doing refactors so I’d rarely do them.
And we all know that if you don’t regularly refactor, you get bit rot, and then the codebase becomes a burden
to work with.

I was surprised to get this exact feeling again this year, when enforcing 100% types coverage in our new project.
Nowadays, when I have to work with code that does not have type hints, it feels icky. When I see an obvious eye sore
I’m afraid to touch it, even if the code has 100% tests coverage, as we all know that the coverage does not guarantee
the test suite will catch all problems. It won’t. Having types feels like having an additional, independent, test suite.
And it makes me far more comfortable to touch parts of code I no longer remember writing.

Does it take you longer to write code? Yes. Is there a learning curve? Sure. Will you be frustrated and annoyed
because you will not understand mypy’s error messages? Absolutely. But it’s an investment your are making
today that the future you will be very glad you did.


z.

[Hynek Schlawack]

> On 26. May 2019, at 11:57, Nejc Zupan <nejc....@gmail.com> wrote:
>
> And I’m glad I did it. The codebase of the new project (while ~4x smaller than the old) is
> vastly more readable and navigate-able. The only drawback that I found so far is that it is a bit harder to
> ask my editor, for example, “I forgot how to build an SQLAlchemy query to do X in tests, find me all tests that do it”
> because I I’ll get non-tests code in my “find in all files” results. This is a super minor drawback. Benefits far outweigh it.

FWIW I made a similar experience.

I’m still keeping tests outside, but I’ve started putting separate features into separate sub-packages and it made my maintenance experience much more pleasant thanks to the locality of common contexts. As for tests, I just mirror the structure in my global tests directory. That’s probably just a matter of taste.

[Jonathan Vanasco]

On Sunday, May 26, 2019 at 6:17:53 AM UTC-4, hynek wrote:

t I’ve started putting separate features into separate sub-packages and it made my maintenance experience much more pleasant thanks to the locality of common contexts. As for tests, I just mirror the structure in my global tests directory. That’s probably just a matter of taste.

The biggest improvement we had to overall efficiency and happiness was splitting the model into it's own package.  We have multiple Pyramid, Python, Celery, Twisted, etc apps all import a unified model.  The model has it's own tests, each package/application has it's own tests against the model as well.  Doing this centralized and simplified the management of the database across all the different components, and more importantly multiple levels of testing.  The end result was a much more frictionless build/deploy.