Proposal to format Django using black

2,352 views
Skip to first unread message

Herman S

unread,
Apr 13, 2019, 11:35:12 AM4/13/19
to django-d...@googlegroups.com
Hi.

I propose that Django starts using 'black' [0] to auto-format all Python code.
For those unfamiliar with 'black' I recommend reading the the projects README.
The short version: it aims to reduce bike-shedding and non value-adding
discussions; saving time reviewing code; and making the barrier to entry lower
by taking some uncompromissing choices with regards to formatting. This is
similar to tools such as 'gofmt' for Go and 'prettier' for Javascript.

Personally I first got involved contributing to Django couple of weeks back,
and from anecdotal experience I can testify to how 'formatting of code' creates
a huge barrier for entry. My PR at the time went multiple times back and forth
tweaking formatting. Before this, I had to research the style used by exploring
the docs at length and reading at least 10-20 different source – and even those
were not always consistent. At the end of the day I felt like almost 50% of the
time I used on the patch was not used on actually solving the issue at hand.
Thinking about code formatting in 2019 is a mental energy better used for other
things, and it feels unnecessary that core developers on Django spend their time
"nit-picking" on these things.

I recently led the efforts to make this change where I work. We have a 200K+
LOC Django code-base with more than 30K commits. Some key take-aways: it has
drastically changed the way we work with code across teams, new engineers are
easier on-boarded, PR are more focused on architectural choices and "naming
things", existing PRs before migration had surprisingly few conflicts and were
easy to fix, hot code paths are already "blameable" and it's easy to blame a
line of code and go past the "black-commit", and lastly the migration went
without any issues or down-time.

I had some really fruitful discussions at DjangoCon Europe this week on this
very topic, and it seems we are not alone in these experiences. I would love to
hear from all of you and hope that we can land on something that will enable
*more* people to easier contribute back to this project.

I've set up how this _could_ look depending on some configurables in Black:

* Default config: https://github.com/hermansc/django/pull/1
* Line length kept at 119: https://github.com/hermansc/django/pull/3
* Line length kept at 119, no string normalization:
https://github.com/hermansc/django/pull/2

Please have a look at the Black documentation. It explains the benefits better
than I possibly could do here.

With kind regards,
Herman Schistad

[0]: https://github.com/ambv/black

Ian Foote

unread,
Apr 13, 2019, 11:56:09 AM4/13/19
to django-d...@googlegroups.com
I'm in favour. Not needing to think much about code style when writing code or when reviewing is very nice.

Ian

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAN%3DnMTx0EE5WfXuccv_e3MBuCxp9u_pAV_ow5MxNST6MptTDBw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Florian Apolloner

unread,
Apr 13, 2019, 12:07:48 PM4/13/19
to Django developers (Contributions to Django itself)
As expressed at Djangocon Europe, I am hugely in favor for adopting black.

If we choose to do this there are a few things to consider:

 * Line-length, we probably want to stay at 119 I guess
 * String normalization, I'd be in favor of following black defaults here, but I am open to be convinced otherwise
 * We will need to adjust the isort configuration ( https://github.com/ambv/black/blob/master/README.md#how-black-wraps-lines )

Cheers,
Florian

Markus Holtermann

unread,
Apr 13, 2019, 12:12:44 PM4/13/19
to Django developers
👍

/Markus
> --
> You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-develop...@googlegroups.com.
> To post to this group, send email to
> django-d...@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/698d8e49-0deb-437c-b6ce-a321fe3988e3%40googlegroups.com <https://groups.google.com/d/msgid/django-developers/698d8e49-0deb-437c-b6ce-a321fe3988e3%40googlegroups.com?utm_medium=email&utm_source=footer>.

Jon Dufresne

unread,
Apr 13, 2019, 12:46:22 PM4/13/19
to django-d...@googlegroups.com
Big +1

I've used black in quite a few projects now and found it has greatly reduced the mental energy involved in code formatting while aligning all contributors expectations.

I'm in favor of adding no configuration. That is, use black's default line length and string delimiters. IMO, the less configuration the better. The more the Django code style aligns with the larger Python community, the better and easier it is few new contributors. But these secondary choices are less important to me than the improvement of adopting a tool to auto-format code.

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

Tobias McNulty

unread,
Apr 13, 2019, 2:19:19 PM4/13/19
to django-developers
On Sat, Apr 13, 2019 at 12:07 PM Florian Apolloner <f.apo...@gmail.com> wrote:
As expressed at Djangocon Europe, I am hugely in favor for adopting black.

A quick glance suggests this would result in ~20k lines changed (~27% of non-whitespace, non-comment code lines) in django/ and ~58k lines changed (~41%) in tests/ (with `--line-length=119`). After the initial pass to fix everything `black -l 119 tests/ django/` takes only 2.3 seconds to run (!), though I'm sure my OS had all the files cached at that point.

Without normalizing strings, the number of changed lines comes down to ~14k (~19%) for django/ and ~23k (~16%) for tests/ (with `--skip-string-normalization --line-length=119`).

I'm sure some people will see this as "a ton" and others "only that much?" :-)

(My line counts are imperfect and approximate, and you'll likely get somewhat different numbers if you try this yourself.)

If we choose to do this there are a few things to consider:

 * Line-length, we probably want to stay at 119 I guess

Perhaps we could compromise and choose something in the middle, for the reasons noted in the docs: https://black.readthedocs.io/en/stable/the_black_code_style.html#line-length

We should also consider adding max-doc-length for flake8 to our setup.cfg, and/or forego the shorter requirement for comments/docstrings if we shorten the maximum line length for code.

This section in the docs will need updating if we hand this off to a 3rd-party formatter (in particular the sentence, "Don’t limit lines of code to 79 characters if it means the code looks significantly uglier or is harder to read." since we'll no longer have the option of shortening lines to 79 characters arbitrarily).

 * String normalization, I'd be in favor of following black defaults here, but I am open to be convinced otherwise

It'd be nice to minimize the diff, but I'd be fine either way. If we're going to do it eventually, better to do so all at once.
 
 * We will need to adjust the isort configuration ( https://github.com/ambv/black/blob/master/README.md#how-black-wraps-lines )

+1

Overall, the main downside I see is the large, fairly useless changeset this would create and the impact it would have, such as reduced utility of `git blame` for some portion of the code for the foreseeable future and interruption to all in-progress PRs. Problems that can be worked around, sure, but it would be an interruption to ongoing work nonetheless.

One question: Would we want to update some/all code samples in the docs with the same code style? I imagine that could be a fairly time-intensive endeavor, unless someone's already found a way to automate the process. In particular, Black's format for chaining method calls differs from Django's preferred style. Applying this standard to the docs without discretion may lead to less readable docs, IMO...on the other hand, it's odd to have code samples that don't pass our own tests...

In any case, while this sounds like a non-trivial project with some questions to work through yet, I too like not having to worry about code formatting and don't have any real objections to this.

Cheers,


Tobias McNulty
Chief Executive Officer

tob...@caktusgroup.com
www.caktusgroup.com

Adam Johnson

unread,
Apr 13, 2019, 2:23:13 PM4/13/19
to django-d...@googlegroups.com
+1

I’d like to see gradual roll out (one module at a time? One rule at a time?) to spread the work of reformatting all the in-flight patches.


For more options, visit https://groups.google.com/d/optout.
--
Adam

Tobias McNulty

unread,
Apr 13, 2019, 2:23:39 PM4/13/19
to django-developers
On Sat, Apr 13, 2019 at 11:35 AM Herman S <herman....@gmail.com> wrote:
I've set up how this _could_ look depending on some configurables in Black:

* Default config: https://github.com/hermansc/django/pull/1
* Line length kept at 119: https://github.com/hermansc/django/pull/3
* Line length kept at 119, no string normalization:
https://github.com/hermansc/django/pull/2

Apologies, somehow I missed these links on the first read. Go here for an overview of lines changed, don't use my numbers. :-)

Tobias

Berker Peksağ

unread,
Apr 13, 2019, 3:41:38 PM4/13/19
to django-d...@googlegroups.com
On Sat, Apr 13, 2019 at 9:19 PM Tobias McNulty <tob...@caktusgroup.com> wrote:
> Overall, the main downside I see is the large, fairly useless changeset this would create and the impact it would have, such as reduced utility of `git blame` for some portion of the code for the foreseeable future and interruption to all in-progress PRs. Problems that can be worked around, sure, but it would be an interruption to ongoing work nonetheless.

I came here to say exactly this. Django's codebase is already pretty
consistent with its own style and I think having a usable "git blame"
is much more important than starting to use a new code formatter. IMO,
with a codebase as big as Django, trying to adapt a different style
guide would only cause unnecessary code churn.

The following style is much more readable (and it also makes future
diffs less noisy since we can add new formats without having reformat
code and comments) than black-formatted code:

TIME_INPUT_FORMATS = [
'%H:%M:%S', # '14:30:59'
'%H:%M:%S.%f', # '14:30:59.000200'
'%H:%M', # '14:30'
]

Reformatted by black:

TIME_INPUT_FORMATS = ['%H:%M:%S', '%H:%M:%S.%f', '%H:%M'] # '14:30:59'
# '14:30:59.000200' # '14:30'

(Taken from https://github.com/hermansc/django/pull/2/files#diff-ba8335f5987fcd81d41c28cd1879a9bfL370)

(Also, thank you for creating PRs with different black configurations, Herman.)

--Berker

Florian Apolloner

unread,
Apr 13, 2019, 4:18:30 PM4/13/19
to Django developers (Contributions to Django itself)
Hi,

On Saturday, April 13, 2019 at 9:41:38 PM UTC+2, Berker Peksağ wrote:
I came here to say exactly this. Django's codebase is already pretty
consistent with its own style and I think having a usable "git blame"
is much more important than starting to use a new code formatter.

I am not sure I agree with this argument. We already have had big code changes which make git blame useless (if you are looking for a commit from like 8 years ago or so). All we are doing to do is adding one commit more to that list (and the tooling to work around one such a commit is good, ie git-hyper-blame, git gui blame etc etc).
 
with a codebase as big as Django, trying to adapt a different style
guide would only cause unnecessary code churn.

Maybe, but I think the benefits outweigh the costs -- and I also do not think that it is unnecessary. Our goal has always been to make contributing easier, well nowadays black is in the position to do just that.

The following style is much more readable (and it also makes future
diffs less noisy since we can add new formats without having reformat
code and comments) than black-formatted code:

Yes, the formats (same for settings files) are imo certainly files we could exclude from black. That said those files are highly specific and hardly representative for python code.

Cheers,
Florian

Florian Apolloner

unread,
Apr 13, 2019, 4:21:23 PM4/13/19
to Django developers (Contributions to Django itself)
On Saturday, April 13, 2019 at 8:23:13 PM UTC+2, Adam Johnson wrote:
I’d like to see gradual roll out (one module at a time? One rule at a time?) to spread the work of reformatting all the in-flight patches.

My thoughts initially. But then I realized that it would be easier for tooling if there were just one commit to exclude (also easier for developers to remember that one) and commits that get merged while we are converting module by module would be very careful over which files to run black… 

James Bennett

unread,
Apr 13, 2019, 4:21:48 PM4/13/19
to django-d...@googlegroups.com
On Sat, Apr 13, 2019 at 1:18 PM Florian Apolloner <f.apo...@gmail.com> wrote:
Maybe, but I think the benefits outweigh the costs -- and I also do not think that it is unnecessary. Our goal has always been to make contributing easier, well nowadays black is in the position to do just that.

I feel like Black is useful for projects that don't already have and enforce a style guide. But we already have and enforce a style guide, and some parts of it are things Black can't auto-enforce. So I'm not sure I see what benefit we'd get out of Black, and the downside is a lot of churn. 

Carlton Gibson

unread,
Apr 13, 2019, 6:34:59 PM4/13/19
to django-d...@googlegroups.com
We spend a lot of time spotting small formatting errors and then asking for those to be fixed and then waiting for an update. This wastes reviewer time and slows down the feedback cycle. Many pull requests drag out because of it. 

For this reason I would be 100% behind adopting black, and applying that in a single commit. 

My only reservation is about the history. I don’t know the tools Florian mentioned but I wonder if there is a clever git blame usage that would allow bringing over a single commit change of this sort?

Even if we can’t solve that I’d still be +1 for the added ease to contributors and the speed benefits to review efforts. 

C.

Berker Peksağ

unread,
Apr 13, 2019, 7:04:22 PM4/13/19
to django-d...@googlegroups.com
On Sun, Apr 14, 2019 at 1:34 AM Carlton Gibson <carlton...@gmail.com> wrote:
>
> We spend a lot of time spotting small formatting errors and then asking for those to be fixed and then waiting for an update. This wastes reviewer time and slows down the feedback cycle. Many pull requests drag out because of it.

But you can now do final edits (cosmetic changes, tweak reST markup
and documentation) via GitHub UI, right? I always do it to not bother
contributors with these changes, especially if they are new to the
project.

--Berker

Nick Sarbicki

unread,
Apr 14, 2019, 3:11:58 AM4/14/19
to Django developers (Contributions to Django itself)
Just going to say that one of the main frustration points I've had when making a contribution is having to fix small formatting errors (often minor things in docstrings which aren't _always_ consistent).

It produces a lot of inertia and can stop PRs from getting merged for extended periods of time. So from a new contributor angle I think black is an obvious choice.

From a maintainers point of view, git blame is very nice to have and this would add some complexity. But it's easy to see the full version history of a file and, let's be honest, not enough people use git bisect.


--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

Curtis Maloney

unread,
Apr 14, 2019, 4:22:46 AM4/14/19
to django-d...@googlegroups.com
So to summarise the discussion so far:

1. automated code formatting will be a great boon - reduce work, lower
barrier for new committers, reduce work for the fellows, etc.

2. there are issues with git history in doing "the great commit".

3. there are issues with black's formatting choices.


So to address 1:

I am entirely +1 for automated code formatting to take the work out of
our hands.

Can such a tool be automated into, say, github in a way that doesn't
create extra commit noise?

To address 2:

I side with those who favour a progressive solution, whereby new code
only has the new tool applied.

That said, there might be cause to suggest a deadline [perhaps a N.0
release?] where all remaining code is "cleaned".

And finally 3:

My perspective on the goal of any code formatting tool is this:

When we as developers approach a piece of code, our goal is the
understand its intent and its implementation of that intent.

In the process of reaching that, we pass through the stages of (a)
identifying the relevant code, (b) understanding the action of the code,
and (c) understanding the intent of the code.

Good code formatting choices will remove or reduce the cognitive load in
(b) and (c).

In my experience with using black [we use it at work], there are
numerous choices (including those demonstrated in this list already)
where it can significantly _increase_ the cognitive load in simply
parsing the code.

As simple as black can make the job of code formatting, I feel I'd
rather see a different tool that retained the benefits of "trivial code
reformatting", but still allowed us to retain some of Django's existing
code formatting rules.

(An interesting [and defensible] choice, we had a module with a lot of
strings wrapped across lines. black opted to push them onto the same
line, but NOT to merge them. This is because in Python prior to 3.7, it
would have altered the generated AST - one of the guides black uses)

--
Curtis

Christian González

unread,
Apr 14, 2019, 5:48:15 AM4/14/19
to django-d...@googlegroups.com
Hi
> 2. there are issues with git history in doing "the great commit".

While I don't have much experience neither in Python/Django nor in git,
maybe a view fom "outside" may add something useful to the discussion.

I'm just seeing that discussion is separated between on the one hand
"black is good. It would help Django, newcomers' entry barrier etc." -
and OTOH "black adds a commit to Django that prevents git blame to work
properly."

So, excuse me to blasphemically ask the question: Isn't that a problem
with git? I always asked myself how to properly fix whitespace "bugs"
without destroying the "git blame" history.

What would in my optinion solve that issue, and hence allow black to be
adopted completely in Django without regrets, would be a feature in git
to exclude certain commits, or even chunks, from git blame history. This
could be done by having a .blameignore file like .gitignore, listing
commit hashes that should be ignored in the blame history. So it could
be made clear and configurable per project, and even modified after
committing - because you can't change git history itself - which code
chunks are just "cosmetical" and have not to be treated as real "code".

I knot this has nothing to do with Django itself - but really, NOT
adopting black because of "git blame" creating wrong results would be
the wrong approach IMHO.

If you're not saying that I'm completely insane, or git already has this
feature, or Ii forgot something, I could try and add a git feature
request for that. Even if implemented in 2 years, It could be made sure
later that git blame worked correctly backwards then.

TL;DR: Use black with Django without compromise, fix git.

Christian

--
Dr. Christian González
https://nerdocs.at

pEpkey.asc

Florian Apolloner

unread,
Apr 14, 2019, 6:06:38 AM4/14/19
to Django developers (Contributions to Django itself)
Hi,


On Sunday, April 14, 2019 at 1:04:22 AM UTC+2, Berker Peksağ wrote:
I always do it to not bother
contributors with these changes, especially if they are new to the
project.

This is imo not something that scales well. Also it is not something I want to pay our fellows for, I rather have them use their time on something else. It is true that it is probably easier and faster to just fix code instead of back & forth with the contributor, but the main argument for me here is that this is work that shouldn't be needed in the first place.

Cheers,
Florian

Florian Apolloner

unread,
Apr 14, 2019, 6:10:56 AM4/14/19
to Django developers (Contributions to Django itself)


On Saturday, April 13, 2019 at 10:21:48 PM UTC+2, James Bennett wrote:
But we already have and enforce a style guide, and some parts of it are things Black can't auto-enforce.
 
We do? I mean sure, we do have a styleguide, but enforcing it? It all depends on how well our fellows (manually) enforce it. So even if there are things black cannot auto-enforce, we are currently not enforcing __anything__. And even our great fellows do make mistakes from time to time, something black could prevent (at least in the areas that it could enforce).

Do not get me wrong, I am not trying to bash our fellows here for sometimes making mistakes. But arguing that we are already enforcing a styleguide seems a bit far stretched, even if our fellows are doing an astonishing job there.

Cheers,
Florian

P.S.: I said fellows a few times in this mail, but reviewer/committers would probably be better, even though I think that most of the final fixing is currently on the fellows.

Florian Apolloner

unread,
Apr 14, 2019, 6:15:39 AM4/14/19
to Django developers (Contributions to Django itself)
Hi Christian,

I think you are making a good argument here. On one hand short-comings in our tool chain should not block us from making changes. On the other hand, we do have to live with them -- so having at least some technical solution towards the "git blame" issue is needed. I guess the chromium team also realized this, which led to the development of tools like git-hyper-blame (see https://commondatastorage.googleapis.com/chrome-infra-docs/flat/depot_tools/docs/html/git-hyper-blame.html -- to be fair I haven't tried it yet).

Cheers,
Florian

Andrew Godwin

unread,
Apr 14, 2019, 6:16:13 AM4/14/19
to Django developers (Contributions to Django itself)
We've used Black for the Channels/ASGI projects for about the last year, and I have nothing but good opinions on it from that perspective. It's made pull requests much easier to get formatted, because fixing it is just a case of running a single command (and if you do the right pre-commit hook, you never even get that far).

As for Django, the single biggest problem I can see is the single reformat commit, but I think this is worth it. We've had a few of those in our history - and indeed at work - and while it is a bit of a pain, GitHub has a "see the blame before this commit" button to enable much easier browsing of files prior to it.

For that reason, I am +1 to using the Black code formatter.

Andrew


--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

Florian Apolloner

unread,
Apr 14, 2019, 6:20:31 AM4/14/19
to Django developers (Contributions to Django itself)
Hi,


On Sunday, April 14, 2019 at 10:22:46 AM UTC+2, Curtis Maloney wrote:
Can such a tool be automated into, say, github in a way that doesn't
create extra commit noise?

Probably, but after blacking (is that even a word ;)) the codebase once there shouldn't be much commit noise.

I side with those who favour a progressive solution, whereby new code
only has the new tool applied.

I think that would make it hard on tools like git-hyper-blame which allow one to skip revisions. Also you cannot just run black over changes, the minimal unit is one file as far as I am aware. Which would make PR very very hard to review because you'd have a bunch of syntax changes next to real changes.

In my experience with using black [we use it at work], there are
numerous choices (including those demonstrated in this list already)
where it can significantly _increase_ the cognitive load in simply
parsing the code.

Generally I am okay with the way black formats code. But I have to admit that for some very tricky codepaths I tend to turn it off from time to time. That does not happen very often though.

As simple as black can make the job of code formatting, I feel I'd
rather see a different tool that retained the benefits of "trivial code
reformatting", but still allowed us to retain some of Django's existing
code formatting rules.

Open to suggestions, did you just offer to write one :)

(An interesting [and defensible] choice, we had a module with a lot of
strings wrapped across lines. black opted to push them onto the same
line, but NOT to merge them.  This is because in Python prior to 3.7, it
would have altered the generated AST - one of the guides black uses)

We can and should fix those occurrences in our codebase then I guess.
 
Cheers,
Florian

Josh Smeaton

unread,
Apr 14, 2019, 9:10:48 AM4/14/19
to Django developers (Contributions to Django itself)
Agree with Florian that the progressive rollout is more trouble than it's worth. Tangling up feature changes with whole file formatting will make it harder to review changes, but will also be more difficult to use tools like git blame.

As for disagreeing with some of Blacks choices - you learn very quickly to live with those choices, and forget those choices. Any set of configuration we could come up with for an existing tool would likely be bikeshedded to death and nothing would eventuate.

For line lengths, projects I work on set Black to 100 columns, but flake8 line length is capped to 120. This lets some long strings (black doesn't break long strings over multiple lines) get past flake8 without being over the top.

I'm in favour of using blacks double quotes for strings. I **hated** this decision when it was first made, but have seriously come around to it, and prefer it aesthetically too. Consistency with other projects (most big projects are adopting black) I feel is a good goal. The diff is already going to be huge, there's not much value in protecting a few strings.

Finally, there are some tricks you pick up if black fights you on some decisions. To use Berkers example:

TIME_INPUT_FORMATS = [ 
    '%H:%M:%S', # '14:30:59' 
    '%H:%M:%S.%f', # '14:30:59.000200' 
    '%H:%M', # '14:30' 


TIME_INPUT_FORMATS = ['%H:%M:%S', '%H:%M:%S.%f', '%H:%M'] # '14:30:59' 
# '14:30:59.000200' # '14:30' 

Break each individual format into its own variable, with appropriate comment, and add the variables to the list.

HMS = "%H:%M:%S"  # 14:30:59
HMSF = ".." 
HM = ".."
TIME_INPUT_FORMATS  = [HMS, HMSF, HM]

Obviously just an example, but something to keep in mind.

Jon Dufresne

unread,
Apr 14, 2019, 9:29:36 AM4/14/19
to django-d...@googlegroups.com
> 2. there are issues with git history in doing "the great commit".

Tools make a real difference here. I know everyone has their preferred tools and workflows, but there are tools that let one move through iterations of git blame. This is something I do quite frequently. Just one example, in Emacs the vc-annotate command (bound to "C-x v g") shows a view of git blame. The user can hit "a" to go to the previous diff the line appeared on which might be more interesting. So one could use this feature to skip over the Black commit on to a more interesting one. This is just a single example. Florian's suggested tools may handle this even more elegantly, but I haven't used them. This only goes to show that the history and git blame concerns don't need to be a show stopper and can be solved. The feature people are asking for, "skipping over" git commits in blame, does exist for some tools. It even exists in GitHub's blame view. The button is labeled "View blame prior to this change".


--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

Aymeric Augustin

unread,
Apr 14, 2019, 9:40:10 AM4/14/19
to django-d...@googlegroups.com
Hello,

I'm strongly in favor of adopting black with the default options.

In my opinion, it's the first Python code formatter that passes the bar of "on average, does better than any single programmer". Trying to enforce something else manually is a waste of energy that produces worse results.

I like nicely formatted code and I hate some of what black produces, typically on Django model definitions. However, I've seen the benefits of an automated code formatter, even on projects where I write almost all of the code.

I'm positive the benefits will be great on Django, where you can tell when a module was (re)written just by looking at the style. Yes, there are some downsides, but it's clearly a good tradeoff.

Regarding the migration strategy, converting one package at a time sounds good, because then we can proof-read the Big Diff and perhaps tweak the code where the result really hurts the eyes.

Most of Django's style guide will still applies, as it's mostly conventions about naming things, ordering declarations, etc.

Best regards,

-- 
Aymeric.



--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

Berker Peksağ

unread,
Apr 14, 2019, 9:48:33 AM4/14/19
to django-d...@googlegroups.com
On Sun, Apr 14, 2019 at 1:06 PM Florian Apolloner <f.apo...@gmail.com> wrote:
> This is imo not something that scales well. Also it is not something I want to pay our fellows for, I rather have them use their time on something else. It is true that it is probably easier and faster to just fix code instead of back & forth with the contributor, but the main argument for me here is that this is work that shouldn't be needed in the first place.

True, having an automated code formatter would definitely help to save
reviewer a bit time. However, Django has already flake8 checker on
GitHub, which would already help a contributor to fix biggest
inconsistencies in their PR. Note that my point in my earlier email
was that there are more minor details than just cosmetic fixes and
unfortunately you can't automatically fix all of them :) So, a final
round of edits still needed.

I've used git-hyper-blame before. Most of the time you need to
manually collect all "mass cleanup" commits in a file (and you need to
maintain that file separately) It's also not easy to integrate it with
your workflow especially if you're using "git blame" from your IDE of
choice.

--Berker

Tobias McNulty

unread,
Apr 14, 2019, 11:00:46 AM4/14/19
to django-developers
On Sun, Apr 14, 2019 at 3:11 AM Nick Sarbicki <nick.a....@gmail.com> wrote:
Just going to say that one of the main frustration points I've had when making a contribution is having to fix small formatting errors (often minor things in docstrings which aren't _always_ consistent).

I constantly need to be reminded to shorten docstrings and comments (and sometimes remove "We" from the beginning of such strings). Unfortunately (though understandably), Black won't help with any of these issues, and we'll still be left with the same back and forth about grammar, wrapping, etc. in comments and docstrings. (Again, as of mid-2018 there is the max-doc-length option for flake8 that could help with this somewhat, but that change can be made independently.)

I'm not arguing against Black adoption, but I don't think it's going to eliminate from the review process what may sometimes feel like inertia...

Tobias

charettes

unread,
Apr 14, 2019, 7:02:49 PM4/14/19
to Django developers (Contributions to Django itself)
I was and and I'm still bugged by how black decided to go for double quotes instead of single ones. Even if it's backed some valid arguments it left all the projects following PEP 8's recommendations over the years with this git blaming loss trade off. From past experimentation with large'ish projects following PEP8's guideline string quotes changes represented well over 50% of the diff and there's no way to turn off only this form of string normalization, it's all or nothing. At $WORK we even went as far as forking the project to switch to use single quotes instead[0]. I understand requiring the use of a fork is not a feasible solution for Django but I just wished there was an option to stick to single quotes to reduce the cost of adoption for such projects without loosing all the other nifty string formatting goodies[1] /rant

Even if I don't completely agree with black's formatting choices and the sometimes quite unnatural code it generates my past month's usage at work and the adoption of the project in the Python ecosystem that made it's way in most IDE makes me mostly share the same feeling as Aymeric; it's not always pretty but not having to think about it makes it worth it. I suggest we stick to disabling string normalization to reduce the noise generated by the migration though.

Cheers,
Simon


-- 
Aymeric.



To unsubscribe from this group and stop receiving emails from it, send an email to django-d...@googlegroups.com.

Scot Hacker

unread,
Apr 15, 2019, 2:40:57 AM4/15/19
to Django developers (Contributions to Django itself)
 Just bringing this up for the sake of thoroughness: 

NOTE: This is a beta product

Black is already successfully used by several projects, small and big. It also sports a decent test suite. However, it is still very new. Things will probably be wonky for a while. This is made explicit by the "Beta" trove classifier, as well as by the "b" in the version number. What this means for you is that until the formatter becomes stable, you should expect some formatting to change in the future. That being said, no drastic stylistic changes are planned, mostly responses to bug reports.

Many/most of us use it in production settings already, as the AST check guarantees it isn't going to break anything, but the beta status means you can't always do a simple `pip install black`, and it means that black's formatting choices could change. 

Does a project as significant as Django want to adopt a beta tool whose "contract" with its users has not yet stabilized?

I heart black and would be happy to see it in Django but wonder if we should wait a bit to hit 1.0.

./s

Adam Johnson

unread,
Apr 15, 2019, 3:06:29 AM4/15/19
to django-d...@googlegroups.com
Lukasz Langa, the developer, said he will tag the first non-beta version in "Early April." https://twitter.com/llanga/status/1106247623802060802 There won't be any significant changes when it leaves beta afaiu.


--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

For more options, visit https://groups.google.com/d/optout.


--
Adam

Florian Apolloner

unread,
Apr 15, 2019, 4:12:13 AM4/15/19
to Django developers (Contributions to Django itself)
Hi Simon,


On Monday, April 15, 2019 at 1:02:49 AM UTC+2, charettes wrote:
 and there's no way to turn off only this form of string normalization, it's all or nothing.
 
So the black --help output lists:
-S, --skip-string-normalization Don't normalize string quotes or prefixes.
Or does that not what you wanted for your company?

Cheers,
Florian

Florian Apolloner

unread,
Apr 15, 2019, 4:15:06 AM4/15/19
to Django developers (Contributions to Django itself)
Hi, as Adam points out 1.0 is on it's way. My mails were written with that knowledge and assumption, sorry for not making this clear. I did actually had in mind to wait till it is out of beta -- which is most likely before we finished our discussions :D

Cheers,
Florian

On Monday, April 15, 2019 at 9:06:29 AM UTC+2, Adam Johnson wrote:
Lukasz Langa, the developer, said he will tag the first non-beta version in "Early April." https://twitter.com/llanga/status/1106247623802060802 There won't be any significant changes when it leaves beta afaiu.


On Mon, 15 Apr 2019 at 08:40, Scot Hacker <scot....@gmail.com> wrote:
 Just bringing this up for the sake of thoroughness: 

NOTE: This is a beta product

Black is already successfully used by several projects, small and big. It also sports a decent test suite. However, it is still very new. Things will probably be wonky for a while. This is made explicit by the "Beta" trove classifier, as well as by the "b" in the version number. What this means for you is that until the formatter becomes stable, you should expect some formatting to change in the future. That being said, no drastic stylistic changes are planned, mostly responses to bug reports.

Many/most of us use it in production settings already, as the AST check guarantees it isn't going to break anything, but the beta status means you can't always do a simple `pip install black`, and it means that black's formatting choices could change. 

Does a project as significant as Django want to adopt a beta tool whose "contract" with its users has not yet stabilized?

I heart black and would be happy to see it in Django but wonder if we should wait a bit to hit 1.0.

./s

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-d...@googlegroups.com.


--
Adam

Dmitriy Sintsov

unread,
Apr 15, 2019, 4:47:01 AM4/15/19
to Django developers (Contributions to Django itself)
Why can't it use mixed quotes, single quotes for all strings except the strings that contain single quote characters? I think mixed syntax of strings is made in various programming languages so both single and double quotes can be used inside strings not having to use ugly backslash escaping. Maybe you or someone can create such feature request in black repo? Why the people do not like mixed quotes strings and try to eliminate such great feature? Aesthetical issues are very much biased and enforcing these automatically is not always a good idea. Does the switching to double quote string makes the code error prone and safer? Probably not. Just some voluntary personal choice.

Kye Russell

unread,
Apr 15, 2019, 4:48:31 AM4/15/19
to django-d...@googlegroups.com
This is something discussed af length on the black issue tracker and not something the author wishes to change. 

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

Josh Smeaton

unread,
Apr 15, 2019, 5:53:38 AM4/15/19
to django-d...@googlegroups.com
Yes, let’s not discuss the tools choice in quotes, that’s for another place. To provide some information though, black will not convert single quote strings if the string itself contains a double quote. It won’t escape the character, it’ll use single quotes. 

You received this message because you are subscribed to a topic in the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-developers/wK2PzdGNOpQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-develop...@googlegroups.com.

To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

René Fleschenberg

unread,
Apr 15, 2019, 7:13:56 AM4/15/19
to django-d...@googlegroups.com
Hi

> HMS = "%H:%M:%S"  # 14:30:59
> HMSF = ".." 
> HM = ".."
> TIME_INPUT_FORMATS  = [HMS, HMSF, HM]

Just my two cents: This particular rule is the main reason why I
personally have not adopted black yet. I really find the
one-item-per-line style *much* more readable in many cases.

In natural-language writing, there are occasions where you use "foo,
bar, baz" and other occasions where you use

- foo
- bar
- baz

IMO, both variants have their place in Python code, too. I'd like to see
a black variant which allows to leave this decision to a human.

--
René

Ned Batchelder

unread,
Apr 15, 2019, 8:05:37 AM4/15/19
to django-d...@googlegroups.com
Try the latest black.  A multi-line lists with comments on each line is
not converted to a one-line list.  The lined-up indentation of the
comments is lost :(, but it's still multi-line.

This is one of my concerns: Black is still beta, and is still adjusting
the way it formats in small ways.  So a boil-the-ocean formatting of the
code may not be the last time the code will change to suit black.

--Ned.

René Fleschenberg

unread,
Apr 15, 2019, 9:12:10 AM4/15/19
to django-d...@googlegroups.com
Hi Ned.

> Try the latest black.  A multi-line lists with comments on each line is
> not converted to a one-line list.  The lined-up indentation of the
> comments is lost :(, but it's still multi-line.

Hmmm, I tried commit cea13f4 (current master at the time of this writing).

It turned

l = [
1, # one
2, # two
3, # three
]

into

l = [1, 2, 3] # one # two # three

Maybe in your case the line limit kicked in and prevented the reformatting?

> This is one of my concerns: Black is still beta, and is still adjusting
> the way it formats in small ways.  So a boil-the-ocean formatting of the
> code may not be the last time the code will change to suit black.

I agree.

--
René

Tim Allen

unread,
Apr 15, 2019, 9:38:51 AM4/15/19
to Django developers (Contributions to Django itself)
Another plus one for Black, without getting too into the weeds. I think the time is will save likely outweigh the valid concerns posted here. I've used Black extensively on several projects, and much like f-strings, the last Pink Floyd album, and broccoli, have found I really like something I didn't think I would.

I've gone through the churn of style edits each time I've made a PR to Django, despite using flake8. Having an automated tool to avoid that back-and-forth time would be a big help. It would also help reduce feelings of impostor syndrome; I know I've kicked myself for making basic formatting mistakes. Submitting to a project like Django can be intimidating, and this would remove some of the intimidation factor. I like to ask with every major change, "What does it mean for the newcomer?" Much like the new 2.0 URL routing syntax, I think this is  a slam-dunk.

While I prefer longer line lengths, I agree we should just stick with Black's defaults. Their commentary on the choice of 88 characters is compelling: it saves quite a bit of space over PEP-8's 79, but stays readable for the greater majority of people. Keeping the code base accessible to as many as possible is a big plus. I also find that with 88 character limits, I can keep three files open on a 16x9 screen in an editor, which is a nice editor workspace while working on Django (models file, views file, template). While Black may change in the future, I doubt the changes would be so large as to cause us to need another "huge commit and merge."

Regards,

Tim


charettes

unread,
Apr 15, 2019, 9:58:28 AM4/15/19
to Django developers (Contributions to Django itself)
Florian,

This option turns off *all* string normalization and not only quote normalization.

We wanted to still have prefixes stripping because of our usage of __future__.unicode_literals for example.

Cheers,
Simon

Emil Madsen

unread,
Apr 16, 2019, 4:12:20 AM4/16/19
to Django developers (Contributions to Django itself)
Hello,

I'm not sure if it's relevant for the discussion, but black requires python 3.6.0+.
Several Linux distros does not ship 3.6.0+ yet, including, but not limited to:

* Debian Stretch (current LTS)
* FreeBSD 12 (current release)
* Ubuntu 16.04 Xenial (previous LTS)
* Linux Mint 18.1 (previous LTS)
* OpenSuse Leap 42 (previous LTS)
* Slackware 14.1 (previous LTS)
* Fedora 27 (two versions old)

While it is ofcourse possible to install python 3.6.0+ on these systems, it requires more effort than a simple package manager install.

Florian Apolloner

unread,
Apr 16, 2019, 5:04:20 AM4/16/19
to Django developers (Contributions to Django itself)
Hi Emil,

this would only affect Django 3.0 which supports only python3.6 -- as such you couldn't even run Django on those systems anyways.

Cheers,
Florian

Christian González

unread,
Apr 16, 2019, 5:49:50 AM4/16/19
to django-d...@googlegroups.com

Am 14.04.19 um 12:15 schrieb Florian Apolloner:
I've filed a feature request for git itself
(https://public-inbox.org/git/5df051ce-007c-cc12...@nerdocs.at/T/),
and it seems that there is a feature within git which is in development
and seems to deal with that blame-before-not-important-commits issues.
Like I said there, it's not perfect IMHO, and would be done easier by
using a config file and individual/project specific commit takeouts for
blaming. That's certainly seen for long-term, but When Django 3.0 will
arrive, hopefully with black 1.x, it's early enough for "git with blake
skip support" too. And, as it's only for development, It would be fully
backwards compatible for git users that don't have support for that
feature - they could always use black/git and skip blame the "old" way.

That said, I hope black will be used in Django.

Christian

--
Dr. Christian González
https://nerdocs.at

pEpkey.asc

Jacob Rief

unread,
Apr 16, 2019, 10:37:13 AM4/16/19
to Django developers (Contributions to Django itself)
To address some of Curtis Maloney's concerns:

1. automated code formatting will be a great boon - reduce work, lower
barrier for new committers, reduce work for the fellows, etc.

2. there are issues with git history in doing "the great commit".

3. there are issues with black's formatting choices.

 
How about, instead of reformatting all the code in one big go, discarding the "blame"-history,
instead we use a tool which only checks for formatting violations and prevents to commit such
code.

Such a tool, for instance is Coala https://coala.io/#/about?lang=Python
This could be added as a Pre-Commit Hook and prevents contributors to create non-conformant
branches on GitHub. This would address the problem of some pull requests not being merged
for a long period, because of minor formatting issues.

It also would enforce new coding styles over a longer period, keeping code annotations until
someone touches a file.

Mariusz Felisiak

unread,
Apr 16, 2019, 10:54:30 AM4/16/19
to Django developers (Contributions to Django itself)

This would address the problem of some pull requests not being merged
for a long period, because of minor formatting issues.

We don't have such PRs. If PRs have tests, docs (when required) and solutions are of good quality we would fix minor formatting issues before merge. It never stopped us from merging.

charettes

unread,
Apr 16, 2019, 10:55:44 AM4/16/19
to Django developers (Contributions to Django itself)
I feel like it would be worse as it would break formatting changes across multiple commits,
creating an inconsistent formatting code base and generating large unrelated changes diff
when altering a still untouched file which would make the review harder.

Cheers,
Simon

Ivan Anishchuk

unread,
Apr 16, 2019, 1:38:19 PM4/16/19
to django-d...@googlegroups.com
My two cents: black's mostly ok but I'm against any auto-formatting. Code style is much more than whitespace arrangement, it requires looking, creative thinking, judging, and sometimes actually compromising rules and looks for readability while the usual attitude with such tools is "I have this autoformatter tool so I don't have to think about style anymore, see, all my whitespace is nice and shining". Using some helpers in your favorite editor is great but relying on automatics without looking is almost completely counter-productive.

I find that PR autocommenter is a great way to keep issues detectable by flake8/pylint to a minimum.

Ivan.

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

Josh Smeaton

unread,
Apr 16, 2019, 5:36:05 PM4/16/19
to django-d...@googlegroups.com
Ivan, what you’re talking about is subjective code formatting, and lends itself to extreme bikeshedding and little consensus. Django already has a formatting standard that mergers try to enforce at review and commit time. But it’s time consuming, prone to missing things, and requires lots of back and forth. 

Not having to think of formatting at all, either as a developer or reviewer, is a major productivity win. Even if some of the formatting choices are not agreeable. 

You received this message because you are subscribed to a topic in the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-developers/wK2PzdGNOpQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-develop...@googlegroups.com.

To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

Ivan Anishchuk

unread,
Apr 16, 2019, 7:45:11 PM4/16/19
to django-d...@googlegroups.com
Yeah, it's very common to confuse style with formatting. Not having to think is a productivity win when and only when you don't care about your product's quality (in which case I'd suggest using generators and stop thinking about code altogether). Language is a communication tool, a programming language is a tool programmers use to talk to each other, and code passed through autoformatters (especially if author _didn't think_ about style) is harder to understand then written by thinking humans. It's pretty much like trying to stop thinking about style in English -- by using some processor to put your commas and whitespace in place you can generate something that is usually possible to understand, at least for the most part, but that and _good style_ would still be worlds apart (yes, natural languages are different, it's much easier to make things worse there, but it's not any easier to make things better in programming languages).

There are situations where compromising style for formatting is acceptable and even a good idea, I don't think here is one of them. Removing a significant expressive tool from a language just to ensure your whitespace is arranged according to some arbitrary rules is about as counter-productive and pointless as it can get. Not because sometimes it produces some bad results but exactly because it prevents humans from thinking and expressing themselves properly. What would you think about some processor renaming all your variables so that you don't have to think about naming them? Must be even better for productivity.

Ivan.

Josh Smeaton

unread,
Apr 16, 2019, 8:17:41 PM4/16/19
to Django developers (Contributions to Django itself)
We aren't talking about code minifiers though, are we? We're talking about a very specific tool with very specific rules. No one will ever agree on one specific code style, which is why subjectivity is anti-productive. Black chooses a specific set of rules and removes ambiguity. Some choices will be agreeable, others will not be. And the ones that are agreeable aren't agreeable to every person.

> and code passed through autoformatters (especially if author _didn't think_ about style) is harder to understand then written by thinking humans

I find this statement to be mostly incorrect with regard to Black. There are edge cases, but they are few.

On Wednesday, 17 April 2019 09:45:11 UTC+10, Ivan Anishchuk wrote:
Yeah, it's very common to confuse style with formatting. Not having to think is a productivity win when and only when you don't care about your product's quality (in which case I'd suggest using generators and stop thinking about code altogether). Language is a communication tool, a programming language is a tool programmers use to talk to each other, and code passed through autoformatters (especially if author _didn't think_ about style) is harder to understand then written by thinking humans. It's pretty much like trying to stop thinking about style in English -- by using some processor to put your commas and whitespace in place you can generate something that is usually possible to understand, at least for the most part, but that and _good style_ would still be worlds apart (yes, natural languages are different, it's much easier to make things worse there, but it's not any easier to make things better in programming languages).

There are situations where compromising style for formatting is acceptable and even a good idea, I don't think here is one of them. Removing a significant expressive tool from a language just to ensure your whitespace is arranged according to some arbitrary rules is about as counter-productive and pointless as it can get. Not because sometimes it produces some bad results but exactly because it prevents humans from thinking and expressing themselves properly. What would you think about some processor renaming all your variables so that you don't have to think about naming them? Must be even better for productivity.

Ivan.

To unsubscribe from this group and stop receiving emails from it, send an email to django-d...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-developers/wK2PzdGNOpQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-d...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-d...@googlegroups.com.

Dan Davis

unread,
Apr 16, 2019, 8:33:55 PM4/16/19
to Django developers (Contributions to Django itself)
+1 isort
-1 black

I think that codestyle checkers are better, because you teach yourself proper style for python.

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

Matthias Kestenholz

unread,
Apr 17, 2019, 2:48:07 AM4/17/19
to django-d...@googlegroups.com
After dozens of mails it's clear that this is certainly a controversial topic -- especially, as always, string quoting.

I think there is an overwhelming benefit in adopting black and not deviating from the defaults even if the only benefit of this is just never having to discuss these choices again.

Adopting black has helped in all projects I (help) maintain. A few of them are well known, but the specific projects do not matter much.

And I'm one of those people who hesitated because I didn't like many of the choices black made but I adapted really quickly.

Thanks,
Matthias

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.


--
www.feinheit.chbera...@feinheit.ch — +41 555 11 11 41

Tobias Kunze

unread,
Apr 17, 2019, 4:26:47 AM4/17/19
to django-d...@googlegroups.com
Hi Dan,

On 19-04-16 20:33:29, Dan Davis wrote:
>+1 isort
>-1 black
>
>I think that codestyle checkers are better, because you teach yourself
>proper style for python.

I appreciate this argument, but: As Django community our primary concern
in this discussion has to be the impact black would have on the Django
code base and the Django development process – educating our
contributors cannot be a primary concern, compared to making
contributions as easy as possible.

Tobias
signature.asc

Tom Forbes

unread,
Apr 17, 2019, 6:56:41 AM4/17/19
to django-d...@googlegroups.com
I would be +1 for Black. I think it makes a lot of sense for a project like Django, and it does seem that a non trivial amount of both contributor and reviewer time is spent on formatting fixes.

The choice of double quotes by default used to annoy me, but after using black for a while I think that the benefits outweigh the downsides.

One thing we have not considered here is that after running black on Django a huge portion of our outstanding merge requests will have conflicts, some of which might be tricky to rebase. I’m not sure there is much we can do about that though.

Tom
> --
> You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
> To post to this group, send email to django-d...@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/20190417082648.miicsigxjwoevmkl%40cordelia.localdomain.

Jon Dufresne

unread,
Apr 17, 2019, 10:25:54 AM4/17/19
to django-d...@googlegroups.com
> One thing we have not considered here is that after running black on Django a huge portion of our outstanding merge requests will have conflicts, some of which might be tricky to rebase. I’m not sure there is much we can do about that though.

With a little bit of git-foo, this is actually not that bad. In past projects, this is the approach I took:

    $ git checkout my-branch
    # Rebase on the parent of the commit introducing Black.
    $ git rebase <black-commit>^
    # Fix normal merge conflicts.
    # ...
    # Rebase on the Black commit, instructing git to prefer the rebased changes
    # over the black changes. For rebase, the terminology is backwards, hence
    # "theirs".
    $ git rebase <black-commit> -X theirs
    # Reformat your changes with black.
    $ black .
    # Amend the commit with the Black formatting changes.
    $ git commit --ammend -a

Brice Parent

unread,
Apr 17, 2019, 6:47:05 PM4/17/19
to django-d...@googlegroups.com
Le 17/4/19 à 2:33, Dan Davis a écrit :
> +1 isort
> -1 black
>
> I think that codestyle checkers are better, because you teach yourself
> proper style for python.

I'm in a team in which we've enforced Black a bit more than a year ago,
and I can tell you that now, everyone code in a blackish style from
start. Black is still always executed, but doesn't make any change most
of the time. It's just there to enforce the style, for newcomers and if
we sometimes make some style mistakes.

A big win here.

[snip]

Jani Tiainen

unread,
Apr 18, 2019, 9:03:41 AM4/18/19
to 'Anoosha Masood Keen' via Django developers (Contributions to Django itself)
Well let me add my two cents here since I was also in the group in DCEU that talked about the usage of black.

Personally I don't like to contribute to Django. And this is why:

Day one: I'll make the fix/patch and create PR
Day two (or four or five depending how busy reviewers are): I missed a comma or some minor indent is wrong
Day three: I fix styles
Day four: PR is accepted.

So whole round trip took about a five days (give a take few usually depending how busy reviewers are).

That gives me a feeling that I'm really wasting my time and since I can't get all the small bits and pieces exactly as Django wants in correct place.

And that's because we have slightly different rules at the work. And some other projects do have different rules.

So it would be great if some of this pain could be relieved with a tool. In my short experience with black (I've been using it for work projects) it does a pretty decent job.

Like others have said black does some decisions I don't agree with. But I don't have to. Black does it for a "greater good". And after a while black actually vanishes from the flow. 

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

For more options, visit https://groups.google.com/d/optout.


--
Jani Tiainen
Software wizard


Always open for short term jobs or contracts to work with Django.

Luke Plant

unread,
Apr 19, 2019, 3:25:24 AM4/19/19
to django-d...@googlegroups.com

Hi all,

An alternative solution to the problem of nit-picky, formatting related PR reviews is simple: let's not do that.

We already have an automated code formatting enforcer, flake8. Let's simply drop the requirement on fixing anything that flake8 doesn't pick up. A committer can fix up style issues if they want to, but shouldn't make anyone else do it. This would mean most of the stuff on our coding style page should just be delete, or at least not enforced - by which I mean almost anything that can't be enforced by a tool (such as isort, flake8, editors via .editorconfig etc.), and has no non-local effects. (So consistent naming of classes/functions *should* be enforced, because that affects other people's ability to use the code).  Large parts of that page are just duplicating of flake8/isort rules anyway. Honestly, does it kill us if someone writes "we" in a code comment? And black couldn't help with some of these things anyway. Let's just stop being code review jerks.

I'm kind of ambivalent on black itself. Certainly there are cases where it makes code less readable (a significant sin in my book) e.g. lists that are better displayed vertically, as mentioned already, and there are cases where it makes your diff larger than it needs to be (e.g. when it decides a list is now too long and needs to be re-formatted vertically). If we adopt black we'd have to live with those annoyances. Alternatively, we can live with the annoyance that code formatting is not perfectly consistent and we accept less than 'perfect' PR. But we should just live with those things:


    A foolish consistency is the hobgoblin of little minds


And if our consistency requirements are causing problems for people attempting to contribute, they are foolish and should be dropped.

My 2 ¢.

Luke

Tim Graham

unread,
Apr 19, 2019, 8:51:58 AM4/19/19
to Django developers (Contributions to Django itself)
I'm sorry for establishing some (foolish?) guidelines that discouraged some people from contributing. I didn't establish those guidelines merely to be pedantic, but perhaps I'm too much of a perfectionist at times.

After GitHub allowed mergers to amend pull requests, I often made cosmetic adjustments myself to eliminate the trivial back and forth that some have lamented. I never blocked a merge merely for trivial cosmetic reasons.
To unsubscribe from this group and stop receiving emails from it, send an email to django-d...@googlegroups.com.


--
Jani Tiainen
Software wizard


Always open for short term jobs or contracts to work with Django.
--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-d...@googlegroups.com.

Tobias McNulty

unread,
Apr 19, 2019, 10:08:35 AM4/19/19
to django-developers
FWIW, I like that we have high standards for concise and readable comments with proper grammar, especially because I myself often need a little extra push in that department. And I'm not easily convinced by the arguments that sound a whole lot like "I've written the code and don't have time to clean this up." In the thick of the moment it's easy to assume a solution will be obvious to others reading the code, when in reality taking a few extra minutes to clearly write down why something is the way it is (not to mention making the code itself readable in the first place) will save potentially hours of someone else's time later on. This is precisely what code reviews are for; to make sure we're handing down well-written, described, and documented code for those who come after us (including perhaps ourselves, a year or two down the road). The benefit to us as contributors is that it makes us stronger developers, and I think that might get forgotten sometimes.

I've continued to follow along in this thread and remain fairly ambivalent about Black. I see how it will simplify many things; on the other hand, I like my multi-line lists formatted my way. (Not poking fun, I really do.) That said, if Black will truly remove some significant hurdles, let's do it, but we should remain clear-headed about the fact that it's not going to remove from the equation what some may feel is nitpicking and others would argue is a necessary part of maintaining our high standards.

Herman, if you haven't been scared away from this just yet, I might suggest trying to summarize the discussion and your recommendation/proposal given the reactions so far (could be a DEP but doesn't have to be, IMO). I do feel there's a pragmatic solution in here somewhere.

Best to all,
Tobias


Tobias McNulty
Chief Executive Officer

tob...@caktusgroup.com
www.caktusgroup.com


To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

Luke Plant

unread,
Apr 19, 2019, 10:17:53 AM4/19/19
to django-d...@googlegroups.com

Hi Tim,

I didn't mean to imply the original decisions were foolish, in their context. Reading it again, my language in the email was too strong - it was directed at myself as much as anyone else, because I'm also the kind of person who loves getting everything consistent, often too much, and that wasn't clear. What I meant to say was that if it turns out that our guidelines are becoming a problem, we should drop them, or change how we regard them (e.g. as guidelines for those who want to go the extra mile, not as rules that will frustrate contributors).

Your work on the Django code base has been tireless and fantastic, and along with the other fellows I don't know what we'd do without you, sorry for not making that clear with my comments.

Thanks,

Luke

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

Claude Paroz

unread,
Apr 19, 2019, 1:25:06 PM4/19/19
to Django developers (Contributions to Django itself)
I must admit I'm rather pleased by Luke's proposal. Let's keep our current isort/flake8 checks and relax other policies (considering the committer can always minor edit patches, as Tim did in the past).

I'm always a bit suspicious of tools that decide the formatting for me.

Claude

Mariusz Felisiak

unread,
Apr 19, 2019, 2:33:13 PM4/19/19
to Django developers (Contributions to Django itself)
I'm against adopting black to Django. I don't like style proposed by black (some of rules make code less readable) and in the same time I really enjoy Django's style, but that's of course my subjective opinion.

I don't think that our code style is any barrier for newcomers. Just like Tim I make a minor stylistic edits before merging patches (if necessary) and it is not really a big issue for me. I double-check all lines of patches so minor edits don't increase the time that I spend on PR. I want to clarify that adopting black will not immediately solve any tickets because we've never blocked any patch due to stylistic nitpicks. I also don't believe that it will increase the number of contributors, if I would like to contribute to any package it wouldn't matter to me.

IMO we should stay with isort, flake8 and hanging indentation (we don't have many extra rules). Of course I'm open for discussion about relaxing our code policy.

Best,
Mariusz

Jacob Rief

unread,
Apr 19, 2019, 2:56:31 PM4/19/19
to Django developers (Contributions to Django itself)
I share the opinion of Mariusz Felisiak, Luke Plant and Claude Paroz, and believe that it is a bad idea to do this in an automatic way without the possibility to interfere manually.
- Jacob

Carlton Gibson

unread,
Apr 20, 2019, 3:11:24 AM4/20/19
to Django developers (Contributions to Django itself)


On Friday, 19 April 2019 20:33:13 UTC+2, Mariusz Felisiak wrote:

I don't think that our code style is any barrier for newcomers. ...we've never blocked any patch due to stylistic nitpicks. I also don't believe that it will increase the number of contributors, if I would like to contribute to any package it wouldn't matter to me.

So, I've been out banging the drum for new contributors for a year or so now. I've had a number of personal reports that this just isn't true. 

Members of our community who want to contribute to Django are finding the review process off-putting. (They've used stronger language than that.) We have a host of people that have made a single contribution and never come back. 

"Wrap to 79 chars", or whatever, would be better gone. I'm not sure I like Black per se, but using an auto-formatter would enable review comments to focus on substantive points. ("Can we rephrase this slightly to more like....")  (From experience in Go and Elm lands, the community using a single formatter has great benefits. After a couple of week your brain gets used to the different style.) 

It's not a panacea. But we have a massive barrier to entry. We need to address that. Using an auto-formatter would be one step. 

Claude Paroz

unread,
Apr 20, 2019, 4:02:44 AM4/20/19
to Django developers (Contributions to Django itself)
Sorry, but I cannot believe that the coding style is a hurdle in our review process. Most Python devs should already write mostly PEP-8 compliant code. Let me repeat what I already said in other threads: contributing to Django today is hard because:
1. The low-hanging fruits are gone and most tickets are now non-trivial to solve.
2. Our quality expectations are high (tests, docs, backwards compatibility) and I don't think we should sacrifice any of those requirements just to get more contributors.

I think that we are at a point where asking a single person to handle a ticket from writing code down to the final commit is becoming a too heavy task (at least for newcomers). In my opinion, we should find a way to better split the various tasks involved in solving a ticket among several contributors.

This should be the subject of another thread probably. It is somewhat related to this thread in the sense that if the syntax polishing is simply ignored at first and deported to a pre-commit step (I'm sure we'll find people willing to do that part), then this issue is moot. We could imagine for example that isort/flake8 patch checks are only run at final stages of a patch process.

Claude

Jamesie Pic

unread,
Apr 22, 2019, 2:44:16 AM4/22/19
to django-d...@googlegroups.com
Just note that the coala python package ( https://coala.io ) seems to be the alternative to black that will let you make some compromises, might be worth considering as well.

Nick Sarbicki

unread,
Apr 22, 2019, 4:57:50 AM4/22/19
to Django developers (Contributions to Django itself)
I'm still quite strongly in support of black. I find it's very nice to read a codebase that has strong consistency (which is historically usually the work of pedantic but very necessary reviews).

It's even nicer to go between multiple codebases which are all consistent with each other. PEP8 and linters like flake8 go quite a long way in achieving this but black goes even further.

There will always be arguments about what standards to use, it's an unsolvable problem. Some people may think the current Django standards are perfect, some will prefer blacks defaults, even more will have their own preferences. I feel that it is arbitrary to make this a major point, it will never be agreed upon.

Consistency in the end is the most important thing (even PEP8 agrees there). If we can contribute towards a global consistency in the python community that would be wonderful. If black is the way the community is going for this, then that is where my vote is. Even moreso if it lowers the barrier to entry (I don't think anyone is saying it raises it?).


On Mon, 22 Apr 2019, 07:44 Jamesie Pic, <jp...@yourlabs.org> wrote:
Just note that the coala python package ( https://coala.io ) seems to be the alternative to black that will let you make some compromises, might be worth considering as well.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

Jamesie Pic

unread,
Apr 22, 2019, 6:01:55 AM4/22/19
to django-d...@googlegroups.com
Thanks for sharing some of your insight.

On Mon, Apr 22, 2019 at 10:57 AM Nick Sarbicki <nick.a....@gmail.com> wrote:
 Even moreso if it lowers the barrier to entry (I don't think anyone is saying it raises it?).

I'm just saying that if "As contributor, I can haz automatic code formatter to lower the barrier" is precisely the story you want to solve, then black may not be the only solution you want to consider deeply ;)

Have a great day ;)
 
--

Melvyn Sopacua

unread,
Apr 22, 2019, 7:58:54 AM4/22/19
to django-d...@googlegroups.com
On maandag 22 april 2019 10:57:19 CEST Nick Sarbicki wrote:

> Consistency in the end is the most important thing (even PEP8 agrees
> there).

Not sure where you got that impression:
https://pep8.org/#a-foolish-consistency-is-the-hobgoblin-of-little-minds

Pep8 clearly states consistency is less important then readability (it's the
first thing mentioned and mentioned repeatedly that you can use as an argument
to break consistency). And this is the primary advantage of black, since
readability is hard to quantify (and therefore lint or format) and I think
this is where black has succeeded (by breaking consistency where it is
needed).
I mostly follow the discussion with interest from the sidelines, but just
wanted to correct this consistency argument: if you want consistent code, go
with autopep8, it'll keep your lines consistent below 79 characters and quite
an unreadable mess.
--
Melvyn Sopacua


Nick Sarbicki

unread,
Apr 22, 2019, 10:14:44 AM4/22/19
to Django developers (Contributions to Django itself)

I'm just saying that if "As contributor, I can haz automatic code formatter to lower the barrier" is precisely the story you want to solve, then black may not be the only solution you want to consider deeply ;)


Jamie, sure, I wasn't responding directly to you about this, more to the general people arguing against blacks style choices. I would happily consider alternatives to black - although (without any formal research to back this claim) it does feel like black has the most community support.

My point is mostly that if there is a growing community consistency through black then I'd be hesitant to choose another tool that goes against this.
Thanks for the correction Melvyn, you're right - aside from readability and backwards compatibility consistency is important.

I'd also note the irony of using PEP8 to argue for consistency with a tool that is (at least on line length) inconsistent with PEP8. Although I really don't want to start a debate on line length.

thinkwel...@gmail.com

unread,
Apr 22, 2019, 2:10:41 PM4/22/19
to Django developers (Contributions to Django itself)
I wonder if there's a middle ground between minimizing code churn and having a standardized formatter. Our team recently switched to yapf after carefully configuring a .style.yapf file that's included in the root directory of every new repo. Once that config file is done, the workflow for yapf vs an unconfigurable formatter is identical.


I experimented a bit with the following .style.yapf settings on the django codebase and think the output is as good as black's without some of the arbitrariness of string quotes:

[style]
based_on_style
= pep8
column_limit
=100
i18n_function_call
=['_']
blank_line_before_nested_class_or_def
=True
join_multiple_lines
=False
indent_dictionary_value
=False

coalesce_brackets
=True
dedent_closing_brackets
=True
align_closing_bracket_with_visual_indent
=False
space_between_ending_comma_and_closing_bracket
=False

split_complex_comprehension
=True
split_before_first_argument
=True
split_before_logical_operator
=False
split_before_bitwise_operator
=False
split_arguments_when_comma_terminated
=True
split_before_expression_after_opening_paren
=True
split_before_named_assigns
=True


Yapf currently has more stars than black, but whether black has more momentum or not, who can say.

Sam Cooke

unread,
Apr 23, 2019, 5:43:31 AM4/23/19
to django-d...@googlegroups.com
On projects I've been working on I started using "prettier" fro JavaScript and it made a huge difference to code consistency across the team, which was great, but my favourite benefit (that I'm not sure I've seen mentioned much here) is I can write code faster if I don't have to format it. Being able to just type without worrying about it, then hit save and it jumps into place, means if I'm working on files without automatic formatters I now feel like I'm doing a machine's job tidying things up.

Having said that - whilst I use black on all of my python projects, it's not had anywhere near as much of a positive impact as prettier has on JavaScript because:
 - python is generally more readable and consistent naturally than JavaScript
 - significant whitespace in python means you still spend time manually formatting code so you save less time than with JavaScript
 - I've found black to be unreliable - it silently fails on quite a few of my files
 - prettier is smarter about allowing you to add whitespace to make things like lists more readable, but black is 100% fixed in its ways

The best thing about automatic formatters, in my opinion, is even if you don't like the style at least you don't have to talk about it any more! And you tend to get used to it eventually.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

Carlton Gibson

unread,
Apr 24, 2019, 2:35:57 AM4/24/19
to Django developers (Contributions to Django itself)
Thanks for the YAPF suggestion (and sample config!) I'll have a go with this this week. 
(If we can get auto-formatting, just on diffs(?), that matches the existing style...)

Josh Smeaton

unread,
Apr 24, 2019, 2:58:57 AM4/24/19
to Django developers (Contributions to Django itself)
FWIW, I'd be opposed to YAPF at all, but especially over Black. The community (that is, large projects that aren't Django) are standardising around Black - a single unambiguous format style. Having a slightly different format to everybody else I see as backwards. Contributors would also likely need to maintain two installations - one for Django, and one for (nearly) everything else. I'm probably being a little dramatic here with regards to the reach Black currently has, but there are many projects using it successfully.

There are also the problems with YAPF as described by Łukasz (black author) himself: https://news.ycombinator.com/item?id=17155205

Copied from link above:

- YAPF would at times not produce deterministic formatting (formatting the same file the second time with no changes in between would create a different formatting); Black treats this as a bug;

- YAPF would not format all files that use the latest Python 3.6 features (we have a lot of f-strings, there's cases of async generators, complex unpacking in collections and function calls, and so on); Black solves that;

- YAPF is based on a sophisticated algorithm that unwinds the line and applies "penalty points" for things that the user configured they don't like to see. With a bit of dynamic programming magic it arrives at a formatting with the minimal penalty value. This works fine most of the time. When it doesn't, and surprised people ask you to explain, you don't really know why. You might be able to suggest changing the penalty point value of a particular decision from, say, 47 to 48. It might help with this particular situation... but break five others in different places of the codebase.


I'm unsure how relevant all of these points still are. For me, I was never able to get YAPF to play nice, or get widespread agreement on a set of rules that all devs were OK with (lots of bikeshedding).

Carlton Gibson

unread,
Apr 24, 2019, 3:06:44 AM4/24/19
to Django developers (Contributions to Django itself)
On Wednesday, 24 April 2019 08:58:57 UTC+2, Josh Smeaton wrote:
lots of bikeshedding

Yeah. 🙂 

But we've already got a style guide, so **IF** we can get a YAPF config to work to that then hopefully the arguments against using a formatter here would be moot. (Note that **IF** — I only said I'd have a play.) 

Equally, **IF** it's true that the Python community is standardising around black then, we should join in. But I get the impression that's not a done deal yet. 

I'm not fussed on the code style. The brain/eye adapts. As I said before, I would like us to have an auto formatter of some kind. 

Matthias Kestenholz

unread,
Apr 24, 2019, 3:35:54 AM4/24/19
to django-d...@googlegroups.com
It's maybe not fair to count all channels-related repositories individually, but when I take a look at the code repositories in the Django organization on Github I see either repositories using only flake8/isort or repositories using black. One could say that the Django organization already is moving towards standardizing on black...

The arguments against a particular code style will never stop. In this case it's a big advantage to choose a tool developed by others which does not allow any bikeshedding.

I can see why one would want to wait on others to choose a standard tool for the Python community. On the other hand, Django could easily help establish the "obvious way to do it" given its visibility.

Rene Fleschenberg

unread,
Apr 24, 2019, 6:22:23 AM4/24/19
to django-d...@googlegroups.com
Hi

> The arguments against a particular code style will never stop. In this
> case it's a big advantage to choose a tool developed by others which
> does not allow any bikeshedding.
>
Will black really stop the arguments (which are not bikeshedding, IMO --
readability counts)? Or will they just be replaced by arguments over
when to use ``# fmt: off``?

--
Rene

Josh Smeaton

unread,
Apr 24, 2019, 7:25:55 AM4/24/19
to django-d...@googlegroups.com
Black does not support disabling formatting by block with a comment. It removes all choice except for the upfront choices of length and string normalisation. 

--
You received this message because you are subscribed to a topic in the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-developers/wK2PzdGNOpQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-develop...@googlegroups.com.

To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

Florian Apolloner

unread,
Apr 24, 2019, 8:51:38 AM4/24/19
to Django developers (Contributions to Django itself)

On Wednesday, April 24, 2019 at 1:25:55 PM UTC+2, Josh Smeaton wrote:
Black does not support disabling formatting by block with a comment. It removes all choice except for the upfront choices of length and string normalisation. 

It does, "# fmt: off" and "# fmt: on" can be used

Josh Smeaton

unread,
Apr 24, 2019, 5:25:39 PM4/24/19
to django-d...@googlegroups.com
Whoops, you are correct. 

It did not at the time I added black to my projects, my information was quite out of date. Sorry for the wrong correction! 

--
You received this message because you are subscribed to a topic in the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-developers/wK2PzdGNOpQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

Herman S

unread,
Apr 25, 2019, 4:24:56 AM4/25/19
to django-d...@googlegroups.com
On Fri, Apr 19, 2019 at 6:33 PM Mariusz Felisiak
<felisiak...@gmail.com> wrote:
> I don't think that our code style is any barrier for newcomers.

As a newcomer, I can tell you that it most definitely is a barrier, at
least it is for me – for the reasons laid out in my original email :)

*

From what I can gather the following arguments have been raised against Black:

- I don't like the formatting style (double quotes, white-space, line
length, wrapping, ...)
- It produces one huge diff
- It, in some situations, produces weird formatting (lists with comments, ...)
- It doesn't solve an issue that we currently have

At the end of the day some of these arguments are just personal
opinions and comes down to preference. However a lot of
counter-arguments have been made and I'll just shortly summarize them
here:

* I don't like the formatting style
We will never collectively agree on "the perfect" formatting and
many do not agree with the *current* formatting. The time spent having
these non value-adding discussions are not worth it, and Black would
eliminate them to
to a large extent.

* It produces one huge diff
There are tools to work around this (git-hyper-blame [0]), Github
has a great UI for browsing diffs and it looks like in the future it
will be possible to exclude certain commits from blame history [1].

* It, in some situations, produces weird formatting
A lot of these can be remedied my using `--target-version=py37` as
the AST does not change in newer versions of Python. Other files (like
settings) can be either disabled from auto-formatting or we can
disable parts of the file with `# fmt: off`.

* It doesn't solve an issue that we currently have
I, and a lot of people have expressed this too, feel like it
definitely solves a very real problem. Formatting of code delays PRs,
wastes time, makes it harder to contribute as a newcomer, is an extra
cognitive load for all of us, and produces
loads of wasteful discussions.

I'm sure there are more arguments made too, I'm sorry if I missed
some, but these seem to be repeated most often.

*

I favor adopting the default configuration (88 width, double quotes)
in one go and just be done with it. It's not as scary as it seems and
it really just makes the day a tiny bit easier :-)

What has usually been the process of democratically deciding these
"controversial" topics in the past? From what I gather there is a
clear majority favoring Black, but I'm curious as to when and how we
decide to end the discussion either for or against.

Kind regards,

[0]: https://commondatastorage.googleapis.com/chrome-infra-docs/flat/depot_tools/docs/html/git-hyper-blame.html
[1]: https://public-inbox.org/git/20190410162409...@google.com/

--
Herman Schistad

Tobias Kunze

unread,
Apr 25, 2019, 4:59:17 AM4/25/19
to django-d...@googlegroups.com
On 19-04-25 08:24:34, Herman S wrote:
>From what I gather there is a clear majority favoring Black, […]

Please don't resort to influencing the discussion by way of presenting a
majority opinion like that. People on django-dev are generally good at
not repeating points that have been made already, or cluttering an
ongoing discussion with a lot of "+1", so there is no way to know who is
nodding along with which arguments.

Tobias
signature.asc

Tom Forbes

unread,
Apr 25, 2019, 6:11:21 AM4/25/19
to django-d...@googlegroups.com
so there is no way to know who is nodding along with which arguments.

That somewhat impedes the point of posting this to get consensus, and if you ascribe to that view then all mailing list discussions about project wide changes are somewhat useless, as there could be a silent majority completely against the idea being proposed?

There is a majority of people who have expressed opinions in this thread in favor of black, so he’s not necessarily trying to influence the discussion as much as pointing out a fact.

That’s not to say consensus has been reached though, judging by the number of replies here there are a lot of strong views either way.

Tom

Tobias

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

James Bennett

unread,
Apr 25, 2019, 6:15:10 AM4/25/19
to django-d...@googlegroups.com
I like Django's style guide.

I like the *idea* of an autoformatter.

I dislike the particular mostly-unconfigurable style Black enforces, and I find that several of its rules negatively impact code readability.

So I would be -1 on applying it to Django.

Josh Smeaton

unread,
Apr 25, 2019, 7:30:30 AM4/25/19
to django-d...@googlegroups.com
To answer the question about decision making...

Usually a decision is made if there’s reasonable consensus in the discussion or after a vote. If no clear consensus can be reached with the core team, then a decision can be escalated to the technical board for a vote. 

In the future, if the core team is dissolved in its primary capacity, consensus in the discussion rather than consensus among the core team would be the first step. 

I figure all reasonable arguments have probably been made at this stage. But there are really two questions we should be proposing. 

1. Should Django adopt an autoformatter?
2. If it should, should that autoformatter be black or something else? 

Considering the rather strong views in this thread, I’d probably recommend that a DEP is drafted for each of the above, where opinions and comments can be gathered more formally. 

--
You received this message because you are subscribed to a topic in the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-developers/wK2PzdGNOpQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-develop...@googlegroups.com.

To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

Curtis Maloney

unread,
Apr 25, 2019, 8:59:40 AM4/25/19
to django-d...@googlegroups.com
I think this very clearly sums up what I was going to say on this topic,
if I hadn't already.

I see the benefits [lower barrier to entry, time saving for the Fellows,
etc], but I don't believe Black is the answer.

Specifically because in my own experience with it (as well as
demonstrated in this list) there are cases where its style makes code
significantly harder to read.

And IMHO, the sole purpose of a consistent style (and thus a style
guide) is to reduce the cognitive load of humans parsing code.

--
Curtis

Aymeric Augustin

unread,
Apr 27, 2019, 3:07:00 AM4/27/19
to django-d...@googlegroups.com
I'm writing a DEP.

-- 
Aymeric.



You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

Max Arnold

unread,
Apr 27, 2019, 6:05:25 AM4/27/19
to Django developers (Contributions to Django itself)
Once the DEP is ready, maybe it would make sense to discuss the above-mentioned concerns with Łukasz Langa? Black is quite opinionated, but Django is a well-known project in the Python ecosystem, and its weight could lead to some adjustments (or at least future stability guarantees) in black?

Aymeric Augustin

unread,
Apr 28, 2019, 9:20:42 AM4/28/19
to django-d...@googlegroups.com
On 14 Apr 2019, at 15:10, Josh Smeaton <josh.s...@gmail.com> wrote:

Finally, there are some tricks you pick up if black fights you on some decisions. To use Berkers example:

TIME_INPUT_FORMATS = [ 
    '%H:%M:%S', # '14:30:59' 
    '%H:%M:%S.%f', # '14:30:59.000200' 
    '%H:%M', # '14:30' 


TIME_INPUT_FORMATS = ['%H:%M:%S', '%H:%M:%S.%f', '%H:%M'] # '14:30:59' 
# '14:30:59.000200' # '14:30' 

Break each individual format into its own variable, with appropriate comment, and add the variables to the list.

HMS = "%H:%M:%S"  # 14:30:59
HMSF = ".." 
HM = ".."
TIME_INPUT_FORMATS  = [HMS, HMSF, HM]

Obviously just an example, but something to keep in mind.

FWIW there's also this possibility:

TIME_INPUT_FORMATS = [ 
    # '14:30:59' 
    '%H:%M:%S',
    # '14:30:59.000200'
    '%H:%M:%S.%f',
    # '14:30'
    '%H:%M',
]

-- 
Aymeric.

Aymeric Augustin

unread,
Apr 28, 2019, 10:51:21 AM4/28/19
to django-d...@googlegroups.com
Hello,

Here's my attempt at summarizing the conversation in a DEP: https://github.com/django/deps/pull/55.


Please let me know if I missed or represented unfairly some ideas!

Best regards,

-- 
Aymeric.

PS: while I'm eager to listen to feedback and iterate on this draft, I would also prefer if this DEP didn't turn into a novel, so let's avoid going into every detail and trust the implementation team to do a good job — thank you :-)


On 13 Apr 2019, at 13:52, Herman S <herman....@gmail.com> wrote:

Hi.

I propose that Django starts using 'black' [0] to auto-format all Python code.
For those unfamiliar with 'black' I recommend reading the the projects README.
The short version: it aims to reduce bike-shedding and non value-adding
discussions; saving time reviewing code; and making the barrier to entry lower
by taking some uncompromissing choices with regards to formatting.  This is
similar to tools such as 'gofmt' for Go and 'prettier' for Javascript.

Personally I first got involved contributing to Django couple of weeks back,
and from anecdotal experience I can testify to how 'formatting of code' creates
a huge barrier for entry. My PR at the time went multiple times back and forth
tweaking formatting. Before this, I had to research the style used by exploring
the docs at length and reading at least 10-20 different source – and even those
were not always consistent. At the end of the day I felt like almost 50% of the
time I used on the patch was not used on actually solving the issue at hand.
Thinking about code formatting in 2019 is a mental energy better used for other
things, and it feels unnecessary that core developers on Django spend their time
"nit-picking" on these things.

I recently led the efforts to make this change where I work. We have a 200K+
LOC Django code-base with more than 30K commits. Some key take-aways: it has
drastically changed the way we work with code across teams, new engineers are
easier on-boarded, PR are more focused on architectural choices and "naming
things", existing PRs before migration had surprisingly few conflicts and were
easy to fix, hot code paths are already "blameable" and it's easy to blame a
line of code and go past the "black-commit", and lastly the migration went
without any issues or down-time.

I had some really fruitful discussions at DjangoCon Europe this week on this
very topic, and it seems we are not alone in these experiences. I would love to
hear from all of you and hope that we can land on something that will enable
*more* people to easier contribute back to this project.

I've set up how this _could_ look depending on some configurables in Black:

* Default config: https://github.com/hermansc/django/pull/1
* Line length kept at 119: https://github.com/hermansc/django/pull/3
* Line length kept at 119, no string normalization:
https://github.com/hermansc/django/pull/2

Please have a look at the Black documentation. It explains the benefits better
than I possibly could do here.

With kind regards,
Herman Schistad

[0]: https://github.com/ambv/black


--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

Markus Holtermann

unread,
Apr 28, 2019, 11:44:38 AM4/28/19
to Django developers

Alexander Hill

unread,
Apr 28, 2019, 11:11:38 PM4/28/19
to Django developers (Contributions to Django itself)
Hi all,

Could we do the work to make black (or another suitable formatter) apply only to a range of lines? Prettier has this feature[0], which is used by precise-commits[1] in the way we'd want to use it for Django. It takes a range of lines, expands the range to statement boundaries, and formats only those lines. This feels like a solution nearly everyone could agree on and would be an asset to the Django and broader Python community.

From my perspective, the DEP as it stands undervalues blame and Django's commit history. In PyCharm, the date, message and entire diff of the commit responsible for every line is immediately available, in the margin of the editor or with a keyboard shortcut. When it's a trivial to get to, this context is incredibly useful. Plugins for vim and other tools provide similar functionality. GitHub is pretty good in this regard too, but it really shines when integrated into your editor. Like auto-formatting, I think you don't know how good this is until it's part of your workflow.

To address the counter-arguments: the "view blame prior to this change" button in GitHub shows you the blame for the entire file before the specified commit, as opposed to the blame for the file in its current state, disregarding the specified commit. Not bad, but not quite what you want. git-hyper-blame has been raised several times, but it's non-standard, not straightforward to install, and not supported by tooling. A git patch was mentioned that incorporates hyper-blame's functionality into blame[2], which looks like it'll be great, but won't help until it's merged, released, and integrated into tooling.

Django is a mature project, which means that many, many lines have remained unchanged for years, and will continue to be attributed to the black mega-commit for many years to come.

I'm firmly -1 to globally applying black to the codebase until it can be done without breaking blame.

Alex


Aymeric Augustin

unread,
Apr 29, 2019, 2:30:56 AM4/29/19
to django-d...@googlegroups.com
Hello Alex,

Could we do the work to make black (or another suitable formatter) apply only to a range of lines? Prettier has this feature[0], which is used by precise-commits[1] in the way we'd want to use it for Django. It takes a range of lines, expands the range to statement boundaries, and formats only those lines. This feels like a solution nearly everyone could agree on and would be an asset to the Django and broader Python community.

Yes, that would alleviate about half of the concerns — the other half being opposition to some style choices made by Black.

The answer to "could we do the work" is pretty much in your hands :-) If you want this to happen, put together a PoC and write DEP 9.

If I see good momentum on a credible alternative, I'm willing to put DEP 8 on standby for a few weeks.

From my perspective, the DEP as it stands undervalues blame and Django's commit history. In PyCharm, the date, message and entire diff of the commit responsible for every line is immediately available, in the margin of the editor or with a keyboard shortcut. When it's a trivial to get to, this context is incredibly useful.

This is a good point.

It's valid insofar as the last commit contains useful information by itself, which requires some luck in my experience with Django.

I triaged a few thousand tickets between 2011 and 2014. Back then many git blame expeditions required jumping through several commits. If I needed git blame to understand the code, often it meant that two different people touched the same piece of code and left an inconsistency, so the last commit by itself wasn't going to give me the answer. I don't think this changed since then.

That said, I don't have the kind of setup you describe, and if I had, I would certainly lean on git blame more heavily.

To address the counter-arguments: the "view blame prior to this change" button in GitHub shows you the blame for the entire file before the specified commit, as opposed to the blame for the file in its current state, disregarding the specified commit. Not bad, but not quite what you want. git-hyper-blame has been raised several times, but it's non-standard, not straightforward to install, and not supported by tooling. A git patch was mentioned that incorporates hyper-blame's functionality into blame[2], which looks like it'll be great, but won't help until it's merged, released, and integrated into tooling.

Yes, let's be honest: DEP 8 doesn't have a much better argument than "oh well we'll survive with the tools we have".

I'm firmly -1 to globally applying black to the codebase until it can be done without breaking blame.

By framing the problem in that way, you imply that git blame currently works and that it will stop working if Black is applied to the codebase. Both are false :-) Rather, we'd be trading one problem for another :

- The current problem is random small reformattings, which I referred to as "a steady stream of pollution" in DEP 8 and which I experienced very regularly when I was an active contributor ; this is not a theoretical issue ;
- The new problem if DEP 8 goes live is the Big Commit ;

I find it easier to determine that a commit isn't significant if the message says "Reformat Django with Black" so the new problem could be easier to deal with than the current problem.

Best regards,

-- 
Aymeric.



Carlton Gibson

unread,
Apr 29, 2019, 3:12:00 AM4/29/19
to Django developers (Contributions to Django itself)
Hi Alex. 

So I use git blame on Django a lot: a new ticket comes in, I have no idea at all how to triage it so I need to look at the history to make a sensible decision. I use git blame to find the commit, to find the original ticket, in which there's a discussion, which 🤞 explains why things are as they are. (Mostly...)

But we're already at the the point where more times than not I need to skip multiple commits to get back to the original decision. 

(Screenshot attached) 

The last three commits from this morning are all "tidy-ups" that obscure the history. This is typical. Rarely now is there such a old block that hasn't been adjusted at all. So it's normal to have to jump commits. 

A big Black commit will add to this. (It's the concern I do have.) But it's not a game changer. It's just more of the same. I'm pretty sure it's the motivation needed to up my git blame foo to the next level, at which point I'll likely save time ironically. 

For this reason, weighed against the benefits, I don't think the git history should be a blocker. I hope that makes sense. 

Kind Regards,

Carlton



Screenshot 2019-04-29 09.02.54.png

Paveł Tyślacki

unread,
Apr 29, 2019, 3:34:48 AM4/29/19
to django-d...@googlegroups.com
Just short hint about git blame.

Github has functionality to get more deep git blame: https://help.github.com/en/articles/tracking-changes-in-a-file, more over pycharm has similar functionality too (potentially another editors and IDE have too). I suggest that git blame is more tooling issue, not changes approach, because for another refactoring and active development you can have the same issue with git blame.

Regards
Paveł

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

J. Pic

unread,
Apr 29, 2019, 4:29:05 AM4/29/19
to django-d...@googlegroups.com
If you don't want to break git blame then I suppose you could create a branch to replay each commit with black.

Laurens A. Bosscher

unread,
Apr 30, 2019, 8:42:00 AM4/30/19
to Django developers (Contributions to Django itself)
The Chrome team fixed this issue with git-hyper-blame: https://commondatastorage.googleapis.com/chrome-infra-docs/flat/depot_tools/docs/html/git-hyper-blame.html

That could be a solution that would work for Django. IDE support for git-hyper-blame is lacking (at the moment) but might improve in the future.

Op maandag 29 april 2019 09:12:00 UTC+2 schreef Carlton Gibson:

J. Pic

unread,
Apr 30, 2019, 12:18:47 PM4/30/19
to django-d...@googlegroups.com
Also note that even for people that don't use an IDE, they might like when GitHub's blame feature work, so that would also be a pro for rewriting the git history rather than creating a new commit for the code rewrite.

Christian González

unread,
Apr 30, 2019, 3:31:03 PM4/30/19
to django-d...@googlegroups.com

Am 30.04.19 um 14:28 schrieb 'Laurens A. Bosscher' via Django developers
(Contributions to Django itself):
> The Chrome team fixed this issue with
> git-hyper-blame: https://commondatastorage.googleapis.com/chrome-infra-docs/flat/depot_tools/docs/html/git-hyper-blame.html
>
>
> That could be a solution that would work for Django. IDE support for
> git-hyper-blame is lacking (at the moment) but might improve in the
> future.

I made a feature request for git itself, and they told me this is
already in the works *within* git dev:

https://public-inbox.org/git/20190410162409...@google.com/

So on the long run, no need for git-hyper-blame.

Christian

--
Dr. Christian González
https://nerdocs.at

pEpkey.asc
It is loading more messages.
0 new messages