Transition Docs to Inline

[Moshe Dicker]

Due to the dynamic nature of Python, features aren’t obvious.
Developers don’t know what features a class has unless they dig through the convoluted source code themselves or going online to check the documentation.

For example when implementing a `forms.ModelForm` hovering over it reveals absolutely nothing useful.

image1000×157 11.5 KB

What do the “Django Lords” have to say about inlining the documentation in the source code itself.

I know the “Django Lords” have been weary of adding type stubs.
However adding a docstring to this

```
class Model(models.Model):
    ...
    class Meta: <<<<< THIS
        ...

```

would require some stubs/.pyi files.

I wouldn’t want to work on this just to find out that there are benefits of the current setup that they aren’t willing to make a tradeoff.

What would be the best way of asking the Django Governing Board about their opinion on this?

[Tim Graham]

Hi Moshe, 

Why is consulting the online documentation insufficient? I think most developers build Django projects while referencing the online documentation rather than while reading Django's source code.

What sort of documentation would be inlined? Would this require a large amount of duplication between docs and source code?

Why does adding docstrings require pyi files? My text editor doesn't have this "hover" feature. Please elaborate on your proposal for the uninitiated. Are there other projects that use the approach you suggest?

[Moshe Dicker]

I will address each part of your questions.

Why is consulting the online documentation insufficient? I think most developers build Django projects while referencing the online documentation rather than while reading Django’s source code.

Having documentation available within the IDE really raises the developer experience.
For those with a Visual IDE it’s shown on hover, and even for those with a text editor they could see that docstring if they “click-into” the class/function/argument/parameter.

What sort of documentation would be inlined?

I will try to explain below how I think the separation of API Reference and Guides should be done.

Would this require a large amount of duplication between docs and source code?

God Forbid! Sphinx has autodoc which convert doctrings to reStructuredText.

Why does adding docstrings require pyi files?

It isn’t required but it would help. I imagine that with some stub magic it would be possible to get docstrings applied onto a Meta class that doesn’t actually inherit anything

class CustomUser(models.Model): class Meta: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< verbose_name = _("user")

Are there other projects that use the approach you suggest?

Pretty much everything else.
Some mix docstrings and handwritten docs into 1 really nice API reference

• FastAPI - API Reference (docs|code)

Others write a regular guide separately, but generate API reference from docstrings

• Flask - API Reference (docs|code)
• Pydantic - API Reference (docs|code)

I could go on and on, but I’ve never come across a package that does not utilize docstrings at all.

Thanks!

​

[Moshe Dicker]

Django’s API Reference is more like a guide than an actual Reference at this point.
I’m proposing as follows:

1. We take any part of the reference that is short and to the point and put it in a docstring.
   I think going the FastAPI route on this would be best, A guide and a separate really robust API Reference that sources the technical info from the docstrings. 
   Example: This, should be in the docstring of null.
   Implementation:class Foo: def __init__(self, value) -> None: """The Foo class. :param value: "Simple string value" :type value: str """ self.value = value foo = Foo(value="bar")
2. If this is rejected for whatever reason, can we please add just docstring to link to the online docs.class Foo: def __init__(self, value) -> None: """https://example.com""" self.value = value foo = Foo(value="bar")

​

I'm coming from Flutter where the documentation is pretty much only inline docs, It's really a pleasure to never need to lookup documentation.

This is a bunch of work. But having a lower barrier to entry is important. I think a docs refresh will really rock!

[Jörg Breitbart]

+1 on the idea for better inline docs, would be some relief for IDEs
figuring out a proper interface story boosting dev speed for rarely used
aspects, where currently one would have to search through the prosaic
online docs or end up browsing the django source.

On the other hand I am not sure, if that an easy task to accomplish
without bigger refactoring. Django uses a lot of bootstrap/runtime
patching under the hood, from custom metaclasses, explicit injection
pattern up to proxies at various ends. Currently IDEs often give up
here, idk if more inline docs alone can change much or if the level of
abstraction within django would have to change to get a significant
improvement.

Cheers,
Jörg

[Moshe Dicker]

Django uses a lot of bootstrap/runtime patching under the hood, from custom metaclasses, explicit injection
pattern up to proxies at various ends.

I understand this is true on low level APIs and on classes we implement ourselves (Like passing the correct arguments when creating a django model).
But almost all the documentation in ref is connected to a class, function, method etc. This can be pushed into the source code and autodoced quite easily.
Anything that can’t be in a class can stay the way it is. Correct me if I’m wrong, It is almost certain that I am.
(Adding inline docs to django-stubs would help mitigate this, but coordinating documentation between 2 different projects would be a nightmare.)

I only see 2 downsides to my proposal:

1. A big refactor to the codebase, There will be strings everywhere.
2. A larger install size.

​

[Jörg Breitbart]

Ah interesting that you mentioned django-stubs. I had good to mediocre
success applying it to my own django apps. It gets the job mostly done
for high level interfaces, but shows rough stub edges as soon as you
have to touch lower interfaces (can only speak for the ORM section in
this regard, which is highly mutating itself).

Type annotations in Python are heavily inspired by Typescript, where
you'd strive for building up you interface types granularily and the
compiler does the rest pulling things together.
I am not sure if Python's type annotations are yet or will ever be
capable to achieve that good enough for most common python paradigm,
seems Python is way too flexible for static type analysis. Typescript
partially solves this by banning/hiding some of JS' flexibility. (Ever
tried to fully type a mutating decorator in Python? Hours of fun ahead,
and you will still end up declaring many Anys effectively disabling
proper type pulling.)

Now back to Django - it makes heavy usage of dynamic python features,
which is poison for any static type deduction. This is also the reason
why IDEs typically fail for more dynamic python code like django, unless
they do a partial runtime inspection (I think IntelliJ tries to do that
to some degree).

So while better typing support is really nice where possible, I'd
suggest to not open this can of worms (yet), but mainly to stick to the
better inline docs idea.


Cheers,
Jörg

[Tim Graham]

I don't think moving docs inline is a good idea. Quoting Aymeric from 2013 regarding django.contrib.admindocs [1] summarizes my feelings:

"""

1) It's called the "documentation generator", but it only operates on docstrings. This promotes the idea that docstrings are appropriate documentation, while the Python and Django communities now favor prose documentation.

2) Even though it's possible to use docstrings to generate API documentation, for instance with Sphinx' autodoc, I find that heavily formatted, Javadoc-style docstrings (or late epydoc-style) tend to be hard to read for humans and I don't want Django to encourage them.

"""

[1] https://groups.google.com/g/django-developers/c/0-qFyCPuSRs/m/kny7Oeu2RdoJ

[dicke...@gmail.com]

I'll argue that right now we don't have documentation. We just have a mix of docs and reference, resulting in a convoluted manual that doesn't fit either need.

Django isn't some fly-by-night framework whose documentation will devolve if we move the technical reference and documentation into separate parts.

Looking at other packages for inspiration, fastapi and pydantic do great work of describing the technical parts and have a manual in a separate section.

Personally I would suggest moving the entire docs to mkdocs, with its better navigation. Although I understand that older frameworks are hesitant to change.

I'll argue that right now we don't have documentation. We just have a mix of docs and reference, resulting in a convoluted manual that doesn't fit either need.

Django isn't some fly-by-night framework whose documentation will devolve if we move the technical reference and documentation into separate parts.

Looking at other packages for inspiration, fastapi and pydantic do great work of describing the technical parts and have a manual in a separate section.

Personally I would suggest moving the entire docs to mkdocs, with its better navigation. Although I understand that older frameworks are hesitant to change.

--------------------------------

From: Tim Graham

Date: Wed, Jan 10, 2024 5:41 PM

To: Django developers (Contributions to Django itself);

Cc:

Subject:Re: Transition Docs to Inline

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/011ce3a2-3936-4e90-b205-5d12b229d536n%40googlegroups.com.

[Mariusz Felisiak]

Agreed with Tim.

> I'll argue that right now we don't have documentation. We just have a mix of docs and reference, resulting in a convoluted manual that doesn't fit either need.

This is a really unfair opinion (not the only one in your comment). Hundreds of folks have put a lot of effort into making Django documentation as great as it is today. It's one of our biggest advantages.

Best,
Mariusz

[Moshe Dicker]

I should not that I'm only being critical to improve.

The docs are well written, it's just disorganized.

I mean no disrespect, we stand on the shoulders of giant, but that doesn't mean we should stop trying to improve. 

[Ken Whitesell]

On 1/11/2024 12:01 AM, Moshe Dicker wrote:
> I should not that I'm only being critical to improve.
> The docs are well written, it's just disorganized.

Something I've learned over the past 47 years of reading computer
documentation is that _no_ documentation is "perfect" - especially not
for everyone.

All documentation is a compromise.

Different people have different ways of reading and understanding
written material. There is no "one right way" to write or organize a set
of manuals. The best you can hope for is to create docs that are
complete, materially correct, and internally consistent.

I have also learned through the years that it's _me_ that needs to adapt
myself to the different styles of documentation available. Yes, I have
preferences for certain styles. But, I also recognize that not everyone
reads and processes information the same way I do. I have no doubt that
_you_ would find it an improvement to change the docs as you describe.
However, I would suggest that you not try to sell the idea as being
objectively true for everyone and that it's "obviously better".

Me? Personally, I have an extreme dislike of docs embedded in code. When
I want to read the code, I want to see the code - not a full set of
reference information that I have to wade through - and ignore - while
I'm trying to find the information I want to see.

I want the code to be free of any clutter that can be better stored
elsewhere.

Likewise, I greatly prefer docs that go well beyond just being an "API
reference". I find sites for libraries showing nothing more than
Javadoc-style pages to be virtually useless to me when I'm first getting
started.

I have long maintained that I am more than twice as productive using
Python than I am in any other language - and part of that is the quality
of the Python docs. I always seem to be able to easily find what I'm
looking for.

In my opinion, the Django docs are close - but not quite as good - as
the Python docs. There are a couple of specific topics where I do find
myself thinking "Where is that documented?" and have to search for it.
It's not that I think they're in the "wrong" place, but it's probably
not where _I_ would have put that information. But that's ok, I probably
should just bookmark those pages...

So no, I'd be a strong -1 to any recommendation suggesting that
docstrings be used to clutter up the code.

[Jörg Breitbart]

I agree that the official Python docs are well maintained and achieve a
tough goal - the right balance of just replaying basic interface facts
on one side, or being overly prosaic on the other side. Nope, they are
well balanced giving enough of both extremes to get you going, place
repo code lookup links, and examples where needed.

And Python properly fills the __doc__ attribute on almost every function
or class, which is a big relief in interactive testing sessions to
quickly grasp a rarely used detail not in your muscle memory.

With Django thats mostly not possible, inspecting __doc__ or using
help(XY) does often not reveal any useful information.

> So no, I'd be a strong -1 to any recommendation suggesting that
> docstrings be used to clutter up the code.

I dont see how informative docstrings can clutter code, they are
visually well separated from code bodies, and most editors ven allow to
fold them.

The bigger issue I see with docstrings is, that they sometimes end up
being written too machine firendly, e.g. full of restructuredText
formatting, which hinders human reading to some degree, esp. if you
never used those formats yourself before.

[Thibaud Colas]

Hi Moshe,

Just so you know there are a lot more discussions about Django these days on the Django Forum than this mailing list. There are also no such things as Django lords or a governing board :) The project is consensus-driven, so doesn’t require any specific seal of approval.

As a Django beginner I’d really like better support for IDEs with those "hover" features you mentioned – and I don’t see why we’d have to move the docs inline to do that. Why not add docstrings where they’re missing, in addition to the docs as they stand currently? Django is a big enough project that there’s room to provide docs in different ways for different needs, and it doesn’t change often enough that keeping docs in sync would be an issue. According to the Django Developers Survey in 2022, 80% of Django devs use an IDE that offers this "inline docs on hover" developer experience nicety.

Django does have plenty of docstrings like this already, so perhaps we can add concise ones where there’s none currently in the public API, rather than having a big documentation refactoring project. Something like this – would be a quick win to add a one-liner for ModelForm:

Cheers,

Thibaud