Experimental APIs DEP

[Andrew Godwin]

Hi everyone,

One of the results of discussions at Django Under The Hood was support for the idea of marking APIs "experimental" - that is, document them and include them in mainline Django releases, but away from the standard Django deprecation cycle while still not hiding them under the "Internal" moniker (these would be APIs for public consumption).

Of particular interest here is the issues we faced with some of the migration APIs during the 1.7 release; 1.7 might have been able to release faster if we could have released early with some parts of migrations and the schema editor marked as "experimental" and iterated on it during the main release cycle, and communicated with third-party database backends more effectively and given them documentation and release notes.

There's an extra consideration here if we should extend the proposed part of the release notes listing experimental changes to also include internal changes, and possibly a separate discussion to be had about marking some of the Django database API as stable (or experimental for a release or two, then upgrade them).

The pull request for the DEP is here: https://github.com/django/deps/pull/10

(If you're not familiar with the updated DEP process, you should read the new DEP 1 at https://github.com/django/deps/blob/master/final/0001-dep-process.rst - but, quickly, the pull request is for both corrections to the layout and spelling and content suggestions, though this thread is the best place for long-form discussion).

Discussion welcome! This is by no means a final proposal, just a first draft.

Andrew

[Carl Meyer]

Hi Andrew,

Thanks for putting together this proposal.

On 12/05/2014 09:32 PM, Andrew Godwin wrote:
> One of the results of discussions at Django Under The Hood was support
> for the idea of marking APIs "experimental" - that is, document them and
> include them in mainline Django releases, but away from the standard
> Django deprecation cycle while still not hiding them under the
> "Internal" moniker (these would be APIs for public consumption).

I must have missed this discussion at DutH (or my memory is just poor).
I recall discussion of better documenting changes to the database
backends API for the benefit of third-party database backends, but I
don't recall discussion of a new "experimental" designation.

> Of particular interest here is the issues we faced with some of the
> migration APIs during the 1.7 release; 1.7 might have been able to
> release faster if we could have released early with some parts of
> migrations and the schema editor marked as "experimental" and iterated
> on it during the main release cycle, and communicated with third-party
> database backends more effectively and given them documentation and
> release notes.
>
> There's an extra consideration here if we should extend the proposed
> part of the release notes listing experimental changes to also include
> internal changes, and possibly a separate discussion to be had about
> marking some of the Django database API as stable (or experimental for a
> release or two, then upgrade them).
>
> The pull request for the DEP is here: https://github.com/django/deps/pull/10

As the DEP notes, our backwards-compatibility policy already includes a
longstanding carve-out for anything documented within the "internals"
section of the docs. We haven't made much use of this for documenting
actual internal APIs (most of that section of the docs is about
contribution process), but we could. What does an "experimental"
designation gain us that the "internals" section doesn't?

(I am hesitant about the term "experimental" -- it implies both that we
don't trust the code in question, and that the code is likely to
"graduate" to fully-backwards-compatible status in the near future. I'm
not sure that those two things will be true in every case of an internal
API that we may want to document.)

Carl

[Aymeric Augustin]

On 6 déc. 2014, at 10:05, Carl Meyer <ca...@oddbird.net> wrote:

> As the DEP notes, our backwards-compatibility policy already includes a
> longstanding carve-out for anything documented within the "internals"
> section of the docs. We haven't made much use of this for documenting
> actual internal APIs (most of that section of the docs is about
> contribution process), but we could.

Like Carl, I believe “internal” is an established concept that fits our needs
better than “experimental”.

To me “experimental” sounds much like an ill-defined gray zone between
private and public.

--
Aymeric.

[Andrew Godwin]

My notes from the meeting say "experimental API language", so I may have taken an adjective too literally when I made this.

Nevertheless, the key thing _I_ want to see is for us to commit to putting release notes out for some of Django's APIs that aren't necessarily considered stable. The 1.7 database API stuff was the main catalyst for this; those are changes and APIs we should have documented, and fall somewhere above "internal" (as we expect people to build third-party database backends), but not tie ourselves into a 3-release deprecation cycle for.

How about we change the label from "Experimental" to something like "Unstable", but keep the same provisions - documentation called out as "unstable feature", separate section in the release notes, etc. That establishes a clear level between "internal and we don't care about it" and "stable and publicly documented". The alternative is to change the DEP to just say we're going to start putting release notes up for certain internal APIs, and then somehow list the ones we will somewhere (probably on the API stability page).

Andrew

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/A3D3D45A-19CE-44D4-82DA-84AA45EDA586%40polytechnique.org.

For more options, visit https://groups.google.com/d/optout.

[Carl Meyer]

On 12/06/2014 07:24 PM, Andrew Godwin wrote:
> My notes from the meeting say "experimental API language", so I may have
> taken an adjective too literally when I made this.
>
> Nevertheless, the key thing _I_ want to see is for us to commit to
> putting release notes out for some of Django's APIs that aren't
> necessarily considered stable. The 1.7 database API stuff was the main
> catalyst for this; those are changes and APIs we should have documented,
> and fall somewhere above "internal" (as we expect people to build
> third-party database backends), but not tie ourselves into a 3-release
> deprecation cycle for.
>
> How about we change the label from "Experimental" to something like
> "Unstable", but keep the same provisions - documentation called out as
> "unstable feature", separate section in the release notes, etc. That
> establishes a clear level between "internal and we don't care about it"
> and "stable and publicly documented". The alternative is to change the
> DEP to just say we're going to start putting release notes up for
> certain internal APIs, and then somehow list the ones we will somewhere
> (probably on the API stability page).

Something doesn't feel quite right to me about the word "unstable"
either. I guess it's technically applicable to something we want to
retain the freedom to change, but it seems pejorative.

I guess I would favor your latter alternative: don't introduce new
terminology, just document some things (probably starting with the
database backend interface) in the internals/ section of the docs, and
clarify that while APIs documented there are not covered by the
back-compat policy, changes to them will be noted in the release notes.

I don't really think there's a problem with the word "internal" here:
from the perspective of the vast majority of Django users, the database
backend API is internal, but there are a few (maintainers of third-party
backends) for whom its a very important interface, so we document it.

Carl

[Shai Berger]

I like the general idea of experimental API, although Carl and Aymeric's notes
are important: If we do this, we need to very picky in its use, or else it
just becomes an easy route to avoid committment. In particular, there should
be a hard-and-fast rule that nothing should be made an "experimental API" if
it can just be done outside of core.

For the example given -- backend API -- I think that, again, Carl and Aymeric
are right; the examples I have more in mind for this are user-facing ORM
features like the expressions, or custom lookups. I think it might be better
to let people play with them more, in a released version, before setting the
APIs in concrete.

With that in mind, I would also reconsider adding a silent warning when the
features are used -- because, if they are for general use (as opposed to the
use of, say, 3rd-party backend developers) then there's a significant use-case
where the person who would use the API is not the person who configures the
warnings on the CI.

My 2 cents,
Shai.

[Russell Keith-Magee]

To my mind, the role of this new status is closer to "provisional", rather than "experimental". It's a recognition of the fact that no matter how many times we ask for pre-release testing, the first *real* test of any API is when it sees real-world use. 

In some respects, it a return to the pre-1.0 status of Django APIs, where we would endeavour to maintain backwards compatibility if at all possible, but we would be willing to break an API if it was in the best interests of future use. As an example of the sorts of things that got changed see:

https://code.djangoproject.com/wiki/BackwardsIncompatibleChanges

The difference here is that we're doing it for a specific API or new feature, not for the entire API.

I agree that "experimental" status shouldn't be used as a way to avoid commitment. IMHO, the "rules of use" are something like this:

 * It can only be used for major feature APIs. If a feature doesn't warrant a significant mention in the release notes, then it's either an internal API, or a minor feature that we can live with being locked down. 

 * There is a strict timeline - if a feature is provisional/experimental in vX, then it should automatically be final in vX+1, unless enough changes are made to the API that warrant another provisional/experimental version cycle. However, at each release, the intention should be to lock down all provisional APIs in the previous release if at all possible. 

 * The corollary of this last point is that the release *before* a stable release can't have any experimental/provisional features. 

 * Even though we have the escape hatch, we only use if if we have to. A rapid-deprecation backwards compatible path is preferable to a fundamental backwards incompatible change.

For my money, the role of this policy change isn't to introduce instability into Django's process. The role is to give us permission to introduce features which we might not otherwise land (or might delay in landing) due to fears over whether the public-facing API is suitable for use. 

Russ %-)

[Michael Manfre]

On Sun, Dec 7, 2014 at 6:37 PM, Russell Keith-Magee <rus...@keith-magee.com> wrote:

 * The corollary of this last point is that the release *before* a stable release can't have any experimental/provisional features. 

 Does this mean that there should be no experimental/provisional features in an LTS?

Regards,

Michael Manfre

[Russell Keith-Magee]

That point is already covered by the DEP, so I didn't repeat it; but yes, to my mind it also follows that corollary is that an LTS has no provisional features.

Yours

Russ Magee %-)

[Carl Meyer]

Thanks Russ (and Shai), this is very helpful.

For one thing, it's good to clarify that we're _not_ talking about the
database backends API here -- I think that's a unique case where
"internal but documented" (a status we already have) is a better fit
than any of "experimental" or "unstable" or "provisional". I agree that
for big new public-facing features, "internal but documented" doesn't
fit as well.

And secondly, if we are going to do this, I think "provisional" is a
much better choice of term than either "experimental" or "unstable".

Nonetheless, I am -1 on this DEP. I certainly see the initial attraction
of the idea from the core developer standpoint, but I don't think in
practice it helps anyone. Nobody (to a reasonable first approximation)
will wait an entire release cycle to use an awesome new feature like
migrations or ORM expressions just because we've labeled the API
provisional. So in effect, all we've done by introducing the provisional
label is add needless complexity to our policies and given ourselves a
reason to break our users' code more cavalierly than we'd have otherwise
done.

Instead, I think we should continue to do exactly as we have done in the
past: add new features, make them as good as we possibly can with the
testing that we get pre-release (and frankly, though it took a few RCs,
we got a _lot_ of user testing of migrations before they were ever in a
final release), and then if major issues crop up after release, do our
best to fix them in as backwards-compatible a way as we can manage.

It's not like our backwards-compatibility policy is an unbreakable
straitjacket. When we have no choice in order to fix a critical issue,
we have broken backwards-compatibility without a deprecation path
before, and I'm sure we will do it again. But that choice should remain
an absolute last resort; usually a deprecation path is possible. For any
feature, whether new or old, we should stick to our
backwards-compatibility policy as closely as we can, and break it only
when we have no other reasonable choice. I don't see this "provisional"
label adding anything new or useful to that decision-making process.

Carl

[Carl Meyer]

On 12/07/2014 04:37 PM, Russell Keith-Magee wrote:

> For my money, the role of this policy change isn't to introduce
> instability into Django's process. The role is to give us permission to
> introduce features which we might not otherwise land (or might delay in
> landing) due to fears over whether the public-facing API is suitable for
> use.

That sounds good in theory -- but are there any examples, ever in
Django's history, of a feature we otherwise wanted, but chose not to
land because we weren't sure enough that the API was right?

"Delay landing" is a different story. If we're unsure enough of an API
to delay landing the code, then I think "delay landing until we're more
confident" is a much better choice than "land it anyway but with a
provisional tag."

Carl

[benjaoming]

Hi guys!

As an external user of Django, relying on its stability, I'd like to comment.... In general, I think this is possibly a good improvement.. but possibly also a very dangerous one...

On Monday, December 8, 2014 12:38:18 AM UTC+1, Russell Keith-Magee wrote:

On Sun, Dec 7, 2014 at 6:35 PM, Shai Berger <sh...@platonix.com> wrote:

I like the general idea of experimental API, although Carl and Aymeric's notes
are important: If we do this, we need to very picky in its use, or else it
just becomes an easy route to avoid committment. In particular, there should
be a hard-and-fast rule that nothing should be made an "experimental API" if
it can just be done outside of core.

For the example given -- backend API -- I think that, again, Carl and Aymeric
are right; the examples I have more in mind for this are user-facing ORM
features like the expressions, or custom lookups. I think it might be better
to let people play with them more, in a released version, before setting the
APIs in concrete.

With that in mind, I would also reconsider adding a silent warning when the
features are used -- because, if they are for general use (as opposed to the
use of, say, 3rd-party backend developers) then there's a significant use-case
where the person who would use the API is not the person who configures the
warnings on the CI.

To my mind, the role of this new status is closer to "provisional", rather than "experimental". It's a recognition of the fact that no matter how many times we ask for pre-release testing, the first *real* test of any API is when it sees real-world use.

What I would fear is not so much that experimentals/provisionals end up in other parts of Django itself: The Django team is well-disciplined and will understand its own release protocol :)

I'm more worried that the application ecology starts suffering from symptoms of an experimental attitude in the Django project...

This DEP would definitely have to be understood by the outside app-developer: For the stability of third-party apps, app maintainers should consider to put "don't use experimental APIs" in their contribution guidelines and make sure to go through the painful process of rejecting well-intended PRs relying on experimental components. Especially if it's decided not to raise Warnings when experimental modules are imported.

For the projects and applications that do use experimentals, it would be nice to have a guarantee of a smooth API transition when the next Django ships. DEP states that APIs can change, but that it shouldn't do so. If stated that the API should not change, that causes for a much more significant guarantee, and sets the bar reasonably high for new experimental components.

The amount of deprecations in Django in general is very high, and currently, supporting both South and db.migrations is quite hard. So adding more complexity to multi-version Django support does not sound very attractive to me and should be avoided at all costs.

Some suggestions for the "experimental" definition / ruleset:

• API has been thoroughly agreed and is not subject to change.
  
• Only stability and performance is a questionable part that makes a feature experimental.
  
• Internal parts of an experimental component are expected to change more than usual
  
• Something does not get pulled out without a deprecation warning, even though it's experimental

If experimental components will be allowed to change their APIs, I would definitely suggest a Warning to be raised so they're kept out of the app ecology.

The relevant part of the DEP reads:

Instead, an Experimental designation will allow APIs to be included in Django and released without being beholden to the full deprecation cycle.

If you allow that to happen, then in essence, anything can be released and removed again. But there should be a clear intention when something is released, that it will be made permanent, right?

DEP also states:

Effort will be taken to communicate to users that the APIs in question are not stable to avoid inadvertent use.

I don't think that's possible. If the feature is attractive, it will be used.

RKM wrote:

due to fears over whether the public-facing API is suitable for use.

This could be rephrased: Because some components do not get the exposure necessary for thorough testing through the availability and use of the beta releases, we need experimental components in the final releases.

The goal is to get a broader test audience for *more rapid* releases and to have *faster* maturing of new features. Just be careful that this stuff doesn't end up where it's not supposed to, that's my main worry.

Other than that, it's really easy to release django applications, and features not agreed to enter the django mainline have always been referred to release as an external component. This is possibly still the best way...

Best,
Benjamin

[Tim Graham]

I am skeptical like Carl, but it seems to me the time to introduce and discuss this DEP is when there is an actual feature that we think could benefit from this policy so we don't have to talk in hypotheticals.

[Marc Tamlyn]

I'm going to agree with Tim, it's a good idea but it is hard to agree on its impact without the specifics.

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/938dd835-3678-483e-ab30-0bfeda9fd3db%40googlegroups.com.