pre-APE: a project-level policy for maintainable *new* public APIs ?

[Clément Robert]

Hi all, I've been thinking about proposing a set of guidelines to improve long-term maintainability in new APIs (new classes, functions, and arguments). These include, but are not necessarily limited to:

- never add positional-or-keyword arguments to existing functions (always use keyword-only arguments)

- prefer positional-only arguments for the first (couple) arguments in a new function

- avoid using (multiple) inheritance when adding a new class that doesn't extend on existing, inheritance-based patterns.

- avoid mixin classes for code reuse, prefer protocols and composable objects

- ...

basically, these are practices that 

* reduce the probability that we paint ourselves in a corner over the years, e.g.

    - when we can't smoothly deprecate parameters that are positional-allowed because doing so 

       might break (valid) caller code that also passes *further* arguments positionally

    - when a class hierarchy is too hard to reason, making bugs harder and harder to fix as

      complexity often grows but never scales back down

* are only applicable to *new* APIs, as enforcing them after-the-fact would always create backward incompatibilities.

In review, I often want point out to some of these, but it'd feel more confortable if there was a well established, project-wide consensus. So, my questions to you guys are: should I write a full fledged APE about this, and would anyone like to co-author it ? Does anyone feel strongly that this is *not* a good idea, and if so, why ?

Cheers,

Clément

[Marten van Kerkwijk]

Hi Clément,

Astropy is fairly mature, and hence I am not sure that enough new
additions to the astropy core package are foreseen that we really need
it, but maybe it is still a good idea to have a kind of best practice
document. If we do have one, ideally it would just point to a generally
accepted one that we follow, a bit like we do for PEP 8, and have
minimal astropy specific content.

Indeed, the only astropy-specific content that I would really like to
have would be to make explicit the current practice that for smaller
contributions to existing sub-packages of astropy, one should follow
whatever the pattern is that is used in those. I'd very much like to no
longer have to discuss "src layout!" or "internal modules should be
prefixed by underscored!"), or to have to ask to please not add no-ops
like ``__all__: list[str] = []``.

On the specific examples you give, I think most are obvious except your
last one, viz., to "avoid mixin classes for code reuse, prefer protocols
and composable objects". We have not done that before (protocols didn't
exist), and at least at the sub-package level I don't think it is
particularly helpful if some newer parts were to follow that, while the
older parts do not, i.e., I think it would be better to change a whole
sub-package if it really turns out that the existing class hierarchies
have become too complicated. (I might add that so far in the cases where
protocols were suggested, it seemed nice in principle, but in practice
it turned out that it ended up requiring the addition of specific
attributes just to make them runtime checkable.)

Anyway, overall my main comment is that if we go ahead with this, any
recommendations in an APE should be for really widely *used* best
practice (e.g., in CPython itself, or in the wider scientific python
ecosystem).

All the best,

Marten

[Juan BC]

Check the scikit learn policies for this kind of api spepcififcation. They come up with a simple and clear rules 

Juan B Cabral

--
You received this message because you are subscribed to the Google Groups "astropy-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to astropy-dev...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/astropy-dev/y2jseg130n0.fsf%40swan.astro.utoronto.ca.

[Pey Lian Lim]

Thanks for the ideas! APE probably is an overkill, unless somehow this can be added to the existing https://github.com/astropy/astropy-APEs/pull/85 . That said, https://docs.astropy.org/en/latest/development/codeguide.html could be updated, or we can point to other more established docs as well from there.

[Thomas Robitaille]

Hi Marten,

On Thu, 2 Oct 2025 at 14:15, 'Marten van Kerkwijk' via astropy-dev
<astro...@googlegroups.com> wrote:
>
> Hi Clément,
>
> Astropy is fairly mature, and hence I am not sure that enough new
> additions to the astropy core package are foreseen that we really need

> it.

Just wanted to chime in to say that even though the core package is
mature, there is a lot of development and additions going on in other
coordinated packages, so having guidelines could still make sense.

Cheers,
Tom

[Clément Robert]

Thanks everyone for engaging in this discussion !

Here's a point-by-point answer to salient points that I saw

> I'd very much like to no
longer have to discuss "src layout!" or "internal modules should be
prefixed by underscored!")

I guess this could be read two ways, but in my opinion:

- discussions around src layout will only die out if/when we actually make the switch (since the problems it solves are real *and* cannot be solved otherwise)

- explicitly marking (new !) private modules with a leading underscore is precisely the kind of thing I'd want to make policy

> I might add that so far in the cases where

protocols were suggested, it seemed nice in principle, but in practice
it turned out that it ended up requiring the addition of specific
attributes just to make them runtime checkable.

In my opinion, if you haven't seen protocols being useful *without* runtime-checkability, you are missing the point of protocols.

Anyway, that's not the core of my proposal so I'm willing to let this one go if it helps reach a consensus !

>  any recommendations in an APE should be for really widely *used* best

practice (e.g., in CPython itself, or in the wider scientific python
ecosystem).

keyword-only, positional-only arguments are indeed widely used in CPython and numpy/scipy (for APIs that were introduced after these tools became available !)

> Check the scikit learn policies for this kind of api specification

thanks for this suggestion, I will !

> APE probably is an overkill ...

I fear that anything less couldn't make the impact I'm aiming for; I'd like these guidelines (the exact content of which should be debated) to always be followed, so I think it needs to be formalized as a somewhat authoritative (but democratically-designed !) policy.

> ... unless somehow this can be added to the existing https://github.com/astropy/astropy-APEs/pull/85

I think it's actually quite orthogonal. Besides, APEs are aimed to be as minimally focused as possible, and our guidelines recommend making more APEs rather than merging conceptually separable ideas into giant proposals.

> Just wanted to chime in to say that even though the core package is

mature, there is a lot of development and additions going on in other
coordinated packages

True, providing guidelines for long term design sustainability in the *ecosystem* is also a goal here. But I should add that, while we don't envision future additions to the core package's API, they still happen overtime, and that's what I'm trying to consolidate here.

Clément

[Aldcroft, Tom]

Thanks Clément for the proposal and working to keep the astropy code base sustainable and modernizing it where possible.

Overall, I agree with Pey Lian that an APE may be overkill [1]. The coding guidelines you outlined originally fit in neatly with existing coding conventions documentation. That already includes a point about avoiding multiple inheritance. 

Another point to remember is that an APE is meant to be static, and the process for updating it is (by design) somewhat difficult. Coding conventions may evolve, or more importantly the way they are presented may need improvement (e.g. new examples, tweaking wording to address developer experience). This is easy to address in the astropy docs, but very hard to do in an APE.

As a way to move forward, I would suggest that you open a PR on the coding guidelines to include salient points. As a secondary goal, that page currently exists as mostly bulleted lists of points without any particular order or flow. I'm not sure how easy it would be to make the content more organized and accessible, but it might be worth trying.

[1] The one point that came up that would deserve an APE is changing to a src folder layout. I've googled the advantages of a src folder layout and IMO they were not so compelling as to justify the effort and disruption. Astropy, numpy, scipy, scikit-learn, pandas, astropy coordinated packages, and many many others all use the flat layout. I'm not aware of any serious problems that astropy or other projects have had in the 10+ years of development due to using the flat layout. I'm with Marten on discontinuing the src folder discussion until we actually experience real and compelling problems. This might be something to mention in the coding guidelines, e.g. "The astropy code package uses a flat source layout. The project source layout has been discussed extensively and we have no plans to migrate to a src folder layout.".

- Tom

To view this discussion visit https://groups.google.com/d/msgid/astropy-dev/65c6e67b-6178-4cd3-945a-c46f307c7cb6n%40googlegroups.com.

[Clément Robert]

> The coding guidelines you outlined originally fit in neatly with existing coding conventions documentation. That already includes a point about avoiding multiple inheritance. 

fair point.

> Another point to remember is that an APE is meant to be static, and the process for updating it is (by design) somewhat difficult. Coding conventions may evolve, or more importantly the way they are presented may need improvement (e.g. new examples, tweaking wording to address developer experience). This is easy to address in the astropy docs, but very hard to do in an APE.

I think you just convinced me with this one.

> As a way to move forward, I would suggest that you open a PR on the coding guidelines to include salient points. As a secondary goal, that page currently exists as mostly bulleted lists of points without any particular order or flow. I'm not sure how easy it would be to make the content more organized and accessible, but it might be worth trying.

I'll give it my best. I'm thinking it could have 2 reading levels: a "TLDR" best-practice-level, and a detailed explanation for each "rule" (including nuanced points on how they might not *always* apply).

I"d like to punt the src layout stuff for now, it was never the point of this discussion to begin with, but I can't resist adding one more point:

>  I'm with Marten on discontinuing the src folder discussion until we actually experience real and compelling problems. 

the issue with this approach is that we're having a strong survivor bias with assessing what counts as "real" problems; src layout is meant to shield against mistakes made (mostly) by *beginners*. To me this isn't about the dev exp of our existing team members: obviously they get by these issues, otherwise they wouldn't be here. Rather, it's a gate-keeping issue.

Anyway that's enough of this. Thanks everyone for your inputs. I'll try to put that out in a PR form soon !

Cheers

Clément