Add more-itertools as a standard package

[Samuel Lelièvre]

Dear sage-devel,

The "more-itertools" Python package extends the "itertools"
module with extra useful iteration tools such as `pairwise`,
`triplewise`, `unique_justseen`, etc.

- https://more-itertools.readthedocs.io/en/stable/index.html

Making it a standard package would allow writing many
algorithms more elegantly and naturally.

Including a new standard spkg requires a sage-devel
discussion, so I am writing about that.

I opened a ticket for that too:

- Add more-itertools as a standard package
https://trac.sagemath.org/ticket/33030

Would anyone else support adding this package?
Looking forward to reading your thoughts. --Samuel

[Michael Orlitzky]

On Thu, 2021-12-16 at 00:38 +0100, Samuel Lelièvre wrote:
> Dear sage-devel,
>
> The "more-itertools" Python package extends the "itertools"
> module with extra useful iteration tools such as `pairwise`,
> `triplewise`, `unique_justseen`, etc.
>

> Would anyone else support adding this package?
> Looking forward to reading your thoughts. --Samuel
>

We already have 214 standard packages. That's 214 pieces of software
copy & pasted into the sage releases... and 214 SPKGs that the sage
developers need to keep updating, and 214 distro packages that every
distro maintainer needs to keep track of as dependencies of the sage
package.

(Even that is not an honest accounting, since a few of our jupyter
dependencies are patched to hide the enormous dependency trees that
distribution packagers cannot avoid.)

More-itertools looks small and useful, but some day we have to stop
allowing that number to grow faster than anyone can chase it.

[Dima Pasechnik]

a pypi package without a version constraint in requirements.txt is not much of burden, maintenance-wise, until it starts breaking things.

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/8414ada2af3758712b8f731b0ec6369dbcd54494.camel%40orlitzky.com.

[Michael Orlitzky]

On Mon, 2021-12-20 at 14:29 +0000, Dima Pasechnik wrote:
> a pypi package without a version constraint in requirements.txt is not much
> of burden, maintenance-wise, until it starts breaking things.
>

It still needs to be upgraded forever, and added to every distro, and
upgraded in every distro forever.

Just because we don't list a version constraint doesn't mean there
isn't one. We need a version that's new enough to have all the features
we use. (Will v1.0 from April 2012 really work?) Distro users will have
a thousand other packages installed that can depend on more-itertools
and might conflict with whatever implicit version constraint sage has.

Each individual papercut is harmless, but if you look at the new ticket
list, it's filled with tickets for package upgrades and
incompatibilities. And that's just within sage.

[Lorenz Panny]

On Mon, 20 Dec 2021 14:41:27 +0100, Michael Orlitzky <mic...@orlitzky.com> wrote:
> We already have 214 standard packages. That's 214 pieces of software
> copy & pasted into the sage releases... and 214 SPKGs that the sage
> developers need to keep updating, and 214 distro packages that every
> distro maintainer needs to keep track of as dependencies of the sage
> package.

It's also 214 software packages which might, for all we know, at any
time be hijacked by The Bad Guys to run arbitrarily malicious code on
every Sage user's machine.

This is terrifying.

(For examples where the modern "import * from internet" mentality has
led to security disasters, just search for terms like "malicious npm".
Luckily it seems less bad with pip packages for now, but not for any
real reason. Every single piece of code we import adds huge security
questions, because updates to the dependency may be published at any
time totally invisible to Sage developers and the review process used
in Sage development. The build scripts will simply pull and run it.)

We should reduce dependencies, not add more. _Especially_ when it's
about non-essential convenience libraries.

Best,
Lorenz

[julian...@fsfe.org]

Hi Samuel,

This is a popular pure Python package. It seems to have a history of non-breaking releases, so I would not mind adding it if it makes our lives much easier (and keeps us from reinventing the wheel when implementing algorithms.) As a maintainer of SageMath in conda-forge, I don't mind new dependencies if they are very easy to package, popular, and actively maintained. While I am very much in favor of making SageMath more modular and I believe that some of our dependencies are a problem, I don't think that such pure Python dependencies are causing any issues here.

I am not too worried about the security implications here. more-itertools is according to GitHub used by 118k projects. So, if it gets compromised we'll know before we release a new version of SageMath and actually before we even consider upgrading our SPKG.

more-itertools is already packaged in the distributions I checked (Debian/Ubuntu, ArchLinux, conda-forge) btw.

julian