New PyTables release - call for discussion
55 views
Skip to first unread message
Antonio Valentino
Dec 8, 2021, 3:29:49 AM12/8/21
to pytables-dev
Dear PyTables developers,
several users requested an official release of PyTables in order to have
compatibility with updated versions of Python, numpy, HDF5 etc.
I think that it is totally reasonable, because the release (2019) is
quite aged.
Probably (not sure) I could manage to find the time to make a new
release during the Christmas break.
Anyway, I can commit only for finalising the release of the source code.
I do not intend to make the packaging job to produce wheels on several
platforms. The reasons are:
* I do not have access to platforms, like windows or OSX in arm, to do
the job
* I *do not want* to spend my time for it
* personally I think that there is already a very good solution for
distributing binary packages that is conda. I could eventually help with
the conda packaging if necessary.
Do not providing official wheels could be an issue, indeed many users
are requesting them explicitly.
For this reason I thing that we should make a decision all together.
To me options are:
1. release source only and make a statement that the PyTables will not
provide official wheel packages (in that case I will close all the
issues related to wheels)
2. another developer steps forward to provide official wheels
3. do nothing
In case we decide for option 3 we should also consider to properly
communicate to users that they can no longer rely on PyTables updates
(at least in the near future).
kind regards
--
Antonio Valentino
several users requested an official release of PyTables in order to have
compatibility with updated versions of Python, numpy, HDF5 etc.
I think that it is totally reasonable, because the release (2019) is
quite aged.
Probably (not sure) I could manage to find the time to make a new
release during the Christmas break.
Anyway, I can commit only for finalising the release of the source code.
I do not intend to make the packaging job to produce wheels on several
platforms. The reasons are:
* I do not have access to platforms, like windows or OSX in arm, to do
the job
* I *do not want* to spend my time for it
* personally I think that there is already a very good solution for
distributing binary packages that is conda. I could eventually help with
the conda packaging if necessary.
Do not providing official wheels could be an issue, indeed many users
are requesting them explicitly.
For this reason I thing that we should make a decision all together.
To me options are:
1. release source only and make a statement that the PyTables will not
provide official wheel packages (in that case I will close all the
issues related to wheels)
2. another developer steps forward to provide official wheels
3. do nothing
In case we decide for option 3 we should also consider to properly
communicate to users that they can no longer rely on PyTables updates
(at least in the near future).
kind regards
--
Antonio Valentino
Josh Moore
Dec 8, 2021, 6:41:10 AM12/8/21
to pytabl...@googlegroups.com
A few quick thoughts from my side, for what their worth:
- If there's no capacity for wheels, I think the "use conda" strategy is a fine one, certainly for the short-term. (It's one we use ourselves, especially for Windows.)
- I don't know to what degree a qemu-based build would be portable to OSX/arm (nor do I have one for testing), but I did have some success in using qemu for GHA builds: https://github.com/zarr-developers/zarr-python/pull/869 (even if I couldn't reproduce my big-endian issue...)
- That being said, I will eventually need to start working on wheels on arm for OME packages. If I'm successful, I'm happy to try to share the workflow. (Better yet, if anyone has a good working example, I'm happy to test it on OME _and_ PyTables)
- Finally, there's also the option of applying e.g. to NumFOCUS for funds for driving wheels. If there's a general sense that that's feasible, I'd be happy to get involved. Francesc could perhaps comment on his experience with this.
~J
--
You received this message because you are subscribed to the Google Groups "pytables-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pytables-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pytables-dev/3ff0aab5-630b-ef38-4a7c-d37af2fb6126%40tiscali.it.
Antonio Valentino
Dec 8, 2021, 7:10:52 AM12/8/21
to pytabl...@googlegroups.com
Dear Josh,
thanks a lot for your feedback
Il 08/12/21 12:40, Josh Moore ha scritto:
most of the cases
Unfortunately duilding in quemu is quite slow
> - That being said, I will eventually need to start working on wheels on
> arm for OME packages. If I'm successful, I'm happy to try to share the
> workflow. (Better yet, if anyone has a good working example, I'm happy to
> test it on OME _and_ PyTables)
I think that we have some basic setup to build wheel on github.
Also there are WIP PRs adding more features (if I remember correctly).
Main technical issue IMHO are:
* not all platforms are currently covered
* may users need wheels with an updated HDF5 version while the current
GHA setup uses the hdf5 version packaged in the container (viw yum or apt)
Probably it could help to automate the correct build process if we start
to vendor HDF5 sources (in the same way we do it for blocs).
Not sure about that anyway.
> - Finally, there's also the option of applying e.g. to NumFOCUS for funds
> for driving wheels. If there's a general sense that that's feasible, I'd be
> happy to get involved. Francesc could perhaps comment on his experience
> with this.
It would be very nice indeed.
kind regards
--
Antonio Valentino
thanks a lot for your feedback
Il 08/12/21 12:40, Josh Moore ha scritto:
> A few quick thoughts from my side, for what their worth:
>
> - If there's no capacity for wheels, I think the "use conda" strategy is a
> fine one, certainly for the short-term. (It's one we use ourselves,
> especially for Windows.)
>
> - I don't know to what degree a qemu-based build would be portable to
> OSX/arm (nor do I have one for testing), but I did have some success in
> using qemu for GHA builds:
> https://github.com/zarr-developers/zarr-python/pull/869 (even if I couldn't
> reproduce my big-endian issue...)
in can confirm that, my experience, qemu-based build are very useful in
>
> - If there's no capacity for wheels, I think the "use conda" strategy is a
> fine one, certainly for the short-term. (It's one we use ourselves,
> especially for Windows.)
>
> - I don't know to what degree a qemu-based build would be portable to
> OSX/arm (nor do I have one for testing), but I did have some success in
> using qemu for GHA builds:
> https://github.com/zarr-developers/zarr-python/pull/869 (even if I couldn't
> reproduce my big-endian issue...)
most of the cases
Unfortunately duilding in quemu is quite slow
> - That being said, I will eventually need to start working on wheels on
> arm for OME packages. If I'm successful, I'm happy to try to share the
> workflow. (Better yet, if anyone has a good working example, I'm happy to
> test it on OME _and_ PyTables)
Also there are WIP PRs adding more features (if I remember correctly).
Main technical issue IMHO are:
* not all platforms are currently covered
* may users need wheels with an updated HDF5 version while the current
GHA setup uses the hdf5 version packaged in the container (viw yum or apt)
Probably it could help to automate the correct build process if we start
to vendor HDF5 sources (in the same way we do it for blocs).
Not sure about that anyway.
> - Finally, there's also the option of applying e.g. to NumFOCUS for funds
> for driving wheels. If there's a general sense that that's feasible, I'd be
> happy to get involved. Francesc could perhaps comment on his experience
> with this.
kind regards
Antonio Valentino
Francesc Alted
Dec 8, 2021, 8:26:52 AM12/8/21
to pytabl...@googlegroups.com
On Wed, Dec 8, 2021 at 1:11 PM Antonio Valentino <antonio....@tiscali.it> wrote:
Dear Josh,
thanks a lot for your feedback
Il 08/12/21 12:40, Josh Moore ha scritto:
> A few quick thoughts from my side, for what their worth:
>
> - If there's no capacity for wheels, I think the "use conda" strategy is a
> fine one, certainly for the short-term. (It's one we use ourselves,
> especially for Windows.)
>
> - I don't know to what degree a qemu-based build would be portable to
> OSX/arm (nor do I have one for testing), but I did have some success in
> using qemu for GHA builds:
> https://github.com/zarr-developers/zarr-python/pull/869 (even if I couldn't
> reproduce my big-endian issue...)
in can confirm that, my experience, qemu-based build are very useful in
most of the cases
Unfortunately duilding in quemu is quite slow
Agreed. I expect GHA (or other CIs) to provide native OSX/ARM soon, but apparently this has not happened yet.
> - That being said, I will eventually need to start working on wheels on
> arm for OME packages. If I'm successful, I'm happy to try to share the
> workflow. (Better yet, if anyone has a good working example, I'm happy to
> test it on OME _and_ PyTables)
I think that we have some basic setup to build wheel on github.
Also there are WIP PRs adding more features (if I remember correctly).
Main technical issue IMHO are:
* not all platforms are currently covered
* may users need wheels with an updated HDF5 version while the current
GHA setup uses the hdf5 version packaged in the container (viw yum or apt)
Probably it could help to automate the correct build process if we start
to vendor HDF5 sources (in the same way we do it for blocs).
Not sure about that anyway.
I think that, provided the amount of maintainers in PyTables (for the last two years, essentially Antonio), it makes little sense to try to tackle the making of wheels from inside the PyTables team. I sincerely think that trusting in other, more experienced binary packagers (like conda-related projects or Linux distributions) is the way to go.
> - Finally, there's also the option of applying e.g. to NumFOCUS for funds
> for driving wheels. If there's a general sense that that's feasible, I'd be
> happy to get involved. Francesc could perhaps comment on his experience
> with this.
It would be very nice indeed.
Yes, we are currently providing wheels for the Blosc project because we have got funds for doing that *and* we have people for doing the job. IIRC, I suggested several times for PyTables developers to ask for NumFOCUS grants for doing a similar thing, but I don't think anybody stepped ahead yet, probably because they have not that much time for maintenance (and, as said, this is why I suggest PyTables maintainers keep doing source releases and let binaries in the hands of more specialized people).
My 2 cents,
Francesc
PS: We will never thank Antonio enough for his amazing work at maintaining PyTables for the past decade (at least) with no reward other than serving the community.
To view this discussion on the web visit https://groups.google.com/d/msgid/pytables-dev/7b570fd1-3144-6905-7b19-9690d23ad449%40tiscali.it.
Francesc Alted
Antonio Valentino
Dec 8, 2021, 12:16:25 PM12/8/21
to pytabl...@googlegroups.com
Dear Francesc, dear all,
Il 08/12/21 14:26, Francesc Alted ha scritto:
available.
For me quemu solution are still the only way to worl on issues locally,
even if it is luckily not so often necessary.
>>> - That being said, I will eventually need to start working on wheels on
>>> arm for OME packages. If I'm successful, I'm happy to try to share the
>>> workflow. (Better yet, if anyone has a good working example, I'm happy to
>>> test it on OME _and_ PyTables)
>>
>> I think that we have some basic setup to build wheel on github.
>> Also there are WIP PRs adding more features (if I remember correctly).
>> Main technical issue IMHO are:
>>
>> * not all platforms are currently covered
>> * may users need wheels with an updated HDF5 version while the current
>> GHA setup uses the hdf5 version packaged in the container (viw yum or apt)
>>
>> Probably it could help to automate the correct build process if we start
>> to vendor HDF5 sources (in the same way we do it for blocs).
>> Not sure about that anyway.
>>
>
> I think that, provided the amount of maintainers in PyTables (for the last
> two years, essentially Antonio), it makes little sense to try to tackle the
> making of wheels from inside the PyTables team. I sincerely think that
> trusting in other, more experienced binary packagers (like conda-related
> projects or Linux distributions) is the way to go.
this is exactly my feeling.
I think that it is important to have a clear and shared position on this
point.
My point is that if someone want to contribute effort to produce wheel
it would be more than welcome, and we could find a way to give it an
"official stamp" meybe.
The point IMHO is that future releases should not be blocked by the
impossiblity to produce wheels whithin this project.
>>> - Finally, there's also the option of applying e.g. to NumFOCUS for
>> funds
>>> for driving wheels. If there's a general sense that that's feasible, I'd
>> be
>>> happy to get involved. Francesc could perhaps comment on his experience
>>> with this.
>>
>> It would be very nice indeed.
>>
>
> Yes, we are currently providing wheels for the Blosc project because we
> have got funds for doing that *and* we have people for doing the job.
> IIRC, I suggested several times for PyTables developers to ask for NumFOCUS
> grants for doing a similar thing, but I don't think anybody stepped ahead
> yet, probably because they have not that much time for maintenance (and, as
> said, this is why I suggest PyTables maintainers keep doing source releases
> and let binaries in the hands of more specialized people).
+1
> PS: We will never thank Antonio enough for his amazing work at maintaining
> PyTables for the past decade (at least) with no reward other than serving
> the community.
Thanks Francesc for your kind words.
Unfortunately also for me the activity on PyTables has become very very
sporadic.
ciao
--
Antonio Valentino
Il 08/12/21 14:26, Francesc Alted ha scritto:
> On Wed, Dec 8, 2021 at 1:11 PM Antonio Valentino <
> antonio....@tiscali.it> wrote:
>
>> Dear Josh,
>> thanks a lot for your feedback
>>
>> Il 08/12/21 12:40, Josh Moore ha scritto:
>>> A few quick thoughts from my side, for what their worth:
>>>
>>> - If there's no capacity for wheels, I think the "use conda" strategy
>> is a
>>> fine one, certainly for the short-term. (It's one we use ourselves,
>>> especially for Windows.)
>>>
>>> - I don't know to what degree a qemu-based build would be portable to
>>> OSX/arm (nor do I have one for testing), but I did have some success in
>>> using qemu for GHA builds:
>>> https://github.com/zarr-developers/zarr-python/pull/869 (even if I
>> couldn't
>>> reproduce my big-endian issue...)
>>
>> in can confirm that, my experience, qemu-based build are very useful in
>> most of the cases
>> Unfortunately duilding in quemu is quite slow
>>
>
> Agreed. I expect GHA (or other CIs) to provide native OSX/ARM soon, but
> apparently this has not happened yet.
yes, I agree with you that native runners should be used in CI when
> antonio....@tiscali.it> wrote:
>
>> Dear Josh,
>> thanks a lot for your feedback
>>
>> Il 08/12/21 12:40, Josh Moore ha scritto:
>>> A few quick thoughts from my side, for what their worth:
>>>
>>> - If there's no capacity for wheels, I think the "use conda" strategy
>> is a
>>> fine one, certainly for the short-term. (It's one we use ourselves,
>>> especially for Windows.)
>>>
>>> - I don't know to what degree a qemu-based build would be portable to
>>> OSX/arm (nor do I have one for testing), but I did have some success in
>>> using qemu for GHA builds:
>>> https://github.com/zarr-developers/zarr-python/pull/869 (even if I
>> couldn't
>>> reproduce my big-endian issue...)
>>
>> in can confirm that, my experience, qemu-based build are very useful in
>> most of the cases
>> Unfortunately duilding in quemu is quite slow
>>
>
> Agreed. I expect GHA (or other CIs) to provide native OSX/ARM soon, but
> apparently this has not happened yet.
available.
For me quemu solution are still the only way to worl on issues locally,
even if it is luckily not so often necessary.
>>> - That being said, I will eventually need to start working on wheels on
>>> arm for OME packages. If I'm successful, I'm happy to try to share the
>>> workflow. (Better yet, if anyone has a good working example, I'm happy to
>>> test it on OME _and_ PyTables)
>>
>> I think that we have some basic setup to build wheel on github.
>> Also there are WIP PRs adding more features (if I remember correctly).
>> Main technical issue IMHO are:
>>
>> * not all platforms are currently covered
>> * may users need wheels with an updated HDF5 version while the current
>> GHA setup uses the hdf5 version packaged in the container (viw yum or apt)
>>
>> Probably it could help to automate the correct build process if we start
>> to vendor HDF5 sources (in the same way we do it for blocs).
>> Not sure about that anyway.
>>
>
> I think that, provided the amount of maintainers in PyTables (for the last
> two years, essentially Antonio), it makes little sense to try to tackle the
> making of wheels from inside the PyTables team. I sincerely think that
> trusting in other, more experienced binary packagers (like conda-related
> projects or Linux distributions) is the way to go.
I think that it is important to have a clear and shared position on this
point.
My point is that if someone want to contribute effort to produce wheel
it would be more than welcome, and we could find a way to give it an
"official stamp" meybe.
The point IMHO is that future releases should not be blocked by the
impossiblity to produce wheels whithin this project.
>>> - Finally, there's also the option of applying e.g. to NumFOCUS for
>> funds
>>> for driving wheels. If there's a general sense that that's feasible, I'd
>> be
>>> happy to get involved. Francesc could perhaps comment on his experience
>>> with this.
>>
>> It would be very nice indeed.
>>
>
> Yes, we are currently providing wheels for the Blosc project because we
> have got funds for doing that *and* we have people for doing the job.
> IIRC, I suggested several times for PyTables developers to ask for NumFOCUS
> grants for doing a similar thing, but I don't think anybody stepped ahead
> yet, probably because they have not that much time for maintenance (and, as
> said, this is why I suggest PyTables maintainers keep doing source releases
> and let binaries in the hands of more specialized people).
> PS: We will never thank Antonio enough for his amazing work at maintaining
> PyTables for the past decade (at least) with no reward other than serving
> the community.
Unfortunately also for me the activity on PyTables has become very very
sporadic.
--
Antonio Valentino
Francesc Alted
Dec 8, 2021, 12:34:02 PM12/8/21
to pytabl...@googlegroups.com
On Wed, Dec 8, 2021 at 6:16 PM Antonio Valentino <antonio....@tiscali.it> wrote:
<snip>
My point is that if someone want to contribute effort to produce wheel
it would be more than welcome, and we could find a way to give it an
"official stamp" meybe.
The point IMHO is that future releases should not be blocked by the
impossiblity to produce wheels whithin this project.
Totally agreed.
Francesc Alted
Josh Moore
Dec 8, 2021, 12:45:41 PM12/8/21
to pytabl...@googlegroups.com
+1 from my side then for a source-only strategy. (And seconded thanks to Anthony) If/when a GHA workflow is submitted for building wheels, we can re-evaluate.
~J
--
You received this message because you are subscribed to the Google Groups "pytables-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pytables-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pytables-dev/CAFrp1vrH_UjHXgAMTNOLYAWCNJvsG7nrV8EzQYBeuo4aWtO4Zw%40mail.gmail.com.
Matthias Voppichler
Dec 11, 2021, 4:21:36 AM12/11/21
to pytables-dev
> 2. another developer steps forward to provide official wheels
In my understanding, it's basically complete (but is obviously missing support for python 3.10, which didn't exist yet when that PR was created).
Based on this let me ask - what would be required to "step forward" to provide official wheels - if a PR adding that functionality to CI does not seem to be sufficient (or maybe it was simply overlooked / forgotten)?
I don't think "conda only" is a good/valid approach, as installation via pypi is the main method of installation for a majority of people.
There are many cases where using an (opinionated) distribution is not possible - and i don't see the "hurdle" of providing wheels as that high (considering it's possible to build in CI pretty easily).
Antonio Valentino
Dec 11, 2021, 5:36:18 AM12/11/21
to pytabl...@googlegroups.com
Dear Matthias,
Il 11/12/21 10:21, Matthias Voppichler ha scritto:
https://github.com/PyTables/PyTables/issues/909#issuecomment-991578385
> Based on this let me ask - what would be required to "step forward" to
> provide official wheels - if a PR adding that functionality to CI does not
> seem to be sufficient (or maybe it was simply overlooked / forgotten)?
I have already said, in my previous post, that if someone provides
properly working wheels I have nothing against including them in an
"official" release.
But still there is a maintenance effort that someone should take in charge.
At the moment we have at least 7 issues regarding wheels:
https://github.com/PyTables/PyTables/issues?q=is%3Aopen+is%3Aissue+label%3Awheel
> I don't think "conda only" is a good/valid approach, as installation via
> pypi is the main method of installation for a majority of people.
> There are many cases where using an (opinionated) distribution is not
> possible - and i don't see the "hurdle" of providing wheels as that high
> (considering it's possible to build in CI pretty easily).
I have my personal ideas about that but I don't want to discuss your point.
As already said the real problem is that no one seems to have the time
and willing to produce and maintain wheels for PyTables.
> On Wednesday, December 8, 2021 at 6:45:41 PM UTC+1 Josh Moore wrote:
>
>> +1 from my side then for a source-only strategy. (And seconded thanks to
>> Anthony) If/when a GHA workflow is submitted for building wheels, we can
>> re-evaluate.
>>
>> ~J
>>
>> On Wed, Dec 8, 2021 at 6:33 PM Francesc Alted <fal...@gmail.com> wrote:
>>
>>> On Wed, Dec 8, 2021 at 6:16 PM Antonio Valentino <antonio....@tiscali.it>
>>> wrote:
>>> <snip>
>>>
>>>> My point is that if someone want to contribute effort to produce wheel
>>>> it would be more than welcome, and we could find a way to give it an
>>>> "official stamp" meybe.
>>>> The point IMHO is that future releases should not be blocked by the
>>>> impossiblity to produce wheels whithin this project.
>>>>
>>>
>>> Totally agreed.
>>>
>>> --
>>> Francesc Alted
Il 11/12/21 10:21, Matthias Voppichler ha scritto:
>> 2. another developer steps forward to provide official wheels
>
> I think you might have overlooked the existing PR
> in https://github.com/PyTables/PyTables/pull/872 - which is attempting to
> provide wheels build via CI.
> In my understanding, it's basically complete (but is obviously missing
> support for python 3.10, which didn't exist yet when that PR was created).
I think that I have provided an answer to this in
>
> I think you might have overlooked the existing PR
> in https://github.com/PyTables/PyTables/pull/872 - which is attempting to
> provide wheels build via CI.
> In my understanding, it's basically complete (but is obviously missing
> support for python 3.10, which didn't exist yet when that PR was created).
https://github.com/PyTables/PyTables/issues/909#issuecomment-991578385
> Based on this let me ask - what would be required to "step forward" to
> provide official wheels - if a PR adding that functionality to CI does not
> seem to be sufficient (or maybe it was simply overlooked / forgotten)?
properly working wheels I have nothing against including them in an
"official" release.
But still there is a maintenance effort that someone should take in charge.
At the moment we have at least 7 issues regarding wheels:
https://github.com/PyTables/PyTables/issues?q=is%3Aopen+is%3Aissue+label%3Awheel
> I don't think "conda only" is a good/valid approach, as installation via
> pypi is the main method of installation for a majority of people.
> There are many cases where using an (opinionated) distribution is not
> possible - and i don't see the "hurdle" of providing wheels as that high
> (considering it's possible to build in CI pretty easily).
As already said the real problem is that no one seems to have the time
and willing to produce and maintain wheels for PyTables.
> On Wednesday, December 8, 2021 at 6:45:41 PM UTC+1 Josh Moore wrote:
>
>> +1 from my side then for a source-only strategy. (And seconded thanks to
>> Anthony) If/when a GHA workflow is submitted for building wheels, we can
>> re-evaluate.
>>
>> ~J
>>
>> On Wed, Dec 8, 2021 at 6:33 PM Francesc Alted <fal...@gmail.com> wrote:
>>
>>> On Wed, Dec 8, 2021 at 6:16 PM Antonio Valentino <antonio....@tiscali.it>
>>> wrote:
>>> <snip>
>>>
>>>> My point is that if someone want to contribute effort to produce wheel
>>>> it would be more than welcome, and we could find a way to give it an
>>>> "official stamp" meybe.
>>>> The point IMHO is that future releases should not be blocked by the
>>>> impossiblity to produce wheels whithin this project.
>>>>
>>>
>>> Totally agreed.
>>>
>>> --
>>> Francesc Alted
Matthias Voppichler
Dec 11, 2021, 2:48:39 PM12/11/21
to pytables-dev
Hi Antonio,
> At the moment we have at least 7 issues regarding wheels:
---
That said - i don't think anyone demanded from you (personally or otherwise) to add/implement/invest the time to provide such wheels, which i fully understand is a time-consuming effort, which sometimes maintainers won't have time for.
Personally, I think it would already help to have some clear points / guidelines / summary what is missing, and what will be required to provide wheels (at least for the major platforms).
In my understanding wheels for the following platforms are currently being built as part of CI - and should therefore be considered available:
* linux (3.6, 3.7, 3.8, 3.9, 3.10) - for i686, x86_64 and aarch64 - all built in github CI [3.10 as of tomorrow - assuming my PR get's merged].
* windows (3.6, 3.7 amd64) - built in appveyor
In my calculation - this leaves:
* windows 3.9 and 3.9 (amd64) - potentially aarch64 (although I'd consider this an "exotic" platform which we can neglect for the moment)
* macOS (x86_64, aarch64) (i think the intend was to build this at least in part in travisCI - however travis seems to not have ran in the last 9ish months - which might require another look...).
Obviously there'll be some manual effort to "scrape together" all the wheels as part of the release process - but i think it's not as bad as you make it sound, as we've got 1.5 out of 3 platforms already covered in the current master branch.
Let me know if you think I've forgotten a platform / combination.
Please note: I'm also willing to put in the work (at least for the moment) on getting wider wheels supported pretty short-term - obviously assuming there is interest from the maintainers to have this in place.
Regards
Matthias
Antonio Valentino
Dec 11, 2021, 4:02:05 PM12/11/21
to pytabl...@googlegroups.com
Dear Matthias,
Il 11/12/21 20:48, Matthias Voppichler ha scritto:
My point is that the list also shows that there are very specific needs:
#912: Release to support HDF5 1.12.1 and
#884 PyPi wheel for tables 3.6.1 on Python 3.9 bundles (old) libraries
I understand that this is a request of having wheels including HDF5
v1.12.1. At the moment I'm not sure what is the status, but I think that
at least for GNU/Linux it is not the case, and the implementation is not
a matter of pinning a specific version in your package manager.
#776 Build libhdf5 with the --enable-threadsafe flag
Probably this is easy because most of the packagers use that flag AFAIK.
The point here is that I don't even know what is the status on wheels
that are currently produced.
#841 PypI wheel tables 3.6.1 for Python 3.9 has been compiled with AVX2
instructions triggering SIGILL
Also this one should be easy but I don't know what is the status for the
wheels currently built (probably GNU/Linux is already fine).
> That said - i don't think anyone demanded from you (personally or
> otherwise) to add/implement/invest the time to provide such wheels, which i
> fully understand is a time-consuming effort, which sometimes maintainers
> won't have time for.
> Personally, I think it would already help to have some clear points /
> guidelines / summary what is missing, and what will be required to provide
> wheels (at least for the major platforms).
To me the basic point would be to have the build process properly set up
in CI and possibly tests on produced wheels in CI.
Also, produced wheels should be "acceptable" form a technical point of view
At the moment the status is that CI workflows to produce wheels that are
already in the master branch probably require some improvement (e.g. to
use specific HDF5 versions and add testing).
Regarding other platform the workflows are still not in the master
branch and it is not clear to me if e.g. the use of "brew" is considered
or not an acceptable solution.
> In my understanding wheels for the following platforms are currently being
> built as part of CI - and should therefore be considered available:
> * linux (3.6, 3.7, 3.8, 3.9, 3.10) - for i686, x86_64 and aarch64 - all
> built in github CI [3.10 as of tomorrow - assuming my PR get's merged].
> * windows (3.6, 3.7 amd64) - built in appveyor
OK, I have merged you PR, thanks a lot.
Please note, anyway, that the generated Linux wheels are built against
HDF5 v1.8.12, that is quite outdated (see also #912).
> In my calculation - this leaves:
> * windows 3.9 and 3.9 (amd64) - potentially aarch64 (although I'd consider
> this an "exotic" platform which we can neglect for the moment)
> * macOS (x86_64, aarch64) (i think the intend was to build this at least in
> part in travisCI - however travis seems to not have ran in the last 9ish
> months - which might require another look...).
I confirm that Travis is out of the games.
And I agree with you that windows on aarch64 is not fundamental.
In principle Mac builders (at least for x86_64) should be available in GHA.
> Obviously there'll be some manual effort to "scrape together" all the
> wheels as part of the release process - but i think it's not as bad as you
> make it sound, as we've got 1.5 out of 3 platforms already covered in the
> current master branch.
Downloading wheels included in CI artifacts and uploading to PyPi is not
a problem for me.
Anyway I'm not in favor of providing an incomplete set of wheels (e.g.
windows 3.6 and 3.7 but not 3.8, 3.8 and 3.10, or windows but not mac).
The reason is that I'm pretty sure that 5 minutes after the upload users
will start complaining for the missing wheels, and in that scenario it
would make perfectly sense.
By the way I don't want to be the one that puts vetoes.
If also other developers agree to provide partial sets of wheels I have
no problems with it.
Of course I will just ignore any complain form users in that regards.
> Let me know if you think I've forgotten a platform / combination.
>
> Please note: I'm also willing to put in the work (at least for the moment)
> on getting wider wheels supported pretty short-term - obviously assuming
> there is interest from the maintainers to have this in place.
I confirm that form my point of view any contribution in this and other
fields of the PyTables development is very welcome.
If you decide to do some job I will try to support you as much as
possible compatibly with the very limited time that I have to dedicate
to PyTables.
I do not commit to do any activity related to direct implementation,
testing and maintenance of wheels.
Also please consider that I plan to make a new release during the
Christmas break because after that I will not have time to do it.
kind regards
antonio
--
Antonio Valentino
Il 11/12/21 20:48, Matthias Voppichler ha scritto:
> Hi Antonio,
>
>> At the moment we have at least 7 issues regarding wheels:
>
> I think this shows that there is a generic need for it.
I totally agree that there is a lot of people asking for wheels.
>
>> At the moment we have at least 7 issues regarding wheels:
>
> I think this shows that there is a generic need for it.
My point is that the list also shows that there are very specific needs:
#912: Release to support HDF5 1.12.1 and
#884 PyPi wheel for tables 3.6.1 on Python 3.9 bundles (old) libraries
I understand that this is a request of having wheels including HDF5
v1.12.1. At the moment I'm not sure what is the status, but I think that
at least for GNU/Linux it is not the case, and the implementation is not
a matter of pinning a specific version in your package manager.
#776 Build libhdf5 with the --enable-threadsafe flag
Probably this is easy because most of the packagers use that flag AFAIK.
The point here is that I don't even know what is the status on wheels
that are currently produced.
#841 PypI wheel tables 3.6.1 for Python 3.9 has been compiled with AVX2
instructions triggering SIGILL
Also this one should be easy but I don't know what is the status for the
wheels currently built (probably GNU/Linux is already fine).
> That said - i don't think anyone demanded from you (personally or
> otherwise) to add/implement/invest the time to provide such wheels, which i
> fully understand is a time-consuming effort, which sometimes maintainers
> won't have time for.
> Personally, I think it would already help to have some clear points /
> guidelines / summary what is missing, and what will be required to provide
> wheels (at least for the major platforms).
in CI and possibly tests on produced wheels in CI.
Also, produced wheels should be "acceptable" form a technical point of view
At the moment the status is that CI workflows to produce wheels that are
already in the master branch probably require some improvement (e.g. to
use specific HDF5 versions and add testing).
Regarding other platform the workflows are still not in the master
branch and it is not clear to me if e.g. the use of "brew" is considered
or not an acceptable solution.
> In my understanding wheels for the following platforms are currently being
> built as part of CI - and should therefore be considered available:
> * linux (3.6, 3.7, 3.8, 3.9, 3.10) - for i686, x86_64 and aarch64 - all
> built in github CI [3.10 as of tomorrow - assuming my PR get's merged].
> * windows (3.6, 3.7 amd64) - built in appveyor
Please note, anyway, that the generated Linux wheels are built against
HDF5 v1.8.12, that is quite outdated (see also #912).
> In my calculation - this leaves:
> * windows 3.9 and 3.9 (amd64) - potentially aarch64 (although I'd consider
> this an "exotic" platform which we can neglect for the moment)
> * macOS (x86_64, aarch64) (i think the intend was to build this at least in
> part in travisCI - however travis seems to not have ran in the last 9ish
> months - which might require another look...).
And I agree with you that windows on aarch64 is not fundamental.
In principle Mac builders (at least for x86_64) should be available in GHA.
> Obviously there'll be some manual effort to "scrape together" all the
> wheels as part of the release process - but i think it's not as bad as you
> make it sound, as we've got 1.5 out of 3 platforms already covered in the
> current master branch.
a problem for me.
Anyway I'm not in favor of providing an incomplete set of wheels (e.g.
windows 3.6 and 3.7 but not 3.8, 3.8 and 3.10, or windows but not mac).
The reason is that I'm pretty sure that 5 minutes after the upload users
will start complaining for the missing wheels, and in that scenario it
would make perfectly sense.
By the way I don't want to be the one that puts vetoes.
If also other developers agree to provide partial sets of wheels I have
no problems with it.
Of course I will just ignore any complain form users in that regards.
> Let me know if you think I've forgotten a platform / combination.
>
> Please note: I'm also willing to put in the work (at least for the moment)
> on getting wider wheels supported pretty short-term - obviously assuming
> there is interest from the maintainers to have this in place.
fields of the PyTables development is very welcome.
If you decide to do some job I will try to support you as much as
possible compatibly with the very limited time that I have to dedicate
to PyTables.
I do not commit to do any activity related to direct implementation,
testing and maintenance of wheels.
Also please consider that I plan to make a new release during the
Christmas break because after that I will not have time to do it.
kind regards
antonio
Antonio Valentino
Reply all
Reply to author
Forward
0 new messages