Packaging questions

[Ghislain Vaillant]

Hi Almar,

I came across imageio following the ANN posted on the Numpy / Scipy ML.

I have got a few questions regarding your package:
 * could you describe the purpose of imageio in a few sentences (which problems are addressed and which solutions are implemented) ?
 * how does imageio differ from the competition (PIL, Pillow, Scipy...)
 * what is the plan for future development / releases ?

Thank you, and keep up the good work :-)

Ghis

[Almar Klein]

Hi Ghis,

I added some text on the origin and outlook of the package to the
readme/website.

> * could you describe the purpose of imageio in a few sentences (which
> problems are addressed and which solutions are implemented) ?

The purpose of imageio is to have a library for reading and writing
images that is easy to install, easy to maintain, and can easily scale
the supported formats. I think the scipy community could use a single
package for reading/writing image, that does it good, and does nothing more.

> * how does imageio differ from the competition (PIL, Pillow, Scipy...)

PIL is a pain. Pillow should be better, but my personal experience is
not very good. All examples (and also the imread package) require Cython
or C/C++ skills to maintain. None of these is fool-proof pip-installable.

> * what is the plan for future development / releases ?

The core is pretty small and seems more or less settled; I expect by far
the most of the maintenance will be in the plugins. I hope to attract
developers, so we can each take one or two plugins to focus on.

To be honest, I made imageio because I thought it was needed, not
because I had an awful lot of time to work on it (which is also the
reason why it took me 2 years to get it to a relatively mature level). I
plan to actively maintain it, but could really use some help.

Regards,
Almar

>
> Thank you, and keep up the good work :-)
>
> Ghis
>

> --
> You received this message because you are subscribed to the Google
> Groups "imageio" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to imageio+u...@googlegroups.com
> <mailto:imageio+u...@googlegroups.com>.
> To post to this group, send email to ima...@googlegroups.com
> <mailto:ima...@googlegroups.com>.
> Visit this group at http://groups.google.com/group/imageio.
> For more options, visit https://groups.google.com/d/optout.

[Ghislain Vaillant]

Good, thanks.

I am intending to package your work for inclusion to the Debian / Ubuntu main archive. Let me know if you have any objection to it.

Cheers,
Ghis

[Almar Klein]

> I am intending to package your work for inclusion to the Debian / Ubuntu
> main archive. Let me know if you have any objection to it.

Ah, nice! No objection at all.

- Almar

>
> Cheers,

[Ghislain Vaillant]

Reason I asked for this set of questions was to help me file the initial ITP bug [1]

I'll keep you up-to-date with this.

Cheers,
- Ghis

[1] https://lists.debian.org/debian-devel/2014/11/msg00737.html

[Almar Klein]

Thanks for this Ghis! Let me know if you need something.

- Almar

[Ghislain Vaillant]

Hi Almar,

I have had a go at it. The Python 2 and 3 versions of the package builds nicely, but there are a few minor issues:

 * The source tarball of the current v1.0.0 does not include the sphinxdoc sources, which are necessary to build the documentation package. Would you consider including them in a future release please ?
 * It seems that the test suite requires downloading extra resources from the internet. This is prevented by the Debian packaging tools for security reasons. I can disable the tests (although this is considered bad practice in Debian) but it might be good to think about a "light" version of the test suite which does carry the problematic tests out. Or maybe include the extra files in the source release tarball.

Cheers,
- Ghis

[Almar Klein]

Hi Ghis,

> * The source tarball of the current v1.0.0 does not include the
> sphinxdoc sources, which are necessary to build the documentation
> package. Would you consider including them in a future release please ?

Sure. Just the .rst sources? Is there a preferred directory name to use?
Otherwise I'll just use "docs".

> * It seems that the test suite requires downloading extra resources
> from the internet. This is prevented by the Debian packaging tools for
> security reasons. I can disable the tests (although this is considered
> bad practice in Debian) but it might be good to think about a "light"
> version of the test suite which does carry the problematic tests out. Or
> maybe include the extra files in the source release tarball.

Ah yes, all tests that use images download these from the web. I could
make the test suite skip such tests based on some parameter. Is there
like an environment variable that I can use?

Can the extra resources be downloaded at install-time (i.e. when running
"setup.py install"), or do they need to be present in the tarball that I
upload to pypi?

- Almar


>
> Cheers,
> - Ghis

[Ghislain Vaillant]

'Morning Almar

Le lundi 17 novembre 2014 07:51:28 UTC, Almar Klein a écrit :

Hi Ghis,

>   * The source tarball of the current v1.0.0 does not include the
> sphinxdoc sources, which are necessary to build the documentation
> package. Would you consider including them in a future release please ?

Sure. Just the .rst sources? Is there a preferred directory name to use?
Otherwise I'll just use "docs".

The full content of the docs/ folder in your current source tree is enough.
Then, the package build system knows it has to cd into docs/, run
sphinxbuild and move the generated html to the documentation package. So
here, all you need is include docs/ in your MANIFEST.in file.

 

>   * It seems that the test suite requires downloading extra resources
> from the internet. This is prevented by the Debian packaging tools for
> security reasons. I can disable the tests (although this is considered
> bad practice in Debian) but it might be good to think about a "light"
> version of the test suite which does carry the problematic tests out. Or
> maybe include the extra files in the source release tarball.


Ah yes, all tests that use images download these from the web. I could
make the test suite skip such tests based on some parameter. Is there
like an environment variable that I can use?

Depends how you want to do it. You can either provide a test command
derived from distutils Command, which may accept a command line arg to
skip the problematic tests...

Can the extra resources be downloaded at install-time (i.e. when running
"setup.py install"), or do they need to be present in the tarball that I
upload to pypi?

Or make sure that when you build the source package, be it with a custom
`make dist` or `python setup.py sdist`, the additional pics necessary for the
tests are downloaded and included somewhere in the source package tree.

 

- Almar

Providing the documentation is more of a pressing issue from a packaging
standpoint than fixing the test suite TBH. You can calmly think about a
solution that works for you, should you want to fix it, for a next minor or
major release. For the documentation however, I'd warmly appreciate if you
could release something like a 1.0.1, containing 1.0.0 + docs.

Thank you very much for spending time on this.

Best,
- Ghis

[Almar Klein]

> > * The source tarball of the current v1.0.0 does not include the
> > sphinxdoc sources, which are necessary to build the documentation
> > package. Would you consider including them in a future release
> please ?
>
> Sure. Just the .rst sources? Is there a preferred directory name to
> use?
> Otherwise I'll just use "docs".
>
>
> The full content of the docs/ folder in your current source tree is enough.
> Then, the package build system knows it has to cd into docs/, run
> sphinxbuild and move the generated html to the documentation package. So
> here, all you need is include docs/ in your MANIFEST.in file.

Ok, will do this soon.

> Ah yes, all tests that use images download these from the web. I could
> make the test suite skip such tests based on some parameter. Is there
> like an environment variable that I can use?
>
>
> Depends how you want to do it. You can either provide a test command
> derived from distutils Command, which may accept a command line arg to
> skip the problematic tests...

BTW: how do you run the test suite, since it is not in the source
distribution?

Or do you download the snapshot from Github and build from there? Maybe
you can explain a bit what steps are involved in building the packages,
so I can see what might be the best moment to download resources.

BTW: do you have a Github account so I can cc you in corresponding issues?

- Almar

>
> Can the extra resources be downloaded at install-time (i.e. when
> running
> "setup.py install"), or do they need to be present in the tarball
> that I
> upload to pypi?
>
>
> Or make sure that when you build the source package, be it with a custom
> `make dist` or `python setup.py sdist`, the additional pics necessary
> for the
> tests are downloaded and included somewhere in the source package tree.
>
> - Almar
>
>
> Providing the documentation is more of a pressing issue from a packaging
> standpoint than fixing the test suite TBH. You can calmly think about a
> solution that works for you, should you want to fix it, for a next minor or
> major release. For the documentation however, I'd warmly appreciate if you
> could release something like a 1.0.1, containing 1.0.0 + docs.
>
> Thank you very much for spending time on this.
>
> Best,
> - Ghis
>

[Ghislain Vaillant]

I'll walk you through the package build process.

It takes an upstream tarball, like the v1.0 available on PyPi, and add a debian/
folder to it which contains all the necessary meta-data and rules to build the pkg.

For Python pkg, we have a helper which automatically cleans, configures, builds,
install and tests the pkg for all supported Python versions (now 2.7 and 3.4, but
later 3.4+). For the test suite, the build system autodetects how the test suite is
implemented, for instance with a separate test/ or tests/ folder, and runs a
`discover` to it. In your case, there is a testing.py in the source tree, which the
build system seems to be trying to run.

You are right though, if you do not ship the test suite with the source release, then
there is no reason to run it. However, Debian is encouraging us, pkg maintainers to
convince upstream devs, like yourself, to ship the test suite as well, so the latter
can be used for continuous integration purposes at the distribution level.

Again, this is entirely up to you and is not a pressing issue. It is only a
recommendation for now, though I foresee it to become more and more "encouraged",
to the point where a working test suite will be a requirement for any package in some
future.

Please let me know whether this was detailed enough.

PS: my github pseudo is ghisvail

Cheers,
- Ghis