Re: fanstatic compilers

[Wolfgang Schnerring]

Hi,

[finally moving to the mailinglist from private emails]

* Martijn Faassen <faa...@startifact.com> [2013-04-10 14:18]:
> > * Jan-Jaap Driessen <jdri...@minddistrict.com> [2013-04-08 09:36]:
> > > Should we put the compilers in separate fanstatic- packages
> > > (fanstatic-jsmin, fanstatic-less, etc?) We could even make the
> > > fanstatic entry point part of the "official" cssmin python package. I
> > > am wondering whether the author of these packages will honor our pull
> > > requests though.
> >
> > I've thought about that while I was sprinting, too. On the one hand,
> > yes, separation of concerns and all that. On the other hand, the
> > implementation of a single concrete compiler/minifier is in the order
> > of less than 10 lines of code, so having a whole package just for that
> > felt a bit like overkill. Additionally, it felt kind of nice for
> > Fanstatic to support ~80% of the common use cases right out of the box...
> >
> > In other words, I'm leaning towards "leave it in core for now" due to
> > laziness (on several axes), but I wouldn't object to breaking out
> > separate packages at all.
>
> I'm leaning towards 'leave in core for now' too at first glance, for the
> reasons Wolfgang gave.
>
> Are the dependencies optional dependencies in setup.py? I'm trying to see
> how to support the deployment scenario Jan-Jaap sketched out where the
> dependencies are not available. Optional dependencies is a first step in
> that direction.

Fanstatic has not gained additional dependencies to support compilers.

I opted to not declare the dependencies in fanstatic's setup.py at
all, i.e. not do a fanstatic[compilers] extra or something, since
a) many compilers require external binaries (node.js-based and so on)
which we couldn't declare anyway and b) collecting all possible
dependencies under one extra feels wrong (wouldn't want to force *all*
on somebody who just wants to use one), so we might as well leave it
off completely and tell people to require Python packages for
compilers "downstream", that is in their application.

Wolfgang

--
Wolfgang Schnerring · w...@gocept.com · Software development
gocept gmbh & co. kg · Forsterstraße 29 · 06112 Halle (Saale) · Germany
http://gocept.com · Tel +49 345 219401-0
Python, Pyramid, Plone, Zope · consulting, development, hosting, operations

[Jan-Jaap Driessen]

Hi Wolfgang,

On Wed, Apr 10, 2013 at 2:36 PM, Wolfgang Schnerring <w...@gocept.com> wrote:
> I opted to not declare the dependencies in fanstatic's setup.py at
> all, i.e. not do a fanstatic[compilers] extra or something, since
> a) many compilers require external binaries (node.js-based and so on)
> which we couldn't declare anyway and b) collecting all possible
> dependencies under one extra feels wrong (wouldn't want to force *all*
> on somebody who just wants to use one), so we might as well leave it
> off completely and tell people to require Python packages for
> compilers "downstream", that is in their application.

Sounds very reasonable. We could use extras_require in setup.py:

extras_require = {
'cssmin': ['cssmin'],
'jsmin': ['jsmin'],
}

Where the keys are the names of minifiers/compilers and the values are
list of python packages. However, this would only work for python
packages, not for node/java dependencies. There is a chance of clashes
between compiler and minifier names, but I am willing to bet there
won't be a 'js compiler' called 'cssmin'.

@Wolfgang, in [a] you hook into sdist_compile; previously we discussed
hooking into zest.releaser. It seems the sdist_compile hook is more
versatile than a zest.releaser pre-release hook. This is a very cool
hack!

a) https://bitbucket.org/fanstatic/fanstatic/src/70594589896703e0f79425e44edb0a841d380c60/fanstatic/compiler.py?at=wosc-compilers#cl-118


I took the liberty of fixing/adding some tests on the wosc-compilers
branch and added these features:

1. Added google closure compiler as a minifier, using the 'closure'
python package. Adding a compiler is a breeze! No problem for me to
keep them in the fanstatic package.

2. When compiling a resource, write a log record of the Resource being
compiled and the time spent. This can be picked up by a console
handler when working on a running app or in the fanstatic-compile
script by passing '-v' or '--verbose' for verbose output. This is a
nice indication to the developer when a server request is slow to load
when resources are compiled.

3. If you don't want the sources of your Resources to be publishable,
you can set your library 'ignores' setting to the globs of the sources
files; I have added this to the documentation.


I would like to discuss the following topics:

1. It is good practice not to do compilation on a production system.
When deploying an app to production, the compiler backend need not be
available. The NeededResources 'compile' flag determines whether
resources need to be compiled. If a fanstatic Library has
compiled/minified Resources, but the resulting files are not added to
the distribution as a result of a human error, the files will not be
served.
At the moment of Resource instantiation it is not known whether the
NeededResources configuration has the 'compile' and or 'minified'
settings.
There is not a lot we can do about this. For minified resources, we
could jump through all kind of hoops and use the original source file
in a last attempt to do "the right thing", but for compiled resources
that is not an option.
My suggestion is to do nothing to correct the human error; the person
who deployed the application can found out about the missing resources
in the server access logs.

2. In earlier discussions about compilers, we talked about storing the
information about the compiler (name and version) that was used to
compile a resource near the output file (in a manifest file or in the
filename), in order to detect that a new version of the compiler or a
compiler with a different name is now active and fanstatic should
recompile the resource. I think we can drop this functionality, the
"fanstatic-compile" script is handy in case of changes in the compiler
backends.

3. Should we compile the resources only for the packages that are
installed in development mode? In the fanstatic.registry code we do
something similar:

https://bitbucket.org/fanstatic/fanstatic/src/35e5f8b9d6e65ed05f4d26a1f2b1ff41f7a3ec9b/fanstatic/registry.py?at=default#cl-36


I am really excited about this feature and think we are nearing a
release! I am particularly happy about the feature of using both a
compiler and a minifier for a Resource, and to see that these steps
are executed in the correct order. And I am also happy that the
resource compilation feature doesn't need any changes in other
fanstatic features such as bundling.

Next steps:

1. fix on python3
2. @Martijn, could you have a look at the changes on the branch?
3. merge to default and release.

Cheers,

JJ

[Wolfgang Schnerring]

Hello,

> > I opted to not declare the dependencies in fanstatic's setup.py at
> > all, i.e. not do a fanstatic[compilers] extra or something, since
> > a) many compilers require external binaries (node.js-based and so on)
> > which we couldn't declare anyway and b) collecting all possible
> > dependencies under one extra feels wrong (wouldn't want to force *all*
> > on somebody who just wants to use one), so we might as well leave it
> > off completely and tell people to require Python packages for
> > compilers "downstream", that is in their application.
>
> Sounds very reasonable. We could use extras_require in setup.py:
>
> extras_require = {
> 'cssmin': ['cssmin'],
> 'jsmin': ['jsmin'],
> }
>
> Where the keys are the names of minifiers/compilers and the values are
> list of python packages. However, this would only work for python
> packages, not for node/java dependencies.

Yes, the non-Python dependencies is why I didn't bother doing extras_require at
all, since I figured, people will need to take care of their dependencies
themselves anyway. But I'd be fine with extras, too.

> @Wolfgang, in [a] you hook into sdist_compile; previously we discussed
> hooking into zest.releaser. It seems the sdist_compile hook is more
> versatile than a zest.releaser pre-release hook. This is a very cool
> hack!

Yes, well, for one thing, zest.releaser is somewhat proprietary, as opposed to
setuptools, so I thought, hey, let's try whether I can do something more
general. And since it wasn't too hard (nor /too/ ugly) I did that, and didn't
spend a second thought on zest.releaser.
I do hope this really works; I don't think I've hacked too much or used stuff
that's is actually setuptools-internal and subject to breakage.

> I took the liberty of fixing/adding some tests on the wosc-compilers
> branch and added these features:
>
> 1. Added google closure compiler as a minifier

> 2. When compiling a resource, write a log record of the Resource being
> compiled and the time spent.

> 3. If you don't want the sources of your Resources to be publishable,
> you can set your library 'ignores' setting to the globs of the sources
> files; I have added this to the documentation.

Superb, thanks!

> 1. It is good practice not to do compilation on a production system.
> When deploying an app to production, the compiler backend need not be
> available. The NeededResources 'compile' flag determines whether
> resources need to be compiled. If a fanstatic Library has
> compiled/minified Resources, but the resulting files are not added to
> the distribution as a result of a human error, the files will not be
> served.

If the compiler.available property returns False (i.e. some dependency is
missing) and the file that the compiler (or minifier) would generate is missing,
the existing code should raise an error. At least that was my intent while
programming it.

> 2. In earlier discussions about compilers, we talked about storing the
> information about the compiler (name and version) that was used to
> compile a resource near the output file (in a manifest file or in the
> filename), in order to detect that a new version of the compiler or a
> compiler with a different name is now active and fanstatic should
> recompile the resource. I think we can drop this functionality, the
> "fanstatic-compile" script is handy in case of changes in the compiler
> backends.

I also think that would be overkill. If the user incompatibly upgrades the
compiler, the burden of manually recompiling things once is not too great, I
think.

> 3. Should we compile the resources only for the packages that are
> installed in development mode?

Humm. I don't have much of an opinion on this. I guess trying to compile files
inside eggs that are installed somewhere would be a bit of a violation of
the spirit of things. So that guard would probably be a good idea.
On the other hand, the Resource/Library at the moment doesn't know anything
about its package, and I'm not sure it's worth introducing all that just so we
can implement this safety belt.

> 1. fix on python3

I thought I took care to keep it py3-compatible. Sorry if I missed something.

> 2. @Martijn, could you have a look at the changes on the branch?

Yes, another set of eyes to look over this would be much appreciated.

> 3. merge to default and release.

Yayy! :)

Wolfgang

[Martijn Faassen]

Hey everybody,

> 2. @Martijn, could you have a look at the changes on the branch?

Yes, another set of eyes to look over this would be much appreciated.

Real quick: I'm a bit busy these coming weeks. Doing Obviel work on Ibiza this week. :)

The good news is that I might be doing a bit of customer work with Fanstatic in a couple of weeks so can spend a bit of time on it. Don't wait for my review for a release though - I can always review
after a release and fix things (if necessary!) then.

I'm really happy this feature is finally landing, it's really really cool. I realized recently that Fanstatic has a unique selling point compared to the client-side tools, as it can also work for CSS dependencies - I understand it that this is hard to in a client-side library.

I think an interesting exploration would be to look into asynchronous module definition and such and see whether we can write some code to automatically create resources for JS packages that use it. We should also look at tools like bower which can be used to pull in JS libraries, using git/github as a kind of PyPI. How could we leverage bower in the creation of Fanstatic packages? Could we also create "virtual" python modules on top of existing JS libraries so we can import resources from them without actually having to install a package?

Regards,

Martijn

[Wolfgang Schnerring]

Hi,

> Real quick: I'm a bit busy these coming weeks. Doing Obviel work on Ibiza
> this week. :)
> The good news is that I might be doing a bit of customer work with
> Fanstatic in a couple of weeks so can spend a bit of time on it. Don't wait
> for my review for a release though - I can always review
> after a release and fix things (if necessary!) then.

That sounds good! :)

> I think an interesting exploration would be to look into asynchronous
> module definition and such and see whether we can write some code to
> automatically create resources for JS packages that use it. We should also
> look at tools like bower which can be used to pull in JS libraries, using
> git/github as a kind of PyPI. How could we leverage bower in the creation
> of Fanstatic packages? Could we also create "virtual" python modules on top
> of existing JS libraries so we can import resources from them without
> actually having to install a package?

Yes! I've long wondered whether/how Fanstatic and AMD/require.js/etc. overlap
and/or might work together. I have this vague notion that they operate in a
similar problem space, so there should be some cool opportunity there, but I
have no clue at all how they might integrate. I haven't really used require.js
yet, either, so all thoughts have been theoretical in nature so far.

Wolfgang

[Jan-Jaap Driessen]

On Mon, Apr 15, 2013 at 3:56 PM, Wolfgang Schnerring <w...@gocept.com> wrote:
> Yes, the non-Python dependencies is why I didn't bother doing extras_require at
> all, since I figured, people will need to take care of their dependencies
> themselves anyway. But I'd be fine with extras, too.

I have added the extras and added the notation to the docs.

>> @Wolfgang, in [a] you hook into sdist_compile; previously we discussed
>> hooking into zest.releaser. It seems the sdist_compile hook is more
>> versatile than a zest.releaser pre-release hook. This is a very cool
>> hack!
>
> Yes, well, for one thing, zest.releaser is somewhat proprietary, as opposed to
> setuptools, so I thought, hey, let's try whether I can do something more
> general. And since it wasn't too hard (nor /too/ ugly) I did that, and didn't
> spend a second thought on zest.releaser.
> I do hope this really works; I don't think I've hacked too much or used stuff
> that's is actually setuptools-internal and subject to breakage.

This setup works like a charm on my projects. However, in order to
have my package "foo" work with the "fanstatic.sdist_compile"
cmdclass, I need to import fanstatic in the setup.py of my package.
As far as I can see, this is not going to work in situations with a
python interpreter with a "clean" environment, such as when using
buildout.
I tried some ways to circumvent this, the least amount of work is to
copy the implementation of fanstatic.sdist_compile to the setup.py of
my foo project and work with late imports of fanstatic...
Any ideas? Am I doing something wrong here?

>> 1. It is good practice not to do compilation on a production system.
>> When deploying an app to production, the compiler backend need not be
>> available. The NeededResources 'compile' flag determines whether
>> resources need to be compiled. If a fanstatic Library has
>> compiled/minified Resources, but the resulting files are not added to
>> the distribution as a result of a human error, the files will not be
>> served.
>
> If the compiler.available property returns False (i.e. some dependency is
> missing) and the file that the compiler (or minifier) would generate is missing,
> the existing code should raise an error. At least that was my intent while
> programming it.

My bad: I misread the code and mis-tested this. In "production" with
no compiler backends, I now see the Resource warnings.

>> 3. Should we compile the resources only for the packages that are
>> installed in development mode?
>
> Humm. I don't have much of an opinion on this. I guess trying to compile files
> inside eggs that are installed somewhere would be a bit of a violation of
> the spirit of things. So that guard would probably be a good idea.
> On the other hand, the Resource/Library at the moment doesn't know anything
> about its package, and I'm not sure it's worth introducing all that just so we
> can implement this safety belt.

Implemented this.

>> 1. fix on python3
>
> I thought I took care to keep it py3-compatible. Sorry if I missed something.

This was also a mental note to myself. I have run all tests and fixed
them. Long live tox! All pythons are happy:

https://travis-ci.org/fanstatic/fanstatic

>> 3. merge to default and release.

Boom!

https://pypi.python.org/pypi/fanstatic/1.0a

www.fanstatic.org is also updated to the 1.0a docs.

Thank you Wolfgang!

--
JJ

[Wolfgang Schnerring]

> >> @Wolfgang, in [a] you hook into sdist_compile; previously we discussed
> >> hooking into zest.releaser. It seems the sdist_compile hook is more
> >> versatile than a zest.releaser pre-release hook. This is a very cool
> >> hack!

> > I do hope this really works; I don't think I've hacked too much or used
> > stuff that's is actually setuptools-internal and subject to breakage.
>
> This setup works like a charm on my projects. However, in order to
> have my package "foo" work with the "fanstatic.sdist_compile"
> cmdclass, I need to import fanstatic in the setup.py of my package.
> As far as I can see, this is not going to work in situations with a
> python interpreter with a "clean" environment, such as when using
> buildout.

There is ``setup_requires`` which if I understand it correctly, should cause
setuptools to download/install the dependencies listed in there "while"
processing the setup.py. However, I haven't used this before, and the
documentation[1] is a little vague ("setuptools will attempt to obtain
these, even going so far as to download them using easy_install, before
processing the rest of the setup script or commands.") -- I don't know how that
relates to "I need to import that stuff before calling setup()".

Wolfgang

[1] http://peak.telecommunity.com/DevCenter/setuptools

[Jan-Jaap Driessen]

On Thu, Apr 18, 2013 at 8:15 AM, Wolfgang Schnerring <w...@gocept.com> wrote:
>> This setup works like a charm on my projects. However, in order to
>> have my package "foo" work with the "fanstatic.sdist_compile"
>> cmdclass, I need to import fanstatic in the setup.py of my package.
>> As far as I can see, this is not going to work in situations with a
>> python interpreter with a "clean" environment, such as when using
>> buildout.
>
> There is ``setup_requires`` which if I understand it correctly, should cause
> setuptools to download/install the dependencies listed in there "while"
> processing the setup.py. However, I haven't used this before, and the
> documentation[1] is a little vague ("setuptools will attempt to obtain
> these, even going so far as to download them using easy_install, before
> processing the rest of the setup script or commands.") -- I don't know how that
> relates to "I need to import that stuff before calling setup()".

The setup.py of my package 'foo' looks like this:

"""
from setuptools import setup
setup(
name='foo',
...
cmdclass={'sdist': fanstatic.sdist_compile},
...
)
"""

The cmdclass['sdist'] needs to be a class, not a string identifier to
an entry point, which is later looked up.
At this point you need to have fanstatic imported; A plain "import
fanstatic" gives an import error, as I am working with a "clean"
python.
A workaround is to try to import fanstatic and catch the importerror:

"""
import setuptools
try:
import fanstatic
except ImportError:
pass

setup(
name='foo',
...
cmdclass={'sdist': fanstatic.sdist_compile},
...
)
"""

This works on my machine. Is this the way to go? Shall we put this in the docs?

[Wolfgang Schnerring]

Hello,

> The cmdclass['sdist'] needs to be a class, not a string identifier to
> an entry point, which is later looked up.
> At this point you need to have fanstatic imported; A plain "import
> fanstatic" gives an import error, as I am working with a "clean"
> python.

Damn. It seems there isn't much we can do here, so we probably want to
document something like this:

try:
import fanstatic
cmdclass = {'sdist': fanstatic.sdist_compile}
except ImportError:
cmdclass = {}

setup(
...
cmdclass=cmdclass,
...
)

I had hoped for something more elegant. :-(

Wolfgang

[Martijn Faassen]

Hi there,

I'm vague on the details, but this cmdclass is a way to compile stuff upon installation time, right?

Instead could we shift the compilation phase of all registered packages to startup time instead?

Regards,

Martijn

[Wolfgang Schnerring]

Hi,

> I'm vague on the details, but this cmdclass is a way to compile stuff upon
> installation time, right?
> Instead could we shift the compilation phase of all registered packages to
> startup time instead?

That wouldn't help, since the problem this is trying to solve is:
Serve compiled assets on a system where the compiler is not available (it is
for example thought to be likely that production systems won't have node.js
installed).

Wolfgang

[Martijn Faassen]

Hi there,

I understand, cmdclass is needed to generate compiled assets into a release tarball. I can't see a way around that.

And the problem is that fanstatic may not be available when setup.py is being run. As long as it's available when doing a release, it's okay.

But shouldn't then we have a boilerplate that fails hard if you try to do a release *without* fanstatic available?

Regards,

Martijn

[Wolfgang Schnerring]

Hello,

> I understand, cmdclass is needed to generate compiled assets into a release
> tarball. I can't see a way around that.
> And the problem is that fanstatic may not be available when setup.py is
> being run. As long as it's available when doing a release, it's okay.
> But shouldn't then we have a boilerplate that fails hard if you try to do a
> release *without* fanstatic available?

You're totally right. So I guess we're back to where I started it: simply
"import fanstatic" in the setup.py, which will break hard if it isn't available.

I'm still miffed about the decided non-elegance of this whole business, but I
guess trying to fix (yet another) Python packaging machinery shortcoming is
more than out of scope for us. :)

Wolfgang

[Martijn Faassen]

Hi there,

It depends on whether 'import fanstatic' is going to be triggered by something before fanstatic is installed, for instance by buildout. If so, we have to catch it and do nothing. Perhaps there is no good way. :)

Regards,

Martijn