The case for binary assets
Arno Hautala
From what little I've seen, it looks like assets will be dropped, or
at least merged with pages and binary assets dropped. The reason (one
of them?) being that filtering of binary assets isn't common or a task
that nanoc needs to be concerned with.
Stop me if I've got the asset situation all wrong, in which case, the
video stuff that comes later is still cool.
Granted, the only filters I've seen for binary assets have been
regarding galleries and thumbnail creation. I haven't even gone that
far and my binary asset usage has been limited to simply copying the
few images to their destinations.[1] Beyond binary assets, I have
made use of embedded ruby to set color variables in my main style
sheet.
However, I saw a link the other day that I realized presents an ideal
case for binary asset filtering. VideoForEverybody [2] specifies a
creative use of the HTML5 video tag to provide media to any browser.
Depending on the capabilities of the browser, the code falls back from
Ogg to h264, Flash, QuickTime, Windows Media, and finally an offer to
download the media. The code is pretty generic and could be easily
populated via a script. Filtering with nanoc would allow the asset to
be specified, encoded to the other formats, and inserted into the
output page with the appropriate template.
I suppose this could all be accomplished with a script which performs
the encoding and template generation ahead of time. I still like the
idea of keeping the output directory clean and separate from
uncompiled assets. But perhaps this is solved by a different approach
to asset management?
A lot of this, for me, is brought on by the uncertainty of what
capabilities Nanoc 3 will bring for assets. There are closed bugs for
removing binary asset support, but also entries for removing the
differences between pages and assets, with binary assets still
referenced. Would the procedure I listed above still be possible? Or
would binary assets be best handled outside of site compiling by a
different task?
I also understand that the features of the next major version may not
be entirely defined and any clarification is appreciated.
Thanks,
--Arno
[1] As these images are just text, it occurs to me that I could
generate the images with an ImageMagick filter.
[2] http://camendesign.com/code/video_for_everybody
--
arno s hautala /-\ ar...@alum.wpi.edu
pgp eabb6fe6 d47c500f b2458f5d a7cc7abb f81c4e00
Denis Defreyne
On 03 Jul 2009, at 06:15, Arno Hautala wrote:
> From what little I've seen, it looks like assets will be dropped, or
> at least merged with pages and binary assets dropped.
Textual assets and pages have indeed been merged and are now known as
simply "items" (represented by Nanoc3::Item). Textual assets are,
after all, very similar to pages.
Binary assets are no longer supported. (This is not a final and
definitive decision; if it turns out to be worth readding support for
binary assets then I likely will.)
> The reason (one of them?) being that filtering of binary assets
> isn't common or a task that nanoc needs to be concerned with.
That is pretty much correct.
> Granted, the only filters I've seen for binary assets have been
> regarding galleries and thumbnail creation. I haven't even gone that
> far and my binary asset usage has been limited to simply copying the
> few images to their destinations.[1]
I personally don't (didn't) use binary assets that were simply copied
to the output directory. Copying binary assets (especially large ones)
takes time, and I find it better to simply drop the binary assets
directly in the output directory instead. This works quite well when
combined with version control, since it removes the danger of
accidentally removing the entire output directory. The wiki has a page
describing some ways of doing this; check http://projects.stoneship.org/trac/nanoc/wiki/Tips/CopyingAssetsIntelligently
for details.
> Beyond binary assets, I have made use of embedded ruby to set color
> variables in my main style sheet.
It is a good idea to let nanoc manage textual assets such as
stylesheets and JavaScript files; there are several interesting
filters that can be applied to them. Simple color variable replacement
is one good use, but there is also Sass and LESS for generating CSS,
RainPress for reducing the size of generated CSS, and minify for
reducing the size of JavaScript scripts.
(nanoc has a 'sass' and a 'rainpress' filter built-in; for LESS
support check out http://projects.stoneship.org/trac/nanoc/wiki/Plugins/LESS
. There is no minify filter yet, but I did find a Ruby implementation
at http://www.crockford.com/javascript/jsmin.rb—perhaps turning this
into a filter would be a good idea.)
> However, I saw a link the other day that I realized presents an
> ideal case for binary asset filtering. VideoForEverybody [2]
> specifies a creative use of the HTML5 video tag to provide media to
> any browser. Depending on the capabilities of the browser, the code
> falls back from Ogg to h264, Flash, QuickTime, Windows Media, and
> finally an offer to download the media. The code is pretty generic
> and could be easily populated via a script. Filtering with nanoc
> would allow the asset to be specified, encoded to the other formats,
> and inserted into the output page with the appropriate template.
Yep, I've seen Video For Everybody, and I'm as excited about HTML5
video as you are.
While playing around with creating different representations of the
same video in different formats (H.264, Theora, …) I've come to the
conclusion that there is no easy way to automate conversions between
different formats. There are a few reasons for this:
* There are many formats and it may not be possible to directly
transcode between them (and may therefore need an intermediary
transcoding step).
* There are a great deal of decoding/encoding parameters involved.
ffmpeg2theora, for example, has options for changing gamma,
saturation, brightness, contrast, sharpness, start/stop time,
deinterlacing, FPS, removing audio, etc. A decent nanoc binary asset
filter would have to support a great deal of these options.
* There is a great variety of software on the market (both open-
source and commercial) for transcoding audio, video and images. nanoc
would have to support at least the most popular transcoding
applications (assuming they can be automated in Ruby).
* There are likely already existing software packages that handle
automated asset management such as transcoding a lot better than nanoc
ever will. I have to admit I barely have any experience with such
packages, though…
> I suppose this could all be accomplished with a script which
> performs the encoding and template generation ahead of time.
That is pretty much what I'm thinking… giving nanoc support for binary
assets does not give nanoc extra functionality that cannot be
recreated in a couple of small scripts.
For example, it would be possible to simply ignore all the complexity
mentioned and only give nanoc some basic binary assets support.
Filters could then map onto commandline executables directly, e.g.
item '/video/blah/', :binary => true do |i|
i.filter :ffmpeg2theora, :videoquality => '9'
end
… would become (with --output automatically derived from the mapping
rules used by nanoc 3):
ffmpeg2theora --videoquality 9 --output ./output/video/blah.ogv
But then I wonder what advantage nanoc provides over alternative
approaches such as rake tasks or shell scripts that simply invoke the
'ffmpeg2theora' executable directly. Such rake tasks or shell scripts
would, I believe, be more flexible and more straightforward to use.
This does not only apply to video, but also to audio, images, and so on.
> I still like the idea of keeping the output directory clean and
> separate from uncompiled assets. But perhaps this is solved by a
> different approach to asset management?
I personally don't mind putting precompiled assets directly in the
output directory, but I understand that not everyone is willing to do
this. You may find the CopyingAssetsIntelligently wiki page, which I
mentioned above, useful.
> A lot of this, for me, is brought on by the uncertainty of what
> capabilities Nanoc 3 will bring for assets.
As I see it, there will be no support for binary assets; only textual
assets will be supported. I have not ruled out support for binary
assets, though, but I'll need convincing arguments in favour of binary
assets for it to come back… :)
> There are closed bugs for removing binary asset support, but also
> entries for removing the differences between pages and assets, with
> binary assets still referenced.
Ticket #46 ("Remove difference between assets and pages", http://projects.stoneship.org/trac/nanoc/ticket/46)
did indeed still reference binary assets. I've updated the
description of the ticket to prevent future confusion (the ticket was
created when binary assets were still supported).
> Would the procedure I listed above still be possible? Or would
> binary assets be best handled outside of site compiling by a
> different task?
I recommend that binary assets be handled by a separate task or
script. You may also find asset-compiler (http://asset-compiler.rubyforge.org/
) to be useful; from what I've seen it is rather limited, though.
> I also understand that the features of the next major version may
> not be entirely defined and any clarification is appreciated.
I should probably start working on a "What's New in nanoc 3" series of
posts on the nanoc site, just like I did with nanoc 2.1. For the time
being, here's a short list of changes:
* Support for binary assets has been removed (in case you forgot :).
* Support for drafts has been removed. nanoc would ignore items
which has the "is_draft" attribute set to true. I may readd this,
though; it would be fairly trivial (simply let the data source not
return such items… perhaps a line or two of extra code).
* Data sources' save_*, delete_*, move_* methods have been removed.
They were never used and I didn't even bother fully implementing them
in the filesystem and filesystem_combined data sources. The idea was
to give data sources the four CRUD operations so that a GUI on top of
nanoc could then use the data sources for managing the data… but there
are no such GUIs for nanoc yet and I doubt there ever will be.
* Support for templates has been removed. I rarely used this
myself; I found it to be easier to simply copy and existing page and
use that as template instead. This can easily be replaced with a rake
task that loads the site and then calls the data source's
#create_item(content, attributes, identifier) method.
* There is a DSL for specifying rules that tell nanoc how items
should be mapped and compiled, stored in a file called 'Rules' in the
site directory. This is described in http://projects.stoneship.org/trac/nanoc/wiki/Dev/Nanoc3DSL
and you can find a great example at http://projects.stoneship.org/hg/sites-moul/file/205fb9bf0dea/Rules
.
* Because of this DSL, the no longer useful old-style routers
(Nanoc::Router subclasses) have been removed, and processing
instructions (filters_pre, filters_post, layout, …) are no longer
stored in the item's meta file.
* Page and asset defaults (asset_defaults.yaml, page_defaults.yaml)
has been removed. They were only really useful for specifying the
default list of filters, the default layout, etc, but the new DSL
pretty much makes said asset and page defaults obsolete.
* Proxies have been removed. The @page variable in nanoc 2.x is not
a Nanoc::Page instance, but it is a Nanoc::PageProxy that allowed
writing @page.title instead of @page.attribute_named(:title). In nanoc
3.x, @page is now known as @item and @item is an actual Nanoc3::Item
instance. Requesting an attribute is done by calling #[], e.g.
@item[:title] (@item.title won't work anymore). Proxies were a real
pain, especially when debugging, and I'm glad that they're finally gone.
* Filter arguments, so you can pass options when calling the
filter, e.g. "item.filter :haml, :format => :html5". :format here is a
Haml-specific option (see the bottom of http://haml.hamptoncatlin.com/docs/rdoc/classes/Haml.html)
.
* A 'filter' helper that allows parts of a page to be filtered.
This is especially useful in combination with the new coderay filter,
because now you can write stuff like this in order to syntax-highlight
a piece of code:
<% filter :coderay, :language => 'ruby' do %>
#!/usr/bin/env ruby
puts "Hello World!"
<% end %>
* Dependency tracking between items and layouts. This eliminates
the need for passing --all to 'nanoc co', because nanoc will
automatically figure out which items are outdated and need
recompiling. For example: if item A includes the content of item B,
which in turn includes content from item C, and if page C is updated,
then nanoc will know that page B and page A will have to be recompiled
as well. (This is, at the moment, still a bit buggy.)
There's probably more, but those are the most significant changes in
nanoc 3.x. As you can see, this is going to be a big release with a
lot of changes (and no backward compatibility, but think of that as a
good thing). It is slowly nearing completion, but there are still a
few unimplemented features in the pipe line (multiple data sources and
data preprocessing, to be more precise).
I hope this answer your question! :)
Regards,
Denis
--
Denis Defreyne
denis.d...@stoneship.org
Cheba
and binary? What's the essential difference between those two?
Denis Defreyne
> May I ask why assets (or items in general) should be separated to
> text and binary? What's the essential difference between those two?
Textual assets are loaded into memory so that their content is a
String object, while binary assets are File objects. Loading binary
assets in memory is impossible because that would require huge amounts
of memory, and most binary filters operate on files (or paths) instead
of the actual content.
Alexander Mankuta
source. Be it file or string it can be accessed through IO interface.
Item#content will just read the IO and return string (text or binary).
Anyway, filter can make it's own decision if it needs string, IO
object or a file. Strings can be easily saved to temp files
transparently if this is needed for filter and data is not present on
the fs. Also this may make network sources a bit easier to implement
because sockets are IOs to.
Denis Defreyne
> I believe it's not that hard to make items hold an IO object of the
> source.
Hi,
You are right that using an IO object would work; this is after all
the way nanoc 2.x handles binary assets (take a look at the
AssetRep#compile_binary method in lib/nanoc/base/asset_rep.rb to see
how it works).
The problem is not that it's hard to implement (which it isn't), but
that there are already existing solutions out there that I'd prefer to
reuse (asset-compiler [1], for instance), rather than write my own
implementation. Most asset-modifying functionality would have to be
implemented using external executables since not many Ruby APIs for
such functionality exist. It would make more sense to use those
external executables directly and use rake to automate everything.
For example, the most common task (for me, at least) for binary assets
is to build thumbnails out of large images. This is pretty much what
asset-compiler does; check out this asset-compiler example [2] and
you'll see that it does that job pretty well.
Regards,
--
Denis Defreyne
denis.d...@stoneship.org
[1]: http://rubyforge.org/projects/asset-compiler/
[2]: http://www.rubyinside.com/asset-management-with-ruby-dsls-10.html
Arno Hautala
Defreyne<denis.d...@stoneship.org> wrote:
>
> For example, it would be possible to simply ignore all the complexity
> mentioned and only give nanoc some basic binary assets support.
> Filters could then map onto commandline executables directly, e.g.
> item '/video/blah/', :binary => true do |i|
> i.filter :ffmpeg2theora, :videoquality => '9'
> end
> … would become (with --output automatically derived from the mapping
> rules used by nanoc 3):
> ffmpeg2theora --videoquality 9 --output ./output/video/blah.ogv
This is actually what I think I had in mind. Provide some minimal
amount of support for processing generic files. Nanoc needn't worry
about handling every binary type, just provide the hooks, and let the
user or Nanoc community provide the rest.
> I personally don't mind putting precompiled assets directly in the
> output directory, but I understand that not everyone is willing to do
> this. You may find the CopyingAssetsIntelligently wiki page, which I
> mentioned above, useful.
I'll be certain to check it, and asset-compiler, etc., out.
> I personally don't (didn't) use binary assets that were simply copied
> to the output directory. Copying binary assets (especially large ones)
> takes time, and I find it better to simply drop the binary assets
> directly in the output directory instead. This works quite well when
> combined with version control, since it removes the danger of
> accidentally removing the entire output directory.
One way to speed the copying process would be to check the inputs and
outputs before compiling. I'm not sure, but it seems like each asset
is compiled regardless of whether the input has been altered or not
and only copied if the result differs from the existing file. Is that
right? Dependency tracking would be necessary to prevent this, to
account for changed templates that would affect every page of that
type. Basic checking could check the timestamp of the input against
the output, or in an extreme case, track hashes of inputs. I've been
meaning to version control my work anyway and should really have a
look at that.
Thanks,
--Arno
Denis Defreyne
After a few days of discussions about binary assets (including
discussions on the IRC channel) I've come to the conclusion that I am
wrong, and that binary assets should indeed be handled by nanoc. There
are two reasons why I've changed my mind:
* Supporting binary assets in nanoc would allow arbitrary metadata
to be given to said assets, which can then be used in nanoc pages. ALT
text for images and captions for media items jump to mind, but there
are more ways of using such metadata.
* With dependency tracking, binary assets should not have to be
needlessly recompiled, speeding up site compilation a great deal. This
was a big issue (for me at least) in nanoc 2.x because it required
using --force (or --all) quite often.
nanoc 2.x's handling of binary assets was less than satisfactory. An
implementation of binary assets in nanoc 3 would have to be different.
There are several aspects about nanoc 2.x's binary asset handling that
I don't like and which should therefore be fixed or solved in a
different way in nanoc 3.
* Using separate directories for textual/binary assets and pages
was not a good idea. There is, as I see it, no real difference between
pages and textual assets, which is the reason why the two were merged
in nanoc 3.
* I am not sure if using a boolean attribute "binary" (which
indicates whether the item content should be treated as a stream or a
string) is the way to go. In nanoc 3, all item metadata is custom;
there are no predefined attributes anymore (such as nanoc 2.x'
filters_pre etc). A possible solution would be to treat everything as
stream by default, and specify a list of file extensions for files
that should be treated as text (.html, .txt, …) but I assume this
won't necessarily work for all data sources. Ideas are welcome.
* How should binary filters differ from textual filters? In nanoc
2.x, binary filters take a stream (File object, more specifically) as
argument, but I think it may be better to supply a filename instead;
some filters, such as those that delegate their functionality to
commandline tools, don't need a stream object. In fact, creating a
stream object has only drawbacks since it takes up unused system
resources (file descriptors).
I imagine that there are more issues like these. Letting nanoc support
binary items is certainly not a trivial task, so suggestions and ideas
are more than welcome.
One of the biggest drawbacks of nanoc 2.x's binary asset handling is
indeed the lack of dependency tracking. nanoc 3's dependency tracker
(while still buggy) speeds up compilation a lot by preventing
unchanged items from being needlessy recompiled.
On 09 Jul 2009, at 05:18, Arno Hautala wrote:
> I'm not sure, but it seems like each asset is compiled regardless of
> whether the input has been altered or not and only copied if the
> result differs from the existing file. Is that right? Dependency
> tracking would be necessary to prevent this, to account for changed
> templates that would affect every page of that type. Basic checking
> could check the timestamp of the input against the output, or in an
> extreme case, track hashes of inputs.
nanoc 2.1 keeps track of file modification times (mtimes) to determine
whether pages and assets should be recompiled. Unfortunately, its
dependency tracking is very limited:
1. It does not understaind dependency chains: for example, if page
A includes page B content which in turn uses page C content, then only
page B will be recompiled when page C changes, even though page A
should have been recompiled as well.
2. It does not track dependencies between attributes; for example,
if page A includes the value of an attribute of page B, then page A
should be recompiled when the attributes of page B changes.
nanoc 2.2 pretty much disables the dependency tracker because it is
fairly useless anyway. Perhaps this wasn't such a bright idea; it
makes site compilation quite slow after all, but at least there is no
need for using --force/--all so often anymore.
The two problems mentioned above have been resolved in nanoc 3's much
more powerful dependency tracker. It still has some bugs which require
using --force/--all once in a while, and it does not yet keep track of
layouts (which causes some pages to be recompiled even though it is
not necessary). I hope to have these issues resolved soon, though.
Denis Defreyne
I have decided not to include binary assets in nanoc 3.0, but it will
likely make it to nanoc 3.1 instead. I'd like to release nanoc 3.0 as
soon as possible, and adding support for binary assets will only shift
the release date back.
Denis Defreyne
On 27 Jul 2009, at 12:29, I wrote:
> I have decided not to include binary assets in nanoc 3.0, but it
> will likely make it to nanoc 3.1 instead. I'd like to release nanoc
> 3.0 as soon as possible, and adding support for binary assets will
> only shift the release date back.
Some of you have wondered whether this means you can still use images
etc. with nanoc. The answer is yes, you can, but these binary assets
won't be handled by nanoc itself. There are a few ways to deal with
binary assets in nanoc 3.0:
=== If you don't need filtering
* You can place the binary assets directly in the `output` directory.
This is the most basic approach but it works quite well; I use it
regularly. Just be sure not to delete the `output` directory or you'll
lose your assets (I recommend adding them to version control).
* You can create a directory whose contents will be copied into the
`output` directory. This can be automated with a rake task (named
"copy_assets" or so). I prefer doing this with rsync, which is rather
fast. Also see http://projects.stoneship.org/trac/nanoc/wiki/Tips/CopyingAssetsIntelligently
.
=== If you need filtering
* You can filter binary assets manually (in a rake task that uses
RMagick or ImageScience or whatever) or use asset-compiler (see http://rubyforge.org/projects/asset-compiler/)
.
* You can wait for nanoc 3.1 :). However, it may still be a while
before 3.1 is out.
Hope this helps,