Binary items: what, why, and how: a brainstorming thread

40 views
Skip to first unread message

Denis Defreyne

unread,
Aug 19, 2009, 2:14:22 PM8/19/09
to na...@googlegroups.com
Hi.

As you may already know, nanoc 3.0 has no support for managing or
filtering binary assets. nanoc 2.1 and 2.2 did, but I don't like its
handling of binary assets well.

So I'm looking for ideas for binary asset handling in nanoc 3! If you
have opinions on how the next release of nanoc should handle binary
assets, share them! There are three questions I would like you to
answer:

* WHY do you want binary asset handling in nanoc 3? Or why not?
Perhaps this question has already been answered a few times, but maybe
some new and interesting answers will turn up anyway.

* WHAT do you want nanoc to do, exactly? What do you _need_ nanoc to
do? What features would you benefit from, make your life easier? What
do you definitely _not_ want nanoc to do? Share some uses cases for
binary assets--I've given a few of my own at the bottom of this post
to kick-start the thread.

* HOW do you want nanoc to handle binary assets? Similar to textual
items, i.e. being able to filter them with (binary) filters? Would
binary layouts make sense (e.g. putting a frame around a picture)?
Should binary assets be applicable to textual items as well? Should
filters apply to only a single "class" of items (image, sound, video,
text, …)?

* * *

Some use cases:

* Being able to generate thumbnails of images. For instance, my
personal web sites has a small gallery of screenshots; the Myst Online
web site also has a screenshot gallery. These are not managed by nanoc
and therefore live in the output directory (under version control,
though). I'd like to make them proper binary assets so I can assign
titles to them and perhaps the name of the location depicted by the
screenshot, and of course so that I can generate the thumbnails
automatically.

* Being able to automatically convert a video into several formats,
e.g. Ogg Theora and H.264 so they can be used in HTML5 <video>
elements. This way, you'd only need a single input file, type "nanoc
co" and not worry about having the right video format for every
browser. One way of achieving this could be by using ffmpeg2theora.

* Being able to assign custom metadata to binary files even though
they aren't filtered. For example, a musician could have a large
collection of music clips (in MP3 format) and each one would have a
track name, album name, track number, a list of links where to buy the
album or track, etc. This information could then be used on pages
generated for each album, which then contain a track preview along
with track metadata.

So, if you have ideas or opinions, please share!

Regards,

Denis

--
Denis Defreyne
denis.d...@stoneship.org

Denis Defreyne

unread,
Aug 19, 2009, 2:23:10 PM8/19/09
to na...@googlegroups.com
Hi.

Let me add a few more aspects of binary asset handling that I (and
others) consider important:

* Dave Dribin said: "Putting [compiled binary assets] in the output
directory seems wrong. I should be able to nuke the build directory,
just as I do when compiling C." (source: http://twitter.com/ddribin/status/3409399171)
. I agree. With even the most basic support for binary assets (just
copying them), this should no longer be a problem.

* Dave Dribin also said: "Perhaps built-in support for "Copying assets
intelligently"?" (source: http://twitter.com/ddribin/status/
3409413892). This is probably the most basic form of binary asset
management, but it actually highlights a small but important
annoyance: every file needs a meta-file, which is not always
necessary. I'd like to be able to drop a hundred JPEGs in content/
assets/images, type "nanoc co" and see them pop up in output/assets/
images without having to create a meta-file for every single item.

Sören Nils 'chucker' Kuklau

unread,
Aug 19, 2009, 3:22:32 PM8/19/09
to nanoc
I shall preface this by saying that I don't yet use nanoc enough to
make a particularly good assessment.

On Aug 19, 8:14 pm, Denis Defreyne <denis.defre...@stoneship.org>
wrote:
> * WHY do you want binary asset handling in nanoc 3?

Two uses that would be interesting for me are thumbnails for an image
gallery, and checksums for a downloads section.

I'd like nanoc to discover/keep track of which assets have thumbnails/
checksums/metadata/whichever are "out of date" or missing, and
automatically (re-)generate those using custom-defined tasks. E.g., to
use your video example, if I were to drop a new source video in there,
it would take care of automatically generating targets in H.264 and
Theora, and if I were to modify an existing source video, it would
update those targets.

On Aug 19, 8:23 pm, Denis Defreyne <denis.defre...@stoneship.org>
wrote:
> * Dave Dribin said: "Putting [compiled binary assets] in the output
> directory seems wrong. I should be able to nuke the build directory,
> just as I do when compiling C." (source:http://twitter.com/ddribin/status/3409399171)
> . I agree. With even the most basic support for binary assets (just
> copying them), this should no longer be a problem.

This is great — the mix of generated and external files in output/
struck me as unclean.

Alexander Mankuta

unread,
Aug 19, 2009, 3:40:02 PM8/19/09
to na...@googlegroups.com
Why
In first place having binary assets is a logical move. This days web
site rarely consists only of text (html/css/js). Almost every modern
site has binary information that needs to be presented to user be it
graphics, audio, video or enything else.

What
As I see it nanoc doesn't have to do any heavy-lifting in here. I'm
pretty sure that there are not that many common use cases for binary
assets. Even such a common task as generating thumbnails can vary
greatly from case to case. That's why I'm proposing nanoc to implement
only API for building custom binary filters.

How
Having in mind that binary assets are going to be filtered just like
regular items it's probably logical to not create separate items for
binary assets but just allow users to use binary filters on any item.
I don't see any problems with this. I believe that common sense in
each of us will stop us creating image thumbnails from sass files.

Some use cases
Denis mentioned the html5 video. The very same arguments are
applicable to html5 audio also because as of now there's no one code
common among all browsers.
Another example is providing archive in alternate formats. Many
platform independent projects pack their source code in a few
archives, e.g. .zip for windows and .tar.bz2 for *nix. You can pretty
easy make a zip out of tar.bz2.

I see binary filer as a piece of code that can do something to item
rep. I say rep because in general filter can use external tools and in
some cases it's much more efficient to point the tool writing directly
to output dir rather than to temp file or stdout and move the data to
output dir. Item rep has the output file name in the raw_path
attribute. Also rep has access to the item. But I'd propose item to be
changed a bit. First it needs to know where from the data come. The is
needed for use with external tools. Also it, probably, needs a lazy
evaluated content property. Also it needs somehow to cache data in
temp file if filter prefers to use file rather than raw data and item
was generated or retrieved over then net. And one more thing that I'm
not really sure about. Item, probably, should provide a mime-type of
it's content so that filter could avoid doing stupid things. For
example, ImageMagick can eat arr your ram if you'll try to feed it
medium size video instead of raw image.

Brian Bowling

unread,
Aug 19, 2009, 11:00:05 PM8/19/09
to na...@googlegroups.com
Denis Defreyne wrote:
> Hi.
>
> <snip>

> * Dave Dribin also said: "Perhaps built-in support for "Copying assets
> intelligently"?" (source: http://twitter.com/ddribin/status/
> 3409413892). This is probably the most basic form of binary asset
> management, but it actually highlights a small but important
> annoyance: every file needs a meta-file, which is not always
> necessary. I'd like to be able to drop a hundred JPEGs in content/
> assets/images, type "nanoc co" and see them pop up in output/assets/
> images without having to create a meta-file for every single item.
>
>
+1
> Regards,
>
> Denis
>
>

Denis Defreyne

unread,
Aug 23, 2009, 4:07:40 AM8/23/09
to na...@googlegroups.com
Hi,

A while ago there was a discussion about how to determine whether an
item is a textual or a binary one. Here's another idea for determining
that.

In the past (nanoc 2.x), this was accomplished by setting a "binary:
true" attribute in the asset's attributes. I would rather not do this
for nanoc 3.x because attributes in nanoc 3.x are completely free-
form; there are no predefined attributes that get special treatment by
nanoc anymore.

Another way to determine whether a certain item should be considered
as "binary" could be the file extension. And, because it is safer to
assume a file is binary instead of textual, you'd define the list of
textual assets in the site configuration or perhaps the data source
configuration. For example:

data_sources:
- type: filesystem_compact
config:
textual_extensions:
- txt
- html
- erb
- rhtml
- md

This may not work for all data sources, though. Some data sources may
load their data from a location that does not provide information
about file extensions. That wouldn't be a problem though: such data
sources could then use the MIME type instead; their data source
configuration would then have a list of MIME types to consider
"textual":

data_sources:
- type: filesystem_compact
config:
textual_mime_types:
- text/*
- application/*+xml
- application/xml
- application/json

The list of textual extensions and textual MIME types could already be
filled in with well-known textual extensions and MIME-types, of
course. What do you think?

Denis Defreyne

unread,
Aug 23, 2009, 5:44:01 AM8/23/09
to na...@googlegroups.com
On 19 Aug 2009, at 20:23, Denis Defreyne wrote:

> I'd like to be able to drop a hundred JPEGs in content/assets/
> images, type "nanoc co" and see them pop up in output/assets/images
> without having to create a meta-file for every single item.

This is now implemented (for textual items, not for binary ones
because binary items aren't implemented yet).

JLF

unread,
Aug 23, 2009, 9:46:46 AM8/23/09
to nanoc
I do not know if it qualifies as 'binary' assets, but I would like to
have textual items processed into binary representations. For example,
it would allow the following tasks

- Generating PDFs from DocBook or LaTeX source.
- LaTeX helper able to process inline LaTeX code, generate an image
and replace the latex code in generated file with an appropriate <img>
tag.
- Produce a compressed representation of an item (the sitemap.xml for
instance) or a group of items.
- A chart/plotter filter taking a csv file and producing an image.
- ...

Denis Defreyne

unread,
Aug 23, 2009, 2:54:13 PM8/23/09
to na...@googlegroups.com

Hi,

Very good point! It should indeed be possible to go from textual to
binary, and probably the other way around as well. This also is
something missing from nanoc 2.x' handling of binary assets.

JLF

unread,
Aug 24, 2009, 6:04:03 AM8/24/09
to nanoc
On Aug 23, 8:54 pm, Denis Defreyne <denis.defre...@stoneship.org>
wrote:
> Very good point! It should indeed be possible to go from textual to  
> binary, and probably the other way around as well. This also is  
> something missing from nanoc 2.x' handling of binary assets.

Indeed, going from binary to textual could allow for instance to
extract exif data from jpegs, to improve gallery pages.

Denis Defreyne

unread,
Aug 24, 2009, 6:32:10 AM8/24/09
to na...@googlegroups.com
On 24 Aug 2009, at 12:04, JLF wrote:

> Indeed, going from binary to textual could allow for instance to
> extract exif data from jpegs, to improve gallery pages.

This is something I'd do in the preprocessing phase: parse the EXIF
headers and then generate extra attributes for the item.

Christopher

unread,
Aug 27, 2009, 11:13:49 PM8/27/09
to nanoc
(apologies if this sent twice, Google Groups ate it the first time)

@cbowns from the twitters here, thought I'd pipe in, since I just set
up my site with nanoc3 a couple weekends ago and had a few ideas on
this very topic.

On Aug 19, 11:14 am, Denis Defreyne <denis.defre...@stoneship.org>
wrote:
> * WHY do you want binary asset handling in nanoc 3? Or why not?
> Perhaps this question has already been answered a few times, but maybe
> some new and interesting answers will turn up anyway.

I have a few binary blobs (a couple source tarballs, a PDF, and a dmg)
that I want to "inject" into the site structure that nanoc is
generating. I've manually linked to those assets in pages around the
site.

> * WHAT do you want nanoc to do, exactly? What do you _need_ nanoc to
> do? What features would you benefit from, make your life easier? What
> do you definitely _not_ want nanoc to do? Share some uses cases for
> binary assets--I've given a few of my own at the bottom of this post
> to kick-start the thread.

My initial internal model of how I -thought- binary assets would work:

1. "Oh sweet, 'content', this must be where I put things! <put binary
stuff here"
2. "Hmmm, that didn't work. Weird."
3. <read docs> "'output'? Isn't that like a build directory, though? I
should be able to delete this and recompile from scratch. But if you
insist..."
4. "Well, that worked, so whatever, cool."

> * HOW do you want nanoc to handle binary assets? Similar to textual
> items, i.e. being able to filter them with (binary) filters? Would
> binary layouts make sense (e.g. putting a frame around a picture)?
> Should binary assets be applicable to textual items as well? Should
> filters apply to only a single "class" of items (image, sound, video,
> text, …)?

Ideally, I'd put binary items in the content folder, and they would
copy to output according to the normal routing rules.

—Christopher

On Aug 19, 11:14 am, Denis Defreyne <denis.defre...@stoneship.org>
wrote:
> denis.defre...@stoneship.org

Christian Plessl

unread,
Sep 20, 2009, 3:04:18 PM9/20/09
to na...@googlegroups.com
Hi everyone

Sorry for resurrecting an old thread, but I somehow lost track of the
plans for supporting binary assets in future nanoc versions. Is the
feature set already decided? If not, it may be still of help, if I
share my use case with you.

I'm currently in the middle of the process of converting my blog and
website from Wordpress to nanoc3. While in this process, I noticed two
places, where I have a need for binary assets.

a) CSS style-sheets

I plan to store the style files of the website in a directory
structure like to following (filesystem_compact data source):

config.yaml
contents/
contents/style/
contents/style/style.css <- to be processed with ERB
contents/style/img/background.png <- no processing needed
contents/style/img/favicon.ico <- no processing needed
...

The style directory should be processed by nanoc into the following:

...
output/style/style.css
output/style/img/background.png
output/style/img/favicon.ico
...

My intention is to keep the CSS style-sheets which are subject to ERB
processing close together with the binary files, in particular images,
that are related to the style-sheets.

b) binary files related to blog articles

My preferred structure for storing the blog posts is like this:

config.yaml
contents/
contents/blog/
contents/blog/2009-09-19.md
contents/blog/2009-09-19.yaml
...
contents/blog/assets/2009-09-19-a-screenshot.png
contents/blog/assets/2009-09-19-another-screenshot.png

Also here, my feeling says that it makes sense to store the binary
assets that are directly related to the blog posts also close in the
directory structure.

Both use cases are very similar and do not require advanced processing
of binary assets but only automated copying binary assets to the
output directory. Still, I see a number of advantages if nanoc would
manage these assets:

* output directory not polluted by static assets
* no rsync or rake copy task needed for copying assets (easier setup
of nanoc for these use cases)
* keep files processed by nanoc close together with binary files used
by the processed files

Best regards,
Christian

Denis Defreyne

unread,
Sep 20, 2009, 4:35:19 PM9/20/09
to na...@googlegroups.com
On 20 Sep 2009, at 21:04, Christian Plessl wrote:

> Sorry for resurrecting an old thread, but I somehow lost track of the
> plans for supporting binary assets in future nanoc versions. Is the
> feature set already decided? If not, it may be still of help, if I
> share my use case with you.

Hi,

Support for binary assets is still in its brainstorming phase. Once I
start implementing, I'll be sure to notify the mailinglist and mention
the features that will be included and the way they will work.

> I'm currently in the middle of the process of converting my blog and
> website from Wordpress to nanoc3. While in this process, I noticed two
> places, where I have a need for binary assets.
>
> a) CSS style-sheets
>
> I plan to store the style files of the website in a directory
> structure like to following (filesystem_compact data source):
>
> config.yaml
> contents/
> contents/style/
> contents/style/style.css <- to be processed with ERB
> contents/style/img/background.png <- no processing needed
> contents/style/img/favicon.ico <- no processing needed
> ...
>
> The style directory should be processed by nanoc into the following:
>
> ...
> output/style/style.css
> output/style/img/background.png
> output/style/img/favicon.ico
> ...
>
> My intention is to keep the CSS style-sheets which are subject to ERB
> processing close together with the binary files, in particular images,
> that are related to the style-sheets.

Yes, this is something I intend to support. Unlike nanoc 2.x, I would
like nanoc 3.x not to have assets stored in a separate directory. In
addition, the meta-file (the .yaml file) will be optional (in which
case the metadata will simply be empty) so you will be able to drop in
files without explicitly creating a YAML file for them.

I may also introduce file extension-based processing of items, but I'm
not entirely sure how to handle this yet. It would be quite useful to
do something along these lines (with + a new wildcard matching 1 or
more characters):

compile '/journal/+', :meta => { :extension => 'md' } do
filter :bluecloth
layout 'default'
end

compile '/journal/+', :meta => { :extension => 'html' } do
layout 'default'
end

In your case, this would make it easy to create a short-circuit
compilation rule that simply copies images that don't need to be
filtered. For example:

compile '*', :meta => { :extension => 'png' } do
end

This filtering-by-metadata approach could be quite useful but I am
unsure about the way it will actually work. More specifically, I
believe that the :meta => {} argument is inflexible; perhaps a lambda
could be used to filter things:

compile(lambda { %w[ png jpg jpeg bmp ].include?(item[:extension]) })
do
# do nothing; they're images so we just copy them
end

But that will likely tend to become ugly soon. If you have other
ideas, feel free to share.

> b) binary files related to blog articles
>
> My preferred structure for storing the blog posts is like this:
>
> config.yaml
> contents/
> contents/blog/
> contents/blog/2009-09-19.md
> contents/blog/2009-09-19.yaml
> ...
> contents/blog/assets/2009-09-19-a-screenshot.png
> contents/blog/assets/2009-09-19-another-screenshot.png
>
> Also here, my feeling says that it makes sense to store the binary
> assets that are directly related to the blog posts also close in the
> directory structure.

Same as above: you'll be able to drop the binary assets in there just
like you showed.

> Both use cases are very similar and do not require advanced processing
> of binary assets but only automated copying binary assets to the
> output directory. Still, I see a number of advantages if nanoc would
> manage these assets:
>
> * output directory not polluted by static assets

This is a very important advantage. In nanoc 3.0.x and below, the
output directory does not behave like a "build" directory that can be
emptied, but in nanoc 3.1.x it definitely should behave so.

> * no rsync or rake copy task needed for copying assets (easier setup
> of nanoc for these use cases)
> * keep files processed by nanoc close together with binary files
> used by the processed files

Both are quite correct. Thanks for sharing your use cases!

Christian Plessl

unread,
Sep 22, 2009, 3:49:32 PM9/22/09
to na...@googlegroups.com
Hi Denis

On 20.09.2009, at 22:35, Denis Defreyne wrote:

> On 20 Sep 2009, at 21:04, Christian Plessl wrote:

>> My intention is to keep the CSS style-sheets which are subject to ERB
>> processing close together with the binary files, in particular
>> images,
>> that are related to the style-sheets.
>
> Yes, this is something I intend to support. Unlike nanoc 2.x, I would
> like nanoc 3.x not to have assets stored in a separate directory. In
> addition, the meta-file (the .yaml file) will be optional (in which
> case the metadata will simply be empty) so you will be able to drop in
> files without explicitly creating a YAML file for them.
>
> I may also introduce file extension-based processing of items, but I'm
> not entirely sure how to handle this yet. It would be quite useful to
> do something along these lines (with + a new wildcard matching 1 or
> more characters):

Excellent. I think the two use cases you mention here are actually
quite frequent: 1) copy assets without processing, or 2) process
assets based on the kind of file and copy them to the output
directory. I'm looking forward to nanoc 3.1 or whatever version will
implement binary asset processing again.

Cheers,
Christian


h3raLd

unread,
Sep 23, 2009, 2:47:32 AM9/23/09
to nanoc
I just wanted to chime in to raise another small concern of mine
concerning assets in general. Personally, I'm a big fan of the
filesystem_combined data source. In other words, putting metadata
_inside files_, not in separate YAML files. This helps me keeping
everything in one place, and it's particularly useful for articles.

Unfortunately though, filesystem_combined _requires_ that you put at
least an empty "embedded yaml file", so in all my textual assets I
have things like this:

-----
-----

# Some javascript, CSS, SASS, etc.

Where this is not a big inconvenience _most_ of the time, things could
get worse: Sass gives me a warning saying that "because there's no
attribute in line -----, it will be ignored". A minor issue, but still
not great.

So, once again:

-> I'd like to be able to put *no metadata* whatsoever in text assets,
with filesystem_combined, without having nanoc complain that such
items are not "nanoc items"

Honestly, there's really not much need of metadata in assets 90% of
the cases, especially now that it is possible to "query them" through
extensions in rules!

As for binary assets, at the moment I have a /resources folder and a
rake task to copy them to /output as a temporary workaround...

Denis Defreyne

unread,
Sep 27, 2009, 8:07:00 AM9/27/09
to na...@googlegroups.com
On 23 Sep 2009, at 08:47, h3raLd wrote:

>
> I just wanted to chime in to raise another small concern of mine
> concerning assets in general. Personally, I'm a big fan of the
> filesystem_combined data source. In other words, putting metadata
> _inside files_, not in separate YAML files. This helps me keeping
> everything in one place, and it's particularly useful for articles.
>
> Unfortunately though, filesystem_combined _requires_ that you put at
> least an empty "embedded yaml file", so in all my textual assets I
> have things like this:
>
> -----
> -----
>
> # Some javascript, CSS, SASS, etc.
>
> Where this is not a big inconvenience _most_ of the time, things could
> get worse: Sass gives me a warning saying that "because there's no
> attribute in line -----, it will be ignored". A minor issue, but still
> not great.
>
> So, once again:
>
> -> I'd like to be able to put *no metadata* whatsoever in text assets,
> with filesystem_combined, without having nanoc complain that such
> items are not "nanoc items"
>
> Honestly, there's really not much need of metadata in assets 90% of
> the cases, especially now that it is possible to "query them" through
> extensions in rules!

Hi,

This would certainly be possible. Such an updated filesystem_combined
data source could check whether the file starts with "---" (or
"-----") and then treat the content between this and the next "---" as
metadata. If it does not start with "---", the file is assumed to have
no metadata.

One issue with this is that "---" is always considered to be special,
and that there cannot be files without metadata but with content
starting with "---". I'm not sure how to solve this problem in a fully
compatible way, and I'm not sure whether this is possible at all.
Perhaps something else than "---" should be used to delimit metadata
sections--something like this, maybe:

[nanoc-meta]
foo: bar
qux: meow
[/nanoc-meta]

content goes here...

It is unlikely that "[nanoc-meta]" will be used at the start of a
document that is not supposed to have a metadata section, so this will
probably work fine. What do you think?

h3raLd

unread,
Sep 27, 2009, 8:17:36 AM9/27/09
to nanoc
Well, I definitely don't like the BBCode-like tag solution, to be
honest :P

Personally, I'd leave "---" or "------" as default, and the maybe add
a setting to let users configure it? So if anyone wants to delimit
their metadata with °°°=§=§=§=°°°, it's their business!

(at worst I guess I could just create a new, custom data source to
take care of that myself, as you implicitly suggested :P)
> denis.defre...@stoneship.org

sickill

unread,
Sep 28, 2009, 6:08:10 AM9/28/09
to nanoc
> This would certainly be possible. Such an updated filesystem_combined  
> data source could check whether the file starts with "---" (or  
> "-----") and then treat the content between this and the next "---" as  
> metadata. If it does not start with "---", the file is assumed to have  
> no metadata.

I was thinking myself about writing such a data source which will do
exactly this (detection).
However I'm wondering what about these binary items. All items in
nanoc3 are being loaded into memory, right? So if we will treat some
png image without "---" at the start as an item without attributes, it
will still be loaded into memory. Am I right?

Marcin

Denis Defreyne

unread,
Sep 28, 2009, 6:31:35 AM9/28/09
to na...@googlegroups.com
On 27 Sep 2009, at 14:17, h3raLd wrote:

> Well, I definitely don't like the BBCode-like tag solution, to be
> honest :P
>
> Personally, I'd leave "---" or "------" as default, and the maybe
> add a setting to let users configure it? So if anyone wants to
> delimit their metadata with °°°=§=§=§=°°°, it's their business!

Hi,

I'd like to keep the number of configuration options to a minimum. I
believe that sensible defaults are more important than flexibility--
after all, making this a configuration option doesn't really solve the
problem but merely delegates it to the end user. I'd like to find a
solution that doesn't involve configuration options, if possible.

Regards,

Denis

--
Denis Defreyne
denis.d...@stoneship.org

Denis Defreyne

unread,
Sep 28, 2009, 6:31:36 AM9/28/09
to na...@googlegroups.com
On 28 Sep 2009, at 12:08, sickill wrote:

> However I'm wondering what about these binary items. All items in
> nanoc3 are being loaded into memory, right? So if we will treat some
> png image without "---" at the start as an item without attributes, it
> will still be loaded into memory. Am I right?

Hi,

That's correct; everything is considered to be textual and therefore
everything is loaded into memory. Because of this, nanoc 3.0 is not
suited for large, non-textual files.

nanoc 3.1 will have support for binary assets, i.e. files that are not
loaded into memory but can still be processed. I am not sure how the
filesystem_combined data source will work with binary files; it is
probably not a good idea to embed metadata directly in binary files.

Bernd Ohr

unread,
Oct 10, 2009, 3:09:29 AM10/10/09
to nanoc
Hi, this is my first post here!

I haven't done much with nanoc yet, but I think it best suited for my
static projects.

One thing I am missing is: I wan't stay binary items at the same place
as html-generating files, when they belong to them (photos for
example). The access of the metadata of items is also very important
for one of my bigger projects. All this is missing in nanoc 3.0, but I
think it can be solved easily.

My suggestion is a new data source like filesystem_meta, which has a
new section in rules.rb (I like the way a user can modify behavior in
rules.rb!) looking like:

meta '/help/' do
# retrieve meta data here
end

This gives flexibilty to the user and the system (the number of
options) is kept minimal.

Now I can do something (rather complex) like this (Mac OS only):

meta '*' do |m|
yaml = 'yaml'
exts = m.extensions - [yaml]
if m.extensions.length == 2 && exts.length == 1
m.load_yaml yaml
m.load_file exts[0] if m.meta[:load] != ':false'
elsif m.extensions.length >= 2
m.err "too much extensions, found #{m.extensions.join ', '}"
else
ext = m.extensions[0]
md_content_type = `mdls -name kMDItemContentTypeTree -raw assets/
favicon.ico`
if ext == 'blob'
m.meta[:load] = false
m.meta[:type] = :blob
elsif `file #{m.filename}.#{ext}` =~ /ASCII text/ # ask OS if it
is ASCII
m.load_file_with_yaml '--'
elsif md_content_type =~ /public.image/
m.meta[:load] = false
m.meta[:type] = :image
# retrieve other meta data here like EXIF etc...
elsif md_content_type =~ /public.zip-archive/
m.meta[:load] = false
m.meta[:type] = :zip
elsif md_content_type =~ /com.adobe.pdf/
m.meta[:load] = false
m.meta[:type] = :pdf
elsif md_content_type =~ /public.ruby-script/
m.meta[:type] = :ruby
m.load_file ext
else
m.err "unknown file type"
end
end
end

Compile rules then can access the meta data and call the desired
filter:

compile '*' do
if item[:pdf]
filter :copyfile
elsif item[:ruby]
filter :ruby
else
rep.filter :redcloth
layout 'default'
#else…..
end
end

The behavior of other filesystems can be mimiced:

# mimics filesystem_compact
meta '/articles/' do |m|
yaml = 'yaml'
ext = m.extensions - [yaml]
if m.extensions.length == 2 && ext.length == 1
m.load_yaml yaml
m.load_file ext[0]
else
m.err "there must be one yaml and one data file, found #
{m.extensions.join ', '}"
end
end

# mimics filesystem_combined
meta '/help/' do |m|
if m.extensions.length != 1
m.err "must be exactly one file, found #{m.extensions.length}"
else
m.load_file_with_yaml '--'
end
end

Of course this all is only some pseudo programming and maybe that I am
overlooking some important issues (most likely, I am a newbie to nanoc
and I even hadn't a look at the filesystem implementation).

Bernd

Bernd Ohr

unread,
Oct 10, 2009, 3:50:00 AM10/10/09
to nanoc
Oh, I made a mistake in the above example. Correction is:

filename = m.filename
filename += "." + ext if ext.length != 0
md_content_type = `mdls -name kMDItemContentTypeTree -raw #
{filename}`

Denis Defreyne

unread,
Oct 11, 2009, 7:45:35 AM10/11/09
to na...@googlegroups.com
Hi,

On 10 Oct 2009, at 09:09, Bernd Ohr wrote:

> One thing I am missing is: I wan't stay binary items at the same place
> as html-generating files, when they belong to them (photos for
> example).

This is indeed something that would be very useful, and it's one of
the goals for nanoc 3.1.

> The access of the metadata of items is also very important
> for one of my bigger projects. All this is missing in nanoc 3.0, but I
> think it can be solved easily.
>
> My suggestion is a new data source like filesystem_meta, which has a
> new section in rules.rb (I like the way a user can modify behavior in
> rules.rb!) looking like:
>
> meta '/help/' do
> # retrieve meta data here
> end

You can actually do that already in a preprocessor block. In such a
block, you can iterate over every item and add additional metadata
which you can get from e.g. Spotlight metadata. Here's an example:

preprocess do
items.each do |item|
# Find and set the UTI for this item
item[:uti] = system('mdls', '-name', 'kMDItemContentType', '-
raw', item[:content_file_path])
end
end

This requires a small change to the data source, though. The data
source should set the 'content_file_path' attribute, but this should
be fairly easy to do. Instead of this line:

attributes = meta.merge(:file => Nanoc3::Extra::FileProxy.new
(content_filename))

… the code would have to say something like this (getting rid of :file
at the same time because it's not very useful anyway):

attributes = meta.merge(
:content_file_path => File.expand_path(content_filename),
:meta_file_path => File.expand_path(meta_filename)
)

Now, you can change the compilation behaviour based on the UTI:

compile '*' do
switch item[:uti]
when 'public.ruby-script'
# ... do stuff here ...
when 'com.adobe.pdf'
# ... do other stuff here ...
else
# ... do default stuff here ...
end
end

So, what you are describing is actually already possible using
preprocessor blocks!

Reply all
Reply to author
Forward
0 new messages