Serving static files, but not from public directory?

[Felix E. Klee]

What is a good module to serve some static (JavaScript) files (including
caching and compression)?

Desired mapping of files to URLs:

scripts/*.js -> http://example.com/scripts/*.js (only for development)
xyz.js -> http://example.com/xyz.js
app.js -> not served (of course)
package.json -> not served (of course)
...

That's it. There is no public directory in my app.

What I already looked at:

* node-static: only allows specification of a public directory which is
mapped to the root of <http://example.com/>. As you can see above,
this is not what I want.

* connect: will not do caching anymore ("connect.staticCache() is
deprecated and will be removed in 3.0 use varnish or similar reverse
proxy caches")

By the way, I plan to deploy to Nodejitsu. Does anyone know whether they
do automatic caching?

[Tim Caswell]

What kind of caching do you want? This is a complicated issue. And
are you also talking about conditional http requests (Etag,
timestamps, 304 responses)? What about range request support?

A simple filter in front of any off-the-shelf static file server
should five you 95% of what you want.

var staticHandler = require('creationix').static("/", __dirname);
function (req, res, next) {
if (!req.uri) { req.uri = urlParse(req.url); } // parse the url if
it hasn't been parsed by another middleware
if (req.uri.pathname === "/app.js" ||
!req.uri.pathname.match(/\.js$/)) return next();
staticHandler(req, res, next);
}

Then you'll mount / -> / but only if it ends in ".js" and is not app.js.

If you don't mind caching your files in memory, then just read the
files to ram (either at startup or on demand) and watch the files for
changes (if you want to pick up changes) to invalidate your cache.

There may be an existing module that does everything you want out of
the box, but most likely, nothing is 100% what you want. Just build
what you want using simple modules (mime or my simple-mime, for
example to get mime type) and code you type. Don't fall into the trap
of using a large library that only does 50% of what you need and then
spend more time tweaking/hacking it than you would spend just building
what you want from scratch.

> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en

[Felix E. Klee]

On Fri, Aug 3, 2012 at 5:16 PM, Tim Caswell <t...@creationix.com> wrote:
> What kind of caching do you want?

Actually it's just one file that needs *caching* and *compression*:

http://example.com/xyz.js

The other files are just exposed for development (process.env.NODE_ENV
!== 'production'):

http://example.com/scripts/*.js

And, then I want to provide Socket.IO. Fortunately, the Socket.IO module
is taking care of serving itself.

What is all this? It's an API that users interface with via an object
provided by `xyz.js`. In a way it's similar to the YouTube API. There
you first load some JavaScript, and then you interact with the `YT`
object, to: load a new video, play it, stop it, etc.

The contents of the directory `script` are a bunch of AMD modules. They
are the building blocks of `xyz.js`.

> And are you also talking about conditional http requests (Etag,
> timestamps, 304 responses)? What about range request support?

Nothing complicated - see above. No redirects, no error page, etc.

> A simple filter in front of any off-the-shelf static file server
> should five you 95% of what you want.

I don't want to use a separate file server.

> var staticHandler = require('creationix').static("/", __dirname);

Looks interesting! Reading the source of `static()`...

> Then you'll mount / -> / but only if it ends in ".js" and is not
> app.js.

There are more files that I don't want to serve in `/`. In fact, for
security reasons, I don't want to mount `/` at all...

> If you don't mind caching your files in memory, then just read the
> files to ram (either at startup or on demand) and watch the files for
> changes (if you want to pick up changes) to invalidate your cache.

I see. That's a start for `xyz.js`. I could just use `fs.watchFile`.
Then I also need to think about compression. But perhaps the Nodejitsu
folks take care of that.

> Don't fall into the trap of using a large library that only does 50%

> of what you [...]

OTOH I don't have any special requirements, and so writing my own static
file serving code - while surely instructional - seems like reinventing
the wheel.

[Weltschmerz]

You can do this with Lactate

https://github.com/Weltschmerz/Lactate 

[Felix E. Klee]

On Sat, Aug 4, 2012 at 8:41 AM, Weltschmerz <chla...@gmail.com> wrote:
> You can do this with Lactate
>
> https://github.com/Weltschmerz/Lactate

Looks interesting! It would be nice, though, if the examples were not
Express specific. As many others, I am using plain Node.js.

[Weltschmerz]

I've added some examples for plain-jane node.js. The conversion is very simple.

On Friday, August 3, 2012 9:46:37 AM UTC-5, Felix E. Klee wrote:

[Felix E. Klee]

I'm testing Lactate now.

Questions and remarks:

* How do I disable client (browser) side caching?

Without the `expires` option being set, just no `Cache-Control`
header is returned. Is that enough? In general I though that
something as the following is required:

Cache-Control:max-age=0, no-cache

* Although I said earlier that I don't need it, I now think it would
be a nice idea to emit a custom 404 page for invalid paths (I check
that myself). So it would be nice if I could serve a file with 404
status code.

Custom headers may be a great addition!

[Weltschmerz]

Lactate doesn't set Cache-Control or Expires headers by default. 

It also doesn't cache in memory by default (but I'm considering to change that).

As for custom 404 handlers, I think that's a great idea. I'll work on it tomorrow, thanks.

[Oliver Leics]

Hallo Felix,

have a look at
https://github.com/oleics/node-filecache
for a simple in-memory filecache designed for node.js http servers.

See
https://github.com/oleics/node-filecache/blob/master/examples/server.js
for a example of a http-server for static files that supports
browser-cache-control, content-encoding and a custom 404-page.

Maybe it is useful for you.

[Felix E. Klee]

On Sun, Aug 5, 2012 at 1:23 AM, Oliver Leics <oliver...@gmail.com>
wrote:

> Maybe it is useful for you.

Thanks - looks definitely interesting, although for serving files still
a lot of code has to be written.

[Oliver Leics]

The code to serve files is in examples/server.js. Maybe it is a good
idea to put that example into a node-package.

[Oliver Leics]

I just released v0.2.0 that adds .httpHandler(options), returning a
handler for http-requests: handler(req, res [, next])

Again, please see
https://github.com/oleics/node-filecache/blob/master/examples/server.js
for a working example of a http-server serving static files cached in memory.

[Felix E. Klee]

On Sun, Aug 5, 2012 at 5:19 PM, Oliver Leics <oliver...@gmail.com>
wrote:
> https://github.com/oleics/node-filecache/blob/master/examples/server.js

This looks really cool! Haven't tried it out in my own project, yet,
though.

[tjholowaychuk]

Don't cache with node, slap varnish in front

[tjholowaychuk]

Though if you are going to cache with node you'll have to do a better

job than the hack I did with staticCache(), node-static does it extremely wrong

and will give you stale responses and invalid ones at times depending on the

request

[Felix E. Klee]

On Mon, Aug 6, 2012 at 6:15 AM, tjholowaychuk <tjholo...@gmail.com>

wrote:
> Don't cache with node, slap varnish in front

I wonder whether Nodejitsu is using something like that.

And I wonder how Varnish will know whether a file has changed and its
cache needs to be updated. With `Cache-Control: no-cache`, I assume
Varnish needs to get an up to date file for *every request*. Then,
again, there would be a read operation from disk per request, and
exactly that I want to avoid.

> node-static does it extremely wrong and will give you stale responses
> and invalid ones at times depending on the request

Why is that?

Eventually I consider watching the source code in `scripts` and - on
change - have it minified to `xyz.js`, or in memory. Oliver's
node-filecache looks interesting for that purpose.

[Abramovick]

I am in a similar situation as yours. Thanks to the guy who suggested node-filecache, because, it worked like a charm. 

N thanks to the guy who suggested lactate too. Although I couldnt get my head around lactate. The Documentation was poor because it was tied down to Express.

But anyways, once again thanks to the guy who suggested filecache, coz he saved my life. :) 

[P. Douglas Reeder]

IIRC, "hard" directives (Expires and cache-control: max-age) take effect before "soft" directives such as cache-control: no-cache (which actually means something like "validate before returning cached copy".
If a response is "fresh", the user agent will not make any request, and if you don't set Expires or Max-Age, browsers will pick a value based on who-knows-what.

So, make sure you have Expires or Max-Age set to an appropriate value before worrying about "soft" directives.