[nodejs] Re-requiring modules?

[Serge in Darkness]

It seems that subsequent `require()` of module does not lead to
recompiling, making hot code swapping impossible without manipulations
with file name. So, even if module was modified after first `require`
call, there is no way to update it in Node other than restart it.

Any ideas how to implement this?

[Jonas Pfenniger]

Look at the Module implementation in "src/node.js". Especially,
process.compile()

http://github.com/ry/node/blob/master/src/node.js#L906

Maybe this will give you some ideas.

--
Jonas Pfenniger (zimbatm) <jo...@pfenniger.name>

[Serge in Darkness]

Hmm, it seems to me that `process.compile(__wrap__, filename)` should
be called *every* time module loads.

There is some sort of module cache at http://github.com/ry/node/blob/master/src/node.js#L801
, maybe this is the case. I will inevstigate further.

Also, I cannot find any info related to module reloading in CommonJS
standart (which Node implements, AFAIK). For Node, which mainly
intended to run in FCGI-like manner, this should be quite useful
addition.

[Felix Geisendörfer]

I'm currently working on a require.hot patch for node. If you feel
adventurous, you might want to try my current version:

http://github.com/felixge/node/commit/86b5354a10026c9c86c3a3250f17f4ed45cbdc6c

It works perfectly fine for stuff like dynamically reloading the
response logic for a http server. However, I'm still hitting issues
integrating it with a production app of mine - once I figure out
what's going on there I'll put the patch up for discussion here.

--fg

[Serge in Darkness]

Thanks, Felix. I will try this.

On Dec 22, 6:44 pm, Felix Geisendörfer <fe...@debuggable.com> wrote:
> I'm currently working on a require.hot patch for node. If you feel
> adventurous, you might want to try my current version:
>

> http://github.com/felixge/node/commit/86b5354a10026c9c86c3a3250f17f4e...

[Jonas Pfenniger]

Felix,

why are you passing that "hot" argument all around ? Is it so that you
can require.hot("fresh") and all required modules in "fresh" will also
be reloaded ?

[Felix Geisendörfer]

> why are you passing that "hot" argument all around ? Is it so that you
> can require.hot("fresh") and all required modules in "fresh" will also
> be reloaded ?

Yeah, that's the idea. But I'm thinking that each module needs its own
cache, so that within the hot reloaded module each child module is
only reloaded once.

The entire patch is just a quick hack, I'll clean it up properly
before suggesting Ryan to have a look at it.

--fg

[Jonas Pfenniger]

Le 23 déc. 2009, 10:57 AM, "Felix Geisendörfer" <fe...@debuggable.com> a écrit :

>> why are you passing that "hot" argument all around ? Is it so that you > can require.hot("fresh") ...

>
> Yeah, that's the idea. But I'm thinking that each module needs its own
> cache, so that within the hot reloaded module each child module is
> only reloaded once.
>
> The entire patch is just a quick hack, I'll clean it up properly
> before suggesting Ryan to have a look at it.

That seem complicated. It seems hard to guess what reload scenarios
are needed for the application in a general way.

Why not implement a load() function, with require() being a memoized
version of it ? Then expose the memoized cache so that you can play
with it when implementing your reload scenarios.

Cheers,
zimbatm

[Felix Geisendörfer]

> That seem complicated.

I agree, its not trivial.

> It seems hard to guess what reload scenarios
> are needed for the application in a general way.

I disagree. My gut feeling says that most users want to be able to
reload a module and all of its children in an easy fashion. And that's
the problem I'm trying to adress.

I could be totally wrong on this, so anybody who cares about hot code
reloading should chime in - now is the time.

> Why not implement a load() function, with require() being a memoized
> version of it ? Then expose the memoized cache so that you can play
> with it when implementing your reload scenarios.

That's too low-level for me. You actually have to make sure that you
never try to reload more than a single module at a time as you will
otherwise run into nodes coroutine limitations / race conditions
easily.

That being said, the new patch I'm almost ready to present here will
expose the current module's cache to you as 'module.cache'. This means
you will be able to purge it according to your needs and then use
require.async to only reload an individual module without reloading
its children. I just don't see that as the common use case as you'll
have to implement your own reload queue or otherwise guarantee that
there is only 1 reload going on at any given time.

Let me know what you think.

--fg

[Marak Squires]

It occurs to me that watchFile might be appropriate as a hot loading option. It would be awesome to git push a .js file and have my node application know to trigger a reload of that file and a custom event closure...

2009/12/27 Felix Geisendörfer <fe...@debuggable.com>

--

You received this message because you are subscribed to the Google Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com.
To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/nodejs?hl=en.

[Felix Geisendörfer]

> It occurs to me that watchFile might be appropriate as a hot loading option.
> It would be awesome to git push a .js file and have my node application know
> to trigger a reload of that file and a custom event closure...

You will be able to do this, yeah. I just don't think it should be
built-in, as its very hard/ugly to update all references to a reloaded
module from "the outside" rather than having your app do it from "the
inside" if that makes any sense.

--fg

> > nodejs+un...@googlegroups.com<nodejs%2Bunsu...@googlegroups.com>

[Tom Robinson]

With CommonJS modules you can't really reload a single module, all dependent modules must be reloaded too, otherwise they'll be holding onto references to the old module.

Narwhal implements it's module system as two parts, loaders (reads, compiles, and caches *module factories* from disk) and sandboxes (instantiates and caches *module objects* from the module factory).

http://github.com/280north/narwhal/blob/master/lib/sandbox.js#L8

New sandboxes can be created and loaded with a fresh set of module objects, in this case on every JSGI request (but it could, and probably should, be done only when a loaded module's source file changes):

http://github.com/280north/jack/blob/master/lib/jack/reloader.js

(note that the loader happens to record and check modification times, so only modified modules get reloaded off disk, but every module gets re-"instantiated")

-tom

> --
>
> You received this message because you are subscribed to the Google Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com.

> To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.

[Daniel Ly]

I created a node.js variant in which require() can skip the cached
module.

Perhaps this is an easier solution but I have to admit that I do not
understand all the finer details. It's at http://github.com/nalply/node.

The idea is that hot loading is not implemented in node.js itself, so
the modifications are a lot more lightweight.

What do you think about this?

[Felix Geisendörfer]

> I created a node.js variant in which require() can skip the cached
> module.

That looks very similar to my first patch, but as Tom pointed out,
there is very little value in only reloading a single module. All
modules included by that module will still point to their old
versions, and you cannot implement a "deep reload" based on your
patch.

> The idea is that hot loading is not implemented in node.js itself, so
> the modifications are a lot more lightweight

I hear you. At this point I still consider my implementation a proof
of concept. I'll do some more work on it once the libeio issue (see
the event loop killer thread) is worked out. After thinking about it a
bit more, I think my reloading queue is not needed and just a
workaround to that. If that's the case, I'd be much more open to
pushing logic to user-land since no global queue would be required
anymore.

> Narwhal implements it's module system as two parts, loaders (reads, compiles, and caches *module factories* from disk) and sandboxes (instantiates and caches *module objects* from the module factory).

Sounds very interesting, I'll have to look into this some more.
However, that would be a major refactoring of the module system and
need a separate discussion.

--fg

On Dec 29, 8:11 pm, Daniel Ly <nal...@gmail.com> wrote:
> I created a node.js variant in which require() can skip the cached
> module.
>
> Perhaps this is an easier solution but I have to admit that I do not

> understand all the finer details. It's athttp://github.com/nalply/node.

[Daniel Ly]

I wrote:

> > The idea is that hot loading is not implemented in node.js itself, so
> > the modifications are a lot more lightweight

Felix Geisendörfer wrote:

> I hear you. At this point I still consider my implementation a proof

> of concept. [...]

Thank you. I realize now that reloading modules is quite a deep
problem and that I scratched only the surface.

--nalp

[Alex Bustin]

I've found that doing something like ...

x = require("./lib/foo");
y = require("./../app/lib/foo");

... will get you a two copies of a module chain.

Is this the correct behavior? Should the cache key be smarter?

- Alex