Why is 'require' not doing callbacks?

[Guoliang Cao]

Hi,

I'm new to Node.js and wrapping my head around the asynchronous programming model. A question comes to mind is why 'require' itself does not use callbacks like below. I believe IO is involved in require and a lot of things can fail.

require('http', 

function(http){

   // do stuff with http

},

function(error) {

   console.log("Failed to load http");

});

Thanks,

Guoliang Cao

[Marak Squires]

require() is a sync call.

If you need an async require ( a require that performs i/o ), you should check out a plugin system like Broadway ( https://github.com/flatiron/broadway )

This will allow you to require modules with an optional .init and .attach methods, which can both accept optional callbacks.

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

--

-- 

Marak Squires

Co-founder and Chief Evangelist

Nodejitsu, Inc.

marak....@gmail.com

[Roly Fentanes]

Even though it's IO, it is very common to require other files during startup, and therefore ok for it to be a synced operation.

[Martín Ciparelli]

So the question would be: why an async-driven platform uses sync-driven way to load modules?

2012/5/15 Marak Squires <marak....@gmail.com>

[Scott González]

The reason require() is synchronous is because asynchronous require() is quite burdensome to work with and the majority of require() calls happen on start up, where blocking I/O isn't a concern. In the early days, there was require.async() but it was a pain for node devs and node users. See https://github.com/joyent/node/commit/bb08f0c219bf4ca99b1129f54ef415ce46b01f22

[Marak Squires]

Node use to have an async ready event for all modules loading.

It wasn't that great.

[Mikeal Rogers]

If you search through the mailing list archive you can see very detailed discussions of why the "setup phase" of node programs should, and do, use synchronous file IO.

This topic has been beat to death, we should get this answer in an FAQ we can link to.

-Mikeal

[Jorge]

Async is always best, but sometimes not :-P

[Angel Java Lopez]

Any relation with require in other context?

For example, require in CommonJs

http://wiki.commonjs.org/wiki/Modules/1.1 

require "was not born" in Node.js, it has a previous story, and I guess Node.js team adopted the previous work.

[Mikeal Rogers]

When you require a module, or when you must load any file in to memory, before your program can actually accept any input, it's better to use sync file IO. In other words, if there is nothing your program can do until this file IO operation is complete then there is no benefit in not blocking the vm until it returns. In this case it's actually faster to do a synchronous file operation and halt the execution until the operation is complete.

After the "setup phase" (the time from when your program starts until the first nextTick()) you should not use synchronous IO because there are other concurrent IO events to process while you do file IO so it's better not to block any execution.

-Mikeal

[Mikeal Rogers]

That is where it started but node.js has several non-standard additions to the module system that are not in the spec.

At this point you should consider node.js modules to be specific to node and not based on another standard. Sure, there are places where it matches Modules/1.1.1 but there are many places it does not and node.js will likely never adopt new commonjs standards like AMD.

[Alexey Petrushin]

Yes, it's more handy to have it synced (although, maybe it should be also allow to be used in async manner, for sake of completeness, but not sure about it).

[Mikeal Rogers]

"completeness" is overrated. it's much better to be able to assume that a handler on nextTick() will be run after the full setup phase and that more required loading is will not happen after. removing features enable us to make better assumptions, eventually those assumptions are then features themselves :)

minimalism, it's a beautiful thing.

-Mikeal

On May 15, 2012, at May 15, 20121:00 PM, Alexey Petrushin wrote:

Yes, it's more handy to have it synced (although, maybe it should be also allow to be used in async manner, for sake of completeness, but not sure about it).

[Mark Hahn]

>   it's much better to be able to assume that a handler on nextTick() will be run after the full setup phase and that more required loading is will not happen after. 

Many times you cannot assume that.  My app often loads a module when already up and running.  When I am doing that I *really* hate blocking my entire app. Making that assumption kills a dynamic feature of the module system that the current api overlooks.

[Mikeal Rogers]

Yeah, don't do that :)

On May 15, 2012, at May 15, 20126:09 PM, Mark Hahn wrote:

>   it's much better to be able to assume that a handler on nextTick() will be run after the full setup phase and that more required loading is will not happen after. 

Many times you cannot assume that.  My app often loads a module when already up and running.  When I am doing that I *really* hate blocking my entire app. Making that assumption kills a dynamic feature of the module system that the current api overlooks.

[Isaac Schlueter]

Yeah, we're not going to resurrect require.async. It's very tricky to
get right, and not beneficial in the vast majority of use cases.

Load all your modules up front.

[Mark Hahn]

>  Yeah, don't do that :) 

How else do you dynamically load a module?

[Mikeal Rogers]

Don't dynamically load modules. Load all the modules you might need in the setup phase, never after.

On May 15, 2012, at May 15, 20128:55 PM, Mark Hahn wrote:

>  Yeah, don't do that :) 

How else do you dynamically load a module?

[Anand George]

In the use case specified here...  http://nodejs.org/api/modules.html#modules_module_exports we perform some initialization steps when we load the module. In the given example we use setTimeout to ensure that the module is loaded before we emit the event. Is there a way to ensure this synchronization if we have more modules to be required, each emitting an event when loaded.

[Bruno Jouhier]

It's a wart in the system, and more generally, Sync calls are warts.

Sync require makes it hard to do things like loading code dynamically or loading modules from a database. There are workarounds. For example you can load code from a database by caching it to local files and requiring afterwards. And you have to accept the fact that your event loop may pause to load code dynamically.

The only reason why things are so is because writing async code with callbacks is more difficult than writing sync-like code. So people take shortcuts.

Bruno

[Guoliang Cao]

Thank you all for replying.

It seems it is reasonable to write synchronous code for special cases like require. Suppose there are other synchronous IO library written for V8, will my node.js app be able to call them?

Regards,

Guoliang

[Jake Verbaten]

There are some valid use-cases for require.async

Can you (or anyone else) recommend a reliable user land implementation of async require

[Axel Kittenberger]

The true question is, why doesn't node follow its own naming conventions, and call it requireSync()?

I know the answer is probably, because some code functions are older than these conventions. Anyway it makes a bad role model for module devs, if the core already takes it quite lenient.

I'd suggest a 2nd optional parameter, which can be a buffer or string what would be the files contents, even the filename is just a phony name to sure uniqueness and reuse. So if you need any kind if asynchronousness, database, web-fetch or whatever, its your responsibility to aquire the contents before calling require which by itself is always a requireSync.

[Isaac Schlueter]

Synchronous require() not a "wart". Nor are fs.*Sync functions warts.
They're a valuable API that's there for a reason.

require() is for the startup time of your application. It's
significantly faster and simpler than loading modules asynchronously.
If you want a more complex module loader, then **build one**, and load
it synchronously at program startup.

I'm not just speaking theoretically here: I was personally very
involved with the development of node's module system, which started
out as a dual sync/async thing. Some of you may remember include(),
include_async(), and require_async().

You know what else has to be read synchronously at startup? The node binary.

Node is for writing servers. Servers have a startup phase where
they're not serving requests.
Node is also for writing command line utilities. Command line
utilities are best when they start up quickly.

Do you think that we just do asynchronous IO because it's some kind of
new fad, a word we say over and over again because it broadcasts
membership in some kind of club? No. That's not rational.

Do you think that "async" is some kind of magic gel that you pour over
your program and it makes it webscale? Please. Computers are not so
easily fooled. You make programs go faster by using resources wisely,
not by praying to some event loop gods.

Why does any of this matter? The point of doing things asynchronously
is so that you can continue to serve requests while IO is in progress.
This is relevant **only** because IO (especially network IO) is many
many orders of magnitude slower than anything else in your program.

You can take advantage of this by serving requests while other
requests are waiting on IO.

If you want to dynamically reload modules at run-time, there's a good
chance that you would be better off running that module as a child
process, and restart it on changes. The cluster module in v0.6 was a
good proof of concept for doing this, and in v0.8 it's much more
fleshed out. Dynamic module reloading is something we explored very
thoroughly in the early days of node's module system. It is fraught
with peril. The correct approach is to restart the server when you
want it to load new code. You can do this with workers in a cluster
pretty easily.

Otherwise? yeah. require() blocks. It's optimized for the startup
phase of your program, where it is important to go as fast as
possible, rather than be available to requests. Your server isn't
serving while it's happening. If you're require()ing something that
is not a native module, and not cached, then this means you'll wait a
few ms (or more!) while it reads from disk. That's why node programs
typically do a bunch of require() calls up at the top of the file, not
inside the server callbacks.

There's been a lot of exploration to come to this point, and figure
out the direction forward. I think we can make these patterns a lot
simpler to take advantage of. Adding an asynchronous require() would
not be correcting a wart. It would be a step backward.

[Marak Squires]

Sync require makes it hard to do things like loading code dynamically or loading modules from a database

Good.

--

Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

[Angel Java Lopez]

Interesting... thanks for the detailed history.

Isaac, this is nice topic for dinner at Buenos Aires, with Argentinean meat and wine ;-) we are waiting for your talk at http://jsconf.com.ar 

[Amjad]

I thought this page would be of interest. It introduces ss.load.code()  that allows you to require modules in the browser in an asynchronous manner. (Of course that is different from an asynchronous require on the server)

Loading Assets On Demand (in socketstream)
https://github.com/socketstream/socketstream/blob/master/doc/guide/en/loading_assets_on_demand.md