a pakage that exports few modules

[Osher E]

Hi all

most modules have an entry point, which is by default index.js.

But what if my module does not have one entry point?

What if it is a colleciton of many small cross-project core utility modules that i would not always want to load them all to use one of them?

I mean, by doing

var core = require('core').logger

we require  whatever is exported on the entry-point of core, and use only logger.

and there could be twenty of them, where in this project I just need the logger here.

Please avoid the discussion of wither to wrap each utility as it's own package...

Just assume that they do not justify that, but do need a package to live in...

I saw somewhere that I should be able to

var core = require('core/logger')

and should be able to direct in my package.json that 'core/logger' is actually available at ./lib/logger

only that now i cant find that place that sais how to do it...

Was I dreaming? Can anybody point it out for me? :)

Thanks

O.

[greelgorke]

make your entry point configurable:

// core entry point:

module.exports.libs = ['logger', 'morelib', /*all other*/];

module.exports.load = function load(libsToLoad){

  var libs = {};

  if(libsToLoad.indexOf('logger') >=0)

   libs.logger = require('./logger');

//proceed for every entry in libsToLoad

  return libs;

};

// client module

var core = require('core');

// check that lib can be loaded with core.libs

var libs = core.load(['logger']);

var logger = libs.logger;

hope it helps, i don't see any way to load a module out of the package. in fact this would contradict the purpose of packages as they are meant 

greetz

[greelgorke]

replace

 //proceed for every entry in libsToLoad

with

 //proceed for every entry in exports.libs

[Dan Milon]

I believe there is no performance drop if you require a huge library and
only use a small part of it. I mean, code memory is a small part of the
total memory you use.

danmilon.

> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en

[Christian Tellnes]

You can use getters like connect does.

https://github.com/senchalabs/connect/blob/master/lib/connect.js#L86-92



2012/8/13 Osher E <oshe...@gmail.com>:

[Osher E]

here, I'm looking at:

http://wiki.commonjs.org/wiki/Packages/1.1 

Specifically - search for "directories", and you'll end up with these texts:

• directories.lib - directory of modules to be loaded under the packages namespace. require(name/subfilename) must return modules from this directory. Definition must be relative to the package description file. 

• directories - Object hash of package directories. Typical directories include "lib", "src", "doc", "jars", "test" and "bin". Package manager tools must use these directory definitions to find various package components.

And from that I learn that I should be able to create a package from which I should be able to require("mypackage/somemodule")

Did anybody here did it before?

[Osher E]

here, I'm looking at: 

     http://wiki.commonjs.org/wiki/Packages/1.1 

Specifically - search for "directories", and you'll end up with the following findings in the texts, all of them under the section

Package Descriptor File 

• directories.lib - directory of modules to be loaded under the packages namespace. require(name/subfilename) must return modules from this directory. Definition must be relative to the package description file. 

• directories - Object hash of package directories. Typical directories include "lib", "src", "doc", "jars", "test" and "bin". Package manager tools must use these directory definitions to find various package components.

And from that I learn that I should be able to create a package from which I should be able to require("mypackage/somemodule")

Did anybody here did it before?

Here's what I try:

{

  "name": "common-core",

  "version": "0.0.1",

  "description": "common tools - wrappers for logger and crypto, a config-base util and more",

  "main": "index.js",

  "directories.lib": "lib",

  "directories": {

    "test": "test",

    "lib" : "lib"

  },

  "scripts": {

    "test": "node node_modules/vows/bin/vows --spec"

  },

  "repository": "hg+http://private-lan:2782/common-core",

  "license": "none",

  "dependencies": {

    "log4js": "0.5.*",

    "crypto": "0.0.3",

    "underscore": "~1.3.3",

    "validator": "~0.4.10"

  },

  "devDependencies": {

    "vows": "*"

  }

}

Classic Folder structure 

/common-core

  /lib

     config.js

     logger.js

     enc.js

     minimo.js

     ...and more 

  /test

     ... all test files

 package.json

 index.js

 README.md

There just have to be a way to do it...

[Dominic Tarr]

each module should to just one thing. if it does a lot of things it
should really be separate modules.

[Osher El-Netanany]

That's really great in theory.

In fact, you confuse between a module and a package.

Try to claim that underscore does only one thing. It does not - it gathers helpers for many different problems.

Underscore has one entry point that loads everything, and because of that - you can find other versions of it in many kind of transmutations, like nimble, backbone, and so forth, that essentially duplicate code with it.

To be more practical - 

If you got a module that all it does is abstract the logger use - a 60 lines of code, you don't publish it as a package, maintain it's own version number, it's own dependencies - that's less then ideal.

I have like 30 such modules that form the infrastructure in my organization, and I want them to be accessible in a smart way, like the spec originally intended to:

var logger = require("my-org-core/logger");

And let the implementation of require worry about loading any of the other 30 modules when it is actually called.

I don't think it's too much to ask - all that is needed is a way to explain which of the modules in the package can be required independently, and which are not :-)

[Osher El-Netanany]

sorry for the jabber - but I have to add to my previous mail - that it's a little more then just declaring what OO programmers would recognize as public and private ...modules in a package.

Technically, one can cheat, and do

var logger = require("./node_modules/my-org-core/lib/logger");

and by the spec, logger can be a lib/logger.js, or be a lib/logger/index.js - which is great - we really got planty of flexibility in that.

And that would work well, - especially under the convention that isaac promotes:

- if you require it - install it locally

- if you run it - install it globally

But that would be coupling with this structure, and I'm not sure we want to do that, because we built so far to allow possibilities, and not close them down.

I believe there should be a way to do the same, but without coupling with the folder structure.

[greelgorke]

> I believe there should be a way to do the same, but without coupling with the folder structure.

well that it about convention over configuration. if you follow the folder structure, you dont have to change or configure anything in 95%-99% of cases. thats how it works in many javaEE libs and somewhere else

the other way is to write more configuration. i'm not sure about if it is a better way 

[Jake Verbaten]

require("kitchen-sink/lib/foo");

You can already require files.

On Thu, Jan 17, 2013 at 10:09 AM, <lis...@millermedeiros.com> wrote:

PS: I do NOT want to start a discussion about keeping everything as separate packages, I really want a "kitchen sink" for this case. I'm not the first one to want this feature and I probably won't be the last one.

[lis...@millermedeiros.com]

I know that, to quote my previous e-mail:

 > The issue is not about the extra 4 chars ("lib/") but about having different require paths depending on the environment, code will be less portable.

I don't want `require("kitchen-sink/lib/foo")` on one environment and `require("kitchen-sink/foo")` on another, the lack of a "directories.lib" setting is hurting compatibility/flexibility. It's a shame to have to follow the less flexible system just because it is the most popular, this is not how "standards" should be created.

[Isaac Schlueter]

On Fri, Jan 18, 2013 at 3:19 AM, <lis...@millermedeiros.com> wrote:
> I don't want `require("kitchen-sink/lib/foo")` on one environment and
> `require("kitchen-sink/foo")` on another, the lack of a "directories.lib"
> setting is hurting compatibility/flexibility. It's a shame to have to follow
> the less flexible system just because it is the most popular, this is not
> how "standards" should be created.

Then you use require(actualFilePath) in node, and
require(fakeFilePath) in other environments. Put the node file at
foo/bar.js and the ringojs file at foo/lib/bar.js.

Or, you can check the environment in your script, and build up a
string using programming. Then pass that string to require(). You
could even get fancy and use a switch statement or something.

No amount of beating will bring this horse back to life.

[Jake Verbaten]

can ringo not require("kitchen-sink/lib/foo") ?

You could just move your files to the root of your folder. If there are too many files then don't have so many files.

On Fri, Jan 18, 2013 at 3:19 AM, <lis...@millermedeiros.com> wrote:

--

[Isaac Schlueter]

On Fri, Jan 18, 2013 at 7:34 PM, Jake Verbaten <ray...@gmail.com> wrote:
> You could just move your files to the root of your folder. If there are too
> many files then don't have so many files.

This. Exactly this.

[Miller Medeiros]

I don't know if you guys are just trolling or think that I'm dumb (maybe
both?). EVERYONE knows that you can use the full path or put the files at
the root folder - both are far from being "optimal" as described on
previous emails. It's not because you can do some silly hack that the
problem doesn't exist, common patterns are usually a "sign" of "flaws" on
the system itself.

Node.js is popular and because of that it has the power to "dictate" its
own way and flexible systems will have to "bend" to fit the dull model.
"Worse is Better"[1] really applies in this case.

"Write once, run everywhere" won't happen anytime soon because some people
insist to not follow the specs[2]. - that's exactly what made client-side
development a chaos, don't commit the same mistake.

I won't push this any further, maybe one day you guys will realize that in
some specific cases this actually makes sense. I should not be spending my
whole day adding more dependencies to my package.json or handling multiple
separate repositories when keeping everything into a single project would
be way easier to maintain. Don't hurt my productivity just because it goes
against your personal preferences, think about the users and what they are
trying to achieve, don't be a wall in the middle of the way.


[1] http://www.jwz.org/doc/worse-is-better.html
[2] http://wiki.commonjs.org/wiki/Packages/1.1#Package_Descriptor_File



PS: Sorry If I got more "emotional" than I should but this really sucks.

[Luke Arduini]

Magically finding files in a /lib directory sounds scary. Maybe I missed it, why not be explicit with your requires if you're including multiple files in your module? (not commenting on if I think this is a good idea)

[Glenn Block]

Check out utile: https://github.com/flatiron/utile

It has helpers requireDir and requireDirLazy which scan for .js files and exposes an object for each file with the key being the filename.

On Fri, Jan 18, 2013 at 10:27 PM, Luke Arduini <lucasa...@gmail.com> wrote:

Magically finding files in a /lib directory sounds scary. Maybe I missed it, why not be explicit with your requires if you're including multiple files in your module? (not commenting on if I think this is a good idea)

[Austin William Wright]

You cannot make the assumption that that's the one correct way to do things. Even if you can't think of a reason that someone would want to stuff their files in a subdirectory, that's no reason at all to not allow it.

A module is just special type of function. How it gets resolved to a file is the job of Node.js, and one method of resolving a file must not be prohibited on the basis of "well I wouldn't do it that way." The more Node.js can be decoupled from filesystem organisation, especially for the sake of implementations that we presently cannot dream of, the better.

[Isaac Schlueter]

On Sun, Jan 20, 2013 at 1:36 AM, Austin William Wright
<diamon...@users.sourceforge.net> wrote:
> You cannot make the assumption that that's the one correct way to do things.
> Even if you can't think of a reason that someone would want to stuff their
> files in a subdirectory, that's no reason at all to not allow it.

Node and npm do not disallow subdirectories. You can use as many
subdirectories as you like. But if you want to do
`require('foo/bar.js')` then 'bar.js' has to be in the root of the
'foo' package. You can put files in the root of the package, if
that's what you want.

All that Jake said was "If there are too many files, then don't have
so many files", which is basically the most simple solution that can
be devised.

On Fri, Jan 18, 2013 at 9:36 PM, Miller Medeiros
<lis...@millermedeiros.com> wrote:
> EVERYONE knows that you can use the full path or put the files at
> the root folder - both are far from being "optimal" as described on
> previous emails.

Correct. "Optimal" is exporting a single thing, which does one thing well.

The ugliness of the long paths and cluttered folders is merely a
reflection of the ugliness of an inelegant design. Node will not
prevent you from doing ugly things, but neither will it paper over
their ugliness. I know that sometimes ugly things have to be done,
and sometimes de-uglifying is not worth the effort. (Take a look at
npm's source some time.)

> It's not because you can do some silly hack that the
> problem doesn't exist, common patterns are usually a "sign" of "flaws" on
> the system itself.

I agree, but in this case, the flaw is in the bag-o-tricks module.
Carve your solution at the joints, and write small modules which do
one thing well, and can be easily assembled. They're easier to test,
easier to document, easier to understand, easier to use (in isolation,
or together, or with other different modules), and easier to fix (or
rewrite) when they're found lacking.

> Node.js is popular and because of that it has the power to "dictate" its
> own way and flexible systems will have to "bend" to fit the dull model.

We like dull models. They're easier to reason about. Programs should
be boring dull things that solve problems in predictable ways.

> "Write once, run everywhere" won't happen anytime soon because some people
> insist to not follow the specs[2]. - that's exactly what made client-side
> development a chaos, don't commit the same mistake.

You say "some people insist to not follow the specs" as if that spec
has EVER mattered. Writing up a bit of documentation on a wiki page
and then calling it a "spec" doesn't confer magical powers.

The documentation for the properties that matter in package.json files
is here: https://npmjs.org/doc/json.html (or `npm help json`). That's
authoritative. If you find any way in which node or npm deviate from
the contents of `npm help json`, then it's a bug.

Specs are a thing of the past. Node is the spec. Node is server-side
JS. This has always been about having a single-minded laser-like
focus on making Node better, regardless of what any other system does
or any other doc says. We will take good ideas from them, if those
ideas make sense for us, but if those ideas don't make sense for Node,
we are happy to ignore them.

Someone with the ability to do so should edit
http://wiki.commonjs.org/wiki/Packages/1.1 to mention that it does not
apply to node, never has, and never will. It's only a confusing
distraction.

> I won't push this any further, maybe one day you guys will realize that in
> some specific cases this actually makes sense. I should not be spending my
> whole day adding more dependencies to my package.json or handling multiple
> separate repositories when keeping everything into a single project would
> be way easier to maintain. Don't hurt my productivity just because it goes
> against your personal preferences, think about the users and what they are
> trying to achieve, don't be a wall in the middle of the way.

Who's a wall? You can either put your files in the root of your
package, or you can put "lib/" or whatever in your require() function
calls. You're saying that you want to add complexity to Node that *no
one else wants*, because you don't want to have to choose between a
bunch of files in the root of your project or an extra *4 characters*
in your require() statements.

How much does this really hamper your productivity? With the
keystrokes spent on this email, you could have typed out "lib/" in
thousands of require() statements already.

What is this really about? Is it just a reaction to the attitude that
Node is better than anything else? Does it seem cocky or arrogant to
reject features simply because they are not something we care about?

What kind of project maintainer would I be if I cared more about silly
things like this than actual real problems that need fixing?

[Austin William Wright]

On Monday, January 21, 2013 7:46:01 PM UTC-7, Isaac Schlueter wrote:

Who's a wall?  You can either put your files in the root of your 
package, or you can put "lib/" or whatever in your require() function
calls.  You're saying that you want to add complexity to Node that *no
one else wants*, because you don't want to have to choose between a
bunch of files in the root of your project or an extra *4 characters*
in your require() statements.

How much does this really hamper your productivity?  With the
keystrokes spent on this email, you could have typed out "lib/" in
thousands of require() statements already.

What is this really about?  Is it just a reaction to the attitude that
Node is better than anything else?  Does it seem cocky or arrogant to
reject features simply because they are not something we care about?

What kind of project maintainer would I be if I cared more about silly
things like this than actual real problems that need fixing?

 

On the contrary, what sort of project could Node.js be if everyone actually cared about doing things the objectively correct way.

How require() works, and indeed similar to how most programming language module loading systems ought to work, is it takes an argument specifying which function (specifically, which module) to evaluate (if not already evaluated), and returns the result. It could already be compiled into the system - this is how the core Node.js modules work. Out of simplicity and extensibility, however, most of the time you're going to use the filesystem to define a module. And if you have to select your require() arguments based around how your filesystem is laid out, then you're Doing It Wrong.

Cross-platform compatibility is a highly desirable feature. Therefore this is something worth investing time into and getting right. Yes, Node.js should do what is right for Node.js. But this is somewhat misleading per se. If what is right for Node.js means cross-platform compatibility, then this means cooperating with others and coming to a common agreement when the decision would otherwise be arbitrary.

[Isaac Schlueter]

On Tue, Jan 22, 2013 at 7:00 PM, Austin William Wright
<diamon...@users.sourceforge.net> wrote:
> On the contrary, what sort of project could Node.js be if everyone actually
> cared about doing things the objectively correct way.

"Objectively correct"? Wow.

Node programs run on filesystems. Every step of indirection away from
"the file on the filesystem" to "the string in the require()
statement" is a cost. Some of those costs provide some benefit, like
how require("foo") can be either locally in node_modules, or up a
level, so that modules can share dependencies (but don't have to, if
they would conflict). Other costs, like how require("foo/bar") can be
either foo/bar, or foo/bar.js, or foo/bar/index.js, are simply
historical mistakes that seemed like a good idea at the time, but
provide no value, and are now too costly to remove.

> Out of simplicity and
> extensibility, however, most of the time you're going to use the filesystem
> to define a module.

100% of the time is "most of the time", yes.

> Cross-platform compatibility is a highly desirable feature.

Cool story.

[Austin William Wright]

On Wednesday, January 23, 2013 1:13:47 AM UTC-7, Isaac Schlueter wrote:

On Tue, Jan 22, 2013 at 7:00 PM, Austin William Wright
<diamon...@users.sourceforge.net> wrote:
> On the contrary, what sort of project could Node.js be if everyone actually
> cared about doing things the objectively correct way.

"Objectively correct"?  Wow.

If you have a problem with my logic, the correct thing to do would be describe what's actually wrong. If you ignore the semantics of what operations like require() actually *mean* then you're going to run into problems, or stagnate growth.

So yes, there's an "objectively correct" way to do things. And you MUST NOT assume that the filesystem is the only way someone would want to dereference a module.

But this response isn't unusual to see when people can't actually attack logic.

 

Node programs run on filesystems.  Every step of indirection away from
"the file on the filesystem" to "the string in the require()
statement" is a cost.  Some of those costs provide some benefit, like
how require("foo") can be either locally in node_modules, or up a
level, so that modules can share dependencies (but don't have to, if
they would conflict).  Other costs, like how require("foo/bar") can be
either foo/bar, or foo/bar.js, or foo/bar/index.js, are simply
historical mistakes that seemed like a good idea at the time, but
provide no value, and are now too costly to remove.

The cost is the value of the next best alternative that must be given up. What's being given up? A few extra CPU cycles, worst case? That can't be it, the current Node.js implementation currently has to synchronously stat() for package.json, which should have been done away with long ago. The fact require() is synchronous far outweighs the magnitude of any few extra lines of code -- if that's even needed. The benefit of a uniform require() far exceeds any cost of implementation.

We were alright with adding a handler for ".json". But a uniform interface, that's not cool? For some reason?

 

> Out of simplicity and
> extensibility, however, most of the time you're going to use the filesystem
> to define a module.

100% of the time is "most of the time", yes.

As mentioned, require() may be extended to load things other than files. Take json, or coffeescript, for example. I think you suggested at one point that it'd be cool for require() to *load files off the Internet*. That's not a good idea, anymore?

 

> Cross-platform compatibility is a highly desirable feature.

Cool story.

Then help do something about it.

[Jacob Groundwater]

Other costs, like how require("foo/bar") can be

either foo/bar, or foo/bar.js, or foo/bar/index.js, are simply
historical mistakes that seemed like a good idea at the time, but
provide no value, and are now too costly to remove.

Is `index.js` considered a non-preferred method of doing package imports? I really like it, but arguing a stable feature is pointless. I'm just curious what the reasons were for including it, and what has changed since? 

FOR SCIENCE!

[Isaac Schlueter]

On Wed, Jan 23, 2013 at 9:13 AM, Jacob Groundwater <ja...@nodefly.com> wrote:
> Is `index.js` considered a non-preferred method of doing package imports? I
> really like it, but arguing a stable feature is pointless. I'm just curious
> what the reasons were for including it, and what has changed since?
>
> FOR SCIENCE!

index.js predates packages by quite a long time. It was the first
"load a packagey folder thing as if it was a single module" feature,
dating back to the 0.1.x days. It's not terrible. But personally I
prefer a main script in package.json, since you can name it something
more appropriate than "index.js", and somewhat like named functions,
it makes the debugging experience a bit nicer.

On Wed, Jan 23, 2013 at 6:18 AM, Austin William Wright
<diamon...@users.sourceforge.net> wrote:
>> "Objectively correct"? Wow.
> If you have a problem with my logic, the correct thing to do would be
> describe what's actually wrong.

Fair enough. There are two major problems with your argument.

The first is a gross category error. You are asserting that the
semantics of the require() function can be in any sense "objectively
correct", when in fact we are discussing complicated trade-offs
involving subjective values.

Second, you are asserting that "correctness" is defined by adherence
to a spec that has nothing to do with Node, and never did.

> If you ignore the semantics of what
> operations like require() actually *mean* then you're going to run into
> problems, or stagnate growth.

Where have you been the last 3 years? "Stagnating growth" isn't a
problem we have. We have the problems that come from *not* stagnating

growth.


> So yes, there's an "objectively correct" way to do things. And you MUST NOT
> assume that the filesystem is the only way someone would want to dereference
> a module.

I don't assume that. But it IS the only way that someone is capable
of dereferencing a Node module. Look at the code. That's how it
works. If you want your modules to live somewhere else, mount it with
fuse. Apart from the buildin modules bundled in the binary, Node only
loads modules using the fs.

> But this response isn't unusual to see when people can't actually attack
> logic.

Ad hominem.

> The cost is the value of the next best alternative that must be given up.
> What's being given up? A few extra CPU cycles, worst case? That can't be it,

The cost is time spent debugging programs you didn't write. That is
the cost I was referring to, not CPU cycles. (Unnecessary stat()
calls at startup are not ideal, but they're also not particularly
costly.)

In the normal case, you have three types of things:

- Builtin module: require('fs')
- Package main: require('request')
- Local module: require('./mine.js')

Package mains and builtins are similar in their appearance, and when
you're working with your program, they're actually somewhat similar in
how you tend to think of them, as well. They're a thing "outside" the
current program that it depends on. This similarity in appearance is
still a bit annoying, but changing it would be overly destabilizing.

The fourth case, is depending on a local module within a file, and
that's what we're talking about here.

So, say that the file is at node_modules/foo/lib/utils/bar.js. Right
now, you can do require('foo/lib/utils/bar.js'). It's pretty clear
what file you're pulling out: it's lib/util/bar.js from the foo
package.

If we add directories.lib then foo/lib/utils/bar.js could become
require('foo/bar.js'). What is the advantage of this? What is the
cost?

The advantage is that you can export more things from your package
slightly more easily. There's a feature designed explicitly for that,
so the API communicates that such a thing is a good idea. (Most in
the node community are of the opinion that it's *not* a good idea!)

Another advantage is that you can have shorter strings in your
require() function. But you can also have that by just putting files
in the root of the foo package. That is a much simpler solution, and
simpler solutions are to be preferred unless they come along with a
cost that outweighs the value of their simplicity. What is the cost
of putting files in the root of your package? Clutter in your project
folder. But is that clutter a bad thing, necessarily, and exactly how
bad is it? The clutter is a motivation to build a more elegant
module. The API is communicating that such an approach is
ill-advised. (And this reflects the feelings of the node community.)

The cost of directories.lib is that it's an extra step in figuring out
how node programs work. Consider a module baz that depends on foo.
In baz, there is the line `var Bar = require('foo/bar.js')`. I am
using baz, and debugging an issue that seems to stem from this Bar
thing. So, I want to look up its implementation. I type `npm explore
foo` to go into that folder, or `npm edit foo` to open it up in my
editor. Now what? Which file should I edit? There's a package.json
here, and a lib folder. No bar.js. In lib, there's a folder called
"exports" and a folder called "utils" and a folder called "browser"
and a folder called "ringo". Each have a file called "bar.js" in
them. Now I have to open up package.json, and understand it.

(God help me if you're using "overlays", another feature that we have
flat out refused.)

It gets much much worse if you use the "modules" feature of
package.json. Now it's not just a folder, but an arbitrary mapping of
module IDs to filenames. The "mapping" feature makes it so that I
can't even reliably read your code, since require("request") might
actually be something entirely different from the "request" module.

It may seem like I'm painting a bleak picture. I'm not. I'm actually
describing a real-world situation that I personally encountered trying
to build a real product for real customers at Joyent, back when npm
supported this feature you guys want to revive.

Supporting the "main" field is as far as we go. Why "main", but not
"directories.lib" or "modules" or "mapping", especially when we
already have "index.js" support? Well, having 25 files named
"index.js" is not very helpful in the debugger, and most packages were
already using the "main" field when we started putting
node_modules/package awareness into the node module loader. Only a
few were using modules or directories.lib, and they didn't have much
traction already.

The API that provides a specific interface for "export one thing"
communicates that the correct way to build programs is with modules
that do one thing. This is widely acknowledged in the node community
as a best practice.

Because of this, the experience of looking up a single "main" module
is much less crazy-making in practice, especially since it's often the
only .js file in the package.

> the current Node.js implementation currently has to synchronously stat() for
> package.json, which should have been done away with long ago.

Why? Because sync IO is "icky"? It's only icky if it prevents you
from servicing requests. At start-up time, it's actually much faster,
and gets you to a running program sooner.

The solution is quite easy in practice: don't do require() after start
time, or suffer the consequences.

> As mentioned, require() may be extended to load things other than files.
> Take json, or coffeescript, for example.

Those aren't examples of "things other than files". JSON and
coffeescript are files.

> I think you suggested at one point
> that it'd be cool for require() to *load files off the Internet*. That's not
> a good idea, anymore?

The node team has rejected several proposals to do exactly that, and
I've been convinced that it's a bad idea, or at least, would require a
massive change to how Node works.

Just because I think something would be neat doesn't mean it's
actually a good idea. I AM a hypocrite, after all.

>> > Cross-platform compatibility is a highly desirable feature.
>>
>> Cool story.
>
> Then help do something about it.

"Cool story" is not a way of communicating agreement. It is a way of
dismissing a non sequitur.

"Desirability" is not an innate property of features. "Desire" is a
feeling a person has. Something is "desirable" if anyone desires it.
But really, the people in the Node community who care about being
compatible with pinf-js and ringo are a rounding error.

So, desirable to whom? Who feels this "desire" sensation? Not enough
people to matter to me, I'm afraid, and certainly not enough to make
the benefits outweigh the costs.

Seriously, if you are so convinced, prove it. Fork node, add this
feature, and then wait for the kudos to roll in. Or better yet, go
get Meteor and Vert.x to be compatible with Node, and let the CommonJS
experiment rest in peace.

[Jacob Groundwater]

index.js predates packages by quite a long time.  It was the first
"load a packagey folder thing as if it was a single module" feature,
dating back to the 0.1.x days.  It's not terrible.  But personally I
prefer a main script in package.json, since you can name it something
more appropriate than "index.js", and somewhat like named functions,
it makes the debugging experience a bit nicer.

Very interesting, thanks. 

I usually lay my modules out like the following: https://gist.github.com/4611494

This gives me a lot of flexibility, and totally abstracts the require statements away from the file system.

[Austin William Wright]

On Wednesday, January 23, 2013 11:34:17 AM UTC-7, Isaac Schlueter wrote:

On Wed, Jan 23, 2013 at 9:13 AM, Jacob Groundwater <ja...@nodefly.com> wrote:
> Is `index.js` considered a non-preferred method of doing package imports? I
> really like it, but arguing a stable feature is pointless. I'm just curious
> what the reasons were for including it, and what has changed since?
>
> FOR SCIENCE!

index.js predates packages by quite a long time.  It was the first
"load a packagey folder thing as if it was a single module" feature,
dating back to the 0.1.x days.  It's not terrible.  But personally I
prefer a main script in package.json, since you can name it something
more appropriate than "index.js", and somewhat like named functions,
it makes the debugging experience a bit nicer.

package.json is, frankly, a terrible idea. Node.js is not and should not be in the business of knowing what a "package" is. That's something that exists several levels up in the abstraction hierarchy.

Nearly the entire changelog of require() behavior from v0.4 to now is a history of how *bad* npm's design was initially. It also happens to be a story of how influential npm was, for better or worse, because nearly all of those patches was a result of npm's abuse of some feature or another. And the *apparent* attitude while such features as require.paths was being removed, and package.json being added, was "well if npm can't do it right then we can't trust anyone to do it right". Now why bring this up?

I'll argue Node.js it right in the beginning, and that Node.js was at a better position to handle packages in v0.2 than it is now, so long as one had NODE_PATH properly exported, the same way one would export PATH. And it was always possible to use local modules, with some hacks. Exempting my own code, not once have I installed any module globally. I'll describe what can be done right further down.

The only reason we're in such a good position now is because of node_modules - a replacement of commonly used behavior with NODE_PATH and require.paths.

 

On Wed, Jan 23, 2013 at 6:18 AM, Austin William Wright
<diamon...@users.sourceforge.net> wrote:
>> "Objectively correct"?  Wow.
> If you have a problem with my logic, the correct thing to do would be
> describe what's actually wrong.

Fair enough.  There are two major problems with your argument.

The first is a gross category error.  You are asserting that the
semantics of the require() function can be in any sense "objectively
correct", when in fact we are discussing complicated trade-offs
involving subjective values.

Second, you are asserting that "correctness" is defined by adherence
to a spec that has nothing to do with Node, and never did.

I'm not trying to say that any specification canonically defines correct behavior. I'm instead saying two things:

(1) There is an objectively correct way to design things, based on their meaning and semantics. This is chosen in part by paying due respect to subjective preference, i.e. just because one can't think of why you would ever want to use an "index.js" file, this does not mean that it is a bad idea.

(2) Where a decision would otherwise be arbitrary, or of acceptable cost to the project, standardization is beneficial. You need not utilize existing standards, especially if the standard are poor or incorrect with respect to the application semantics. (If the specification were incorrect with respect to application semantics, then it follows that implementing the standard would not be an acceptable cost.)

Note that the RequireJS functionality doesn't (or has no authority to) define which files get looked up when a module name is dereferenced. It's just a (rather ugly) API in two parts, defining (1) how to load a module, and (2) how to export a module body.

Also, if some feature were indeed a bad idea, then take it out and call the next version 1.0. This is exactly what we have semver for.

 

> If you ignore the semantics of what 
> operations like require() actually *mean* then you're going to run into
> problems, or stagnate growth.

Where have you been the last 3 years?  "Stagnating growth" isn't a
problem we have.  We have the problems that come from *not* stagnating
growth.

Just because you can point to growth doesn't mean you can demonstrate a certain feature was responsible for it, or that you can excuse any decision that would negatively impact growth more than the alternative, i.e., a post hoc ergo propter hoc.

I'm making any assertion that there's no growth, I am saying continued lack of deference for proper design for will have a negative impact all other things held equal.

 

> So yes, there's an "objectively correct" way to do things. And you MUST NOT 
> assume that the filesystem is the only way someone would want to dereference
> a module.

I don't assume that.  But it IS the only way that someone is capable
of dereferencing a Node module.  Look at the code.  That's how it
works.  If you want your modules to live somewhere else, mount it with
fuse.  Apart from the buildin modules bundled in the binary, Node only
loads modules using the fs.

The point of is to make lives easier. Yes, you could do really costly things like custom filesystems or obscure linking. But those are hacks, and you shouldn't have to.

 

> But this response isn't unusual to see when people can't actually attack 
> logic.

Ad hominem.

On the contrary, I was saying "I don't blame you", albeit being needlessly snarky in doing so.

 

> The cost is the value of the next best alternative that must be given up. 
> What's being given up? A few extra CPU cycles, worst case? That can't be it,

The cost is time spent debugging programs you didn't write.  That is
the cost I was referring to, not CPU cycles.  (Unnecessary stat()
calls at startup are not ideal, but they're also not particularly
costly.)

I'm not seeing the cost here, to the point there's no way I could have known you were talking about debugging, or anything else you're referring to. I've never had a problem with locating the source files that a module is referring to, nor do I anticipate one. People aren't going to extend require() in some bizarre fashion unless it's a net benefit somehow.  Especially if you're using a debugger, no matter how convoluted module dereferencing becomes, you still are handed off the correct source file. (Even if it is crazily uglified, as with compiled Coffeescript.)

But for the sake of argument, maybe there is some case where this really will add a significant (measurable), marginal (on the margin) cost to development. Then, likely, the development process for your use-case was cheap and will become expensive, and as such the development process was likely chosen not because it was correct, but precisely because it was cheap, and wrongly chosen, though through no fault of your own, and as such we can fix whatever is causing the excess burden. 

Node.js by its nature defines what a module resolves to out-of-bounds from the file naming what is to be loaded. In Node.js, it's up to the external environment of the application to define which function "http" or "mysql" is, exactly. You seem to be of the opinion that this is partly a bad thing.

Contrast this to how most frameworks and programming languages work: The module to be loaded is globally named somehow in a central namespace, defined in a system directory (maybe with an option to install things in a user directory), or both.

The former functionality, defining modules out of bounds or otherwise without a central namespace, is far more powerful. This means that two applications can both refer to a "mysql" module, and it each resolves to a different module, no matter if it's byte-for-byte the same (actually the same thing as being the same module, since modules exist per-process), or different versions of each other, or completely different, competing projects.

This exact behavior, though, is what enables us to do things like depend on certain versions of modules, a certain Git tree/commit, or a certain semver, or even more esoteric patterns. It allows each module to use a different name. It makes sure that your application always runs the code you endorse. (The exception being Node.js which is often installed system-wide. I don't see a particular problem with this, so long as Node.js properly marks breakage according to the semver specification, and especially considering usually people running Node.js tend to have root access anyways, though this is not an assumption we should make.)

Now I may have been a bit confusing because I haven't actually thoroughly explained what I'm actually favoring. I'm not saying Node.js should add "lib/" to its search path (not necessarily, at least). I'm advocating that Node.js, wherever possible, should have less say in what a module name resolves to. You're arguing that this leads to unpredictable behavior. But what you're thereby arguing is that Node.js package management schemes should not be allowed to evolve towards better setups. That is, the cost of letting people define their own behavior is negative. Or another way, diversity and evolution is good in the long run.

Nor do I believe your assertion that such freedom presents a significant development cost. If I want to find out which executable is going to be run when I type `git` or `ffmpeg`, I type `which ffmpeg`. That sounds like a much better solution than not letting people specify their own $PATH. (Obviously Node.js still has NODE_PATH, I refer to the more complex functionality I will describe shortly.)

This probably means:

Remove package.json and similar lookups from consideration, and replace index.js behavior. If, upon stat, the target is a directory, then recursively look for name+'/index' or maybe name+'/lib'. This would continue to permit symlinks e.g. index -> lib/ or index -> lib/module.js. This behavior is more flexible and just as readable as current solutions.

I'd like to see people be able to replace the dereferencing functionality entirely. However I don't have any particular proposal to do so or how to do so. But the dereferencing process is nonetheless done within application space, and not globally (unlike, say, a URI), and while I don't anticipate anyone needing to extend the functionality beyond what I laid out, that possibility must not be eliminated.

It gets much much worse if you use the "modules" feature of
package.json.  Now it's not just a folder, but an arbitrary mapping of
module IDs to filenames.  The "mapping" feature makes it so that I
can't even reliably read your code, since require("request") might
actually be something entirely different from the "request" module.

I'm not even sure why that exists.

 

It may seem like I'm painting a bleak picture.  I'm not.  I'm actually
describing a real-world situation that I personally encountered trying
to build a real product for real customers at Joyent, back when npm
supported this feature you guys want to revive. 

Supporting the "main" field is as far as we go.  Why "main", but not
"directories.lib" or "modules" or "mapping", especially when we
already have "index.js" support?  Well, having 25 files named
"index.js" is not very helpful in the debugger, and most packages were
already using the "main" field when we started putting
node_modules/package awareness into the node module loader.  Only a
few were using modules or directories.lib, and they didn't have much
traction already.

The API that provides a specific interface for "export one thing"
communicates that the correct way to build programs is with modules
that do one thing.  This is widely acknowledged in the node community
as a best practice.

Because of this, the experience of looking up a single "main" module
is much less crazy-making in practice, especially since it's often the
only .js file in the package.  

> the current Node.js implementation currently has to synchronously stat() for
> package.json, which should have been done away with long ago.

Why?  Because sync IO is "icky"?  It's only icky if it prevents you
from servicing requests.  At start-up time, it's actually much faster,
and gets you to a running program sooner.

I didn't mean synchronous operations, I meant all reliance on package.json, as described above.

 

The solution is quite easy in practice: don't do require() after start
time, or suffer the consequences.


> As mentioned, require() may be extended to load things other than files.
> Take json, or coffeescript, for example.

Those aren't examples of "things other than files".  JSON and
coffeescript are files.

> I think you suggested at one point
> that it'd be cool for require() to *load files off the Internet*. That's not
> a good idea, anymore?

The node team has rejected several proposals to do exactly that, and
I've been convinced that it's a bad idea, or at least, would require a
massive change to how Node works.

Just because I think something would be neat doesn't mean it's
actually a good idea.  I AM a hypocrite, after all.


>> > Cross-platform compatibility is a highly desirable feature.
>>
>> Cool story.
>
> Then help do something about it.

"Cool story" is not a way of communicating agreement.  It is a way of
dismissing a non sequitur.

I too am communicating my displeasure that you don't see the gravity of the argument. We made a case that having a certain style of require() arguments is a desirable feature because it would often allow programs to work cross-platform natively, without modification. This is a very desirable function for Node.js, being an ECMAScript based platform. The argument being made is precisely for cross-platform so I don't see how you can call it a non sequitur.

If you don't see the use of cross-platform ES scripting then I'm not sure what to say.

 

"Desirability" is not an innate property of features.  "Desire" is a
feeling a person has.  Something is "desirable" if anyone desires it.
But really, the people in the Node community who care about being
compatible with pinf-js and ringo are a rounding error.

I use "desirable" to mean, "this feature is pretty darned important or useful function to a program or developer".

 

So, desirable to whom?  Who feels this "desire" sensation?  Not enough
people to matter to me, I'm afraid, and certainly not enough to make
the benefits outweigh the costs.

Just because you don't develop cross-platform doesn't mean you ought to exclude people who do (me and the several others of us in this thread). Again, about the whole inhibiting growth thing.

 

Seriously, if you are so convinced, prove it.  Fork node, add this
feature, and then wait for the kudos to roll in.  Or better yet, go
get Meteor and Vert.x to be compatible with Node, and let the CommonJS
experiment rest in peace.

 

Now I don't really like CommonJS at all, though admittedly it's better than any other thing I've come across. I'd like to see "return" values honored, but that's trivial enough it doesn't need a fork from me.

But I can't see making progress on anything else as is, without development work in other areas first. Right now I'm stuck at convincing people that pragmatism isn't good.

[Isaac Schlueter]

On Wed, Jan 23, 2013 at 7:30 PM, Austin William Wright
<diamon...@users.sourceforge.net> wrote:
> ...

Look, here's the bottom: The module system is locked. We're not
interested in changing it. The source is open, so you can go nuts on
it if you'd like. Your feelings about node's module system are rare,
and not shared by the development team, or a significant fraction of
users.

We don't destabilize node-core for vocal minorities.

> Right now I'm stuck at convincing people that pragmatism isn't good.

That's confusing. Why are you here if you don't think pragmatism is a
good thing? Node is all about pragmatism.

[Miller Medeiros]

Since it's still being discussed...


## To give some context

I'm all favor of small modules that does one thing and does it well,
"single responsibility" is usually my mantra except when it doesn't.
Almost all my open source projects have a single responsibility:

- http://millermedeiros.github.com/js-signals/
- http://millermedeiros.github.com/crossroads.js/
- https://github.com/millermedeiros/nodefy/
- https://github.com/millermedeiros/rocambole/
- https://github.com/millermedeiros/Hasher
- https://github.com/millermedeiros/esformatter/

But there are exceptions to all rules...

Around 1 year ago I decided to split some of my JavaScript helpers into
individual files so I could load/require only the methods I needed - I do
mostly front-end development so filesize/memory is a big concern - think
of it as a modular version of underscore.js, it eventually became the
moutjs project ( http://moutjs.com ), we currently have almost 200
individual modules/functions. Imagine if I decided to flood npm?

With a single package/repository maintenance can be way easier - we all
know that git submodules can be a huge PITA. You can have a single build
script, scaffolding is easier, publishing updates is easier, managing
packages is easier (less places to look for), running tests is easier,
only a single travis-ci instance required, etc... It also helps on
**discoverability** and **consistency** (you have a clear sense of what
names are already used and everything follows same convention), less time
spent on `npm search my-helper-method`, less time updating package.json,
create a sense of "community", centralized bug tracker, same identity,
centralized docs...

Maybe you are working somewhere without internet connection and remember
that you need some helper method that wasn't on your dependency list -
damn.. If you had downloaded the kitchen sink (mout) instead of individual
methods you could just "require" what you need, all the files you need are
already inside the node_modules folder (including docs).


## Why "directories.lib" is such a big deal?

It decouples the file location with the value that you are "requiring", if
you decide to change the source folder structure you don't need to do a
new major release, renaming folders/files should not be a reason for a
breaking change. The `require()` syntax SHOULD be flexible, it's just a
string ID used to map into some module, there is no good reason for it to
not allow a different scheme than the fs.

I would go even further and add an "alias" setting (similar to the old
"modules"[1]) to have local remap of modules so you could
`require("foo/bar")` and load "foo/ipsum" instead. This would solve issues
with "long-and-bad-module-names" and would make the whole namespace
discussion irrelevant. It would also reduce backwards incompatibilities
just because you decided to rename an internal file.


## "Hard to debug" myth

I call it BS. If `npm explore foo` and `npm edit foo` doesn't work as
expected then it is an npm issue, fix/improve/adapt the tool, don't limit
the package structure just because the tools aren't good enough. Each
top-level package contains a package.json with ALL the config, just need
to read the closest one... It's like when you create an alias to a method,
it doesn't get harder to debug, it just have a different name, call stack
still works the same, line numbers match, file path match. It is not a big
deal if you app requires "foo" and in fact it loads "bar" it's YOUR app,
just open the package.json and read what has been remapped, or add a new
command to npm that displays which module is being loaded for a given ID,
just like `which npm` works, problem solved.


## Namespaces are old school

Saying that the user should put everything into the same "namespace" when
we have a real module system is silly.
`I.dont.want.to.write.code.like.this()` Also saying that every package
should consist of a single script or all files should be on the root
folder is naïve... "A reprogrammable system is incredibly powerful.
Abusing the power is always possible and it's a point of principle in a
reprogrammable system that people must be able to abuse it." - @nickferrier


---

I solved this "issue" on moutjs by converting all modules to node.js
format and putting the files on the root folder during "npm prepublish" so
the same require paths are used on AMD and node.js, but it feels dumb to
have to place everything into the root folder, also made it slightly
harder to cleanup the folder after `npm publish` (more paths to `rm -r`).
The system have flaws, it's not because we can be "creative" about how to
solve it that the problem goes away.


[1] how "modules" worked in npm a couple years ago:
https://github.com/isaacs/npm/commit/a12ead34615bf62eae106010c8b27069e98e5f
63

[Mariusz Nowak]

I'm also getting confused, but still I'm after simplicity and rather happy that 'directories' are not supported any more.

Isaac do you see any semantics in 'lib' folder? Do you feel that some modules should rather be placed in 'lib' than at 'root' fo the package (or other way) or it's totally "whatever works" in your opinion?

I always have problem with that, firstly lib makes my root clean, but on contrary it makes require import verbose and not that clean.

Additionally I don't agree with suggestion to put all package code into one big module. Smaller modules are much more maintainable and provide better customizability for users of the package, it's especially important if you port your modules for the browser.

[Austin William Wright]

On Wednesday, January 23, 2013 9:04:52 PM UTC-7, Isaac Schlueter wrote:

On Wed, Jan 23, 2013 at 7:30 PM, Austin William Wright
<diamon...@users.sourceforge.net> wrote:
> ...

Look, here's the bottom: The module system is locked.  We're not
interested in changing it.  The source is open, so you can go nuts on
it if you'd like.  Your feelings about node's module system are rare,
and not shared by the development team, or a significant fraction of
users.

We don't destabilize node-core for vocal minorities.

Absolutely not: If something is wrong, you change it and increment the major version number. That's not "destabilizing" in any sense of the word, that's improving functionality, and telling people that you broke reverse compatibility in doing so.

There's nothing "vocal minorities" about cross-platform coding. Most people won't use 'crypto'. Does that make them a vocal minority? Most people won't use 'domain'. Does that make them a vocal minority? Most people won't be affected by any of the bugs being filed from now on, but we still regard those as important. But ill-defined require() is just as much a bug as any of those.

The story of the growth of Node.js is getting support and functionality out for as many people as possible. You don't regard if it's going to be a "majority" or not, you regard if it's *profitable at the margin*.

 

> Right now I'm stuck at convincing people that pragmatism isn't good. 

That's confusing.  Why are you here if you don't think pragmatism is a
good thing?  Node is all about pragmatism.

When you say "pragmatism" I hear "I don't understand the the system works together well enough to understand the consequences of my actions." That there are no facts, no fixed laws of logic, no certainty, no objectivity. It is the antithesis of semantics and objectivity. And it is not an appropriate design strategy for any software system.

[Mark Hahn]

tl;dr

Skimming over this thread I don't see anyone jumping on listas'/austin's bandwagon.  I, for one, certainly agree with Isaac's arguments.  I also think he has been very nice in putting in so much effort in replying to such a stupid concern.

In short, +1 for Isaac's arguments.

--

[Isaac Schlueter]

>> We don't destabilize node-core for vocal minorities.
> Absolutely not: If something is wrong, you change it and increment the major
> version number. That's not "destabilizing" in any sense of the word, that's
> improving functionality, and telling people that you broke reverse
> compatibility in doing so.

You're about 2 years too late, Austin.

We only break reverse-compatibility when absolutely necessary, and
never for modules that are marked with the "Stable" stability level or
higher.

There are some actual real-world programs running on node making real
money that pays real rents. Like it or not, Node is a relevant
platform now, not just any old program. The time for major breaking
changes is behind us. Even with the new streams API, which is by all
accounts better, we only made one semantic change to existing
functionality, and only because it is *literally impossible* to make
the desired fixes without doing so. (And we provided workarounds for
the most common cases where it's an issue, so that it's not an issue.)
To justify that change, we have heaps of feedback from existing
expert users, and confusion from new users.

The amount of work that a breaking change involves today is
astronomical, because we care about the people using it. I'm not
talking about the code change, either, but the social upheaval of
"Why'd you break my shit!?" I fear that there will be some semantic
changes necessary to fix the HTTP client's problems in 0.12, but I
hope that we can do everything we need to in a non-breaking way.

The goal of node has *always* been to get to a state where it's
effectively "done". We want node to be like awk or sed or grep:
something that's just there, and taken for granted.

> There's nothing "vocal minorities" about cross-platform coding.

As far as I can tell, there are 3 people who care about putting back
the directories.lib to package.json. All three of you are vocal.
Everyone else either seems to not care, or agrees that it was a bad
idea in the first place. That's the paradigm case of a "vocal
minority".

> Most people won't use 'crypto'. Does that make them a vocal minority?

So, first of all, everyone who uses https uses crypto. That's not a
minority. They just don't realize they're using it.

Secondly, they're not vocal, because it's working for them just fine.
They'll *become* vocal if we suddenly break their programs, though,
you can bet on that! And the majority will agree with them that
breaking crypto was a bad idea.

> But ill-defined require() is just as much a bug as any of those.

How is require() ill-defined? There's even detailed pseudocode in the
documentation explaining exactly how it works.

> When you say "pragmatism" I hear "I don't understand the the system works
> together well enough to understand the consequences of my actions." That
> there are no facts, no fixed laws of logic, no certainty, no objectivity. It
> is the antithesis of semantics and objectivity. And it is not an appropriate
> design strategy for any software system.

Well, I think I understand how Node's module system and package
paradigms work better than most people. I certainly have a unique
perspective on the complications of debugging node module code you
didn't write (since many reported npm bugs are actually package bugs,
but I spend time tracking them down anyway). I work at a company that
is one of the biggest node users, and I am in constant communication
with the node core development team, as well as engineers at Voxer,
Nodejitsu, and several others.

When I say "pragmatic", I mean that practice and theory inform one
another, rather than being a one-way relationship, and that empirical
learnings from practice always trump conclusions drawn from theory.
That means that userland program breakage is worse than any
ideological breakage, and that context is more important than
consistency. The "consequences of our actions" that we care about are
making life easier for those people using Node for real applications
running on real networks.

Though I do occasionally monologue for 57 pages, I am not a character
in an Ayn Rand novel, and as such, don't really care much about these
theories regarding facts, fixed laws of logic, certainty, objectivity,
or antitheses of semantics. I happen to think that getting stuff done
in simple ways is an ideal design strategy for software.

[Austin William Wright]

On Thursday, January 24, 2013 12:57:13 PM UTC-7, Isaac Schlueter wrote:

You're about 2 years too late, Austin. 

We only break reverse-compatibility when absolutely necessary, and
never for modules that are marked with the "Stable" stability level or
higher. 

 

There are some actual real-world programs running on node making real
money that pays real rents.  Like it or not, Node is a relevant
platform now, not just any old program.  The time for major breaking
changes is behind us.  Even with the new streams API, which is by all
accounts better, we only made one semantic change to existing
functionality, and only because it is *literally impossible* to make
the desired fixes without doing so.  (And we provided workarounds for
the most common cases where it's an issue, so that it's not an issue.)
 To justify that change, we have heaps of feedback from existing
expert users, and confusion from new users.

The amount of work that a breaking change involves today is
astronomical, because we care about the people using it.  I'm not
talking about the code change, either, but the social upheaval of
"Why'd you break my shit!?"  I fear that there will be some semantic
changes necessary to fix the HTTP client's problems in 0.12, but I
hope that we can do everything we need to in a non-breaking way.

Node.js has broken reverse compatibility three times on me, and not in a minor version update, during a *patch* update. I'm not asking for an end to this, on the contrary, in all the cases it was a good and proper change. I am asking that we be told when such a change is made. Release Node.js 1.0. Do it now. And never stop making improvements when they are necessary.

 

The goal of node has *always* been to get to a state where it's
effectively "done".  We want node to be like awk or sed or grep:
something that's just there, and taken for granted.

Ignoring entire segments of use-cases will mean Node.js is either not "done", or, is permanently buggy.

Obviously it's not exactly like shell utilities, some improvements will need to be kept maintained. Like crypto, for instance. I assume people know this.

 

As far as I can tell, there are 3 people who care about putting back 
the directories.lib to package.json.  All three of you are vocal.
Everyone else either seems to not care, or agrees that it was a bad
idea in the first place.  That's the paradigm case of a "vocal
minority".

I'm not advocating that Node.js use package.json for anything. Quite the opposite.

I do agree with the position that there are changes to be made to enhance cross-platform compatibility  And you must realize that the use case is far more than three people. How about, more like, everyone who wants to deploy a module to a web browser?

 

So, first of all, everyone who uses https uses crypto.  That's not a 
minority.  They just don't realize they're using it.

I consider that using 'https', not 'crypto'. I don't believe that 'crypto' should be used directly by users, except HMAC and maybe PBKDF2. In all other cases you should be using 'tls' or 'https', or a library by someone who knows what they're doing that has correctly implemented a functionality that depends on the use of e.g. vanilla SHA-1. I.e. I'm an advocate that 'crypto' remain used only by a vocal minority, and that this is a good thing.

 

Secondly, they're not vocal, because it's working for them just fine.
They'll *become* vocal if we suddenly break their programs, though,
you can bet on that!  And the majority will agree with them that
breaking crypto was a bad idea.

Breaking crypto was a necessary and proper idea. Except, where is version 1.0?

 

How is require() ill-defined?  There's even detailed pseudocode in the 
documentation explaining exactly how it works.

Maybe not ill-defined, rather, inconsistent with its own semantics, as previously described.

 

Well, I think I understand how Node's module system and package 
paradigms work better than most people.  I certainly have a unique
perspective on the complications of debugging node module code you
didn't write (since many reported npm bugs are actually package bugs,
but I spend time tracking them down anyway).  I work at a company that
is one of the biggest node users, and I am in constant communication
with the node core development team, as well as engineers at Voxer,
Nodejitsu, and several others.

Knowing lots of use cases is helpful, it means you can realize more quickly when mistakes were made, and add more optional functionality when the lack of such functionality would make accommodating such use-cases impossible. But that won't mean that you're producing ideal code per se.

 

When I say "pragmatic", I mean that practice and theory inform one
another, rather than being a one-way relationship, and that empirical
learnings from practice always trump conclusions drawn from theory.
That means that userland program breakage is worse than any
ideological breakage, and that context is more important than
consistency.  The "consequences of our actions" that we care about are
making life easier for those people using Node for real applications
running on real networks.

Though I do occasionally monologue for 57 pages, I am not a character
in an Ayn Rand novel, and as such, don't really care much about these
theories regarding facts, fixed laws of logic, certainty, objectivity,
or antitheses of semantics.  I happen to think that getting stuff done
in simple ways is an ideal design strategy for software.

The ideal design strategy *is* being aware of what your software fundamentally *means*. Doing the correct thing with the design of software will always produce an ideal result in the end. Semantics are how we keep track of more variables than we can keep track of in our heads (for instance, modularity and functionality are some of the tools used to deploy semantics, though of course they're not automatically good tools to use). Logic, when it has assertions, if it has assertions, is by definition correct, and computer science is an inductive science, like when someone presents evidence that the squares of sides of a right triangle are greater than the square of the hypotenuse, we don't say "Oh, huh, maybe we should re-consider the Pythagorean theorem". No, we say "Well we know that can't be right, which of the premises did we get wrong?" (Probably the one where we must use a flat plane and Euclidean geometry.) Being pragmatic about it may lead you in the right direction most of the time (repeated tests "prove" that the value of pi is 22/7... maybe a good enough estimate for most people, but by no means going to work when you venture into advanced applications, and are left wondering why your buildings are toppling over), it may even be good enough to deploy to large corporations, but it won't /continue/ to lead you in that direction, and you won't know /why/ you ended up where you did. And then you're stuck repeating the same practices that led one to unwittingly stumble across success, but in reality do not. Like my example, npm ended up using local packages not because anyone realized that's the correct way to deploy software, but because the global install system and tangle of require.paths shell files was a complete and utter train wreck.

[Austin William Wright]

On Thursday, January 24, 2013 11:51:47 AM UTC-7, Mark Hahn wrote:

tl;dr

Skimming over this thread I don't see anyone jumping on listas'/austin's bandwagon.  I, for one, certainly agree with Isaac's arguments.  I also think he has been very nice in putting in so much effort in replying to such a stupid concern.

In short, +1 for Isaac's arguments.

I very much appreciate Isaac's time and effort.

And idk, what else can I say? If being in the minority automatically made one wrong, I wouldn't be making the effort to reason like I am. Taking the time to reason your design isn't the first thing people tend to do, it's non-obvious.

Though I wouldn't say I'm the only one here, I'm jumping to the defense of at least two others here in favor of some fraction of their inquiry.

[Mikeal Rogers]

On Jan 24, 2013, at January 24, 20131:03 PM, Austin William Wright <diamon...@users.sourceforge.net> wrote:

Node.js has broken reverse compatibility three times on me, and not in a minor version update, during a *patch* update.

What broke, and in which versions? I'm not aware of compatibility changes in minor versions since pre-0.4

[Isaac Schlueter]

If it was in an unstable version, then that is by design. That's where we make API changes. But, we still try to minimize them, even there. 

[Mikeal Rogers]

oh yeah, 0.odd are all unstable and subject to change. for the most part they shouldn't be targeted by developers.

[Austin William Wright]

I'll have to dig up exactly when it was, but the most recent incident was unfortunately with crypto, in the middle of the v0.6 series, when behavior of 'binary' changed. Nothing serious, but it locked me out of my application, and (theoretically) reduced the security of my tokens by about a dozen bits (concern worthy but not much more when I'm using 256 bits or more).

Before that was in v0.4 and had something to do with events.

And another which is probably so old I'm not sure it's going back and figuring out what happened.

And now that I'm thinking of it, there's one more involving pause()ing HTTP streams. If that was during a minor version increment, it still should have been a major version increment.

Again I have no problems with improvements, but people really need to be in the habit of marking their breakages. Otherwise version numbers are meaningless, and we should move to Git commit ids (or "commits in ancestry" if you really like growing numbers).

[Miller Medeiros]

On 1/24/13 5:57 PM, "Isaac Schlueter" <i...@izs.me> wrote:

>>> We don't destabilize node-core for vocal minorities.
>> Absolutely not: If something is wrong, you change it and increment the
>>major
>> version number. That's not "destabilizing" in any sense of the word,
>>that's
>> improving functionality, and telling people that you broke reverse
>> compatibility in doing so.
>
>You're about 2 years too late, Austin.
>
>We only break reverse-compatibility when absolutely necessary, and
>never for modules that are marked with the "Stable" stability level or
>higher.
>
>There are some actual real-world programs running on node making real
>money that pays real rents. Like it or not, Node is a relevant
>platform now, not just any old program. The time for major breaking
>changes is behind us.

There are some things that can't be foreseen, you will only find out if it
works or not after broad adoption and when people use it to solve problems
of very different domains and in ways that you never imagined... I have
40+ github repositories and this feature is only a big deal in 1 of them,
if you take this as a sample it would mean that this feature is only
useful for 2% of the projects. Don't expect a "vocal majority" or
"experts" to complain about it, most people when faced with problems like
this just "hack their way around it" and move on...

>The goal of node has *always* been to get to a state where it's
>effectively "done". We want node to be like awk or sed or grep:
>something that's just there, and taken for granted.

Awk, sed and grep *sucks*, that's why there is always someone coding
alternatives (see: ag, ack, replace :P) frustration is a huge motivator
for innovation/competitors...


I hope this thread made you guys realize that some users are not happy
with the current solutions. Maybe one day some of you guys will find
yourselves on some situation where a feature like this would solve real
issues and will change your opinion, maybe not; at least I expressed my
desires and got a clear answer.


Cheers.