what are the merits / demerits of require.extensions?

[~flow]

i recently opened an issue https://github.com/joyent/node/issues/5430
concerning require.extensions, and got told that

"People should not be using require.extensions. It's officially deprecated", "Compile your code to JavaScript prior to running it." "There is never any need for additional filetype extensions. Node runs JavaScript"; "we're not bothered with tens of other dialects that require compilation and are written in code that's not readable for JavaScript programmer"; "stop relying on this horrible feature"

in no unclear words. i was a little shocked, since my perception has always been that require.extensions was the strike of a genius. with little effort, you can hook in filetypes and make it so that they are require'd transparently, no matter whether they represent data / programs as javascript, json, coffeescript, whatever.

ok it's a global hook which, in theory might lead to problems (but in years of writing coffeescript hasn't been a problem even once.

i love the fact that i do not have to compile everything in advance / add a buildscript / are forced to keep a coffeescript watch process in the background. coming from python, i was also very happy to see that those pernacious *.pyc files that used to litter my directories were now a thing of the past—i mean, that is code duplication enforced by the system, utterly avoidable.

to me, javascript is a wonderful language with some rough edges and a horribly cluttered syntax. now here comes nodejs and all those wonderful home-grown programming languages that take advantage of the great compiling target that javascript running on nodejs is. dumbing down `require` will be sad news for all the many people that are using those new languages daily.

as a user of coffeescript, the fact that coffeescript compiles to javascript is a fact that i have to be aware of, but it is not something that i want to be (or need be) constantly reminded of. doing on-the-fly, transparent compilation is the way to go; it has never been any appreciable drain on resources, either. it sure will continue doing things that way.

if they kill require.extensions with no sensible replacement, things will just get more difficult. it won't be too difficult to come up with a sensible replacement—one could even predict that sooner or later, there will be modules in npm that will allow you to `require 'old-require'` to replace the new require with the good ole'. gone a bit of efficiency, gone a bit of standardization.

so what are the good, the bad and the ugly facts about require.extensions?

[Mark Hahn]

I have written node code in nothing but coffeescript for over two years.  

I have never been tempted to use that hook.  IMHO using a build process is natural and coffeescript compiling is just a small part of that process.  I never see the .js files.  My editing environment is clean with only .coffee files.  The files are compiled transparently on the server with no effort or visibility on my part.  And now I don't even see the JS code in the chrome inspector.  I'm JS free.

[Eldar]

I also don't get what's cool about compiling in advance for private modules. It's just an addition of pain and complexity. All reasons for deprecation of require.extensions are illegal for me. If core team decided to fight against coffescript modules in npm they can do that in many other ways.

[Ben Noordhuis]

You're essentially saying that deprecating require.extensions makes
your life as a module author harder, right? That's the wrong way to
look at it: it makes life for _users_ of your module easier.

The quintessential example is where someone has a project that depends
on two modules, both written in FooScript(TM), but with module A
depending on foos...@1.0.0 and module B depending on
foos...@2.0.0.

require.extensions is global; if FooScript 1 and 2 are incompatible,
then your user is between a rock and a hard place - there is no way
for him to use both modules at the same time.

That's why we _strongly_ recommend that you compile to JS before
publishing to npm. You are doing your users a disservice if you
don't.

[Eldar]

No one argues about modules for npm. What about apps? What about components private for organization?

There is never any need for additional filetype extensions. Node runs JavaScript. Ultimately, you have to compile it to JS before it runs anyway, so you may as well do that up front, rather than adding a global hook for it.

Ok, Then we'd like to see setups as simple as `mocha --requrie FooLang` for running tests and `node ./support/foo script.foo` for everything else. 

[Sam Roberts]

On Fri, May 10, 2013 at 7:02 AM, Eldar <elda...@gmail.com> wrote:
> No one argues about modules for npm. What about apps? What about components
> private for organization?

Don't you package as modules and publish them to a private npm repo?

[Eldar]

It doesn't matter if there is no problem mentioned by @Ben

[Bruno Jouhier]

+1

The context is different for people building libraries and people building apps.

We are building an app and not having require.extensions would make our dev life more difficult because we would have to deal with the generated JS files somewhere in our source tree. We cannot just blindly exclude .js files from source control because we have a mix of .js, ._js, .coffee, etc. modules - sometimes in the same directory.

But there is an even bigger problem for us: we need to distribute some of our source files in encrypted form. How could we do it without require.extensions?

Bruno

[Alexey Petrushin]

> it makes life for _user_ of your module easier

Everyone say the same mantra again and again - "easier for other users" - I care for the comfort for other users, but I care for my own comfort even more.

If some methods make my life easier but life of other users that use my modules harder - well, that's sad, but why should I solve it at the cost of my own convenience?

[Ben Noordhuis]

I was reasonably sure someone was going to post the reply you did.

If you don't care much about users of your module, sure, that's fine.
Just don't expect us to care much about you*. Only fair, right?

I'm not going to repeat the reasons why require.extensions is bad,
just that I fully support Isaac's move of deprecating it.

* Generic you, as in require.extensions users.

[Alex Kocharin]

I agree with that. npm modules should be compiled to js. I even suggest them to be concatenated and minified for performance reasons, why don't anybody do that?

But it's NOT about published npm modules. It's about developing. When I develop my coffeescript libraries (see https://github.com/rlidwka/asset-pipeline for example), I put 'module.exports = require("./src/index");' in index.js and using on-the-fly compilation. On production system I change it to require("./lib/index") to use compiled version.

So, how am I supposed to do this without require.extensions? Oh... well, you're right, it'll probably be require("good-old-require")("./src/blablabla.coffee") with a help of a library that would reimplement all filesystem searching/reading stuff by itself all over again. 

How will require("something.json") be dealt with anyway?

[Bruno Jouhier]

I'm not going to repeat the reasons why require.extensions is bad, 
just that I fully support Isaac's move of deprecating it.

What about people who need to encrypt source code. What do you propose?
 

[Alex Kocharin]

 

How do you do that? Do you use some kind of C++ native module for this? Because from I've seen so far, most "encrypted" javascripts can be "decrypted" by replacing eval with console.log...

 

--
// alex

 

 

11.05.2013, 14:30, "Bruno Jouhier" <bjou...@gmail.com>:

I'm not going to repeat the reasons why require.extensions is bad, 
just that I fully support Isaac's move of deprecating it.

What about people who need to encrypt source code. What do you propose?
 

 

--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
 
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Bruno Jouhier]

Yes, we do it with a C++ module. And fn.toString() won't give you the source of exported functions.

[Alex Kocharin]

Can you post a hello-worldish example of such working encrypted module? I'm just curious how it is possible to use such module without source code appearing somewhere.

[Bruno Jouhier]

Sorry but I cannot do it. You'll have to buy our product to get it :-(

There is a mix of encrypted and non-encrypted modules. The loader and the modules that we share between browser and server are not encrypted.

[Alexey Petrushin]

> If you don't care much about users of your module, sure, that's fine. 
> Just don't expect us to care much about you*.  Only fair, right? 

Will be with one additional detail. I don't hide my intentions and declared my priorities openly and honestly. To be fair you should also do the same and also declare it clearly. 

So, official politics of yours (what exactly did you meant by "us"?) is to not care much about users of other dialect in node.js?

[~flow]

for some reason my previous post appears to have get lost.

so let me say again that the one reason to remove require.extensions that i fully dig is: it's global. meaning that #1 you can't have, say, one package written in coffeescript use one version of that language and another package use another version of that language, as long as they share extensions. likewise, when loading a certain module introduces a new extension `.foo`, then that might clash with the semantics of `.foo` files in another package. all these conflicts are fine and to be expected, but they can not be solved using a global require.extensions.

let me again say that i am all for using languages that transpile to javascript, and i am all for the renewed interest for homegrown languages—it is exciting to see that!

i find the often-aired disgust at distributing modules in languages other than javascript a bit narrow-minded. after all, one great thing about having an interpreted / a scripting language is that—and this is an important point!—you do not have to 'compile' anything in the traditional meaning of the word, and also that you have a vm with known properties; this means that you can (1) program in a platform-independent way, (2) keep the semantics of your code open for everyone to read.

in the C world, the code you write is pretty remote from the code you execute, and the code you execute (the 'binaries', aptly called so for their being 'unintelligible' to the average human programmer) is devoid of all the names and the higher-level constructs that the programmer introduced. javascript, perl, python—these languages are great because they literally 'execute what you wrote'.

when people choose coffeescript over javascript, they might do so because they see value in the clarity of its syntax (i certainly do). i certainly believe that a transpiling language should strive to retain that clarity (readability) as much as possible when producing their javascript target. still, i do not want to bother about that representation of my code too much, and having it done invisibly, on the fly, in memory is just fine for me.

nodejs is a very young community, and it has so far never failed to amaze me how people in this community arrive at very sensible, pragmatic, and open-minded solutions and conventions that empower people rather than putting them on a very narrow track (java comes to mind). so lest this thread veers off to shouting 'just compile before you publish' and discussing obfuscation techniques, let us rather discus:

given that it is a good thing that new languages are sprouting up—languages that may be beneficial in their expressiveness for computing at large and getting stuff done;
given that this expressiveness is there for humans to read;
given that this expressiveness can always be verified / inspected / reasoned about in a unified way when needed—by explicit translation into the known entity that is javascript;
given that it has been shown that it is feasible and of utility to have the translation be done transparently, in the background;
given that, quite possibly, for many in the transpiling community the blunt call to 'compile before you publish' is, at best, problematic and, at worst, exclusivist;
given that require.extensions is problematic for being a global:

what conventions / best practices / techniques can we come up that will allow us to 'publish in my language, run as javascript' the day that require.extensions is gone?

i envision that i will put a standardized, constant index.js in each of my module folders that will do the bootstrap transpiling business quite transparently. but i'm not sure what do to when i just want to require a single sub-module (say, `require 'timezones/berlin.coffee'` as a stupid example).

[Eldar]

i envision that i will put a standardized, constant index.js in each of my module folders that will do the bootstrap transpiling business quite transparently

That's wrong because

1. That means that additional typically huge dependency must be added
2. Compiler may just don't work in a target environment.

Once again and finally I'd like to clarify the claim to the core team:

1) require.extensions is an easy way to define custom executables that support additional languages, e.g. coffeenode, wispnode, sagenode, etc.

Then coffeenode@1.0 relates to coffeenode@2.0 like ruby@1.8 to ruby@2.0 or like python@2 to python@3. It's hard to see what's wrong with that.

2) The above post shows that deprecation of require.extensions just fails to solve the problem it's intended to solve.

[Ben Noordhuis]

On Sat, May 11, 2013 at 9:05 AM, ~flow <wolfga...@gmail.com> wrote:
> what conventions / best practices / techniques can we come up that will
> allow us to 'publish in my language, run as javascript' the day that
> require.extensions is gone?

What seems to be getting lost in this discussion is that
require.extensions is not actually going away, certainly not in the
near future and probably never.

Deprecation in API-locked parts of node.js core means "you really
shouldn't use this", not "we'll remove it whenever the fancy strikes
us."

[José F. Romaniello]

It will be good to have a log with a warning.. " some module is overriding your require.extension" for the reasons you described before.

[Bruno Jouhier]

Cool cause I need this to work forever: https://github.com/bjouhier/node-lol

OTHERWIZ DAZ NO FUN!

Sorry for my lolcat, it's a bit weak.

Bruno

[Isaac Schlueter]

My thoughts on the matter are articulated in the issue linked in the
OP. Also everything that Ben has written in this thread is 100%
correct.

Furthermore:

We only remove/break deprecated APIs if they're in the way. If it's
buried in a locked module, then that's highly unlikely. (Someone
would have to find an extremely severe bug that could only be fixed by
removing require.extensions.)

However, we also aren't likely to *fix* any issues that you have with
that deprecated API. That's the issue that started this round of
require.extensions discussion. The only change that we'd make to it
is to remove it or add a deprecation warning, and we won't be doing
that, since there's no reason to.

In the near future, the module rating system that we're building will
penalize any module that uses deprecated APIs. It will be ranked
lower in searches, have a smaller score next to its name, and perhaps
even print a warning when it is installed by npm.

If you would like to ensure that your program is not doing anything
stupid, you can use the `node-strict` module, and add
`require('node-strict')` to your main file. It's still a work in
progress, but it prevents require.extensions and other poor choices.

[Brad Carleton]

As an author of a couple of coffeescript projects in node, I agree the code that goes into npm should NOT be coffeescript.  That being said when you are developing an application, adding an explicit build step is not fun.  I mean I wouldn't want to see all the assembly or binary files that my javascript code could be compiled into.  

I think other dialects of javascript are a great thing.  There are a lot of developers from varying communities ruby, python, .NET, java, who may never come over to the node/javascript world unless these transpiled languages exist.  I came from python, and being able to walk into coffeescript felt vary comfortable for me.  Now, I contribute to the node community, but I don't write javscript.  Yes, in an ideal purist world everyone writes javscript, but then there are less people in the community because people tend to be biased towards languages.  Beyond that, some people are just more or less productive in some languages versus others.

It would be nice if the wonderful core guys would tell us that it's cool to use transpiled languages in our applications, as long as we keep it out of npm.  Furthermore, if they truly don't want to support "require.extensions", they should at least be supportive of userland setting up a hack to bring it back.  I mean that would be dead simple.  Replace the global require with a custom object that adds the require.extensions functionality back, and then package it up in an npm module, called "good-ole-require"!

[Isaac Schlueter]

> It would be nice if the wonderful core guys would tell us that it's cool to use transpiled languages in our applications, as long as we keep it out of npm.

We are telling you it's fine to use transpiled languages as long as
you keep them out of Node. Transpile them to JavaScript. Run
JavaScript in Node.

Relying on require.extensions is explicitly not encouraged, nor
supported. It has known bugs *today*, which will not be fixed, ever.
It is very brittle and cannot be made any less brittle. It is a
global switch by which one module can introduce subtle bugs in another
module. It is tight coupling and unnecessary run-time complexity.
It's everything we try to *avoid* in Node.

Take it up with the CoffeeScript maintainers to stop relying on and
encouraging this dangerous, unsafe, unwise, deprecated practice. Even
long-time coffeescript users are teling you here to avoid using the
require hooks. The "build step" takes milliseconds, and can be set up
to run in the background automatically. There is absolutely zero
reason to insist on using techniques that are known to be problematic,
and even less to insist that we support them.

Of course, it's your computer, it's your application, you can do
whatever foolish thing you want. But if you're looking for my
blessing, then don't do foolish things. It would be irresponsible of
me to say this is OK when I know it isn't, since I see the fallout
from the bugs it causes.

[Brad Carleton]

That seems like a strongly worded, no.

Maybe I'm unsafe, foolish, degenerate, and lacking in basic social skills (most likely true), but I still don't see the point.

By their very nature most of the compilers run in Node, so the idea of keeping them out of Node is kind of a moot point.  Which means you have two paths:

Current:

Filesystem -> Node A (transpiles)  -> Node A (program)

Future:

FIlesystem: -> Node A (transpiles) -> Filesystem -> Node B (program)

If I distribute all of my code as javascript, I just don't see why in my own personal environment whether dev or on the server, that I wouldn't choose the first option there.  Especially, if from my viewpoint, "coffeescript", "typescript" or whatever is the first class language, not javascript.

And you can say it's bad and there are bugs, which is cool, but if you guys won't support it, I can pretty much guarantee you that userland will bring it back anyway.

[Mikeal Rogers]

On May 12, 2013, at 3:20PM, Brad Carleton <br...@techpines.com> wrote:

If I distribute all of my code as javascript, I just don't see why in my own personal environment whether dev or on the server, that I wouldn't choose the first option there.  Especially, if from my viewpoint, "coffeescript", "typescript" or whatever is the first class language, not javascript.

And you can say it's bad and there are bugs, which is cool, but if you guys won't support it, I can pretty much guarantee you that userland will bring it back anyway.

This is exactly what he's saying.

It won't be supported in core in a way that is less buggy than today, it is now deprecated, and it *belongs* in userland, which "not in core" means.

-Mikeal

[Brad Carleton]

Cool, that makes sense.  

"require.extensions" = bad

whatever userland wants to do = good (maybe)

[Floby]

for those  who need a quick and available everywhere solution, this is the Makefile line for coffeescript.

%.js: %.coffee

    coffee -c $<

[Bradley Meck]

Encrypting source code is not something that can be done in v8 due to Function.toString and how v8 stores code for recompilation. The source will be present in v8's heap somewhere. However, if all JS is encrypted with a secret key and that must be provided before v8 starts it is possible to cause difficulty in obtaining actual source code. That and turning off the debugger are a must, but even in C++ you can disassemble and decompile to a somewhat functional point these days.

[Brian Takita]

Deprecating `require.extensions` is unfriendly to developers who want to require a template file.

No, I don't want to have to add more layers of complexity to my build process, thank you. I just want to use *custom* logic for my app/platform. I don't have other users, it's my sandbox, thank you. Please don't tell me what's good for me, I know what's good for me, thank you!

[Zlatko]

On Sunday, October 11, 2015 at 6:37:16 AM UTC+2, Brian Takita wrote:

Deprecating `require.extensions` is unfriendly to developers who want to require a template file.

No, I don't want to have to add more layers of complexity to my build process, thank you. I just want to use *custom* logic for my app/platform. I don't have other users, it's my sandbox, thank you. Please don't tell me what's good for me, I know what's good for me, thank you!

 Now, that sounds exclusivist, too. I could say something like "you don't want to introduce complexity to your build process? Well me too, I prefer to keep your .coffee (or whatever) bugs out of mine."

But that is not constructive, so I'll proceed with an example.

I write ES6 code, the whole team does here. I even write my tests in ES6. And we don't see the .js files, either. We have transpilers as part of our build and test process, and I don't see any obstacle with that.

You do something like style checkers, linters, documentation compilers when you develop, right? Maybe a small browser syncing server, if you're full stack? Putting a transpiler step in one of those steps is a trivial thing to do. You do it already, anyway, so another step or two, another few milliseconds should not matter.

Yeah, yeah, I know people are saying they only use it for development, right? Just like people only don't write tests for development, right? Only tested modules get published to npm. I know there would be gains for people writing in other languages then JavaScript by having a full support of require.extensions. But there are also clear advantages of not doing it. The main thing - Node.js core will remain much more stable and with less bugs in the system overall.

You know of JRuby, Jython and other languages that run on JVM? I would take a hint there. I think that if this is really a useful feature, sooner or later a native module or addon or similar will appear that will resolve this. So, the core remains clean, the addon is clear in what it does, and everything works as intended. That's how I would like this to play out.

Though, I'm not the core maintainer, not even a contributor, so my complains and wishes wouldn't count anywhere as it is, would they?

Zlatko