JAR like format for nodejs

[Jan Flyborg]

Hi,

I am wondering if there are any plans to change the module system of nodejs to make it able to import modules from a single archive (much like Java's jar which actually is a zip in disguise).

We are planning on deploying code on a space constrained device and since our node_module folder is very large, we are looking for ways to reduce its size. We can of course address this with normal minification, but if a jar-like archive format were available that would probably also help us a lot.

Any thoughts on this?

Cheers

    //Jan Flyborg

--

Sent from my mobile device. Please excuse my brevity.

[Daniel Lo Nigro]

I'd like to have something like this as well. 

In the meantime, the closest thing you can get is probably combining and minifying all your JavaScript. If you combine your app into a single .js file using a tool such as Webpack, it avoids the need to deploy the node_modules directory (assuming there's no additional non-JS files in there that are required).

[Alain Mouette]

You could try
  Npm dedupe

-----
Alain Mouette

--
Job board: http://jobs.nodejs.org/
New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md
Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
To post to this group, send email to nod...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/CACrfS1AcaqsM2EjicTtsJMxC3PG0wxCaE%3DDSvK4JCvCGM%2BrWVA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Jeremy Darling]

Since you can overload the way that Require works you don't actually need a "change" to Node you can just add in your own loader, as an example here is a very simple YAML loader:

const fs = require('fs');

const yaml = require('js-yaml');

require.extensions['.yaml'] =

require.extensions['.yml'] = (module, filename)=>{

  const content = fs.readFileSync(filename, 'utf8');

  module.exports = yaml.load(content);

};

You could do something similar with whatever and have it basically decompress say a Zstandard archive or something.  I only suggest Zstandard as I've seen great compression with it against extremely large JSON structures, so would probably excel at JS as well.

--

Job board: http://jobs.nodejs.org/
New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md
Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
---
You received this message because you are subscribed to the Google Groups "nodejs" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscribe@googlegroups.com.

[Jan Flyborg]

I did not know that it was that easy to plug in new functionality in the module loader. With this new information it seems very doable to package an application with Rollup (and have the benefit of their tree-shaking and other goodies), and then write a new module loader that can load the few native add-ons it cannot handle (and a wrapper around the stuff that Rollup  has generated, that first loads this new module loader).

I will do some experimentation tomorrow and see if I can get it to work for our project.

fre 21 okt. 2016 kl 20:25 skrev Jeremy Darling <jeremy....@gmail.com>:

Since you can overload the way that Require works you don't actually need a "change" to Node you can just add in your own loader, as an example here is a very simple YAML loader:

const fs = require('fs');

const yaml = require('js-yaml');

require.extensions['.yaml'] =

require.extensions['.yml'] = (module, filename)=>{

  const content = fs.readFileSync(filename, 'utf8');

  module.exports = yaml.load(content);

};

You could do something similar with whatever and have it basically decompress say a Zstandard archive or something.  I only suggest Zstandard as I've seen great compression with it against extremely large JSON structures, so would probably excel at JS as well.

On Thu, Oct 13, 2016 at 9:12 AM, Jan Flyborg <jan.f...@gmail.com> wrote:

Hi,

I am wondering if there are any plans to change the module system of nodejs to make it able to import modules from a single archive (much like Java's jar which actually is a zip in disguise).

We are planning on deploying code on a space constrained device and since our node_module folder is very large, we are looking for ways to reduce its size. We can of course address this with normal minification, but if a jar-like archive format were available that would probably also help us a lot.

Any thoughts on this?

Cheers

    //Jan Flyborg

--

Sent from my mobile device. Please excuse my brevity.

--

Job board: http://jobs.nodejs.org/
New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md
Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
---
You received this message because you are subscribed to the Google Groups "nodejs" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.

To post to this group, send email to nod...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/CACrfS1AcaqsM2EjicTtsJMxC3PG0wxCaE%3DDSvK4JCvCGM%2BrWVA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Job board: http://jobs.nodejs.org/
New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md
Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
---
You received this message because you are subscribed to the Google Groups "nodejs" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.

To post to this group, send email to nod...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/CAAhs7EgwZEFJZcr7rF%2B0XcS688UAxPpgP-p2goxoYgiSxvEJPA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Jan Flyborg]

Thanks for this.

We have tried this, but as soon has you have add-on modules, things starts to get complicated. I have made some experiments with webpack and Rollup and while distributing pure JS projects is not a problem, AFAIK neither of them support the loading of add-ons, since how that procedure is carried out is up to each package maintainer to decide. Often the require statements contains logic that builds up the path to the *.node files programmatically and those are not easily parsed.

This is doable since other languages has succeeded with solving the same problem. For example does PyInstaller (http://www.pyinstaller.org/) contain logic to cope with quirks like these for most common Python packages. It is simplier on Python though, since you cannot load a module from a given path (without using tricks like changing the sys.path variable).

Cheers

    //Jan

--

Job board: http://jobs.nodejs.org/
New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md
Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
To post to this group, send email to nod...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/42ae3c4f-8b05-4ba7-945a-d88e7c8d0a3b%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jan Flyborg]

Thanks for the tip. This gives me like a 25% reduction in size for our project which is very valuable to us.

Cheers

    //Jan

To view this discussion on the web visit https://groups.google.com/d/msgid/nodejs/157bef31828.2782.962c06d43ba9274da034cf7d02472879%40bonseletrons.com.br.

For more options, visit https://groups.google.com/d/optout.

[Matt]

Aside from the other tips, have you considered using a filesystem that supports transparent compression? I have no idea what system you are deploying to, but NTFS, ZFS and Btrfs support this.

--

Job board: http://jobs.nodejs.org/
New group rules: https://gist.github.com/othiym23/9886289#file-moderation-policy-md
Old group rules: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
---
You received this message because you are subscribed to the Google Groups "nodejs" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+unsubscribe@googlegroups.com.