Porting fs to the browser via browserify

[Adam Crabtree]

Browserify's pretty sweet. Not because of anything special that it's doing to make it work (though there's plenty of that to go around), but because of what it reveals about the nature of node.js and the implications it has for browser-based JavaScript, specifically that node core is heavily written in JavaScript. (Shocker I know)

In other words, a lot of core functionality just works with no change in the browser like, Streams, util.inherit, path, etc... and several with only moderate tweaks like EventEmitter and Buffer. If you don't believe me, just compare the files in the following two directories for a possibly illuminating experience:

https://github.com/joyent/node/tree/master/lib

https://github.com/substack/node-browserify/blob/master/builtins/

Anywho, I've been playing around with the new FileSystem API in the browser recently,

http://www.w3.org/TR/file-system-api/
http://www.html5rocks.com/en/tutorials/file/filesystem/

but I can't shake just how unpleasant it is compared with node's, which got me thinking, how much work would it take to port fs to the browser? It turns out, the answer is simultaneously not much, and quite a lot actually. Let me explain...

Since fs.js is written entirely in JavaScript, and since node doesn't augment V8, porting fs to the browser requires only that we to port the node C++ bindings utilized in fs.js and add process.binding support to browserify. Though I haven't gone too far in depth yet, it seems quite doable. The bindings are exposed via `process.binding('fs')`. See the following thread:

https://groups.google.com/group/nodejs-dev/browse_thread/thread/5699149204ed0445/fc280b6509b57cc6?lnk=gst&q=browser+fs#fc280b6509b57cc6

For convenience, I've copied the list of functions and constants needing to be ported here:

// from process.binding('fs') 
[ 'Stats', 
  'close', 
  'open', 
  'read', 
  'fdatasync', 
  'fsync', 
  'rename', 
  'truncate', 
  'rmdir', 
  'mkdir', 
  'sendfile', 
  'readdir', 
  'stat', 
  'lstat', 
  'fstat', 
  'link', 
  'symlink', 
  'readlink', 
  'unlink', 
  'write', 
  'chmod', 
  'fchmod', 
  'chown', 
  'fchown', 
  'utimes', 
  'futimes', 
  'StatWatcher' ] 

// from process.binding('constants') 
{ 
O_RDONLY: 0, 
  O_WRONLY: 1, 
  O_RDWR: 2, 
  S_IFMT: 61440, 
  S_IFREG: 32768, 
  S_IFDIR: 16384, 
  S_IFCHR: 8192, 
  S_IFBLK: 24576, 
  S_IFIFO: 4096, 
  S_IFLNK: 40960, 
  S_IFSOCK: 49152, 
  O_CREAT: 512, 
  O_EXCL: 2048, 
  O_NOCTTY: 131072, 
  O_TRUNC: 1024, 
  O_APPEND: 8, 
  O_DIRECTORY: 1048576, 
  O_NOFOLLOW: 256, 
  O_SYNC: 128, 
  O_SYMLINK: 2097152 

} 

So the question to the community is....

Who's with me?! =D

For the curious, Substack's on board with including the eventual (assuming it works) builtin into browserify. 

My fork of node-browserify for fs.js development (to eventually submit a pull request) (there's nothing there yet):

http://github.com/CrabDude/node-browserify

I'm also interested in porting the zlib module to the browser as well, but that's another thing entirely.

Cheers,

Adam Crabtree

--
Better a little with righteousness
       than much gain with injustice.
Proverbs 16:8

[Tim Caswell]

Why port it at such a low level?  Wouldn't it be easier to just re-implement the same API as node exposes publicly?

Also, FYI, when I get back to working on LuvMonkey, I intend to make it API compatible with the current V8 based node, I haven't decided how much of node's C++ API I want to mirror internally though.

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

[Adam Crabtree]

That's one approach, but honestly my ultimate goal / desire is to utilize existing npm modules in the browser that may depend upon fs. If I mirror at the high-level API, it seems like it would be far more prone to edge cases, and I'll ultimately be using a lot of fs.js anyways to get there.

The fs bindings on the other hand are very low-level and simple, and in cases where there's no browser equivalent, they can simply be a no-op or some appropriately acceptable default. 

The reason I mentioned porting zlib too, is because I want to be able to use the ZipStream module as is, in the browser and stream the zip to the filesystem.

Maybe I'm wrong though, and it would be easier to mirror the higher-level APIs. Initially though, this doesn't seem to be much of an issue. I've already ported fs.writeFile, which only required porting bindings.open, write and close. =)

Cheers,

Adam

[Mikeal Rogers]

On May 22, 2012, at May 22, 20121:31 PM, Tim Caswell wrote:

Why port it at such a low level?  Wouldn't it be easier to just re-implement the same API as node exposes publicly?

In some cases this is actually what browserify does. It can browserify modules that require('http') so clearly it has code that reimplements the http client API on top of XHR.

[substack]

On Tuesday, May 22, 2012 12:55:48 PM UTC-7, Adam Crabtree wrote:

For the curious, Substack's on board with including the eventual (assuming it works) builtin into browserify. 

My fork of node-browserify for fs.js development (to eventually submit a pull request) (there's nothing there yet):

http://github.com/CrabDude/node-browserify

It would also be nice if this new development was its own module instead of a fork in the same way that http-browserify, vm-browserify, and crypto-browserify are all their own modules. 

[Alexey Petrushin]

I don't exactly understand - it seems for me that there are two separated projects:

- browserify - build tool to serve JS, Templates, NPM modules and other assets.

- node.js environment.

So, wouldn't it be better to separate it? As right now Node.JS and NPM package system is different projects (although they shipped together).

By the way You can use node.js browser environment even without browserify (and maybe browserify without node.js environment).

[Louis Santillan]

I suggest taking a look at NestedVM (http://nestedvm.ibex.org/) which does something similar for java byte code within a JVM

-L

[billywhizz]

+1 for emulating the low level bindings

+1 for making this a project independent of but compatible with browserify

[Adam Crabtree]

At first blush decoupling this into fs-browserify makes sense, but ATM the fork is really for my own convenience / out of necessity. Porting fs to the browser requires a couple things that aren't currently, to my knowledge, available in browserify: process.binding support, TypeArrays in Buffer.js / SlowBuffer port to the browser, and then of course process.binding('fs'). The fs module needs all of these to work. Of course, Substack's not going to turn down someone porting SlowBuffer to the browser I'm sure, it still remains to be done, thus the fork. 

So I suppose the question is really how do we want to do this.

Here's the link to the eventual fs-browserify:

http://github.com/CrabDude/fs-browserify

I've also added an issue to browserify to come up with a convention for process.binding:

https://github.com/substack/node-browserify/issues/148

Cheers,

Adam

--

Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en