PhantomJS 1.3 FileSystem API feedback

[James Roe]

Hi,

For PhantomJS 1.3 we're working on a FileSystem API, which allows you
to do things such as read files, open files, save files, read
filesystem paths, use symlinks, amongst a lot of other nifty
features. :)

We would like some feedback from our user base. Please let us know
what YOU think is missing, what you'd like to see, and what you think
isn't needed in the API.

For this API, we're following CommonJS specs*.

Our current plans for the API are:

Attributes
size - returns the size of a file in bytes
lastModified - returns the time that a file was last modified

Files / Directories
copy - copy a file / directory from src to dest
move - move a file / directory from src to dest
rename - rename a file / directory

Directory
copyTree - copy a directory tree
makeDirectory - make a directory
makeTree - make a directory tree
removeDirectory - remove a directory
removeTree - remove a tree of directories

Files
open - open a file and return an object which you can manipulate to do
different actions
- atEnd: find if the file is at the end or not
- close: close the file
- flush: write all buffered output into file
- read: read the entire files contents
- readLine: return the next line of the file
- write: write data to file
- writeLine: write the next line of the file
read - read a files contents
write - write to a file
newline - get the native lineending (scheduled for removal)
remove - delete a file / directory

Listing
list - get an entire directories list of files / directories
listTree - list an entire tree (schedules for removal)
listDirectoryTree - list an entire directory tree (scheduled for
removal)

Links
symbolicLink - create a symbolic link
hardLink - create a hard link
readLink - read a symbolic link

Paths
absolute - return an absolute path to pathname
base - returns the part of the path that is after the last directory
separator. If an extension is provided and is equal to the file's
extension, the extension is removed from the result.
canonical - returns the canonical path to a given abstract path.
changeWorkingDirectory - change the working directory
directory - returns the path of a file's containing directory, albeit
the parent directory if the file is a directory.
extension - returns the extension of a file.
join - takes a variadic list of path Strings, joins them on the file
system's path separator, and normalizes the result.
normal - removes '.' path components and simplifies '..' paths, if
possible, for a given path.
normalCase - Normalize the case of a pathname. On Unix and Mac OS X,
this returns the path unchanged; on case-insensitive filesystems, it
converts the path to lowercase. On Windows, it also converts forward
slashes to backward slashes. (scheduled for removal)
workingDirectory - return the current working directory (CWD)
relative - returns the relative path from one path to another using
only ".." to traverse up to the two paths' common ancestor. If the
target is omitted, returns the path to the source from the current
working directory.
separator - return the native path separator
split - returns an array of path components.

Permissions
changeGroup - change the group of the path to groupName or GID
changeLinkGroup - same as above but doesn't follow symlinks
changeOwner - change the owner of the path to ownerName or UID
changeLinkOwner - same as above but doesn't follow symlinks
changePermissions - change permissions on a path
changeLinkPermissions - same as above but doesn't follow symlinks
group - return the group name and GID of a path
linkGroup - same as above but doesn't follow symlinks
owner - return the owner name and UID of a path
linkOwner - same as above but doesn't follow symlinks
permissions - return the permissions of a path
linkPermissions - same as above but doesn't follow symlinks

Tests
exists - find out if a path exists
isAbsolute - test that path is absolute or not
isDirectory - test that path is directory or not
isExecutable - test that path is able to be executed (using users
permissions)
isFile - test that path is a regular file
isLink - test that path is a symlink
isMount - test that path is on a mounted volume
isReadable - test that path is able to be read (using users
permissions)
isWritable - test that path is able to be written to (using users
permissions)
linkExists - same as exists but doesn't follow symlinks
same - test that pathA is same file as pathB

If you'd like more information on how these work, you are welcome to
see an already implemented draft API in PyPhantomJS**, and also
CommonJS specs*.
* http://wiki.commonjs.org/wiki/Filesystem/A
** https://github.com/ariya/phantomjs/blob/master/python/pyphantomjs/filesystem.py

[Ariya Hidayat]

Thanks for starting this thread!

A lot of CommonJS specs/API is just "convenience" for the purpose of
web testing with PhantomJS. Combining PhantomJS with external
script/shell will solve the problem, albeit less elegant.

I suggest coming up with a strict small set of API that answers the
following question: "Which IO functions are very critical for web
testing?".

For example, I can imagine that file open, read, write, and close
would be absolutely critical. However, things like symbolic link,
permission, file copy, etc are not show stopper at all. Nothing stops
someone from using PhantomJS together with other scripting languages
for more post-processing.

PhantomJS #1 goal is always headless WebKit and not general purpose
JavaScript shell.

--
Ariya

[Ivan De Marino]

I have to say that I broadly agree here.

There is still a bit of API I'd rather have in addition to the "minimal set" described by Ariya, but nothing substantial.

I think the risk here is "let's add it just because we can".

And this is bad, because it drives PhantomJS in lands where it shouldn't.

I mean, think about it: if we go "all in" with the FS api, and at the same time, someone makes a module for Node.js to manipulate WebPages, suddenly PhantomJS becomes a "cut down version of Node.js". Or, at least I'd see it that way.

I prefer to keep it "small and simple", with major emphasis on making "scriptable" as much of the browser as we can.

--

Ivan De Marino

Front-End Developer @ Betfair

email: ivan.de...@gmail.com | detron...@gmail.com | ivan.d...@betfair.com

web: blog.ivandemarino.me | www.linkedin.com/in/ivandemarino | twitter.com/detronizator

mobile: +44 (0)7515 955 861

[Ariya Hidayat]

> There is still a bit of API I'd rather have in addition to the "minimal set"
> described by Ariya, but nothing substantial.

Can you list what you'd like to have?

--
Ariya Hidayat
http://www.linkedin.com/in/ariyahidayat

[Ivan De Marino]

I can't right now but I'll be on that this evening. I have to do a
second hand of paint to my balcony Walls ;)

I have been on hold this past days, waiting for more feedback on this
topic, but seems like only 3/4 of us care. So we should just move on
and do it.

Later
Ivan

Ivan De Marino
Front End Developer @ Betfair

Sent from my iPhone 4

[Ivan De Marino]

So, painting done, dinner as well, it's time to list what API, in my opinion, should be part of PhantomJS, out of the ones defined in CommonJS/Filesystem currently.

I marked with "DONE" what is already on the upstream repo :)

Attributes 
- size()
- lastModified()

Files / Directories
- copy()
- move()
- rename()

Directory
- copyTree()
- makeDirectory()     DONE
- makeTree()     DONE
- removeDirectory()     DONE
- removeTree()     DONE

Files
- open()     DONE
fs:
   - atEnd()     DONE
   - close()     DONE
   - flush()     DONE
   - read()     DONE
   - readLine()     DONE
   - write()     DONE
   - writeLine()     DONE
- read()     << this is for very lazy coders :)
- write()    << this is for very lazy coders :)
- remove()     DONE

Listing
list()     DONE

Paths
- separator()     DONE
- changeWorkingDirectory()     DONE
- workingDirectory()     DONE

Tests 
- exists()     DONE
- isAbsolute()
- isDirectory()
- isExecutable()
- isFile()
- isLink()
- isMount()
- isReadable()
- isWritable()

I removed from James' list based on the following questions:

- can JavaScript do it already?

- should this be done by Bash Script or "full programming languages"?

What do you guys think?

Ivan

[Ivan De Marino]

Not at all.

That is totally unrelated to this.

This is about Filesystem interaction.

On 13 July 2011 21:20, Scott Rosenthal <sc...@castlepointsolutions.com> wrote:

Does the above cover the scripting of file uploads to a web app in the
dom, i.e., ajax?

For example, a web app that is using an ajax based multiple file
upload, phantomjs allows parsing the dom for the next element to
script files locally to be uploaded...

On Jul 10, 4:45 pm, Ivan De Marino <detroniza...@gmail.com> wrote:
> So, painting done, dinner as well, it's time to list what API, in my
> opinion, should be part of PhantomJS, out of the ones defined in
> CommonJS/Filesystem currently.
>
> I marked with "DONE" what is already on the upstream repo :)
>

> *Attributes *
> - size()
> - lastModified()
>
> *Files / Directories*

> - copy()
> - move()
> - rename()
>

> *Directory*
> - copyTree()
> - makeDirectory()*     DONE*
> - makeTree()*     DONE*
> - removeDirectory()*     DONE*
> - removeTree()*     DONE*
>
> *Files*
> - open()*     DONE*
> *fs:*
>    - atEnd()*     DONE*
>    - close()*     DONE*
>    - flush()*     DONE*
>    - read()*     DONE*
>    - readLine()*     DONE*
>    - write()*     DONE*
>    - writeLine()*     DONE*

> - read()     << this is for very lazy coders :)
> - write()    << this is for very lazy coders :)

> - remove()*     DONE*
>
> *Listing*
> list()*     DONE*
>
> *Paths*
> - separator()*     DONE*
> - changeWorkingDirectory()*     DONE*
> - workingDirectory()*     DONE*
>
> *Tests *
> - exists()*     DONE*

> - isAbsolute()
> - isDirectory()
> - isExecutable()
> - isFile()
> - isLink()
> - isMount()
> - isReadable()
> - isWritable()
>
> I removed from James' list based on the following questions:

> *- can JavaScript do it already?*
> *- should this be done by Bash Script or "full programming languages"?*
> *
> *

> What do you guys think?
>
> Ivan

[Ivan De Marino]

So, can James and Me assume this list is accepted and go ahead?

I'd like to mark this new feature as "done and dusted" :)

Or do you guys have objections or things to change?

This thread as been on for quite sometimes now...

IVan

[Ariya Hidayat]

I'm scared because the list is pretty long. Without having our testing
framework in place yet, this is even scarier.

I also don't know if anyone actually uses various API. For example, I
know I want to use file open/read/write/close (and its variants), but
that's about my needs.

One very good practice back when I was with Trolltech/Nokia-Qt was
"can you have three examples/use cases?" everytime a public API is
introduced. If we use this approach, file read/write makes sense but
I'm not sure about other functions.

--
Ariya

[Ivan De Marino]

In terms of tests, It's my intention to do just that afterwards: I know we shouldn't be writing tests AFTER the code, but at the moment that's what I can manage with my spare time.

In terms of use cases, I can come up with some more.

But I can admit that I might have issue with some (stuff like "isMount").

Let's trim a bit this list...

[Ariya Hidayat]

What about rolling it our a group at a time, rather than everything at
once? That way we can exercise the API + build the test suite as we
go.

--
Ariya

[Ivan De Marino]

Fine.

I'll add this to my todo:

- roll out group 1

- roll out tests for group 1

I'll use Jasmine for the tests, if you don't have a specific reason not to.

Ivan

[Ariya Hidayat]

> I'll add this to my todo:
> - roll out group 1
> - roll out tests for group 1

Sounds good. Which group did you mean by group 1? My suggestion would
be to start by what everyone needs most :)

> I'll use Jasmine for the tests, if you don't have a specific reason not to.

It should be fine till we have something in issue 146 implemented.

--
Ariya

[Ivan De Marino]

On 15 July 2011 16:51, Ariya Hidayat <ariya....@gmail.com> wrote:

> I'll add this to my todo:
> - roll out group 1
> - roll out tests for group 1

Sounds good. Which group did you mean by group 1? My suggestion would
be to start by what everyone needs most :)

Files is probably the best #1:

Files
- open()     DONE
fs:
   - atEnd()     DONE
   - close()     DONE
   - flush()     DONE
   - read()     DONE
   - readLine()     DONE
   - write()     DONE
   - writeLine()     DONE
- read()     << this is for very lazy coders :)
- write()    << this is for very lazy coders :)

- remove()     DONE

 

> I'll use Jasmine for the tests, if you don't have a specific reason not to.

It should be fine till we have something in issue 146 implemented.

Well, I guess what I'd do is to kickstart Issue #146 with tests for this part.

Ideally, our test runner can be simply a bash script that takes every single file in the "/tests" directory and launches them in phantomjs.

 

--
Ariya