Interested in helping expose new filesystems to nacl_io

[Patrick Chan]

Hi,

 

I would like to help contribute to the Native Client project - expose new filesystems to nacl_io.

 

My current status is that I took a look of the technical overview of Native Client, and the  nacl_io library but I have not dived into the technical details of the Native Client technology, how to get the code built, etc.

 

I am a newcomer of working on a open-source project.

I am not very sure what information, such as what my technical skill set I have, why I would like to contribute the Native Client project, how much time a week I have for the Native Client project, etc I should provide.

 

Could you please get in touch with me to get me started?

 

Thanks,

Patrick

[Ben Smith]

Hi Patrick,

I think the best way to start working on an open-source project is to pick a project that you actually need/want. This will give you the desire to get things done without getting bogged down in all the code.

A good place to get started with NaCl is to try porting an application. We have many you can look at for inspiration in webports.

It sounds like you are considering making a proposal for GSOC? If so, I should tell you that I believe we are not part of the GSOC this year. Naturally you're still welcome to contribute. :)

-Ben

[Patrick Chan]

Ben,

 

I actually want a project that is technically tough.  I would like to apply for software developer jobs at large software firms such as Google/Microsoft/Amazon/etc in the future.  I think a project that is seeking contributors, has a code base with high-quality code, is active, and has active members with high technical skills is going to be helpful. 

 

I am going to look at webports to get started.

 

I am not considering making a proposal for GSOC, Google Summer of Code.

 

Thanks,

Patrick

[Patrick Chan]

Ben,

I think I have completed porting an application.  I built webports/src/ports/openal-ogg-demo (*), written as openal-ogg-demo below, and verified the application in Chrome.   openal-ogg-demo is an application that has already been ported to Native Client, so to see how an application that has not been ported becomes ported, I compared openal-ogg-demo to the nacl_io demo application, written as nacl-io-demo below.

The comparison was comparing the difference of the .c files and Makefiles between openal-ogg-demo and nacl-io-demo.  I did not see any extra codes for porting, so I think porting an application means rewriting the application to a Native Client application and I think I have completed porting an application.

Could you please comment given what I actually want to work on technically tough projects and would like to work at a dev in big firms in the future, whether or not to proceed on the project - expose new filesystems to nacl_io - or change to another project?

If I may proceed on the project - expose new filesystems to nacl_io, could you please advise what to do next?

Thanks,

Patrick

(*) I would just like to add a note on my build experience in Cygwin/Windows to possibly help others in the future.  After spending several hours building a ported application in Cygwin/Windows, I was not able to build.  Similar to what another person wrote in the group, building in Linux is a lot more time saving. 

[Ben Smith]

Hi Patrick,

What I was trying to say before was this: if you have a practical need for adding new filesystems to nacl_io, then by all means, work on it! I'll be happy to help you through technical issues along the way. If you don't yet have a reason for adding a filesystem to nacl_io, I'd suggest coming up with one. What would you use this new filesystem for? It's difficult to develop a feature in a vacuum, and a new filesystem without a reason would likely not be very useful. So my advice is this; don't just think of what project to work on, also think about why. Why work on this project, instead of any other one? Why does this project matter to you? Only you can answer these questions. But I believe answering them will lead you in a better direction than picking your project because it is technically tough.

Cheers,

-Ben

[Patrick Chan]

Ben,

 

The cause for adding a filesystem to nacl_io,is for your developers. It would be great to have new filesystems to mount in the NaCl Dev Env: https://developer.chrome.com/native-client/nacldev, as written here.

 

While searching for the cause, I saw another post that described the approaches of adding a filesystem

 

I look to be able to start to work on the technical side from the described approaches, could you please reply right now, which file systems, Google Drive, Githib, and Dropbox, are good to be mounted?

 

Could you please reply if I should sign Google Individual Contributor License Agreement now?

 

Could you please reply an issue should be created for tracking?  If yes, which bug database Chromium or Native Client needs to be opened for tracking?

In case there are any things in your reply I miss to get back any of that, please let me know.

Thanks,

Patrick

 

 

On Wednesday, April 13, 2016 at 11:56:37 AM UTC+9, Patrick Chan wrote:

[Ben Smith]

On Monday, April 18, 2016 at 5:36:46 PM UTC-7, Patrick Chan wrote:

Ben,

 

The cause for adding a filesystem to nacl_io,is for your developers. It would be great to have new filesystems to mount in the NaCl Dev Env: https://developer.chrome.com/native-client/nacldev, as written here.

 

While searching for the cause, I saw another post that described the approaches of adding a filesystem

 

I look to be able to start to work on the technical side from the described approaches, could you please reply right now, which file systems, Google Drive, Githib, and Dropbox, are good to be mounted?

Any of those would be useful, feel free to choose the one that you're most interested in.

 

 

Could you please reply if I should sign Google Individual Contributor License Agreement now?

That won't be necessary until you submit a patch.

 

 

Could you please reply an issue should be created for tracking?  If yes, which bug database Chromium or Native Client needs to be opened for tracking?

This isn't necessary either, though if you would like to create a bug, use the Chromium issue tracker. If you do, please post the link here and I'll add the appropriate tags.

[Patrick Chan]

On Wednesday, April 20, 2016 at 2:52:48 AM UTC+9, Ben Smith wrote:

On Monday, April 18, 2016 at 5:36:46 PM UTC-7, Patrick Chan wrote:

Ben,

 

The cause for adding a filesystem to nacl_io,is for your developers. It would be great to have new filesystems to mount in the NaCl Dev Env: https://developer.chrome.com/native-client/nacldev, as written here.

 

While searching for the cause, I saw another post that described the approaches of adding a filesystem

 

I look to be able to start to work on the technical side from the described approaches, could you please reply right now, which file systems, Google Drive, Githib, and Dropbox, are good to be mounted?

Any of those would be useful, feel free to choose the one that you're most interested in.

After double checking which file systems have already been mounted in native-client-discuss, some file systems may have been already mounted.  I am going to verify with each person who wanted to mount a file system so duplicate work is not done.

[Patrick Chan]

On Wednesday, April 20, 2016 at 6:22:00 PM UTC+9, Patrick Chan wrote:

On Wednesday, April 20, 2016 at 2:52:48 AM UTC+9, Ben Smith wrote:

On Monday, April 18, 2016 at 5:36:46 PM UTC-7, Patrick Chan wrote:

Ben,

 

The cause for adding a filesystem to nacl_io,is for your developers. It would be great to have new filesystems to mount in the NaCl Dev Env: https://developer.chrome.com/native-client/nacldev, as written here.

 

While searching for the cause, I saw another post that described the approaches of adding a filesystem

 

I look to be able to start to work on the technical side from the described approaches, could you please reply right now, which file systems, Google Drive, Githib, and Dropbox, are good to be mounted?

Any of those would be useful, feel free to choose the one that you're most interested in.

After double checking which file systems have already been mounted in native-client-discuss, some file systems may have been already mounted.  I am going to verify with each person who wanted to mount a file system so duplicate work is not done.

 

I am going to choose Google Drive to get started.

After double checking, there was another individual who did some work on Google Drive 4-5 months ago.

He is now working on another project.  

I told him about my working on Google Drive that he had worked on before, and he did not tell me to work on another file system.

[Patrick Chan]

Ben,

Around April 20th, you were helping me get started on the Native Client project - expose new filesystems to nacl_io.  Google Drive is the file system to be mounted.

I am seeking help on calling Google Drive API v3 through URLLoader.

I am attempting to create a directory in Google Drive. 

I call Google Drive API v3 on the methods of URLLoader, URLRequestInfo, etc.

To set up an API request, 

an URLRequestInfo instance urlRequestInfo_ is created with 

SetMethod("POST"),

SetURL("https://www.googleapis.com/drive/v3/files"),

AppendDataToBody(json.data(), json.length())

while json is a std::string equal to:

{

"mimeType": "application/vnd.google-apps.folder",

"name": "createnewfolder"

}

and some other methods SetAllowCrossOriginRequests(true), SetHeaders("Bearer <token>"), etc.

After setting up the API request, to call the API request, an URLLoader instance urlLoader_ is called as urlLoader_.Open(urlRequestInfo_, pp::BlockUntilComplete()).

After calling the API request, to get the response, an URLResponseInfo instance urlResponseInfo_ is created from urlLoader_->GetResponseInfo().

After getting the response, the response status code and the response body are gotten.

The response status code is gotten from urlResponseInfo_->GetStatusCode().

The response body is gotten from 

FileRef fr = urlResponseInfo_.->GetBodyAsFileRef(),

FileIO fio(<InstanceHandle>),

fio.Open(fr, PP_FILEOPENFLAG_READ, pp::BlockUntilComplete()),

and fio.Read(0, <buffer>, 8192, pp::BlockUntilComplete()).

The response status code returns 200.

The response body returns:

{

 "kind": "drive#file",

 "id": "0BwVFLjfqf6AYRGNCczFVRFJGYXc",

 "name": "Untitled",

 "mimeType": "application/octet-stream"

}

However, in OAuth 2.0 Playground (https://developers.google.com/oauthplayground/?code=4/ROS1cJkQYDxcEHTfNIvqGpScN4X-n10l-7zHJ1Ssv6M#),

when I send a POST request

to https://www.googleapis.com/drive/v3/files

with the request body:

{

"mimeType": "application/vnd.google-apps.folder",

"name": "createnewfolder"

}

which is the same request body set up in urlRequestInfo_,

the response status code returns 200.

And the response body returns:

{

  "mimeType": "application/vnd.google-apps.folder", 

  "kind": "drive#file", 

  "id": "0BwVFLjfqf6AYZjdKQl9GVW5QYVU", 

  "name": "createnewfolder"

}

I am not very sure why setting the same API request inputs in URLLoader and in OAuth 2.0 Playground results in a file creation in URLLoader but a directory creation in OAuth 2.0 Playground.

Could you please help?

Thanks,

Patrick

[Ben Smith]

Hi Patrick,

I don't know why you'd be getting different results. Try debugging using the Chrome devtools network tab. You should be able to see exactly what is being sent in both cases, and compare them to see what is different.

You can also take a look at the Google Drive example in the SDK. It uses the Rest v2 API, and IIRC doesn't create folders. But it may be a useful reference anyway.

-Ben

[Patrick Chan]

Ben,

Thank you for your help.  The reason was that the content type was not set in headers. SetHeaders has to be called as SetHeaders("Authorization: Bearer <token>\nContent-type: application.json\n").

Thanks,
Patrick

[Patrick Chan]

Ben,

I saw questionable open_flags values in this KernelProxy open() method and did some investigation of that; the investigation found a typo in the nacl_io_demo index.html file.

nacl_sdk/pepper_49/examples/demo/nacl_io_demo/index.html:

        <option value="w+">Read/Write New File (w+)</option>

        <option value="a">Append Write (a)</option>

        <option value="w+">Append Read/Write (a+)</option>

The fix is going to be "a+" but not "w+" in the second "w+":

        <option value="w+">Read/Write New File (w+)</option>

        <option value="a">Append Write (a)</option>

        <option value="a+">Append Read/Write (a+)</option>

I searched All issues for "nacl_io_demo" in the bug database Chromium and Native Client but could not find a filed bug report.

Could you please tell me if I should file a bug report?

Thanks,
Patrick

[Ben Smith]

Thanks! Looks like a typo. I've created a fix here: https://codereview.chromium.org/1995763004

[Patrick Chan]

Ben,

I would like to seek another help. 

I attempt to let the mounted file system to Google Drive be able to perform the file operations and directory operations listed in nacl_io_demo.

My code for handling the mounted file system to Google Drive is put under nacl_sdk/pepper_49/src/nacl_io/ and in a file nacl_sdk/pepper_49/src/nacl_io/googledrivefs/googledrivefs.cc.  A class GoogleDriveFs that extends Filesystem is defined in googledrivefs.cc.

When fopen(<filename>, “r”) is called in a directory /dir in nacl_sdk/pepper_49/examples/demo/nacl_io_demo/handlers.c, GoogleDriveFs::OpenWithMode(const Path& path, int open_flags, mode_t mode, ScopedNode* out_node) in googledrivefs.cc is run with the passed arguments:

path contains /dir/<filename>
open_flags is 0
mode is 0

The same arguments are passed when opendir(<dirname>) is called in a directory /dir in nacl_sdk/pepper_49/examples/demo/nacl_io_demo/handlers.c, which runs GoogleDriveFs::OpenWithMode(const Path& path, int open_flags, mode_t mode, ScopedNode* out_node) in googledrivefs.cc with the passed arguments:

path contains /dir/<dirname>
open_flags is 0
mode is 0

The information from OpenWithMode is not sufficient to see if a user would like to open a file or a directory.  

When Google Drive has a directory that has the same name a user is opening a file with, OpenWithMode does not know whether to return an error for the absence of a file or to return PP_OK.

Could you please tell me what you think is practical for resolving?

Thanks,
Patrick

[Ben Smith]

Hi Patrick,

This is because it is OK to call open on a directory. Ultimately, fopen calls open and so does opendir. It isn't until you call read or getdents that you should fail if the file type is incorrect. Take a look at the Html5FsNode, it uses this strategy for handling files vs. directories.

-Ben

[Patrick Chan]

Ben,

Around 06/01/2016, you helped resolve my question about OpenWithMode not having enough information to see if a user is opening a file or a directory on the Native Client project - expose new filesystems to nacl_io.

Afterward, I kept coding and now the code has been completed.  I would like to check with you on my implementation of the code.

The implementation is simple.  I quickly get a simple implementation done to see if a simple implementation is enough for the use of your developers and submitting a patch.  I need to complete the open source project as soon as possible.  I would like to put the open source project in my resume when applying for jobs so to show that I am able to do a project with an engineer in a large software company.

The implementation contains some code from this piece of code and that piece of code from other codes.  I sometimes was writing the code but the code had been shown before.  

I attached the zip file with the code based on nacl_sdk/pepper_49

googledrivefs.cc – please put at src/nacl_io/googledrivefs/googledrivefs.cc

googledrivefs.h – please put at include/nacl_io/googledrivefs/googledrivefs.h

Makefile – please put at src/nacl_io/Makefile

kernel_proxy.cc – please put at src/nacl_io/kernel_proxy.cc

node.h - please put at include/nacl_io/node.h

googledrivefs_demo – please put at examples/demo/googledrivefs_demo

googledrivefs_demo/examples.js contains 2 variables CLIENT_ID and CLIENT_SECRET, that have to be set from the client ID and client secret made from Google Developers Console.

Could you please reply what you think is expected in the code but is not found in the code?

I don't know about certain implementation details and would like to ask for your comments.

The access time, modified time, and creation time are set as 0 in my implementation of GetStat().

I based the code on HttpFsNode setting all 3 types of times to 0.

Google Drive has sharedWithMeTime, viewedByMeTime, and modified time to set the access time and modified time of a file, but creation time cannot be requested.

Could you please reply what you think about setting all 3 types of times to 0?

stat.st_mode is not set in my implementation of GetStat().

I based the code on HttpFsNode not setting stat.st_mode in HttpFsNode::GetStat().

Could you please reply what you think about my implementation of not setting stat.st_mode?

stat.st_nlink is not set in my implementation of GetStat().

I based the code on Html5FsNode not setting stat.st_nlink in Html5FsNode::GetStat().

Could you please reply what you think about my implementation of not setting stat.st_nlink?

Each function call of GoogleDriveFsNode::Write() writes to Google Drive, but compared to writing to a buffer in GoogleDriveFsNode::Write() and writing to Google Drive on Node::Destroy(), it may be slow.

Could you please reply what you think about my implementation of GoogleDriveFsNode::Write()?

When there are too many directory entries in Google Drive, in GoogleDriveFsNode::GetDents(), Google Drive API response is large. When a buffer stores the entire response in GoogleDriveFsNode::GetDents(), an out-of-memory exception is thrown. I commented a TODO in the code.  The TODO is a task to do in the future.  My current implementation handles only a small number of directory entries.

Could you please reply if this is OK not to deal with too many directory entries at this moment?

GoogleDriveFsNode::Remove(), GoogleDriveFsNode::Rename(), and GoogleDriveFsNode::Unlink() are not supported.

I coded just enough to minimize the time before getting comments from you.

Could you please reply what you think about my not supporting the 3 functions?

Thanks,

Patrick

[Ben Smith]

Thanks for working on this, Patrick. Sorry for the lateness of my reply, I've been on vacation for the past few weeks. :)

I'm happy to review your changes, but it is inconvenient in the current form, as I can't easily see a diff between your source and the source in the repository. Follow the instructions here to get a Chromium checkout, then follow the instructions here to upload your change for review. Don't worry about the part where it describes building Chrome, you don't need to do that.

After you've uploaded your change, please add the notes you've written below. It will be much easier for us to communicate about the change there than in the forums here.

Thanks again,

-Ben

[Patrick Chan]

Thanks for giving me the review.

I would just like to add a note on my experience to possibly help others who would like to port a file system in nacl_io in the future.  The native client SDK code can be downloaded from the zip file nacl_sdk.zip and the Chromium code base.  The code of nacl_sdk.zip is fast to compile, but the code of the Chromium code base is more updated and is easier to compare diff in the review.