GSoC 2015 Ninia

[Muhammad Bhatti]

Hi,

My name is Muhammad Bhatti, and I am a my 3rd year undergraduate of Computer Science from National University of Science and Technology, Pakistan.

I do not have a lot of experience regarding open source development and I wish to change that. I want to contribute and give back to the community.

I found this project extremely interesting, and started to look for ways to contribute, as soon as I discovered it. 

I recently created a pull request, implementing the functionality of the UNPACK_SEQUENCE bytecode instruction and also created some tests to go along with it.

Here is the link: https://github.com/plasma-umass/Ninia/pull/24

I would definitely love to hear your feedback. It would also like to contribute to this project for GSoC 2015 and any advice regarding that would be very much appreciated.

In the meantime, I will keep trying my best to get a better understanding of how Ninia works and how I can further improve it.

Regards,

Muhammad Bhatti

[John Vilk]

Hello Muhammad,

Thanks for submitting your pull request! I've made a number of comments on your PR; once you address them, it should be ready to merge.

Regarding applying for GSoC 2015, the Ninia project is looking for people who are able to understand interpreter design and how the JavaScript event loop works. I replied to another student who is interested in Ninia regarding the project described in the Ideas List, and why the current interpreter isn't structured correctly for the web:

https://groups.google.com/forum/#!topic/plasma-umass-gsoc/BJpHk_Y0EYs

I encourage you to continue looking at Ninia's code. It's in very early stages, and is likely incorrect in many places. If you see anything that looks wrong, feel free to propose a fix! We're still missing a number of important features (e.g. user-defined classes), and some of the things are currently implemented in very hacky ways (e.g. we're not handling error conditions correctly).

Let me know if you have any questions.

John

[Muhammad Bhatti]

Hello John,

I read your reply to the other student, and read more on how the Javascript event loop works, and your workaround to that in Doppio. I also skimmed through your paper: https://plasma-umass.github.io/doppio-demo/paper.pdf

To support multi-threading, does Doppio utilize the WebWorkers? I understood the part about WebWorkers and their limitations, but how exactly is multi-threading achieved when we are explicitly storing the call stack in Javascript objects?

I understand that this project requires one to migrate Ninia to use Doppio's threads. The ideas page mention that Ninia currently does not use all of Doppio's resources, and is limited to trivial python programs.

I wanted to ask, which Doppio resources does Ninia currently use?

I am also in the process of writing my proposal and am currently unsure on how to give specific details for the timeline. Can you please give me any advice regarding that?

Regards,

Muhammad Bhatti

[John Vilk]

Hey Muhammad,

> To support multi-threading, does Doppio utilize the WebWorkers?

Nope! WebWorkers are more like processes than threads; they do not share any memory with other contexts, so it is infeasible to implement threads atop them.

> how exactly is multi-threading achieved when we are explicitly storing the call stack in Javascript objects?

In the same manner that a single-core CPU juggles multiple threads! Each Doppio thread time-shares the single JavaScript thread. One thread executes for a bit, then another thread executes for a bit. We have a simple thread scheduler in Doppio that handles figuring out which thread should run next.

Due to this design, we must explicitly store the JavaScript call stack in a JavaScript object so we can:

• Suspend the running thread after it runs for too long (which we check at method-call boundaries using an adaptive counter https://github.com/plasma-umass/doppio/blob/new-objects/src/threading.ts#L746)
• Suspend the running thread if it needs to wait for an asynchronous API to complete (e.g. opening a file with BrowserFS or the Node FS: https://github.com/plasma-umass/doppio/blob/new-objects/src/natives/java_io.ts#L95)

> I wanted to ask, which Doppio resources does Ninia currently use?

Ninia currently only uses Doppio's file system, BrowserFS, mainly for its emulation of the NodeJS Buffer class for manipulating binary data efficiently in JavaScript. e.g. the unmarshaler relies on it extensively:

https://github.com/plasma-umass/Ninia/blob/335dd96a3a0eea1159db9fecbc3ddc4b2dbd37e0/src/unmarshal.ts#L40

Eventually, Ninia will use Doppio's file system to load *.pyc files and to implement Python's file API.

> I am also in the process of writing my proposal and am currently unsure on how to give specific details for the timeline

Sure. The primary goal of the project is to migrate Ninia to use Doppio's threads. Now, Doppio's threads are not implemented in a standalone library at all; they are tightly coupled with DoppioJVM in the Doppio repository. And they are currently not generic enough to simply move to their own library. I don't expect a GSoC student to be able to jump into the DoppioJVM codebase and make them generic, since they are somewhat hairily intertwined into hairy JVM routines. :)

So, here's what I expect a student working on this project would be able to do:

• Develop from scratch a basic Thread object for Ninia, similar to Doppio's JVMThread. No need to worry about multithreading at all. Like in Doppio, a Thread is a glorified array of StackFrames; the thread has a runMethod function that takes a stack frame, pushes it onto the stack, and executes the code associated with it; the thread has a throwException function that performs stack unwinding to find which stack frame can catch the exception; the thread uses the same adaptive algorithm as DoppioJVM to suspend the thread if it runs for too long...
  
  
• Refactor Ninia to use this Thread. The biggest change here is that all Python bytecode method calls will need to be made asynchronous, because it's impossible to always determine if Ninia will need to suspend execution during the method call.
  

That's the basic, bare-bones version of the summer project. It's a fundamental change to the interpreter, and a nontrivial amount of work. I would view this as a "moderate success".

Once Ninia is using Doppio-like threads, I will work with the student to figure out what a generic Doppio Thread API should look like so that it supports both Ninia and DoppioJVM, and I will work toward making a standalone Doppio Thread library that both language implementations can use.

While that is happening, the student will have the freedom to work on other things. Here are some ideas:

• Multithreading support. Python has a couple of APIs for multithreading. The student could implement a comprehensive test suite for these APIs (which will be tricky -- you need to make sure the tests are robust to arbitrary thread scheduling), and then proceed to implement multithreading in Ninia. The student would need to implement or steal the thread scheduler from DoppioJVM (which would eventually make its way into the generic library I'd be working on), and the Python thread object might need to be modified to support locks.
  
  The hardest part of adding multithreading will be debugging everything.
  
  
• Improving Ninia's conformance to Python's behavior in some other way. You could hook up the file system, you could add Doppio socket support (which also aren't broken off into their own library at the moment)...

For your timeline, I expect you to carefully consider what is required to complete the basic goals of the project, and what additional work past the basic goals you believe you can complete over the summer. You should have clear-cut and dated goals, with concrete ways to determine how you know that you have met those goals.

Over the summer, students will use their timelines to track their progress, and adjust them as they encounter roadblocks or get ahead of where they expected to be. It's a really useful thing to have when working on a project, as it keeps your progress in-line with the project's goals.

Make sense? Let me know if you have any additional questions.

John

[Muhammad Bhatti]

Hello John,

I submitted a draft of my proposal on the google-melange website. Please let me know what you think of it and how I can further improve it.

Regards,

Muhammad Bhatti

[John Vilk]

Hey Muhammad,

Sorry for the delay. I'm going to look at this a little later today, and will email you comments.

John