GSOC 2014 - 3D Visualization

[Simon Danisch]

Hi,

I'm not sure if this is the right mailing-list, but before I officially apply for GSOC 2014, I wanted to make sure, that I'm on the right way.

In the last months I mainly worked with the OpenGL package and started working on my own GLGraphics package.

I basically wrote modules for text-rendering, shapes, shaders, window creation, events, cameras, textures and some matrix math.

Before I read the Julia GSOC projects, I was actually just in the middle of trying out some way of rendering functions that manipulate planes, volumes and particles ( or just render them without applying a function).

This means I'm pretty close to actually implementing the wanted matlab like plotting functions, which makes me a good match for the 3D Graphic project.

I'm also a student (in Germany), so I guess I should just apply on the GSOC site with Julia as the mentor and a nice description on what I plan on doing.

Am I missing anything?

Kind regards,

Simon

[Keno Fischer]

> Am I missing anything?

Nope, sounds like you know exactly what you're doing. Looking forward to your application.

- Keno

[Tim Holy]

Glad to hear that someone is interested in working on graphics/visualization.

--Tim

[Tobi]

Since I originally wrote the proposal for 3D graphics I am pretty excited if we can get a student with that background. I could mentor this project but have not yet entered the application for it. But I am also fine with someone else doing it.

@Simon: I am also from germany (Lübeck) by the way. Where are you from?

[Simon Danisch]

I'm studying in Osnabrueck ;)

Cool, guess I should write my application soon, then.

[Jake Bolewski]

Simon, have you looked at vispy (http://vispy.org/)?  Having something similar would be a great addition to Julia.

[Chris Foster]

Hey guys,

I've just posted a (rather long) note about my project displaz which
is coming at the same problem from a slightly different direction. It
looks like some of my general goals (particularly around modern
OpenGL) are similar to vispy which I wasn't aware of until now.

Hopefully there's some scope to collaborate, or at least share ideas.

Cheers,
~Chris

[Tobi]

For inspiration here is a movie of 3D graphics display that has been developed at my company

http://www.youtube.com/watch?v=pl-d8zE_Mhw#t=69

It is developed for a specific task and not as general as the Julia 3D display should be.

Here are some random notes from me on the development of a 3D display.

- I would be great if the list of dependencies could be kept small. The display itself should actually only require OpenGL. This is quite similar to Winston which has cairo as its core dependency.

- Similar to Winston it would be great if the OpenGL display could be written to a file (png/pdf) or be embedded into Gtk or Tk.

- Around the core display one can write a small interactive GUI (similar to Matlab and Matplotlib).

- The code has to be tested on different graphic cards...

[Chris Foster]

That's really cool Tobi. I can definitely see where you're coming
from if you're building bespoke applications with OpenGL like the one
you've got here.

My aim with displaz is currently focused on the needs of interactive
technical computing: When you're hacking away in the repl and want to
display a whole heap of geometry, provide a really nice simple and
interactive way to do that. I definitely agree that dependency
management/minimization is a constant battle. It's also a matter of
perspective - for better or worse I still work in several different
languages, so having some C++ dependencies isn't necessarily a big
deal to me, especially if it means infrastructure which can easily be
shared between those languages.

~Chris

[Tobi]

Hi Chris,

I hope you did not get my comment as a critique of your work. I just wanted to express my thoughts how I would see 3D graphics in "the most Julian way". When working in multi-language environments one always faces the issue where to draw the line between the languages. It is a usual pattern to draw it in a way that the scripting language only sees a higher level API of the high performance C code. In Julia this is also possible but usually we see that the line is drawn at a very low level. The reason is that a) the ffi is extremely efficient, in fact the overhead is compiled away. b) its more fun to program everything between low level and high level in Julia :-)

In many cases this means that one is reimplementing something which probably already exists in C/C++. But if its finished one usually has a much more flexible code which is easy to adapt and usually has minimal dependencies.

[Chris Foster]

On Sun, Mar 16, 2014 at 9:01 AM, Tobi <tobias...@googlemail.com> wrote:
> I hope you did not get my comment as a critique of your work.

No problems, there are many ways to approach this.

> I just wanted
> to express my thoughts how I would see 3D graphics in "the most Julian way".
> When working in multi-language environments one always faces the issue where
> to draw the line between the languages. It is a usual pattern to draw it in
> a way that the scripting language only sees a higher level API of the high
> performance C code. In Julia this is also possible but usually we see that
> the line is drawn at a very low level. The reason is that a) the ffi is
> extremely efficient

Yeah, to the extent that I'd hesitate to call it an FFI... Julia turns
the "foreign" in FFI into "native" :)

> In many cases this means that one is reimplementing something which probably
> already exists in C/C++. But if its finished one usually has a much more
> flexible code which is easy to adapt and usually has minimal dependencies.

There's definitely a lot to be said for having native julia packages.
Not only all the things you mentioned, but that having native julia
code on hand makes things that much more hackable, which in turn
increases the speed of development.

On the other hand, I'm often frustrated by the amount of reinventing
of wheels which goes on. For example, vispy looks like an awesome
start, but it's unlikely to feel natural to use from anything other
than python (possibly even julia, despite the amazing python interop.)
IMHO it's easy to do a quick 3D interface but making it really top
notch will require man years of effort. Writing a lidar point cloud
viewer is a good example: every man and his dog have their own point
cloud viewer - heck, even some dogs' fleas have written the couple of
calls to glDrawArrays(GL_POINTS, ...) that are required :) On the
other hand, making it flexible, fast and intuitive to use is actually
not that easy.

~Chris

[Mike Innes]

One thing that'd be really good to see in our 3D libraries is support for pluggable backends, and in particular good support for a WebGL backend. That way we could use them from our interactive frontends; IJulia, Light Table, etc.

It's not a huge deal if you need to open a new window to see a plot, but being able to display powerful interactive graphics inline could be a really killer feature. Plus, being able to treat a plot as an HTML element means we can integrate it with the support for interactive controls (sliders etc.) that we're working on for these environments. Ideally we'd want to be able to show the same plot in IJulia, in a GTK.jl window, or rendered to an image, all easily and consistently.

Simon, you'll have to decide for yourself want you want to prioritise during the GSoC period, but it's definitely worth bearing these things in mind when thinking about the implementation.

[Tobi]

I am not sure how feasible this is. At a higher level there definitely should be a common interface, which will likely follow the Matlab API.

But under the surface it gets really fast OpenGL specific. For instance one has to think about the data representation that is uploaded to the graphic card. Further using shader programming one can offload a lot of work to the graphics card leading to a snappy display.

[Simon Danisch]

Thank you all for the inspiration :)
Well, I’m all in for Julia at the moment, so I want OpenGL to feel as native and natural as possible.
I actually don’t have a science visualization background.
I’m studying Cognitive Science and am a hobby 3D graphic designer.
So I always saw the need for performant visualizations from two perspectives: data visualization and art visualization.
Well, there’s actually a third one for me, which are the needs from image recognition: having fast and interactive tools to tweak parameters of a filter pipeline.
My conclusion: They always overlap, but they don’t really exist in one package.
For 3D programs like 3DS Max, you have all the simple interactivity, but you’ll always find the programming, libraries and low-level missing, OpenCV is lacking of good ways to make 3D plots, and scientific visualizations are way too often ugly and you can’t really tweak them.
So I want to create something that has high interactivity and can directly work with the datastructures of the language, but also has good lighting (raytracing and the like).
I actually searched long for a language that can make this possible in a nice way.
Julia was the first language, where I was satisfied with performance and how you can interface with low-level API’s, but also have nice high level features ( and no classes, I really love to leave the Java/C++ world for multiple dispatch).
Also code sharing and working on the same project must be very easy, as one or two persons could never realize such a huge functionality.
I think to go away from classes is one of the best ways of making code sharing easier.
But that is very far out, and just my ultimate motivation.

At the moment I write a bachelor thesis about an IDE, that is supposed to blur the lines between debugging, print-eval loop and interactive visualizations.
So for the start I want something more like this : https://code.google.com/p/glumpy/ (a friend pointed me out this package, thanks for that)
This actually nails the functionality that I want to develop in the next few months.
On top of that I want to implement some GUI like elements and views with manipulation possibilities for different Julia datastructures.
One of the first datastructures will be the Julia Expression tree, allowing for editing your code while you run it.
Its very nice to have interactive views to a bundle of Julia’s datastructures. Like this you can just traverse the tree and display the right view for the leaves, having a functionality similar to Ijulia or Wolfram Mathematica.
My Package is still a little broken, as it depends on a newer version of the OpenGL package and things are a little ugly inside as I rushed to show you something, but this is what I have so far:

Event Management:
# EventAction{EventToReactTo}(id, event -> bool, (additional, params, to condition function), actionFunction, (additional params))
registerEvent(EventAction{MouseClicked}("", x -> x.key == 4 || x.key == 3, (), zoom, (cam,)))

This is a window with a view on the syntax tree of the actual Julia source file that opened this window, and a 3D gauss. The formula for the gauss and color is hidden inside the shader. At some point I want to allow for very easy automated shader generation, so that you basically just define functions on your array and then execute them on the gpu.

Also, I work together with a friend on the IDE. We want to parse the packages, so that you can search through packages from inside your IDE.
It will be a multipurpose database, in which you can store sample data, images, 3d models, shader, shader functions, julia code, etc, which will ultimately be accessible from inside your IDE.

I guess I should just write all of this minus the IDE part into my application for GSOC =)

At the moment I need to work on the interactivity and source code display, as it concerns my bachelor thesis, so that I’ll have time for GSOC and 3D visualization in Mai.

Best regards,
Simon

[Tobi]

Hi Simon,

Could you share a little what the IDE will look like? And what technology you are using. I have myself written a prototype of an IDE called Julietta (https://github.com/tknopp/Julietta.jl) and things like package browsing already work. But the developement currently has stalled as I currently have not found time for it.

Cheers,

Tobi

[Simon Danisch]

Hi Tobi,

I have the crazy plan of actually implementing it natively with Julia and OpenGL.

So basically doing it from scratch.
It gives me huge flexibility to incorporate completely new features.

Also having a native GUI package would be quite awesome!

The downside is, that it might have some issues with different graphic cards and platforms in the beginning.

But I'm willing to risk that, especially because if we fix all the bugs, we might have solid platform independent OpenGL support.

This was my first layout sketch:

I can keep you updated on the progress if you want ;)

Kind regards,

Simon

[Tobi]

Sounds all exciting. I am interested in a pdf of your thesis when your finished.

I started with Gtk as I think that this will get me things faster but as part of a bachelor thesis your approach is very interesting.

For the GSoC you will have to limit the scope of the project (as you already said).

[Tim Holy]

Not only the scope of GSoC, but have you given any thought to narrowing the
scope of your bachelor thesis a bit? :-) Implementing a GUI toolkit from
scratch in OpenGL sounds like a 10-year project, or something to submit when
you're asking for $10M of venture-capital funding.

--Tim

On Monday, March 17, 2014 08:24:19 AM Tobi wrote:
> Sounds all exciting. I am interested in a pdf of your thesis when your
> finished.
>
> I started with Gtk as I think that this will get me things faster but as
> part of a bachelor thesis your approach is very interesting.
>
> For the GSoC you will have to limit the scope of the project (as you
> already said).
>
> Am Montag, 17. März 2014 15:54:15 UTC+1 schrieb Simon Danisch:
> > Hi Tobi,
> > I have the crazy plan of actually implementing it natively with Julia and
> > OpenGL.
> > So basically doing it from scratch.
> > It gives me huge flexibility to incorporate completely new features.
> > Also having a native GUI package would be quite awesome!
> > The downside is, that it might have some issues with different graphic
> > cards and platforms in the beginning.
> > But I'm willing to risk that, especially because if we fix all the bugs,
> > we might have solid platform independent OpenGL support.
> > This was my first layout sketch:
> >
> >
> >

> > <https://lh4.googleusercontent.com/-jWRHdEPxbB8/UycLVDjkwcI/AAAAAAAAINM/Wo
> > J5eb7bid0/s1600/editor.jpg> I can keep you updated on the progress if you

> > want ;)
> >
> > Kind regards,
> > Simon
> >
> > Am Montag, 17. März 2014 10:19:44 UTC+1 schrieb Tobi:
> >> Hi Simon,
> >>
> >> Could you share a little what the IDE will look like? And what technology
> >> you are using. I have myself written a prototype of an IDE called

> >> Julietta ( https://github.com/tknopp/Julietta.jl) and things like

> >>> <https://code.google.com/p/glumpy/>https://code.google.com/p/glumpy/ (a

> >>>>> It's not a *huge* deal if you need to open a new window to see a

[Tobi]

:-) Had a similar though. But honestly, doing proof-of-principle is absolutly fine for a bachelor thesis.

[Simon Danisch]

Yeah scope is definitely an issue, but I hope that I can keep things simple and show some nice concepts quickly.

But without (a lot of) support at some point, this will be surely going nowhere.

@Chris Foster 

Displaz looks very interesting! 

It's not really what I want to use as I want to stay in Julia, but maybe we can find some way of working together :)

@Tobi 

I need to put in a mentor for the project. As no one else offered mentoring so far I will mention you there.

Hope that's fine with you ;)

[Tobi]

Am Dienstag, 18. März 2014 00:23:23 UTC+1 schrieb Simon Danisch:

@Tobi 

I need to put in a mentor for the project. As no one else offered mentoring so far I will mention you there.

Hope that's fine with you ;) 

Sure. I have just registered at the GSOC page.

[Simon Danisch]

Hi,

I submitted my proposal! =)

http://www.google-melange.com/gsoc/proposal/public/google/gsoc2014/simon_danisch/5629499534213120

I'm a little under time pressure with University right now, but I hope it still covers all the important points and doesn't include to many grammatical mistakes.

Looking forward to this!!!

Best regards,

Simon

[Steven G. Johnson]

On Friday, March 14, 2014 2:23:36 PM UTC-4, Simon Danisch wrote:

Before I read the Julia GSOC projects, I was actually just in the middle of trying out some way of rendering functions that manipulate planes, volumes and particles ( or just render them without applying a function).

Could you narrow the scope a bit so that it doesn't sound like you are trying to re-implement all of VTK (http://www.vtk.org/) in one summer?

Have you considered just developing VTK bindings for Julia?  (This is non-trivial because VTK is C++, but there are tools out there to help.   A related possible project would be to develop Julia output for SWIG, in order to generate Julia code from C++ libraries like this.)

[Tobi]

Why is the scope of the project too wide? Drawing volumetric and surface data in OpenGL is not really so hard.

I would strongly vote against making this GSoC project just wrapper around VTK. And this not only BecauseOfThisVeryLongFunctionNames. Wrapping OpenGL directly gives a lot more flexibility and the direct access to shaders opens to do computations directly on the graphic card. Further VTK is a pretty large dependency and being C++ doesn't make it more attractive.

[Isaiah Norton]

VTK uses a very approachable subset of C++, and has getter/setter functions for *everything*, so wrapping it is a good deal simpler than less disciplined C++ libraries. The approach I used here:

https://github.com/ihnorton/VTK.jl

should be feasible now that Julia has package-level static compilation support; with ~50,000 lines of generated code, it was totally unusable before (adding to that, old laptop at the time). There are some other issues (mainly around distribution), but they are tractable. I've been using the pure-Julia-vtable-dereference approach in another project recently and have come back around to the idea.

I'm planning to work on this soon, but would be thrilled to see forks or PRs - or any other approach to wrapping VTK. As Tobi points out, any VTK binding will severely want for a Julian layer on top with less of the CamelCasedJiminyCricketWhenWillThisClassNameEnd syndrome. The Enthought "tvtk" layer sets a nice example (underlying MayaVi).

That said, the pure OpenGL approach has some big selling points, even if it will take a longer time to mature. Not least, it will be a "highly visible" (cough) demonstration of Julia's performance.

[Jake Bolewski]

I've used TVTK a lot for a project analyzing 3D image stacks for confocal microscopy.  Tvtk doesn't really offer too much over the default swig generated python bindings if you are not going get the benefits of Traits, and it has some weird corner cases where the name simplification introduces subtle bugs so beware :).  I'm glad that VTK support is on the radar for Julia.  There really is no substitute for scientific 3D visualization despite it's quirks. 

[Tobi]

Hi Isaiah,

great to see your work on wrapping VTK I was not aware of. My post was not meant as a rant about VTK.

Its just that the pure OpenGL approach is very attractive especially if someone like Simon is going to work on it who has quite some background in OpenGL and already contributed code to the OGL packages. I am pretty sure that we will get a sweet package after this summer if we let him continue the route that he has already taken.

Cheers,

Tobi

[Isaiah Norton]

Yes, definitely +1 on that from me (fwiw). Just wanted to point out one possible approach in case someone was interested in going that way.

Best,

Isaiah

[Simon Danisch]

Thanks for all the support and feedback!

For me, things are pretty simple.
I'm really glad that I found Julia. 

I programmed for OpenGL mostly in C++ so far, which was sometimes a very slow and painful process.

That's why I started looking for a high performance language, which has a sane type system without classes and some high level abstractions.

Julia felt from the beginning very intuitive and easy to use, enabling me to program things faster (Hopefully without any performance loss).

Also, I'm not interested in quick results. I don't care about getting a volume visualization as quick as possible.

What I do care about is to have something that anyone can extend, ultimately adding value to the language itself.

If I'd write a VTK wrapper and something doesn't really work like expected, you would have to go inside VTK, change the (C++) source code and recompile a huge package.

Probably you would even need to change something in the Julia wrapper on top of it.

If its all inside Julia, you make the change while you stay inside the language. 

This will make it a lot more likely, that people will actually contribute to the package.

And that is what I really want. 

I want to create something people can build upon, making the native OpenGL package grow together with Julia.

For the scope:

It is a little ambitious I must admit, but I don't think its that far out.

(Thanks for your supportive opinion there, Tobi! )

One must not forget, that I don't start from scratch here. 

Also, a lot of things are already solved and just need to be rewritten in Julia.

It might be a little tiresome, redundant work. 

But I cant wait to have a visualization package with native performance, that is easy to extend and written in a sane language.

Best,

Simon

[Tim Holy]

I also think there's merit in starting at the bottom. VTK is of course amazing
in many ways, but a student of mine spent quite a lot of time trying to corral
it (admittedly, mostly via ParaView) into something conceptually
straightforward but nontraditional: volumetric rendering using skewed voxels.
This proved painful, and ultimately unsuccessful. I personally had a similar
experience with ImageVis3d. I think it will be easier to get at the plumbing
if Julia has good low-level support for OpenGL.

--Tim

> > <tobias...@googlemail.com<javascript:>>

[Steven G. Johnson]

On Thursday, March 20, 2014 3:01:47 PM UTC-4, Tobi wrote:

Why is the scope of the project too wide? Drawing volumetric and surface data in OpenGL is not really so hard.

 

It depends on what you mean by "drawing volumetric data".   There are many ways to visualize volumetric data (slices, isosurfaces, streamlines etc. for vectorial volumetric data...).

You could even more justifiably say, "plotting y vs. x in 2d is not so hard," but rewriting all of matplotlib would not be a 1-summer project.   Plotting in general is hard, not because any particular plot type is conceptually difficult (though isosurface algorithms for volumetric data are somewhat tricky), but because there are so many possible plot types and so many possible options for each plot type (for axes, labels, colors, etc.) that a implementing a production-quality library (as opposed to an "I drew a plot!" proof of concept) is a big task.

[Tobi]

Steven, you are absolutely right that getting a production quality 3D library done is not an easy task. And I have to admitt that I am so thankful for your work on PyPlot that enables me to make publication level 2D plots in Julia now.

Still I find it very valuable that we have the Winston package which has a lot less dependencies. I have integrated this with Gtk which was so easy and I am not sure if it would have been the same with Pyplot. Application deployment is another important aspect.

If this GSoC is accepted am quite sure that we will get a nice package that allows basic 3D visualizations only one Pkg.add away without any hassles to think about binary dependencies.

[Simon Danisch]

Hello,

I drowned a little in organisational work lately, so I couldn't really work on this.

But there are a few things I want to share.

For example some totally hacked together code, showing how I imagine to make things work in the future.

I've written this very hastily from what I have been thinking about in the past days.

So it's not working and possibly with a lot of flaws.

But I hope I can demonstrate the concept that I have in mind.

the render object:          https://gist.github.com/SimonDanisch/9956371

creating a vertshader:    https://gist.github.com/SimonDanisch/9956214

rendering:                     https://gist.github.com/SimonDanisch/9956371

putting things together:  https://gist.github.com/SimonDanisch/9956363

Hope things are at least a little self explaining.

In my little world, this should be a fairly flexible design, making it very easy implementing things like the surf function.

I hope we can figure out what the flaws and limitations are with this design, in this early stage.

This way, I don't waste time with writing too much spaghetti code =)

Also, I started putting some more work into the OpenGL package.

I'm discussing how to solve a few problems over at julia-users.

Hope we soon have the fruits of that work in form of a fast, platform independent OpenGL package!!

Best,

Simon