Call for contributors

[Redd Kamel]

Hi all, I'm Redd Kamel, a member of the Google Body team. It's great to see so many people signing up for the mailing list.

Quick announcements:

- If you're interested in contributing to open-3d-viewer, please check out the BigIdeas page or feel free to propose your own improvements.

- If anyone in this group is an iOS engineer or knows one, I strongly recommend that someone port open-3d-viewer to iOS. We know that doctors and patients have used the Android tablet version of Google Body in amazing and innovative ways, and feel it would be transformative if the same experience were available on the iPad. Please pass this on to any/all iOS/3D people you may know.

Comments, questions, etc., feel free to let us know.

RK

[Rich Stoner]

Hi Redd,

I was passed this note by a colleague - I'm an iOS developer/scientist (http://wholeslide.com/projects.html) and am interested in getting involved.

Several years ago I wrote a 3d brain atlas for iPhone using a modified game engine ("Cerebrii", http://vimeo.com/4095348 , available on the store for free http://itunes.apple.com/us/app/cerebrii/id309653027?mt=8 ) with the same motivations as Google Body. Fast forward a few years and I've released a few other science/medical apps, most with a focus on high resolution pathology image data. I'm most interested in the incorporation of the technology/app into the clinic and would be glad to help out with the iOS port. 

Feel free to contact me directly with questions/comments/ideas'

Best -

Rich Stoner

Postdoctoral Fellow

UCSD Autism Center of Excellence

[Redd Kamel]

Great, I'll send you some details directly. (If there are others with iOS interest, don't hesitate to jump in as well.)

[Nico Weber]

Why not discuss this on-list, this being open source and all? :-)

Nico

[Redd Kamel]

Didn't want to but people with excess detail, but sure, why not indeed. :-)

Rich, here's an overview of what the WebGL version is capable of doing. I'll post this on the Wiki as well.

  [1] Core capabilities:

       - Load the geometry data (proprietary format) and image textures using the provided models.js file and models/cow_anatomy directory.

       - From that same directory, load the JSON metadata that is used for layering, grouping, and searching.

       - Display and navigate around the model with basic swipe (up/down/rotate-left/rotate-right) and pinch (zoom) gestures.

       - Hide/show each layer individually.

       - Switch between two models, again using the models.js file and models/ directory structure.

  [2] Rounding out a usable viewer:

       - Ability to tap an individual item and see what it is, via a label.

       - Full independent opacity control (0.0 to 1.0) of each layer, controlled by a UI comparable to the WebGL version.

       - Ability to search for an individual item by name and have it highlighted, with autocomplete search.

       - Highlighting rules that mirror the WebGL version (i.e., search for or select something and see a ghost version of the rest of the model).

  [3] Full feature parity with WebGL version:

       - Ability to search for groups of entities as well as individual ones.

       - Smoothed motions and transparency changes.

       - The ability to "pin" a selected item so it stays visible no matter what other opacity changes may happen.

       - The ability to hide any item.

       - "Capsule" navigation of the entire model, meaning that if you go up or down far enough, the camera starts moving as if it were a hemisphere over the top or bottom of the model, thus (in the Body case) allowing you to look down on the top of the head.

       - "Capsule" navigation around whatever item or items are currently selected.

If you're interested, my suggestion would be to work on getting the core capabilities implemented and checked in. (We can get you or others a second data set to ensure that switching between models works.) After that I guess it's a question of how much time you have to commit and whether others want to help.

Which approach to use for porting is an interesting question.

  - The developer of the Android version of Google Body is on this thread; he analyzed the state of the v1 codebase (which differs from open-3d-viewer primarily in the complexity of the metadata model) and ported it to Java wholesale. You could do the same, or you might investigate running some of the open-3d-viewer Javascript code in a wrapped webview -- this might make it faster to get the metadata model up and running, since it's pretty self-contained.

  - You might also look at the Android source itself as checked in. My understanding is that it's not quite buildable, but the gap to making it work is not large.

  - If you have time to embark on this, we can probably get Body team members to provide more detailed notes on the JS/Java source files.

Does this interest you?

RK

[Rich Stoner]

Redd,

Thanks for the information.  I spent some time looking at both code bases - some additional questions inline below:

  [1] Core capabilities:

       - Load the geometry data (proprietary format) and image textures using the provided models.js file and models/cow_anatomy directory.

I'm assuming the app should run standalone with no required network connection? or is the goal to provide an online viewer for an expanding set of models+metadata?

By proprietary format, are you referring to the utf8 encoded vertex data? I don't anticipate too many problems in loading either utf8 or obj models. 

I was not able to find any textures in the SVN - what format will these come in?

 

       - From that same directory, load the JSON metadata that is used for layering, grouping, and searching. 
       - Display and navigate around the model with basic swipe (up/down/rotate-left/rotate-right) and pinch (zoom) gestures.
       - Hide/show each layer individually.
       - Switch between two models, again using the models.js file and models/ directory structure.
  [2] Rounding out a usable viewer:

       - Ability to tap an individual item and see what it is, via a label.

I noticed that the tap-to-select method is index-color based. It should be possible to re-use the shaders with little modification.

 

       - Full independent opacity control (0.0 to 1.0) of each layer, controlled by a UI comparable to the WebGL version.

       - Ability to search for an individual item by name and have it highlighted, with autocomplete search.

       - Highlighting rules that mirror the WebGL version (i.e., search for or select something and see a ghost version of the rest of the model).

  [3] Full feature parity with WebGL version:

       - Ability to search for groups of entities as well as individual ones.

       - Smoothed motions and transparency changes.

       - The ability to "pin" a selected item so it stays visible no matter what other opacity changes may happen.

       - The ability to hide any item.

       - "Capsule" navigation of the entire model, meaning that if you go up or down far enough, the camera starts moving as if it were a hemisphere over the top or bottom of the model, thus (in the Body case) allowing you to look down on the top of the head.

       - "Capsule" navigation around whatever item or items are currently selected.

Most of the features listed are pretty straight-forward, especially with the existing android + webgl source. The unfortunate part is that there are very few good openGL ES2.0 engines that take advantage of the modern iOS frameworks (e.g. GLkit). Ideally, most of the engine code would be written from scratch but I'm open to suggestions from other coders (see: oolong, isgl3d, sio2, cocos3d, etc). 

 

If you're interested, my suggestion would be to work on getting the core capabilities implemented and checked in.

(We can get you or others a second data set to ensure that switching between models works.) After that I guess it's a question of how much time you have to commit and whether others want to help.

 

Which approach to use for porting is an interesting question.

  - The developer of the Android version of Google Body is on this thread; he analyzed the state of the v1 codebase (which differs from open-3d-viewer primarily in the complexity of the metadata model) and ported it to Java wholesale. You could do the same, or you might investigate running some of the open-3d-viewer Javascript code in a wrapped webview -- this might make it faster to get the metadata model up and running, since it's pretty self-contained.

  - You might also look at the Android source itself as checked in. My understanding is that it's not quite buildable, but the gap to making it work is not large.

  - If you have time to embark on this, we can probably get Body team members to provide more detailed notes on the JS/Java source files.

The wrapped javascript option is intriguing but definitely not the first thing that comes to mind. I'd feel better writing it in ObjC to just have more control over the allocations and access to GCD/operation queue. This is the area where docs would assist most.

 

Does this interest you?

Most definitely. Unfortunately I probably won't have a ton of bandwidth to devote to it until the end of January (startup + academia = no fun). That said, I'll start mapping out some of the core components over the next week or two.

Rich

 

[Redd Kamel]

> I'm assuming the app should run standalone with no required network connection? or is the goal to provide an online viewer for an expanding set of models+metadata?

Yes, standalone is top priority.

> By proprietary format, are you referring to the utf8 encoded vertex data? I don't anticipate too many problems in loading either utf8 or obj models. 

Yes, utf8.

> I was not able to find any textures in the SVN - what format will these come in?

Correct, the sample model has no textures. If you have time to work on this, we'll make certain to get you a model with textures. (We should have a sample with textures anyway.)

> The wrapped javascript option is intriguing but definitely not the first thing that comes to mind. I'd feel better writing it in ObjC to just have more control over the allocations and access to GCD/operation queue. This is the area where docs would assist most.

OK, we'll see about adding more docs.

> Most definitely. Unfortunately I probably won't have a ton of bandwidth to devote to it until the end of January (startup + academia = no fun). That said, I'll start mapping out some of the core components over the next week or two.

Sounds great!

RK

[Rich Stoner]

Redd,

An additional follow-up: Are you (and this project) committed to having a standalone 3D engine that is (relatively) conserved across all three platforms or is there room to play with other open 3rd-party options (GPL safe, of course)? 

RMS

[Nico Weber]

Hi RMS,

I don't think we're married to a specific 3d engine, particularly not
for the mobile versions. I'd love if an eventual iOS version could
share some rendering code with the Android version (that means the
Android rendering code would have to move to C), but that's not a
requirement either I think.

Note that open-3d-viewer is Apache License 2.0, so we're probably not
interested in depending on a GPL library.

Nico

[Rich Stoner]

Hi Nico,

Sounds good - I'll review some options and go with the most rapid option. 

By GPL safe, I meant free of any GPLed code. Sorry for the mix-up. Anything I consider will be MIT/BSD/ASL. 

Best -

Rich

[Redd Kamel]

Sounds great.

[Won]

Hi Rich,

Thanks for volunteering here. It sounds like I should spend a bit of time on a C implementation of the UTF8 mesh decompressor on the webgl-loader project to be included in an iOS version. Or, if you manage it, I'd gladly include it.

open-3d-viewer and webgl-loader are both Apache 2.0 and GPL-free.

-Won

[Nico Weber]

On Tue, Jan 17, 2012 at 9:19 AM, Won <won...@gmail.com> wrote:
> Hi Rich,
>
> Thanks for volunteering here. It sounds like I should spend a bit of time on
> a C implementation of the UTF8 mesh decompressor on the webgl-loader project
> to be included in an iOS version. Or, if you manage it, I'd gladly include
> it.

(I continue to be unconvinced that UTF8 is the best model distribution
format for mobile devices ;-) )

[Won]

Thanks for reminding me! Alternatively, I can add a code path to webgl-loader that emits uncompressed 16-bit vertex data so you can just mmap the data and forward the pointer to glBufferData. I know that's something I promised Nico for Android a long time ago. It's something other folks are interested in as well, now that XMLHttpRequest2 supports typed array buffers. In any case, it depends on whether you expect the download models on-the-fly (eventually, yes!) or whether you want to pre-package the model in the app.

Note that for the uncompressed data, even though there won't be a download space savings,  there would still be a runtime benefit because of the quantization and vertex order optimization.

-Won

[Taylor Wilson]

Hi Nico,

Are you proposing a better format? ;)

Thanks,
-Taylor

[Won]

Hi Taylor,

One realization that came from the Android port was that a bunch of time was spent in decompression; there are some annoying issues (that Nico knows more about) that causes lots of inefficiencies due to the Java to OpenGL ES API boundary.

So, if you only have to download once (e.g. for a pre-packaged app) then it is a better user experience to leave it uncompressed so that it is possible to mmap directly from the stored representation to a VBO, which lets the underlying system handle all the copies for you.

-Won

[Taylor Wilson]

Great observation, and this is probably a naive question, but can these uncompressed files live on the server and participate in server-side zip compression (reduce download size) and still perform well in Android through client side zip decompression in the case where you want to dynamically download content?

Thanks,

-Taylor

[Nico Weber]

On Tue, Jan 17, 2012 at 3:26 PM, Taylor Wilson <tay...@gmail.com> wrote:
> Great observation, and this is probably a naive question, but can these
> uncompressed files live on the server and participate in server-side zip
> compression (reduce download size) and still perform well in Android through
> client side zip decompression in the case where you want to dynamically
> download content?

If you grab the resources from a server, you probably want to use
utf8: Then only one format needs to be on the server, and you're IO
bound anyway.

Nico

[Pat Marion]

Hi Guys,

Rich Stoner pointed me to this discussion. I am a lead developer on
Kitware's VES project. VES is a lightweight, pure C++ OpenGL ES 2.0
rendering engine with an Apache 2.0 license. You can check it out in
action in our newly released KiwiViewer app-

https://market.android.com/details?id=com.kitware.KiwiViewer

I think that VES could be a good fit for open-3d-viewer's Android and
iOS ports, if you're open to external library dependencies. If
someone wants to help me get started by sending a simple C/C++ program
that reads your file format and puts the data into c arrays (vertex
position, normal, color, triangle indices, etc), then I'd be happy to
put together a basic demo that runs on iOS and Android.

In addition to the VES render engine, we offer a higher level C++
application library named kiwi. Kiwi incorporates VTK (the
Visualization Toolkit), cURL, and cJSON. With cURL and cJSON, we can
communicate and fetch data from a web server just as easily as reading
bundled asset files. Having this in the C++ library helps reduce the
burden of maintaining app code in both Java & Objective-C.

Pat

[spirit]

Redd Kamel 

hi friend i am very intered in this project but i am having problem generating the JSON and .JS files for my model i 

tryed the webgl project which has some form of compressor but was not able to use it. i will be very greatful if you will help

me get my site runing since i have decided to use it form my project at school.thanks sooo much