Immersive protein structure analysis with Microsoft's HoloLens

[Scott]

Hey all,

We had an awesome and unexpected evening last night exploring a pre-release version of Microsoft's HoloLens at Open Science Network here in Vancouver. I think this was the first public demo of the HoloLens in Vancouver outside of Alex Kipman's talk at the TED conference. It was totally last minute and the VR Roundtable group at UBC join us to get first hand exposure to the device. Even they haven't seen the HoloLens outside of the web. You can view some photos I took during the evening on our Facebook page.

Having access got me thinking about how the HoloLens can be used for protein structure analysis. Can you imaging walking through a field of protein structures imported from PDB and introspecting how different molecules interact in 3D? Can we build a 3D rendering of the inside of a cell, at least in part? Maybe follow a signalling event as it traverses from the cell surface to the nucleus. Imagine that!

Microsoft's HoloStudio app allows one to import standard STL 3D model files used for 3D printing. The awesome free VMD molecular modelling app allows us to import the publicly available protein structures and, critically, to export 3D structures as STL files. Some of you may have exposure to the traditional publicly available 3D protein modelling tools but it is a huge difference to be able to see a 3D hologram that you can interact with. Eventually I can envisage gestural commands that allow one to modify the amino acid sequence and see how that plays out in the 3D context. Then maybe we can sync those changes to gene synthesis companies to have the DNA for real world work. I love this blue sky stuff!

What would you do if you could build a room full of protein structures? How do you see this tech evolving?

Cheers,

Scott

[Bryan Jones]

It's not an entirely new idea. I'm not sure if VMD has built in 3D support, but other molecular viewers (PyMol and Coot, for example) have built in hardware support for 3D displays. I regularly use a 3D display to look at my proteins and build the models. It's definitely helpful to be able to view them in 3D, it makes it easier to see how things interact and orient. I'm not familiar with the full functionality of HoloLens, but I'd imagine it could extent the 3D functionality beyond the monitor. If it improves the interface, that would be huge. The mouse and keyboard were not meant for interacting in 3 dimensions, and can make manipulating proteins difficult when working in 3D.

--
-- You received this message because you are subscribed to the Google Groups DIYbio group. To post to this group, send email to diy...@googlegroups.com. To unsubscribe from this group, send email to diybio+un...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/diybio?hl=en
Learn more at www.diybio.org
---
You received this message because you are subscribed to the Google Groups "DIYbio" group.
To unsubscribe from this group and stop receiving emails from it, send an email to diybio+un...@googlegroups.com.
To post to this group, send email to diy...@googlegroups.com.
Visit this group at https://groups.google.com/group/diybio.
To view this discussion on the web visit https://groups.google.com/d/msgid/diybio/789f5fa9-e21e-444e-8bdd-6f809201483e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Scott]

Hi Brian,

The HoloLens actually projects the images into your eyes so you "see" the 3D model in front of you. You can walk around the model as well as reach out and "touch" it.  It is different from VR devices like the Oculus in that you can actually see and walk around the room - an ability that is great for in situ gaming but may not be necessary for protein modelling. The TED talk linked in my first post above gives you an idea of how it works. 

Last Saturday evening we tried to upload STL files (the Zika viral coat protein and the smaller Ubiquitin) exported from VMD to the HoloLens. This required uploading first to Microsoft's OneDrive. Unfortunately, the HoloLens couldn't connect to the MS guy's OneDrive account and he thinks it is because the development version of the HoloLens was configured to only connect from within the MS internal network (or it was a bug). He is back in Redmond so when he has time he will try uploading.

I opened the STL protein structure files in the Slic3r 3D printer app and a lot of the goodness in the Protein DataBase file format was stripped out. Not surprising since STLs were designed for 3D printing and not molecular modelling. I think the STLs are fine for basic introspection of the structure model but if you want to, say, highlight acidic amino acids in the structure then you are stuck. This means there is an opportunity for someone to write a HoloLens app that the reads in the native PDB xml file format and provide gestural tools for handling and manipulating the models.

Cheers,

Scott