Brekek Pointcloud + Unity Visual Effect Graph

[Charlie]

Hey,

Does anyone have any ideas on how I could render a video / point cloud from Brekel in Unity's VFX Graph?

I've asked Keijiro (https://github.com/keijiro), as he has projects available for Depthkit, 4DViews, Azure Kinect and Realsense, but he wasn't sure. 

For example, in his Depthkit project (https://github.com/keijiro/Dkvfx) you convert the combined-per-pixel .mp4 to a HAP format .mov, and use colour and position attribute maps in the VFX Graph.

Thanks!

[Daniel]

Hey Charlie,

There's asset on the unity asset store that you might want to check out for that:

https://assetstore.unity.com/packages/tools/integration/azure-kinect-examples-for-unity-149700 

Cheers, 

[Charlie]

Hey Daniel, I've seen that asset and it looks really cool, but I'm mainly interested in recording with multiple sensors (Kinect V2 and Realsense D415) in Brekel and editing and cropping there before bringing it in to VFX Graph.

[Charlie]

Is it possible to export UV textures in this format?

 

If so, I could probably bring this in to VFX graph by exporting an alembic mesh and converting the texture .pngs to a HAP .mov file.

[Brekel]

Alembic files have the option to be exported with UVs, they automatically match the exported textures.

There's also the option to export Alembic with vertex colors now, not sure if Unity supports those.

Also not sure what the VFX graph supports.

Op zaterdag 15 augustus 2020 om 15:31:34 UTC+2 schreef Charlie:

[Charlie]

I've been experimenting with the Alembic UVs and vertex colors, but haven't had any luck.

Keijiro's 4DViews example is the only one that uses Alembic. His Depthkit, Kinect and Realsense projects all convert the color/depth data into two dynamically animated attribute maps (textures), position map and color map, which can be used in the "Set Position/Color from Map" blocks in the VFX graph. 

Hope you can add this as a feature at some point, or that Keijiro makes a Brekel project too! Here are some tests with the Realsense Rsvfx project -

https://www.youtube.com/watch?v=fidfhljp9vY

Cheers!

[Charlie]

Hi again,

Just checking in a year later to see if anyone has managed to get Brekel point clouds in to VFX graph somehow? I can see that quite a few export options have been added to Brekel recently. 

Depthkit, EF EVE, Keijiro's depth sensor projects, and the Record3D iOS app can all be used in VFX graph by converting the videos to 2D textures for colour and position, so would really love this option for Brekel too. 

Cheers,

Charlie

[Leonardo Scarin]

There is an abcvfx repository by Keijiro on their github! That, and his 'hap player' used to render video texture, should do the job. The only thing I had trouble with was really just the texture mapping. It's always off and I can't manage to fix it to the particle system.

[Stefano Casella]

Hey Charlie,

I was reaading your post and I know we are about Brekel, but how do you convert videos to 2D textures for colour and position with Depthkit exports? I can't make Keijiro DKVFX run on my machine...

thanks!

[Charlie]

Thanks for the reply. I tried this method using Keijiro's 4DViews VFX projects but also couldn't figure it out. I asked him a while back and he thought alembic mesh would work, but that there would be lots of pitfalls and problems to solve!

[Charlie]

Hey, you need to convert the Depthkit exported video to HAP mov as described here - https://forums.depthkit.tv/t/working-with-hap/299

Place the converted mov in StreamingAssets folder, right click the txt file and select “Depthkit” -> “Convert Metadata”, and follow the setup in the Dkvfx sample scene.