Multiple Kinects + Unity + Playback (Live and/or Recorded) in Unity in VR

[ty weaver]

Background

I have Brekel Pro PointCloud v2

I have Brekel Multi Sensor – Alpha

I have 4 Kinects

What I would like to be able to do:

1. Live Stream the 4 Kinect streams (that are already aligned so they present a single, 3D space) to a Unity application and then view that single, 3D space with my VR headset in Unity (Vive headset).

2. Record the 4 Kinect streams (again...already aligned) and then play it back as a single, 3D space in Unity and view it with my VR Headset (Vive) in Unity.

For #1 - I can't seem to find anything that says Brekel can do live streaming into Unity.

Can I livestream to Unity from Brekel Pro Point Cloud 2?

Can I livestream to Unity Multiple Kinects (using Multi Sensor Alpha)

If I can, can you direct me to the tips, guides, documentation, forum posts, etc?

For #2 - I have recorded from 1 Kinect and been able to play that recording back into Unity and then view it in VR with my headset.

I am reading in the forums that playing back recordings from multiple Kinects is problematic (in my case, I want to do 4 kinect sensor recordings at the same time, aligned, so they present a single, 3D space).

It sounds like getting all 4 streams to stay in Sync in Unity is not easy (or maybe not possible?

I read someone suggested just editing all 4 recordings to have the same # of frames.

I also read that even that may not guarantee they stay in Sync.

Someone suggested writing their own code to try and keep all the kinect recordings sync'd.

Is there a way to ensure the 4 recorded streams stay in Sync while playing back in Unity?

If there is, can you direct me to this method, guide, document, etc?

[Brekel]

#1 No, network streaming to Unity is not supported. Also live data from 4 Kinects would generate a tremendous amount of data, even for a LAN.

#2 Sure, that should be possible, especially with the new BRF file format and Unity plugin.

You may need to fiddle a bit to optimize sync, for example by playing with the first frame setting.

The script can of course also be altered to your own liking.

Let me know if you encounter problems and if you have ideas or code that improves things.