Dynamic mapping of seamless visual projection

[Lisanne Schulze]

Dear group members, 

I am currently trying to build a seamless presentation of a pattern as a visual stimulus, which is being dynamically mapped to a screen, depending on behavioral output
(here for example the mouse position in XY).

The goal is to have a fish swim in an endless environment.

Approach:
I am testing 2 approaches in order to dynamically change the position and perspective of the stimulus in response to behavioral output:
1) via the projection of a loaded model (using DrawModel, to load the exemplary object file "tinker") and

 
2) via trying to fill the entire shader content (meaning screen) using Bonvision's DrawCheckerboard to generate a pattern.

Status:
1) Loading an object file into the workflow allows controlled dynamic mapping depending on the mouse position, however it does not enable me to have a seamless stimulus. Certain behavioral outputs can cause the object to disappear from the screen.
Increasing the object dimension compensates for this effect to a certain degree, however it doesn't fulfill the actual task.

2) When trying to fill the entire screen with an ideally endless stimulus by generating a Bonvision pattern using DrawCheckerboard (NormalizedView) I get the error "Disposable already assigned ''. What does this mean exactly and how can I circumvent it? Is this connected to the Behavioral Subject created during the stimulus generation (in Draw_Checker) and the published Subject integrated into the mapping pipeline (including CubemapView, RenderCubemap, DynamicViewWindow and PerspectiveMapping)?

I'd be thrilled about your help regarding the generation of an endlessly ongoing, visual pattern, linked to behavioral output for controlling the animal's virtual position and perspective. 

Filenames:
1) "DynamicMapping_DrawModel_Obj-file_to_CubemapView" (+ "tinker.obj")
2) "DynamicMapping_DrawCheckerboard_Subject_to_CubemapView"

Best regards, 
Lisanne

[Lisanne Schulze]

Dear group, 

I'm using the following workflow (posted above) to display a 3D object using DrawModel.

This enables me to show a visual environment which is being regulated dynamically according to input deriving from behavioral parameters (e.g. outputs from the computer mouse or fish tail movements as in my specific).

To better understand what's happening experimentally I've attached a video file.

Unfortunately, my three-dimensional object isn't infinite which is why the stimulus can disappear from the shader window after a while depending on the animal's movement.

My aim is to create an endless visual environment, meaning an infinite visual pattern (any pattern will do at this point). 

In order to get there, I’m thinking of generating a stimulus, which I can “repeat” in X and Y (for example an endless checkerboard) - resulting in a seamless representation of a dynamic visual environment.

When trying to create and connect a visual stimulus directly in Bonsai though (I’ve tried Bonvision’s DrawCheckerboard, which works on it's own but not in my workflow), I get the error "disposable has already been assigned", which I can't overcome. 

My 2 questions are:

1) Is it possible to create an infinite visual environment (when avoiding to use DrawModel of a finite object, and instead start) using e.g. Draw(TexturedModel or Image) and trying to repeat it? (Again, the goal is a pattern which does not end in XY, no matter the directional input.)

2) How to avoid the error "disposable has already been assigned"? Should I store my behavioral and visual environment objects in a ResourceSubject node to solve this problem?

Help would be greatly appreciated.

Thanks so much and best regards, 

Lisanne

[Gonçalo Lopes]

Hi Lisanne and welcome to the forums,

This looks like a really cool setup, hopefully we'll be able to get it working. Re. your questions in reverse order:

2) The reason for the error is you are trying to use CreateWindow twice in the same workflow. Bonsai only supports rendering to one shader window at a time for now. If you disable the Draw_Checker group or the duplicate CreateWindow inside, the workflow will run successfully. I'm not sure it is doing what you want though. If I understand correctly, you want to draw the checkerboard stimulus in 3D. To use the DrawCheckerboard primitive you need a single input matrix, whereas for 3D models you use both a view and a perspective matrix. You can workaround this by premultiplying both (attached), but I'm not sure if this will achieve the desired effect... 

1) It is possible to draw an "infinite" plane by taking advantage of the mathematics of homogeneous coordinates. For example, see opengl - Rendering infinitely large plane - Stack Overflow. However, to implement this you would have to modify the default shader pipeline of BonVision to allow for model vertices specified in homogeneous coordinates (which is very doable), and you would then have to probably write your own pixel shader to tile the texture (also doable). Otherwise if you just try to just apply a texture across such an infinite plane it would get hopelessly distorted unless you generate the pattern programmatically as well.

The above is all doable, but I still somehow think the easiest way will be to just draw a very large plane object that will tile the space for all meaningful values of the rotation parameters. This needs to be large, but not really infinite due to finite floating point precision.

At some point I did toy around with rendering such infinite planes and I can try to pull up an example, but hope the above helps in the meantime.

--
You received this message because you are subscribed to the Google Groups "Bonsai Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/67fc179c-63ac-4d0d-9048-20ac5f59a42bn%40googlegroups.com.

[Lisanne Schulze]

Dear Gonçalo,
I hope you're well! Thank you so much for your feedback! This was of great help!
Having played around with the setup I'm current not sure if a 3D solution for the visual presentation of a referential pattern e.g. a checkerboard is best for giving the tethered fish the impression to be freely swimming. A 2D display of a checkerboard (or another pattern e.g. speckles loaded from an image) might actually suffice, especially when MeshMapped to a non-planar projection screen surrounding the fish. I'll send you updates on this once implemented.
One issue in 3D was the fact that the pattern could "drift" away - while keeping it in the same plane when moving it in X and Y is rather what we desire - which kind of makes 3D redundant I assume.

Question:
However, in 2D I'm running into the following problem. I'd like to address one variable in the ViewMatrix in order to update the orientation of the pattern, meaning its alignment in respect to the animal's swim direction. While the pattern moves in X and Y (see video: pattern can change in left/ right & forward/backward directions) it does not align angularly to the animals orientation (basically a short tilt of the horizontal plane), meaning its horizontal always remains at perfect 90 degrees to the animals midline. I would be very grateful if you could indicate how to address a rotation on this axis. I'm currently generating a newX and newY from behavioral parameters, which creates a VirtualPosition for each datapoint. This VirtualPosition feeds into the CubeMapping of the pattern via Eye.
I wonder if I can use a third variable to address the angular adjustment of the pattern? And if so, would this variable address the third column of the ViewMatrix?

Also, I'm wondering if the lacking "tilt" of the pattern could be connected to the fact that I'm simply not able to see it due to the Screen properties used for the CubeMapping, even though the pattern MIGHT BE "tilting". (I CreateExtrinsics for my Screen using Rotation: 0, 0, 0 and Translation 1, 1, 1. I think my Eye is standing on face 3 and looks at face 0.)
Would you advise PerspectiveMapping to be a better solution for projecting a pattern and regulating its navigation by using Eye and possibly Target to control for the different behavioral inputs? So far, I haven't succeeded in regulating a combination of 1) left/ right movements, 2) forward/backwards movements and 3) angular adjustments with Perspective Mapping either.

Attached, I'm sending the core elements of the workflow.

I haven't translated it into a computer mouse example, as the fish's behavior regulating pattern navigation is slightly different, so I'm not sure if it makes sense. But please let me know if you think this would make things easier for you in order to judge. 

I'd be very happy about your input. I'm especially curious if and how you would address a third variable for a VirtualPosition to regulate the pattern in an angular manner.

Best regards from UCSD, Lisanne

[Gonçalo Lopes]

Hi Lisanne,

Do you mind attaching this updated workflow file so I can experiment and try to understand it? I'm not sure yet if I am clear about what you are trying to achieve: would you like the pattern to rotate dynamically around an arbitrary 3D axis (i.e. the animal viewpoint?) or is a 2D rotation around the Z axis enough? You can always remap the 3D orientation of the Z axis using the ViewWindow node, and if you are struggling with the automatic calibration you can always just tweak the values manually.

A schematic of the intended effect might also be very helpful.

--

You received this message because you are subscribed to the Google Groups "Bonsai Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/0391d4c5-4dd2-4422-b612-4ef35a1ba1a1n%40googlegroups.com.

[Lisanne Schulze]

Dear Goncalo, 

Thanks a lot for your response! Absolutely, attached I'm sending a scheme of the three desired effects for controlling the visual environment (reference pattern, to be navigated by motor behaviour). It is supposed to 1) move left/ right (X, pink), 2) move backwards for simulating forward swims (Y, pink), and 3) tilt temporally for angular alignment when the fish changes its direction (possibly via a third variable, Z, yellow). In the fish workflow, X and Y work by creating a VirtualPosition, but the tilting is not implemented. I have generated a subject called ViewAngle (which could be Z?), to tackle this goal, but didn't succeed. I'm not sure if generating a variable Z to build a Vector 3 could solve controlling movement on that third axis to "tilt" the pattern. Does this make sense?

Additionally attached, the fish workflow and a closely related computer-mouse workflow (which is slightly different in terms of input, but maybe it helps, since you won't have a fish).

I hope this explains my intention, but please don not hesitate to ask me for additional information if I wasn't clear. I'm more than happy to send more information asap.

Thanks so much for your help and best regards from UCSD, 

Lisanne

[Gonçalo Lopes]

Hi Lisanne,

If I understand the problem correctly, you could be better off simply drawing your checkerboard plane into a textured 3D object and then just shifting the texture and tilting the plane by rotating around the X axis. To do this you need to render the checkerboard not to a cubemap, but to a normal texture using RenderTexture. Because DrawCheckerboard does not currently allow parameterizing the horizontal and vertical shifts, you also need to render it first to a normal texture so you can then use DrawImage to introduce the shift. The main bits will look like so:

I'm also attaching an example workflow that you could try to tweak to achieve this.

Hope this helps.

To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/aa22a0ac-e069-4c03-9e2d-072f5e92f91en%40googlegroups.com.

[LBLab]

Dear Goncalo,
Thank you so much for your reply in between the years! I hope 2022 will be a good one for you and Bonsai! :)
This looks very interesting the way you did it, thanks a lot!!

Attached I'm sending how I have tackled this quest in the meantime, as well as a new question regarding an error.

I have implemented a pseudo freely swimming sensation for the fish (as well as I could so far) by loading a video from which I read-out the tail movements that control rotation and translation of the checkerboard - which doesn't work too bad.

My question:

1) When switching from VideoInput (FileCapture) to recording a real fish using the node LiveFish (SpinnakerCapture), the same script crashes after running a few seconds and I do get an error saying:

"Insufficient memory: Failed to allocate 3145728 bytes in function cv::OutOfMemoryError, alloc.cpp(52)".

Is there an easy fix to this? Or does this mean the PC (new and empty) is over-challenged?   

Also, one of my next goals  will be to translate the projection (currently 2D on a screen below the fish) onto a spherical arena, for which I will use Mesh Mapping. Is it necessary/ recommended to render the checkerboard to a normal texture (just like you did) instead of cubemap as I'm currently doing, in order to do so?

(The ultimate idea: the animal perceives to be freely navigating through its' visual environment in closed loop. Later I will switch to open loop by disabling the control for navigating at certain angles meaning  the ability to swim into certain directions).

I'd be very grateful for your input. It would be super cool if we could get this scenario going in Bonsai I think. :)

Thanks a lot for your prior help and best regards from San Diego!

Lisanne

[LBLab]

As an addition, I'm sending a video if that helps to run it.