coordinate system relationships

[Michael Graupner]

Hello, 

I have spend some considerable time trying to fix all the relative coordinate relationships in ACQ4. However, I did not converge to a satisfying solution. I would appreciate some help regarding the following points:

- I can now overlay images from the Camera and the Multiphoton Imager in the Imager module. I tried to align (in x,y) both images using the 

transform:

        position: (shiftX*um,shiftY*um)

in the Camera or the Scanner configuration. However, both ROIs remain centered and cannot be displaced with respect to each other. How can I achieve the alignment? 

- I want to have the global coordinate system stick with the sample (which is the default I guess). However, the image is mirrored with the default settings. When I add a minus sign in the scale settings, then the image is correct but the mapping to the coordinate system is screwed up - it moves in the opposite direction as the actual movement. 

- What is wrong when the anticipated movement of the 2p image in imager is opposite (or perpendicular) to the actual movement? When it is opposite, the image moves in one direction in response to a stage movement. However, once the image is updated through another scan, it jumps in the opposite direction. 

As I wrote, I have tried multiple days to wrap my head around the coordinate system but I didn't manage yet. 

Thank you in advance,

Michael 

[Luke Campagnola]

After having spent many years working with coordinate system transformations in graphics and hardware control, I can say this: it does not get much easier with practice  ;)

On Wed, Mar 2, 2016 at 8:00 AM, Michael Graupner <graupner...@gmail.com> wrote:

- I can now overlay images from the Camera and the Multiphoton Imager in the Imager module. I tried to align (in x,y) both images using the 

transform:

        position: (shiftX*um,shiftY*um)

in the Camera or the Scanner configuration. However, both ROIs remain centered and cannot be displaced with respect to each other. How can I achieve the alignment? 

Ideally, the scanner should be automatically aligned with the camera as described here:

http://acq4.org/documentation/userGuide/devices/ScannerDevice.html#calibration

If this is done correctly, then it will not be necessary to make any manual corrections. If for some reason you are unable to use the automatic calibration, then you should adjust the parameters in `config/devices/Scanner/index` to get the correct alignment.

 

- I want to have the global coordinate system stick with the sample (which is the default I guess). However, the image is mirrored with the default settings. When I add a minus sign in the scale settings, then the image is correct but the mapping to the coordinate system is screwed up - it moves in the opposite direction as the actual movement. 

Yes, the sample is always considered fixed to the global coordinate system, and all other devices move relative to the sample. 

When configuring your camera, you should select scale factors (in config->devices->Camera->transform->scale) that give the image the correct scale and orientation when displayed on screen. On most systems I have worked with, the scale factors turn out to be close to the physical pixel size of the sensor, with negative signs as needed to flip the image. The per-objective scale factors are then handled by the Microscope device (see config/example/devices.cfg). On my systems, I like to have the sample axis that points away from the experimenter to be mapped to the +y axis on-screen, but you're free to choose whatever orientation suits you. To verify the orientation is correct, I usually print a very small "R" on paper and place that in the sample plane.

Once the scale factors are correct, you should be able to draw an ROI around an object and the ROI will follow that object when the stage is moved. If that is not the case, then probably your stage transformation is incorrect. Usually you can fix this by changing config->devices->Stage->scale. NOTE: this is not the same as config->devices->Stage->transform->scale. The former is used when converting the hardware's reported position values into the stage device transformation, whereas the latter is actually applied directly to the stage transformation itself. For example, setting Stage->scale = (1, -1) tells the stage that when it's hardware's y-coordinate increases, then it should translate itself (and its children, including the camera) in the negative y direction. In contrast, setting Stage->transform->scale = (1, -1) is equivalent to telling ACQ4 that you have inserted a mirror at this point in the system that causes the y-axis to become inverted, and would thus cause the camera image to flip upside-down.

 

- What is wrong when the anticipated movement of the 2p image in imager is opposite (or perpendicular) to the actual movement? When it is opposite, the image moves in one direction in response to a stage movement. However, once the image is updated through another scan, it jumps in the opposite direction. 

Hopefully, addressing the first two issues will solve this as well. 

Once you get it working, let's discuss ways to make this process more clear. Do we need more documentation? Perhaps a step-by-step guide to calibration (maybe even a video tutorial) ? Are there ways we can direct this process in software?

 

Luke

[Michael Graupner]

Dear Luke,

thank you very much for your suggestions. They allowed me to fix all the described problems and made me wish your clear instructions were/became part of the ACQ4 documentation. 

One small follow-up questions: In order for a ROI to follow an object, I needed to set  config->devices->Stage->scale to (1,-1,1) . Movement and orientation in ACQ4 are perfect with this setting. However, the coordinate system in ACQ4 is not Cartesian. Could that cause an internal problem? 

Could I change the orientation of the y-axis in ACQ4 but leave everything else the same? 

Cheers,
Michael

--
You received this message because you are subscribed to the Google Groups "ACQ4" group.
To unsubscribe from this group and stop receiving emails from it, send an email to acq4+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/acq4/CACZXET9AJknskxCmAu4aahBSXyF67m-%2Beedn05pRg66Payh87g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Luke Campagnola]

On Mon, Mar 7, 2016 at 7:21 AM, Michael Graupner <graupner...@gmail.com> wrote:

One small follow-up questions: In order for a ROI to follow an object, I needed to set  config->devices->Stage->scale to (1,-1,1) . Movement and orientation in ACQ4 are perfect with this setting. However, the coordinate system in ACQ4 is not Cartesian. Could that cause an internal problem? 

Could I change the orientation of the y-axis in ACQ4 but leave everything else the same? 

I'm not sure what you mean by the coordinate system being "not Cartesian".  If you're concerned about the orientation of the y-axis, though, look at the x,y coordinates of the mouse pointer displayed in the bottom-right of the camera window. These give out the cursor position mapped to the global coordinate system, and you should find that +y points upward on the screen  (and hopefully also that +y on the screen maps naturally to the axis that you would consider to be +y on the sample).  Is this not the case?

Luke

[Michael Graupner]

With Cartesian I mean that the x,y,z coordinate directions are aligned according to the right hand rule. 

The +z direction goes "into" the sample in my system. As a result the +y direction would be downward on the screen in ACQ4. The config->devices->Stage->scale to (1,-1,1) setting achieves that +y movements are mapped onto -y movements in ACQ4. I have attached a sketch to better illustrate the situation. 

Maybe this is all OK. My question is simply if the current coordinate system potentially can cause an issue in ACQ4.

Thanks,
Michael

--
You received this message because you are subscribed to the Google Groups "ACQ4" group.
To unsubscribe from this group and stop receiving emails from it, send an email to acq4+uns...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/acq4/CACZXET_yBHmUKYAUyOJMAWBt3fjWR-G%2BFZJ_7PsTXTKD-_D54w%40mail.gmail.com.

[Luke Campagnola]

Ah, I understand. I usually have my +z axis pointing upward, out of the sample.

The camera module displays +y pointing upward by default. You can reverse this from the context menu for the camera module's viewbox (or you can use viewbox.invertY() from python). Then it should make a bit more sense how to configure the stage and camera transformations to match. I have not tried this, but I can't think of any reason it wouldn't work initially..   It would probably cause some headaches if you decide to use ACQ4's manipulator automation in the future, since this relies on the assumption that -z goes into the sample.

To view this discussion on the web visit https://groups.google.com/d/msgid/acq4/CAFVjdWK-AUruduKAyatCMYGYzKm3ZVmNAtqWfxS01gN0GVJGqg%40mail.gmail.com.

[Michael Graupner]

Ok. Thank you for the clarifications. I will leave my coordinate system as it is for now, as it is working without hitches. 

Your comment made me aware of the Pipettte device, which I haven't noticed so far. I managed to configure a pipette and get the GUI (cameraModTemplate.py) in the Imager/Camera interfaces. Playing around with it gave a lot of errors (which might be due to my socketStage implementation). 

Do you have some descriptions/manual for the Pipette device? For example, why does the camera roi rotate when the pipette orientation is adjusted?

Is it possible to mark the location of a cell and then have a indication where the pipette tip should be in order to reach that target cell when entering the surface and all the way to the target? 

Thanks,
Michael

 

To view this discussion on the web visit https://groups.google.com/d/msgid/acq4/CACZXET9CksBB_TOHoB_mtfbWx98vrR8vhVFMHBx9KAhrAe9R9A%40mail.gmail.com.

[Luke Campagnola]

On Wed, Mar 23, 2016 at 3:06 AM, Michael Graupner <graupner...@gmail.com> wrote:

Ok. Thank you for the clarifications. I will leave my coordinate system as it is for now, as it is working without hitches. 

Your comment made me aware of the Pipettte device, which I haven't noticed so far. I managed to configure a pipette and get the GUI (cameraModTemplate.py) in the Imager/Camera interfaces. Playing around with it gave a lot of errors (which might be due to my socketStage implementation). 

Do you have some descriptions/manual for the Pipette device? For example, why does the camera roi rotate when the pipette orientation is adjusted?

Pipette is pretty new; I've been working on that a lot in the last few months. No documentation yet, but you can see how it is intended to be configured in config/example/devices.cfg (look for the MicroStar and Pipette devices). The major assumption is that you have a micromanipulator that holds a pipette, and you'd like to be able to point to a location in the camera module and have the pipette go there. So to start off, you need a stage device that controls the micromanipulator, and a pipette device that has the stage as a parent. Both of these should probably be independent of the stage that moves the microscope, so your device hierarchy might look like:

    - Stage

         - Microscope

              - Camera

    - Manipulator1

        - Pipette1

    - Manipulator2

        - Pipette2

When you adjust the orientation of the pipette, it actually changes the transformation of its parent stage. If you had a pipette with its parent set to the microscope stage, then that would explain why the camera appears to rotate with the pipette orientation.

After devices are configured, I do the following to calibrate the pipette:

1. Bring the pipette tip to the center of the camera view

2. Click "set center", and then click on the tip of the pipette. 

3. Click "set orientation", then move the pipette forward across the screen, and then rotate the orientation arrows such that the x-axis points directly to the new position of the pipette.

4. Drag the arrows by the handle at their origin to the new position of the tip, and un-click "set orientation".

5. Try moving the pipette. If all is well, then a blue arrowhead should follow the tip around. If the y-axis motion (perpendicular to the pipette) is backward, then click "set orientation" again and reverse the direction of the y-axis.

The above orientation calibration should only need to be done once. After that, just use "set center" whenever you need to recalibrate the pipette position. The rest of the buttons have various functions related to pipette handling:

"set surface" is a single lonely button that the microscope adds to the camera window. Use it to mark the location of your sample surface.

"set target" lets you click in the camera window to set the final destination for the pipette (usually a cell). The target is a point in 3D, so you do need to have access to the focus depth via your microscope stage or focus drive. 

"search" causes the microscope to focus 1 mm above the sample surface and brings the pipette under the center of the objective based on its last known location. From there, you should be able to find the tip and recalibrate its position. I do this step immediately after placing each new pipette into its holder.

"above target" moves the pipette directly over its target (which should be set using "set target"), and about 50 um above the sample surface. This allows one last recalibration before going to the cell. This step may or may not be necessary depending on the accuracy of your manipulator.

"approach" brings the pipette into axial alignment with its target, about 50 um above the surface. From here, you should be able to manually drive the pipette diagonally directly to the target.

"target" brings the pipette directly to its target using a diagonal approach. I generally don't use this function, as I prefer to do this step manually from the "approach" position.

Somewhere I have video of this system in action.. if I find it, I'll post it online.

[Michael Graupner]

Dear Luke,

thank you for the comprehensive answer and explanations. This is a very exciting new tool. I was thinking of something similar lately and I am thrilled to find that you already implemented it ... . 

On Wed, Mar 23, 2016 at 6:40 PM, Luke Campagnola <luke.ca...@gmail.com> wrote:

On Wed, Mar 23, 2016 at 3:06 AM, Michael Graupner <graupner...@gmail.com> wrote:

Ok. Thank you for the clarifications. I will leave my coordinate system as it is for now, as it is working without hitches. 

Your comment made me aware of the Pipettte device, which I haven't noticed so far. I managed to configure a pipette and get the GUI (cameraModTemplate.py) in the Imager/Camera interfaces. Playing around with it gave a lot of errors (which might be due to my socketStage implementation). 

Do you have some descriptions/manual for the Pipette device? For example, why does the camera roi rotate when the pipette orientation is adjusted?

Pipette is pretty new; I've been working on that a lot in the last few months. No documentation yet, but you can see how it is intended to be configured in config/example/devices.cfg (look for the MicroStar and Pipette devices). The major assumption is that you have a micromanipulator that holds a pipette, and you'd like to be able to point to a location in the camera module and have the pipette go there. So to start off, you need a stage device that controls the micromanipulator, and a pipette device that has the stage as a parent. Both of these should probably be independent of the stage that moves the microscope, so your device hierarchy might look like:

    - Stage

         - Microscope

              - Camera

    - Manipulator1

        - Pipette1

    - Manipulator2

        - Pipette2

When you adjust the orientation of the pipette, it actually changes the transformation of its parent stage. If you had a pipette with its parent set to the microscope stage, then that would explain why the camera appears to rotate with the pipette orientation.

Ok, you are right. I use the microscope stage as stage device for the pipette, hence the rotation. 

On my setup, the MicroscopeStage and the PipetteStages are controlled with a different computer. That is the reason I wrote the socketStage device class which allows to send and receive stage commands over the network. It works without a hitch for the MicroscopeStage. 

However, I would have to use the same stage device - the network socket - , with a different set of commands for the Pipette control. Do yo have any idea how I could implement this? 

Thanks,

Michael

 

After devices are configured, I do the following to calibrate the pipette:

1. Bring the pipette tip to the center of the camera view

2. Click "set center", and then click on the tip of the pipette. 

3. Click "set orientation", then move the pipette forward across the screen, and then rotate the orientation arrows such that the x-axis points directly to the new position of the pipette.

4. Drag the arrows by the handle at their origin to the new position of the tip, and un-click "set orientation".

5. Try moving the pipette. If all is well, then a blue arrowhead should follow the tip around. If the y-axis motion (perpendicular to the pipette) is backward, then click "set orientation" again and reverse the direction of the y-axis.

The above orientation calibration should only need to be done once. After that, just use "set center" whenever you need to recalibrate the pipette position. The rest of the buttons have various functions related to pipette handling:

"set surface" is a single lonely button that the microscope adds to the camera window. Use it to mark the location of your sample surface.

"set target" lets you click in the camera window to set the final destination for the pipette (usually a cell). The target is a point in 3D, so you do need to have access to the focus depth via your microscope stage or focus drive. 

"search" causes the microscope to focus 1 mm above the sample surface and brings the pipette under the center of the objective based on its last known location. From there, you should be able to find the tip and recalibrate its position. I do this step immediately after placing each new pipette into its holder.

"above target" moves the pipette directly over its target (which should be set using "set target"), and about 50 um above the sample surface. This allows one last recalibration before going to the cell. This step may or may not be necessary depending on the accuracy of your manipulator.

"approach" brings the pipette into axial alignment with its target, about 50 um above the surface. From here, you should be able to manually drive the pipette diagonally directly to the target.

"target" brings the pipette directly to its target using a diagonal approach. I generally don't use this function, as I prefer to do this step manually from the "approach" position.

Somewhere I have video of this system in action.. if I find it, I'll post it online.

--

You received this message because you are subscribed to the Google Groups "ACQ4" group.
To unsubscribe from this group and stop receiving emails from it, send an email to acq4+uns...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/acq4/CACZXET9BWoRKCO7esB8KUmXkwTVxXVBJv5jPy6qy5fcb0ske-Q%40mail.gmail.com.

[Luke Campagnola]

On Thu, Mar 24, 2016 at 1:29 AM, Michael Graupner <graupner...@gmail.com> wrote:

Dear Luke,

thank you for the comprehensive answer and explanations. This is a very exciting new tool. I was thinking of something similar lately and I am thrilled to find that you already implemented it ... . 

You might also want to have a look at my `multipatch` branch. This has some updates to the pipette device as well as a new module that is meant for managing multiple patch pipettes simultaneously. It's pretty early in development right now, but we have big plans for it..

 

On my setup, the MicroscopeStage and the PipetteStages are controlled with a different computer. That is the reason I wrote the socketStage device class which allows to send and receive stage commands over the network. It works without a hitch for the MicroscopeStage. 

However, I would have to use the same stage device - the network socket - , with a different set of commands for the Pipette control. Do yo have any idea how I could implement this? 

My recommendation would be to implement the device as if it were local first, and then we can think about how to connect together two instances of ACQ4 running on different machines. It should be relatively straightforward to make remote devices appear exactly like local devices using an object-proxying system (this is how I was able to connect a 32-bit python instance with a MultiClamp device to a separate 64-bit ACQ4). I have been working on a system that might work well for this as part of another project:  http://github.com/pyacq

If this sounds interesting to you, then I'll try to set up a demonstration that you can work from..

[Michael Graupner]

I feel like getting ACQ4 running on the other computer (#2) would be some work. But maybe I am wrong. Let me maybe describe my system in more detail. 

I already have a program running on computer #2 which allows me to control pipettes and stage ( https://github.com/mgraupe/ManipulatorControl ) . This program can listen, receive and execute commands which it receives over the network socket. The ACQ4 counterpart on computer #1 to this program is the  socketStage device ( https://github.com/mgraupe/acq4/tree/socketStage ). It gives allows to control the stage from within ACQ4 on computer #1. 

It is very simple for me to add functionality to the ManipulatorControl program on computer #2 to control pipette movement. What I would need that in ACQ4 the socketStage device has three instances with all the three talking to the same socket on computer #2. 

I hope it is clearer now. 

Thanks,
Michael

 

--
You received this message because you are subscribed to the Google Groups "ACQ4" group.
To unsubscribe from this group and stop receiving emails from it, send an email to acq4+uns...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/acq4/CACZXET-fczZRTbsC0ss3bMVOBn2zKMRbXhJKzESN-deEzgR%2BVw%40mail.gmail.com.

[Luke Campagnola]

Your setup is clear, but I don't understand what the problem is -- what's preventing you connecting multiple socketStage devices? You said that you need a different set of commands to control the pipette, but it seems like if you wrote socketStage for one set of commands, then it should be straightforward to write another with a different set of commands?

To view this discussion on the web visit https://groups.google.com/d/msgid/acq4/CAFVjdWLjf0wf-wMiwtf%3D6HqO3W1BF763mzUp8GrCi-kVze4f6w%40mail.gmail.com.