[micro-manager-general] Andor iXon+, iXon3 bug?, need help
Lukas Hille
I need help with two Andor cameras.
Two Andor cameras are used in dual view (Multi Camera adapter with
Andor: iXon+ and Andor: iXon3).
Acquisition is done by a burstAcquisition macro (equal to the script),
but the described behavior applies also to MDA.
With Frame transfer enabled and at higher Acquisition speeds (binning 2,
0.9us vertical clock, 20ms exposure, full ROI) i get an buffer overflow.
It appears that the device adapter before 14.April2019 "old" returns
very quick from mmc.popNextTaggedImage() and therefore produces no
buffer overflow, while the adapter after 14.April2019 "new" takes a very
long time to return from this function.
This also occurs with only one Camera in use, but with two cameras the
return time is to long to handle the data flow.
A second issue is, that with Multi Camera adapter, the camera referring
to channel 0 (independent which one) stops acquiring after some time
(~60.000 images in Trigger: software).
This applies to the "old" and the "new" Andor device adapter.
With Multi Camera adapter and external trigger (one source for both
cameras, Trigger set to External on both cameras), the camera referring
to channel 0 stops acquiring within 0 to some thousand images)
I don't get any debugging error message with the new adapter.
With the old adapter i get for example: "GetNumberNewImages error:
first: 3473 last: 3473" for the camera on channel 0 before it stops working.
The goal would be to acquire about 200,000 time points with external
trigger at 20ms exposure time.
I tried to understand the differences in the code from commits on
14.April, but I didn't figure out why this makes a difference for the
popNextTaggedImage() function.
Also: the error message "GetNumberNewImages error:" doesn't make any
sense to me, referring to the situation that no new images are acquired.
I would be very happy to get assistance on this one.
Software:
AndorDriverPack2 - 2.103.30031.0
Micro-Manager 2.0.0-gamma1 20200222
mmgr_dal_Andor.dll from 1.04.2019, 16.04.2019 and 22.02.2019 (tried many
many more)
best
Lukas
--
____________________________________________
Lukas Hille, M.Sc.
microscopy engineer
Lukas...@imp.ac.at
M +43 660 349 169 3
Department of Neurobiology, University of Vienna
Campus-Vienna-Biocenter 1
1030 Vienna
AUSTRIA
https://neuro.univie.ac.at/
https://www.imp.ac.at/groups/manuel-zimmer/
Part of Vienna BioCenter
www.viennabiocenter.org
____________________________________________
_______________________________________________
micro-manager-general mailing list
micro-mana...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/micro-manager-general
Lukas Hille
Hi Pedro,
I am using the 64bit version.
ImageJ.cfg is set with -Xmx6000m
The Sequence Buffer Size is 512MB which can hold 2048 images.
The computer has 32Gb of RAM.
Single Andor camera:
With the old adapter the average return time of
mmc.popNextTaggedImage(); is 1ms (2x binning --> 256x256 pixel,
16bit)
With the new adapter (past 14.April19) the average return time of
mmc.popNextTaggedImage(); is 18ms (exactly the same config file,
just changed mmgr_dal_Andor.dll)
With the Multi Camera adapter:
old adapter: 2ms
new adapter: 36ms --> not sufficient for 100fps (2x 50fps)
I am also curious why the Multi Camera adapter doubles the time to return a single image.
With the "new" adapter the Sequence Buffer fills up quite fast
(observed with Sequence Buffer Monitor).
With the "old" adapter the Sequence Buffer Monitor shows rarely
1-2 images.
I can't just increase the buffer size, it would exceed due to the
long recording time.
Also: the System is tracking on the camera stream, therefore a
large amount of images in a huge buffer causes a fatal delay for
the tracker.
Further Interesting:
With "Trigger: Software", on the "new" adapter the cameras are
slowing down a lot and therefore the buffer is kept below 100
images..
But with external Trigger the buffer overflow occurs quite quick
(but the cameras stay at pace).
The readout time of the cameras is indicated with about 16ms.
The script i used to investigate in the timings:
mm.scripter().resetInterpreter(); //Reset Interpreter to get rid
of old stuff
mm.scripter().clearMessageWindow(); //Clear Script output window
/**
* A burst aquisition test script
*/
import org.micromanager.interal.utils.JavaUtils;
//import org.micromanager.data.Datastore;
// User defined variables:
int nrFrames = 1000;
int loopSleepTime = 3;
// defining variables
long taggedTime = 0;
long endSequenceTime = 0;
int loopCount = 0;
// stop running acquisitions
if (mmc.isSequenceRunning()) {
mmc.stopSequenceAcquisition();
}
// RW ram store
storeRW = mm.data().createRewritableRAMDatastore();
// Create a display to show images as they are acquired.
mm.displays().createDisplay(storeRW);
// actual acqusition in a try block to catch errors
try{
// start acqusition and run until it is stopped by command
// the sequence acquistion with preset number of Images did cause
problems
mmc.startContinuousSequenceAcquisition(0);
// Set up a Coords.CoordsBuilder for applying coordinates to each
image.
builder =
mm.data().getCoordsBuilder().z(0).channel(0).stagePosition(0);
while (mmc.getRemainingImageCount() > 0 ||
mmc.isSequenceRunning(mmc.getCameraDevice())) {
if (mmc.getRemainingImageCount() > 0) {
timeStart = System.currentTimeMillis();
tagged = mmc.popNextTaggedImage();
timeEnd = System.currentTimeMillis();
taggedTime = taggedTime + (timeEnd - timeStart);
frameNumber = tagged.tags.getInt("ImageNumber");
// stop sequence after nrFrames images
if (frameNumber >= nrFrames) {
timeStart = System.currentTimeMillis();
print("stop sequence");
mmc.stopSequenceAcquisition();
print("sequence stopped");
timeEnd = System.currentTimeMillis();
endSequenceTime = taggedTime + (timeEnd - timeStart);
}
// for visualisation
image = mm.data().convertTaggedImage(tagged,
builder.time(0).build(), null);
storeRW.putImage(image);
print("image Nr.: " + frameNumber);
loopCount++;
}
else {
java.lang.Thread.sleep(loopSleepTime);
}
}
}
catch (Exception exception) {
print(exception + " - something went wrong");
}
if (mmc.isSequenceRunning()) {
mmc.stopSequenceAcquisition();
}
storeRW.close();
print ("acqusition done");
print ("sum tagged time: " + taggedTime);
print ("end sequence time: " + endSequenceTime);
Thanks for helping me.
kind regards Lukas
Hi Lukas,
Just as a sanity check, have you set the circular memory buffer to an appropriate size?
Kind regards,Pedro Almada
Lukas Hille
good to know, thanks!
I will contact Andor.
Apart from the "speed / buffer overflow" issue, can you also verify that
one of the cameras (camera 0) stops acquiring after some images?
(~60,000 with Trigger:software and varying with Trigger:external)
This applies to all adapter versions.
I am fine with the old driver, but dealing with breaks every some
thousand images is a pain.
We have some "professional" software too, but they do even worse (i am
super happy with MM).
I have to check if the "dual camera stop" on channel 0 is also a Andor
issue, i didn't try long lasting acquisitions with other cameras.
best
Lukas
On 2020-03-12 18:55, Nico Stuurman wrote:
> Hi Lukas,
>
> I am afraid that I can not help much at this point in time, but we ran
> into similar issues with our dual camera setup (2 iXons), and we are
> still using the "old" Andor adaptor code on our dual camera
> microscope. The Andor engineer at the time told me that he could
> reproduce the issues I was seeing (will have to go back to see if
> those were the same you were seeing) with VS2010, but that they went
> away with the newer VS that he was using. Although that did not make
> a lot of sense to me at the time, I very much do want to see MM to
> migrate soon to new compilers, to at least make sure whether or not
> that is an issue.
>
> It would be great if you could contact Andor technical support about
> these issues, and see if they can raise it to a point that their
> software developers get involved. At that point, please do get me in
> the loop as well. Andor maintains the device adaptor code, and the
> April 2019 update was quite drastic, but certainly left a number of
> issues behind. I am afraid that they are in the best position to fix
> this (but I'll be happy to advice).
>
> Best,
>
>
> Nico
>>> Lukas...@imp.ac.at <mailto:Lukas...@imp.ac.at>
>>> M +43 660 349 169 3
>>>
>>> Department of Neurobiology, University of Vienna
>>> Campus-Vienna-Biocenter 1
>>> 1030 Vienna
>>> AUSTRIA
>>>
>>> https://neuro.univie.ac.at/
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__neuro.univie.ac.at_&d=DwMDaQ&c=iORugZls2LlYyCAZRB3XLg&r=UwP8SWqih8VHO1LwZpgcx83I4o21yLj6V6QD-25Dt4I&m=Azgfttsqypz0DvZFBuWwdGg8HC4NaQyiaa1fqvYEv0A&s=vOEXQZsKGgi3nnbjzvgQn42ESKwizdX8-w-cF9pJs3k&e=>
>>> https://www.imp.ac.at/groups/manuel-zimmer/
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.imp.ac.at_groups_manuel-2Dzimmer_&d=DwMDaQ&c=iORugZls2LlYyCAZRB3XLg&r=UwP8SWqih8VHO1LwZpgcx83I4o21yLj6V6QD-25Dt4I&m=Azgfttsqypz0DvZFBuWwdGg8HC4NaQyiaa1fqvYEv0A&s=asG_6hmBgrmDz6Rbwm0hll60QnyPPogr1kqoz9eEq6Y&e=>
>>> Part of Vienna BioCenter
>>> www.viennabiocenter.org
>>> ____________________________________________
>>>
>>>
>>>
>>> _______________________________________________
>>> micro-manager-general mailing list
>>> micro-mana...@lists.sourceforge.net
>>> <mailto:micro-mana...@lists.sourceforge.net>
>>> https://lists.sourceforge.net/lists/listinfo/micro-manager-general
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.sourceforge.net_lists_listinfo_micro-2Dmanager-2Dgeneral&d=DwMDaQ&c=iORugZls2LlYyCAZRB3XLg&r=UwP8SWqih8VHO1LwZpgcx83I4o21yLj6V6QD-25Dt4I&m=Azgfttsqypz0DvZFBuWwdGg8HC4NaQyiaa1fqvYEv0A&s=9i6pdTjg_qT2toVJkbzjoQnylKRxs8CVRcuuhVjnU9Q&e=>
>>>
>>>
>>>
>>> _______________________________________________
>>> micro-manager-general mailing list
>>> micro-mana...@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/micro-manager-general
>> --
>> ____________________________________________
>> Lukas Hille, M.Sc.
>> microscopy engineer
>> Lukas...@imp.ac.at
>> M +43 660 349 169 3
>>
>> Department of Neurobiology, University of Vienna
>> Campus-Vienna-Biocenter 1
>> 1030 Vienna
>> AUSTRIA
>>
>> https://neuro.univie.ac.at/
>> https://www.imp.ac.at/groups/manuel-zimmer/
>> Part of Vienna BioCenter
>> www.viennabiocenter.org
>> ____________________________________________
>>
>>
>> _______________________________________________
>> micro-manager-general mailing list
>> micro-mana...@lists.sourceforge.net
Lukas Hille
with the demo cameras everything works fine (which is already a good thing).
I will investigate more on that on Monday.
Have a nice weekend!
best
Lukas
On 2020-03-12 23:00, Nico Stuurman wrote:
> Hi Lukas,
>>
>> Apart from the "speed / buffer overflow" issue, can you also verify
>> that one of the cameras (camera 0) stops acquiring after some images?
>> (~60,000 with Trigger:software and varying with Trigger:external)
>
> May be a while before I can get behind a microscope again;( That
> number sounds a bit like an overflow (awfully close to the maximum of
> an unsigned short: 65535), and checking with the democamera (possibly
> using the fastimage=1 option to avoid generating new images all the
> time) would be very helpful to localize the problem.
>
>> This applies to all adapter versions.
>> I am fine with the old driver, but dealing with breaks every some
>> thousand images is a pain.
>>
>> We have some "professional" software too, but they do even worse (i
>> am super happy with MM).
>
> Great to hear! Spending so much time supporting MM that now and then
> hearing from people that it is actually useful helps me stay motivated
> to continue doing so;)
>
>> I have to check if the "dual camera stop" on channel 0 is also a
>> Andor issue, i didn't try long lasting acquisitions with other cameras.
>
> If possible, check with the democamera, although external triggers
> are not an option there, and I do not know how well that code has been
> exercised with multiple cameras.
>
>
> Best,
>
>
>
> Nico