V4 Miniscope sensor image flickering and LED instability - expected behavior or bug?

[Matthias Klumpp]

Hello!

I am currently working on making the V4 Miniscope work with our active
Doric Lenses commutator. While working on this, I noticed severe image
flickering, and even LED flickering (LED dimming down, then reaching
its set brightness again, then breaking down again).

So, I removed all connectors and commutator-related things, but the
issue persists. I am quite sure that this issue has always existed
like this, but it feels like it got worse in recent months, to the
point of making any of my Miniscope recordings very hard to analyze.

I have attached two sample videos of the issue to this email.
The Miniscope I am using is one of the first-generation V4 Miniscopes
shipped by OEPS, which means that it does not yet have the VDD-PIX-FC
board soldered on to clean up the imaging sensor power supply. So, I
expect the horizontal stripes that are moving up and down the image,
and those are relatively easy to filter out.

What I do not expect however and what makes the data really hard to
use is the whole image changing in brightness, as you can see at the
edges of the `21-02-05-1327_scope_1.mkv`[1] recording, but even more
visibly in `21-02-05-1347_scope_1.mkv`[2]. This happens even when the LED
is turned off completely, so the issue is intrinsic to the sensor and
not related to any LED power problems.

Besides that though, when turning down the LED power to a certain
level, the LED itself starts to flicker as well.

I have replaced all cables, connectors, tried different USB cables,
different computers and even added a small circuit to clean up the USB
power supply, just in case there was noise on the USB power channels
interfering with the Miniscope data transmission. The latter seemed to
help briefly, but ultimately nothing made a lasting difference. The
issue also exists across all Miniscope DAQ software and operating
systems.
So, at this point I am quite sure that the issue is actually on the
Miniscope itself.
Furthermore, occasionally when I increase the LED brightness to its
maximum value too quickly, and record with the highest framerate, the
image breaks down completely and I get large individual parts of the
image flickering. This is rare behavior though, but could possibly be
related.

So, my questions are the following:
* Is the whole image changing in brightness normal for the V4? Does
this look similar for you when recording no neurons?
* If the whole-image-brighness-noise is not normal, is it fixed by
the VDD-PIX-FC change? (I will know by next week when the parts for it
arrive)
* Do you have any idea what is going on with the LED itself
flickering? It's not an issue with the power supply to the Miniscope
itself or its sensor, but it looks like the LED itself is on unstable
power, which given the circuitry of the Miniscope is very strange,
unless I have a faulty part somewhere.

I also tried a second V4 Miniscope from the same batch, and that
exhibits the exact same behavior as well. The noise also gets a bit
worse with the Miniscope heating up over time, although the effect of
that is not huge (it can be measured though).

I do not know if the LED issue and the sensor issue are separate from
each other or related problems - the latter doesn't seem unlikely, but
it's hard to test this.

I am glad for any information and help!
With kind regards,
Matthias Klumpp

Attachments:
[1]: At https://drive.google.com/file/d/1N9k-BD8nQb8HrxjpB-w-QH4LvsPIbYl0/view?usp=sharing
[2]: Attached to this mail
All recordings have been compressed with a lossy codec, which did not
impact the observed effect for the human eye though (but the filesize
is now 20x smaller ^^)

--
I welcome VSRE emails. See http://vsre.info/

[Daniel Aharoni]

Hi Matthias,

Thanks for all the details and sorry that you are dealing with an issue with the Miniscope platform.

This sounds possibly like a damaged electrowetting lens driver which is ending up drawing way too much current and messing up the voltage regulators on the PCB. Does the portion of the PCB with the excitation LED get really really hot even with the LED turned down very low? Are you able to adjust the electrowetting lens through software?

The issue you are experiencing isn't normal and the video stream from the Miniscope should not flicker. The VDD-PIX hot fix flex board will clean up the horizontal row noise but your scopes seem to be experiencing much more significant issues. 

Another possibility is that the LED has become damaged but this is likely very unlikely. Or the Python480 image sensor is damaged but again, this is very unlikely.

[Matthias Klumpp]

Hi Daniel!

Am Di., 9. Feb. 2021 um 06:57 Uhr schrieb Daniel Aharoni <dbah...@gmail.com>:
>
> Hi Matthias,
> Thanks for all the details and sorry that you are dealing with an issue with the Miniscope platform.

No worries, besides the flickering being a pain for any analysis, it
looks like the V4 Miniscope actually makes my experiment viable, so I
am very happy about that!

> This sounds possibly like a damaged electrowetting lens driver which is ending up drawing way too much current and messing up the voltage regulators on the PCB. Does the portion of the PCB with the excitation LED get really really hot even with the LED turned down very low? Are you able to adjust the electrowetting lens through software?

Yes to both questions! Actually the serializer part and the
electrowetting lens driver part both get so hot that I can barely
touch these parts of the Miniscope without burning my fingers. This
has been like this ever since I got the PCB, so I thought it was
normal and was making jokes that for the mouse the Miniscope is more
like a portable radiator than anything else...
I was a bit suspicious about this though, so I actually ran the
Miniscope for many hours, monitoring for temperature changes. The
temperature just caps off at some point and doesn't climb any further,
and the scope doesn't seem to be impacted by the heat at all (I'd need
to get to the lab for notes on this test run, I also did it a while
back).
This behavior is shown by both of my Miniscopes, which also both
exhibit the same fault though (the primary reason why I wasn't sure
whether this was expected behavior). The LED was always turned off.

With both Miniscopes, adjusting the EWL works absolutely fine though,
I can adjust focus through the whole range of values available.

> The issue you are experiencing isn't normal and the video stream from the Miniscope should not flicker. The VDD-PIX hot fix flex board will clean up the horizontal row noise but your scopes seem to be experiencing much more significant issues.
>
> Another possibility is that the LED has become damaged but this is likely very unlikely. Or the Python480 image sensor is damaged but again, this is very unlikely.

That's what I thought, about the VDD-PIX board... One interesting
thing after looking through some of my older data is that the behavior
I am seeing does - subjectively - appear to get worse. It has been
there from the start, but initially was very difficult to spot, while
now it is extremely obvious even without any analysis. Also, some
recordings don't exhibit this fault at all, but most of them do. When
trying to reproduce the effect now, it is observable 100% of the time.
So this kind of feels like whatever is broken may actually be getting
worse...

I could ask OEPS for replacement boards, but it's still extremely odd
that both Miniscopes are affected and I would like to rule out any
usage error or easy electronics fix on my side first.

Thank you very much for your reply & help!
Cheers,
Matthias

[Daniel Aharoni]

This very much sounds like a damaged electrowetting lens driver or supporting hardware. We have heard from a few people that this component has started to heat up, draw way too much current, and fail after the Miniscope was dropped. I would have assumed the electrowetting lens driver IC would be pretty robust to moderate impact but it seems like there are certain rare-ish cases that can cause damage. I think there are also rare cases where the EWL driver begins to act up without anything in particular causing the initial issue.

The LED/EWL driver PCB should really stay rather cool if functioning properly. The serializer/coax cable connector PCB will get rather warm even during normal operation but it shouldn't be too hot to touch. Once the EWL driver begins to malfunction, it will draw a ton of excess current, causing both PCBs to heat up much more than normal.

[Matthias Klumpp]

Am Do., 11. Feb. 2021 um 18:44 Uhr schrieb Daniel Aharoni <dbah...@gmail.com>:
>
> This very much sounds like a damaged electrowetting lens driver or supporting hardware. We have heard from a few people that this component has started to heat up, draw way too much current, and fail after the Miniscope was dropped. I would have assumed the electrowetting lens driver IC would be pretty robust to moderate impact but it seems like there are certain rare-ish cases that can cause damage. I think there are also rare cases where the EWL driver begins to act up without anything in particular causing the initial issue.
>
> The LED/EWL driver PCB should really stay rather cool if functioning properly. The serializer/coax cable connector PCB will get rather warm even during normal operation but it shouldn't be too hot to touch. Once the EWL driver begins to malfunction, it will draw a ton of excess current, causing both PCBs to heat up much more than normal.

Hmm... Do you think making an attempt to replace that chip would work
to fix this issue? It's a BGA chip and replacement would need to be
done on an already assembled Miniscope, which would make this
extremely challenging, but it appears like obtaining the chip in the
first place, or even just a datasheet is an issue.
Just to make sure, we are talking about the MAX14574EWL+ IC (U4) here, right?
I had no luck finding a datasheet for this chip, and only references
to a datasheet for the MAX14574 series. Also, no vendor appears to
have it in stock.
The reason for that seems to be: "The MAX14574 is now a custom part
only for Corning Varioptic. Device customer(s) must go directly to
Corning Varioptic for technical support." - But Corning Varioptic
doesn't have a datasheet for this chip, or an option to order it, so I
would need to contact them directly.
Is this the only way to obtain this chip? Do you think this has even a
chance of success? I'll also write to OEPS for their opinion, as this
issue existed on the boards since I first had them, and I am very sure
that I never dropped any of the Miniscopes (yet...).

Cheers,
Matthias

[Federico Sangiuliano]

Hello Matthias. Replacing the EWL driver is very challenging but doable. Unfortunately, as you said, the IC itself is only available for purchase from Corning but if you manage to get one I can write some recommendations on how to replace it.

Thank you

Federico

[Matthias Klumpp]

Am Do., 11. Feb. 2021 um 21:44 Uhr schrieb Federico Sangiuliano
<fsan...@gmail.com>:

>
> Hello Matthias. Replacing the EWL driver is very challenging but doable. Unfortunately, as you said, the IC itself is only available for purchase from Corning but if you manage to get one I can write some recommendations on how to replace it.

That would be very nice! I see what I can do about the chip.
Fortunately, I glued the side parts of the Miniscope circuit board
onto the plastic with some neutral-curing silicone, as I didn't trust
hot glue due to the extreme heat from the board and I didn't want to
use epoxy to make any repairs easier. I was very suspicious of the
extreme Miniscope heat even back then, looks like not tightly gluing
everything may pay off now :-)

[Matthias Klumpp]

Hi!

So, we replaced the EWL IC now (I feared it would be very difficult,
but it's actually okay using a hot air station and lots of kapton
tape). Unfortunately this did not fix the issue at all, the Miniscope
still shows the same issue as before (on the plus side, the EWL focus
is also still working, so nothing lost, nothing gained). Is there
anything else that we could try to investigate this issue?