Misaligned "striping" on V4 images

[William Mau]

Hi all, 

I've noticed that my V4 scope images will sometimes become misaligned and look like this:

https://drive.google.com/file/d/1HuMSbopTgu9gFG7b_tbjbvp0hJfdKH5Y/view?usp=sharing

It seems to occur later in a session more often as opposed to earlier (due to overheating?) and most commonly when connected to a commutator. It's hard to reproduce so I'm not sure what the cause is but has anyone else observed this? Is there a way to prevent this from happening? Also since it seems like the data is intact, is there a way to recover these images? 

Thanks in advance!

Will

[Simon Fisher]

Hi William,

Not sure if you've tried it, but Daniel published some python code that can help to remove these striping artefacts: https://groups.google.com/g/miniscope/c/QTcACTTiD-E/m/E1eXOrF-AQAJ

I'm guessing this is an issue relating to the hardware fix in later releases of the V4 scopes.

By the way, what commutator do you use?

--
You received this message because you are subscribed to the Google Groups "Miniscope" group.
To unsubscribe from this group and stop receiving emails from it, send an email to miniscope+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/miniscope/03be269b-57d3-4fb4-bc52-8761d47abedfn%40googlegroups.com.

[William Mau]

Hi Simon, 

Thanks, but those striping artifacts are a separate issue that have indeed been resolved in the latest releases. My issue is intermittent and cannot be corrected with a simple filter.

We use the Neurotek commutator that they've recently developed. It works well mostly, but we've had this issue come up on the Dragonfly commutator as well, so I think it's a signal line stability issue that gets particularly exposed by commutators in general. 

Will

[Zeeshan]

Hi William,

We've been experiencing this issue as well. It was more frequent with the older v4 PCBs, but can still happen time to time on the newer ones. I've correlated it with the ufl connection moving around too much during behaviour, but overheating might also be a factor. The best fix we've has is to use some kind of silicone substance like Kwik Sil just above the ufl connection (but not touching it) to hold the wire in place better, so it doesn't move or rotate during behaviour. On the rare occasion it still happens (usually once every week or two, given daily recordings) we are trying to figure out a preprocessing script that can detect these bad frames and remove them before running the videos though the calcium analysis pipeline. If the bands occur at the end of the recording, we just trim off the last few videos. I'm not sure if there's any way to recover data from these frames as they look pretty messy. 

Ideally, it would be nice if when this problem occurs, the software simply skips the frames rather than record these horizontal bands. I'm not sure if that's something that can be integrated into the software somehow, @Daniel?

Best, 

Zeeshan 

To view this discussion on the web visit https://groups.google.com/d/msgid/miniscope/CAGaVC%2B08x5spwQKkjQq5AY_omFHU5GhErsA%2BQdpZEBO8qJXjWQ%40mail.gmail.com.

[Daniel Aharoni]

Hi Will,

We haven't been able to fully track down this issue yet as it is super inconsistent for us and very hard to reproduce. The misalignment you see is due to one of two USB buffers not getting fully flushed out which causes a shift in few rows of pixels within this buffer. This technically shouldn't really be possible as these buffers are forced to be reset at the end of each frame acquisition but something funky must happen which causes these buffer resets to get missed. I am not sure if this occurs within the DAQ itself or within the UVC driver buffers on the computer. When I have time I will keep trying to track down the source of this issue but as Zeeshan said, it very likely comes from an unstable U.FL connection on the V4 Miniscope. Adding some silicone or epoxy between the coax cable and PCB can help stability this. Your U.FL connector might also be loose, in which case you should replace it. Finally, you could try removing the U.FL connector on the PCB and directly soldering the coax cable to the PCB. Definitely a commutator can also cause this issue as the slip ring connections inside the commutator can also add some intermittent instability. 

Some good news though: No data is actually lost and it should be pretty easy to detect and correct this misalignment when it happens. When we have time we will write a script that can do this offline but in the meantime I would be happy to give some guidance on how to fix this offline. Let me know if you are interested.

[Zeeshan]

Hi Daniel,

Thanks so much for this reply! We've got at least a couple dozen recordings with these bands and were thinking we had no choice but to toss them/delete chunks of frames. Any guidance on how to fix these frames with a script would be most appreciated. 

Thanks! 

Zeeshan 

To view this discussion on the web visit https://groups.google.com/d/msgid/miniscope/bae4d034-112f-4945-a772-e33ab84084ccn%40googlegroups.com.

[William Mau]

Hi Daniel and Zeeshan, 

Thank you for the tip to secure the U.FL connector. I'll definitely try this! 

I already have some code that can detect these frames, but I don't know how to approach fixing them. The jupyter notebook for detecting the frames can be found here: https://github.com/wmau/V4_stripe_cleaner/blob/main/stripe_cleaner.ipynb. Currently it replaces the misaligned frame with the most recently acquired good frame, so it should be straightforward to substitute the replacement_frame variable with the fixed version of the misaligned frame. Daniel, can you advise what we could do to fix the misalignment in code? 

Will 

To view this discussion on the web visit https://groups.google.com/d/msgid/miniscope/CAE%2BY4UVSyb6cRH65tU2nd3Jzn6p-gKuLvtDSQPNxaoRuk8inaw%40mail.gmail.com.

[Johannes Hahn]

Hi Daniel,

I just had this happen as well for the first time today. It would be great if you could share the info on how to fix this.

[Daniel Aharoni]

Hi Everyone,

I'll try to block out some time to write up a script to detect and correct recorded frames that show this issue. A few tips for those of you that want to tackle this yourselves:

• Pixel data is stored in the DAQ in 2 interleaved buffers. Once a buffer is full, it is sent over USB to the Miniscope DAQ software using a USB Bulk Endpoint (this isn't really important here). Every buffer sent over USB should be completely full of pixel data except for the final buffer of a frame.
• The buffer size of these buffers is 1024 bytes * 16 = 16384bytes. A single pixel being sent over USB takes up 2 bytes so a full buffer holds 8192 pixels.
• When you look at a recorded frame that shows this issue you will see that the misalignment only occurs for pixels in one of these two buffers. In other words, you will have 8192 pixels in their correct location but the next 8192 pixels will be offset by some amount. This is because one of the two buffers somehow does not get fully cleared at the end of a frame and basically holds a bit of data from the previous time the buffer was filled. Another way of thinking about this is one of the two buffers gets data written to it at a location starting at some point above the 0 location of the buffer.
• So for the buffer that shows this misalignment you want to detect how many of the initial pixels in this buffer are from the last time the buffer was filled and then shift all the pixels from this buffer over by that number of pixels.

[William Mau]

Thanks for posting these tips. I have a follow up question. If I black out every 8192 pixels starting from [0, 0], I still see a few pixels that still seem to be misplaced (see red arrows in attached image). Are these pixels the "overflow" from the bad buffer, i.e. pixels displaced by the data that failed to be flushed? Or does the buffer not actually start at [0,0]? Or is there something I'm not understanding on how the pixels should be shifted? 

You received this message because you are subscribed to a topic in the Google Groups "Miniscope" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/miniscope/pr7PMiDbK68/unsubscribe.
To unsubscribe from this group and all its topics, send an email to miniscope+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/miniscope/5afa47e1-158d-406c-af9d-761bace9b697n%40googlegroups.com.

[Daniel Aharoni]

Oh whoops you are right. The UVC data protocol we use uses the first set of bytes in each buffer as a header and not as pixel data. This means the number of pixels sent in each buffer is not 8192 but rather 8180 since the header uses the first 12 bytes. 

[William Mau]

I think I'm still missing something. If I scrub every 8180 pixels starting from [0,12], there are still some misplaced pixels from the supposedly functional buffer. 

[Daniel Aharoni]

Ok. I had to go through some of the Miniscope DAQ firmware code to double check buffer and header/footer sizes. Here is what should be correct:

• Buffer size is 16384 bytes.
• The buffer has a 12 byte header along with a 4 byte empty footer (I was missing the footer in my previous post)
• So each buffer should have 8184 pixels in it. My math in the previous post was incorrect (sorry about that).

I'll try to do the same sort of thing you are doing this weekend on the original video you posted to see if I am missing anything else.

[William Mau]

If I zoom into the bottom of the frame (attached), there are some bright pixels that shouldn't be there, so I don't think the pixel interval between buffers is contributing to this (otherwise, these pixels would be more prevalent at later indices, and absent in the first 0:8184ish interval). I'm wondering whether these pixels are overflow from the data that failed to be cleared from the bad buffer? Or does the buffer not start at index 0 on the frame? 

To view this discussion on the web visit https://groups.google.com/d/msgid/miniscope/889cc0d9-62b5-4417-82b5-4bbdb7aff242n%40googlegroups.com.

[Daniel Aharoni]

Hey Will,

I started to work on this today and this is what I came up with using a similar approach to yours. I boosted up the brightness of the frame to make it more obvious if we were missing any pixels by coloring every 8184 pixels red but it looks good to me. I also zoomed into the transitions between red and gray and do not see any issues.

Here is also the Python code used for generating this:

frame = frame/255 * 4

bufferSize = 8184

for r in range(frame.shape[0]):

    for c in range(frame.shape[1]):

        pixNum = (r) * frame.shape[1] + c

        bufNum = int(pixNum/bufferSize)

        if ((bufNum % 2) == 0):

            # frame[r,c,0] = 255

            frame[r,c,1] = frame[r,c,1]*0.7

            frame[r,c,2] = frame[r,c,2]*0.7

plt.figure()

plt.imshow(frame)

I probably won't get around to doing anything more on this till the end of the week but let me know if you figure out how our approaches differ.

[William Mau]

Thanks for posting that Daniel, I was able to replicate the above so that is reassuring. Turns out we are doing essentially the same thing. The only difference is that I flatten the frame, select 8184 pixels, skip the next 8184, and finally reshape it: 

buffer_size = 8184
intervals = np.arange(0, (608*608), buffer_size, dtype=int)
frame = np.asarray(original[:,:, 0], dtype=float).flatten()
for i in intervals[::2]:
    frame[i:i+buffer_size ] = np.nan

frame = np.reshape(frame, (608, 608))

The reason why I was having trouble earlier was because the previous examples I posted were from another clip where neither of our codes appears to work:

https://drive.google.com/file/d/1ltTIjrp-Zxx1wiTwE_mJFcX3iJZIzn43/view?usp=sharing

I'm curious what's different about this clip? I recall this had only a few frames displaying this issue (frames 195-201), and it fixed itself spontaneously during the recording (as opposed to most of the time, where it's a long stretch of misaligned frames lasting seconds, and only gets fixed if I deliberately try to noisify the recording by yanking on the commutator connection). 

I'll try to see how often the videos defy the 8184 pixel rule. But now that we can identify the pixel locations that need shifting, I'm confident we can shift those pixels into their rightful place so I'll work on writing something that will do that. Before I dive in, Daniel, do you happen to have a guess on how many pixels I should be shifting (8184?) or whether you think this value is consistent from incident to incident? 

Will

--
You received this message because you are subscribed to the Google Groups "Miniscope" group.
To unsubscribe from this group and stop receiving emails from it, send an email to miniscope+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/miniscope/9a5a105b-a7ae-430c-80a3-1d3e76247b9an%40googlegroups.com.

[Daniel Aharoni]

Hi  Will,

So I think I might have figured it out (at least for some cases). From the original video you posted, it appears that the misalignment is due to the first buffer being from the previous frame. To correct the misalignment you just need to shift all even buffers down one while not touching the odd buffers. In other words, for every other 8184 pixels, shift those pixels back by 2 * 8184. I have attached an example image of one corrected frame along with the code.

[William Mau]

Thanks Daniel, that was really helpful. I took your code and integrated into my jupyter notebook for anyone to use: https://github.com/wmau/V4_stripe_cleaner/blob/main/stripe_fixer.ipynb. This should be able to detect and correct some cases of this buffer issue, but not all -- sometimes, the pixels shift by an amount that is not 2 buffer sizes. Also, it looks like the shift amounts are different for each frame.

Two examples: 

https://drive.google.com/file/d/1GpaZAfTKm6BiESi4uvty0ngu9Hj1FNtp/view?usp=sharing

https://drive.google.com/file/d/1ltTIjrp-Zxx1wiTwE_mJFcX3iJZIzn43/view?usp=sharing

It looks like the only way to get the correct shift amount for each frame is to iteratively try every value, unless Daniel, you can think of a more elegant solution? Integer multiples of 8184 also don't work. 

To view this discussion on the web visit https://groups.google.com/d/msgid/miniscope/48a3698e-149d-4c53-b530-d3c66720b84an%40googlegroups.com.

[Daniel Aharoni]

Thanks for these additional examples Will. I will look into them to see if I can track down what is happening.

[William Mau]

Thanks Daniel. I managed to find one particular shift (8515 pixels) that produced this image in the 37.avi file (frame 195): 

It looks like the good buffer got duplicated in the bad buffer and the bad buffer's data might be lost. I'm not sure if these frames are fixable anymore if that's the case, but maybe this will help you get at the source of the issue. 

You received this message because you are subscribed to a topic in the Google Groups "Miniscope" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/miniscope/pr7PMiDbK68/unsubscribe.
To unsubscribe from this group and all its topics, send an email to miniscope+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/miniscope/15217d0d-3b15-4c75-9477-d49dc356bab0n%40googlegroups.com.

[Phil Dong]

Hey everyone,

I've been trying to solve this "moving" type of stripes and here are some findings I got:

1. The buffer size scales with the frame/ROI size. For me I have a 600x600 pixels frame size and my buffer size is 8076 == 8184 / 608 * 600.

2. The buffer size may not be consistent. I have attached a snapshot of buffer size that's working for a particular frame I have, and it seems the buffer size tend to fluctuate +/- 4 pixels, although on average it's still 8076 pixels.

3. The amount of shift of the "current" buffer is not consistent. Although it seems the shift increase monotonically across the frame and the amount of increase seem to be somewhat consistent. I'd need more testing on this.

4. There is an additional gap/shift between the current buffer and the previous buffer, and this gap is not consistent, although it seem to be correlate with the amount of increase in shifts of "current" buffer across the frame. I'd need more testing on this.

So basically what I've done is to get a reference frame (e.g., the one right before the stripping frame), calculate a difference between the ref. frame and bad frame, use the difference to figure out the buffer sizes, and find out shifts for each buffer by cross-correlating the current buffer with corresponding portion of the reference frame.

I've attached some plot of a single corrected frame, as well as some functions to do this. It works fairly well, except there still seem to be some sort of stripping that needs further investigation.

Hopefully this would be helpful in solving this problem!

[Zeeshan]

Thanks so much for sharing this Phil! Indeed, I am still occasionally getting these stripes and haven't figured out how to fix them. If anyone else has an update on how to deal with this post-processing, it would be greatly appreciated!

Zeeshan 

--

You received this message because you are subscribed to the Google Groups "Miniscope" group.
To unsubscribe from this group and stop receiving emails from it, send an email to miniscope+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/miniscope/4cf6fc0d-8aaa-4d96-801b-0a97e67840b9n%40googlegroups.com.

[Phil Dong]

somehow I couldn't get plot to attach last time. trying this png here

[thomas.p...@gmail.com]

Hi all,

I would like to ask if there has been made some progress on this issue when using a commutator?

When I connect the miniscope directly to the daq box everything is fine but everytime I let it run through the commutator it sporadically creates these lines. Is there a way to fix this on the hardware side?

Thanks,

Thomas

[Daniel Aharoni]

We haven't been able to track down a fix for this on the hardware side. It is often times super hard to get to happen in a reproducibly way so hard to track down reliable solutions. I think some commutators cause this to happen much more often than others... but that likely isn't that helpful to you.

If we ever come across a solution we will post it here.

[Serena Wu]

Has anyone made any more progress on this? I tried Will's script to identify and fix frames that have this issue but it wasn't identifying all of the problematic frames. Does anyone have any idea why? I think it is because the striping pattern shows up in a few different ways (see photos below). I've also encountered some problems down the line in Minian with deleting multiple frames at different time points across the video, so would appreciate it if anyone can help me fix these frames!

[J Quinn Lee]

Hi everyone,

This is still an issue in our lab and we really need a solution. Has anyone officially solved this with some example code in Python? Neither Daniel or Phil's scripts seem to be generalized solutions based on the examples I have been working with. I think this is a big issue that many folks still need help addressing. Any help would be very appreciated!

Many thanks,

Quinn