Colorization of miniscopeV4 video

[Naoki Tomita]

hello

I want to change the color sensor of "miniscopeV4.40" to "NOIP1SF0480A (PYTHON480 Bayer Color)" and colorize the video, referring to the method described in the following forum.

(https://groups.google.com/g/miniscope/c/RvrsUiN3qfc/m/08-phBanBgAJ)

I have checked the source code of the DAQ firmware.

When converting the data from miniscope to "YUY2" video-format  with "FX3" to send it in "UVC" standard, it looked like it was converting to grayscale by using the image data (8bit) from miniscope as the Y value and setting the U and V values to "0x80" ( 8bit camera data + 8bit 0x80 = 16bit data).

This appeared to be done regardless of the status of the "isColor" key.

When using PYTHON480 (Bayer Color), does the DAQ firmware support outputting color video?

Also, do you have any information about this?

Thanks in advance.

[Daniel Aharoni]

Hi,

The color Python480 image sensor will output pixel data identically to the monochrome version.  The difference is only that the color version has a Bayer filter pattern in front of the pixels. All you need to do is convert the grayscale recorded images into a color images by applying a Bayer filter transformation. This can also be done directly in real-time but we haven't written that functionality yet to our software. Our old software can do this though and you can check out the source code here, https://github.com/daharoni/Miniscope_DAQ_Software. I don't suggest using our old software but it will show you how it can be done on and offline.

[Naoki Tomita]

Dear Daniel,

Thank you so much for your kind concern and valuable information.

As you suggested, I tried the old software for the Bayer filter transformation. 

(https://github.com/daharoni/Miniscope_DAQ_Software)

However, I could not use it to process images from miniscopeV4.

Therefore, I used the new one by adding the old source code for the transformation part. 

(https://github.com/Aharoni-Lab/Miniscope-DAQ-

QT-Software)

I explained the details of how I added the code at the end of the email.

Next, I tested this modified code for images from MiniscopeV4 (with the original monochrome sensor). By doing so, I expected that the pseudo-color on the monochrome image. Attached are the resulted images from the original and the modified codes. It seems that black lines (enlarged in the figures) show some trace of colorization. 

Would you please let me know what you think whether it is working or not? Alternatively, would it be possible to send some original images from the color CMOS so that I could test them using the modified code, please? I am asking you this because it seems to cost more than 10K$ to test-order PCB for the color CMOS sensor via OpenEphys, so I would like to be sure before ordering. 

Thank you so much again for your great help! 

I am looking forward to hearing from you.

Details of the code modification:

The test code changes lines 247 to 253 of "videostreamocv.cpp".

The test code I added

    if (m_isColor) {

        QString str2 = "miniscopeDeviceName";

        if (m_deviceName.compare(str2, Qt::CaseInsensitive) == 0) {

            if (true) {

                cv::cvtColor(frame, frame, cv::COLOR_BGR2GRAY);

                cv::cvtColor(frame, frame, cv::COLOR_BayerRG2BGR);

            }else {

            //

            }

            frame.copyTo(frameBuffer[idx%frameBufferSize]);

        }else {

            frame.copyTo(frameBuffer[idx%frameBufferSize]);

        }

    }else {

        // frame = cv::repeat(frame,4,4);

        cv::cvtColor(frame, frameBuffer[idx%frameBufferSize], cv::COLOR_BGR2GRAY);

    }

2021年9月23日木曜日 7:44:53 UTC+9 dbah...@gmail.com:

[Daniel Aharoni]

Thanks for the detailed update. Yes this looks pretty much correct to me. Unfortunately I don't currently have any Bayer color image sensor data on my computer. If we collect any soon I will let you know.

As for getting color CMOS image sensor PCBs made, if you have experience reworking PCBs you could swap out the PYTHON480 monochrome image sensor yourself for a color version. Nothing else should be needed.

[Naoki Tomita]

Dear Daniel.

Thank you very much for your kind attention and valuable advice.

I will try to colorize the images based on the advice you gave me.

Please let me know if you have any new information about colorizing the miniscope in the future.

Thank you once again.

2021年10月7日木曜日 6:22:37 UTC+9 dbah...@gmail.com: