Testing and Validation (MarineVision Restoration)

[Ali Awad]

Hello,

We are confused about the dataset structure. In the incrustations, it says there are train, validation and testing sets. However, we could not see any testing set. It's mentioned 'Generate restored images and detection predictions on the RSUIGM test images' do you mean by this the 71 images in new-val-set folder?

Similarly, should the object detector be used to infer the same 71 images of the new-val-set folder?

Kind regards.

[Nikhil Akalwadi]

Hi Ali, 

Yes, the participants are to use the new-val-set (set of 71 images) for leaderboard rankings. 

The evaluation for restoration and detection shall be based on the same. 

On Dec 17, 2024, at 06:53, Ali Awad <aa...@mtu.edu> wrote:

Hello,

We are confused about the dataset structure. In the incrustations, it says there are train, validation and testing sets. However, we could not see any testing set. It's mentioned 'Generate restored images and detection predictions on the RSUIGM test images' do you mean by this the 71 images in new-val-set folder?

Similarly, should the object detector be used to infer the same 71 images of the new-val-set folder?

Kind regards.

--
Sie erhalten diese Nachricht, weil Sie in Google Groups E-Mails von der Gruppe "MaCVi Support" abonniert haben.
Wenn Sie sich von dieser Gruppe abmelden und keine E-Mails mehr von dieser Gruppe erhalten möchten, senden Sie eine E-Mail an macvi-suppor...@googlegroups.com.
Diese Diskussion finden Sie unter https://groups.google.com/d/msgid/macvi-support/a7ab41ab-5cd3-4574-9bfa-101330a86f8fn%40googlegroups.com.
Weitere Optionen finden Sie unter https://groups.google.com/d/optout.

[Ali Awad]

Thank you very much for your support. We submitted yesterday (#14160) with our enhanced images, but it shows extremely low CCF and UIQM values which we believe to be wrong.

Please advise,

Regards.

[Nikhil Akalwadi]

Hi Ali,

The codebase for computing UIQM and UCIQE is derived from the GitHub repository.

For CCF, we utilize the official implementation provided by the paper’s authors. We have modified it to provide normalized value ranges for ease of evaluation. We have thoroughly verified the logic of the evaluation scripts on multiple occasions. I assure you that the values you observe should be largely accurate. 

Feel free to let us know if there is anything we can assist you with. 

Happy Coding!

Diese Diskussion finden Sie unter https://groups.google.com/d/msgid/macvi-support/7345d446-e084-4c77-b631-8ffd52d762aan%40googlegroups.com.

Weitere Optionen finden Sie unter https://groups.google.com/d/optout.

Regards,
--
Nikhil Neelkanth Akalwadi
Researcher
(M) +91 87921 88443 

https://nikhilakalwadi.github.io

[Ali Awad]

We appreciate your support,

We verified our enhanced images using a Matlab implementation of the metrics that we have been using for years. The values we got for UIQM, UCIQE, CCF are as follows:

4.02, 0.62, and 23.52, respectively, Vs. 0.013, 0.595, and 0.000, respectively, on your test server.

We think there is a problem with the codes on the server side. Please check our previous work:

https://ieeexplore.ieee.org/abstract/document/10028983

https://arxiv.org/abs/2411.14626

and many other papers.

For some reason, the portal in now asking for an email verification, but no email is being sent. We have already checked the spam folder. Please provide support on this as well.

Kind regards.

[Nikhil Akalwadi]

Dear Ali, 

As previously mentioned, the codebase for the metrics we have been utilizing for UIQM and UCIQE adheres to the equations presented in the original paper, where the metrics were initially proposed.

For UIQM: https://ieeexplore.ieee.org/document/7305804, for UCIQE: https://ieeexplore.ieee.org/document/7300447

Additionally, it is noteworthy that the outcomes for UIQM and UCIQE metrics are anticipated to be within the range [0,1], akin to SSIM. In contrast, CCF can assume values across various ranges. For the purpose of simplicity and fair evaluation, we normalize the range (0-1) for CCF exclusively for the challenge.

Furthermore, we have recently revised the UIQM metric implementation based on the recommendation of one of the contestants. Kindly re-submit your submissions and observe if you obtain suitable values for the metric.

Please do not hesitate to contact me if you require any further assistance. 

Researcher, 

MVRC Team
(M) +91 87921 88443 

https://nikhilakalwadi.github.io

[Nikhil Akalwadi]

Dear Ali, 

Regarding the email verification process for the portal, I will consult with my team and provide an update as soon as possible. In the meantime, I believe you are able to submit new submissions. 

[Nikhil Akalwadi]

Dear Ali, 

We have two records in our database.

* User_id 733 with Email ID: aa...@mtu.edu

* User_id 736 with Email ID: aa...@mtu.com

I suspect the verification failure is for the account with user_id 736, which also likely has a potential typographical error in the email address. The domain name appears to be incorrect. Please update your email address, and you should be able to verify your account/email address. 

Program Committee, 

MarineVision Restoration Challenge, 

MaCVi, @WACV 2025