Quality Control Consensus

[Vidit Agarwal]

Hello CVAT Community,

Myself Vidit Agarwal, I am among the contributors selected for the Google Summer of Code 2024 program for CVAT. I am writing this to share about my project, Quality Control Consensus. 

The project aims to add functionality to annotate the same job multiple times and then aggregate these multiple annotations. This reduces the bias and mislabelling in the final annotated data. Here's the high level design of how we plan to implement this functionality, and also my proposal for this project.

The community bonding period is over, and the coding period began a week ago. As of now,

1. we have finalised the HLD and proposal 
2. are about to windup the feature implementation of creating consensus jobs based on the Consensus Configuration parameters passed while creating the task and updating the editable parameters among them after the task has been created. You can find the pull request for this feature here: https://github.com/cvat-ai/cvat/pull/7974. 

If you're interested in following the progress of this project, subscribing to this issue will keep you updated: https://github.com/cvat-ai/cvat/issues/7973.

This wouldn't have been possible without Nikita (CVAT admin), Maxim and Roman (project mentors), the CVAT team and the CVAT community.

Thanks,

Vidit Agarwal

[Vidit Agarwal]

Hello again, CVAT Community,

It has been a week since my since my last update.

1. We completed implementing features related to configuring consensus properties and creating consensus jobs. A detailed list of these functionalities can be found here.
2. Additionally, with this, we implemented the feature to merge consensus jobs into their parent job. The pull request can be found here. Please refer to the commits made on and after June 18th for changes specific to this feature.

Now, one can configure and create consensus jobs and merge their annotations in CVAT to improve the quality of annotations.

Currently, we are running on track with the planned project timeline mentioned in the proposal.

Every feature added to the existing code requires an understanding of the surrounding code, which can become very daunting when dealing with such a large codebase. Thanks to Maxim for making this less daunting.

Thanks,
Vidit Agarwal

[Vidit Agarwal]

Hello CVAT Community,

Phew, a lot has happened in the past two weeks,

1. While creating the task, you can only set the value of `consensus jobs per segment`; thus, the consensus configuration collapse component has been removed  
   
   1.  
2. A separate tag was added along the task name when viewing tasks with consensus jobs.
   1. 
3. There's a separate option to configure the rest of consensus configuration like `agreement score threshold`, `quorum`, `min overlap threshold` and the Merge Button
   1.           
4. The tab to view the analytics of the consensus process is also there, though non-functional, until merge reports are implemented.
5. Worked on improving the code added by incorporating various reviews received on the two PRs (Creating Consensus Jobs [PR] and Merge Consensus Jobs [PR])

Summarising, the user can now configure the consensus settings to control how the merging occurs for their task. Along with this, there are a few improvements in the code quality and UI cosmetics.

Thanks,
Vidit Agarwal

[Vidit Agarwal]

Hello CVAT Community,

The past two weeks included the GSoC mid-term evaluation period. I am glad to mention that I have passed the mid-term evaluation and look forward to continuing with the project and collaborating with the CVAT community.

In the past two weeks, I worked on:

1. The initial version of the consensus report is now available via an API request. These consensus reports are created when merging occurs. They consist of a summary of frame-wise consensus conflicts and a summary of consensus conflicts.
2. Improved various UI components:
   1. Colours and hover actions under the Consensus Configuration section of tasks.
   2. The consensus jobs per normal job parameter have been shifted under the Advanced Configuration block in task creation.
   3. Notifications have been replaced with a loading action when saving consensus settings.
3. Structured the code, handling the consensus part better by making it coherent with the rest of the components.
4. By default, quorum is set to half of the consensus jobs.
5. Did you know that a job with no annotations can also be an annotated job? This happens when the job contains frames with no objects with the given labels. Thus, I ignored consensus jobs that haven't been opened, i.e., with the state as NEW.
6. Additionally, I worked on resolving feedback provided in GitHub PR(s) other than the tasks mentioned above.

In summary, the user can now view the consensus report to analyze the consensus quality. Along with this, there are improvements in code quality and UI cosmetics.

Thanks,
Vidit Agarwal

[Vidit Agarwal]

Hello CVAT Community,

In the past two weeks, I worked on:

1. Creating Figma slides for how the Consensus Analytics page and related components will look like. Based on the feedback, a final UI deck will be formed, based on which I will create the consensus analytics page and update the existing UI accordingly
2. Added a feature to merge Skeleton-based annotation, which wasn't present in Datumaro's implementation
3. Improved matching algorithm by replacing Datumaro's annotation matching algorithm with the one in CVAT's quality reports
4. Added sigma parameter in Consensus Settings for OKS distance calculation

I am left with four weeks to complete this project. Time flies.

Making the consensus analytics page, adding test cases, documentation, product paper, and a GSoC blog are things to be done.

Thanks,

Vidit Agarwal

[Vidit Agarwal]

Hello CVAT Community,

In the past two weeks, I worked on:

1. Implemented the UI drafted in the Figma slides for the Consensus Analytics Page.
2. Wrote test cases for creating consensus jobs.
3. While writing test cases, I encountered various errors in the current implementation and fixed them, such as the task backup annotations in the consensus job not being saved.
4. Made the UI functional by adding various components in the cvat_ui and the Django models to ensure they work. This includes passing information about the consensus score at both the consensus job level and the assignee level.

I am left with two weeks to complete this project.

TODO: Test cases for consensus reports, documentation, product paper, and a GSoC blog.

Thanks,

Vidit Agarwal