HCML Reading Group - Upcoming Session

[Diego Miguel]

Hello everyone!

As previously announced, we invite you to save the date for the next reading session:

    22nd January at 10 am CET: ELLIS-ELLIOT reading group on Human-Centric ML

    🔗 Link to join:  https://bit.ly/ellis-hcml-rg

We'd also like to remind everyone that we are actively looking for paper suggestions for future sessions, which can be submitted through this form: https://forms.gle/zDjtKyNyjnBu2tcH8. We really thank anyone who wants to take the initiative!

Since we still did not have any paper suggestions, I will be presenting a paper that I was reading recently for my own work. Hope you also find it interesting :)

    Title: Vision-Language Models Do Not Understand Negation

    Abstract: Many practical vision-language applications require models that understand negation, e.g., when using natural language to retrieve images which contain certain objects but not others. Despite advancements in vision-language models (VLMs) through large-scale training, their ability to comprehend negation remains underexplored. This study addresses the question: how well do current VLMs understand negation? We introduce NegBench, a new benchmark designed to evaluate negation understanding across 18 task variations and 79k examples spanning image, video, and medical datasets. The benchmark consists of two core tasks designed to evaluate negation understanding in diverse multimodal settings: Retrieval with Negation and Multiple Choice Questions with Negated Captions. Our evaluation reveals that modern VLMs struggle significantly with negation, often performing at chance level. To address these shortcomings, we explore a data-centric approach wherein we finetune CLIP models on large-scale synthetic datasets containing millions of negated captions. We show that this approach can result in a 10% increase in recall on negated queries and a 28% boost in accuracy on multiple-choice questions with negated captions. 

     Link: https://negbench.github.io/

You can find the reading group's website here and join the Google Group here.

Kind regards,

Diego Miguel Lozano

PhD Student

ELLIS Alicante