Contrastive Learning

[Aly Kotb]

Hi All,

I am know using contrastive learning with HCP data, so is it normal that a contrastive loss converges to negative values, or should be something wrong? I hope anyone experienced with deep learning can answer me.

Ali.

[Yubo Wang]

Hi Aly,

Would you like to try some a small subset of the data with a very small learning rate first to validate the model/training setting?

Which kind of contrastive loss are you using? It’s hard to say without much details.

Yubo

[Aly Kotb]

I tired it first with the MNIST graph benchmark dataset to make sure it is converging properly, before going to the HCP data. With MNIST it is converging but reaching negative values. The formula I am using is attached, it is the one used in simCLR paper, the positive anchor is the original graph and the positive pair is its augmentation.

Ali.

[Wang, Yubo]

You may want to check if the similarity terms give the value as expected. The formula doesn’t seem to have a negative value. The similarity term within the group ( or within class depends on how the labels are defined ) should be higher than the rest.

Yubo

获取 Outlook for iOS

---

发件人: Aly Kotb <alyko...@gmail.com>
发送时间: Monday, December 1, 2025 2:00:47 PM
收件人: HCP-Users <hcp-...@humanconnectome.org>
抄送: Wang, Yubo <w.y...@wustl.edu>; Aly Kotb <alyko...@gmail.com>
主题: Re: Contrastive Learning

 

[Tim Coalson]

While I don't have the entire context for this, I will note that since the sum in the denominator excludes the "n prime = n" case that defines the numerator, it is theoretically possible for the fraction to exceed 1, and therefore for the log to flip sign.  The "n prime = n" case would presumably have to have a much larger similarity value than the other cases to overcome the sum over N-1 cases, though.

Tim

--
You received this message because you are subscribed to the Google Groups "HCP-Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hcp-users+...@humanconnectome.org.
To view this discussion visit https://groups.google.com/a/humanconnectome.org/d/msgid/hcp-users/IA0PR02MB9170761B39E970EF7053EFA991DBA%40IA0PR02MB9170.namprd02.prod.outlook.com.