VICReg:(Self-supervised Learning) Tutorial and Lightweight PyTorch Implementation (Yann LeCun, Chief AI Scientist at Meta)

[RAO MUNZIR]

Introduction 

Self-supervised representation learning has made significant progress over the last years, almost reaching the performance of supervised baselines on many downstream tasks ... VICReg is the latest in a progression of self-supervised methods for image representation learning

Tutorial/Implementation https://generallyintelligent.ai/open-source/2022-04-21-vicreg/

Authors of the paper : Adrien Bardes, Jean Ponce, and Professor Yann LeCun (Chief AI Scientist at Meta) 

https://arxiv.org/pdf/2105.04906.pdf

Self-Supervised Learning Series

Self-Supervised Learning Series

Background

;;;;;;;;;;;;

The key insight underpinning these new methods is simple: input images that are similar according to a human should be similar according to the model. By augmenting an image in some semantics-preserving way (meaning the pixel values are not necessarily the same, but a human would still register them as being versions of the same image) we can generate pairs of images that should be encoded as similar vectors by the model.

;;;;;

VICREG: INTUITION 

We introduce VICReg (Variance-Invariance-Covariance Regularization), a self-supervised method for training joint embedding architectures based on the principle of preserving the information content of the embeddings. 

The basic idea is to use a loss function with three terms: 

• Invariance: the mean square distance between the embedding vectors. 

• Variance: a hinge loss to maintain the standard deviation (over a batch) of each variable of the embedding above a given threshold. This term forces the embedding vectors of samples within a batch to be different. 

• Covariance: a term that attracts the covariances (over a batch) between every pair of (centered) embedding variables towards zero. This term decorrelates the variables of each embedding and prevents an informational collapse in which the variables would vary together or be highly correlated.

Learn More

If you found above tutorial interesting, you might enjoy reading the original paper or playing with our PyTorch implementation (which is also running in this simple Colab).

[RAO MUNZIR]

Respected Sirs/Admins , Assalamu Alaikum (may peace be upon you) 

(JazakALLAH for your contribution/time)

The group looks more like a job portal than a group with its main purpose written below (at the end). Job emails seem to easily get through ? May i know ,doesn't , at least , the current email (don't know about other people's similar emails not getting through) titled "VICReg:(Self-supervised Learning) Tutorial and Lightweight PyTorch Implementation (Yann LeCun, Chief AI Scientist at Meta) " pass the following criteria ?  Kindly have a look at all the recent titles https://groups.google.com/g/pakgrid .   Thanks 

    

 " Pakgrid is a forum to discuss & disseminate information about artificial intelligence, cloud & fog computing, distributed systems, HPC and 5G/6G networks. 

The aim is to develop a collaborative environment and an excellent research infrastructure through knowledge sharing and discussions." 

  

[RAO MUNZIR]

2304.12210.pdf (arxiv.org) 

A cookbook of Self Supervised Learning (24 April 2023)

by authors from  

Meta AI, FAIR **New York University †University of Maryland +University of California, Davis ‡Universite de Montreal, Mila §Univ Gustave Eiffel, CNRS, LIGM ?Univ. Rennes, Inria, CNRS, IRISA italicEqual contributions, randomized ordering

--
Please join Pakgrid by visiting https://groups.google.com/forum/#!forum/pakgrid
 
You can post your messages by sending an email at pak...@googlegroups.com
---
You received this message because you are subscribed to the Google Groups "Pakgrid" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pakgrid+u...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/pakgrid/384834523.862674.1650840542621%40mail.yahoo.com.