[Wav2Vec2] How to freeze the encoder in Wav2Vec2CTC audio_finetuning task?

[Igor Shalyminov]

Hi, the XLSR paper says that weights of the feature encoder are not updated at the fine-tuning time.

However, I don't see this setting anywhere in the code, and the closest thing in the config is freeze_finetune_updates: 10000 which I think is for the entire model (encoder + output projection).

What's the way to freeze the encoder only, and make it permanent (apart from altering the model's code directly)?

[alex...@gmail.com]

you set feature_grad_mult to 0 which disables updates to feature extractor:

https://github.com/pytorch/fairseq/blob/c71870f370455e6154c730e8822ea323b5f266f6/fairseq/models/wav2vec/wav2vec2.py#L597

[Igor Shalyminov]

Got it, thanks!