Questions regarding adapters and stepwise pretraining

[Tong Su]

Hi everyone,

 

Hope this email finds you well. My name is Tong, one of the students working with Sarubi and Surangika for the Pivot Adapter Project. I have some questions regarding the code you have worked previously.

 

1. Sarubi has forwarded me the email from Pasindu about specifying a list of layers where we do not want the adapters to be applied. Thanks for answering that! My follow-up question would be, since leave_out: Optional[List[int]] uses a list of integers to specify layers, how do we know the exact layer number we want to keep/leave out? For example, if I only want to apply the adapter to the last layer, how to find the number that represents that layer?
2. Another question is regarding the stepwise pretraining code. Initially the stepwise pretraining is for houslby adapter only. I want to apply the pretraining to other adapters by changing the adapter parameter from “houslby” to other one (such as “mam”). When I tried to run the "Step 2: Copy encoder from step 1 si-en model, decoder from mbart50 model, freeze encoder, fine tune using en-ta dataset (+ adapter)" Code. (code here: https://github.com/pengxinnn/adapter-transformers/blob/PivotBasedStepwisePreTrainingUsingTransformers/examples/pytorch/translation/run_translation.py)

It gives me the error message “Cannot use {adapter_setup} with prefix tuning “ as seen in the attachment. May I know why this happens?

 

Thank you so much for your help!

 

Best regards,

Tong      

[Pasindu Udawatta]

Dear Tong,

You can obtain the number of encoder layers using the `num_hidden_layers` parameter in the model config. Assuming the model is saved in the variable `model` you can access the number of hidden layers using `model.config.num_hidden_layers`.

For example, MBart has 12 encoder layers To apply adapters only to the last layer you can exclude the first 11 layers by giving `leave_out` the list [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

Ref: https://huggingface.co/docs/transformers/model_doc/mbart#transformers.MBartConfig.encoder_layers

--

Pasindu Udawatta

(CIMA Dip MA)

Undergraduate

Department of Computer Science & Engineering

University of Moratuwa

+94 714 793 047