Fine tuning a model with add_adapter==True resulted in training loss=0 , validation loss=nan

Question

I am trying to fine tune wav2vec model using transformers library. The usual setup works fine but once I change add_adapter parameter to true I am getting training loss 0, validation loss nan. Any idea why this can happen? nothing else is changed besides the add_adapter parameter.

from transformers import Wav2Vec2BertForCTC

model = Wav2Vec2BertForCTC.from_pretrained(
    "facebook/wav2vec2-large-960h",
    attention_dropout=0.0,
    hidden_dropout=0.0,
    feat_proj_dropout=0.0,
    mask_time_prob=0.0,
    layerdrop=0.0,
    ctc_loss_reduction="mean",
    **add_adapter=True,**
    pad_token_id=processor.tokenizer.pad_token_id,
    vocab_size=len(processor.tokenizer),