I trained a Transformer model for translation. The saved model achieves ~27 BLEU on my test set. When I reload the model in a new session the BLEU score dips to ~0.5. I managed to find out that the issue is probably the fact that I build my vocab anew everytime I start a session instead of loading it from memory, and the itos mapping is changing everytime (I randomly split loaded data into train,dev,test and build vocab from train).
That got me thinking: is there a correct way of loading a vocab from memory so that my inference results are consistent? Minding that I also need to continue training the model so the vocab needs to be part of the TabularDataset.