I am using PyTorch DataParallel to run my model on two GPU cards. So, in every batch, the input batch data is split into two and run separately on two cards. However, I have some special types of operations, that need to run at certain iterations only (say after 1000 iterations) and it is not clear to me how to run these.
- An operation that needs to be run only once and is not related to the input data in the batch. But because the same function will be called twice, once on each GPU card, I think this function will be called twice. It is not a problem except that it is a waste of time. How can I avoid this?
- I need to modify some parts of my model. Again this needs to be run only once and should happen exactly the same to the models on all the GPU cards. How can I do this?