In deep learning, can the prediction speed increase as the batch size decreases?

Question

We are developing a prediction model using deepchem's GCNModel.

Model learning and performance verification proceeded without problems, but it was confirmed that a lot of time was spent on prediction.

We are trying to predict a total of 1 million data, and the parameters used are as follows.

model = GCNModel(n_tasks=1, mode='regression', number_atom_features=32, learning_rate=0.0001, dropout=0.2, batch_size=32, device=device, model_dir=model_path)

I changed the batch size to improve the performance, and it was confirmed that the time was faster when the value was decreased than when the value was increased.

All models had the same GPU memory usage.

From common sense I know, it is estimated that the larger the batch size, the faster it will be. But can you tell me why it works in reverse?

We would be grateful if you could also let us know how we can further improve the prediction time.

Answer 1

let's clarify some definitions first.

Epoch
Times that your model and learning algorithm will walk through your dataset.
(Complete passes)

BatchSize
The number of samples(every single row of your training data) before updating the internal model. in other words, the number of samples processed before the model is updated.

So Your batch size is something between 1 and your len(training_data)

Generally, more batch size gives more accuracy of training data.

Epoch ↑ Batch Size ↑ Accuracy ↑ Speed ↓

So the short answer to question is more batch size takes more memory and needs more process and obviously takes longer time to learn.

https://stats.stackexchange.com/questions/153531/what-is-batch-size-in-neural-network

Here is the link for more details.

Answer 2

There are two components regarding the speed:

• Your batch size and model size
• Your CPU/GPU power in spawning and processing batches

And two of them need to be balanced. For example, if your model finishes prediction of this batch, but the next batch is not yet spawned, you will notice a drop in GPU utilization for a brief moment. Sadly there is no inner metrics that directly tell you this balance - try using time.time() to benchmark your model's prediction as well as the dataloader speed.

However, I don't think that's worth the effort, so you can keep decreasing the batch size up to the point there is no improvement - that's where to stop.