Very long training times in pyTorch compared to Gensim

Question

I have trained a word2vec model using the Brown corpus with gensim as follows:

model = gensim.models.Word2Vec(brown.sents(),min_count = 5,
                              vector_size = 30, window = 5, negative=5) 

This took just a couple of seconds to train.

Then I tried to build a word2vec model using pyTorch. The training takes more than 5 hours according to my estimation.

I was expecting a speed difference since Gensim's word2vec uses optimized C routines. However, I was not expecting this much difference.

Is it normal for the training to take this much time with pyTorch? For both cases I'm not using any GPU's. What should I expect the running time of PyTorch for this task on a standard 4-cpu Mac?

Answer 1

You haven't mentioned what PyTorch code you're running.

However, back when Gensim had both a plain, pure-Python implementation of its word2vec routines, and the alternate Cython- and BLAS- optimized path, it was typical for the plain-Python to run 80-120 times slower.

If the PyTorch code you're running is pure-Python, and further might be mainly instructional in its purpose – without even Python efficiency considered, or perhaps without any attempt to use multiple threads/processes – then a slowdown from "under a minute" to "several hours" would be plausible.