Document classification using pretrained models like BERT

Question

I am looking for methods to classify documents. For ex. I have a bunch of documents with text and I want to label the document on whether it belongs to sports, food, politics etc. Can I use BERT (for documents with words > 500) for this or are there any other models that do this task efficiently?

Answer 1

BERT has a maximum sequence length of 512 tokens (note that this is usually much less than 500 words), so you cannot input a whole document to BERT at once. If you still want to use the model for this task, I would suggest that you

1. split up each document into chunks that are processable by BERT (e.g. 512 tokens or less)
2. classify all document chunks individually
3. classify the whole document according to the most frequently predicted label of the chunks, i.e. take a majority vote

In this case, the only modification you have to make is to add a fully connected layer on top of BERT.

This approach might be quite expensive, though. There is the alternative of representing the text documents as bag of word vectors and then train a classifier on the data. If you are not familiar with BOW, the Wikipedia entry to it is a good starting point. It can serve as a feature vector for all kinds of classifiers, I would suggest you try SVM or kNN.