ValueError: inconsistent shapes after using MultiLabelBinarizer

Question

I'm trying to create a Performance Evaluation Results for my CRF model, which depicts what part of speech does the word belong to. I've created a function to transform the data in a more 'datasetish' format. This function returns the data as two lists, one of Dicts of features and the other with the labels.

def transform_to_dataset(tagged_sentences):
    X, y = [], []
    for sentence, tags in tagged_sentences:
        sent_word_features, sent_tags = [], []
        for index in range(len(sentence)):
            sent_word_features.append(extract_features(sentence, index)),
            sent_tags.append(tags[index])
        X.append(sent_word_features)
        y.append(sent_tags)
    return X, y

then I divide the set BEFORE encoding to have full sentences in training/testing sets.

penn_train_size = int(0.8*len(penn_treebank))
penn_training = penn_treebank[:penn_train_size]
penn_testing = penn_treebank[penn_train_size:]
X_penn_train, y_penn_train = transform_to_dataset(penn_training)
X_penn_test, y_penn_test = transform_to_dataset(penn_testing)

and then I load the model for it to train and test my data

penn_crf = CRF(
    algorithm='lbfgs',
    c1=0.01,
    c2=0.1,
    max_iterations=100,
    all_possible_transitions=True
)
#The fit method is the default name used by Machine Learning algorithms to start training.
print("Started training on Penn Treebank corpus!")
penn_crf.fit(X_penn_train, y_penn_train)
print("Finished training on Penn Treebank corpus!")

and then I test it with

y_penn_pred=penn_crf.predict(X_penn_test)

But when I try

from sklearn.metrics import accuracy_score

print("Accuracy: ", accuracy_score(y_penn_test, y_penn_pred))

It gives off an error:

ValueError: You appear to be using a legacy multi-label data representation. Sequence of sequences are no longer supported; use a binary array or sparse matrix instead - the MultiLabelBinarizer transformer can convert to this format.

But when I try to use MultiLabelBinarizer;

from sklearn.preprocessing import MultiLabelBinarizer

bin_y_penn_test = MultiLabelBinarizer().fit_transform(y_penn_test)
bin_y_penn_pred = MultiLabelBinarizer().fit_transform(y_penn_pred)

It gives me an error of:

ValueError: inconsistent shapes

Here's the full traceback

--------------------------------------------------------------------------- ValueError Traceback (most recent call last) /tmp/ipykernel_5694/856179584.py in 1 from sklearn.metrics import accuracy_score 2 ----> 3 print("Accuracy: ", accuracy_score(bin_y_penn_test, bin_y_penn_pred))

~/.local/lib/python3.8/site-packages/sklearn/utils/validation.py in inner_f(*args, **kwargs) 61 extra_args = len(args) - len(all_args) 62 if extra_args <= 0: ---> 63 return f(*args, **kwargs) 64 65 # extra_args > 0

~/.local/lib/python3.8/site-packages/sklearn/metrics/_classification.py in accuracy_score(y_true, y_pred, normalize, sample_weight) 203 check_consistent_length(y_true, y_pred, sample_weight) 204 if y_type.startswith('multilabel'): --> 205 differing_labels = count_nonzero(y_true - y_pred, axis=1) 206 score = differing_labels == 0 207 else:

~/.local/lib/python3.8/site-packages/scipy/sparse/base.py in sub(self, other) 431 elif isspmatrix(other): 432 if other.shape != self.shape: --> 433 raise ValueError("inconsistent shapes") 434 return self._sub_sparse(other) 435 elif isdense(other):

ValueError: inconsistent shapes

What should I do so that I could produce the confusion matrix of my model?

Answer 1

Try doing

penn_train_size = int(0.7*len(penn_treebank))

Then check if the shape is still inconsistent

bin_y_penn_test.shape
bin_y_penn_pred.shape
if bin_y_penn_test.shape == bin_y_penn_pred.shape:
    print('Consistent Shape')
else:
    print('Inconsistent Shape')

If it gives off consistent shape, do

from sklearn.metrics import multilabel_confusion_matrix
multilabel_confusion_matrix(bin_y_penn_test, bin_y_penn_pred)

However if it still isn't consistent, try to tinker on your data.