How can I find and print unmatched/dissimilar words from the documents(dataset)?

50 Views Asked by At

I am trying to rewrite algorithm that basically takes a input text file and compares with different documents and results the similarities.

Now I want to print output of unmatched words and output a new textile with unmatched words.

From this code, "hello force" is the input and is checked against the raw_documents and prints out rank for matched document between 0-1(word "force" is matched with second document and ouput gives more rank to second document but "hello" is not in any raw_document i want to print unmatched word "hello" as not matched ), But what i want is to print unmatched input word that was not matched with any of the raw_document

import gensim
import nltk

from nltk.tokenize import word_tokenize

raw_documents = ["I'm taking the show on the road",
                 "My socks are a force multiplier.",
             "I am the barber who cuts everyone's hair who doesn't cut their own.",
             "Legend has it that the mind is a mad monkey.",
            "I make my own fun."]

gen_docs = [[w.lower() for w in word_tokenize(text)]
            for text in raw_documents]

dictionary = gensim.corpora.Dictionary(gen_docs)

corpus = [dictionary.doc2bow(gen_doc) for gen_doc in gen_docs]

tf_idf = gensim.models.TfidfModel(corpus)
s = 0
for i in corpus:
    s += len(i)
sims = gensim.similarities.Similarity('/usr/workdir/',tf_idf[corpus],
                                      num_features=len(dictionary))
query_doc = [w.lower() for w in word_tokenize("hello force")]

query_doc_bow = dictionary.doc2bow(query_doc)

query_doc_tf_idf = tf_idf[query_doc_bow]
result = sims[query_doc_tf_idf]
print result
0

There are 0 best solutions below