Problem Importing and running pyLDAvis 3.4.1. (A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.)

Question

I am trying to run pyLDAvis.prepare() but gives me the following error

---------------------------------------------------------------------------
_RemoteTraceback                          Traceback (most recent call last)
_RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/joblib/externals/loky/process_executor.py", line 426, in _process_worker
    call_item = call_queue.get(block=True, timeout=timeout)
  File "/usr/lib/python3.10/multiprocessing/queues.py", line 122, in get
    return _ForkingPickler.loads(res)
ModuleNotFoundError: No module named 'pandas.core.indexes.numeric'
"""

The above exception was the direct cause of the following exception:

BrokenProcessPool                         Traceback (most recent call last)
<ipython-input-158-253cf86428cd> in <cell line: 9>()
      7 
      8 # Initializing pyldavis
----> 9 lda_panel = pyLDAvis.prepare(topic_term, doc_topics,doc_len,vocab,tf)
     10 
     11 # Displaying pyldavis

7 frames
/usr/local/lib/python3.10/dist-packages/joblib/parallel.py in _return_or_raise(self)
    752         try:
    753             if self.status == TASK_ERROR:
--> 754                 raise self._result
    755             return self._result
    756         finally:

BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.

I tried using dill to make files picklable, but that doesn't seem to make any difference.

Also when i do

pip install pyLdAvis  

The code that I am running is as follows: -

# Creating Parameters for pyldavis
topic_term = best_lda_model.components_ / best_lda_model.components_.sum(axis=1)[:, np.newaxis]
doc_topics= best_lda_model.transform(input_matrix)
doc_len = np.ravel(np.sum(input_matrix,axis=1))
vocab = dictionary.keys()
tf = dictionary.values()

# Initializing pyldavis
lda_panel = pyLDAvis.prepare(topic_term, doc_topics, doc_len, vocab, tf)

# Displaying pyldavis
lda_panel

Answer 1

I finally found the solution. All I had to do bypass the dependency resolution error while installing it. In google colab notebook every dependency requirement was satisfied except funcy, tzdata. So I did this

pip install funcy

pip install tzdata

pip install --no-deps pLDAvis

and this did the bypass of installing numpy and pandas which is already installed in colab.

This worked for me.