I repeatedly get jupyter lab sessions that start to hang while spamming tracebacks to the terminal that end with:
zmq.error.ZMQError: Too many open files
An example complete traceback is:
[E 2024-03-17 17:40:36.843 ServerApp] Uncaught exception GET /api/kernels/059a770b-436f-40b5-b41b-95a1748ef7fd/channels?session_id=f60b1486-f926-4c96-924d-692a4e50d1e4 (172.16.206.48)
HTTPServerRequest(protocol='http', host='localhost:23456', method='GET', uri='/api/kernels/059a770b-436f-40b5-b41b-95a1748ef7fd/channels?session_id=f60b1486-f926-4c96-924d-692a4e50d1e4', version='HTTP/1.1', remote_ip='172.16.206.48')
Traceback (most recent call last):
File "/orange/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/tornado/websocket.py", line 944, in _accept_connection
await open_result
File "/orange/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/jupyter_server/services/kernels/websocket.py", line 77, in open
await self.connection.connect()
File "/orange/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/jupyter_server/services/kernels/connection/channels.py", line 363, in connect
self.create_stream()
File "/orange/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/jupyter_server/services/kernels/connection/channels.py", line 154, in create_stream
self.channels[channel] = stream = meth(identity=identity)
File "/orange/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/jupyter_client/ioloop/manager.py", line 25, in wrapped
socket = f(self, *args, **kwargs)
File "/orange/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/jupyter_client/connect.py", line 664, in connect_iopub
sock = self._create_connected_socket("iopub", identity=identity)
File "/orange/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/jupyter_client/connect.py", line 654, in _create_connected_socket
sock = self.context.socket(socket_type)
File "/orange/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/zmq/sugar/context.py", line 362, in socket
s: ST = socket_class( # set PYTHONTRACEMALLOC=2 to get the calling frame
File "/orange/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/zmq/sugar/socket.py", line 159, in __init__
super().__init__(
File "zmq/backend/cython/socket.pyx", line 332, in zmq.backend.cython.socket.Socket.__init__
zmq.error.ZMQError: Too many open files
This lab session is only running a single notebook. While that notebook is large, it's not extremely large.
I've investigated some, and lsof reveals that there are >30 copies of various library files being opened. For example:
$ lsof | grep scipy/sparse/_sparsetools.cpython-310-x86_64-linux-gnu.so | wc
33 361 8613
$ lsof | grep indexing.cpython-310-x86_64-linux-gnu.so | wc
33 361 8481
These individually look like this:
COMMAND PID TID TASKCMD USER FD TYPE DEVICE SIZE/OFF NODE NAME
python 1325008 adamginsburg mem REG 2445,764964 4384216 180149904096755221 /blue/adamginsburg/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/scipy/sparse/_sparsetools.cpython-310-x86_64-linux-gnu.so
python 1325008 1325015 ZMQbg/Rea adamginsburg mem REG 2445,764964 4384216 180149904096755221 /blue/adamginsburg/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/scipy/sparse/_sparsetools.cpython-310-x86_64-linux-gnu.so
python 1325008 1325016 ZMQbg/IO/ adamginsburg mem REG 2445,764964 4384216 180149904096755221 /blue/adamginsburg/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/scipy/sparse/_sparsetools.cpython-310-x86_64-linux-gnu.so
python 1325008 1325017 python adamginsburg mem REG 2445,764964 4384216 180149904096755221 /blue/adamginsburg/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/scipy/sparse/_sparsetools.cpython-310-x86_64-linux-gnu.so
python 1325008 1325018 python adamginsburg mem REG 2445,764964 4384216 180149904096755221 /blue/adamginsburg/adamginsburg/miniconda3/envs/python310/lib/python3.10/site-packages/scipy/sparse/_sparsetools.cpython-310-x86_64-linux-gnu.so
i.e., they are all loaded by the same PID but different TIDs.
So, my question: What would cause a jupyter python session to load 30 copies of the libraries, and how do I prevent it?