PyQt5: single QProgressBar with multiprocessing gets stuck

Question

I have a very long list to deal with, so I use multiprocessing to speed up the process. Now I want to show the progress in a PyQt5.QtWidgets.QProgressBar. This is the code:

import sys
from PyQt5.QtWidgets import *
import multiprocessing as mp
import threading

targets = list(range(0, 100))


def process(i):
    print("target:", i)
    # do something, for example:
    for c in range(1000):
        for r in range(1000):
            c = c * r + 4


class MainWin(QWidget):

    def __init__(self):
        super(MainWin, self).__init__()
        self.setupUi()

    def setupUi(self):
        self.setFixedSize(500, 90)
        self.layout = QGridLayout()
        self.main_widget = QWidget(self)
        self.progressBar = QProgressBar()
        self.progressBar.setValue(0)

        self.btn = QPushButton('start')

        self.layout.addWidget(self.progressBar, 0, 0, 1, 1)
        self.layout.addWidget(self.btn, 1, 0, 1, 1)
        self.setLayout(self.layout)

        self.btn.clicked.connect(self.run)

    def display(self, args):
        self.progressBar.setValue(self.progressBar.value() + 1)
        print("process bar:", self.progressBar.value())
        # QApplication.processEvents()  # I've tried this function and it has no effect

    def run(self):
        def func(results):
            pool = mp.Pool()
            for t in targets:
                pool.apply_async(process, (t,), callback=self.display)
                results.append(t)
            pool.close()
            pool.join()

        results = []
        t = threading.Thread(target=func, args=(results,))
        t.start()
        # everything is fine without t.join(), but the progress bar always gets stuck when t.join() is called
        # t.join()
        # pass  # do something with the results


if __name__ == '__main__':
    app = QApplication(sys.argv)
    main_win = MainWin()
    main_win.show()
    sys.exit(app.exec_())

Everything is fine without the "t.join()" called. But to gain a complete "results", I have to wait for the end of the thread, in which condition the processBar always gets stuck at around 40%.

How to fix this?

Answer 1

PyQt5 (and Qt in general) needs to continuously run the event loop in the main thread in order to function: redraw the GUI, react to user inputs, etc. If you call t.join(), the main thread is stuck inside the run method, blocking the thread and all the GUI updates, include redrawing of the progress bar. In order to function properly, the code should exit run as soon as possible, so t.join() is not a good solution.

One of the ways to deal with that is by using Qt signals. First, waiting for them dos not block the main loop, so the GUI remains responsive. Second, they are executed in the main thread, so there is no problem with accessing Qt widgets from a non-main thread (which is, generally, a bad idea). Here's how I would suggest rewriting the code:

import sys
from PyQt5.QtWidgets import *
from PyQt5.QtCore import pyqtSignal, pyqtSlot
import multiprocessing as mp
import threading

targets = list(range(0, 100))


def process(i):
    print("target:", i)
    # do something, for example:
    for c in range(1000):
        for r in range(1000):
            c = c * r + 4


class MainWin(QWidget):

    def __init__(self):
        super(MainWin, self).__init__()
        self.setupUi()
        self.done = False

    def setupUi(self):
        self.setFixedSize(500, 90)
        self.layout = QGridLayout()
        self.main_widget = QWidget(self)
        self.progressBar = QProgressBar(self.main_widget)
        self.progressBar.setValue(0)

        self.btn = QPushButton('start',self.main_widget)

        self.layout.addWidget(self.progressBar, 0, 0, 1, 1)
        self.layout.addWidget(self.btn, 1, 0, 1, 1)
        self.setLayout(self.layout)

        self.btn.clicked.connect(self.run)
        self.single_done.connect(self.display)
        self.all_done.connect(self.process_results)

    single_done = pyqtSignal()
    @pyqtSlot()
    def display(self):
        self.progressBar.setValue(self.progressBar.value() + 1)
        print("process bar:", self.progressBar.value())
    
    all_done = pyqtSignal()
    @pyqtSlot()
    def process_results(self):
        print("Processing results")
        pass  # do something with the results

    def run(self):
        def func(results):
            pool = mp.Pool()
            for t in targets:
                pool.apply_async(process, (t,), callback=lambda *args: self.single_done.emit())
                results.append(t)
            pool.close()
            pool.join()
            self.all_done.emit()

        results = []
        t = threading.Thread(target=func, args=(results,))
        t.start()


if __name__ == '__main__':
    app = QApplication(sys.argv)
    main_win = MainWin()
    main_win.show()
    sys.exit(app.exec_())

I've added two signals: single_done, which is emitted every time a single target execution is done, and all_done, which is emitted when all the processing is done. In the end of setupUi they are connected to the corresponding methods for updating the progress bar and for processing the results. run no longer sticks around and exits immediately, and the processing of the results is now done in process_result method, which is called upon completion.

Incidentally, using signals to report intermediate results also gets rid of QObject::setParent: Cannot set parent, new parent is in a different thread warning which you might have been getting. This is because now display is called in the proper thread (since it's invoked using signals), whereas before it was called directly in the thread t, which does not own any of the widgets.