Why multithreading pyFFTW is slower?

Question

Recently, I need to conduct the parallel FFT on images which is just the two dimensional matrix, and I used pyFFTW in my python programs.However, the result shows that multithreading pyFFTW is slower, and I don't known where the problem is. The followings are my testing code:

def pyfftw64(inp):
    u0 = pyfftw.empty_aligned(inp.shape, dtype='float64')
    u1 = pyfftw.empty_aligned((inp.shape[0], inp.shape[-1]//2 + 1), dtype='complex128')
    
    fft_plan = pyfftw.FFTW(u0, u1, axes=(-2,-1), direction='FFTW_FORWARD', flags=('FFTW_MEASURE',), threads=4)
    ifft_plan = pyfftw.FFTW(u1, u0, axes=(-2,-1), direction='FFTW_BACKWARD', flags=('FFTW_MEASURE',), threads=4)
    
    u0[:] = inp

    start = time.time()
    for i in range(0,1000):
        fft_plan()
        ifft_plan()
    
    #fft_plan()
    #ifft_plan()

    end = time.time()
    
    return end-start

I have tried different matrix size and the threads argumnet, but there is no apparent accelerating effect. So, I want to ask for your help to give some adivces and the method to check the problem.