Procuring a more stable center from outputs obtained from an object segmentation model

Question

So I have a program in which I detect a guitar from a live camera feed, then detect its fret board, and then segment its fret board, of which I then find the contours and fit it in the smallest rectangle possible with the function cv2.minAreaRect(). I lastly extract the center of this rectangle for further use in my program;

The red dot indicates the center

I use the center with several other factors to cast the virtually generated hands to their correct positions on the guitar

def object_segment(img, model):
    global dev_mode
    results = model(img)
    mask_rgba = np.zeros_like(img)
    center = [0, 0]
    angle = 0
    width_rect = 0
    height_rect = 0

    if results[0].masks is not None:
        for j, mask in enumerate(results[0].masks.data):
            mask = (mask.cpu().numpy() * 255).astype(np.uint8)
            mask = cv2.resize(mask, (img.shape[1], img.shape[0]))
            mask_rgba[:,:,0] += mask
            mask_rgba[:,:,1] += mask
            mask_rgba[:,:,2] += mask

            contours, hierarchy = cv2.findContours(mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
            max_area = 0
            max_contour = None
            for contour in contours:
                area = cv2.contourArea(contour)
                if area > max_area:
                    max_area = area
                    max_contour = contour

            if max_contour is not None:
                rect = cv2.minAreaRect(max_contour)
                center = [int(rect[0][0]), int(rect[0][1])]
                angle = rect[2]
                box = cv2.boxPoints(rect)
                box = np.intp(box)
                width_rect = rect[1][0]
                height_rect = rect[1][1]

                if dev_mode:
                    for img_draw in [mask_rgba, img]:
                        cv2.drawContours(img_draw, [box], 0, (0, 0, 255), 2)
                        cv2.circle(img_draw, center, radius=5, color=(0, 0, 255), thickness=-2)

    return mask_rgba, center, angle, width_rect, height_rect

The center serves as a reference point for nearly every other step that complements it. The thing is, this center might show little fluctuations depending on those three models that I mentioned above. And these fluctuations, even though they might not seem like much, do interfere with my obtained accuracy as the reference point is in a constant state of change.

I need to prevent these fluctuations as much as possible. So far, the only thing that has come to mind is to maybe record the past instances of the calculated centers and then maybe perform linear interpolation on them to at least get a smoother change or a better center that does not fluctuate much. But I don't think that's the best solution that should be applied here. Could you guys be so kind to share your wisdom with me?

Thank you kindly.

P.S. After a more thorough research I've come to the conclusion that calculating the moving average of my center might contribute in some way, but I'm still uncertain whether this is the best solution