Poor results from matching against key points and descriptors in OpenCV Python using FAST, BRIEF and FLANN

Question

I'm trying to write a scale and rotation insensitive pattern matcher using a key point approach but I'm struggling to get good results even without changes in scale and rotation.

I'd like to avoid SIFT and SURF if possible as I'd prefer the approach to be patent free. I've experimented with FAST and ORB and have found that FAST seems to be picking up suitable key points. I've then used Brief to extract the descriptors for those points and fed those descriptors into a FLANN pattern matcher (I also tried Brute Force). Despite the template image coming from the same image the search is being conducted on, the results are terrible. Using matchTemplate works well.

This is the code I'm using:

import cv2 as cv
import numpy as np
from matplotlib import pyplot as plt

directory = "C:\\Users\\MickeyCrozier\\Documents\\Digital Microscope\\Default\\Picture\\"
image = "1.bmp"
tmplImage = "tmpl1.bmp"

img = cv.imread(directory+image,flags = cv.IMREAD_GRAYSCALE)
template = cv.imread(directory+tmplImage,flags = cv.IMREAD_GRAYSCALE)

fast = cv.FastFeatureDetector_create() 
fast.setNonmaxSuppression(0)
kp1 = fast.detect(img,None)
kp2 = fast.detect(template,None)

brief = cv.xfeatures2d.BriefDescriptorExtractor_create()

kp1, d1 = brief.compute(img,kp1)
kp2, d2 = brief.compute(img,kp2)

d1 = np.float32(d1)
d2 = np.float32(d2)

output = cv.drawKeypoints(img, kp1, -1, (255,0,0))
cv.imshow('Original',output)
cv.waitKey(0)

output = cv.drawKeypoints(template, kp2, -1, (255,0,0))
cv.imshow('Original',output)
cv.waitKey(0)

matcher = cv.FlannBasedMatcher() 
knn_matches = matcher.knnMatch(d1, d2, 2)


#-- Filter matches using the Lowe's ratio test
ratio_thresh = 0.8
good_matches = []
for m,n in knn_matches:
    if m.distance < ratio_thresh * n.distance:
        good_matches.append(m)


print(good_matches)


output = cv.drawMatches(img,
                        kp1,
                        template,
                        kp2,
                        good_matches,None,
                        flags = cv.DrawMatchesFlags_NOT_DRAW_SINGLE_POINTS)

cv.imshow('Original',output)
cv.waitKey(0)

Here are the results of the key points and match:

Any idea how to improve the match results?