How would I go about detecting brightness in a specific area

Question

I would like to make a robot to automatically play a guitar hero type of game, however all of the notes are the same color. These notes are bright and stand out, so I think that finding the average brightness between two points would be the best way of detecting notes. I am open to new suggestions on how to detect these notes though and any support would be appreciated. I would like to do this project in python, due to how its one of the only coding languages I know. Detecting the color however could work as well, because the notes would be fast, and it shouldn't take more than a millisecond to decide if its a note or not

I already tried googling for my answer, however it seems that there isn't a specific library I should use and there aren't many reputable sources about this topic.

Answer 1

I set up a little code that should get you started at what you want to do. I tried to keep it light on dependencies, but it does require mss for screenshots, PIL for image manipulations and numpy for some numerics.

In brief:

• Take a screenshot and calibrate where the buttons are
• Use those positions to draw lines along each coloured button
• Each frame read the intensity along each line
• Interpret the intensity vector to determine when to issue inputs to play the game

I didn't have rockband, so I looked up a youtube video of someone playing it to get some video, so it should be a pretty close match to your game, credit at the bottom.

Here's the mockup:

from PIL import Image, ImageDraw
import matplotlib.pyplot as plt
import mss
import numpy as np

def get_screenshot(sct: mss.base.MSSBase, monitor_dict: dict) -> Image:
    sct_img = sct.grab(monitor_dict)
    pil_img = Image.frombytes('RGB', sct_img.size, sct_img.bgra, 'raw', 'BGRX')
    return pil_img

def setup(img: Image, num_lines: int, lines: list[tuple[int, int, int, int]] = None):
    # If lines was not already provided, ask the user to provide the info to generate them
    # This is the calibration
    if lines is None:
        plt.imshow(img)
        plt.title('Select corners of bar, starting with bottom left going clockwise')
        points = plt.ginput(4, timeout=-1)
        plt.close()
        lower_line = int((points[0][1] + points[3][1]) / 2)
        upper_line = int((points[1][1] + points[2][1]) / 2)
        lower_space = (points[3][0] - points[0][0]) / num_lines
        upper_space = (points[2][0] - points[1][0]) / num_lines
        lines = [
            (int(points[0][0] + (i + 0.5) * lower_space), lower_line, int(points[1][0] + (i + 0.5) * upper_space), upper_line)
            for i in range(num_lines)
        ]
    # Draw the image with lines on it to let user verify good calibration
    draw = ImageDraw.Draw(img)
    for line in lines:
        draw.line(line, fill=128, width=3)
    plt.imshow(img)
    plt.title('Verify that lines align with screen, abort if not.')
    plt.show()
    return lines


def get_raster_lines(img, lines_expanded):
    # This function is set up to be pretty amenable to a numba improvement if necessary
    raster_lines = np.empty(shape=lines_expanded.shape[:2], dtype=float)
    for line_num in range(lines_expanded.shape[0]):
        for point_num in range(lines_expanded.shape[1]):
            raster_lines[line_num, point_num] = np.linalg.norm(img[lines_expanded[line_num, point_num][1], lines_expanded[line_num, point_num][0]])
    return raster_lines


def _main():
    # How many guitar lines, seems like 5 based on youtube videos
    num_lines = 5
    # The colours to make plots more understandable
    colour_order = ['green', 'red', 'yellow', 'blue', 'orange']
    # Set up the screen capturing library
    sct = mss.mss()
    monitor = sct.monitors[2]
    monitor['mon'] = 1
    img = get_screenshot(sct, monitor)
    # Run this the first time with lines set to None, then after you get the output, you can put it here
    lines = [(540, 799, 675, 470), (639, 799, 707, 470), (738, 799, 739, 470), (838, 799, 771, 470), (937, 799, 803, 470)]
    lines = setup(img, num_lines, lines)
    # lines = setup(img, num_lines)
    # lines follows: [(x0, y0, x1, y1), ...]
    #   [(bottom left), (top left), (top right), (bottom right)]
    print('lines: ', lines) # print so you can put it above and not have to calibrate every time
    # Generate all the points along the lines
    lines_expanded = np.array([
        [(int(np.interp(yval, (line[3], line[1]), (line[2], line[0]))), yval) for yval in range(line[3], line[1])]
        for line in lines
    ])
    # Wait to start so that you can set up the game or any other initial setup
    input('Press Enter when ready to start')
    print('Starting ...')
    # While true so that it will run for the whole game
    while True:
        try:
            # Take the screenshot
            img = get_screenshot(sct, monitor)
            # Pull out the raster lines
            raster_lines = get_raster_lines(np.array(img), lines_expanded)
            # Process raster_lines to generate your commands
            # In this case just plot to show that the information was captured
            for i in range(num_lines):
                plt.plot(raster_lines[i], color=colour_order[i])
            plt.legend([f'bar {colour_order[i]}' for i in range(num_lines)])
            plt.show()
        except KeyboardInterrupt:
            break
        break # comment this out to continue running

if __name__ == '__main__':
    _main()

When I run that I get:

lines:  [(540, 799, 675, 470), (639, 799, 707, 470), (738, 799, 739, 470), (838, 799, 771, 470), (937, 799, 803, 470)]
Press Enter when ready to start
Starting ...

Process finished with exit code 0

Here is the calibration points I used approximately:

And here is the calibration graph, where you can see the red lines run straight up each coloured key lane:

And here is the raster lines it generates:

I've annotated it here to show some features of interest:

And that should be enough to get you going!

Credit to this link for the video I tested with: https://www.youtube.com/watch?v=TIs9x8MfROk