I want to be able to extract data from a screenshot of a game's scoreboard and am looking for guidance on how to start. I want it to work with many combinations of resolutions and aspect ratios (the scoreboard is scaled and moved depending on the game resolution and aspect ratio).
I've attached an example screenshot below, including what I want to extract.
- Green circles: Whether each player on 'survivor' team escaped (first 3) or died (last one)
- Red box: Character name
- Yellow box: left to right, 4 perks (each identifiable), 1 offering, 1 item and then 2 addons
- Orange box: Score for each player
Note that I've obscured player names and chat.
To start, I've tried using OpenCV to detect where the horizontal lines are on the scoreboard to try and estimate where it is to no avail and think I'm heading in the wrong direction.
Before I spend more time trying to learn something new, I want a plan for how to do it. Loosely in my head, it's the following, but I'd like some suggestions on the best way to achieve it.
I think I need some object detection (probably using Tensorflow - suggestions encouraged, please) to locate items on the scoreboard (e.g. perks, offerings, items, addons).
Then after I've got the locations of the perks etc., I can use some object classification to identify which perk/offering/item/addon of each is. I can then use those locations to find areas to OCR to find out the character names, scores etc.
I've not really done anything to do with Computer Vision and TensorFlow in the real world so guidance would be much appreciated. Thanks!
If you're doing this for streaming or recording, you may want to check out the free OCR plugin for OBS or ScoreSight which allow for setting up OCR for regions of a screen capture. You can then use the OCR'd information to display on screen or save to file.
I'm the developer of these tools. Reach out with questions!