Identifying and recording data from screen

Question

I am new to coding and I might be in over my head. I want to write a program that automatically generates the best optimal in-game choice for me to make. I can manage the calculations, but I need a way for my program to automatically identify and record certain things (like damage) by looking at my screen.

Here is what I was thinking:

I can run the program and it will search the screen for values (damage, type of attack, etc.). It will then record those values and run calculations, then return the optimal attack for me to make.

To start, I need to write something that can look at my screen and identify values based on subsections of screenshots. How can I do this? Do I need PIL?

Answer 1

You can train a convolutional neural network to look for certain images on screen. This model could theoretically identify a 'damage' number from an image as well as read the text and convert the image to an actual value. There would be a lot of difficulty here with training as you need a large enough data set to teach your network to identify the correct objects as well as read the number into a value that you can process.

If you are new to coding this would probably be too much to take on. I am sure there are better ways of approaching this by creating a mod for the game that can actually monitor damage values your character inflicts, which would be infinitely more easier than trying to get a neural network to do this. I have no experience creating mods, just machine learning stuff.

Answer 2

you might want to try pyautogui.

Here are some features, as mentioned in the documentation:

PyAutoGUI has several features:

* Moving the mouse and clicking or typing in the windows of other applications.
* Sending keystrokes to applications (for example, to fill out forms).
* Take screenshots, and given an image (for example, of a button or checkbox), find it on the screen.
* Locate an application’s window, and move, resize, maximize, minimize, or close it (Windows-only, currently)
* Display message boxes for user interaction while your GUI automation script runs.