How do root apps access the ui of other apps?

Question

I am looking to build an app that (with root privileges) can access the ui on another app and click buttons on the screen. I have seen this done with many of the macro apps and some "bot" apps. I know they use the 'device administrator' and i have seen some that need the 'overlay over other apps' permission. I'm not looking for a comprehensive guide, and i know i would have to use some kind of ocr software to process the ui elements. Can someone give me some pointers? Example: https://forum.xda-developers.com/showthread.php?t=2241770 Thanks for any help! Taine

Answer 1

I don't know how THEY do it, but one possible way is through the input shell command. I expect that normal programs don't have permission to use it, but adb shell does, and rooted programs presumably would. Usage is printed as follows:

Usage: input [<source>] <command> [<arg>...]

The sources are:
      mouse
      keyboard
      joystick
      touchnavigation
      touchpad
      trackball
      stylus
      dpad
      touchscreen
      gamepad

The commands and default sources are:
      text <string> (Default: touchscreen) [delay]
      keyevent [--longpress] <key code number or name> ... (Default: keyboard)
      tap <x> <y> (Default: touchscreen)
      swipe <x1> <y1> <x2> <y2> [duration(ms)] (Default: touchscreen)
      press (Default: trackball)
      roll <dx> <dy> (Default: trackball)

You'd do something like

List<String> commandLine = Arrays.asList("input touchscreen swipe 500 10 500 500 100".split(" "));
Process process = Runtime.getRuntime().exec(commandLine);
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(process.getInputStream()));

as mentioned in Any way to run shell commands on android programmatically? . (I've adapted it a bit, and haven't actually tested it, so it might take some fiddling.)

Note that it doesn't let you, e.g, see where buttons are, so you'll have to know what you're doing beforehand.

You might also try Instrumentation, as mentioned here: Android simulate key press