How can you draw custom ViewGroup in Android (API Level 23+) and outside of that ViewGroup should be still accessible?

Question

I am working on a project where I need to do mainly 4 stuff:

1. Get screen source
2. Find out on which button/text/image user has clicked
3. Draw the red rectangle around that object.
4. Repeat

So, what I have achievid so far.

1. to get the screen source, I am using AccessibilityService where I override onAccessibilityEvent
2. from onAccessibilityEvent I get the information about the bounds where to draw. (Like in Rect class, top, left, bottom, right)
3. I have created custom view which extends ViewGroup
4. I have overloaded onMeasure, onDraw methods.

I successfully draw red Rects around the button/text/image, but unfortunately, I am not able to access any other stuff on the screen (i.e the screen gets overloaded by my view and I can't click anymore on the screen)

I have tried:

1. making the custom view's height and width so small that I thought it wouldn't overload the whole screen. The idea actually worked, but it wasn't drawing anymore where I wanted.

Here is my AccessibilityService's meta-data (xml config)

<?xml version="1.0" encoding="utf-8"?>


<accessibility-service xmlns:android="http://schemas.android.com/apk/res/android"
    android:description="@string/app_name"
    android:accessibilityEventTypes="typeAllMask"
    android:packageNames="XX XXX" <!-- I've removed these for some purposes -->
    android:accessibilityFeedbackType="feedbackAllMask"
    android:canRequestFilterKeyEvents="true"
    android:accessibilityFlags="flagIncludeNotImportantViews|flagRequestTouchExplorationMode|flagReportViewIds"
    android:notificationTimeout="100"
    android:canRetrieveWindowContent="true"
    android:canRequestTouchExplorationMode="true"
    android:canPerformGestures="true"
    android:canTakeScreenshot="true">

</accessibility-service>

This is how I technically draw something on screen:

val drw = DrawingView(this)
val params = WindowManager.LayoutParams()
params.flags = WindowManager.LayoutParams.FLAG_WATCH_OUTSIDE_TOUCH
params.type = WindowManager.LayoutParams.TYPE_ACCESSIBILITY_OVERLAY
params.format = PixelFormat.TRANSLUCENT
val wm = getSystemService(WINDOW_SERVICE) as WindowManager
wm.addView(drw, params)

I insert the image as well:

After drawing this red box (Rectangle) the whole screen is inaccessible.

I want to: When I draw the red rectangle, I want other buttons to be accessible as well, because to remove the current red rectangle and draw a new one...

Imagine that this app is something similar to TalkBack (Android's integrated accessibility service)

Answer 1

It does look like you're trying to call the setAccessibilityFocusAppearance, which has been added in Android S - so that might not be all that helpful.

Looking at the AccessibilityService codelab, they add their view without overriding the layout params type field:

       // Create an overlay and display the action bar
       WindowManager wm = (WindowManager) getSystemService(WINDOW_SERVICE);
       mLayout = new FrameLayout(this);
       WindowManager.LayoutParams lp = new WindowManager.LayoutParams();
       lp.type = WindowManager.LayoutParams.TYPE_ACCESSIBILITY_OVERLAY;
       lp.format = PixelFormat.TRANSLUCENT;
       lp.flags |= WindowManager.LayoutParams.FLAG_NOT_FOCUSABLE; // <--- here
       lp.width = WindowManager.LayoutParams.WRAP_CONTENT;
       lp.height = WindowManager.LayoutParams.WRAP_CONTENT;
       lp.gravity = Gravity.TOP;
       LayoutInflater inflater = LayoutInflater.from(this);
       inflater.inflate(R.layout.action_bar, mLayout);
       wm.addView(mLayout, lp);

So I would recommend you try changing

    params.flags = WindowManager.LayoutParams.FLAG_WATCH_OUTSIDE_TOUCH

to the following:

   params.flags = params.flags or WindowManager.LayoutParams.FLAG_NOT_FOCUSABLE