UiAutomatorViewer doesn't work with secondary/multiple displays

Question

I have a multiple display scenario, which I want to test with Appium in the end. However, right now I am struggeling already with UiAutomatorViewer.

This is my scenario:

• I have setted up Android Emulator with a secondary display: The left screen has Chrome launched, the right side YouTube (just some abitrary apps).

• When I launch uiautomatorviewer (or appium using uiautomator2), it just retrieves me the content of the "main" display and I can just access the elements on this screen. Appium is showing me the same content.

I don't know, how I can access the secondary display as well. In a perfect approach, I would basically be able to identify all apps on all displays. Is this somehow possible?

In addition, I need to test multiple apps. Because of that single app approaches like Espresso don't work for me.

Answer 1

According to the source code, UI automator viewer dumps the view hierarchy of the active window and the screenshot is the one from the default display.

If you want to use ui automator viewer and analyze the right screenshot (and the corresponding UI hierarchy), a possible workaround might be to create both files programmatically and provide them to ui automator viewer:

adb shell am start com.google.android.calendar --display 0
adb shell screencap -d 0 -p /sdcard/screencapture0.png
adb pull /sdcard/screencapture0.png .
adb shell uiautomator dump 
adb pull /sdcard/window_dump.xml
mv window_dump.xml window_dump0.uix

now you can select "Open" on ui automator viewer and select screencapture0.png and window_dump0.uix.

By opening an app on the secondary display the focus moves to the other display and you can reinvoke the same commands to get the 2 files:

adb shell am start com.google.android.calendar --display 1
adb shell screencap -d 1 -p /sdcard/screencapture1.png
adb pull /sdcard/screencapture1.png .
adb shell uiautomator dump 
adb pull /sdcard/window_dump.xml
mv window_dump.xml window_dump1.uix

and this time the files to provide to ui automator viewer are screencapture1.png and window_dump1.uix.

Not sure if the same approach can be leveraged also to execute the tests against a specific screen.

Answer 2

Below we will be discussing about one of the workaround solution that has been identified. In this approach one will be relying on adb (Android databridge command) to switch between primary and secondary display. So when ever the scripts needs to perform action on the secondary screen the app activity running on that screen will be switched to primary screen so that appium will be able to continue the execution without interruption.

Switching of screens can be achieved using adb shell cmd activity display move-stack <taskID> <displayID> which will move the taskID (Task Id of the activity) to displayID (display 0 represents primary, 1 represents secondary or so on).

In order to capture the taskId we can use another adb command named adb shell am stack list which will list out all the stack number along with task id of all activities running on the device ( The command need to be execution by having the application running either in emulator or real device).

The above adb command can be integrated into the appium scripts so that we can dynamically capture the taskID and flip the screen at run time. These call will be done once we start the application or startexisting application.

Challenges with the approach

1. Switching the screens of the application may affect the behaviour of the application because of display size, activity setting etc – its advice to discuss this approach with the team and should be adopted after sufficient testing.
2. Switching screen can lead to flaky test – so do test the execution behaviour of the scripts before fully adopting this approach into your framework.
3. adb command may not get executed from the appium scripts due to different reasons. a. Improper setting of adb Path b. JDK version issues c. Permission issue within android Manifest.xml – In order to enable this following line need to be added and then apk file need to be recompiled and reinstalled. Basically a different version