Tuesday, November 27, 2012

AndroidViewClient: UiAutomator support

AndroidViewClient v2.3.1 has been released recently providing UiAutomator compatibility, when supported by the device or emulator. UiAutomator is supported since Android API 16.

This is a great improvement over previous version in two different aspects:

  • it can now be run on non-rooted devices not demanding application modification (as previous version required to use LocalViewServer)
  • the change in the backend now frees the client from port redirection, as ViewServer requests, and at the same time the performance of dumping the View tree is greatly improved
 As an introduction of this new release we will be using a simple example that demonstrates some of the new features. This example us based on the demo application AndroidSampleUi.apk.

This example demonstrates:

  • automatic device connection, handling command line parameters if present
  • automatic View tree dump
  • finding Views using regular expressions or text
  • touching found Views
As a precondition to run this example, install and run AndroidSampleUI.


Zoom buttons let you increase or decrease the margins and consequently move the toggle buttons to demonstrate that they will be found whatever their coordinates are.

#! /usr/bin/env monkeyrunner
'''
Copyright (C) 2012  Diego Torres Milano
Created on Aug 31, 2012

@author: diego
'''


import re
import sys
import os

# This must be imported before MonkeyRunner and MonkeyDevice,
# otherwise the import fails.
# PyDev sets PYTHONPATH, use it
try:
    for p in os.environ['PYTHONPATH'].split(':'):
       if not p in sys.path:
          sys.path.append(p)
except:
    pass

try:
    sys.path.append(os.path.join(os.environ['ANDROID_VIEW_CLIENT_HOME'], 'src'))
except:
    pass
from com.dtmilano.android.viewclient import ViewClient, ViewNotFoundException

vc = ViewClient(*ViewClient.connectToDeviceOrExit())

# Find the 3 toggle buttons, because the first 2 change their text if they are selected
# we use a regex to find them.
# Once found, we touch them changing their state
for t in [re.compile('Button 1 .*'), re.compile('Button 2 .*'), 'Button with ID']:
    try:
        vc.findViewWithTextOrRaise(t).touch()
    except ViewNotFoundException:
        print >>sys.stderr, "Couldn't find button with text=", t

Once this script is run, ViewClient will find a device, which can be specified using its serial number in the command line invoking the script, connects to, automatically dump the tree and then use regular expressions to find two of the ToggleButtons because we couldn't use a fixed text because it changes when the button is clicked. If, for some reason the Button is not found, perhaps because it was move outside the screen using Zoom buttons, a message is printed.

This screenshot show the state of the Buttons after the script has run.



More articles and examples will be coming soon, but I didn't want to miss the opportunity to introduce this new version. One of the most remarkably advantages over plain UiAutomator is the  simplification of the script or test creation and the expressiveness gain of using Python instead of Java.

dump.py

dump.py is also present in AndroidViewClient examples. It now supports several command line options now

usage: dump.py [-u|--uniqueId] [-x|--position] [-d|--content-description] [serialno]

so we can use it to verify the content of the screen. If everything went well running

dump.py --content-description

will show the View tree including also the content descriptions, as given by the following dump


android.widget.FrameLayout id/no_id/1  
   android.widget.LinearLayout id/no_id/2  
      android.widget.FrameLayout id/no_id/3  
         android.view.View id/no_id/4  
            android.widget.FrameLayout id/no_id/5  
               android.widget.ImageView id/no_id/6  
            android.widget.LinearLayout id/no_id/7  
               android.widget.LinearLayout id/no_id/8  
                  android.widget.TextView id/no_id/9 Sample UI v2.0 
      android.widget.FrameLayout id/no_id/10  
         android.widget.RelativeLayout id/no_id/11  
            android.widget.Button id/no_id/12 Show Dialog show_dialog
            android.widget.LinearLayout id/no_id/13  
               android.widget.TextView id/no_id/14  
               android.widget.ToggleButton id/no_id/15 Button 1 OFF button_1
               android.widget.TextView id/no_id/16 v=(75.0,82.0) lw=(115,272) ls=(115,272) wxh=(290,72) margin=(40,80) button_1_info
               android.widget.ToggleButton id/no_id/17 Button 2 OFF button_2
               android.widget.TextView id/no_id/18 v=(75.0,273.0) lw=(115,463) ls=(115,463) wxh=(290,72) margin=(40,80) button_2_info
               android.widget.ToggleButton id/no_id/19 Button with ID button_with_id
            android.widget.ZoomControls id/no_id/20  zoom
               android.widget.ZoomButton id/no_id/21  
               android.widget.ZoomButton id/no_id/22  


UPDATE:
Changed linked version to AndroidViewClient 2.3.1 as some latest commits were not in version 2.3 as found by Durairaj.

30 comments:

Durairaj said...

Thank you for your continued work on UI automation. Will try this and let you know

Regards
Durai

Durairaj said...

Hi Milano,
I tried running the sample as it is, but I could not get it to work.

I got the following error:

Couldn't find button with text=
Couldn't find button with text=
Couldn't find button with text= Button with ID

I verified window_dump.xml from sdcard, I could see the button id/text dumped properly. I am running in a JB non-rooted device, also tried in emulator with the same results.

Diego Torres Milano said...

@Durairaj,
Hi, thanks for your comment and reporting back.
I updated the post including how to obtain the dump and how it should look like.

Let me know if your is different.

Durairaj said...

Hi Milano, the problem is solved after applying "Fixed text property use with UiAutomator" from your git repository. Now it is able to identify the views, and sending the touch events, but still it is not sending the touch events at the right co-ordinates. I suspect Line number 451 in viewclient.py "if not self.useUiAutomator:"

Communicating via comments is difficult :-)

Diego Torres Milano said...

@Durairaj,

You will need commit 1b37d23dc5543507ea07ab0cc69b96f9c3245818 as well to be able to run the same example as described in the post.

I strongly agree, this comments are really a pain. Feel free to use Google+ which is a little less painful.

Durairaj said...

Thank you. It is working correctly now.

Diego Torres Milano said...

@Durairaj,
Updated the link in the post to include these latest commits. Thanks for reporting it.

etherjoe said...

Hey Diego + co., thanks for this latest update. I'm making partial progress and then running into a com.android.ddmlib.ShellCommandUnresponsiveException.

I am able to connect to an initial screen, but when I go to a subsequent screen, which is much more content-rich, I'm getting this exception after about 5 seconds.

I'm thinking there is a timeout somewhere in the android code that could be extended?

etherjoe said...

ah, I found this SO post which sheds some light

http://stackoverflow.com/questions/10502833/androids-monkeyrunner-occasionally-throws-exceptions

Requires doing a custom Android jar build though. Is there a less painful way to address this?

Diego Torres Milano said...

@etherjoe,
Are you using ViewClient's UiAutomator backend?
It should happend if you are. Just to verify this, use the following lines in your script:

vc = ViewClient(*ViewClient.connectToDeviceOrExit())
if vc.useUiAutomator:
print "ViewClient: using UiAutomator backend"

etherjoe said...

@Diego, looks like I am:

>>> vc = ViewClient(*ViewClient.connectToDeviceOrExit())
>>> print vc.useUiAutomator
True

still getting the exception though. :(

Unknown said...

Hi Diego-

We love the software, but are facing an issue on JB MR1 devices when using UIAutomator backend: When calling .touch() on a view, the touch event is sent 25 pixels lower (y-axis) than it should be. I believe it is due to the 25 pixel status bar at the top of the screen.

Do you have a recommendation for how we should work around this?

etherjoe said...

oops, regarding the com.android.ddmlib.ShellCommandUnresponsiveException, this is the correct SO link:

http://stackoverflow.com/questions/4264057/android-cts-is-showing-shellcommandunresponsiveexception-on-emulator

This article provides some insight on extending the default DDMS timout.

Rasmikanta Guru said...

I am getting the following error:
view = vc.findViewWithAttributeOrRaise('ENGLISH', 'English')
File "D:\Telus\Softwares\AndroidViewClient-master\AndroidViewClient\src\com\dtmilano\android\viewclient.py", line 1409, in findViewWithAttributeOrRaise
raise ViewNotFoundException("Couldn't find View with %s='%s' in tree with root=%s" % (attr, val, root))
com.dtmilano.android.viewclient.ViewNotFoundException: Couldn't find View with ENGLISH='English' in tree with root=ROOT

Tara said...

Hi Diego,
view.touch() not working properly, if the view is part of an activity which is not full screen. getXY() also returns wrong co-ordinates(might be returning relative co-ordinates). Same issue is there with monkeyrunner touch(). Is there any workaround for this in androidviewclient?

Thanks & Regards,
Tara

Diego Torres Milano said...

@Tara,
AndroidViewClient suppports the case you mention in your post, however there are tons of variations.

Please specify your case properly, with all the details and screenshots if possible. Create the issue at https://github.com/dtmilano/AndroidViewClient/issues so it can be tracked properly.

srikanth said...

your providing such a valuable information about studying..and also have some good key points to every student.

IBM MQ online training

rohit_blog said...

Is there any way how can I run a adb command in UI Automator

Diego Torres Milano said...

@rohit_blog,
I don't understand what you are trying to do.
Can you please provide more information, examples, etc.?

If your comment is related in any way with AndroidViewClient (which I guess it is) please take a look at this post: http://dtmilano.blogspot.com/2012/10/androidviewclient-google-pages.html

I really appreciate if you use Google+ for the comments as they are easier to track there.

Diego Torres Milano said...

@Manasa Muchchatti,
You dind't provide much information about your problem (i.e. source code) but I think I can guess.

After dragging you have to capture the new screen content again using ViewClient.dump():

drag(...)
sleep(...)
vc.dump()
vc.findViewById(...)


Also, please take a look at this post: http://dtmilano.blogspot.com/2012/10/androidviewclient-google-pages.html.
I would really appreciate if you place your AndroidViewClient comments there as they are much easier to handle.

Magento Ecommerce Development said...

Wow Seems like you are using the Python programming languages. But Any how I prefer to use the JAVA Programming to make the Android Apps. Looking for some JAVA Codes from you.

Mobile App Development

Jerry Chen said...

HI Milano,
I got this error very often when dump view:
Error executing command: uiautomator dump /mnt/sdcard/window_dump.xml

is there a better way to avoid this error?
Thanks

Diego Torres Milano said...

@Jerry Chen,
I moved the discussion to Google+.
Please visit https://plus.google.com/111731764904697052166/posts/bXEH4a57RWD

keenos said...

Hi Diego,

I've been attempting to use findViewById('id/idName')

but have been failing. I have access to the source code and know that the activity I'm loading should have the view with android:id="@+id/idName" but I continue to get an exception.

When I run dump.py, I see that all my views on the page have ids similar to id/no_id/#

Does this mean I lose idName as a way to identify my view and have to use id/no_id/# instead? Is there something I'm missing here?

Diego Torres Milano said...

@keenos,
This discussion was moved to Google+: https://plus.google.com/111731764904697052166/posts/3qWrHsdfJSG
See the answer to your question there.

Matheus Sousa Faria said...

Hey, Can you help me?
That is my problem, When I pass the command
vc = ViewClient(*ViewClient.connectToDeviceOrExit())

It raises this error:
ERROR: UIAutomator doesn't contain 'dumped'

Someone knows how to solve this?

Diego Torres Milano said...

@Matheus Sousa Faria,
Latest version (2.3.6) adds a slightly verbose error message that perhaps helps you to detect the problem.

Do you have an SD card ?

Octav Chipara said...

Dear Milano,

This is a great tool. It make automation much easier. However, there is a bug in findViewWithText. I fails to find the text in the last element of a layout. So if my xml layout looks like ..

linear layout
===> button text="start"
===> button text="stop"
===> button text="pause"

It will find start/stop but not pause.

Diego Torres Milano said...

+Octav Chipara,
Thanks for reporting this problem.
Can you provide a test case?
Are all the buttons visible?
Android version?

Tracking bug/issues reports is much easier at https://github.com/dtmilano/AndroidViewClient/issues, please consider creating one there and provide the relevant information.

Zellman said...

Hi Diego,

Have you been able to do multi-touch (e.g. pinch/zoom, or two-finger swipe) via monkeyrunner?

Thanks,
Andrew