Tuesday, November 27, 2012

AndroidViewClient: UiAutomator support

AndroidViewClient v2.3.1 has been released recently providing UiAutomator compatibility, when supported by the device or emulator. UiAutomator is supported since Android API 16.

This is a great improvement over previous version in two different aspects:

  • it can now be run on non-rooted devices not demanding application modification (as previous version required to use LocalViewServer)
  • the change in the backend now frees the client from port redirection, as ViewServer requests, and at the same time the performance of dumping the View tree is greatly improved
 As an introduction of this new release we will be using a simple example that demonstrates some of the new features. This example us based on the demo application AndroidSampleUi.apk.

This example demonstrates:

  • automatic device connection, handling command line parameters if present
  • automatic View tree dump
  • finding Views using regular expressions or text
  • touching found Views
As a precondition to run this example, install and run AndroidSampleUI.


Zoom buttons let you increase or decrease the margins and consequently move the toggle buttons to demonstrate that they will be found whatever their coordinates are.

#! /usr/bin/env monkeyrunner
'''
Copyright (C) 2012  Diego Torres Milano
Created on Aug 31, 2012

@author: diego
'''


import re
import sys
import os

# This must be imported before MonkeyRunner and MonkeyDevice,
# otherwise the import fails.
# PyDev sets PYTHONPATH, use it
try:
    for p in os.environ['PYTHONPATH'].split(':'):
       if not p in sys.path:
          sys.path.append(p)
except:
    pass

try:
    sys.path.append(os.path.join(os.environ['ANDROID_VIEW_CLIENT_HOME'], 'src'))
except:
    pass
from com.dtmilano.android.viewclient import ViewClient, ViewNotFoundException

vc = ViewClient(*ViewClient.connectToDeviceOrExit())

# Find the 3 toggle buttons, because the first 2 change their text if they are selected
# we use a regex to find them.
# Once found, we touch them changing their state
for t in [re.compile('Button 1 .*'), re.compile('Button 2 .*'), 'Button with ID']:
    try:
        vc.findViewWithTextOrRaise(t).touch()
    except ViewNotFoundException:
        print >>sys.stderr, "Couldn't find button with text=", t

Once this script is run, ViewClient will find a device, which can be specified using its serial number in the command line invoking the script, connects to, automatically dump the tree and then use regular expressions to find two of the ToggleButtons because we couldn't use a fixed text because it changes when the button is clicked. If, for some reason the Button is not found, perhaps because it was move outside the screen using Zoom buttons, a message is printed.

This screenshot show the state of the Buttons after the script has run.



More articles and examples will be coming soon, but I didn't want to miss the opportunity to introduce this new version. One of the most remarkably advantages over plain UiAutomator is the  simplification of the script or test creation and the expressiveness gain of using Python instead of Java.

dump.py

dump.py is also present in AndroidViewClient examples. It now supports several command line options now

usage: dump.py [-u|--uniqueId] [-x|--position] [-d|--content-description] [serialno]

so we can use it to verify the content of the screen. If everything went well running

dump.py --content-description

will show the View tree including also the content descriptions, as given by the following dump


android.widget.FrameLayout id/no_id/1  
   android.widget.LinearLayout id/no_id/2  
      android.widget.FrameLayout id/no_id/3  
         android.view.View id/no_id/4  
            android.widget.FrameLayout id/no_id/5  
               android.widget.ImageView id/no_id/6  
            android.widget.LinearLayout id/no_id/7  
               android.widget.LinearLayout id/no_id/8  
                  android.widget.TextView id/no_id/9 Sample UI v2.0 
      android.widget.FrameLayout id/no_id/10  
         android.widget.RelativeLayout id/no_id/11  
            android.widget.Button id/no_id/12 Show Dialog show_dialog
            android.widget.LinearLayout id/no_id/13  
               android.widget.TextView id/no_id/14  
               android.widget.ToggleButton id/no_id/15 Button 1 OFF button_1
               android.widget.TextView id/no_id/16 v=(75.0,82.0) lw=(115,272) ls=(115,272) wxh=(290,72) margin=(40,80) button_1_info
               android.widget.ToggleButton id/no_id/17 Button 2 OFF button_2
               android.widget.TextView id/no_id/18 v=(75.0,273.0) lw=(115,463) ls=(115,463) wxh=(290,72) margin=(40,80) button_2_info
               android.widget.ToggleButton id/no_id/19 Button with ID button_with_id
            android.widget.ZoomControls id/no_id/20  zoom
               android.widget.ZoomButton id/no_id/21  
               android.widget.ZoomButton id/no_id/22  


UPDATE:
Changed linked version to AndroidViewClient 2.3.1 as some latest commits were not in version 2.3 as found by Durairaj.

61 comments:

Durairaj said...

Thank you for your continued work on UI automation. Will try this and let you know

Regards
Durai

Durairaj said...

Hi Milano,
I tried running the sample as it is, but I could not get it to work.

I got the following error:

Couldn't find button with text=
Couldn't find button with text=
Couldn't find button with text= Button with ID

I verified window_dump.xml from sdcard, I could see the button id/text dumped properly. I am running in a JB non-rooted device, also tried in emulator with the same results.

Diego Torres Milano said...

@Durairaj,
Hi, thanks for your comment and reporting back.
I updated the post including how to obtain the dump and how it should look like.

Let me know if your is different.

Durairaj said...

Hi Milano, the problem is solved after applying "Fixed text property use with UiAutomator" from your git repository. Now it is able to identify the views, and sending the touch events, but still it is not sending the touch events at the right co-ordinates. I suspect Line number 451 in viewclient.py "if not self.useUiAutomator:"

Communicating via comments is difficult :-)

Diego Torres Milano said...

@Durairaj,

You will need commit 1b37d23dc5543507ea07ab0cc69b96f9c3245818 as well to be able to run the same example as described in the post.

I strongly agree, this comments are really a pain. Feel free to use Google+ which is a little less painful.

Durairaj said...

Thank you. It is working correctly now.

Diego Torres Milano said...

@Durairaj,
Updated the link in the post to include these latest commits. Thanks for reporting it.

etherjoe said...

Hey Diego + co., thanks for this latest update. I'm making partial progress and then running into a com.android.ddmlib.ShellCommandUnresponsiveException.

I am able to connect to an initial screen, but when I go to a subsequent screen, which is much more content-rich, I'm getting this exception after about 5 seconds.

I'm thinking there is a timeout somewhere in the android code that could be extended?

etherjoe said...

ah, I found this SO post which sheds some light

http://stackoverflow.com/questions/10502833/androids-monkeyrunner-occasionally-throws-exceptions

Requires doing a custom Android jar build though. Is there a less painful way to address this?

Diego Torres Milano said...

@etherjoe,
Are you using ViewClient's UiAutomator backend?
It should happend if you are. Just to verify this, use the following lines in your script:

vc = ViewClient(*ViewClient.connectToDeviceOrExit())
if vc.useUiAutomator:
print "ViewClient: using UiAutomator backend"

etherjoe said...

@Diego, looks like I am:

>>> vc = ViewClient(*ViewClient.connectToDeviceOrExit())
>>> print vc.useUiAutomator
True

still getting the exception though. :(

Unknown said...

Hi Diego-

We love the software, but are facing an issue on JB MR1 devices when using UIAutomator backend: When calling .touch() on a view, the touch event is sent 25 pixels lower (y-axis) than it should be. I believe it is due to the 25 pixel status bar at the top of the screen.

Do you have a recommendation for how we should work around this?

etherjoe said...

oops, regarding the com.android.ddmlib.ShellCommandUnresponsiveException, this is the correct SO link:

http://stackoverflow.com/questions/4264057/android-cts-is-showing-shellcommandunresponsiveexception-on-emulator

This article provides some insight on extending the default DDMS timout.

Unknown said...

I am getting the following error:
view = vc.findViewWithAttributeOrRaise('ENGLISH', 'English')
File "D:\Telus\Softwares\AndroidViewClient-master\AndroidViewClient\src\com\dtmilano\android\viewclient.py", line 1409, in findViewWithAttributeOrRaise
raise ViewNotFoundException("Couldn't find View with %s='%s' in tree with root=%s" % (attr, val, root))
com.dtmilano.android.viewclient.ViewNotFoundException: Couldn't find View with ENGLISH='English' in tree with root=ROOT

Tara said...

Hi Diego,
view.touch() not working properly, if the view is part of an activity which is not full screen. getXY() also returns wrong co-ordinates(might be returning relative co-ordinates). Same issue is there with monkeyrunner touch(). Is there any workaround for this in androidviewclient?

Thanks & Regards,
Tara

Diego Torres Milano said...

@Tara,
AndroidViewClient suppports the case you mention in your post, however there are tons of variations.

Please specify your case properly, with all the details and screenshots if possible. Create the issue at https://github.com/dtmilano/AndroidViewClient/issues so it can be tracked properly.

Blog_Rohit said...

Is there any way how can I run a adb command in UI Automator

Diego Torres Milano said...

@rohit_blog,
I don't understand what you are trying to do.
Can you please provide more information, examples, etc.?

If your comment is related in any way with AndroidViewClient (which I guess it is) please take a look at this post: http://dtmilano.blogspot.com/2012/10/androidviewclient-google-pages.html

I really appreciate if you use Google+ for the comments as they are easier to track there.

Diego Torres Milano said...

@Manasa Muchchatti,
You dind't provide much information about your problem (i.e. source code) but I think I can guess.

After dragging you have to capture the new screen content again using ViewClient.dump():

drag(...)
sleep(...)
vc.dump()
vc.findViewById(...)


Also, please take a look at this post: http://dtmilano.blogspot.com/2012/10/androidviewclient-google-pages.html.
I would really appreciate if you place your AndroidViewClient comments there as they are much easier to handle.

Martin Whisely said...

Wow Seems like you are using the Python programming languages. But Any how I prefer to use the JAVA Programming to make the Android Apps. Looking for some JAVA Codes from you.

Mobile App Development

Jerry Chen said...

HI Milano,
I got this error very often when dump view:
Error executing command: uiautomator dump /mnt/sdcard/window_dump.xml

is there a better way to avoid this error?
Thanks

Diego Torres Milano said...

@Jerry Chen,
I moved the discussion to Google+.
Please visit https://plus.google.com/111731764904697052166/posts/bXEH4a57RWD

keenos said...

Hi Diego,

I've been attempting to use findViewById('id/idName')

but have been failing. I have access to the source code and know that the activity I'm loading should have the view with android:id="@+id/idName" but I continue to get an exception.

When I run dump.py, I see that all my views on the page have ids similar to id/no_id/#

Does this mean I lose idName as a way to identify my view and have to use id/no_id/# instead? Is there something I'm missing here?

Diego Torres Milano said...

@keenos,
This discussion was moved to Google+: https://plus.google.com/111731764904697052166/posts/3qWrHsdfJSG
See the answer to your question there.

Unknown said...

Hey, Can you help me?
That is my problem, When I pass the command
vc = ViewClient(*ViewClient.connectToDeviceOrExit())

It raises this error:
ERROR: UIAutomator doesn't contain 'dumped'

Someone knows how to solve this?

Diego Torres Milano said...

@Matheus Sousa Faria,
Latest version (2.3.6) adds a slightly verbose error message that perhaps helps you to detect the problem.

Do you have an SD card ?

Unknown said...

Dear Milano,

This is a great tool. It make automation much easier. However, there is a bug in findViewWithText. I fails to find the text in the last element of a layout. So if my xml layout looks like ..

linear layout
===> button text="start"
===> button text="stop"
===> button text="pause"

It will find start/stop but not pause.

Diego Torres Milano said...

+Octav Chipara,
Thanks for reporting this problem.
Can you provide a test case?
Are all the buttons visible?
Android version?

Tracking bug/issues reports is much easier at https://github.com/dtmilano/AndroidViewClient/issues, please consider creating one there and provide the relevant information.

Zellman said...

Hi Diego,

Have you been able to do multi-touch (e.g. pinch/zoom, or two-finger swipe) via monkeyrunner?

Thanks,
Andrew

Maria said...

I've been attempting to use findViewById('id/idName')

but have been failing. I have access to the source code and know that the activity I'm loading should have the view with android:id="@+id/idName" but I continue to get an exception.

When I run dump.py, I see that all my views on the page have ids similar to id/no_id/#

Augmented Reality App development

Maria said...

You dind't provide much information about your problem (i.e. source code) but I think I can guess.

After dragging you have to capture the new screen content again using ViewClient.dump():

drag(...)
sleep(...)
vc.dump()
vc.findViewById(...)

Download our free ebook for mobile app development

Mark said...

Very informative article. You are providing such a valuable information about studying..and also have some good key points to every student.

Free guide for macular degeneration diseases

James Harsel said...

Wow Seems like you are using the Python programming languages. But Any how I prefer to use the JAVA Programming to make the Android Apps. Looking for some JAVA Codes from you.

Find Content writers

James Harsel said...

view.touch() not working properly, if the view is part of an activity which is not full screen. getXY() also returns wrong co-ordinates(might be returning relative co-ordinates). Same issue is there with monkeyrunner touch(). Is there any workaround for this in androidviewclient?

Eye Health

Maria said...

This is a great tool. It make automation much easier. However, there is a bug in findViewWithText.

svrtechnologies said...

Really thanks for sharing such an useful stuff...

mq tutorial

Akshay Joshi said...

The language of this blog is very technical and the only android developer can relate to this blog. Can you introduce same blog in simple language so lame person also can relate to the blog and gain some additional knowledge from the same.

Isa Bonita said...

A backlink is a link created when one website links to another. Backlinks are important to SEO & impact for higher ranking. In my 7+ years seo Career i see, without backlinks a website doesn't rank higher on google SERP.

Get Your 300+ High Quality DoFollow Backlinks Here!

Order Now with Full Confidence & 100% satisfaction.

Ashwini said...

Thanks for sharing the post.
We are an experienced team in one of the Best software company and product specialist for software development and implementation. Sovereign provides Website Design, Wordpress Development and Mobile App Development, Digital marketing and SEO Services.

Info World said...

Thanks for sharing a great article.
You are providing wonderful information, it is very useful to us.
Keep posting like this informative articles.
Thank you.

Get to know about 13377x

kumarfield29 said...

Computer networks are the basis of communication in IT. They are used in a huge variety of ways and can include many different types of network. A computer network is a set of computers that are connected together so that they can share information. The earliest examples of computer networks are from the 1960s, but they have come a long way in the half-century since then.

computer network

Anonymous said...

Thanks for Writing this informative content it will help me to understand this topic, a long time I looking for informative content.
To be able to acquire information about some of the best companies around the globe or to
get contact information about companies' executives, you can refer DataGuru
Marketing head database to conserve your valuable time wandering here and there.

Knowledge Hacker said...

This is a great post - so clear and easy to read.if you ever done research on Subscription based grocery delivery app solution. This service provides strategies to follow the online delivery Business. After looking over the few articles on your site, I truly like your technique of Hyperlocal Business App Ideas.

Infoventure Technologies said...

Become an EC-Council Certified Ethical Hacker Training(CEH) with this certification training course. Learn tools & techniques to protect your network. Attend onsite, in-class or customized training. Contact Now! +1 905-625-2727

Anonymous said...

Great post, thank you for sharing with us.
Scrum Master Training
Professional Scrum Master Training
Product Owner Training
SAFe® Agilist Training
Certified Agile Coaching

Flixtor Movies said...

Visit our website flixor to watch full free movies and and Tv shows online.

AFDAH - WATCH FREE MOVIES ONLINE said...

Check out our website afdahtv and watch online free movies and Tv shows without any membership.

Myflixer Movies - Watch Online Free Movies and Tv Shows said...
This comment has been removed by the author.
Mike Johnson said...

Thank you for your useful blog. I think you should lead twitter blog. You can buy twitter followers to promote your blog.

Credeb Advisors said...


Your Blog is very informative, seriously I like your blog. If you want to Application for your business visit ourmobile app development company in delhi

Amritsar digital academy said...

Amritsar Digital Academy is a licensed digital marketing academy that offers advanced online business courses in Amritsar. Digital marketing generates a pool of excellent work chances for students while also assisting large corporations in increasing sales. Our professionals will assist you in understanding digital marketing strategies and gaining hands-on experience with various long term projects so that you may obtain in-depth expertise.

Hulexo said...

Hulexo ERP allows you to effectively streamline all your retail departments. Hulexo ERP helps companies cut down on excessive resources & costs while saving valuable time, money, and manual effort that gets wasted on managing the day-to-day business operations and activities while losing out on the opportunity to grow the business. ERP Dubai

Flixtor Movies said...

The new kid on the block is flixtor movies. It's one of the few websites like 123movies that don't have any adverts or pop-ups. If you have a look around this website, you'll find a big collection of TV shows and movies that you can watch for free online without having to register.

Kito Infocom said...

Website Development Company In Dubai
Website Development Company In Australia
Website Development company in ranchi
Social Media Marketing Agency in Delhi

CG said...

Hii,
This is great and awsome post for me. i loved to read your blog. it's really-really amazing. thanks for inspired me by your blog.CapitalGrocery
Tata Sampann 100% Chana Dal Fine Besan (500 gm)
Aashirvaad Sugar Release Control Atta (5 kg)
Aashirvaad Multigrain Atta (5 kg)
Saffola Active Pro Weight Watchers Edible Oil (Jar) + Active Pro Weight Watchers Edible Oil (5 + 1 litre)

Franticpro said...

Hi, I am John Smith I am Web Developer, It is an amazing blog thanks for the sharing the blog. Frantic infotech provide the native mobile app development such as an information about software development for costumer service. Franti infotech also provide the wearable app development. The development of advanced web applications is Orient Software’s specialty and we will successfully fulfill all your web application development requirements, from small-sized to wider-ranged projects. We Also do work multiple platforms like:
mobile app development services
android app development on mobile
ios app development services
react native app development
ionic app development company
flutter app development company
php web development services in india

Unknown said...

We’re a Web Design Bali with over a decade of experience in building user friendly, secure websites that help strengthen your business.

MednetLabs said...

Great Info! I Recently Came Across Your Blog And Have Been Reading Along. I Thought I Would Leave My First Comment. I Don’t Know What To Say Except That I Have Best Hospital Software in India.
Best Hospital Management Software in India
Best Hospital Software in India

Alfa Designs said...

Thanks for sharing!!!
Your blog is very nice and we hope you are providing more information in future times.
Wine Glass
Bathroom Mirror
Wall Mirror

JacobHarman said...

Quite possibly of the best cross breed stage that permit making a mixture application is Ionic. Ionic isn't simply a structure, an environment incorporates all that improvement needs, beginning from admittance to portable working elements itself to sending and fabricates a cycle, and even dissemination. Made by the organization Drifty Co. in 2013, Ionic is a genuinely late open-source structure that has since laid out its image as a capable half breed application improvement system. Drifty recently was centered around building UI instruments for web and portable and was ultimately determined by client input to make a full cross breed versatile creation device that would zero in on boosting execution>> ionic developers for hire

arjun said...

elza energy claims that it is one of the biggest green energy suppliers. Its energy comes from a variety of solar, wind, renewable sources, hydro, and geothermal facilities.