## Tuesday, February 07, 2012

### monkeyrunner: interacting with the Views

Amazingly, this post is at the top of the stats meaning that still a lot of people are still using monkeyrunner and haven't discovered yet AndroidViewClient/culebra. Come on! It's time to evolve.

The time may come when you want your tests to interact with the Application Under Test (AUT) in a more clever way than just guessing the View's coordinates on the screen and sending the events.

Furthermore, this is sometimes not possible at all because in order to send the events you may need to obtain some View state. Let me give you an example to illustrate this and if you want you can try to solve it using monkeyrunner.

We can use Development Settings as our AUT and our intention could be to activate the Show running processes and Immediately destroy activities settings.
As we may accustom to do, we can obtain the coordinates of these Views on the screen and send the corresponding touch events using MonkeyDevice.touch() as usual. Sonner or later, maybe sooner, we will discover that because these settings are persistent we should know the state before sending the event, otherwise we will be changing its state other than just enabling this settings as is this example's intention.

We introduced a way of doing things like that in
Automated Android testing using Sikuli using visual comparison and obtaining properties like the text in EditText's in monkeyrunner: testing views properties (which depends on a patch to chimpchat that has not yet been approved) but now we are craving for a more general approach unless we had the intention to patch chimpchat to support all of the properties in the different Views.

This approach is AndroidViewClient that you can download and install from github. Still has its rough edges but I wanted to show its functionality here and be open to comments.
AndroidViewClient adds to monkeyrunner the ability of
• finding Views by ID, very much like you normally do in your Android Activity (using ViewClient.findViewById())
• obtaining the value for almost any of the Views properties (using for example View.isChecked() or View.mText())
• sending touch events to the Views by simply invoking View.touch()
The following script is a case of these abilities in action.

-->
#! /usr/bin/env monkeyrunner
'''
Copyright (C) 2012  Diego Torres Milano
Created on Feb 3, 2012

@author: diego
'''

import re
import sys
import os

# this must be imported before MonkeyRunner and MonkeyDevice,
# otherwise the import fails
try:
ANDROID_VIEW_CLIENT_HOME = os.environ['ANDROID_VIEW_CLIENT_HOME']
except KeyError:
print >>sys.stderr, "%s: ERROR: ANDROID_VIEW_CLIENT_HOME not set in environment" % __file__
sys.exit(1)
sys.path.append(ANDROID_VIEW_CLIENT_HOME + '/src')
from com.dtmilano.android.viewclient import ViewClient

from com.android.monkeyrunner import MonkeyRunner, MonkeyDevice

# Displayed com.android.development/.DevelopmentSettings: +379ms
package = 'com.android.development'
activity = '.DevelopmentSettings'
componentName = package + "/" + activity
device = MonkeyRunner.waitForConnection(60, "emulator-5554")
if not device:
raise Exception('Cannot connect to device')

device.startActivity(component=componentName)
MonkeyRunner.sleep(5)

vc = ViewClient(device)
vc.dump()

showCpu = vc.findViewById("id/show_cpu")
alwaysFinish = vc.findViewById("id/always_finish")

if not alwaysFinish.isChecked():
alwaysFinish.touch()

if not showCpu.isChecked():
# WARNING: Show CPU usage is de-activated as soon as it's activated, that's why it seems it
# is never set
showCpu.touch()

Once we run this script against a running emulator or device we will be able to see how the settings are enabled.

Moreover, if you run the script again you will see how the settings remain unchanged.
AndroidViewClient is a work in progress but it is already useful for many cases where the monkeyrunner scripts are not enough or where the complexity of achieving the desired goals is too high.

Give AndroidViewClient a try, share your comments, suggestions, patches and scripts here to help improve it.

Joe Bowbeer said...

Thanks for the writeup. This same information is available in Java from the full hierarchyviewerlib.

The hierarchy viewer DeviceBridge can be initialized from ChimpChat as follows:

DeviceBridge.acquireBridge(
AndroidDebugBridge.getBridge());

Have you observed that sending DPAD physical button presses does not work in monkeyrunner on android-15 emulators, whereas it worked on android-10?

chimpDevice.press(
TouchPressType.DOWN_AND_UP);

// nope

Joe Bowbeer said...
This comment has been removed by the author.
Joe Bowbeer said...

See Issue 25369 for a workaround.

Canvas said...

@Joe Bowbeer, I'm a freshman to Chimchat. Could you give advice how to use DeviceBridge in chimchat?

Joe Bowbeer said...

ChimpChat is the internals of MonkeyRunner that has recently been refactored so that it can be accessed directly in Java.

A basic sequence is:

ChimpChat cc = ChimpChat.getInstance();
IChimpDevice cd = cc.waitForConnection(5000, ".*");
if (cd != null) {
cd.wake();
TouchPressType.DOWN_AND_UP);
cc.shutdown();
}

A simple HierarchyViewer can be obtained from IChimpDevice:

HierarchyViewer hv = cd.getHierarchyViewer();

However, this hv has a bug (see patch submitted by Diego) and it is missing some functionality.

DeviceBridge is a more powerful interface. You can pretty much write your own hierarchyviewer tool using DeviceBridge.

DeviceBridge.acquireBridge(AndroidDebugBridge.getBridge());
IDevice[] devs = DeviceBridge.getDevices();
IDevice dev = devs[0];

Android app development said...

Excellent post.I got a fantastic idea about Android iPhone developer by read this post.This is one of the thoughtful post.
Android app developers

Diego Torres Milano said...

Thnaks Joe Bowbeer for your contributions to this post.

Dear Diego,

Many thanks for your great sharing of Android knowledge.

For MonkeyRunner, I would like to confirm two basic concepts first:
1. Did your MonkeyRunner work on Windows platform? I only success on Ubuntu environment.
2. If MonkeyRunner could work and control GUI event. Once the event was done, how to you measure the result if correct or not? For example, if MonkeyRunner launch a browser with an URL, how to confirm the behavior was correct? This cannot be done by MonkeyRunner.

This comment has been removed by the author.

This comment has been removed by the author.

This comment has been removed by the author.
evgeny9 said...

Hello, Diego!
Thank you for another great post.
Couldn't you help me out with installing your tool AndroidViewClient in Windows?
Where should I place it to have a successful import in my monkeyrunner script?

evgeny9 said...

I opened an issue with an error in your project: https://github.com/dtmilano/AndroidViewClient/issues/2.

Diego Torres Milano said...

@evgeny9: I added an extra debug message to help us find the issue. Check github.

Diego Torres Milano said...

@小腳掌 ,
1) monkeyrunner (lowercase) should run on Windows as well as Linux and OSX, and BTW it's Android's not mine ;-)

2) I added the example (browser-open-url.py) to (github). It's exactly what you are trying to do, so I hope it helps.

sridhar said...

Hi Thanks for the valuable post.

My question is regarding hierarchyviewer.

I am able to get windows name as Joe Bowbeer said.
how to get Children size and children names of ViewNode.

Garry said...

I would like to thank Diego. In this blog, discovered for himself useful information.
I wish to draw attention to the use of AndroidViewClient in landscape mode.
Perhaps someone can help solve this problem?

PP said...

Hi Diego,
now i have a question about how to implement a method like
touchByText(self, text)
getXY of the text
device.touch with the (x,y)

I just want to simply use text instead of android id.

Thank you.

bharat said...

Hello,

Why am I getting a ERROR: ANDROID_VIE
W_CLIENT_HOME not set in environment

while running your example script browser-open-url.py

What I am doing is monkeyrunner.bat "C:\path\AndroidViewClient\examples\browser-open-url.py"

Diego Torres Milano said...

ANDROID_VIEW_CLIENT_HOME is mandatory. You must set it up in the environment specifying the installation directory of AndroidViewClient.

ビクトル said...

When I try to load ChimpChat in my Android app I get the following error, any idea?

05-04 16:45:10.488: E/AndroidRuntime(4890): java.lang.NoClassDefFoundError: com.android.chimpchat.ChimpChat

Diego Torres Milano said...

Are you following Using monkey from Java ?

Chimpchat is not intended to be used in your Android application but on a Java one.

Anand P said...

I getting a ERROR: ANDROID_VIE
W_CLIENT_HOME not set in environment

I have set the environment variable in eclipse as well as in the code.

Code:

try:
ANDROID_VIEW_CLIENT_HOME = os.environ['C:/Users/Dell/Desktop/dtmilano-AndroidViewClient-f93204c/AndroidViewClient']
except KeyError:
print >>sys.stderr, "%s: ERROR: ANDROID_VIEW_CLIENT_HOME not set in environment" % __file__
sys.exit(1)
sys.path.append(ANDROID_VIEW_CLIENT_HOME + '/src')

Running the script from command prompt:

monkeyrunner.bat C:\workspace_jython\JythonSample\src\Sample.py

Diego Torres Milano said...

ANDROID_VIEW_CLIENT_HOME is not set in your environment. Try:

set ANDROID_VIEW_CLIENT_HOME='\path\to\androidviewclient'

in your command prompt before running the script.

robin zhou said...

Diego:
The AndroidViewClient is a great module, it give a chance to take monkeyrunner into more testing. and make it possible to reuse the script even the UI screen have diffrent.
the core concept is dump the infomation of current view, and store it into the vc.views. then pick up the position and id informaiton from it. BUT, the data is so large about 2M,each dump action will use more than 10s via the adb port forward.
So I think it is the key trouble.
whether I have something wrong in my test? how about the performance in your test for a dump operation?

robin zhou said...
This comment has been removed by the author.
rilwan said...

Dear Diego,
Excellent tool,Expanding monkeyrunner to new areas.thanks.
Is there any documentation or webpage how it is taking the dump.dump action is taking 8-10 secs,sometimes UI is changing by the time.
Is there any way to improve dump action.?
thanks.

Diego Torres Milano said...

@rilwan, Thanks for your comment. Unfortunately the time it takes to get the dump is mainly due to the ViewServer and not much can be done on the client side. If you run hierarchyviewer you can see that it takes almost the same time because both tools use the same strategy.

rilwan said...

itsmyworld said...

hii diegi i have done all th ethings but it is throwing one last error but i was unable to solve this............i think so the jar file is missing
120606 19:45:36.152:S [main] [com.android.monkeyrunner.MonkeyRunnerOptions] Scri
pt terminated due to an exception
120606 19:45:36.152:S [main] [com.android.monkeyrunner.MonkeyRunnerOptions]Trace
back (most recent call last):
File "C:\Users\279281\Desktop\raju1.py", line 14, in
from com.dtmilano.android.viewclient import ViewClient
ImportError: No module named dtmilano

itsmyworld said...

can u plz let me know what would be the error

Smitha said...

Hi Milano,

I am getting the following error:

C:\Users\smgs\Desktop\MonkeyRunner_Bangalore\dtmilano-AndroidViewClient-f93204c\dtmilano-AndroidViewClient-f93204c.

I have done this:
set ANDROID_VIEW_CLIENT_HOME='\path\to\androidviewclient'

Avinash Rao said...

Hi Diego,
this is my doubt:
showCpu = vc.findViewById("id/show_cpu")
when u use this command, the id "show_cpu" that you are using in findViewByID, where is that coming from?
is it from the android application(the xml files in resources)?

Diego Torres Milano said...

@Avinash Rao,
You are right, the IDs are the ones of the Views in the layout. Usually they are set in the XML file.
That's precisely the advantage of AndroidViewClient's findViewById(), instead of using screen coordinates you can use the IDs.

myfirstblog said...

Hi
Now is.checked() method is not there in viewclient , Can u confirm is it remvoe from the view client

Diego Torres Milano said...

@myfirstblog,
Do you mean isChecked() ?
No, it wasn't removed.
What's failing?

Mallik Mahankali said...

Diego,

with your viewclient.py. parseTree () is not alculating the parents and tree strucure properly which causes it to return wrong values in getXY(). I see issue happen when treeLevel - newLevel is more than 1. Although print of received shows right heirachy the traverse heirarchy being printed is different.

Hope so said...

Hi Diego

If Any View with NO_ID , than how to touch that view or how to find the text in that view

Thanks
Mukesh

Mallik Mahankali said...

Mukesh,

If view doesnt have an id/xxx but has a mText Value or any unique attribute value you can use the getViewWithAttribute() and get the corresponding View.

Diego Torres Milano said...

@Mallik Mahankali,
Thanks for reporting the issue with parseTree().
Do you have any test case I could use to see the problem?

Hope so said...

Hi Mallik
i already tried findviewwithAttribute() and getting view ,
but i have to make touch on that text ,
1.for that i am using easyDevice.touch(), in that i have to pass id
2.if i using getXY function then it giving relative cordinate from that view ,

i am not able to touch to that text

Please let me know any other method

Mallik Mahankali said...

@Diego,

simple example I tried was to touch Phone Icon in the home screen on Google Nexus ICS device. As we know we get only the relative coordinates of the parent which is id/hotseat view for this case. We need to traverse back and add x and y values to get right value until we reach phonedecorView. I didnt get a chance yet as i am to see to find a way to make monkeydevice getViewsByText() Functionality to work but it is not going too well either Thank you for your support Diego.

Diego Torres Milano said...

The latest AndroidViewClient patch (https://github.com/dtmilano/AndroidViewClient/commit/75cf5acafc7c24bcd944424c205ced95301a80ff) may help to identify NO_ID Views.

Feedback is welcome.

Hope so said...

Hi Diego

good to see handle No_id cases,
But this id we cant use in Easy Monkey device , and not able to do the touch event for the No_id cases

Thanks

Diego Torres Milano said...

With the latest patch, every View having NO_ID is assigned a new unique ID in the uniqueId filed. This should help in most of the cases.

Sometimes it's a bit trickier to guess which uniqueId belongs to the View you are looking for (they are assigned consecutively starting from 'id/no_id/1'). But once you get it you can simply do

uniqueId = 'id/no_id/5'
view = vc.findViewWithAttribute('uniqueId', uniqueId);

Mallik Mahankali said...

@Diego,
Thanks for the patch, I was able to ues findviewwithattribute and use mText to get the desired view but unfortunately as i mentioned in phone icon example the tree construction doesnt give me right coordinates, I modified monkeyrunner itself to find view by attribute and i see the same thing may be its a special case for hotseat apps

Mallik Mahankali said...

@Diego,

Any idea how monkeyrunner can be used to handle force closes and ANR ? I found advices on internet to use python subprocess module but not sure how you can do that. Just run a subprocess for logcat and parse for strings should do i guess, I will try and see if it works, but is there a better way monkeyrunner can tell us there is an ANR ?

Ashwani Kumar said...

Hi Diego, while executing this tutorial code i encountered the following error:
] Script terminated due to an exception
]Traceback (most recent call last):
ewClient\src\com\dtmilano\android\viewclient.py", line 160, in touch
(x, y) = self.getXY()
ewClient\src\com\dtmilano\android\viewclient.py", line 128, in getXY
x = self.getX()
ewClient\src\com\dtmilano\android\viewclient.py", line 105, in getX
x += int(self.map['layout:mLeft'])
KeyError: layout:mLeft

Ashwani Kumar said...

Hi Diego,

What is 'self'? How i can use it? Is there is any way using which i can type text in edittext views of an app.

thanks
Ashwani

Diego Torres Milano said...

@Ashwani Kumar,
I couldn't reproduce the problem you mentioned (KeyError: layout:mLeft) but one of the latest patches (https://github.com/dtmilano/AndroidViewClient/commit/62175fa64bf6e978863feefda48b407085d3700b) should fix your problem.
Let me know if this is true.

Diego Torres Milano said...

@Mallik Mahankali,
The latests patches to Android View Client 62175fa64bf6e978863feefda48b407085d3700b) should fix the touch issues.
If you can find a test case where the latest AndroidViewClient version fails please let me know.

Ashwani Kumar said...

Thanks Diego for your efforts. One more case i would like to highlight here suppose we have a preference activity. In such activities we don'd defines id's instead keys are used. AndroidViewClient will not work with such activities.

Diego Torres Milano said...

@Ashwani Kumar,
It doesn't matter if they are Preferences, as long as they are shown on the screen Views are the containers. You will probably have a lot of repeated IDs but you can use ViewClient.findViewWithText() to find what you want.

I will post an example using ApiDemos soon.

Ashwani Kumar said...

Hi, I had tired that also. I my case i had ended up with textviews. For example i had a preference activity, which contains some check boxes and name associated with those checkboxes. When i used findviewbytext() method it pointed to the textview not the checkbox. Hence, i was unable to check the state of check box. I tried to use alternatives also by adding checks and so on but nothing worked because my preference activity was using multiple layouts pinned up within each other due to which it was not possible to do that. By the way i will wait for the demos you are about to post. Thanks

Diego Torres Milano said...

@Ashwani Kumar,
True. But if you touch the TextView the Checkbox is changed. That's what my examples are doing.
Let me know if this works for you.

Ashwani Kumar said...

I created two views objects. One for Text view and second for checkbox. I obtained their IDs using the following code:
text = findViewWithText(Launch)
check = findViewByID("id/checkbox")
if not check.isChecked():
text.touch()

if check.isChecked():
print "I am checked"

Bit I am checked is never printed. reason being my activity contains two views with id/checkbox. I tried to use other properties of the checkbox i wanted to check. but nothing worked. If you want i will mail you the code i printed using AndroidViewClient's getViewIds() method.

Anonymous said...
This comment has been removed by a blog administrator.
Anonymous said...
This comment has been removed by a blog administrator.
Anonymous said...

I am trying to install and use AndroidViewClient in ubuntu.But it is giving me NO MODULE named java error.I have searched and searched for how to setup AndroidViewClient in ubuntu but to no avail!! :-( :-(.Please give a precise details of how to install and setup AndroidViewclient in ubuntu.it would be of gr8 help to me!Thanks in advance..am a android newbie..so please bear with me!

Anonymous said...
This comment has been removed by a blog administrator.
Ashwani Kumar said...

Just extract AndroidViewClient and set the classpaths to the root directory of AndroidViewClient. To use Android view client you have to run monkeyrunner from android SDK in shell to execute python files.

Anonymous said...

i have already DONE THAT if you read my post.MY ISSUE IS THAT it is giving me error no module named java.what should be done about that?where n how do i get the module java

Ashwani Kumar said...

Is java installed on your machine. try running "java -version" from shell. If it displays version information that mean java is installed. You also need to check the Java path is set properly i.e. JAVA_HOME. If problem still persists please send complete error log that will help me analyze the error.
Also, i would also like to know how you are executing your python code?
Are you using any IDE?

Anonymous said...
This comment has been removed by a blog administrator.
Anonymous said...

I am not using any IDE.I am invoking the script from shell (python scriptname.py).java version "1.6.0_22" is installed on my machine.i have set ANDROID_JAVA_HOME to /usr/bin/java..is that right?Also,i have set ANDROID_VIEW_CLIENT_HOME to pathtomyhome/pythonmods/dtmilano-AndroidViewClient-579227f/dtmilano-AndroidViewClient-579227f/AndroidViewClient
This is the script i am running-
import re
import sys
import os
try:
sys.path.append(os.path.join(os.environ['ANDROID_VIEW_CLIENT_HOME'], 'src'))
print sys.path
except:
pass

from com.dtmilano.android.viewclient import ViewClient, View

after running this test script i get this -
File "pathtomyhome/pythonmods/dtmilano-AndroidViewClient-579227f/dtmilano-AndroidViewClient-579227f/AndroidViewClient/src/com/dtmilano/android/viewclient.py", line 27, in
import java
ImportError: No module named java

Diego Torres Milano said...

I guess the error
ImportError: No module named java
comes from executing the scripts with python and not monkeyrunner.

Ashwani Kumar said...

Hi Prachi,
As Diego mentioned you have to run monkeyrunner as your pyhton interpreter. You can do that using following command:
monkeyrunner help.py
http://developer.android.com/tools/help/monkeyrunner_concepts.html

Anonymous said...

Thanks!its works when run with monkeyrunner.Where can i get a documentation/tutorial of androidviewclient to start with?it would be of great help!

Diego Torres Milano said...

@Prachi Bhagat,
Some of the documentation can be found
Searching for label AndroidViewClient.

The examples provided in the source distribution also show a lot of aspects of AVC usage.

Anonymous said...
This comment has been removed by a blog administrator.
Anonymous said...
This comment has been removed by a blog administrator.
Anonymous said...

i am playing a video on my android target,i want to touch the views while the video is being played,but that is not happening,i can only touch the views only when the player is paused.can anybody please suggest anything here?

Ashwani Kumar said...

You can do this without AndroidViewClient. You can use sendKeyEvents(KEYCODE) or sendRepeatedKeyEvents(value, KEYCODE, value, KEYCODE)

Anonymous said...
This comment has been removed by a blog administrator.
Anonymous said...

Also,i want to know why i cant touch the views when the video is being played,but only when it is paused??

Anonymous said...

@Ashwani,
i am trying to touch the seekbar to seek the video and it is not happening with the Keycode-"KEYCODE_MEDIA_FAST_FORWARD" using the device.press(Keycode, MonkeyDevice.DOWN_AND_UP)) function,right now what i ve done is i am using the drag(co-ordinates of the seekbar) function,using which i am able to seek the video.But that would be a very hardcoded way to do it.So,i want to touch the seekbar in a generic way,probably using androidviewclient ,if possible

Also,where is this sendKeyEvent() func?is it in monkeyrunner or what?

Anonymous said...

i am importing-
"from com.dtmilano.android.viewclient import ViewClient" in a script.I have set ANDROID_VIEW_CLIENT_HOME.It works fine when i am running the script as main script,but if i import the script in another script,then it says NO module named dtmilano!i just dont get why,Help?

Ashwani Kumar said...

You need to add the module to path variable. As you have done for monkeyrunner and other modules.

Anonymous said...

Diego,

Is it possible to detect or press a particular button or frame using the findviewbytext or View.mText, if so please provide us with a sample code.

Diego Torres Milano said...

@Deepak,
These lines are taken from AndroidViewClient: UiAutomator support. They show how you can use text and even regular expressions to find the Views:

for t in [re.compile('Button 1 .*'), re.compile('Button 2 .*'), 'Button with ID']:
try:
vc.findViewWithTextOrRaise(t).touch()
except ViewNotFoundException:
print >>sys.stderr, "Couldn't find button with text=", t

This works independently of the backend version used (ViewServer, UiAutomator).

rohit_blog said...

Whenever I use vc=ViewClient(device) Error Comes
TypeError: _init_() takes at least 3 arguments(2 given).
What am I doing wrong

Anonymous said...

hI..

I'm able to find a text view and get its co-ordinate and touch it if view is present in 1st page/screen.

If it couldnt find that text in that page/screen, it should scroll to next page/screen. Hence i gave a device.drag() method(which worked fine). But is not able to identify that text. Hence throws error that text is not found.

Why i'm not able to find and touch a text after scroll.

Diego Torres Milano said...

@Manasa Muchchatti,
You dind't provide much information about your problem (i.e. source code) but I think I can guess.

After dragging you have to capture the new screen content again using ViewClient.dump():

drag(...)
sleep(...)
vc.dump()
vc.findViewById(...)

I would really appreciate if you place your AndroidViewClient comments there as they are much easier to handle.

Anonymous said...

Thanks Diego. It worked...And next time will post on particular threads..

Anonymous said...

Hi,

I have a scroll window, i need to find a particular text and click on it so i did this..

tv = self.vc.findViewWithText(text)

MonkeyRunner.sleep(10.0)

if tv:
print tv
self.ApplyTouch(tv,text)

else:
self.device.drag((400,600), (300, 200), 0.01, 120)
MonkeyRunner.sleep(5)
self.vc.dump()
tv = self.vc.findViewWithText(text)
if tv:
print tv
self.ApplyTouch(tv,text)

else:
self.device.drag((400,600), (300, 200), 0.01, 120)
MonkeyRunner.sleep(5)
self.vc.dump()
tv = self.vc.findViewWithText(text)
if tv:
print tv
self.ApplyTouch(tv,text)
else: