Sunday, April 10, 2011

monkeyrunner: visual image comparison

We have discussed taking screenshots with monkeyrunner in previous posts, but now we are taking a step further by comparing the screenshots obtained during the test run against some reference image.

Firstly, we need these reference images that you can obtain by running specific cases of the tests specially designed to do it. It's not recommended to use DDMS to take the reference image as in some cases the image format and compression may slightly differ from the ones taken by monkeyrunner leading to false positive identification of the dissimilarity.

As a fairly simple example we will be creating a test to verify the correct drag and drop to a new position of a screen widget  , in this case the Home screen tips. We will be sending touch events to move it from its original position in row 2:


to row 4:


This second image, showing the Home screen tips widget in row 4 will be our reference image or the final state we want to verify in our test.
There are some items we can anticipate will be different, like the time in the status bar, the battery level,  the connectivity, etc. Thus why we expect some degree of flexibility in our test not to fail under these circumstances.

We also use compare from the great ImageMagick package that you should have installed for this script to work.
In Debian/Ubuntu and derivatives
   $ sudo apt-get install imagemagick

Having this in mind, our monkeyrunner test would look something like this.

#! /usr/bin/env monkeyrunner import sys import subprocess from com.android.monkeyrunner import MonkeyRunner, MonkeyDevice, MonkeyImage TIMEOUT = 30 SERIALNO = 'emulator-5554' REF = 'reference.png' SCR = 'screenshot.png' CMP = 'comparison.png' ACCEPTANCE = 0.9 device = None def testDropWidgetIntoFourthRow():     reference = MonkeyImage.loadFromFile(REF)     print "moving widget..."     device.drag((230, 300), (230, 600), 2.0, 125)     MonkeyRunner.sleep(3)     screenshot = device.takeSnapshot()     screenshot.writeToFile(SCR)     if not screenshot.sameAs(reference, ACCEPTANCE):        print "comparison failed, getting visual comparison..."        subprocess.call(["/usr/bin/compare", REF, SCR, CMP]) def main():     global device     print "waiting for connection..."     device = MonkeyRunner.waitForConnection(TIMEOUT, SERIALNO)     if device:        testDropWidgetIntoFourthRow() if __name__ == '__main__':      main()

We are using MonkeyImage.loadFromFile() to load the reference image from a file.

This method was not available in monkeyrunner so I implemented it myself and I decided to do it in MonkeyImage, but  it is now included in latest verions of monkeyrunner but as MonkeyRunner.loadImageFromFile().

In case you don't have it in your version you can build it from source (being sure that this patch is included).
monkeyrunner is a SDK component and thus is included in this project that can be downloaded from android SDK source.
Anyway, if you can't do it you may just avoid this step and replace the condition in the if by True.


A brief explanation of the script is:

  1. in the main method we obtain the connection with the device using the serial number specified in SERIALNO. A more sophisticated script should use parameters.
  2. if the connection was successful we run the test
  3. in the test method we load the reference image from the specified file
  4. we send a simulated drag event to move the widget from the second row to the fourth row, using screen coordinates
  5. sleep for a while
  6. get the screenshot
  7. if the images are not the same, considering 90% acceptance value, we use the visual comparison to get a clue of the difference
If you run the script you can verify that the screenshot is taken and the test succeed because they are the same. They only have minor differences in the time and battery level but they lie above the acceptance value.
Now, if we run the test again but this time using (230, 400) instead of (230, 600) in the drag command the widget will be dropped in the third line instead of the fourth and we expect this difference be detected as our reference image for the final state contains the widget in the fourth row, and thus precisely what happens.
This is the comparison image highlighting in red the differences:


Check that as we mentioned before the battery level and time in the status bar are also detected as differences but they are usually under the threshold.

This is not a real repeatable test as we are not leaving the system in the same state that was found but the idea is to present the subject in the simplest possible way to avoid deviating from the main goal which is demonstrating how you can add visual comparison to your tests using monkeyrunner and ImageMagick.
I'm sure this will give you lots of ideas to implement you own tests.
As always, comments are gladly welcome.

40 comments:

Unknown said...

Hi Diego,
How do you get the latest version of monkeyrunner?could you send me the download link?

Diego Torres Milano said...

Added SDK source link to the page.

Unknown said...

Do you build the Jar file by yourself?I got the source code,but I don't know how to convert it to Jar file.I wish Bill Napier could integrate the patch to the SDK tool and launch it.Then everyone can use it,I found the function is very useful,and your artical is also helpful.

Diego Torres Milano said...

You can build the whole SDK and it will contain monkeyrunner.

Unknown said...

I still don't know how to build SDK.could you please send the monkeyrunner to my email?
My email is botong2010@gmail.com.
Thanks a lot

Tiago Maluta said...

MonkeyImage [1] doesn't have loadFromFile() method. Is this correct?

[1] http://developer.android.com/guide/developing/tools/MonkeyImage.html

Diego Torres Milano said...

I updated the post:
"We are using MonkeyImage.loadFromFile() to load the reference image from a file.

This method was not available in monkeyrunner so I implemented it myself and I decided to do it in MonkeyImage, but it is now included in latest verions of monkeyrunner but as MonkeyRunner.loadImageFromFile()."

rharya said...

1. checking=MonkeyRunner.loadImageFromFile(chk)
2.
checking=MonkeyRunner.loadFromFile(chk)
3.
checking=MonkeyImage.loadFromFile(chk)

all the above give error
Traceback (most recent call last):
File "", line 1, in
AttributeError: type object 'com.android.monkeyrunner.MonkeyRunner' has no attri
bute 'loadFromFile'



Please help

rharya said...

1. checking=MonkeyRunner.loadImageFromFile(chk)
2.
checking=MonkeyRunner.loadFromFile(chk)
3.
checking=MonkeyImage.loadFromFile(chk)

all the above give error
Traceback (most recent call last):
File "stdin", line 1, in
AttributeError: type object 'com.android.monkeyrunner.MonkeyRunner' has no attri
bute 'loadFromFile'



Please help

Diego Torres Milano said...

If MonkeyRunner.loadImageFromFile() doesn't work maybe you don't have the latest monkeyrunner version (built from source).

Unfortunately, monkeyrunner has no -V or --version to be able to tell exactly. It would be a nice addition, as well as the support for -u that we commented in this blog before.

Anyway, if you need to be absolutely sure about what's in and what's not, build from source.

Unknown said...

Is there a way to highlight the difference on the reference image, for later review?

Diego Torres Milano said...

The comparison is saved in comparison.png in this example (the value of CMP) and you can use it for later review.
The idea here is not to overwrite the reference image, but you can do it if this is your intention.

Does this answer your question ?

rharya said...

Hello Diego Torres Milano

Could you please upload your jar file so that we could replace our jar file and make it running.

Murali said...

Hi Diego,
The MonkeyImage.sameAs() API is throwing a ClassCastException (java.lang.ClassCastException: org.python.core.PySingleton cannot be cast to com.android.monkeyrunner.core.IMonkeyImage) for me.

Currently I'm using android 3.2 sdk.
Do you have any idea how to resolve this?

Thanks in advance.

Anonymous said...

I'm getting the same issue as Murali above... Do you know if there is a fix for this?

I've also tried going over the images byte-by-byte by using MonkeyImage.convertToBytes('png') but the byte arrays are always different lengths for some reason.

Diego Torres Milano said...

It's a known monkeyrunner bug, it should be fixed soon.

On your second problem, remember that you are comparing PNGs, not bitmaps. If they are not exactly the same their lengths will be different.

Anonymous said...

Thanks Diego,

The odd thing is, is that I am just testing it by taking snapshots about 5 seconds apart. Nothing should have changed, but it is still off by sometimes 100 bytes. I guess its not a good workaround for this issue. I'll be waiting for the fix.

I've also heard about this new tool called ChimpChat. I haven't actually seen it, just saw Bill N comment on it at one point not long ago.

Diego Torres Milano said...

Even the most unnoticeable difference (like the battery icon or the clock) could generate very different PNG file sizes.

I recommend using Imagemagick's compare to do the visual comparison.

Romeo said...

Dear Diego
Do you know how to install the ImageMagick under the Windows environment and use monkeyrunner to check it?
I just downloaded the ImageMagick for Win and installed but I am still confused to use it.

Diego Torres Milano said...

I should say that I've never tested in on Windows, but I think it should work with no or minor changes only.

Take a look at imagemagick usage under windows.

Romeo said...

Dear Diego
Thanks for your kindly reply.
I've searched some information about running it under Windows but still could not help, I just want to know if we could install the package in monkeyrunner. By the way, thank you very much, I will study for it.

Diego Torres Milano said...

Hi Romeo,
ImageMagick is a set of stand-alone programs that you can run, in your case, from the Windows command prompt. There is no need to install anything in monkeyrunner.

Romeo said...

Dear Diego
Thanks. I have checked the usage of ImageMagick but it is not what I want.
Do you know that there is a way sameAs?
I searched that there is a bug about sameAs now. Do you have any comment on it?

Smitha said...

Hi Diego,
Thanks for the information on your blog.
I am working on Monkeyrunner. Screenshot is a very good option for regression testing.
Wanted to know if there is any way to crop the images and compare? Since Acceptance value is not always preferred. Thanks for your time

Diego Torres Milano said...

Hi Smitha,
ImageMagick convert has also the ability of cropping your image, for example:

$ convert input.png -crop 400x200+0+200 output.png

will crop a 400x200 area at (0, 200).

Blaine said...

Hi Diego,

I was wondering how to set up imageMagic to interface with the Python script.

Thanks,

Blaine

Diego Torres Milano said...

There are python bindings available for ImageMagick that could be used, but I've never tried (see http://www.imagemagick.org/download/python)

Blaine said...

I'm confused. The script you wrote in your example is in Python, isn't it? I'm just curious how you were able to call the compareAs() function.

Diego Torres Milano said...

Do you mean sameAs() ?
It's a MonkeyImage method.

Blaine said...

I'm confused how you are using ImageMagic in your scripting.

Did you use MonkeyRunner to compare images or ImageMagic?

If you used imagemagic, how were you able to call it from your script?

Blaine said...

I figured it out! Thank you for all your responses and support! It's been helpful!

Diego Torres Milano said...

Great!

Smitha said...

Hi Milano,

Thanks for the update.
I wanted to know if there is a way to retrieve the state of the android device in order to make a test case complete rather than comparing screen shots.
Good to know your view.

Diego Torres Milano said...

There are plenty of alternatives.
Have you read the other articles in this blog ?

Smitha said...

Hi Milano,

I have read quite a few other topics. But unable to get what you are referring to. Could you please point out.
Thanks,
Smitha

Diego Torres Milano said...

@Smitha,
You question is not very specific so I cannot give you a more detailed answer, but I think what you are looking for is described in monkeyrunner: running unit tests.

Smitha said...

Hi,

Thanks for the response. To get more specific for what I was looking at is, to use getCallState(),getDataActivity() etc
[Reference:http://developer.android.com/reference/android/telephony/TelephonyManager.html] is Monkeyrunner scripts. Please let me know if this is a possible idea,

Diego Torres Milano said...

@Sathia,
Yes, it's possible and very simple by the way. You have to invoke the phone service from monkeyrunner (as ViewClient does with the window server). This probably is a good example for a post by itself. I will be posting it soon.

Smitha said...

Thanks Milano. Would be of great help to me

Diego Torres Milano said...

@Smitha, the answer was published as a new post: http://dtmilano.blogspot.ca/2012/06/monkeyrunner-q.html.