Obtaining code coverage of a running Android application

How can we obtain the code coverage of a running application, not just its tests ?
I have been asked this question many times. Recently, Jonas posted a similar question as comment to Eclipse, Android and EMMA code coverage. So we will elaborate the solution to this problem.
But firstly, let's do a brief introduction of the concepts.

EMMA: a free Java code coverage tool

EMMA is an open-source toolkit for measuring and reporting Java code coverage. EMMA distinguishes itself from other tools by going after a unique feature combination: support for large-scale enterprise software development while keeping individual developer's work fast and iterative.

Android includes EMMA v2.0, build 5312, which includes some minor changes introduced by Android to adapt it for the platform specifics.

Android Instrumentation

The instrumentation framework is the foundation of the testing framework. Instrumentation controls the application under test and permits the injection of mock components required by the application to run.

Usually, an InstrumentationTestRunner, a special class the extends Instrumentation, is used to run various types of TestCases, against an android application.

Typically, this Instrumentation is declared in the test project's AndroidManifest.xml and then run from Eclipse or from the command line using am instrument.

Also, to generate EMMA code coverage -e coverage true option is added to the command line.

Basically, we have all the components but in different places because we want to obtain the code coverage from the running application not from its tests.

EmmaInstrumentation
The first thing we need to do is to create a new Instrumentation that starts the Activity Under Test using EMMA instrumentation and when this Activity is finished the coverage data is saved to a file.
To be notified of this Activity finish we need a listener that we can set extending the AUT because one of our objectives is to keep it unchanged.

To illustrate this technique we will be using the Temperature Converter application that we have used many times in other posts. The source code is as usual available through github.

package com.example.instrumentation;

import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;

import com.example.i2at.tc.TemperatureConverterActivity;
//import com.vladium.emma.rt.RT;

import android.app.Activity;
import android.app.Instrumentation;
import android.content.Intent;
import android.os.Bundle;
import android.os.Looper;
import android.util.Log;

public class EmmaInstrumentation extends Instrumentation implements FinishListener {

    private static final String TAG = "EmmaInstrumentation";

    private static final boolean LOGD = true;

    private static final String DEFAULT_COVERAGE_FILE_PATH = "/mnt/sdcard/coverage.ec";

    private final Bundle mResults = new Bundle();

    private Intent mIntent;

    private boolean mCoverage = true;

    private String mCoverageFilePath;

    /**
     * Extends the AUT to provide the necessary behavior to invoke the
     * {@link FinishListener} that may have been provided using
     * {@link #setFinishListener(FinishListener)}.
     * 
     * It's important to note that the original Activity has not been modified.
     * Also, the Activity must be declared in the
     * <code>AndroidManifest.xml</code> because it is started by an intent in
     * {@link EmmaInstrumentation#onStart()}. This turns more difficult to use
     * other methods like using template classes. This latter method could be
     * viable, but all Activity methods should be re-written to invoke the
     * template parameter class corresponding methods.
     * 
     * @author diego
     * 
     */
    public static class InstrumentedActivity extends
    TemperatureConverterActivity {
        private FinishListener mListener;

        public void setFinishListener(FinishListener listener) {
            mListener = listener;
        }

        @Override
        public void finish() {
            if (LOGD)
                Log.d(TAG + ".InstrumentedActivity", "finish()");
            super.finish();
            if (mListener != null) {
                mListener.onActivityFinished();
            }
        }

    }

    /**
     * Constructor
     */
    public EmmaInstrumentation() {

    }

    @Override
    public void onCreate(Bundle arguments) {
        if (LOGD)
            Log.d(TAG, "onCreate(" + arguments + ")");
        super.onCreate(arguments);

        if (arguments != null) {
            mCoverage = getBooleanArgument(arguments, "coverage");
            mCoverageFilePath = arguments.getString("coverageFile");
        }

        mIntent = new Intent(getTargetContext(), InstrumentedActivity.class);
        mIntent.setFlags(Intent.FLAG_ACTIVITY_NEW_TASK);
        start();
    }

    @Override
    public void onStart() {
        if (LOGD)
            Log.d(TAG, "onStart()");
        super.onStart();

        Looper.prepare();
        InstrumentedActivity activity = (InstrumentedActivity) startActivitySync(mIntent);
        activity.setFinishListener(this);
    }

    private boolean getBooleanArgument(Bundle arguments, String tag) {
        String tagString = arguments.getString(tag);
        return tagString != null && Boolean.parseBoolean(tagString);
    }

    private void generateCoverageReport() {
        if (LOGD)
            Log.d(TAG, "generateCoverageReport()");

        java.io.File coverageFile = new java.io.File(getCoverageFilePath());

        // We may use this if we want to avoid refecltion and we include
        // emma.jar
        // RT.dumpCoverageData(coverageFile, false, false);

        // Use reflection to call emma dump coverage method, to avoid
        // always statically compiling against emma jar
        try {
            Class<?> emmaRTClass = Class.forName("com.vladium.emma.rt.RT");
            Method dumpCoverageMethod = emmaRTClass.getMethod(
                    "dumpCoverageData", coverageFile.getClass(), boolean.class,
                    boolean.class);
            dumpCoverageMethod.invoke(null, coverageFile, false, false);
        } catch (ClassNotFoundException e) {
            reportEmmaError("Is emma jar on classpath?", e);
        } catch (SecurityException e) {
            reportEmmaError(e);
        } catch (NoSuchMethodException e) {
            reportEmmaError(e);
        } catch (IllegalArgumentException e) {
            reportEmmaError(e);
        } catch (IllegalAccessException e) {
            reportEmmaError(e);
        } catch (InvocationTargetException e) {
            reportEmmaError(e);
        }
    }

    private String getCoverageFilePath() {
        if (mCoverageFilePath == null) {
            return DEFAULT_COVERAGE_FILE_PATH;
        } else {
            return mCoverageFilePath;
        }
    }

    private void reportEmmaError(Exception e) {
        reportEmmaError("", e);
    }

    private void reportEmmaError(String hint, Exception e) {
        String msg = "Failed to generate emma coverage. " + hint;
        Log.e(TAG, msg, e);
        mResults.putString(Instrumentation.REPORT_KEY_STREAMRESULT, "\nError: "
                + msg);
    }

    /* (non-Javadoc)
     * @see com.example.instrumentation.FinishListener#onActivityFinished()
     */
    @Override
    public void onActivityFinished() {
        if (LOGD)
            Log.d(TAG, "onActivityFinished()");
        if (mCoverage) {
            generateCoverageReport();
        }
        finish(Activity.RESULT_OK, mResults);
    }

}

We are also implementing the FinishListener interface, which is defined as

package com.example.instrumentation;

/**
 * Listen for an Activity to finish and invokes {@link #onActivityFinished()} when this happens.
 * 
 * @author diego
 *
 */
public interface FinishListener {

        /**
         * Invoked when the Activity finishes.
         */
        void onActivityFinished();

}

Running the instrumented application
Once we have the EmmaInstrumentation class in place we need a few more adjustments to be able to get the coverage report of the running application.

Firstly, we need to add the new Activity to the manifest. Secondly, we should allow our application to write to the sdcard if this is where we decided to generate the coverage report. To do it you should grant the android.permission.WRITE_EXTERNAL_STORAGE permission.

Then, it's time to build and install the instrumented apk:

$ ant clean

$ ant instrument

$ ant installi

Everything is ready to start the instrumented application

$ adb shell am instrument -e coverage true \

     -w com.example.i2at.tc/\

        com.example.instrumentation.EmmaInstrumentation

If everything went well, the Temperature Converter application will be running and we can use it for a while

when we exit by pressing the BACK button we can see that the coverage data was written to the file and reflected in the logcat

I/System.out(2453): EMMA: runtime coverage data written to [/mnt/sdcard/coverage.ec] {in 975 ms}

this file can then be moved to the host computer using adb pull.

Hope this helps you obtaining the code coverage for your application to help you understand its usage patterns. As always, comments and questions are always welcome.

Android: Using monkey from Java

The latest version of the Android SDK and tools include chimpchat, a library that facilitates the use of monkey from Java. This is equivalent to monkeyrunner, which is the bridge between monkey and the Python scripting language.
While Python is an incredibly powerful and expressive scripting language and will permit you creating tests with just a few statements, there are some occasions when you don't want to introduce a new language to the project leaving your Java confort zone or you prefer to leverage the use of previously created libraries instead of writing new ones.
In such cases, you can now have the same access to monkey running on the device with the help of chimpchat, as we are going to demonstrate.


Creating a Java project
Our first step will be to create a new Java project and we will add the required libraries to the Java Build Path as External Jars.
We are naming the project JavaMonkey, for obvious reasons.

We are adding these libraries from Android SDK, which are used directly or indirectly by our project, to the Java Build Path:

• chimpchat.jar
• ddmlib.jar
• guavalib.jar
• sdklib.jar

JavaMonkey.java
Our intention is to create a simple class, serving the purpose of a simple example to get as started. We will be simply:

1. Creating a JavaMonkey object
2. initializing it, this implies creating the connection with any emulator or device found or throwing an exception is not connection was made before the timeout expires
3. listing all the properties in the device or emulator
4. shutting down the connection

Following, is the JavaMonkey class: 

/**
 * Copyright (C) 2011  Diego Torres Milano
 */
package com.example.javamonkey;

import java.util.TreeMap;

import com.android.chimpchat.ChimpChat;
import com.android.chimpchat.core.IChimpDevice;

/**
 * @author diego
 *
 */
public class JavaMonkey {

        private static final String ADB = "/Users/diego/opt/android-sdk/platform-tools/adb";
        private static final long TIMEOUT = 5000;
        private ChimpChat mChimpchat;
        private IChimpDevice mDevice;

        /**
         * Constructor
         */
        public JavaMonkey() {
                super();
        TreeMap<String, String> options = new TreeMap<String, String>();
        options.put("backend", "adb");
        options.put("adbLocation", ADB);
        mChimpchat = ChimpChat.getInstance(options);
        }

        /**
         * Initializes the JavaMonkey.
         */
        private void init() {
                mDevice = mChimpchat.waitForConnection(TIMEOUT, ".*");
                if ( mDevice == null ) {
                        throw new RuntimeException("Couldn't connect.");
                }
                mDevice.wake();
        }

        /**
         * List all properties.
         */
        private void listProperties() {
                if ( mDevice == null ) {
                        throw new IllegalStateException("init() must be called first.");
                }
                for (String prop: mDevice.getPropertyList()) {
                        System.out.println(prop + ": " + mDevice.getProperty(prop));
                }
        }

        /**
         * Terminates this JavaMonkey.
         */
        private void shutdown() {
                mChimpchat.shutdown();
                mDevice = null;
        }

        /**
         * @param args
         */
        public static void main(String[] args) {
                final JavaMonkey javaMonkey = new JavaMonkey();
                javaMonkey.init();
                javaMonkey.listProperties();
                javaMonkey.shutdown();
        }

}

Configuration
One of the important things you have to adapt to your environment is the location of the adb command. Otherwise if you don't set it you will receive:

E/adb: Failed to get the adb version: Cannot run program "adb": error=2, No such file or directory

Hope this helps you get started with chimpchat. As always, comments and questions are always welcome.

Android Application Testing Guide: Q&A

Q:  Hi Diego, I wanted to ask that can i write a monkey runner script which controls a web based apk ?
Eg; I install youtube.apk which is nothing but a browser with hardcoded youtube url.
Now my monkeyrunner script shall install this apk and then pass events such as a search string etc on this web based application.
All this i want to do and control externally through the monkey runner script. Is this possible? If yes, then could you please guide me by some pseudo code? 


Comment on Using Android monkeyrunner from Eclipse

Posted by latha

A:  This is an interesting question and a good monkeyrunner example, so here we go. monkeyrunner has the ability of installing APKs after obtaining the connection with the device. Then we start Youtube main activity, sleep for a bit to let things settle down.
Once we have the activity running is time to start our search. To do it, we touch the Search icon, enter the desired search string, 'android' in this particular case and the we touch the Search button again to actually start the action.
Following, is the script that translates our plan to monkeyrunner:

#! /usr/bin/env monkeyrunner

import sys
import os
from com.android.monkeyrunner import MonkeyRunner, MonkeyDevice

YOUTUBE = 'com.google.android.youtube-2.1.6.apk'
prog = os.path.basename(sys.argv[0])

def usage():
        print >>sys.stderr, "usage: %s" % prog
        sys.exit(1)

def main():
        if len(sys.argv) != 1:
                usage()

        print "waiting for connection..."
        device = MonkeyRunner.waitForConnection()

        print "installing youtube"
        device.installPackage(YOUTUBE)

        device.startActivity(component="com.google.android.youtube/.HomeActivity")
        MonkeyRunner.sleep(3)
        # search
        device.touch(450, 80, MonkeyDevice.DOWN_AND_UP)
        MonkeyRunner.sleep(5)
        device.type('android')
        # done
        device.touch(450, 740, MonkeyDevice.DOWN_AND_UP)



if __name__ == '__main__':
    main()

This script covers the case described in the question but it could be easily adapted for other cases and application.

I hope this is the answer you were looking for.