Tuesday, November 15, 2011

Obtaining code coverage of a running Android application

How can we obtain the code coverage of a running application, not just its tests ?
I have been asked this question many times. Recently, Jonas posted a similar question as comment to Eclipse, Android and EMMA code coverage. So we will elaborate the solution to this problem.
But firstly, let's do a brief introduction of the concepts.

EMMA: a free Java code coverage tool
EMMA is an open-source toolkit for measuring and reporting Java code coverage. EMMA distinguishes itself from other tools by going after a unique feature combination: support for large-scale enterprise software development while keeping individual developer's work fast and iterative.

Android includes EMMA v2.0, build 5312, which includes some minor changes introduced by Android to adapt it for the platform specifics.

Android Instrumentation
The instrumentation framework is the foundation of the testing framework. Instrumentation controls the application under test and permits the injection of mock components required by the application to run.
Usually, an InstrumentationTestRunner, a special class the extends Instrumentation, is used to run various types of TestCases, against an android application.
Typically, this Instrumentation is declared in the test project's AndroidManifest.xml and then run from Eclipse or from the command line using am instrument.
Also, to generate EMMA code coverage -e coverage true option is added to the command line.
Basically, we have all the components but in different places because we want to obtain the code coverage from the running application not from its tests.

EmmaInstrumentation
The first thing we need to do is to create a new Instrumentation that starts the Activity Under Test using EMMA instrumentation and when this Activity is finished the coverage data is saved to a file.
To be notified of this Activity finish we need a listener that we can set extending the AUT because one of our objectives is to keep it unchanged.

To illustrate this technique we will be using the Temperature Converter application that we have used many times in other posts. The source code is as usual available through github.


package com.example.instrumentation;

import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;

import com.example.i2at.tc.TemperatureConverterActivity;
//import com.vladium.emma.rt.RT;

import android.app.Activity;
import android.app.Instrumentation;
import android.content.Intent;
import android.os.Bundle;
import android.os.Looper;
import android.util.Log;

public class EmmaInstrumentation extends Instrumentation implements FinishListener {

    private static final String TAG = "EmmaInstrumentation";

    private static final boolean LOGD = true;

    private static final String DEFAULT_COVERAGE_FILE_PATH = "/mnt/sdcard/coverage.ec";

    private final Bundle mResults = new Bundle();

    private Intent mIntent;

    private boolean mCoverage = true;

    private String mCoverageFilePath;

    /**
     * Extends the AUT to provide the necessary behavior to invoke the
     * {@link FinishListener} that may have been provided using
     * {@link #setFinishListener(FinishListener)}.
     * 
     * It's important to note that the original Activity has not been modified.
     * Also, the Activity must be declared in the
     * <code>AndroidManifest.xml</code> because it is started by an intent in
     * {@link EmmaInstrumentation#onStart()}. This turns more difficult to use
     * other methods like using template classes. This latter method could be
     * viable, but all Activity methods should be re-written to invoke the
     * template parameter class corresponding methods.
     * 
     * @author diego
     * 
     */
    public static class InstrumentedActivity extends
    TemperatureConverterActivity {
        private FinishListener mListener;

        public void setFinishListener(FinishListener listener) {
            mListener = listener;
        }

        @Override
        public void finish() {
            if (LOGD)
                Log.d(TAG + ".InstrumentedActivity", "finish()");
            super.finish();
            if (mListener != null) {
                mListener.onActivityFinished();
            }
        }

    }

    /**
     * Constructor
     */
    public EmmaInstrumentation() {

    }

    @Override
    public void onCreate(Bundle arguments) {
        if (LOGD)
            Log.d(TAG, "onCreate(" + arguments + ")");
        super.onCreate(arguments);

        if (arguments != null) {
            mCoverage = getBooleanArgument(arguments, "coverage");
            mCoverageFilePath = arguments.getString("coverageFile");
        }

        mIntent = new Intent(getTargetContext(), InstrumentedActivity.class);
        mIntent.setFlags(Intent.FLAG_ACTIVITY_NEW_TASK);
        start();
    }

    @Override
    public void onStart() {
        if (LOGD)
            Log.d(TAG, "onStart()");
        super.onStart();

        Looper.prepare();
        InstrumentedActivity activity = (InstrumentedActivity) startActivitySync(mIntent);
        activity.setFinishListener(this);
    }

    private boolean getBooleanArgument(Bundle arguments, String tag) {
        String tagString = arguments.getString(tag);
        return tagString != null && Boolean.parseBoolean(tagString);
    }

    private void generateCoverageReport() {
        if (LOGD)
            Log.d(TAG, "generateCoverageReport()");

        java.io.File coverageFile = new java.io.File(getCoverageFilePath());

        // We may use this if we want to avoid refecltion and we include
        // emma.jar
        // RT.dumpCoverageData(coverageFile, false, false);

        // Use reflection to call emma dump coverage method, to avoid
        // always statically compiling against emma jar
        try {
            Class<?> emmaRTClass = Class.forName("com.vladium.emma.rt.RT");
            Method dumpCoverageMethod = emmaRTClass.getMethod(
                    "dumpCoverageData", coverageFile.getClass(), boolean.class,
                    boolean.class);
            dumpCoverageMethod.invoke(null, coverageFile, false, false);
        } catch (ClassNotFoundException e) {
            reportEmmaError("Is emma jar on classpath?", e);
        } catch (SecurityException e) {
            reportEmmaError(e);
        } catch (NoSuchMethodException e) {
            reportEmmaError(e);
        } catch (IllegalArgumentException e) {
            reportEmmaError(e);
        } catch (IllegalAccessException e) {
            reportEmmaError(e);
        } catch (InvocationTargetException e) {
            reportEmmaError(e);
        }
    }

    private String getCoverageFilePath() {
        if (mCoverageFilePath == null) {
            return DEFAULT_COVERAGE_FILE_PATH;
        } else {
            return mCoverageFilePath;
        }
    }

    private void reportEmmaError(Exception e) {
        reportEmmaError("", e);
    }

    private void reportEmmaError(String hint, Exception e) {
        String msg = "Failed to generate emma coverage. " + hint;
        Log.e(TAG, msg, e);
        mResults.putString(Instrumentation.REPORT_KEY_STREAMRESULT, "\nError: "
                + msg);
    }

    /* (non-Javadoc)
     * @see com.example.instrumentation.FinishListener#onActivityFinished()
     */
    @Override
    public void onActivityFinished() {
        if (LOGD)
            Log.d(TAG, "onActivityFinished()");
        if (mCoverage) {
            generateCoverageReport();
        }
        finish(Activity.RESULT_OK, mResults);
    }

}

We are also implementing the FinishListener interface, which is defined as


package com.example.instrumentation;

/**
 * Listen for an Activity to finish and invokes {@link #onActivityFinished()} when this happens.
 * 
 * @author diego
 *
 */
public interface FinishListener {

        /**
         * Invoked when the Activity finishes.
         */
        void onActivityFinished();

}

Running the instrumented application
Once we have the EmmaInstrumentation class in place we need a few more adjustments to be able to get the coverage report of the running application.
Firstly, we need to add the new Activity to the manifest. Secondly, we should allow our application to write to the sdcard if this is where we decided to generate the coverage report. To do it you should grant the android.permission.WRITE_EXTERNAL_STORAGE permission.
Then, it's time to build and install the instrumented apk:

$ ant clean
$ ant instrument
$ ant installi

Everything is ready to start the instrumented application

$ adb shell am instrument -e coverage true \
     -w com.example.i2at.tc/\
        com.example.instrumentation.EmmaInstrumentation

If everything went well, the Temperature Converter application will be running and we can use it for a while


when we exit by pressing the BACK button we can see that the coverage data was written to the file and reflected in the logcat

I/System.out(2453): EMMA: runtime coverage data written to [/mnt/sdcard/coverage.ec] {in 975 ms}

this file can then be moved to the host computer using adb pull.

Hope this helps you obtaining the code coverage for your application to help you understand its usage patterns. As always, comments and questions are always welcome.

18 comments:

Canvas said...
This comment has been removed by the author.
Gagan said...

Android tools come with ant build script to get the code coverage... it should be really simple!!

http://code-gotcha.blogspot.com/2011/11/android-code-coverage.html

Canvas said...

Hi diego,
I followed the instruction against my project above but it could not generate the coverage file and reported error below. Could you give some advice? I also tried the sample project calculator which worked well. My app has a Splash activity(Just a picture) before the Main Activity start (in another word, Splash Activity is “android.intent.category.LAUNCHER”, Splash Activity will start Main Activity after display a picture a few seconds and then it will call finish() )
"error report in logcat":
"Crash of app com.tencent.mtt running instrumentation ComponentInfo{com.tencent.mtt/com.example.instrumentation.EmmaInstrumentation}


BTW, there is a typo in your post, "adb clean, adb instrument, adb installi", it should be "ant clean, ant isntrument, ant installi"

Diego Torres Milano said...

Hi Canvas,
Thanks for spotting the typo. Fixed now.
Regarding your problem, can you post the code somewhere (the minimum version that shows the problem) ?

Canvas said...

Diego,please check my app on github,
https://github.com/canvasding/calculatorWithSplashActivity
Thanks in advance!

Vincent said...

Hello,

I am trying to get your project to work but I am facing some problems:
A coverage.ec file is never generated. It seems like the process is killed immediately before that.
Logcat:
D/EmmaInstrumentation( 736): onActivityFinished()
I/ActivityManager( 79): Force stopping package com.example.i2at.tc uid=10040
I/ActivityManager( 79): Killing proc 736:com.example.i2at.tc/10040: force stop

I have been trying to figure it out but couldn't find a solution.
I would be thankful for any advice or idea.

Jose Ángel Zamora said...
This comment has been removed by the author.
Jose Ángel Zamora said...

Hi,

When I run the ant in order to get the .ec for an Android Test Project, after that when I tried to import the .ec into the eclipse, I get the message "class X appears to be instrumented already" and the class is related to the Android Project (not Android Project Test).

Has someone any idea?

Thanks.
Best regards

Diego Torres Milano said...

The example presented here is to instrument a running application itself with EMMA, not its test project.

You should follow the steps described in the presentation Introduction to Android Testing.

Hope this helps.

peter said...

Diego any chance to see this example in Maven based structure?
So far I had no luck getting it working with Emma or Cobertura

MAK said...

i followed exactly same steps to instrument another App, but got this error:

INSTRUMENTATION_STATUS: id=ActivityManagerService
INSTRUMENTATION_STATUS: Error=Unable to find instrumentation info for: ComponentInfo{com.example.android.musicplayer/com.example.instrumentation.EmmaInstrumentation}
INSTRUMENTATION_STATUS_CODE: -1
android.util.AndroidException: INSTRUMENTATION_FAILED: com.example.android.musicplayer/com.example.instrumentation.EmmaInstrumentation

Can you please let me know where i am screwing this up?

btw, i did include emma.jar in my class path of init.rc..

Marek said...

Disclaimer: I'm an Atlassian.

Great blog post, however I must admit that it's quite a huge hack :-) I believe you should try this one:

https://confluence.atlassian.com/display/CLOVER/Clover-for-Android

Cheers
Marek

Unknown said...

Thanks for the review! Android apps development.I agree with you regarding the limitations of the built-in tools when it comes to time tracking.

Android Application

2009 DREAM said...

Your computer is rattling instructive and your articles are wonderful.
go to the website

EffantTerrible said...

Thank you for your posting!
It was very useful to me.

I always appreciate your help.

and..I have one question.

My question is that is there a way to generate code coverage start signal?

It means that after ant clean, ant instrument, ant installi...

I should type command that "adb shell am instrument -e coverage true -w [project name]/[project activity]".


But, If I want to get code coverage then, Every time I have to use that command.

But, I want to execute like that command when my test application execute.

So, When I just execute the test-project in my real-device.(touch the app icon) Then, Code coverage will start!...like this...

Is there a way?(no using command)

Eric Tseng (豪豪豪豪) said...

Hi Diego,
Thanks for your sharing. I've followed the steps you posted. But there's one error.

INSTRUMENTATION_RESULT: shortMsg=java.lang.RuntimeException
INSTRUMENTATION_RESULT: longMsg=java.lang.RuntimeException: Unable to resolve activity for: Intent { flg=0x10000000 cmp=com.example/.test.EmmaInstrumentation$InstrumentedActivity }
INSTRUMENTATION_CODE: 0

Would you help share the AndroidManifest.xml content of EmmaInstrumentation project?

It's my AndroidManifest.xml content from now.

















Thanks.
Eric

Eric Tseng (豪豪豪豪) said...

Hi Diego,
Thanks for your sharing. I've followed the steps you posted. But there's one error.

INSTRUMENTATION_RESULT: shortMsg=java.lang.RuntimeException
INSTRUMENTATION_RESULT: longMsg=java.lang.RuntimeException: Unable to resolve activity for: Intent { flg=0x10000000 cmp=com.example/.test.EmmaInstrumentation$InstrumentedActivity }
INSTRUMENTATION_CODE: 0

Would you help share the AndroidManifest.xml content of EmmaInstrumentation project?

It's my AndroidManifest.xml content from now.

















Thanks.
Eric

Unknown said...

I am sure you have a great fan following out there.Learn About it