Android Cts Manual R5

User Manual:

Open the PDF directly: View PDF .
Page Count: 17

Download
Open PDF In Browser	View PDF

Compatibility Test Suite (CTS) Framework User
Manual

Android 4.1 CTS r1
Open Handset Alliance

Google Confidential

Contents

1. Why be compatible?
2. How can I become compatible?
2.1. Comply with Android Compatibility Definition document
2.2. Pass the Compatibility Test Suite (CTS)
2.3. Submit report
3. How does the CTS work?
3.1. Workflow
3.2. Types of test cases
3.3. Areas Covered
4. Setting up and using the CTS
4.1. Configuring the CTS
4.2. Setting up your device
4.3. Using the CTS
4.4. Selecting CTS Plans
5. Interpreting the Test Results
6. Release Notes
6.1. General
6.2. Known Framework issues
6.3. Known Test issues
7. Appendix: CTS Console Command Reference

Google Confidential

1. Why be compatible?

1. Give your users the best possible experience with the applications they run.

When a device is compatible with Android, users can choose from among many
high-quality applications. Applications that take full advantage of Android's
features are likely to perform best on compatible devices.
2.

Make it easy for developers to write top-quality applications for your device.
Developers want to streamline their applications for Android, and this is easiest for
them when they are writing for a predictable platform.
3.

Take advantage of the Android Market.
Compatible handsets can give users access to the Android Market.
Android compatibility is free, and it's easy.

Google Confidential

2. How can I become compatible?

2.1. Comply with Android Compatibility Definition document
To start, read the Android compatibility definition for the Android platform version
that you want. This document enumerates the software and the hardware features
in a compatible Android device. Except where noted, the features are all required
for Android compliance. To learn more about Android compatibility definition in
general, and to locate and download a particular definitions document, see the current
Compatibility Definition. Archived versions of older Compatibility Definitions may be
found on the Downloads page.

2.2. Pass the Compatibility Test Suite (CTS)
The Compatibility Test Suite (CTS) is a downloadable open-source testing harness that
you can use in any way you like as you develop your handset; for example, you could
use the CTS to do continuous self-testing during your development work. For more
about the CTS and the compatibility report that it generates, see the Compatibility Test
Suite page. For instructions on using the CTS, see the CTS User Guide.

2.3. Submit report
When you are ready to claim compatibility for your device, you can submit the CTSgenerated report to cts@android.com. When you submit a CTS report, you can also
request access to the Android Market.
* This is an early preview of CTS. The compatibility site and the service to certify your
compatibility reports are work in progress - we will update you when these are ready.

Google Confidential

3. How does the CTS work?

The CTS is an automated testing harness that includes two major software
components:
● The CTS test harness runs on your desktop machine and manages test execution.
● Individual test cases are executed on attached mobile devices or on an emulator.
The test cases are written in Java as JUnit tests and packaged as Android .apk
files to run on the actual device target.
3.1. Workflow
1. Use the bundled CTS release or download the CTS from the Android Open

Source Project onto your desktop machine.
2. Install and configure the CTS.
3. Attach at least one device (or emulator) to your machine.
4. Launch the CTS. The CTS test harness loads the test plan onto the attached

devices. For each test in the test harness:
○ The test harness pushes a .apk file to each device, executes the test
through instrumentation, and records test results.
○ The test harness removes the .apk file from each device.

Google Confidential

5. Once all the tests are executed, you can view the test results in your browser and

use the results to adjust your design. You can continue to run the CTS throughout
your development process.
When you are ready, you can submit the report generated by the CTS to
cts@android.com. The report is a .zip archived file that contains XML results and
supplemental information such as screen captures.

3.2. Types of test cases
The CTS includes the following types of test cases:
● Unit tests test atomic units of code within the Android platform; e.g. a single class,
such as java.util.HashMap.
● Functional tests test a combination of APIs together in a higher-level use-case.
● Reference application tests instrument a complete sample application to exercise
a full set of APIs and Android runtime services
Future versions of the CTS will include the following types of test cases:
● Robustness tests test the durability of the system under stress.
● Performance tests test the performance of the system against defined
benchmarks, for example rendering frames per second.

3.3. Areas Covered
The unit test cases cover the following areas to ensure compatibility
Area

Description

Signature tests

For each Android release, there are XML files describing the
signatures of all public APIs contained in the release. The CTS
contains a utility to check those API signatures against the APIs
available on the device. The results from signature checking are
recorded in the test result XML file.

Platform API Tests

Test the platform (core libraries and Android Application
Framework) APIs as documented in the SDK Class Index to
ensure API correctness:
● correct class, attribute and method signatures
● correct method behavior
● negative tests to ensure expected behavior for incorrect
parameter handling

Dalvik VM Tests

The tests focus on testing the Dalvik VM

Google Confidential

Platform Data Model The CTS tests the core platform data model as exposed
to application developers through content providers, as
documented in the SDK android.provider package:
● contacts
● browser
● settings
● more...
Platform Intents

The CTS tests the core platform intents, as documented in the
SDK Available Intents.

Platform
Permissions

The CTS tests the core platform permissions, as documented in
the SDK Available Permissions.

Platform Resources The CTS tests for correct handling of the core platform resource
types, as documented in the SDK Available Resource Types.
This includes tests for:
● simple values
● drawables
● nine-patch
● animations
● layouts
● styles and themes
● loading alternate resources

Google Confidential

4. Setting up and using the CTS
4.1. Configuring the CTS
Note: the steps to configure and run CTS have changed in the 4.0 release.
To run CTS, make sure you have a recent copy of adb installed on your machine.
To install adb, download and install the Android SDK tools. Then add the SDK Platform
tools component.
Ensure 'adb' is in your system path.
Example:
export PATH=$PATH:/home/myuser/android-sdk-linux_x86/
platform-tools

4.2. Setting up your device
CTS can be executed only on consumer devices.
This section is important as not following these instructions will lead to test timeouts/
failures:
1. Your phone should be running a user build (Android 4.0 and later) from
source.android.com
2. Please refer to this link on the Android developer site and set up your device
accordingly.
3. Make sure that your device has been flashed with a user build (Android 4.0 and
later) before you run CTS.
4. You need to ensure the Text To Speech files are installed on the device. You can
check via Settings > Speech synthesis > Install voice data before running CTS
tests. (Note that this assumes you have Android Market installed on the device, if
not you will need to install the files manually via adb)
5. Make sure the device has a SD card plugged in and the card is empty. Warning:
CTS may modify/erase data on the SD card plugged in to the device.
6. Do a factory data reset on the device (Settings > SD Card & phone storage >
Factory data reset). Warning: This will erase all user data from the phone.
7. Make sure no lock pattern is set on the device (Settings > Security > Screen Lock
should be 'None').
8. Make sure the "USB Debugging" development option is checked (Settings >
Developer options > USB debugging).

Google Confidential

9. Make
10. Make
11. Make
12. Make

sure Settings > Developer options > Stay Awake is checked
sure Settings > Developer options > Allow mock locations is checked
sure device is connected to a functioning Wi-Fi network (Settings > Wi-Fi)
sure the device is at the home screen at the start of CTS (Press the home
button).
13. While a device is running tests, it must not be used for any other tasks.
14. Do not press any keys on the device while CTS is running. Pressing keys or
touching the screen of a test device will interfere with the running tests and may
lead to test failures.
15. Set up accessibility tests:
1. adb install android-cts/repository/testcases/
CtsDelegatingAccessibilityService.apk

2. On the device, enable Settings > Accessibility > Delegating Accessibility

Service
16. Set up device administration tests:
1. adb install android-cts/repository/testcases/CtsDeviceAdmin.apk
2. On the device, enable Settings > Security > Device Administrators >

android.deviceadmin.cts.CtsDeviceAdmin* settings
17. Copy the CTS media files to your device using ADB with the instructions below:

The CTS media stress tests require video clips to be on external storage (/sdcard).
Most of the clips are from Big Buck Bunny which is copyrighted by the Blender
Foundation ( http://www.bigbuckbunny.org) under the Creative Commons Attribution
3.0 license. The required space depends on the maximum video playback resolution
supported by the device. By default, 176x144 and 480x360 SHOULD be supported.
Note that the video playback capabilities of the device under test will be checked via
the android.media.CamcorderProfile APIs.
Here are the storage requirements by maximum video playback resolution:
● 480x360: 91.4MB
● 720x480: 151.9MB
● 1280x720: 401.6MB
● 920x1080: 1008.2MB
Instructions to copy the media files to a device:
1. Download the android-cts-media-X.Y.zip file from http://source.android.com/
compatibility/downloads.html and unzip it.
2. Connect the device to the computer and check that adb can connect to it.
3. cd to the unzipped folder.
4. chmod 544 copy_media.sh
5. Run copy_media.sh
○ To copy clips for just the default resolutions, run ./copy_media.sh.

Google Confidential

To copy clips up to a resolution of 720x480, run ./copy_media.sh 720x480.
If you are not sure about the maximum resolution, try 1920x1080 so that all
files are copied.
○ If there are multiple devices under adb, add -s serial option to the end.
For example, to copy up to 720x480 to device with serial 1234567, run
copy_media.sh 720x480 -s 1234567.
○
○

4.3. Using the CTS
To run a test plan:
1. Make sure you have at least one device connected. Launch the CTS console by

running the cts-tradefed script e.g.
$ ./android-cts/tools/cts-tradefed
2. You may start the default test plan (containing all of the test packages) by
typing run cts --plan CTS. This will kick off all the CTS tests required for
compatibility.
Type list plans to see a list of test plans in the repository.Type list packages to see
a list of test packages in the repository.
See the CTS command reference or type help for a complete list of supported
commands.
3. Alternately, you can just run a CTS plan from the command line using ctstradefed run cts --plan 
4. You should test progress and results reported on the console.

4.4. Selecting CTS Plans
For this release the following 7 test plans are available.
1. CTS - contains all tests and will run ~17,000 tests on your device. These tests are

required for compatibility. At this point performance tests are not part of this plan
(this will change for future CTS releases).
2. Signature - contains the signature verification of all public APIs
3. Android - contains tests for the android APIs
4. Java - contains tests for the Java core library
5. VM - contains tests for the Dalvik virtual machine
6. RefApp - contains reference application tests (more coming in future CTS
release)
7. Performance - contains performance tests for your implementation (more
coming in future CTS releases)
These can be executed with the run cts command as mentioned earlier.

Google Confidential

Google Confidential

5. Interpreting the Test Results
The test results are placed in the file:
$CTS_ROOT/repository/results/.zip
Inside the zip, the testResult.xml file contains the actual results -- open this file in
any web browser (Firefox 3.x recommended) to view the test results.

The 'device information' section provides details about the device and the firmware
(make, model, firmware build, platform) and the hardware on the device (screen
resolution, keypad, screen type).
The details of the executed test plan are present in the 'test summary' section which
provides the CTS plan name and execution start and end times. It also presents an
aggregate summary of the number of tests that passed, failed, time out or could not be
executed.
The next section also provides a summary of tests passed per package.

Google Confidential

This is followed by details of the the actual tests that were executed. The report lists the
test package, test suite, test case and the executed tests. It shows the result of the test
execution - pass, fail, timed out or not executed. In the event of a test failure details are
provided to help diagnose the cause. Further, the stack trace of the failure is available
in the XML file but is not included in the report to ensure brevity - viewing the XML file
with a text editor should provide details of the test failure (search for the  tag
corresponding to the failed test and look within it for the  tag).

Google Confidential

6. Release Notes
6.1. General
Note the CTS test harness has changed signficantly in the Android 4.0 release.
Some new features have been added included support for sharding a CTS test
run onto multiple concurrent devices, as well as general faster performance.
● This CTS release contains approximately 17,000 tests that you can execute on
the device.
● Please make sure all steps in section 4.2 "Setting up your device" have been
followed before you kick off CTS. Not following these instructions may cause tests
to timeout or fail.
●

6.2. Known Issues
●

Google Confidential

7. Appendix: CTS Console Command Reference

Host
help

Display a summary of the most
commonly used commands.

help all

display the complete list of available
commands

exit

Gracefully exit the CTS console. Console
will close when all currently running tests
are finished

Run

Google Confidential

run cts

Run the specified tests and displays
progress information. One of --plan, -package, --class or --continue-session-id
needs to be specified.
The CTS console can accept other
commands while tests are in progress.
If no devices are connected, the CTS
host will wait for a device to be connected
before starting tests.
If more than one device is connected,
CTS host will choose a device
automatically.

--plan 
Run the specified test plan
-- package/-p  [-package/-p ...]
--class/-c  [--method/-m


Runs the specified test packages.

Runs the specified test class and/or
method

--continue-session-id

--shards 

--serial/-s 

Runs all not executed tests from
previous CTS session. The sessions
testResult.xml will be updated with the
new results.

Shard a CTS run into given number of
independent chunks, to run on multiple
devices in parallel.
Run CTS on the specific device

List

Google Confidential

list packages

List all available test packages in the
repository.

list plans

Lists all available test plans in the
repository

list invocations

Lists 'run' commands currently being
executed on devices.

list commands

List all 'run' commands currently in the
queue waiting to be assigned to devices

list results

List CTS results currently stored in
repository

list devices

List currently connected devices and their
state.
'Available' devices are functioning, idle
devices, available for running tests.
'Unavailable' devices are devices visible
via adb, but are not responding to adb
commands and won't be allocated for
tests.
'Allocated' devices are devices currently
running tests.

Add
add derivedplan --plan 
--result/-r
[pass | fail | timeout | notExecuted]
[--session/-s ]

Google Confidential

Create a plan derived from given result
session.



---

Source Exif Data:

File Type                       : PDF
File Type Extension             : pdf
MIME Type                       : application/pdf
PDF Version                     : 1.4
Linearized                      : No
Format                          : application/pdf
Producer                        : iText 2.1.6 by 1T3XT
Create Date                     : 2012:07:10 21:32:00-07:00
Modify Date                     : 2012:07:10 21:32:00-07:00
Creator Tool                    : Documill Publishor 6.3.12 by Documill (http://www.documill.com/)
Page Count                      : 17
Creator                         : Documill Publishor 6.3.12 by Documill (http://www.documill.com/)

EXIF Metadata provided by EXIF.tools