标签：
The Compatibility Test Suite (CTS) is a free, commercial-grade test suite, available for download. The CTS represents the "mechanism" of compatibility.
The CTS runs on a desktop machine and executes test cases directly on attached devices or an emulator. The CTS is a set of unit tests designed to be integrated into the daily workflow (such as via a continuous build system) of the engineers building a device. Its intent is to reveal incompatibilities early on, and ensure that the software remains compatible throughout the development process.
The CTS is an automated testing harness that includes two major software components:
The CTS tradefed test harness runs on your desktop machine and manages test execution.
Individual test cases are executed on the Device Under Test (DUT). The test cases are written in Java as JUnit tests and packaged as Android .apk files to run on the actual device target.
The Compatibility Test Suite Verifier (CTS Verifier) is a supplement to the CTS available for download. CTS Verifier provides tests for APIs and functions that cannot be tested on a stationary device without manual input (e.g. audio quality, accelerometer, etc).
The CTS Verifier is a tool for manual testing and includes the following software components:
The CTS verifier app that is executed on the DUT and collects the results.
The executable(s) or script(s) that are executed on the desktop machine to provide data or additional control for some test cases in the CTS Verifier app.
Figure 1. How to use CTS
This diagram summarizes CTS workflow. Please refer to the subpages of this section starting with Setup for detailed instructions.
The CTS includes the following types of test cases:
Unit tests test atomic units of code within the Android platform; e.g. a single class, such as java.util.HashMap.
Functional tests test a combination of APIs together in a higher-level use-case.
Future versions of the CTS will include the following types of test cases:
Robustness tests test the durability of the system under stress.
Performance tests test the performance of the system against defined benchmarks, for example rendering frames per second.
The unit test cases cover the following areas to ensure compatibility:
| Area | Description |
|---|---|
| Signature tests | For each Android release, there are XML files describing the signatures of all public APIs contained in the release. The CTS contains a utility to check those API signatures against the APIs available on the device. The results from signature checking are recorded in the test result XML file. |
| Platform API Tests | Test the platform (core libraries and Android Application Framework) APIs as documented in the SDK Class Index to ensure API correctness, including correct class, attribute and method signatures, correct method behavior, and negative tests to ensure expected behavior for incorrect parameter handling. |
| Dalvik Tests | The tests focus on testing the Dalvik Executable Format. |
| Platform Data Model | The CTS tests the core platform data model as exposed to application developers through content providers, as documented in the SDK android.provider package: contacts, browser, settings, etc. |
| Platform Intents | The CTS tests the core platform intents, as documented in the SDK Available Intents. |
| Platform Permissions | The CTS tests the core platform permissions, as documented in the SDK Available Permissions. |
| Platform Resources | The CTS tests for correct handling of the core platform resource types, as documented in the SDK Available Resource Types. This includes tests for: simple values, drawables, nine-patch, animations, layouts, styles and themes, and loading alternate resources. |
CTS tests require a Wi-Fi network that supports IPv6, can treat the Device Under Test (DUT) as an isolated client, and has an internet connection. An isolated client refers to a configuration where the DUT does not have visibility to the broadcast/multinetwork messages on that subnetwork, either by a Wi-Fi AP configuration or by running the DUT on an isolated sub-network without other devices being connected.
If you don‘t have access to a native IPv6 network, an IPv6 carrier network, or a VPN to pass some tests depending on IPv6, you may instead use a Wi-Fi access point and an IPv6 tunnel. See Wikipedia list of IPv6 tunnel brokers.
If the DUT supports the Bluetooth LE feature, then at least three Bluetooth LE beacons should be placed within five meters of the DUT for Bluetooth LE scan testing. Those beacons can be any kind, do not need to be configured or emit anything specific, and can include iBeacon, Eddystone, or even devices simulating BLE beacons.
CTS currently supports 64-bit Linux and Mac OS host machines.
Before running the CTS, make sure you have recent versions of both Android Debug Bridge (adb) and Android Asset Packaging Tool (AAPT) installed and those tools‘ location added to the system path of your machine.
To install ADB, download the Android SDK Tools package for your operating system, open it, and follow the instructions in the included README file. For troubleshooting information, see Installing the Stand-alone SDK Tools.
Ensure adb and aapt are in your system path. The following command assumes you‘ve opened the package archive in your home directory:
$ export PATH=$PATH:$HOME/android-sdk-linux/build-tools/<version>
Note: Please ensure your starting path and directory name are correct.
Install the proper version of the Java Development Kit (JDK). See the JDK requirements.
Download and open the CTS packages matching your devices‘ Android version and all the Application Binary Interfaces (ABIs) your devices support.
Download and open the latest version of the CTS Media Files.
Follow the step to set up your system to detect your device, such as creating a udev rules file for Ubuntu Linux.
A compatible device is defined as a device with a user/release-key signed build, so your device should be running a system image based on the known to be compatible user build (Android 4.0 and later) from Codenames, Tags, and Build Numbers.
Caution: When used to confirm Android compatibility of your final system image, CTS must be executed on devices with a user build.
The CTS media stress tests require video clips to be on external storage (/sdcard). Most of the clips are from Big Buck Bunny which is copyrighted by the Blender Foundation under the Creative Commons Attribution 3.0 license.
The required space depends on the maximum video playback resolution supported by the device (See section 5 in the compatibility definition document for the platform version of the required resolutions.) Note that the video playback capabilities of the device under test will be checked via the android.media.CamcorderProfile APIs for earlier versions of Android and the android.media.MediaCodecInfo.CodecCapabilities APIs from Android 5.0.
Here are the storage requirements by maximum video playback resolution:
Warning: CTS may modify/erase data on the SD card plugged into the device.
In order to run CTS carrier API tests, the device needs to has a SIM card with carrier privilege rules on it. SeePreparing the UICC.
Warning: This will erase all user data from the device.
Note: On Android 4.2 and later, Developer options is hidden by default. To make them available, go toSettings > About phone and tap Build number seven times. Return to the previous screen to find Developer options. See Enabling On-device Developer Options for additional details.
Note: This mock locations setting is applicable only in Android 5.x and 4.4.x.
Note: This verify apps step became required in Android 4.2.
Note: When you connect a device running Android 4.2.2 or later to your computer, the system shows a dialog asking whether to accept an RSA key that allows debugging through this computer. Select Allow USB debugging.
Note: For CTS versions 2.1 R2 through 4.2 R4, set up your device (or emulator) to run the accessibility tests with:adb install -r android-cts/repository/testcases/CtsDelegatingAccessibilityService.apk
On the device, enable: Settings > Accessibility > Accessibility > Delegating Accessibility Service
Note: For CTS 2.3 R4 through CTS 6.0 on devices that declare the android.software.device_admin feature, set up your device to run the device administration tests with:adb install -r android-cts/repository/testcases/CtsDeviceAdmin.apk
On the device, enable only the two android.deviceadmin.cts.CtsDeviceAdminReceiver* device administrators under: Settings > Security > Select device administrators. Make sure theandroid.deviceadmin.cts.CtsDeviceAdminDeactivatedReceiver and any other preloaded device administrators stay disabled in the same menu.
Note: For CTS 2.3 R12 and beyond, if the device supports video codecs, the CTS media files must be copied to the device.
chmod u+x copy_media.shcopy_media.sh:
./copy_media.sh 720x480./copy_media.sh all so that all files are copied../copy_media.sh 720x480 -s 1234567See the Trade Federation Overview for an explanation of the Trade Federation (tradefed or TF for short) continuous test framework.
To run a test plan:
$ ./android-cts/tools/cts-tradefedrun cts --plan CTS (or just run cts for v2). This kicks off all the CTS tests required for compatibility. Enter list plans to see a list of test plans in the repository. Enter list packages to see a list of test packages in the repository. See the CTS command reference or type help for a complete list of supported commands.cts-tradefed run cts --plan<plan_name>
Note: When running Android 6.0 (Marshmallow) CTS only, we recommend you use the --skip-preconditionsoption to skip the experimental pre-conditions feature that may cause issues for when executing CTS tests.
For Android 6.0 or earlier, you‘ll use CTS v1.
The following test plans are available:
These can be executed with the run cts command.
Table 1. This table summarizes the CTS v1 console commands for various uses.
| Host | Description |
|---|---|
help |
Display a summary of the most commonly used commands |
help all |
Display the complete list of available commands |
exit |
Gracefully exit the CTS console. Console will close when all currently running tests are finished |
| Run | Description |
run cts |
Run the specified tests and displays progress information. One of --plan, --package, --class or --continue-session needs to be specified
The CTS console can accept other commands while tests are in progress If no devices are connected, the CTS desktop machine (or host) will wait for a device to be connected before starting tests If more than one device is connected, the CTS host will choose a device automatically |
--plan <test_plan_name> |
Run the specified test plan |
--package/-p <test_package_name> [--package/-p <test_package2>...] |
Run the specified test packages |
--class/-c <class_name> [--method/-m <test_method_name> |
Run the specified test class and/or method |
--continue-session |
Run all not executed tests from previous CTS session; the sessions testResult.xml will be updated with the new results |
--shards <number_of_shards> |
Shard a CTS run into given number of independent chunks, to run on multiple devices in parallel |
--serial/-s <deviceID> |
Run CTS on the specific device |
-t <class_name>#<test_method_name> |
Run a specific test method |
--force-abi 32|64 |
On 64-bit devices, run the test against only the 32-bit or 64-bit ABI |
| List | Description |
list packages |
List all available test packages in the repository |
list plans |
List all available test plans in the repository |
list invocations |
List ‘run‘ commands currently being executed on devices |
list commands |
List all ‘run‘ commands currently in the queue waiting to be assigned to devices |
list results |
List CTS results currently stored in repository |
list devices |
List currently connected devices and their state
‘Available‘ devices are functioning, idle devices, available for running tests
‘Unavailable‘ devices are devices visible via adb, but are not responding to adb commands and won‘t be allocated for tests
‘Allocated‘ devices are devices currently running tests |
| Add | Description |
add derivedplan --plan <plan_name> |
Create a plan derived from given result session; use this option to rerun reports and validate test issues |
For Android 7.0 or later, you‘ll use CTS v2.
Available test plans include the following:
Other available configurations include the following:
All of these plans and configs can be executed with the run cts command.
Table 1. This table summarizes the CTS V2 console commands for various uses.
| Host | Description |
|---|---|
help |
Display a summary of the most commonly used commands |
help all |
Display the complete list of available commands |
version |
Show the version. |
exit |
Gracefully exit the CTS console. Console will close when all currently running tests are finished. |
| Run | Description |
run cts |
Run the default CTS plan (that is, the full CTS invocation). The CTS console can accept other commands while tests are in progress. If no devices are connected, the CTS desktop machine (or host) will wait for a device to be connected before starting tests. If more than one device is connected, the CTS host will choose a device automatically. |
--plan <test_plan_name> |
Run the specified test plan. |
-- module/-m <test_module_name> [--module/-m <test_module2>...] |
Run the specified test module or modules. For example, run cts --module CtsGestureTestCases executes the gesture test module (this can be shortened to run cts -m Gesture).run cts -m Gesture --test android.gesture.cts.GestureTest#testGetStrokes runs the specific package, class, or test. |
-- module/-m <test_module_name> -- test <test_name> |
Run the specified module and test. For example, run cts -m Gesture --test android.gesture.cts.GestureTest#testGetStrokes runs the specific package, class, or test. |
--retry |
Retry all tests that failed or were not executed from the previous sessions. Use list results to get the session id. |
--shards <number_of_shards> |
Shard a CTS run into given number of independent chunks, to run on multiple devices in parallel. |
--serial/-s <deviceID> |
Run CTS on the specific device. |
--include-filter <module_name> [--include-filter <module2>...] |
Run only with the specified modules. |
--exclude-filter <module_name> [--exclude-filter <module2>...] |
Exclude the specified modules from the run. |
--log-level-display/-l <log_level> |
Run with the minimum specified log level displayed to STDOUT. Valid values: [VERBOSE, DEBUG, INFO, WARN, ERROR, ASSERT]. |
--abi <abi_name> |
Force the test to run on the given ABI, 32 or 64. By default CTS runs a test once for each ABI the device supports. |
--logcat, --bugreport, and --screenshoot-on-failure |
Give more visibility into failures and can help with diagnostics. |
--device-token |
Specifies a given device has the given token eg. --device-token 1a2b3c4d:sim-card.. |
--skip-device-info |
Skips collection of information about the device. Note: do not use this option when running CTS for approval. |
--skip-preconditions |
Bypasses verification and setup of the device‘s configuration, such as pushing media files or checking for Wi-Fi connection. |
| List | Description |
list modules |
List all available test modules in the repository. |
list plans or list configs |
List all available test plans (configs) in the repository. |
list invocations |
List ‘run‘ commands currently being executed on devices. |
list commands |
List all ‘run‘ commands currently in the queue waiting to be assigned to devices. |
list results |
List CTS results currently stored in repository. |
list devices |
List currently connected devices and their state.
‘Available‘ devices are functioning, idle devices, available for running tests.
‘Unavailable‘ devices are devices visible via adb, but are not responding to adb commands and won‘t be allocated for tests.
‘Allocated‘ devices are devices currently running tests. |
| Dump | Description |
dump logs |
Dump the tradefed logs for all running invocations. |
The CTS test results are placed in the file:
$CTS_ROOT/android-cts/repository/results/<start_time>.zip
If you have built the CTS yourself, $CTS_ROOT will resemble the path out/host/linux-x86/cts but differ by platform. This reflects the path where you have uncompressed the prebuilt official CTS downloaded from this site.
Inside the zip, the testResult.xml file contains the actual results—open this file in any web browser (HTML5 compatible browser recommended) to view the test results. It will resemble the following screenshots.
Note: The results are provided to help you ensure the software remains compatible throughout the development process and act as a common format for communicating the compatibility status of your device with other parties.
Figure 1. CTS test summary
If testResult.xml displays a blank page when using the Chrome browser, change your browser configuration to enable the --allow-file-access-from-files command line flag.
The Device Information section provides details about the device, firmware (make, model, firmware build, platform), and device hardware (screen resolution, keypad, screen type). To access device information, click the link above Test Summary.
The Test Summary section provides executed test plan details, like the CTS plan name and execution start and end times. It also presents an aggregate summary of the number of tests that passed, failed, timed out, or could not be executed.
The next section also provides a summary of tests passed per package.
Figure 2. CTS test report
This is followed by details of the the actual tests that were executed. The report lists the test package, test suite, test case, and the executed tests. It shows the result of the test execution—pass, fail, timed out, or not executed. In the event of a test failure details are provided to help diagnose the cause.
Further, the stack trace of the failure is available in the XML file but is not included in the report to ensure brevity—viewing the XML file with a text editor should provide details of the test failure (search for the <Test> tag corresponding to the failed test and look within it for the <StackTrace> tag).
标签：
原文地址：http://www.cnblogs.com/CoderTian/p/5975991.html
