How it works

With Motion Tracking alone, the device tracks its movement and orientation through 3D space and tells you where it is and which way it’s facing, but it retains no memory of what it sees. Area Learning gives the device the ability to see and remember the key visual features of a physical space—the edges, corners, other unique features—so it can recognize that area again later. To do this, it stores a mathematical description of the visual features it has identified inside a searchable index on the device. This allows the device to quickly match what it currently sees against what it has seen before without any cloud services.

When a Tango device has learned an area, there are two key things it can do to improve upon the information provided by Motion Tracking alone:

1. Improve the accuracy of the trajectory by performing "drift corrections."

2. Orient and position itself within a previously learned area by performing "localization."

Improving the trajectory

As mentioned on the Motion Tracking Overview page, motion estimates become less accurate over time. The device corrects for some errors by orienting itself to gravity, but errors in other aspects of its pose cannot be detected through Motion Tracking alone.

With Area Learning turned on, the Tango device remembers the visual features of the area it has visited and uses them to correct errors in its understanding of its position, orientation, and movement. This memory allows the system to perform drift corrections (also called loop closures). When the device sees a place it knows it has seen earlier in your session, it realizes it has traveled in a loop and adjusts its path to be more consistent with its previous observations. These corrections can be used to adjust the device‘s position and trajectory within your application.

The illustration below shows an example of drift correction. As you begin walking through an area, there are actually two different trajectories occurring simultaneously—the path you are walking (the "real trajectory") and the path the device estimates that you are walking (the "estimated trajectory"). The green line is the real trajectory that the device is traveling; the red line shows how, over time, the estimated trajectory has drifted away from the real trajectory. When the device returns to the origin and realizes it has seen the origin before, it corrects the drift errors and adjusts the estimated trajectory to better match the real trajectory.

Without drift correction, a game or application using a virtual 3D space aligned with the real world may encounter inaccuracies in Motion Tracking after extended use. For example, if a door in a game world corresponds with a door frame in the real world, drift errors can cause the game door to appear in the middle of the real-world wall instead of in the door frame.

Area descriptions and localization

After you have walked through an area with Area Learning turned on, you can save what the device has seen in an Area Description File (ADF). Learning an area and loading it as an ADF has a number of advantages; for example, you can use it to intentionally align the device‘s coordinate frame with a pre-existing coordinate frame so that content in a game or app always appears in the same physical location.

There are two ways to create an ADF. You can use any application that can save area descriptions, including Tango Area Learning sample projects (see more information about sample projects for C, Java, or Unity). Your second choice is to use the Tango APIs to handle the learning, saving, and loading all within your application.

If you want to create a consistent experience within the same mapped space, such as having virtual objects appear in the same location as the last time the user visited an area, you must perform localization. This is a two-step process:

1. Load a previously saved ADF.

2. Move the device into the area that was saved in the ADF.

When the device "sees" that it is in the area covered by the ADF, it instantly knows where it is relative to the origin in the file (that is, the point where original learning started in the saved area)—this is localization. Without localizing to an area description, a device‘s starting point is lost every time you end the session.

Usability tips

• Tango devices depend on the visual diversity of the area to localize. If you are in an area with many identical rooms or in a completely empty room with blank walls, it is difficult to localize.

• An environment can look quite different from different angles and positions, and can change over time (furniture can be moved around, lighting will be different depending on the time of day). Localization is more likely to succeed if the conditions at the time you localize are similar to the conditions that existed when the ADF was created.

• Because environments can and do change, you might create multiple ADFs for a single physical location under different conditions. This gives your users the option to select a file that most closely matches their current conditions. You could also append multiple sessions onto the same ADF to capture visual descriptions of the environment from every position and angle and under every variation of lighting or environmental change.

Our UX Best Practices page has additional tips on creating ADFs and using Area Learning.

Common use cases

Multi-player experiences: Two or more users in the same physical location share an ADF through a cloud service and then localize to the same coordinate frame. This allows multiple people to interact in the same physical space where all of their relative positions are known. The Tango APIs do not natively support data sharing in the cloud, but you can implement this through Google Cloud Storage and the Google Play Games API.

Location-aware shopping or other activities: A retail store manager makes an ADF of their store and then makes the ADF publicly available. Customers load the ADF, localize, and then use the device to navigate directly to products they are interested in.

Important: While saved area descriptions do not directly record images or video, the data can be reconstructed into viewable images. Therefore, to protect the user‘s privacy, you must ask for permission before saving any of their learned areas to the cloud or sharing areas between users just as you would with any other images or video.

Area Learning and using area descriptions are powerful features, and we’re excited to see how developers use them to offer new user experiences.

Using learning mode and loaded ADFs

The behavior of some aspects of the Tango APIs will vary depending on your settings for learning mode or whether you loaded an ADF.

In the table below, the two left columns specify whether you have learning mode on and whether you have loaded a previously stored ADF. You may or may not be able to save an ADF depending on the status of those two things. For example, if you don‘t have learning mode on, you cannot save an ADF. If you have learning mode on and have loaded an ADF, you can only save again after you have localized against the loaded ADF.

Also, if you aren‘t in learning mode and don‘t have an ADF loaded, you cannot get pose data using the TANGO_COORDINATE_FRAME_AREA_DESCRIPTION frame of reference. If you have an ADF loaded, you can get pose data from that frame of reference after the device localizes to the loaded ADF.

*If tracking is lost, these frame of reference pairs will no longer be available. After service reset, the session functions as if learning mode is True and an ADF was loaded, where the area descriptions are those that were learned up to the loss of tracking. To continue using the area description frame of reference, you must localize against what you learned before the loss of tracking. You must also localize to include what you learned before tracking was lost when saving an ADF.

Configuration

There are two configuration options that are relevant to area learning.

Loading area descriptions

To load an area description file, your TangoConfig must have KEY_STRING_AREADESCRIPTION set to the UUID of the ADF you wish to load. When the TangoService starts, it loads this area description.

try {
    mConfig.putString(TangoConfig.KEY_STRING_AREADESCRIPTION, uuid);
} catch (TangoErrorException e) {
    // handle exception
}

Here is an example of how you could load the latest saved area description file. It retrieves a list of all area description files and takes the last one in the list, which would be the most recent:

ArrayList<String> fullUUIDList = new ArrayList<String>();
// Returns a list of ADFs with their UUIDs
fullUUIDList = mTango.listAreaDescriptions();

// Load the latest ADF if ADFs are found.
if (fullUUIDList.size() > 0) {
     mConfig.putString(TangoConfig.KEY_STRING_AREADESCRIPTION,
                       fullUUIDList.get(fullUUIDList.size() - 1));
}

Learning mode

In order to use area learning, your TangoConfig must have KEY_BOOLEAN_LEARNINGMODE set to true. In the default TangoConfig, KEY_BOOLEAN_LEARNINGMODE is set to false.

try {
    TangoConfig mConfig = mTango.getConfig(TangoConfig.CONFIG_TYPE_CURRENT);
    mConfig.putBoolean(TangoConfig.KEY_BOOLEAN_LEARNINGMODE, true);
} catch (TangoErrorException e) {
    // handle exception
}

You can save an ADF by calling Tango.saveAreaDescription(). This call can be long-running, so don‘t call it on the UI thread.

Learning mode with a loaded area description

If both options are used, an area description of the current session will be built. If localization occurs against the loaded ADF and Tango.saveAreaDescription() is called, a new ADF will be saved containing the combination of both.

Note: You cannot save an area description in this way until you have localized against the loaded ADF. See Area Learning Mode and Loaded Area Description Files for more information.

Motion tracking with area learning

In general, using motion tracking with area learning is very similar to basic motion tracking. You need to define the coordinate pairs you are interested in, then use either callbacks or polling to get data.

There are two coordinate frame pairs for area learning. The TangoCoordinateFramePair

with baseFrame == TangoPoseData.COORDINATE_FRAME_AREA_DESCRIPTION and targetFrame == TangoPoseData.COORDINATE_FRAME_DEVICE gets you the pose of the device from the area description‘s origin, including drift corrections. If an ADF was loaded, pose data is only available after you have localized to the ADF and the origin will be the same as the ADF. If learning mode enabled without loading an ADF, pose data will be available immediately after initialization and the origin will be the same as the start of service.

The frame pair with baseFrame == TangoPoseData.COORDINATE_FRAME_AREA_DESCRIPTION and targetFrame == TangoPoseData.COORDINATE_FRAME_START_OF_SERVICE provides the offset between the area description and start of service at the requested time. This pair updates only when a localization occurs.

Applications should use TangoPoseData.COORDINATE_FRAME_DEVICE with respect toTangoPoseData.COORDINATE_FRAME_AREA_DESCRIPTION for tracking a device‘s motion, and use TangoPoseData.COORDINATE_FRAME_START_OF_SERVICE with respect to TangoPoseData.COORDINATE_FRAME_AREA_DESCRIPTION only as a localization signal.

@Override
public void onPoseAvailable(TangoPoseData pose) {

if (pose.baseFrame == TangoPoseData.COORDINATE_FRAME_AREA_DESCRIPTION
        && pose.targetFrame == TangoPoseData.COORDINATE_FRAME_DEVICE) {
    // Process new ADF to device pose data.
}
else if (pose.baseFrame == TangoPoseData.COORDINATE_FRAME_AREA_DESCRIPTION
        && pose.targetFrame == TangoPoseData.COORDINATE_FRAME_START_OF_SERVICE) {
    // Process new localization.
}

Working with ADFs

Using metadata

ADFs have metadata you can use, so you don‘t have to work with UUID alone. The metadata consists of key-value pairs; for a list of supported keys see the constant values for TangoAreaDescriptionMetaData.

Get the name field for the ADF identified by uuid:

public String getName(String uuid) {

    TangoAreaDescriptionMetaData metadata = new TangoAreaDescriptionMetaData();
    metadata = mTango.loadAreaDescriptionMetaData(uuid);
    byte[] nameBytes = metadata.get(TangoAreaDescriptionMetaData.KEY_NAME);
    if (nameBytes != null) {
        String name = new String(nameBytes);
        return name;
    } // Do something if null
}

Set the name field to name, for the ADF identified by uuid:

public void setName(String name, String uuid) {

    TangoAreaDescriptionMetaData metadata = new TangoAreaDescriptionMetaData();
    metadata = mTango.loadAreaDescriptionMetaData(uuid);
    metadata.set(TangoAreaDescriptionMetaData.KEY_NAME, name.getBytes());
    mTango.saveAreaDescriptionMetadata(uuid, metadata);
}

Listing all the UUIDs of currently accessible ADFs:

ArrayList<String> mFullUUIDList;
try {
    mFullUUIDList = mTango.listAreaDescriptions();
} catch(TangoErrorException e) {
    // handle the errors
}
if (mFullUUIDList.size() == 0) {
    // handle the fact that there‘s no ADFs stored
}