AirSig Gesture and Signature Recognition SDK for Daydream (繁中|简中)

Import AirSig Assets

  • From standalone unitypackage: In Unity, go to Assets > Import Package > Custom Package then select AirSig_Daydream_[ver].unitypackage file and import all items.
  • From the Asset Store: In Unity, open Asset Store and search for AirSig Daydream and click "Download" then "Import".

How to Use in Unity

Before using the AirSig package, you must install Daydream SDK first. Download it from Here and import it (Assets > Import Package > Custom Package > Select unitypackage you just downloaded).

Once Daydream SDK and AirSig are both imported, you need to:

  1. In Unity, set following settings:
    1. Switch Platform to Android
    2. Set Minimium SDK version to API 24
    3. Add 'Daydream' to Virtual Reality Supported
    4. Set your package name
    5. Add your keystore
  2. Copy 'res' folder from Assets/AirSig/Plugins/Android/res to Assets/Plugins/Android/
  3. In Unity, select Assets/AirSig/Plugins/Android/libs/ARMv7/libAirSig and change CPU to 'ARMv7' in the inspector.
  4. In Unity, select Assets/AirSig/Plugins/Android/libs/x86/libAirSig and change CPU to 'x86' in the inspector (Sometimes these settings just went off. It's necessary to update them to avoid duplicated library error).

Before you start using AirSig function, you have to call AirSigManger's SetMode API to decide which function you want to use when user triggers from daydream controller. Setting to Mode.None means turning off AirSig function which is default. The basic lifecycle of using a AirSig function is:

  1. Call SetMode to indicate which function is going to use
  2. Player triggers data input from daydream controller
  3. Get result from callback

The plugin automatically collects data when player hold down the trigger (Can be defined using API or Unity Editor) and identify when player release.

Use cases

AirSigManger provides 4 major functions:

  1. Identify player's identity by verifying against Player's Signatures with adjustable signature strength.
  2. Identify input gestures against Developer-defined Gestures. The feature is
    • Developers customize and define gestures. Players don't have to train.
    • With high generality. Even different players with different drawing habit(e.g. different size, shape), system can recongnize it.
    • In order to use this feature, gestures sample must be collected first and then generate a gesture profile in Here, or developers can use the gesture profile generated by others.
  3. Learn and identify Player-defined Gestures. The feature is
    • Players can create their own gestures in application real-time.
    • The training progress is needed. Players have to repeat the same gesture for about 5 times.
    • This kind of gesture focus on the drawing feature of individual. Less likely to be shareable between players.
  4. Smart Gesture algorithm that combines the result of Developer-defined Gestures and Player-defined Gestures (2 and 3). The feature is
    • Developers define the gesture and design a "Tutorial phase" in application. Players have to repeat the gesture and system learns the drawing habit in the same time. The accuracy is further improved.
    • In "Tutorial phase", if the players' drawing habit is very different from the gestures defined by developers, system will not learn it. That can avoid the wrong gesture affects the recognition result.
    • We suggest developers use this mode in most scenario. Players usually don't familiar with using gestures. The "Tutorial phase" helps players practice it and system can learn the drawing habit in the same time.

How to use the demo scene

  1. Open demo from Assets/AirSig/Demo/Scene
  2. Connect your daydream device and then File > Build and Run

API References

Classes and Types

Mode - indicates which AirSig function are going to use when player triggered the controller input

  • None: Do nothing

  • TrainPlayerSignature: use the input data to set player signature (multiple input data are required to complete the training)

  • IdentifyPlayerSignature: use the input data to identify player signature

  • DeveloperDefined: use the input data to identify against Developer-defined Gestures. The gestures are calculated by Here with developers collecting their own gesture samples. Using this API, developer will be able to provide infinite amount of gestures for improving experience of their products.

  • AddPlayerGesture: use the input data to collect Player-defined Gesture at runtime. Notice that data is only collected into a cache, until SetPlayerGesture(index) method is called which insert data into the engine.

  • IdentifyPlayerGesture: use the input data to identify Player-defined Gesture that set into the engine. This function identifies whether a gesture is one of given targerts rather than telling is or is not a gesture.

    For example, using this function to identify a gesture against one target - index 1, will almost always return match of index 1. If two targets - index 1 and 2 are set, will return match of either one.

  • SmartTrainDeveloperDefined: use the input data to identify against the Developer-defined Gesture and learn Player-defined Gesture at the same time. A excellent place to apply is like tutorial or practice session.

  • SmartIdentifyDeveloperDefined: use the input data to verify against both the Developer-defined Gesture and learnt Player-defined Gesture. The result is more accurate than just using one.

Methods

public void SetMode(Mode mode);

set which AirSig function is going to use when player triggered the controller input. Please reference Mode above to see a list of available modes.

public void SetTarget(List<int> target);

  • Player's Signatures and Player-defined Gestures share the same storage space. Please use the index to indicate which storage slot to access. target is a List of indexs.
  • For TrainPlayerSignature mode, set candidate index to train. Valid value are 1 - 1000.
  • For IdentifyPlayerSignature mode, set identification candidates to identify aganist. Valid value are 1 - 1000.
  • For AddPlayerGesture mode, set candidate index to add to. Valid values are 1 - 1000.
  • For IdentifyPlayerGesture mode, set identifcation candidate index to verify aganist. Valid values are 1-1000.
  • For DeveloperDefined, SmartTrainDeveloperDefined, SmartIdentifyDeveloperDefined mode, please use SetDeveloperDefinedTarget instead.

public void SetDeveloperDefinedTarget(List<string> target);

This API is specific for DevloperDefined mode. This tells the engine which candidate to perform against to. These targets are gesture name that was used in the collection app.

public void SetClassifier(string classifier, string subClassifier)

This API is specific for DevloperDefined mode. This tells the engine which classifier (database name) to use. Currently, the subClassifier is not used and should be set to empty string.

public Dictionary <int, int> SetPlayerGesture(List<int> targets);

Call this method to complete the AddPlayerGesture process. The method will set gestures at the target index to the engine and return how many gestures are set for each index.

public void ResetSmartTrain();

Reset smart training cache data

public void DeletePlayerRecord(int targetIndex);

Delete player signature or gesture

public float[] GetFromCache(long gestureId)

Get the gesture data given a gestureId (gestureId is available for all callback function). Only latest 10 gestures are stored.

public bool IsPlayerGestureExisted(AndroidJavaObject data)

Check whether a Player-defined Gesture data exist in the database or not. Gesture data can be obtain from GetFromCache.

TriggerStartButton

Set the trigger key to start collecting gesture. If None is used, then no gesture will be collected and no identification will be triggered. The default is Touchpad and other acceptable settings are:

  • TriggerButton.None
  • TriggerButton.Touchpad,
  • TriggerButton.App

TriggerEndButton

Set the trigger key to stop collecting gesture. If None is used, then releasing the TriggerStartButton will be used to end the collecting. The default is None and other acceptable settings are:

  • TriggerButton.None
  • TriggerButton.Touchpad,
  • TriggerButton.App

UseTouchTrigger

Set use Touch or Press to trigger. Note only Touchpad can support Touch.

Example:

  1. Use Touchpad Press to start collecting and releasing to end collecting.

    TriggerStartButtton = TriggerButton.Touchpad;
    UseTouchTrigger = false;
    
  2. Use Touchpad Press to start collecting and Touchpad Press again to end collecting.

    TriggerStartButtton = TriggerButton.Touchpad;
    UseTouchTrigger = false;
    
    TriggerEndButtton = TriggerButton.Touchpad;
    UseTouchTrigger = false;
    

Event Delegates

public delegate void OnGestureTriggered(long gestureId, GestureTriggerEventArgs eventArgs);

Called when player triggered a gesture input, and this callback is called before processing the AirSig function.

  • gestureId: This id is associated with the input gesture data, you can use this to know this result callback is for which input.

public delegate void OnDeveloperDefinedMatch(long gestureId, string gesture, float score);

Called when received an DeveloperDefined result.

  • gestureId: This id is associated with the input gesture data, you can use this to know this result callback is for which input.
  • gesture: The best match from candidates (the target list), see more in SetDeveloperDefinedTarget(List<string> target).
  • score: The match score of the best match, higher means higher confidence level, you can make a threshold to decide match or not. This value depends quality of samples collected during the collection phase. You could try out the value and adjust a best acceptable value. In normal situation, higher than 1 means match.

public delegate void OnPlayerSignatureTrained(long gestureId, Error error, float progress, SecurityLevel securityLevel);

Called when the training progress result is sent.

  • gestureId: This id is associated with the input gesture data, you can use this to know this result is for which input.
  • error: error object with its error code as
    • Error Code and Meaning
      • 0: no error
      • -1: not found or invalid
      • -2: invalid license
      • -3: invalid action index
      • -201: in signature setting or validation, the signature is too short as false input
      • -202: in signature setting or validation, the signature is too long
      • -203: in signature setting or validation, player used wrong holding posture to hold the controller to write, it would make player hard to perform the same writing
      • -204: in signature setting or validation, the signature is too short to meet security criteria
      • -205: in signature setting or validation, player's wrist is too stiff when writing, it would make player hard to perform the same writing
      • -207: in signature setting, the input signature sample is too different than previous
      • -208: the signature setting is failed because of the input signature samples don't have enough similarity, should start over the training again
      • -209: in signature setting, the 2nd input signature sample is too different than the first one, should start over the training again
      • -301: in signature verification, already tried too many times and failed, the verification is temporally blocked
  • progress - indicates the training progress. It usually takes 3 - 7 cycles to complete the training.
  • securityLevel - indicates how difficult is the signature to duplicate by others according by the complexity of the signature higher value means higher security level, means more complex the signature is

public delegate void OnPlayerGestureAdd(long gestureId, Dictionary<int, int> count);

Called when player triggers a gesture and it's stored in the cache. You must explicitly call SetPlayerGesture(index) in order to flush cache data to database.

  • gestureId: This id is associated with the input gesture data, you can use this to know this result is for which input.
  • count: a map of index id and cache count of gesture of this index.

public delegate void OnPlayerGestureMatch(long gestureId, int match);

Called when a gesture is matched against with custom gesture.

  • gestureId - This id associated with the input gesture data, you can use this to know this result is for which input.
  • match - index that match, or -1 if controller data is invalid (for example, not moving)

public delegate void OnPlayerSignatureMatch(long gestureId, bool match, int targetIndex);

Called when received an IdentifyPlayerSignature result.

  • gestureId: This id is associated with the input gesture data, you can use this to know this result callback is for which input.
  • match: The result is matched or not.
  • targetIndex: The id of the identity that been verified.

public delegate void OnSmartIdentifyDeveloperDefinedMatch(long gestureId, string gesture)

Called when received a SmartIdentifyDeveloperDefinedMatch result.

  • gestureId: This id is associated with the input gesture data, you can use this to know this result callback is for which input.
  • gesture: the result matched gesture name.

Sample Code

Train Player Signature

Sample code that demostrates how to train signature and handle result (For completed example, see GestureHandle.cs):

// Use Unity inspector to drag AirSigManager reference here
public AirSigManager airsigManager;

// Define callback for listening training progress
AirSigManager.OnPlayerSignatureTrained signatureTrained;

void HandleOnPlayerSignatureTrained(
    long gestureId, AirSigManager.Error error, float progress, 
    AirSigManager.SecurityLevel securityLevel) {
    // Handle training result
    if(1.0f >= progress)
        // training complete
    else
        // more gesture data is required
}

void Awake () {
    // 1. Use SetMode to configure AirSig function
    airsigManager.SetMode(
        AirSigManager.Mode.TrainPlayerSignature);

    // 2. Use SetTarget to configure index for train
    // or identification function
    airsigManager.SetTarget(100);

    // 3. Register callback for identification result
    signatureTrained = new AirSigManager.OnPlayerSignatureTrained(
        HandleOnPlayerSignatureTrained);
    AirSigManager.onPlayerSignatureTrained += signatureTrained;
}

Use Developer-defined Gestures

Gesture profile must be generated first in Here.

  1. Once the gesture profile is generated, place it under Assets/Plugins/Android/res/raw/ folder.
  2. Use SetClassifier to set the gesture profile name which is the name of gesture profile you downloaded from List All Gesture Profiles, and this value is case sensitive.
  3. Use SetDeveloperDefinedTarget to set the gesture type name that we want to perform identification against. Please set the gesture type names which compose the gesture profile. The gesutre type name is case sensitive, please make sure the values you set are same as List All Gesture Types.
  4. Use SetMode to set Mode.DeveloperDefined.
  5. Register Event Delegate to obtain the identification result.

Sample code that demostrates how to use developer predefined gesture and handle matched result using gesture profile in the SDK package (For completed example, see GestureHandle.cs):

// Use Unity inspector to drag AirSigManager reference here
public AirSigManager airsigManager;

// Define callback for listening Developer-defined Gesture match event
AirSigManager.OnDeveloperDefinedMatch
    developerGesture;

// Callback method that will handle the event
void HandleOnDeveloperDefinedMatch(
    long gestureId, string gesture, float score) {
    // handle match or fail to match
}

void Awake () {
    // 1. Use SetMode to configure AirSig function
    airsigManager.SetMode(
        AirSigManager.Mode.DeveloperDefined);

    // 2. Set classifier and sub classifier
    airsigManager.SetClassifier(
        "sample_gesture_profile", "");

    // 3. Use SetDeveloperDefinedTarget to configure
    // targets for identification function
    airsigManager.SetDeveloperDefinedTarget(
        new List {
            "C", 
            "HEART", 
            "DOWN" }
    );

    // 4. Register callback for identification
    // result
    developerGesture =
    new AirSigManager.OnDeveloperDefinedMatch(
        HandleOnDeveloperDefinedMatch);
    airsigManager.onDeveloperDefinedMatch +=
        developerGesture;
}