AirSig Gesture and Signature Recognition SDK for Vive (繁中|简中)

Import AirSig Assets

  • From standalone unitypackage: In Unity, go to Assets > Import Package > Custom Package then select AirSig_[ver].unitypackage file and import all items.
  • From Asset Store: In Unity, open Asset Store and search for AirSig Vive and click "Download" then "Import".

Environment Setup

Make sure your environment has ".Net Framework 4.6.1" and "Microsoft Visual C++ 2017 Redistributable (x64)" installed. You can find the installation packages in our decompressed "Extrenal_Library" folder or download them from Microsoft web site.

How to Use in Unity

Before using the AirSig package, you must install SteamVR first. Open Asset Store in Unity and search for SteamVR then download/import it.

Once SteamVR and AirSig are both imported, you need to:

  1. Drag AirSigManager prefebs (under Assets/AirSig/prefebs/) into your Scene.
  2. Copy everything from Assets/AirSig/StreamingAssets to your Assets/StreamingAssets

You can check the video of installation process.

Before you start using AirSig function, you have to call AirSigManger's SetMode API to indicate which function you want to use when player triggers gestures. Setting to Mode.None means turning off AirSig function which is default. The basic lifecycle of using a AirSig function is:

  1. Call SetMode to indicate which function is going to use.
  2. Player triggers data input from the vive controller.
  3. Get result from callback.

The plugin automatically collects data when player hold down the trigger (Can be defined using API or Unity Editor) and identify when player release.

Use Cases

AirSigManger provides 4 major functions:

  1. Identify player's identity by verifying against Player's Signatures with adjustable signature strength.
  2. Identify input gestures against Developer-defined Gestures. The feature is
    • Developers customize and define gestures. Players don't have to train.
    • With high generality. Even different players with different drawing habit(e.g. different size, shape), system can recongnize it.
    • In order to use this feature, gestures sample must be collected first and then generate a gesture profile in Here, or developers can use the gesture profile generated by others.
  3. Learn and identify Player-defined Gestures. The feature is
    • Players can create their own gestures in application real-time.
    • The training progress is needed. Players have to repeat the same gesture for about 5 times.
    • This kind of gesture focus on the drawing feature of individual. Less likely to be shareable between players.
  4. Smart Gesture algorithm that combines the result of Developer-defined Gestures and Player-defined Gestures (2 and 3). The feature is
    • Developers define the gesture and design a "Tutorial phase" in application. Players have to repeat the gesture and system learns the drawing habit in the same time. The accuracy is further improved.
    • In "Tutorial phase", if the players' drawing habit is very different from the gestures defined by developers, system will not learn it. That can avoid the wrong gesture affects the recognition result.
    • We suggest developers use this mode in most scenario. Players usually don't familiar with using gestures. The "Tutorial phase" helps players practice it and system can learn the drawing habit in the same time.

How to use the demo scene

  1. There are 4 Demo Scene in Assets/AirSig/Demo Scene/:
    • PlayerSignature: Demo the funciton of Player's Signatures,the source code is AirSig/Script/Demo/PlayerSignature.cs.
    • DeveloperDefined: Demo the funciton of Developer-defined Gestures,the source code is AirSig/Script/Demo/DeveloperDefined.cs.
    • PlayerGesture: Demo the funciton of Player-defined Gestures,the source code is AirSig/Script/Demo/PlayerGesture.cs.
    • SmartGesture: Demo the funciton of Smart Gesture,the source code is AirSig/Script/Demo/SmartGesture.cs.
  2. Copy everything under Assets/AirSig/StreamingAssets/ to Assets/SteamingAssets/
  3. Select one of the Demo Scenes, and hit the play button.

You can check the video of Demo Scene.

API References

Classes and Types

Mode - indicates which AirSig function are going to use when player triggered the controller input

  • None: Do nothing

  • TrainPlayerSignature: use the input data to set player signature (multiple input data are required to complete the training)

  • IdentifyPlayerSignature: use the input data to identify player signature.

  • DeveloperDefined: use the input data to identify against Developer-defined Gestures. The gestures are calculated by Here with developers collecting their own gesture samples. Using this API, developer will be able to provide infinite amount of gestures for improving experience of their products.

  • AddPlayerGesture: use the input data to collect Player-defined Gesture at runtime. Notice that data is only collected into a cache, until SetPlayerGesture(index) method is called which insert data into the engine.

  • IdentifyPlayerGesture: use the input data to identify Player-defined Gesture that set into the engine. This function identifies whether a gesture is one of given targerts rather than telling is or is not a gesture.

    For example, using this function to identify a gesture against one target - index 1, will almost always return match of index 1. If two targets - index 1 and 2 are set, will return match of either one.

  • SmartTrainDeveloperDefined: use the input data to identify against the Developer-defined Gesture and learn Player-defined Gesture at the same time. A excellent place to apply is like tutorial or practice session.

  • SmartIdentifyDeveloperDefined: use the input data to verify against both the Developer-defined Gesture and learnt Player-defined Gesture. The result is more accurate than just using one.

Methods

public void SetMode(Mode mode);

set which AirSig function is going to use when player triggered the controller input. Please reference Mode above to see a list of available modes.

public void SetTarget(List<int> target);

  • Player's Signatures and Player-defined Gestures share the same storage space. Please use the index to indicate which storage slot to access. target is a List of indexs.
  • For TrainPlayerSignature mode, set candidate index to train. Valid value are 1 - 1000.
  • For IdentifyPlayerSignature mode, set identification candidates to identify aganist. Valid value are 1 - 1000.
  • For AddPlayerGesture mode, set candidate index to add to. Valid values are 1 - 1000.
  • For IdentifyPlayerGesture mode, set identifcation candidate index to verify aganist. Valid values are 1-1000.
  • For DeveloperDefined, SmartTrainDeveloperDefined, SmartIdentifyDeveloperDefined mode, please use SetDeveloperDefinedTarget instead.

public void SetDeveloperDefinedTarget(List<string> target);

This API is specific for DevloperDefined mode. This tells the engine which candidate to perform against to. These targets are gesture name that was used in the collection app.

public void SetClassifier(string classifier, string subClassifier)

This API is specific for DevloperDefined mode. This tells the engine which classifier (database name) to use. Currently, the subClassifier is not used and should be set to empty string.

public Dictionary<int, int> SetPlayerGesture(List<int> targets);

Call this method to complete the AddPlayerGesture process. The method will set gestures at the target index to the engine and return how many gestures are set for each index.

public void ResetSmartTrain();

Reset smart training cache data.

public void DeletePlayerRecord(int targetIndex);

Delete player signature or gesture.

public float[] GetFromCache(long gestureId)

Get the gesture data given a gestureId (gestureId is available for all callback function). Only latest 10 gestures are stored.

public bool IsPlayerGestureExisted(float[] data)

Check whether a Player-defined Gesture data exist in the database or not. Gesture data can be obtain from GetFromCache.

public void SetTriggerStartKeys(Controller ctrl, ulong buttonMask, PressOrTouch pressOrTouch)

Set how to trigger a gesture or signature from the controller. If the start key is not specified, no gesture will be collected and no function will be performed. The SDK set the default to the trigger key.

  • ctrl: Controller ID. It can be either Controller.RIGHT_HAND or Controller.LEFT_HAND.
  • buttonMask: one of value in SteamVR_Controller.ButtonMask class. For example, SteamVR_Controller.ButtonMask.Trigger or SteamVR_Controller.ButtonMask.Touchpad.
  • pressOrTouch: It can be either PressOrTouch.PRESS or PressOrTouch.Touch.

public void SetTriggerEndKeys(Controller ctrl, ulong buttonMask, PressOrTouch pressOrTouch)

Set how to stop collecting a gesture or signature from the controller. Note that it is possible not specifying the end key. If the end key is not specified, a gesture is collected immediately when player release the start key.

  • ctrl: Controller ID. It can be either Controller.RIGHT_HAND or Controller.LEFT_HAND.
  • buttonMask: one of value in SteamVR_Controller.ButtonMask class. For example, SteamVR_Controller.ButtonMask.Trigger or SteamVR_Controller.ButtonMask.Touchpad.
  • pressOrTouch: It can be either PressOrTouch.PRESS or PressOrTouch.Touch. Note not all keys support touch mode.

Example:

  1. Use Touchpad press to trigger gesture collecting and stop when release the touchpad.

    SetTriggerStartKeys(
        Controller.RIGHT_HAND,
        SteamVR_Controller.ButtonMask.Touchpad,
        PressOrTouch.PRESS);
    
  2. Use Touchpad press to trigger gesture collecting and stop when press the trigger key.

    SetTriggerStartKeys(
        Controller.RIGHT_HAND,
        SteamVR_Controller.ButtonMask.Touchpad,
        PressOrTouch.PRESS);
    
    SetTriggerEndKeys(
        Controller.RIGHT_HAND,
        SteamVR_Controller.ButtonMask.Trigger,
        PressOrTouch.PRESS);
    

public void ClearTriggerStartKeys(Controller ctrl)

Clear the start key for the specified controller.

  • ctrl: Controller ID. It can be either Controller.RIGHT_HAND or Controller.LEFT_HAND.

public void ClearTriggerEndKeys(Controller ctrl)

Clear the end key for the specified controller.

  • ctrl: Controller ID. It can be either Controller.RIGHT_HAND or Controller.LEFT_HAND.

Event Delegates

public delegate void OnGestureTriggered(long gestureId, GestureTriggerEventArgs eventArgs);

Called when player triggered a gesture input, and this callback is called before processing the AirSig function.

  • gestureId: This id is associated with the input gesture data, you can use this to know this result callback is for which input.

public delegate void OnDeveloperDefinedMatch(long gestureId, string gesture, float score);

Called when received an DeveloperDefined result.

  • gestureId: This id is associated with the input gesture data, you can use this to know this result callback is for which input.
  • gesture: The best match from candidates (the target list), see more in SetDeveloperDefinedTarget(List<string> target).
  • score: The match score of the best match, higher means higher confidence level, you can make a threshold to decide match or not. This value depends quality of samples collected during the collection phase. You could try out the value and adjust a best acceptable value. In normal situation, higher than 1 means match.

public delegate void OnPlayerSignatureTrained(long gestureId, Error error, float progress, SecurityLevel securityLevel);

Called when the training progress result is sent.

  • gestureId: This id is associated with the input gesture data, you can use this to know this result is for which input.
  • error: error object with its error code as
    • Error Code and Meaning
      • 0: no error.
      • -1: not found or invalid.
      • -2: invalid license.
      • -3: invalid action index.
      • -201: in signature setting or validation, the signature is too short as false input.
      • -202: in signature setting or validation, the signature is too long.
      • -203: in signature setting or validation, player used wrong holding posture to hold the controller to write, it would make player hard to perform the same writing.
      • -204: in signature setting or validation, the signature is too short to meet security criteria.
      • -205: in signature setting or validation, player's wrist is too stiff when writing, it would make player hard to perform the same writing.
      • -207: in signature setting, the input signature sample is too different than previous.
      • -208: the signature setting is failed because of the input signature samples don't have enough similarity, should start over the training again.
      • -209: in signature setting, the 2nd input signature sample is too different than the first one, should start over the training again.
      • -301: in signature verification, already tried too many times and failed, the verification is temporally blocked.
  • progress - indicates the training progress. It usually takes 3 - 7 cycles to complete the training.
  • securityLevel - indicates how difficult is the signature to duplicate by others according by the complexity of the signature higher value means higher security level, means more complex the signature is.

public delegate void OnPlayerGestureAdd(long gestureId, Dictionary<int, int> count);

Called when player triggers a gesture and it's stored in the cache. You must explicitly call SetPlayerGesture(index) in order to flush cache data to database.

  • gestureId: This id is associated with the input gesture data, you can use this to know this result is for which input.
  • count: a map of index id and cache count of gesture of this index.

public delegate void OnPlayerGestureMatch(long gestureId, int match);

Called when a gesture is matched against with custom gesture.

  • gestureId - This id associated with the input gesture data, you can use this to know this result is for which input.
  • match - index that match, or -1 if controller data is invalid (for example, not moving)

public delegate void OnPlayerSignatureMatch(long gestureId, bool match, int targetIndex);

Called when received an IdentifyPlayerSignature result.

  • gestureId: This id is associated with the input gesture data, you can use this to know this result callback is for which input.
  • match: The result is matched or not.
  • targetIndex: The id of the identity that been verified.

public delegate void OnSmartIdentifyDeveloperDefinedMatch(long gestureId, string gesture)

Called when received a SmartIdentifyDeveloperDefinedMatch result.

  • gestureId: This id is associated with the input gesture data, you can use this to know this result callback is for which input.
  • gesture: the result matched gesture name.

Sample Code

Train Player Signature

Sample code that demostrates how to train signature and handle result (For completed example, see PlaySignature.cs):

// Use Unity inspector to drag AirSigManager reference here
public AirSigManager airsigManager;

// Define callback for listening training progress
AirSigManager.OnPlayerSignatureTrained signatureTrained;

void HandleOnPlayerSignatureTrained(
    long gestureId, AirSigManager.Error error, float progress, 
    AirSigManager.SecurityLevel securityLevel) {
    // Handle training result
    if(1.0f >= progress)
        // training complete
    else
        // more gesture data is required
}

void Awake () {
    // 1. Use SetMode to configure AirSig function
    airsigManager.SetMode(
        AirSigManager.Mode.TrainPlayerSignature);

    // 2. Use SetTarget to configure index for train
    // or identification function
    airsigManager.SetTarget(100);

    // 3. Register callback for identification result
    signatureTrained = new AirSigManager.OnPlayerSignatureTrained(
        HandleOnPlayerSignatureTrained);
    AirSigManager.onPlayerSignatureTrained += signatureTrained;
}

Use Developer-defined Gestures

Gesture profile must be generated first in Here.

  1. Once the gesture profile is generated, place it under Assets/StreamingAssets folder.
  2. Use SetClassifier to set the gesture profile name.
  3. Use SetDeveloperDefinedTarget to set the gesture name that we want to perform identification against.
  4. Use SetMode to set Mode.DeveloperDefined.
  5. Register Event Delegate to obtain the identification result.

Sample code that demostrates how to use developer predefined gesture and handle matched result using gesture profile in the SDK package (For completed example, see DeveloperDefined.cs):

// Use Unity inspector to drag AirSigManager reference here
public AirSigManager airsigManager;

// Define callback for listening Developer-defined Gesture match event
AirSigManager.OnDeveloperDefinedMatch
    developerGesture;

// Callback method that will handle the event
void HandleOnDeveloperDefinedMatch(
    long gestureId, string gesture, float score) {
    // handle match or fail to match
}

void Awake () {
    // 1. Use SetMode to configure AirSig function
    airsigManager.SetMode(
        AirSigManager.Mode.DeveloperDefined);

    // 2. Set classifier and sub classifier
    airsigManager.SetClassifier(
        "sample_gesture_profile", "");

    // 3. Use SetDeveloperDefinedTarget to configure
    // targets for identification function
    airsigManager.SetDeveloperDefinedTarget(
        new List {
            "C", 
            "HEART", 
            "DOWN" }
    );

    // 4. Register callback for identification
    // result
    developerGesture =
    new AirSigManager.OnDeveloperDefinedMatch(
        HandleOnDeveloperDefinedMatch);
    airsigManager.onDeveloperDefinedMatch +=
        developerGesture;
}