Using the Leap Motion controller with Adobe Air Desktop App

In this short post I am going to show you how you can use the Leap Motion controller with Adobe Air desktop applications.

As it turns out this is much easier that i originally anticipated thanks to two facts:

1. The leap software on the desktop has a little WebSocket server running on it so you can get data from the hardware.
2. Someone has already written a basic library to deal with the web socket data and return this as a flash event (https://github.com/slikland/LeapMotionAIR/).

I have created a simple example project which you can get from my Github repository here....
https://github.com/nicholasjackson/LeapAir


The first thing we need to do is to connect to the server to start receiving data, the below code creates a LeapSocket and wires up the events.  The main event we are interested in is the Data event, the leap motion periodically updates this with the current state of the device such as hand and finger positions.

    public function LeapTest() {

        var leapSocket:LeapSocket = new LeapSocket();
        leapSocket.addEventListener(Event.CONNECT, socketConnect);
        leapSocket.addEventListener(Event.CLOSE, close);
        leapSocket.addEventListener(IOErrorEvent.IO_ERROR, error);
        leapSocket.addEventListener(SecurityErrorEvent.SECURITY_ERROR, error);
        leapSocket.addEventListener(LeapEvent.DATA, leapData);

        leapSocket.connect("127.0.0.1", 6437);

        _hand = new Hand();
        _hand.visible = false;
        addChild(_hand);
    }

Also in this event we are going to create a simple sprite which is a white circle, this will mark the position of our hand.

Lets have a quick look at the leap data event.

    private function leapData(event:LeapEvent):void
    {
        // Returns a Frame Object
        if(event.data.hands.length > 0) {

            if(!_hand.visible)
                _hand.visible = true;

            var handPos = event.data.hands[0].palmPosition;
            var interactionBox = event.data.interactionBox;
            var pos:Point = leapToScene(interactionBox,handPos);

            _hand.x = pos.x;
            _hand.y = pos.y;

        } else {
            if(_hand.visible)
                _hand.visible = false;
        }


    }

The data payload of this event is of type object and corresponds to the Leap Javascript API for the Frame object (https://developer.leapmotion.com/documentation/cpp/api/Leap.Frame.html).

We are checking to see if there is currently a hand within the controllers view and getting the position of this hand which we then set the position of our Sprite to.  One thing to note is that the leap returns the position in Leap space not in the 2D space of your application.  The Javascript examples on the Leap developer site shows how you can convert this to view space.  The method leapToScene does this for us.

    private function leapToScene(interactionBox,leapPos):Point {
        var pos:Point = new Point();

        var left = interactionBox.center[0] - interactionBox.size[0]/2;
        var top  = interactionBox.center[1] + interactionBox.size[1]/2;
        var x = leapPos[0] - left;
        var y = leapPos[1] - top;

        x = x / interactionBox.size[0];
        y = y / interactionBox.size[0];

        x = x * 1280;
        y = y * 720;

        pos.x = x;
        pos.y = -y;

        trace("left:" + left + " top:" + top + " x:" + x + " y:" + y);

        return pos;
    }

For the speed of rapid prototyping I have just hard coded the height and width of my application here 1280x720 if this was working code I would get these from the application.

And that is it, if you run the app and place your hand within the Leaps view you will see the white circle move with the position of your hand.  It is possible to take this much further and translate these movements to a virtual mouse with click and gesture but that will have to wait for next time.

Have fun.