Connecting the Leap Motion controller to Mathematica Using JLink

Ideas
, , , , , ,
1

So the Leap Motion controller has just been released and I thought I might as well post some code that Todd Gayley and I wrote to connect the Leap Motion and all of its functionality to Mathematica!

Here are the setup instructions:

  1. (If you haven’t already) download and install the leap motion
    software from leapmotion.com/setup

  2. Download the leap motion SDK off their developer page.

  3. Connect the Leap Motion to your computer.

  4. Find the file “LeapJava.jar” inside of the Leap SDK (It should be under lib.)

  5. Open Mathematica and paste this in:

Needs["JLink`"];
ReinstallJava[CommandLine -> "java", 
  JVMArguments -> 
   "-Djava.library.path=[path to directory CONTAINING LeapJava.jar]"];

Now put the path to “LeapJava.jar” in the indicated position, and remember this is the path to the directory containing LeapJava.jar, not the path to LeapJava.jar itself and run the code.

  1. Now paste this in:
AddToClassPath["[path to LeapJava.jar]"];

Replace the indicated area with the path to LeapJava.jar itself and run it.

  1. To setup the controller, run this:
controller = JavaNew["com.leapmotion.leap.Controller"]

You should now be fully connected and ready to go! To see if it is working you can run:

Methods[controller]

and it should return a list of all the methods under the controller.

The main method you want to look at is frame[] which contains all of the information about what is going on.

To see the methods under frame[] run this:

Methods[controller@frame[]]

From here you should be able to figure things out like this:

controller@frame[]@fingers@count[]

which should count the visible fingers in the scene.

Also in a final note I want mention that I was having problems accessing the fingers with this:

controller@frame[]@finger[finger number]

and so you may want to try:

controller@frame[]@fingers[]@get[finger number]

instead. The same goes for palms.

Well, if you write anything cool using this please post it back up here along with any questions you have.

(I am going to try to post some of the code I have written with this soon :smiley: )

10 Likes
Connection to Sphero robot from Mathematica
2

Trying this out on OS X, Mm 9.0.1.0, I get the following error when setting up the controller:

Java::excptn: A Java exception occurred: java.lang.ExceptionInInitializerError
       at com.leapmotion.leap.Controller.(Controller.java:39)
       at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
       at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
       at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
       at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
Caused by: java.lang.SecurityException:
           J/Link does not allow code called from Mathematica to call System.exit().

at java.lang.Runtime.exit(Runtime.java:88)
       at java.lang.System.exit(System.java:921)
       at com.leapmotion.leap.LeapJNI.(LeapJNI.java:18)
       … 11 more.

JavaNew::fail: “Error calling constructor for class \!\(\“com.leapmotion.leap.Controller\”\).”

I presume this is a permissions issue, but I haven’t been able to figure out what file or files permissions need to be changed and to what. I’m running Mathematica from an account with Admin privleges.

Any suggestions?

Thanks,
Mike

3

Are you sure that the file path in either of the earlier lines were right? And that you didn’t move any files around inside the leap SDK including files other than LeapJava.jar? Also you might want to check that you installed the program from the LeapMotion setup page and (I am not sure this is really necessary) have it running.

4

I double checked the file paths - first reference is to directory and second is to the .jar file itself. I’m pretty sure I didn’t move any of the files around inside the SDK folder - which is one level deep on my desktop. I’m running the Leap app and it connects automatically with other Leap programs. I can successfully compile and run the Leap java Sample app from the terminal, so all seems well there and working nominally.

Thanks for the suggestions. I’ll keep poking around and see if I can figure out what my problem is.

5

Edit: Found the solution!
On Windows (x64), LeapJava.dll from LeapDeveloperKit\LeapSDK\lib\x64 needs to be in the same directory as your LeapJava.jar. This should let you run JavaNew according to the instructions in the original post.

------ Disregard below ----
Loading the Java class instead of JavaNew worked for me:

controller = LoadJavaClass["com.leapmotion.leap.Controller"];

Instead of:

controller = JavaNew["com.leapmotion.leap.Controller"]

But this only lets you view the methods, and does not create an object of the class.

1 Like
6

For me this works fine on Windows (x64), no copying around of dll’s:

Needs["JLink`"];
With[{
(* the location of the developer kit : *)
lmlib = "C:\\leapmotion\\LeapDeveloperKit_release_win_1.0.8+7665\\LeapDeveloperKit\\LeapSDK\\lib"
},
(* for 64 bit Windows : *)
AddToClassPath[lmlib, FileNameJoin[{lmlib, "x64"}]]
];
(* now start Java *)
InstallJava[];
(* and things work *)
controller = JavaNew["com.leapmotion.leap.Controller"];
Dynamic[controller[frame[][fingers[][count[]]]], UpdateInterval -> 0.1]
1 Like
7

A small if crude example of data from Leap Motion, when the controller is initialized like in this thread:

getFrameData =
  Module[{getFrame, getFingerList, getVector, makeRule, fun},
   getFrame[controller_] := controller@frame[];
   getFingerList[frame_] :=
    With[{f = frame@fingers[]},
     Table[f@get[finger], {finger, 0, f@count[] - 1}]];
   getVector[v_] := {v@getX[], v@getY[], v@getZ[]};
   makeRule[finger_, map_, method_] := method -> map[finger@method[]];
   fun[controller_] :=
    With[{fr = getFrame[controller]},
     {makeRule[fr, #/1000000. &, timestamp],
      fingers ->
       ({makeRule[#, Identity, length],
           makeRule[#, getVector, direction],
           makeRule[#, getVector, tipPosition],
           makeRule[#, getVector, tipVelocity],
           makeRule[#, getVector, stabilizedTipPosition]} & /@ 
         getFingerList[fr])}];
   fun];

This function returns some of data available in the API ( https://developer.leapmotion.com/documentation/Languages/Java/API/annotated.html ) in Mathematica-style manner:

getFrameData[controller]

(*
{timestamp -> 15370., 
 fingers -> {{length -> 65.1096, 
    direction -> {0.0140035, -0.33028, -0.943779}, 
    tipPosition -> {-24.4047, 179.213, -115.658}, 
    tipVelocity -> {-0.199876, 0.767241, 4.08685}, 
    stabilizedTipPosition -> {-26.4247, 178.62, -116.503}}, 
    ...,
    {length -> 41.934, 
    direction -> {0.71685, -0.00299873, -0.697221}, 
    tipPosition -> {58.0986, 148.038, -0.762505}, 
    tipVelocity -> {2.57099, -0.732685, 3.28637}, 
    stabilizedTipPosition -> {57.0482, 145.986, -4.76263}}}}
*)

Starting point for visualization of this data might be this, showing fingertips (correcting coordinate directions makes it a bit messy oneliner):

Graphics3D[
  Dynamic[Sphere[{1, -1, 1} #[[{1, 3, 2}]], 15] & /@
    ((tipPosition /. #) & /@ (fingers /. getFrameData[controller])), UpdateInterval -> 0.05],
  PlotRange -> {{-150, 150}, {-150, 150}, {0, 300}}]

7469fingertips.png
7469fingertips.png360×391 11.8 KB

1 Like
8

How could I improve the swipe detection robustness? Should I change the frame rate? Or is there some other parameter I should change?
This at least does only work sometimes:

 (* this is the directory where LeapSDK is *)
    $leapdir =
          FileNameJoin[{$UserBaseDirectory, "Applications",
            "LeapDeveloperKit"}];
        If[FileExistsQ[$leapdir], Needs["JLink`"];
      Block[{nativelibs},
       With[{(*the location of the developer kit:*)
         lmlib = FileNameJoin[{$leapdir, "LeapSDK", "lib"}]},
            nativelibs =
             FileNameJoin[{lmlib,
               Switch[$SystemID, "Windows-x86-64", "x64", "Windows", "x86"]}];
        (*now start Java*)
        JLink`ReinstallJava[
         JLink`ClassPath -> FileNameJoin[{lmlib, "LeapJava.jar"}],
         JLink`JVMArguments ->
          "-Djava.library.path=\"" <> nativelibs <> "\""]]]];

    controller = JavaNew["com.leapmotion.leap.Controller"];
    LoadJavaClass["com.leapmotion.leap.Gesture$Type"];
        If[Not[controller@isGestureEnabled[Gesture$Type`TYPEUSWIPE]],
      controller@enableGesture[Gesture$Type`TYPEUSWIPE]
          ];
        controller@isGestureEnabled[Gesture$Type`TYPEUSWIPE];
    gesturePrint[controller_] := Module[{gesture},
      gesture = controller@frame[]@gestures[]@get[0];
      If[(gesture@type[]@name[]) === "TYPE_SWIPE",
       swipe = JavaNew["com.leapmotion.leap.SwipeGesture", gesture];

       {swipe@state[]@toString[],
        (swipe@position[]@#) & /@ {getX[], getY[], getZ[]},
        (swipe@direction[]@#) & /@ {getX[], getY[], getZ[]},
        swipe@speed[]
        }]]
    ud = .05;
    Dynamic[gesturePrint[controller], UpdateInterval -> 0.1]
1 Like
9

Hello Christopher,
We had a quick discussion about Leap Motion at the Computer Based Math Conference. I think you have some more code examples of Leap Motion… would you mind sharing the examples? Thanks, Craig