A LEAP of faith

Alright, now with that terrible pun out of the way, let’s actually start this post. Yesterday we had a tech workshop, or something like that. I don’t know if it even had a proper name. The point was to play around with gadgets and see how they can be used in practical ways. We formed groups, and everyone had to draw two little cards, one of which had a piece of technology on it, while the other was a client we had to “work” for.

We got the LEAP Motion. Which is this little thingy:


The best way I can describe it is it’s like a terrible vertical version of the already pretty bad Kinect. Except this thing only tracks your hands, not your whole body. And it somehow feels even MORE inaccurate than the Kinect, which is an accomplishment. Alright, to be fair, the LEAP had its moments, like when we tried a sculpting app that comes with it. That one worked surprisingly well on a basic level. Until I wanted to control the camera, at which point I was waving my hands in front of the screen like an idiot trying to make the view turn the way I wanted it to. I liked the whole sculpting idea and I actually made a fairly interesting glass object. (I might update this post with a picture if I can render the .obj file somehow.)

EDIT: I got it to work!


At least it’s almost up plug and play, unlike the Kinect, which the group next to us seemed to have some trouble setting up. You connect it to a computer, download some drivers (which are of course not the latest ones but we’ll get to that), and it’s set up! We spent some time trying out the default tutorial apps and had some fun with it, but it was more because of the novelty and less for how useful it is. The same thing happened with the Kinect: sure it had been fun to jump around and play with it for the first few weeks, but after that it just gathered dust on the shelf unless some friends came over and we wanted to play Dance Central, and even then it was really frustrating just how inaccurate the thing was at detecting your movements.

And the kicker is: we got “childhood obesity” as our “client”. So now we had to figure out how to make kids less fat, with the use of a gadget that encourages people to move and have a healthy lifestyle about as much as World of Warcraft or Counter-Strike. I mean, the thing has two modes, one of which is ‘desktop mode’, which requires you to sit (or if you really feel like burning some calories, stand) in front of your computer screen and do some slight hand movements in the air. The other is ‘VR mode’, which works with the Oculus Rift headset by sitting on the front of it, and thus incorporates neck movement too! That’s great and all, but still encourages kids to sit instead of moving around, and VR headsets are also REALLY expensive, and require a high-end PC to even make them work.

With no other real option, we were trying to figure out how to use this thing to engage kids and trick them into learning about healthy lifestyles. If you tell a kid not to eat so much candy/chocolate/junk food/whatever cause it will make them fat, they will probably just ignore you, no matter how many elaborate “before-after” pictures you found using Google Images. No, I don’t have scientific data to back that up, just personal experience, from which I’ve also learned that when you’re telling kids to “go out and play” they can most likely look forward to the most boring part of their day, unless you work hard to make that outside-time more appealing than whatever they were doing inside.

So naturally we turned to games, because those are fun, right?

Okay, I admit I kinda lost track of what we were doing at this point, but I’ll try to sum up the parts I remember:

  • Make it an arcade-machine sort of deal, so the kids playing it have to stand up to reach it. At least if they are playing our game they can’t sit down, which makes us jerks because that’s uncomfortable, but also good guys because hey, they are burning a few calories and stuff I guess.
  • Make it a cooking game with other distracting minigames to break up the monotony of doing just one task.
    • The cooking part should give the player a bunch of ingredients, and the healthier food they make out of them the more points they get. This can be done with multiple levels, each with different ingredients and challenges. One that came up was what kind of toppings to put on a pizza. Obviously leaderboards with friends and stuff can be implemented.
    • As for the minigames, I think basketball was the only one mentioned while I was focused on the discussion, but that’s one thing that definitely can be done with the LEAP. This could even have some VR implementation if we could make that work.

Here are some pictures from the poster. Since I was doing other stuff in the meanwhile, I don’t know what a lot of this stuff is, but I’ll put them here just in case.

whatsapp-image-2016-11-07-at-3-41-12-pm-2 whatsapp-image-2016-11-07-at-3-41-09-pm-1 whatsapp-image-2016-11-07-at-3-41-10-pm whatsapp-image-2016-11-07-at-3-41-11-pm whatsapp-image-2016-11-07-at-3-41-12-pm whatsapp-image-2016-11-07-at-3-41-12-pm-1

So what was I doing while all this productive brainstorming and drawing was going on? Well, I was busy trying to set up a prototype in Unity. There were only two slight problems with that plan:

  • I have never used Unity before.
  • I have never used the LEAP Motion SDK before.

Thankfully there was a Unity specific section of the LEAP site, which had some plugins I downloaded. Not so thankfully the documentation was split across a bunch of disconnected pages I had to find by searching on Google. Once I found the required pages for setting up the motion control with Unity, I went to work. By “work”, I mean “sit in front of the computer and wait for green bars to load”. Oh, and launching Unity was a multiple stage process of starting the damn thing at least three times, because whenever I progressed once small step, like logging in, or choosing a project, or whatever, it just got stuck in what I like to call Task Manager Limbo, which is when the program is running somewhere in the background but you can’t tell that without checking the Processes tab in the Task Manager.


Anyways, once I got everything running, I placed the required hand models in front of the camera object, and tried running it to see if it works. And then nothing happened. I went back to check the documentation, and reproduced every required step very carefully. Still nothing. After the third or fourth attempt I noticed that whenever I tried running the “game” with the play button on the top of the screen, a really small notification popped up on the Windows tray. Which is of course on the other side of the monitor, so I kinda didn’t notice the first few times. But now I did, and it said something about Unity trying to access “v2 functions” and that I should probably update my drivers. I was a bit dumbfounded by this, since I got the drivers from the official site, so I went into the LEAP main menu, and checked the version number. It was 2.something, I don’t remember the numbers exactly.

Well, off to Google again. After some hunting on forums I found the answer. As it turns out, when LEAP told me that Unity wanted to access v2 functionalities, it was actually talking about v3 stuff. Now I knew that there was a newer version, so I went to the LEAP site again, and sure enough, there was a newer version on there somewhere. Why the newest driver is not the default download is beyond me.

Finally, after about an hour of messing around, I was able to get the damn thing working on a very basic level. I could run the game and see my hands moving around. Here’s a rough recap of how the next 20 minutes went:

“Great, let’s just add that table I downloaded from the asset store for free, and put a ball on top of it which I can then pick up with my virtual hands! Oh wait, now both of those objects are just fixed in the air, uhhhh… how do I apply physics to an object?” – short googling intermission – “Alright, now I have gravity applied to both the table and the ball, let’s try this! Wait, why did the ball just fall through the desk?” – long intermission, during which I tried to set up a collision box for the table – “Nice, the ball actually stays on the table now, oh wait, no, it rolls off, damn it. Okay, let’s just add a wall around the desk so the ball can’t fall off. Great, now I can… wait, why is the table flying in the sky in the background? Aw, crap, one of the walls accidentally touched its collision box. Alright, that’s fixed now, let’s try to pick the ball up!”

Needless to say, it didn’t work, but I had no idea why. It didn’t help that whenever my virtual hands touched the ball they just disappeared and I just assumed they were out of reach for the sensor. (Well, looking back I now assume it was due to some kind of rendering glitch). Some not so quick Google searches later I found out that actually the SDK I downloaded earlier was just the core part, which is basically good for nothing, and I have to download additional modules to make it work. One of which is the “Interactivity Module”. I have to question why that is separate from the core module, since interactions seem to be the point of this gadget, but whatever.

Point is, after downloading that module, reading its documentation, it worked! Here’s a picture of me picking up a ball (which was a premade example asset included with the interaction stuff) with one hand and trying to take said picture with the other.


Then I made another ball, this time with the built-in Unity shape-builder tool thingy, and I applied all the attributes the premade ball had to this new one, and it worked, great!

Next, I wanted to make some kitchen objects that can be picked up and thrown around, We were trying to make a cooking game after all, and throwing fragile plates or sharp knives at the wall is clearly a relevant activity. I had a plate model that came with the kitchen table, so I tried to apply the same attributes to it as the other balls. On one hand, the plate had physics, which was great. On the other hand it fell through the table. As it turns out, the “Mesh Collider” attribute just doesn’t work. Like, at all. So I replaced it with a “Box Collider” that roughly encased the plate. Now it had some rudimentary physics, and it stayed on the table. Which is great and all, but I couldn’t pick it up, no matter what attributes I applied to it.

And since the end of the workshop was approaching fast, I just made a ball pit instead. Here’s an in-engine demonstration which I should’ve recorded when I had access to the LEAP, cause right now all it does is showcase the wonky physics.


Thankfully Alice recorded a video of me playing with the actual thing. Yes, the LEAP is actually just as inaccurate and hard to use as it seems here. At least that was my experience with it.

To sum it up: the LEAP is a cool idea, but unless they make it a LOT more precise it’s hard to imagine it being used for anything other than showing off its novelty. Analogue controls are just way too precise to be replaced by something like this. As for child obesity, there are probably a million better (and cheaper) ways to help with that.

(Sorry about that terrible title. Just so you know, I was also considering “An extra hour in the ball pit” but I decided that that reference was probably a bit too obscure.)

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.