Flying crazyflie with a HoloLens

[theseankelly]

Hey Community

My day job is as a software/firmware engineer for Microsoft HoloLens, so naturally I had to put together a HoloLens proof of concept! It's a really simple port of the UWP app I wrote (on the bitcraze GitHub now) for windows 10 and windows phone 10 that hooks in hand gestures as the input.

Demo video here:
https://1drv.ms/v/s!AiLl1yrI9l9HtZ4jx5qxpb10wV698A

It's pretty crude and difficult to fly. I'm doing the hand gestures at a high layer by handling the "navigation gesture" which presents a value from -1 to 1 on each axis relative to where the tap gesture occurred. Ideally I'd hook in one layer lower and process the raw hand input and do smoothing there - would have more control over the physical range, etc. That layer also supports multi-hand tracking so I could get yaw control also.

It'd also be fun to do tracking -- the HoloLens can self localize, and if I can localize the crazyflie with the hololens' sensors, then that'd make for a pretty sweet indoor positioning system.

Sean

[tobias]

This is so cool! Didn't know about the HoloLens before, and now I really want one!

Are you controlling the CF2 directly over bluetooth from the HoloLens?

[theseankelly]

Yup, no external PC in this setup. Just HoloLens and the crazyfile. The video was recorded on the HoloLens too (it has a front facing video/photo camera).

[danielvf]

That's awesome! I wonder if you could use the HoloLens's camera to track the drone and take care of controlling vertical component of the CF2's velocity.... The LED's should be easy to track - just control yaw to always keep the blue one's facing you.

[bjorn]

Awesome!

[bjorn]

Would you mind if we put this on our website?

[theseankelly]

Sure!

I thought about the LED idea -- I think it would be a good start but not good enough for a robust solution. In particular, it would be very difficult to disambiguate between deltas in yaw and deltas in distance based on two LEDs alone. I don't have much first hand experience in computer vision though -- perhaps there are tricks for solving these problems that I'm not aware of. Short of that, I could always program the LED ring with a fixed pattern, or add a styrofoam ball on top of the crazyflie (like is done in the kinect-based tracking schemes).

Where this would *really* shine though, is coupled with the work bitcraze is doing on the positioning system. It wouldn't be a big feat to sync up the HoloLens' coordinate system with that of the PWM system. Without knowing too much about how that system works, I imagine if I have some setup step where I register the position of the marker tags with HoloLens (simply walking around and air-tapping on them -- the APIs allow you to translate a tapped position into the HoloLens' coordinate system. If this is done indoors, long term drift will not be a problem. Once the coordinate systems are aligned, you could do all sorts of cool things. Some ideas I've had:
- wall and obstical avoidance using HoloLens' surface detection + position information
- a "follow me" feature where the crazyflie literally follows you as you walk around
- a "Come here" feature that computes the optimal path from the crazyflie to your position, also avoiding walls and things
- walk around and tap on the floor to set waypoints

I definitely intend to pick up a loco positioning system once they become available..

[Technogen]

This is crazy cool, I linked a friend to it that also has HoloLens.

[Dookien]

That's awesome! I wonder if you could use the HoloLens's camera to track the drone and take care of controlling vertical component of the CF2's velocity.... The LED's should be easy to track - just control yaw to always keep the blue one's facing you.

That looks crazy cool. I'd love to give it a try. Maybe this is the future of flying.