Flying crazyflie with a HoloLens

[theseankelly]

Hey Community

My day job is as a software/firmware engineer for Microsoft HoloLens, so naturally I had to put together a HoloLens proof of concept! It's a really simple port of the UWP app I wrote (on the bitcraze GitHub now) for windows 10 and windows phone 10 that hooks in hand gestures as the input.

Demo video here:
https://1drv.ms/v/s!AiLl1yrI9l9HtZ4jx5qxpb10wV698A

It's pretty crude and difficult to fly. I'm doing the hand gestures at a high layer by handling the "navigation gesture" which presents a value from -1 to 1 on each axis relative to where the tap gesture occurred. Ideally I'd hook in one layer lower and process the raw hand input and do smoothing there - would have more control over the physical range, etc. That layer also supports multi-hand tracking so I could get yaw control also.

It'd also be fun to do tracking -- the HoloLens can self localize, and if I can localize the crazyflie with the hololens' sensors, then that'd make for a pretty sweet indoor positioning system.

Sean

[tobias]

This is so cool! Didn't know about the HoloLens before, and now I really want one!

Are you controlling the CF2 directly over bluetooth from the HoloLens?

[theseankelly]

Yup, no external PC in this setup. Just HoloLens and the crazyfile. The video was recorded on the HoloLens too (it has a front facing video/photo camera).

[danielvf]

That's awesome! I wonder if you could use the HoloLens's camera to track the drone and take care of controlling vertical component of the CF2's velocity.... The LED's should be easy to track - just control yaw to always keep the blue one's facing you.

[bjorn]

Awesome!

[bjorn]

Would you mind if we put this on our website?

[theseankelly]

Sure!

I thought about the LED idea -- I think it would be a good start but not good enough for a robust solution. In particular, it would be very difficult to disambiguate between deltas in yaw and deltas in distance based on two LEDs alone. I don't have much first hand experience in computer vision though -- perhaps there are tricks for solving these problems that I'm not aware of. Short of that, I could always program the LED ring with a fixed pattern, or add a styrofoam ball on top of the crazyflie (like is done in the kinect-based tracking schemes).

Where this would *really* shine though, is coupled with the work bitcraze is doing on the positioning system. It wouldn't be a big feat to sync up the HoloLens' coordinate system with that of the PWM system. Without knowing too much about how that system works, I imagine if I have some setup step where I register the position of the marker tags with HoloLens (simply walking around and air-tapping on them -- the APIs allow you to translate a tapped position into the HoloLens' coordinate system. If this is done indoors, long term drift will not be a problem. Once the coordinate systems are aligned, you could do all sorts of cool things. Some ideas I've had:
- wall and obstical avoidance using HoloLens' surface detection + position information
- a "follow me" feature where the crazyflie literally follows you as you walk around
- a "Come here" feature that computes the optimal path from the crazyflie to your position, also avoiding walls and things
- walk around and tap on the floor to set waypoints

I definitely intend to pick up a loco positioning system once they become available..

[Technogen]

This is crazy cool, I linked a friend to it that also has HoloLens.

[Dookien]

That's awesome! I wonder if you could use the HoloLens's camera to track the drone and take care of controlling vertical component of the CF2's velocity.... The LED's should be easy read this bathmate review post to track - just control yaw to always keep the blue one's facing you.

That looks crazy cool. I'd love to give it a try. Maybe this is the future of flying.