Flying crazyflie with a HoloLens

Discussions and questions about the Crazyflie Nano Quadcopter
Post Reply
theseankelly
Expert
Posts: 153
Joined: Mon Dec 28, 2015 3:23 pm
Contact:

Flying crazyflie with a HoloLens

Post by theseankelly »

Hey Community

My day job is as a software/firmware engineer for Microsoft HoloLens, so naturally I had to put together a HoloLens proof of concept! It's a really simple port of the UWP app I wrote (on the bitcraze GitHub now) for windows 10 and windows phone 10 that hooks in hand gestures as the input.

Demo video here:
https://1drv.ms/v/s!AiLl1yrI9l9HtZ4jx5qxpb10wV698A

It's pretty crude and difficult to fly. I'm doing the hand gestures at a high layer by handling the "navigation gesture" which presents a value from -1 to 1 on each axis relative to where the tap gesture occurred. Ideally I'd hook in one layer lower and process the raw hand input and do smoothing there - would have more control over the physical range, etc. That layer also supports multi-hand tracking so I could get yaw control also.

It'd also be fun to do tracking -- the HoloLens can self localize, and if I can localize the crazyflie with the hololens' sensors, then that'd make for a pretty sweet indoor positioning system.

Sean
http://www.thejumperwire.com
Tips, tutorials, and science about DIY electronics, drones, and embedded software.
tobias
Bitcraze
Posts: 2339
Joined: Mon Jan 28, 2013 7:17 pm
Location: Sweden

Re: Flying crazyflie with a HoloLens

Post by tobias »

This is so cool! Didn't know about the HoloLens before, and now I really want one!

Are you controlling the CF2 directly over bluetooth from the HoloLens?
theseankelly
Expert
Posts: 153
Joined: Mon Dec 28, 2015 3:23 pm
Contact:

Re: Flying crazyflie with a HoloLens

Post by theseankelly »

Yup, no external PC in this setup. Just HoloLens and the crazyfile. The video was recorded on the HoloLens too (it has a front facing video/photo camera).
http://www.thejumperwire.com
Tips, tutorials, and science about DIY electronics, drones, and embedded software.
danielvf
Beginner
Posts: 3
Joined: Mon May 16, 2016 2:23 pm

Re: Flying crazyflie with a HoloLens

Post by danielvf »

That's awesome! I wonder if you could use the HoloLens's camera to track the drone and take care of controlling vertical component of the CF2's velocity.... The LED's should be easy to track - just control yaw to always keep the blue one's facing you.
bjorn
Bitcraze
Posts: 12
Joined: Mon Nov 02, 2015 10:16 am

Re: Flying crazyflie with a HoloLens

Post by bjorn »

Awesome!
bjorn
Bitcraze
Posts: 12
Joined: Mon Nov 02, 2015 10:16 am

Re: Flying crazyflie with a HoloLens

Post by bjorn »

Would you mind if we put this on our website? :-)
theseankelly
Expert
Posts: 153
Joined: Mon Dec 28, 2015 3:23 pm
Contact:

Re: Flying crazyflie with a HoloLens

Post by theseankelly »

Sure!

I thought about the LED idea -- I think it would be a good start but not good enough for a robust solution. In particular, it would be very difficult to disambiguate between deltas in yaw and deltas in distance based on two LEDs alone. I don't have much first hand experience in computer vision though -- perhaps there are tricks for solving these problems that I'm not aware of. Short of that, I could always program the LED ring with a fixed pattern, or add a styrofoam ball on top of the crazyflie (like is done in the kinect-based tracking schemes).

Where this would *really* shine though, is coupled with the work bitcraze is doing on the positioning system. It wouldn't be a big feat to sync up the HoloLens' coordinate system with that of the PWM system. Without knowing too much about how that system works, I imagine if I have some setup step where I register the position of the marker tags with HoloLens (simply walking around and air-tapping on them -- the APIs allow you to translate a tapped position into the HoloLens' coordinate system. If this is done indoors, long term drift will not be a problem. Once the coordinate systems are aligned, you could do all sorts of cool things. Some ideas I've had:
- wall and obstical avoidance using HoloLens' surface detection + position information
- a "follow me" feature where the crazyflie literally follows you as you walk around
- a "Come here" feature that computes the optimal path from the crazyflie to your position, also avoiding walls and things
- walk around and tap on the floor to set waypoints

I definitely intend to pick up a loco positioning system once they become available.. :D
http://www.thejumperwire.com
Tips, tutorials, and science about DIY electronics, drones, and embedded software.
Technogen
Beginner
Posts: 3
Joined: Sat May 21, 2016 3:29 am

Re: Flying crazyflie with a HoloLens

Post by Technogen »

This is crazy cool, I linked a friend to it that also has HoloLens.
Dookien
Beginner
Posts: 1
Joined: Sun Aug 28, 2016 8:09 am

Re: Flying crazyflie with a HoloLens

Post by Dookien »

danieelvf wrote:That's awesome! I wonder if you could use the HoloLens's camera to track the drone and take care of controlling vertical component of the CF2's velocity.... The LED's should be easy to track - just control yaw to always keep the blue one's facing you.
That looks crazy cool. I'd love to give it a try. Maybe this is the future of flying.
Post Reply