Flying crazyflie with a HoloLens

Discussions and questions about the Crazyflie Nano Quadcopter
Post Reply
theseankelly
Expert
Posts: 143
Joined: Mon Dec 28, 2015 3:23 pm
Contact:

Flying crazyflie with a HoloLens

Post by theseankelly » Mon May 16, 2016 1:54 am

Hey Community

My day job is as a software/firmware engineer for Microsoft HoloLens, so naturally I had to put together a HoloLens proof of concept! It's a really simple port of the UWP app I wrote (on the bitcraze GitHub now) for windows 10 and windows phone 10 that hooks in hand gestures as the input.

Demo video here:
https://1drv.ms/v/s!AiLl1yrI9l9HtZ4jx5qxpb10wV698A

It's pretty crude and difficult to fly. I'm doing the hand gestures at a high layer by handling the "navigation gesture" which presents a value from -1 to 1 on each axis relative to where the tap gesture occurred. Ideally I'd hook in one layer lower and process the raw hand input and do smoothing there - would have more control over the physical range, etc. That layer also supports multi-hand tracking so I could get yaw control also.

It'd also be fun to do tracking -- the HoloLens can self localize, and if I can localize the crazyflie with the hololens' sensors, then that'd make for a pretty sweet indoor positioning system.

Sean
http://www.thejumperwire.com
Tips, tutorials, and science about DIY electronics, drones, and embedded software.

tobias
Bitcraze
Posts: 1563
Joined: Mon Jan 28, 2013 7:17 pm
Location: Sweden

Re: Flying crazyflie with a HoloLens

Post by tobias » Mon May 16, 2016 11:26 am

This is so cool! Didn't know about the HoloLens before, and now I really want one!

Are you controlling the CF2 directly over bluetooth from the HoloLens?

theseankelly
Expert
Posts: 143
Joined: Mon Dec 28, 2015 3:23 pm
Contact:

Re: Flying crazyflie with a HoloLens

Post by theseankelly » Mon May 16, 2016 2:51 pm

Yup, no external PC in this setup. Just HoloLens and the crazyfile. The video was recorded on the HoloLens too (it has a front facing video/photo camera).
http://www.thejumperwire.com
Tips, tutorials, and science about DIY electronics, drones, and embedded software.

danielvf
Beginner
Posts: 3
Joined: Mon May 16, 2016 2:23 pm

Re: Flying crazyflie with a HoloLens

Post by danielvf » Mon May 16, 2016 3:10 pm

That's awesome! I wonder if you could use the HoloLens's camera to track the drone and take care of controlling vertical component of the CF2's velocity.... The LED's should be easy to track - just control yaw to always keep the blue one's facing you.

bjorn
Bitcraze
Posts: 12
Joined: Mon Nov 02, 2015 10:16 am

Re: Flying crazyflie with a HoloLens

Post by bjorn » Tue May 17, 2016 9:45 am

Awesome!

bjorn
Bitcraze
Posts: 12
Joined: Mon Nov 02, 2015 10:16 am

Re: Flying crazyflie with a HoloLens

Post by bjorn » Tue May 17, 2016 9:49 am

Would you mind if we put this on our website? :-)

theseankelly
Expert
Posts: 143
Joined: Mon Dec 28, 2015 3:23 pm
Contact:

Re: Flying crazyflie with a HoloLens

Post by theseankelly » Tue May 17, 2016 3:34 pm

Sure!

I thought about the LED idea -- I think it would be a good start but not good enough for a robust solution. In particular, it would be very difficult to disambiguate between deltas in yaw and deltas in distance based on two LEDs alone. I don't have much first hand experience in computer vision though -- perhaps there are tricks for solving these problems that I'm not aware of. Short of that, I could always program the LED ring with a fixed pattern, or add a styrofoam ball on top of the crazyflie (like is done in the kinect-based tracking schemes).

Where this would *really* shine though, is coupled with the work bitcraze is doing on the positioning system. It wouldn't be a big feat to sync up the HoloLens' coordinate system with that of the PWM system. Without knowing too much about how that system works, I imagine if I have some setup step where I register the position of the marker tags with HoloLens (simply walking around and air-tapping on them -- the APIs allow you to translate a tapped position into the HoloLens' coordinate system. If this is done indoors, long term drift will not be a problem. Once the coordinate systems are aligned, you could do all sorts of cool things. Some ideas I've had:
- wall and obstical avoidance using HoloLens' surface detection + position information
- a "follow me" feature where the crazyflie literally follows you as you walk around
- a "Come here" feature that computes the optimal path from the crazyflie to your position, also avoiding walls and things
- walk around and tap on the floor to set waypoints

I definitely intend to pick up a loco positioning system once they become available.. :D
http://www.thejumperwire.com
Tips, tutorials, and science about DIY electronics, drones, and embedded software.

Technogen
Beginner
Posts: 3
Joined: Sat May 21, 2016 3:29 am

Re: Flying crazyflie with a HoloLens

Post by Technogen » Sat May 21, 2016 3:30 am

This is crazy cool, I linked a friend to it that also has HoloLens.

Dookien
Beginner
Posts: 1
Joined: Sun Aug 28, 2016 8:09 am

Re: Flying crazyflie with a HoloLens

Post by Dookien » Mon Aug 29, 2016 6:48 am

danieelvf wrote:That's awesome! I wonder if you could use the HoloLens's camera to track the drone and take care of controlling vertical component of the CF2's velocity.... The LED's should be easy read this bathmate review post to track - just control yaw to always keep the blue one's facing you.
That looks crazy cool. I'd love to give it a try. Maybe this is the future of flying.

Post Reply