Crazyflie with Kinect 2 and Matlab

[AllanB]

We recently developed a project for a first-year engineering class in which students control the flight of a crazyflie using a Kinect 2 and Matlab.

A nice example of student work is at https://www.youtube.com/watch?v=Wi1KXfN ... e=youtu.be. The system is programmed to measure the position of the green ball, and then fly the crazyflie over to pick it up and deliver it (the ball attaches to the crazyflie with velcro).

Everything is running in Windows 10 - we use the Bitcraze cf API to do the communications, and use the python matlab engine (with mex files developed by Juan Terven http://www.mathworks.com/matlabcentral/ ... for-matlab) to read the kinect and extract 3D coordinates from the color and depth images.

Some information about the image processing and designing the PID controller is posted on the course website at http://www.brown.edu/Departments/Engine ... jects.html.

We are indebted to many in this forum who have posted solutions to small problems we ran into along the way while developing the project.

AB

[arnaud]

Hi Allan,

Thanks for the feedback. The video looks good, it is really great job! We know first hand how hard it is to control the thrust and the altitude so is great that they manage to keep is up after carrying some weight (I interpret the slight delay in the celebration that there has been many instance of crash just after picking-up the ball ).

It would be very interesting if you could document somehow the connection between Matlab an the python lib! Either on the wiki or just here and I can add it to the wiki. Matlab connection has been a problem for a lot of people and since we are not Matlab user we have no way to help-out.

Do you mind if we add some info about the course and maybe the video in the education part of the webpage? This is a great example of what can be done with the Crazyflie.

Best regards,
Arnaud

[AllanB]

Hi Arnaud,

Thanks for responding! We would be happy to be listed on the education page.

"crash just after picking-up the ball" Yes It is actually not a problem with the crazyflie which is quite stable, but with the green ball - it needs to be very light and loosely attached to the stand, but then the prop downwash blows it off the stand. If we try to attach it too firmly to the stand the crazyflie gets stuck. Maybe next time we can use some sort of electromagnetic device to hold and release the payload at the right time. But for now Amazon does not need to be afraid of competition from Crazyflie package deliveries....

We get quite good control of altitude if we compensate the motor thrust for the dropping battery voltage. Specifically, instead of using
thrust = (Kp + Kd*s + KI/s) * (z*-z)
in the PID controller we use
thrust = 2903*m - 15300*V + (Kp + Kd*s + KI/s) * (z*-z)
where m is the crazyflie mass in grams and V is the battery voltage (our crazyflies with the training wheels and holder for the ball are 35 grams). The compensation seems to be surprisingly consistent for different crazyflies.

Here are some short instructions for using MATLAB from within the API. I can provide a modified ramp.py example file that uses matlab to calculate the thrust setting as well if that is useful but didnt want to upload that here....

Allan

USING MATLAB WITH THE CRAZYFLIE API

Using Matlab with the Crazyflie API is easy – you just need to install the python ‘matlab engine’ and then can access all matlab commands directly from python.

PREREQUISITES:
1. MATLAB 2014b or later
2. 64 bit python 2.7, 3.3 or 3.4
3. The Crazyflie API


INSTALLING THE MATLAB PYTHON ENGINE

1. Find the path to the MATLAB folder. To do this, start MATLAB and type matlabroot in the command window. Copy the path returned by matlabroot
2. To install the engine open a command window and on Windows type cd "matlabroot\extern\engines\python"
On Mac or Linux systems type cd "matlabroot/extern/engines/python"
3. Finally type
python setup.py install

Step 3 sometimes fails if you do not have write permission to the default build directory. If this happens:

3b Create a new directory to store the build directory where you have write permission.
3c Then set up the python engine with
python setup.py build --build-base builddir install --user

Here builddir is the path of the directory created in step 3b. Note the double dashes (no space) before ‘build-base’ and ‘user’.

More information (from Mathworks MATLAB documentation):

System reqirements: http://www.mathworks.com/help/matlab/ma ... hon.html

Installation: http://www.mathworks.com/help/matlab/ma ... ython.html

Installation in non-default locations
http://www.mathworks.com/help/matlab/ma ... tions.html

USING THE MATLAB PYTHON ENGINE

Once you have installed the matlab engine, you can use any matlab commands (or your own matlab scripts) from within the Crazyflie API. To do this:
1. Import the matlab engine with
import matlab.engine

2. Create a matlab engine object (which starts matlab – the initial startup will be slow) with (eg)
self.eng = matlab.engine.start_matlab()

3. OPTIONAL: You might want to add a directory with your matlab scripts to the matlab path:
self.eng.addpath("directory name",nargout=0)


4. OPTIONAL: MATLAB errors and console output will usually appear in the console window of IDEs such as pycharm or Eclipse. To divert them to a file you can create StringIO objects. To do this you must first import StringIO, then create the StringIO objects, eg
self.matlabout = StringIO.StringIO()
self.matlaberr = StringIO.StringIO()

5. You can then run a matlab script or function with

output = self.eng.function_name([argument list],[stdout=self.matlabout],[stderr=self.matlaberr],[nargout=n])

Here ‘output’ is the output from the matlab function (if multiple arguments are returned by the function then output is an array - see the matlab python engine documentation for more information),
[argument list] is the input arguments to the matlab function,
self.matlabout and self.matlaberr are StringIO objects to capture Matlab errors and console output (optional) and
n is the number of output arguments from the MATLAB script (the default is 1).

More information:

Matlab engine API documentation from Mathworks: http://www.mathworks.com/help/matlab/ma ... ython.html

Call MATLAB functions from python http://www.mathworks.com/help/matlab/ma ... ython.html


LIMITATIONS:

Not all data structures in matlab and python have direct equivalents. You can pass most primitive data types between them, as well as arrays. You can also usually pass matlab data types from one matlab function to another (eg you can save an output variable from a matlab function such as a file handle or plot handle in a python variable, and then pass it to another matlab function).

For speed it is best to minimize communications between python and matlab - i.e. try to do all matlab computations through a single script, instead of calling many matlab functions from inside python.

More information

Pass data to matlab from python http://www.mathworks.com/help/matlab/ma ... ython.html

[arnaud]

Thanks a lot for the documentation, I copied it in the wiki: https://wiki.bitcraze.io/doc:crazyflie:dev:env:matlab. This will hopefully be very useful to many more!

We are going to try the new battery compensation. There already was a battery compensation in the Crazyflie did you disable it or is this on top of it?

[AllanB]

Thanks!

Our compensation was on top of anything already there (our class is quite large and needs multiple setups so it was easiest for us just to 'plug and play' everything).

Allan