Problem connecting multiple quads to multiple Crazyradios

[reiinakano]

Hello there! I am a university student from the Philippines using the CrazyFlie 2.0 as a platform for testing swarm algorithms.

We use an external motion capture system for controlling the movement of each CrazyFlie. In order to communicate with the CF, we use a single cfzmq server (https://wiki.bitcraze.io/doc:crazyflie: ... fzmq:index) for each CF with different ports open (e.g. port 2000-2004 for one quad, port 3000-3004 for the other, etc) and we simply push the yaw,pitch,roll,thrust commands from our own swarm algorithm program written in Python. Since we have a lot of Crazyradios, we use a separate CrazyRadio per CrazyFlie at 1Mbps rate with different channels.

So far, we are only able to successfully control two of them. When we add the third CrazyRadio, each radio is indeed able to establish a connection to each quadrotor, however, the quads can no longer fly properly and sometimes the connection just drops out on one or more Crazyflies ("too many packets lost" error printed on cfzmq server). We also observe that the Crazyradio LED flashes both red and green, probably indicating poor connection.

Also note that we are not able to successfully control the two CrazyFlies 100 percent of the time. Maybe one out of five times, one of the connections will randomly drop out and then we have to switch the Crazyradios around the USB ports or even restart the computer to control it properly again. (Weird, I know)

So some questions:
1. I have recently looked into the radio_multilink branch although it doesn't seem to be updated. Is using this still a valid option?
2. Another solution we are currently looking at is using a network of computers to limit the number of radios per computer to two. Is it valid to think that perhaps our computer's USB drivers simply could not handle controlling three separate CrazyRadios?
3. Any other suggestions or comments regarding our problem? Any help would be very much appreciated.
4. Unrelated but still an interesting question: When a Crazyflie positions itself below another Crazyflie, the air from the quad on top will push down hard on the quad below and ruin its flight pattern. Is this a problem other Crazyswarms have faced? If so, how did you guys fix it?

Thanks in advance for answering my questions!

[whoenig]

Hi,

0.) Make sure you use different addresses (see blog post here: https://www.bitcraze.io/2015/04/towards-crazyswarms-2/). Otherwise there might be some cross-talk between the crazyflies.
1.) It is mainly useful if you want to control multiple CFs with a single radio. However, it is outdated, so you might need to update it yourself.
2.) 3 Crazyradios shouldn't be a problem. You should watch your CPU utilization though - for me the bottleneck was python at some point.
3.) You can switch to ROS and use my crazyflie_ros package (https://github.com/whoenig/crazyflie_ros). It is specifically designed for swarms. It is written in C++ and hence there are less CPU utilization issues. Furthermore it supports to control multiple Crazyflies per Crazyradio out-of-the-box.
4.) This is a hardware problem with any quadcopter. The solution is to avoid it:-) There needs to be a fairly large safety distance in the z-direction, and any possible motion planner you use to plan the trajectories should take that phenomena into account.

Bad Crazyradio connections can also happen if there is interference with other networks. If you can, try to disable WiFi (and/or switch to 5Ghz wifi).

Best,

Wolfgang

[reiinakano]

Hi Wolfgang, thanks for the reply.

We do use different addresses: E7E7E7E7E7, E7E7E7E7E0, and E7E7E7E7E2 etc. using channels 80, 100, and 120 respectively. Unfortunately, we can't disable the WiFi because it is the university's campus WiFi. What channels do you recommend we use?

I believe this radio-multilink branch is yours, correct? https://github.com/whoenig/crazyflie-cl ... _multilink It seems to be more recently updated than the official client's radio-multilink branch. Do we still need to modify your branch for it to work?

By python being the bottleneck, do you mean that running the multiple python "cfzmq" servers could be the bottleneck? Would that explain our error "too many packets lost"?

We'll take a look at ROS and see if we can keep most of our existing code. Is there a way to use ROS simply as a substitute for cfzmq? As in, instead of our system being Optitrack --> Swarm Algorithm --> commands to CFZMQ --> Crazyradio --> multiple quads, can we simply make it Optitrack --> Swarm Algorithm --> commands to ROS (hopefully also using ZMQ or some other communication protocol) --> Crazyradio --> multiple quads. Sorry for all the questions, and thanks very much for answering them!

EDIT: Just checked the CPU utilization. Even when running the entire system with three cfzmq servers, the utilization never goes beyond 40 percent. We've also used channels 107, 100, and 120 so as not to get interference from WiFi. Connections are still dropping out. What could the problem be?

[tobias]

We've also used channels 107, 100, and 120 so as not to get interference from WiFi. Connections are still dropping out. What could the problem be?

Try switching data rates to 1Mbit or 2MBit.

[reiinakano]

We are running on 1Mbps bit rate

[whoenig]

* The branch is mine and most-likely still works. However, I didn't rebase it in a while (i.e. it does not have the latest features from the Bitcraze client). You only need that if you want to try fewer radios. If you just have 3 CFs and 3 radios, there is probably no need to trying it.
* If your CPU utilization is that low, there is most likely no issue with that regard. If you test just a single CF on each of the 3 frequencies, does it work fine? Also try using 2MBit as suggested by Tobias.
* If you switch to ROS, it would be easiest to write your swarm algorithm as a "ROS package". The core idea of ROS is similar to ZMQ, but using pure ZMQ would require some more work. You could try if ROS works better, but to me it sounds more like an interference problem of the radios.
* There are no channels I can truly recommend, because it highly depends on your environment. You can try a few and look at the Signal Strength indicator in the client application. Unfortunately, WiFi uses channel hoping (and the CF does not yet), so it can happen that you find a "good" channel one day and it doesn't work well the next day.

[reiinakano]

We actually have 6 CFs and 6 radios and are planning to add more moving forward. What do you think is the best way for us to scale up the application? Is it still feasible to use only one computer to control all quadrotors? Regardless, we are stuck at just two working for now.

All frequencies work fine one by one. Any combination of two frequencies also works fine. However, when running two, the connection still randomly drops out roughly one out of five times and oddly won't be fixed until we switch around the USB ports or reset the computer. Together with the fact that unrelated USB hubs malfunction when the error "too many packets lost" shows up, this behavior leads us to believe that the problem might lie in USB.

Am I correct in thinking that channels above 100 should not get any interference from WiFi since WiFi is limited to channels below 100?

Also, we use a Windows environment instead of Linux since the OptiTrack software only works on Windows. Is this relevant?

[whoenig]

I had 6 working with 3 radios and there was no limit in terms of computing power (using ROS). I think it would be certainly interesting to try the ROS approach then.
The way it typically works is that you have two machines: One for OptiTrack/Vicon running Windows and a second one with Ubuntu, connected over Ethernet. On the Ubuntu side you can use VRPN to get the data from the OptiTrack software - either using ROS or any other custom software.

If your WiFi follows the specs you are correct. In the US channels above 100 are not allowed for public use (this is different in every country!).

[dinglei]

we want to communicate with my multiple crazyflies by using the motive of the optitrack system.,I have one question about how to set up two rigid bodies to ask you.When I set up the second rigid body ,it gives me the prompt. The error message I put in the attachment。

Can you give me some help.How can I solve these problems.
I’m looking forword to your reply,Thank you very much!

[whoenig]

Looks like it might be related to the last four comments in https://github.com/whoenig/crazyflie_ros/issues/17.