Communication problems when running multiple Crazyflies

[nagaban2]

Hello!

(I have read multiple relevant posts on this forum, but haven't been able to solve my problems, so this isn't a repeated post)

I am running 7 Crazyflies at a time, and I am using 7 Crazyradios, so I have 1 radio for each crazyflie.
I developed my own trajectory tracking controller, which (using a motion capture system) accurately flies a Crazyflie in a lissajous pattern.
Note that I have no issue when I run just 1 Crazyflie on its own... the path is accurate!

------------------------------------------------

Problems:

1) I frequently get "Connection lost- too many packets lost" when I'm running multiple crazyflies
This error message shows when the connection to a Crazyflie is lost, and this causes the Crazyflie to just drop out and stop flying.
Note that although this happens almost all the time, there isn't a clear pattern (generally a different Crazyflie each time, sometimes they all work fine, etc...)

2) Assuming I get lucky and the connections are good enough to not drop out, my crazyflies go through a take-off formation routine, and then begin to fly together in a lissajous pattern.
Due to some communication error, there is a problem with the commands (buffering? queue? timing? IDK!) that causes the paths of the crazyflies to not be as accurate as when running solo.
Due to these small deviations from the desired path, the Crazyflies in my pattern collide and crash (accuracy is essential for me, so I need to solve this problem!)

------------------------------------------------

Thing I've done:

1) I spaced out the channels for each crazyflie/crazyradio as much as possible. It works much more consistently than before. Since they're spaced out by 20+ channels, I feel that channel interference isn't an issue.
2) Wifi in the room is turned off, and I don't know of other things in here that would cause interference
3) Tested the controller on a single crazyflie at a time- works well. Tested on 2/3 at a time, and the paths are still accurate.


------------------------------------------------

I have no idea how to go about solving this because I do not understand the root of the problem.

When I run 1 Crazyflie on its own, I have no connection problems.
When I run a few, they work much better than when I run all 7.
What I'm gathering is that I'm having some sort of communication problem when running a lot of Crazyflies all at once.

However, each crazyflie has its own
-nodes (controller, etc.)
-firmware
-crazyradio
-channel (not too close to other crazyflies' channels)

So I do not understand what the communication problems are that I'm having.
Any thoughts?

[DataBoy]

I'm new to CF, but I do RF comms professionally, so maybe I can offer some minor assistance.

Presume you're using 250 KB channels (not 1 or 2 MB)?

There is an effect known as Intermodulation (IM) in which frequencies add and subtract from each other forming an algebraic mess (2A-B, etc.). Way too much info here: http://en.wikipedia.org/wiki/Intermodulation

The net net is that various frequencies add and subtract from each other, creating unintended carriers. If one of those unintended carriers happens to land on-channel with another you're running, it makes interference. It sounds like that might be your issue.

Although the math is pretty simple, and there are tons of online calculators to see if you have IM, I would suggest you:

- determine in the large swarm if any specific flies are errant, or if it's the whole fleet.
- hopefully it will be less than the whole fleet. If it is a subset, try changing the frequencies of just those to some odd value.
- interestingly, exactly spacing the frequencies out as you've done is probably not the best system-design decision, because it's just asking for high-order IM (2x one frequency, minus another, equals a third).

I don't know the Crazyradio well (yet!) but in general I wouldn't expect adjacent channels to interfere with each other. (that said, WiFi channels overlap, and I don't know the bandwidth etc. of CR).

You could test that in isolation with just two CF's. You might do better to pair up closer channels, and stagger them in the larger picture.

Because the Crazyradio does not have a shield, it may be possible that they are IM sensitive. (Rusty bolts on radio towers cause trouble because they turn into diodes and create IM, so it's all fuzzy...).

I think that this can be solved, hopefully without shield on the CF itself, because tinfoil won't work too well, so it'd be heavy.

Very cool project you're working on, and I'm happy to help as best I'm able. I have some high priced test equip and two CF's here that I can conduct some experiments on, should the need arise.

[whoenig]

I would suggest time-slicing your communication. This can be easily done by using the following branch https://github.com/bitcraze/crazyflie-c ... _multilink. If you just need to send set-points, it might be sufficient to use 1 or 2 crazyradios for your 7 flies. If you need to stream out sensor data, the bandwidth of that solution is not enough (still 2 flies per radio work fine). Another thing you can try (if money is not an issue) is to switch to Crazyradio PA (and Crazyflie 2.0).

[nagaban2]

Thank you both for the replies, I really appreciate the help!

I'll try time-slicing today and see if that helps me (I'm not using sensor data so hopefully this helps)

For IM, that makes sense- I'll try to randomly space out my channels.
As far as which flies are errant, this is fairly random between each run, which is frustrating.
I was hoping to plug the antennas into a spectrum analyzer and search for overlap (while sending commands).... I have an oscilloscope that can handle high frequencies, so I'll try that.

Also, after looking into the firmware and radio drivers a bit more, I discovered that my problem is that the Crazyflie is sending an "ack" that is "False," which causes the connection to end.
What does ack=False indicate (as opposed to not receiving an ack at all)? Also, does that indicate that this is a bandwidth/speed problem, as opposed to interference?

[DataBoy]

There's a Quality signal/display on the client that you might be able to plot. Normally there's a means to determine Bit Error Rate (or block errror rate) so you can try stuff and see if it helps.

One thing to determine is: is this an uplink (computer to flie) or downlink problem. Or both.

If your scope does FFT's you might be able to resolve frequencies high enough, but a high quality spectrum analyzer would do a better job. I happen to have one of these: http://www.anritsu.com/en-US/Products-S ... S412E.aspx and can help characterize the channel in my lab, if you need it.

The NAK probably means there were (too many) bit errors on a pack to be decoded. That's just a guess.

If you're going to timeslice the comms you'll be dividing your throughput by 7, so that might be problematic. But the fact that above (3?) CF's cause the problem is going to hold your root cause. Is it the same CF's and is it channel-independent would be key questions. Also, can you set up to fly them further apart, and maybe see if there is some local Flie-to-Flie interference, or whether this is a PC-based problem.

Interesting problem!

[dbrgn]

If you get your group of CFs to work, I'd really like to see a video and maybe a blogpost about it