Page 1 of 1

packets queued but not sent

Posted: Fri Jul 03, 2015 8:06 am
by absol
Hello everyone,

I've rewritten some firmware, and am struggling to send packets reliably. It feels like a queue must be involved somewhere, but I'm no expert!!

I send a packet out for each motor (for example), and if I pause for at least 5ms between these four packets, they are all received nicely.
If I send them without pausing, only the first is received by the CF. Yet they are being queued somewhere. For example.

Send M1 on; Send M2 on; Send M3 on; Send M4 on;
>observe one motor on.
Pause; Send M1 off;
>observe additional motor on
Pause; Send M2 off;
>observe third motor on
Pause; Send M3 off;
>observe last motor on
Pause; Send M4 off;
>observe first motor off

I don't fully understand where this queue is happening, or why a delay is prompting the radio to send packets. Notably the queue isn't just unloading itself slowly: the additional packets (with sufficient delay) are prompting the original ones to be sent.

I'm very new to thread management, so I expect anyone with insight into the radiodriver operations to be more clued in than me!

Thanks in advance
absol

Re: packets queued but not sent

Posted: Thu Jul 09, 2015 2:58 pm
by marcus
Hi,

On the way down to the client there's two queues. The first queue is in the CRTP layer (queued here) and this will drop packets if it's full. The second queue is in the radio driver (queued here) and this queue will drop, but it will be retried over and over (from here) so the packet will not be dropped.

Re: packets queued but not sent

Posted: Fri Jul 10, 2015 3:02 pm
by absol
Hi marcus,

Marathon answering spree you've been on. I can't thank you enough.

You've significantly improved my understanding of the packet sending pipeline, but I still haven't crushed my bug!

I was wondering if the send_packet resend feature (seen here) is functional, and if it's used in the client implementation at all.
I'm unclear as to how it works, and feel it would help me pinpoint which side of CF / my client I should focus on debugging.
Namely:
where is _answer_patterns is filled?
are the responses automatic or do they need to be explicitly sent back by firmware addition?

Thanks again!
Absol

Re: packets queued but not sent

Posted: Fri Jul 10, 2015 6:44 pm
by absol
-edit

Re: packets queued but not sent

Posted: Tue Jul 14, 2015 11:51 am
by marcus
First a quick background to the problem. In the current implementation the up-link is considered "safe", but not the down-link. This means that if you send a message (like a commander packet) you will get an ACK back on the radio, this means the message got though. Until you get the ACK back the radio driver will continue to try (first in the Crazyradio firmware and then in the Python lib). If there's no ACK after a certain number of retries the link is deemed dead. But some messages (requests) will be ACKed directly, and then at a later point in time data will be sent back. The Crazyradio will poll the Crazyflie for messages continuously and then when the reply is ready it's sent with an ACK. But here's the problem. The Crazyflie will poll (do you want to send something?), the Crazyflie will reply with an ACK (with data in it) and this could be lost. Now the Crazyradio will poll again, but the Crazyflie will say there's nothing to be sent.

Now back to your question. The expected_reply in send_packet(..) is used to work around the issue described above. When calling the function this variable should contain the first bytes of the response you are expecting (at a later point). This pattern will then be checked for every message that comes in. Once you find a packet that fits the pattern the request-response cycle is deemed finished. Until the pattern is found the original request (that's ACKed immediately) will be resent over and over using a timer.

The pattern differs a bit for each port, let's take the parameters as an example. Here the unique pattern for each packet is three bytes: the header (with port and channel), the command byte (get, set or toc), and the id of the parameter. Then this request will be re-sent until we get an answer with the same header, command and id.

This adds a lot of complexity for the stack, so we've been looking into solving this on a lower layer (in the radio). We're pretty sure that it can be solved, but we haven't had the time yet.