Crazyflie restarts in air using experimental firmware

Firmware/software/electronics/mechanics
PhilipH
Beginner
Posts: 8
Joined: Tue Jun 27, 2017 9:11 pm

Crazyflie restarts in air using experimental firmware

Post by PhilipH »

Hi,

I have been trying to integrating Mike Hammer's nonlinear quaternion controller into the firmware for more aggressive control. I have looked into both ICRA demo source code and Mike Hammer's code on https://github.com/mikehamer/crazyflie- ... controller, and modified parts of the firmware to include the new controller.

However, whenever I used my custom firmware (or the crazyflie-firmware-experimental on https://github.com/bitcraze/crazyflie-f ... /icra-2017), I started experiencing an annoying problem: Crazyflie would restart randomly when its landed or flying.

Whenever this happens, it does not show any "ASSERT FAIL" , it just stop flying, restart and immediately starts reconnect to my ROS server again. The only part I have modified in my custom firmware is adding the code for the nonlinear controller and "drag" term in the kalman filter, and I am currently using a modified CrazyflieROS to fly with Vicon motion capture system.

I suspect that this may be a numerical issue in the new controller because I have never experienced crazyflie restarting midair before. However, I have tried using

Code: Select all

 DEBUG_PRINT
in the controller_new.c and power_distrubition_cf2.c to check if there is any negative square root or divide by zero, but they do not seem to the issue.

So I have two questions:
1) Is there anyone who has experience similar problem? (restart when having crazyflie-firmware-experimental)
2) Where in the firmware will trigger a restart?

Thanks in advance,

Philip
arnaud
Bitcraze
Posts: 2538
Joined: Tue Feb 06, 2007 12:36 pm

Re: Crazyflie restarts in air using experimental firmware

Post by arnaud »

Hi Philip,

1) I have not seen that with Mike's controller in particular.

2) The reboot is caused by the watchdog: if the IDLE task is not run often enough (ie. you are locked in an assert or something is taking too much time), the watchdog will reset the Crazyflie. If you hit an assert, the filename and location of the assert will be printed at restart. You could try to run with DEBUG=1 which disable the watchdog or disable it manually (https://github.com/bitcraze/crazyflie-f ... tem.c#L248). Be a bit careful with disabling the watchdog: if you are really locked somewhere in the code you can crash with motors still running, it is not very dangerous with Crazyflie but is always a bit exciting :-).

/Arnaud
japreiss
Beginner
Posts: 12
Joined: Thu Mar 24, 2016 1:17 am

Re: Crazyflie restarts in air using experimental firmware

Post by japreiss »

You might also try increasing STABILIZER_TASK_STACKSIZE in src/config/config.h. I haven't used Mike's controller, but I assume it's more complex than the default and thus might use more stack.
PhilipH
Beginner
Posts: 8
Joined: Tue Jun 27, 2017 9:11 pm

Re: Crazyflie restarts in air using experimental firmware

Post by PhilipH »

Thanks Arnaud and James!

I increased the stack size of CRTP_RX_TASK_STACKSIZE to (3 * configMINIMAL_STACK_SIZE) and STABILIZER_TASK_STACKSIZE to (5 * configMINIMAL_STACK_SIZE). The good news is that the times where the process got stuck somewhere(and thus watchdog time out and restart...) occurs much less frequently. However, in the extreme cases where I left the CF connected for a long period of time, it still restart sometimes.

If my understanding is correct, variables stored in the stack will be automatically freed if we exit the current function/loop in c. Is that the case with Crazyflies? Now, it almost seems as if the stack size keeps increasing and will eventually exceed the limit?

Philip
arnaud
Bitcraze
Posts: 2538
Joined: Tue Feb 06, 2007 12:36 pm

Re: Crazyflie restarts in air using experimental firmware

Post by arnaud »

Do you get any information in the console when the Crazyflie restarts?

We are not using dynamic memory (malloc) in the Crazyflie, so memory leaks should not be possible. The stack will grow and shrink with function call. One possible random problem is with interrupts: the interrupt routines are run in the stack of the current task, if one task is a little bit on the limit of its stack and an interrupt occurs there could be a possibility of stack overflow there. To verify that this is not happening you could multiply by 2 the stack size of all the tasks (we have a lot of free memory in the Crazyflie :)), then you can run and if you still observe crashes it it very unlikely they come from a stack overflow.
PhilipH
Beginner
Posts: 8
Joined: Tue Jun 27, 2017 9:11 pm

Re: Crazyflie restarts in air using experimental firmware

Post by PhilipH »

Sorry about replying a bit late,

1) Before, the Crazyflie would restart and print ("no assert information found"). I have increased the stack size of all the tasks by 2, and it still restarts and crashes into the ground, which means a stack overflow is unlikely.

2) However, I start to see an interesting problem. The CF still restart and now it starts printing

Code: Select all

SYS: The system resumed after 
watchdog timeout [WARNING]

SYS: Assert failed at src/driv
ers/src/nvic.c:147
However, I don't know where does the error occur in the firmware and caused a hard fault. Do you have any ideas for the reason? If not, how can I debug this problem?
tobias
Bitcraze
Posts: 2339
Joined: Mon Jan 28, 2013 7:17 pm
Location: Sweden

Re: Crazyflie restarts in air using experimental firmware

Post by tobias »

Tracking down the hardfaults can be a big challenge without a SWD debugger. You could try and change in nvic.c:

Code: Select all

#else
#include "uart1.h"
#define UART_PRINT    uart1Printf
#endif
to

Code: Select all

#else
#include "debug.h"
#define UART_PRINT    consolePrintf
#endif
And see if you get the print information to the cfclient console. It might not work though depending on how the system crashed.
meyigo2018
Beginner
Posts: 10
Joined: Wed Jun 14, 2017 1:43 pm

Re: Crazyflie restarts in air using experimental firmware

Post by meyigo2018 »

Thanks Tobias and Arnaud!

I'm Philip's colleague. We're still working on the problem.

I've tried to let crazyflie print in the console. Unfortunately, nothing was there. We've been trying to use a ST-Vlink to debug but we are not expert on it. So I'm wondering,
1) with the debugger, Is it possible to find out which line in the code causes the watch dog time out?
2) or do you have any suggestions on debugging hard fault w/ the debugger?

Tracy
tobias
Bitcraze
Posts: 2339
Joined: Mon Jan 28, 2013 7:17 pm
Location: Sweden

Re: Crazyflie restarts in air using experimental firmware

Post by tobias »

Putting a break-point in the hard fault function could reveal the problem and looking at the stack trace. However sometimes the stack trace is corrupt as well. You will also get the program counter where the hard fault happened by looking at stacked_pc variable.
meyigo2018
Beginner
Posts: 10
Joined: Wed Jun 14, 2017 1:43 pm

Re: Crazyflie restarts in air using experimental firmware

Post by meyigo2018 »

Hi Toblas,

Thanks for your suggestions. I've found the problem. It got stuck at the infinite loop where it checks whether the DMA stream for uart6 is disabled thus ready for next transfer, https://github.com/bitcraze/crazyflie-f ... ink.c#L248. In Debug mode, the watch dog is disabled so I can see it stops at that point after a few minutes of normal operation.

However, I don't quite understand why the stream wasn't disabled. Isn't it reset to Disable after all data in DMA stream is transferred to the UART6 Data register?

Just to briefly introduce our setup. I'm broadcasting vicon information from ground station and sending ping at a really low rate. The broadcasting and pinging are running on different thread on the ground station computer. Radios are properly guarded so that only one thread could use the corresponding radio. If I registered a lot of log blocks, there is a high probability that the crazyflie got stuck at that point and then reboot. However, if I do not register any log blocks but still send ping, the problem was solved. Also, I'm using the older firmware which does not include the most recent updates which moves syslink packet parsing into ISR.

I have been digging down into the firmware but nothing came up to my mind. It would be really helpful if you could share me your opinions.

Thanks in advance!
Post Reply