CF2.1 firmwire query for machine learning (ML) applications.

Discussions about quadcopters/multi-rotors
imranmomtaz
Member
Posts: 36
Joined: Tue Mar 17, 2020 7:01 pm

CF2.1 firmwire query for machine learning (ML) applications.

Post by imranmomtaz »

Hello,
I was planning to do some machine learning code in the CF2.1 firmwire. I have following query regarding this:

1. How much memory in RAM, and ROM is available to implement ML approach?
2. Please suggest me a ML package which will be suitable for CF2.1 and be compatible with its core? I want to have a solution which will need minimal code change in existing CF2.1 firmwire, and it should also be cheap in terms of memory and computation overhead.

Thanks a lot!
kimberly
Bitcraze
Posts: 1050
Joined: Fri Jul 06, 2018 11:13 am

Re: CF2.1 firmwire query for machine learning (ML) applications.

Post by kimberly »

I think this blogpost has all the answers you are looking for: https://www.bitcraze.io/2019/10/learnin ... crazyflie/. It explains implementing a small neural network onboard of the stm processor of the crazyflie with TFlite. It also includes a overview of the remaining ram, but you do need to know that that is probably less these days since there is a lot of adde d functionality.

Also consider using the app layer for implementing your NN if you do not want to change too much to the standard firmware.
imranmomtaz
Member
Posts: 36
Joined: Tue Mar 17, 2020 7:01 pm

Re: CF2.1 firmwire query for machine learning (ML) applications.

Post by imranmomtaz »

Thank you for your reply. How much RAM is available in the current CF2.1 firmwire? And, also how can I find this out myself?

Additional question:
1. I was searching the manual for STM32F405 (https://www.st.com/resource/en/referenc ... ronics.pdf) and found that there are 112KB SRMA1 and 16KB SRAM2 (total: 128KB). But CF2.1 page at https://store.bitcraze.io/products/crazyflie-2-1 says it has 192KB. Which one is correct? How to understand that myself?
2. I was able to verify that the CF2.1 firwire is utilizing SRAM1 only in its current version. What do I have to do to use SRAM2? Can I just augment my code and the additional resource will automatically go SRAM2? Please advise how to use SRAM2.

I have ran arm-none-eabi-size --format=SysV -x cf2.elf in command promt and got the following output (I thought this output may help you answer with more information):

Code: Select all

cf2.elf  :
section               size         addr
.isr_vector          0x188    0x8004000
.flashtext             0x0    0x8004188
.text              0x4d828    0x80041c0
.libc                  0x8    0x80519e8
.flashpatch            0x0          0x0
.endflash              0x0          0x0
.data               0x125c   0x20000000
.bss               0x19e48   0x20001260
.nzds                 0x28   0x2001b0a8
._usrstack           0x100   0x2001b0d0
.comment              0x31          0x0
.ARM.attributes       0x32          0x0
.debug_aranges      0x5f28          0x0
.debug_info       0x122dee          0x0
.debug_abbrev      0x2a4e2          0x0
.debug_line        0x56168          0x0
.debug_frame        0xfaa8          0x0
.debug_str         0xbe4ab          0x0
.debug_loc         0x4a9ee          0x0
.debug_ranges       0x9238          0x0
.debug_macro       0x3c2fb          0x0
Total             0x36fdbb
kimberly
Bitcraze
Posts: 1050
Joined: Fri Jul 06, 2018 11:13 am

Re: CF2.1 firmwire query for machine learning (ML) applications.

Post by kimberly »

Currently I'm unsure how much RAM is still left exactly, but we do know it is getting less and less. This issue on the github repo is also discussing this current problem.

About your questions:
1. The additional 64 kb is called CCRAM, which is used for uploading trajectories for the highlevel commander. There is a separated issue about that on the github repo here to discuss easier accessibility.
2. I think so yes, but I will need to double check this. I'm not sure how the SRAM1 and 2 are handled in the crazyflie.

I believe that the output you showed is more about the flash size than the amount of RAM used in runtime. If you go to the cfclient > Console tab, you can press the button 'task dump', which shows the CPU load and stack left. The latter indicates the load on RAM and might also be an good indication where we can lower the total stack if we assigned too much to a certain module.

I will try to come back with some info, but do also keep an eye on those discussions on the ticket.

Btw, are you aware of the AIdeck coming out soon (https://www.bitcraze.io/2019/10/sneak-p ... e-ai-deck/)
kimberly
Bitcraze
Posts: 1050
Joined: Fri Jul 06, 2018 11:13 am

Re: CF2.1 firmwire query for machine learning (ML) applications.

Post by kimberly »

here is some more info with some help of @kristoffer!

When the firmware is build, you see this information:

Code: Select all

text	  data	  bss	  dec	  hex	filename
 213344	  5148	 110236	 328728	 50418	cf2.elf
the bss stands for the ram used by variables. So these are allocated with static task macros (also queues and timers)

There are also task that are allocated dynamically like deck driver tasks which are not showed in the bss. They will be allocated on the heap. This can be seen in the taskdump that I explained before which indicates the stack usage per task.

Hope this is a bit clear?
imranmomtaz
Member
Posts: 36
Joined: Tue Mar 17, 2020 7:01 pm

Re: CF2.1 firmwire query for machine learning (ML) applications.

Post by imranmomtaz »

Hello,

I ran the 'task dump' and obtained the following:

Code: Select all

SYSLOAD: Task dump
SYSLOAD: Load	Stack left	Name
SYSLOAD: 2.98 	175 	Tmr Svc
SYSLOAD: 69.16 	127 	IDLE
SYSLOAD: 2.53 	207 	FLOW
SYSLOAD: 0.0 	75 	PWRMGNT
SYSLOAD: 1.01 	214 	CRTP-RX
SYSLOAD: 5.94 	169 	SENSORS
SYSLOAD: 2.15 	77 	ZRANGER2
SYSLOAD: 0.0 	220 	MEM
SYSLOAD: 0.0 	109 	LOG
SYSLOAD: 0.0 	91 	PARAM
SYSLOAD: 0.0 	177 	LH
SYSLOAD: 0.05 	158 	SYSTEM
SYSLOAD: 0.04 	62 	CRTP-TX
SYSLOAD: 2.96 	20 	SYSLINK
SYSLOAD: 4.64 	311 	STABILIZE
SYSLOAD: 8.47 	258 	KALMAN
SYSLOAD: 0.0 	113 	USBLINK
SYSLOAD: 0.0 	204 	CMDHL
Please help me understand the meaning of this. I see different task at the right, and there are stack left for each of the tasks. Does this mean this much stack is left in each task? Which one is the stack available to use? And, also what is the unit of 'stack left'? Is this in bits or bytes or something else?

Followup about previous questios:
1. Can I use CCRAM for my ML codes? If yes, I would like to be able to use this, and I am ok with quick and dirty way to do this. Please let me know what needs to be done to be able to solve this.
2. Could you confirm me how to use SRAM2?

Additional questions:
3. I am thinking about all ways to free up spaces. Could you give me a list of the files which are not used in the CF2.1 but are in the codebase anyway and taking up space? I am not sure if there is any but would like to confirm with you. I am using on regular PID controller (So, I think I can get rid of melinger controller code from firmwire, and so on and so forth). Am I thinking correctly? If yes, then can I curb extra code for other part (for example, I see there are different functions in Kalman filter which are defined for different sensors. But I am using only lighthouse and flow deck. So, I can remove the other parts, and save space right? This is not very convenient but would work for now.)

Thanks.
tobias
Bitcraze
Posts: 2339
Joined: Mon Jan 28, 2013 7:17 pm
Location: Sweden

Re: CF2.1 firmwire query for machine learning (ML) applications.

Post by tobias »

Hi,

I think using the CCM is the way to go. It can not be used for storing instructions, only data, but should not be a problem? I know we played a bit with this before but I can not find any traces of it in the code or linker files so it should be all free to use. The linker scripts needs to be updated and the variables using it needs to be attributed but should not be hard to do. Found two good resources here and here.
tobias
Bitcraze
Posts: 2339
Joined: Mon Jan 28, 2013 7:17 pm
Location: Sweden

Re: CF2.1 firmwire query for machine learning (ML) applications.

Post by tobias »

I was wrong about not being able to execute from the CCM. Found this AN4296 app note that explains the CCM in detail.
"The CCM SRAM is tightly coupled with the Arm® Cortex® core, to execute the code at the maximum system clock
frequency without any wait-state penalty. This also brings a significant decrease of the critical task execution time,
compared to code execution from Flash memory."
imranmomtaz
Member
Posts: 36
Joined: Tue Mar 17, 2020 7:01 pm

Re: CF2.1 firmwire query for machine learning (ML) applications.

Post by imranmomtaz »

Thank you. I will go through the material you shared, and will comeback if I have any queries. In the meantime, I have following query:
Where is the linker file located in the firmwire? I found several files (having .ld extension) in the whole directory.
Additionally, do I have update any startup file? If yes, where is that file in the firmware?

Also I asked following queries before but did not receive your answer.
Please help me understand the meaning of this. I see different task at the right, and there are stack left for each of the tasks. Does this mean this much stack is left in each task? Which one is the stack available to use? And, also what is the unit of 'stack left'? Is this in bits or bytes or something else?

2. Could you confirm me how to use SRAM2?

Additional questions:
3. I am thinking about all ways to free up spaces. Could you give me a list of the files which are not used in the CF2.1 but are in the codebase anyway and taking up space? I am not sure if there is any but would like to confirm with you. I am using on regular PID controller (So, I think I can get rid of melinger controller code from firmwire, and so on and so forth). Am I thinking correctly? If yes, then can I curb extra code for other part (for example, I see there are different functions in Kalman filter which are defined for different sensors. But I am using only lighthouse and flow deck. So, I can remove the other parts, and save space right? This is not very convenient but would work for now.)
tobias
Bitcraze
Posts: 2339
Joined: Mon Jan 28, 2013 7:17 pm
Location: Sweden

Re: CF2.1 firmwire query for machine learning (ML) applications.

Post by tobias »

The linker files in use for CF2.X are located here and the startup script here. If startup file might have to be changed to zero fill the CCM, this was described in one of the links.

2. I'm pretty sure SRAM2 (16k) is already in use. As I've understood it the reason for having this is so that the bus matrix could be utilized better e.g. when using DMA but it also requires designing the code so it has an effect. Not interesting for you I think.

3. It would be a quite big work to compile a list like that. There is plenty of flash so no reason to exclude code for that but could free static mem. Is the 64k CCM not enough?
Post Reply