High frequency streaming

[david]

Hi everyone,

I'm a student and I just begun a project in my university where we are trying to stream audio from a microphone which would be added to the Crazyflie 2.0 using the Crazyradio to stream the sound.

For now my top priority is to understand how to achieve the highest possible transmission rate.

I have gone through all the documentation I found, search the forum topics and read the Camera Addon master thesis which has a lot of information on the Crazyflie/Crazyradio communications.

Until now I have been exploring the possibilities of logging but, to my understanding, logging is limited to 100 Hz and even by removing the multiplication by 10 mentioned here it only gets to 1000 Hz. Due to this I'm now moving away from logging and focusing on understanding how to use the communication protocol at a lower level so I can better control the transmission rate.

From the Camera Addon master thesis I know I can expect an effective throughput from the Crazyflie to the computer of about 200 Kbps.

My questions are:

1. is anyone working on high frequency streaming from the Crazyflie?

• is there any project that I can look into that might help?

• is any of the code used in the Camera Addon master thesis available?

• is there a better way to understand how to use the protocol other than going though the crtp.c/log.c code and the explanation of the CRTP found here?

Please feel free to comment on any thing regarding the project.

Thank you,
David

[tobias]

Hi David and welcome to the forum!

Interesting project you are looking into. Something we have been wanting to do for a long time as well, just no time to do it.

Depending on how much time you have got I guess there are different approaches. To get to the full bandwidth I think the USB driver would have to be improved as it only can do 1000 packets/sec and 32byte packets (for simplicity). This is not obvious to do either but definitely doable. For the master thesis I think they only tested the radio bandwidth and never forwarded the data over usb.

Other option would be to use some low bitrate transfer and not modify the radio drivers. You could either use the memory port and create a audio memory to read the data (maybe not so efficient) or create a new streaming port.

I think we have the source code from the master thesis somewhere but don't know the state of it and the license. Could you send us an email to: contact (at) bitcraze.io and let's take that there.

[david]

Hi Tobias!

First of all thank you for the reply!

I'm aware that to achieve the full bandwidth I would have to change the USB firmware, I have gone through the code and it didn't seamed to be that easy to understand, that is one of the reasons why I was asking if the code of the master thesis was available. Since is mentioned that they changed the code in the radio and after display results for the "Effective throughput from quadcopter to computer at different radio link rates and distances.", I assumed that they had coded something similar to what I'm trying to achieve.

I believe the way to go is to create a new streaming port, but still haven't started working with the microphone, so I'm not sure yet. Is that something that is hard to do?

In the case of using a streaming port, I would have to then send the data to the computer. To do this using the Crazyradio I would have to go around logging and send the packages directly right?

Thanks!

[arnaud]

Hi,

I also think a new streaming crtp port is the way to go. It is fairly easy to do you just have to call the crtpSendPacket function using an unused port: https://github.com/bitcraze/crazyflie-f ... #L104-L118, the packet is then going to be sent.

On the computer side, the packet will be received by the python code in the CRTP stack and you can setup a function callback to be called each time a packet arrives at your port, for example you can look at how the console port is implemented: https://github.com/bitcraze/crazyflie-l ... ole.py#L52.

This can be achieved with the current Crazyradio and maybe you can give it a try to see what kind of bandwidth you can get without any modification (please report the results here .

For Crazyradio, I think the master thesis guys implemented the double buffering functionalities of USB endpoints. I am not so sure of what they did since I was not involved in the thesis back then. Tobias is going to try to recover the code.

For double buffering to be really efficient I believe we need to start streaming data over the USB port (instead of putting one radio packet per USB packet as we do right now, USB interprets a non-full packet as end of transition). I have been looking a bit at the radio lately and my current idea would be to implement a usb serial port and to stream packed and settings on this serial port (in a similar way to what we do between the two CPU in Crazyflie). If we keep a buffer for input and output data in the Crazyradio this should allow to remove the USB bottleneck.

[david]

Hi Arnaud,

Thank you for your answer!
I have managed to implement the streaming port with the information that you provided, I'll leave the code here just in case someone needs it. The first byte of data is just a number that I use to track the sent packets.

Code: Select all

static void micTask(xTimerHandle timer)
{
  fillBuffer();
  // Copies data to package
  CRTPPacket p;
  p.header = CRTP_HEADER(STREAM_PORT, STREAM_CHANNEL);
  p.size = DATA_BYTES + COUNT_BYTES;
  p.data[0] = packetCount;
  memcpy(&(p.data[1]), dataBuffer, DATA_BYTES);
  crtpSendPacket(&p);
  packetCount++;
}

This function is then being called by a timer with a predetermined frequency Fx.

I have done a few tests and for now, and without any modifications to the Crazyradio, these are my results:
(The packets I'm sending have 30 Bytes of payload)
CRBrandwidth | Fx | % of received packets | Effective bandwidth = Fx*%*30*8
1M | 1000 Hz | 47 % | 112.8 Kbps
2M | 200 Hz | 99 % | 47.5 Kbps
2M | 500 Hz | 97 % | 116.4 Kbps
2M | 1000 Hz | 55 % | 132 Kbps

Are this values what you were expecting? Regarding the Crazyradio, I really don't know how USB works but what you said makes sense to me, but if I change the way data is received in the Crazyradio, I'll have to change the way received data gets unpacked by the cflib right? I'll look into the Crazyradio code and I'll get back to you about it.

I now have a few questions that hopefully you can answer:

1. When I looked into the CRTPPacket structure I found that the <data> vector size is 30 Bytes and not the 31 Bytes I was expecting. Yet there is another vector <raw> which has the size of 31 Bytes. Can you please explain what the vector <raw> is for and how can I achieve the 31 Bytes of payload?

• Regarding logging, since I haven't asked the Crazyflie to log anything from the computer side, can I assume that it isn't sending any logging information?

• In the master thesis I read something about the number of retries to send a packet. Is this value hard coded into the Crazyradio, or can I specify how many times the Crazyradio has to try to send a packet before is drooped? I ask this because in audio streaming sometimes is more important to continue to receive data than the integrity of the data itself.

Thank you again!

[arnaud]

Hi,

Thanks a lot for the measurements! I was expecting either 1000 or 500 packets per second (it depends how the USB host is scheduling packets ...), apparently it is more like 500. This is a good benchmark of the current state, radio-wise we should be able to do better.

For the Crazyradio I have started looking at re-implementing double buffering. I will keep you updated if I get anywhere. Are you interested in testing alpha firmware and do you have more than one radio?

Since we are only going to change the very low level of the radio there is no reason there will be to be changes higher up in the stack. I expect that the only files to change are going to be crazyradio.py and the crtp radiodriver.py.

For the questions:
1. In theory, the radio packets are 32Bytes, we use 1Byte for the header and so the data part should be 31Bytes. However there has been some confusion very early in development that lead to 30Bytes data length being used in the firmware. Each time we have tried to fix it we broke the link and since no-one used very high bandwidth on this link there has been no big incentive to fix it. So to summarize this is a bug .

2. If you connect Crazyflie using the lib there should be no log setup automatically.

3. These values are Crazyradio parameters. It is set in the radio crtp driver: https://github.com/bitcraze/crazyflie-l ... er.py#L194. It is also possible to set the time to wait for a retry (there should be enough time for the full ack packet to come back, we use ack packets to carry downlink data): https://github.com/bitcraze/crazyflie-l ... #L196-L219. If you change the number of ack rety and see a difference of bandwidth then we are not yet at the limit of USB, I expect you are running in good radio condition though so I do not expect that to be the bottleneck for now.

[david]

Thank you for your reply!

I just noticed that I forgot to mention which version of the Crazyradio PA firmware I'm using. It is v0.53, I still haven't got around to compile and flash the lasted firmware , so I'm not sure how up-to-date it is and I only found this change log which only goes up to v0.52 so I'm a bit confused. Once I get everything running in the Crazyflie I'll take a better look at how the USB scheduling is being done and why is closer to the 500 than to 1000.

Thank you. Yes, I'm interested in testing Alpha firmware, I think I can get one more Crazyradio.

Meanwhile and since I'm still working at a relatively low sampling frequency, I have been looking into how to increase it. Until now I have been using the xTimerCreate function to create a timer to sample periodically an analog signal connected to the Crazyflie. Since the configTICK_RATE_HZ of the FreeRTOS is set to 1000 Hz this is the maximum frequency at which I can use this configuration. (I have looked at ways to increase the configTICK_RATE_HZ and I have manage to do it, but to my understanding this is not the proper way to do it, so I'm looking into alternatives.) If you have any idea of how I can achieve this please feel free to comment, I have been exploring CMSIS and the TIMx timers provided by the STM MCU but I'm still not convinced I'm looking in the right place.

Thanks again.

[arnaud]

Hi,

For the radio it is possible to 'brick' it when developing with the USB so this was the reason of my question. There is way to unbrick it using a raspberry pi or Arduino but it is not very convenient yet. I was using an SPI programmer at the beginning of the development, I am looking at making a new one using an Arduino.

For the analog acquisition this is indeed not the way to do it, I would advise against touching configTICK_RATE_HZ ! I will ask Tobias to pass-by since he has been the one to play with ADC in the past but the solution most likely involve a timer: it is possible to either launch the ADC from a hardware timer interrupt or from the timer itself directly. It is even possible to use the DMA to automatically write the reading in memory and trigger an interrupt when the buffer is full, this way you can have a precise fast sampling rate without overloading the CPU.

Edit: Out of curiosity, what sample rate do you need? With 500 packet/sec you can already have usable sound transfers with 8bit at 8KHz

[david]

Hi,

I've read about it and have seen your hack in hackster.io. It doesn't look convenient, but at least we have a way to unbrick it if we ever get to it and don't have an SPI programmer! I guess using a Bus Pirate or something like it is much more convenient.

Yeah, only after I read a bit more about it I realized what I was doing. FYI i think it worked at least until the 10000 Hz. I just realized that, I'm now looking into this example and I think I'll give it a try. I'm just a bit worried about the integration of everything since I haven't done anything like this and don't have any experience working at such low level with MCU's.

That will depend on the application which will depend on how much we can get out of the Crazyradio. In the beginning we weren't really aware of the constraints and the idea was to stream audio from 2 or more microphones to do Sound Source Localization and for this we would need a higher sampling frequency, maybe around 20 KHz maybe more since the microphones are close together. Once we got a better idea of the constraints we thought about just streaming audio from one microphone and use the Crazyflie in collaborative robotics and for that a lower sampling frequency would be ok. We are still exploring possibilities and the final focus will depend on the maximum sampling frequency and bandwidth we can achieve. That is why I'm focusing on increasing the sampling frequency and once I saturate the Crazyradio bandwidth, on how to increase it.

Thanks again for taking the time to reply.

[RickyTerzis]

Hi...i am a new user here. As per my knowledge to get to the full bandwidth I think the USB driver would have to be improved as it only can do 1000 packets/sec and 32byte packets. This is not obvious to do either but definitely doable. For the master thesis I think they only tested the radio bandwidth and never forwarded the data over usb.