Latency Test for deviation vs ersky vs opentx

That's over 400Hz ? sure?

Still it is important in any case, that always "freshest possible data" is entered into the protocol, just in time, synced without buffering.....

It is interesting what Hexfet are doing, best would be that it would be generic/reliable fix, so for any protocol it would work benefit.

Hi Hexfet,
I updated the latency test to include that test build. It looks to be about ~1ms slower then the hack I did. Could the 5ms callback just be updated to 1ms and have it affect all the protocols? It seems like that should(?) give you roughly the same results but without having to touch all the protocols.


Also included the redpine protocol I've been working on.

www.deviationtx.com/forum/protocol-devel...r-the-lowest-latency

Fernadez, it is around 715hz right now. It would be fun to get it up to 1kHz

Updating the mixers every millisecond is not workable for a couple reasons. The model I tested with many mixers takes more than 1 ms to run the mixer update code. Even if it takes half a millisecond some time is still needed to run screen updates, button scans, etc. Also need to consider that the mixer update is paused by the protocol callback interrupt, extending its run time.

The mixer update being interrupted by the protocol callback is even an issue in the current code because it means sometimes the Channels values are used when they're only partially updated (some channels updated with the most recent inputs, others not). In my opinion the packet updates happen so quickly that this is never noticed when flying. Still it would be more technically correct to have the mixer update synchronized with the protocol so the channel values are all updated when used. And that's the best kind of correct...

When implementing protocols that schedule the mixer calculations this interrupt overlap has to be considered. I've done Devo and SFHSS also. Since Devo has such a fast update rate the mixer is only updated once for each two packets sent. With SFHSS I changed the state machine timing a bit to fit in the mixer update. For the moment I'm putting in fixed delays, and don't mind an extra millisecond or two left on the table. May look at updating timing dynamically after this is all working.

Test build reduce_latency (21674bd) has these changes. Only FrskyX and SFHSS are tested. All the protocols should work in this build - ones without optimized timing still use the 5ms periodic mixer updates. The source is here and this is the branch that will turn into a PR if all goes well. Comments invited.

Also plan to update Flysky and Frsky since I can test those but running out of free time for this month...

Hi hexfet,
Could the channel data be be double buffered (ping/pong buffer)? You only point to the new buffer when the mixer is completely finished so you don't get 1/2 updated packets. You'll always switch the pointer at the end of the mixer calculation so you'll always have the newest complete packet.
That might allow the speed to be bumped up in the callback without breaking other things.
Could also the drop the calculation down to the main loop and let it run as fast as possible while letting the buttons and screen updates run periodically.

Just a couple ideas, I know your are busy!

Bryce

Double-buffering solves the synchronization issue but doesn't help latency. The data is still either brand-new or an update-interval old, just not a mix. Just running the mixers fast as possible is not attractive when a synchronous solution is at hand. That's a lot of wasted calculations, and if it takes a millisecond to run them the latency will never get below that.

Still think a synchronous solution is best, even though it requires modifying the protocols. It's not hard to get better latency performance than we have now (for the optimized protocols). It doesn't waste cpu cycles. It has the potential for the lowest latency in all situations (by adjusting timing according to mixer runtime).

Have the code modified for Frsky (D8) and Flysky, which are the other two I can test. Will update test build after testing.

The reduce_latency test build (5220847) is updated. Tested the following protocols with reduced latency code: FrskyX, Frsky, SFHSS, Devo, Flysky.

These are the only hobby protocols I'm able to test. Any volunteers to test other hobby protocols, shout 'em out. Otherwise I plan to make a PR based on these changes.

Hi Hexfet, could you give a risk rating for this testbuild?
Is it to be tested on foamy and whoops?

Any particular things to verify, simple control and with advanced models mixers curves?

It's fairly low risk, but be sure to check all controls on the ground before flying. Observable behaviour should be exactly the same as for the nightly build. The latency-optimized protocols will have a few milliseconds less latency than the nightly, but don't think that will be noticeable while flying.

Test build is updated ( ffba662 ). Fixed Frsky telemetry. Fixed an issue that caused the mixer calculations to be run by protocol and timer at the same time. Not sure this would explain the large deviation in latency measurements. Haven't gotten to measuring within the tx yet.

I've done some timing testing. Still not sure why the min-max difference is large in brycej's tests, but can explain the minimum and maximum latencies within the radio. Keep in mind the numbers here don't include time through a receiver and flight controller, which are included in brycej's measurements. tl;dr the current reduce_latency test build is close to best latency for the modified protocols.

The test rig drives the throttle analog input pin to either 0 or 3 volts. One surprise was that the time spent in ADC conversion is more than what I thought. Didn't dig enough on my first look. On the t8sg with 6 analog inputs one ADC conversion cycle takes 1.26 milliseconds (ms). Depending on when the analog input changes within a conversion cycle the delay for having enough samples to cause a change in ADC filter output would be between about 0.8 to 1.89 ms. (This is for a min-to-max or max-to-min input value change that's detected by watching for a mid-point crossing)

The total delay from analog input change to data change in protocol output is the sum of ADC conversion time, ADC filter and mixer calculation time, and time needed to build the packet data and deliver to the radio. To test on the bench it was easier to send the data packet and detect the channel change with an external box, which adds the protocol packet transmission time to the total, plus the time in the test jig to decode the protocol.

I tested with serial SBUS which has a 3 ms packet transmit time. The test jig needed about 500 microseconds (us) to decode the SBUS packet and detect the channel change. From the previous discussions we know a simple 4-channel model needs about 350-400 microseconds (us) for ADC filter and mixer calculations. So, for t8sg, the minimum predicted delay is 0.8 + 0.35 + 3.0 + 0.5 = 4.65 ms. The best measured value during testing was 4.7 ms.

To consider the worst possible delay the packet period comes into play. The worst delay happens when the input change is just barely missed for the current timing cycle. The input changes just after halfway through an ADC conversion cycle so it can't be detected until another ADC conversion cycle is complete - delay about 1.8 ms. Thus the change will be missed by the current ADC filter and mixer calculation run (400us). The mixer calculation doesn't run again until a packet period has elapsed (for testing I set 6ms period for SBUS). Then a packet transmission time before it can be detected at the receiver (3ms for SBUS). So for SBUS with 6ms packet period the max delay before a change arrives at the test rig is 1.8 + 0.4 + 6 + 3 = 11.2ms. The measured value in testing was 11ms.

For CRSF with packet period 4ms and serial data transmit time of 0.65ms the best and worst values should be about 2.3 and 6.85ms. This is latency from input change to delivering updated serial data to the radio. Additional delays would be introduced in the crossfire transmitter, receiver, and the flight controller.

The ADC conversion time could be reduced with ADC configuration changes but without knowing how the current configuration was chosen it's not something to change without thorough testing. I think we'd need some kind of "ADC quality test", then get a baseline with the current configuration, then experiment with changes. The ADC configurations that could be changed are ADCCLK prescaler, sampling time setting, and number of samples per cycle. The ADC_Filter algorithm might be another place for improvement - maybe there's something better than simple average.

Here are the ADC details. The ADCCLK is derived from PCLK2 which is running at 72MHz on the t8sg (and all other radios best I can tell). Deviation configures the prescaler to divide by 6 which puts the ADCCLK at 12MHz. The input sample time can be adjusted per channel. On all transmitters except 12s it is set to 239.5 cycles. On 12s it is set to 13.5 cycles. Conversion time is sample time + 12.5cycles. So for 12s the single channel conversion time is 2.17us, while on other transmitters it is 21.0us. However the 12s is configured to collect 100 samples per conversion cycle while other transmitters collect 10, so the conversion cycle times are about the same (1.26 ms).

I'd like to test with a receiver in the loop to see why deviation has the large...deviation, but out of time for now.

Interesting research you did!

Especially the ADC part is interesting, would that be possible to synchronize to the protocol loop, taken into account mixer processing, so always freshest possible sample being sent.
Averaging also poor for latency....

Synchronization is possible. It would reduce the max latency for ADC to 1.26 ms (from about 1.8 ). Some old commented-out code indicates something along this line might have been considered. Planning to look at why there's the large delay spread before thinking about ADC changes.

Just out of curiosity is there any news on this reduce latency to minimum topic?

@Bryce, more SPI boards are coming, f.i. the new Brushless Whoops like mobula7.
Did you do some more testing with your "Redpine fast protocol", will it pulled into Betalight and Deviation?

I'm traveling unexpectedly and unable to test. Probably be several more weeks before landing pull requests. A couple other changes are mixed into the reduce_latency branch and need to be separated.

I have a mobula7 and it works great. The updated 2s board is a crazybee f3 pro that uses the same firmware as the 1s version.

I've been flying pretty much exclusively with the redpine protocol for the last month, so it works well. Both with the crazybee boards and the matek board. No telemetry yet, but I'll probably skip that for the first release. I was working on the multi protocol version a bit last weekend and wanted to have that in as well for getting it out there.

Brycej, did you go down to 1kHz?

About 666-715hz was stable (1.4ms-1.5ms a packet) with the mixer calculation preformed before each packet in my branch. The time to do the mixer calucation and ADC sampling was too long to bring it to 1khz. When I unsynced the mixer calc from the protocol, I could send at 1khz, but it was likely sending redundant data. We need to put a STM32F4 or F7 on there Or maybe there is other optimizations to be had. The RF looptime is flexible so you can adjust it in the options.

The Deviation changes to reduce latency have been merged and will be in the next nightly build.

The protocols with optimized latency are: CRSF, DEVO, DSM2, DSMX, Flysky, Frsky, FrskyX, SBUS, and SFHSS.