Assert on reconnect via Crazyradio

[gemenerik]

There have been occasional reports of an assert occurring when reconnecting to the Crazyflie from the client or lib via Crazyradio. It seems more likely to happen when reconnecting multiple times in quick succession, though reproduction is currently unreliable.

This issue is being tracked based on past observations and recent community discussions (e.g., at ICRA). Additional insights or reproduction steps from others would be very helpful.

[matejkarasek]

Hi, here one example (Bolt 1.0 - Flapper platform, with Lighthouse deck), the assert happens shortly after power-up when attempting to connect from cfclient.
We are getting many of these, also when connecting from cflib.
The assert is (more often) in radiolink, will post here when it occurs again

Here the Console:

SYS: Flapper (Bolt 1.0) is up and running!
SYS: Build 6:28f4efe0d77a (2024.10 +6) CLEAN
SYS: I am 0x20373347344D500E002B002B and I have 1024KB of flash!
CFGBLK: v1, verification [OK]
DECK_INFO: Ignoring all OW decks because of compile flag.
DECK_INFO: CONFIG_DECK_FORCE=bcLedRing:bcFlapperDeck:bcLighthouse4 found
DECK_INFO: compile-time forced driver bcLedRing added
DECK_INFO: compile-time forced driver bcFlapperDeck added
DECK_INFO: compile-time forced driver bcLighthouse4 added
DECK_CORE: 3 deck(s) found
DECK_CORE: Calling INIT on driver bcLedRing for deck 0
DECK_CORE: Calling INIT on driver bcFlapperDeck for deck 1
DECK_CORE: Calling INIT on driver bcLighthouse4 for deck 2
IMU: BMI088: Using SPI interface.
IMU: BMI088 Gyro connection [OK].
IMU: BMI088 Accel connection [OK]
IMU: BMP388 I2C connection [OK]
ESTIMATOR: Using Kalman (2) estimator
CONTROLLER: Controller type forced
CONTROLLER: Using PID (1) controller
PWR_DIST: Using Flapper power distribution | PCB revD (2022) or newer
MTR-DRV: Using brushless motor driver
SYS: About to run tests in system.c.
SYS: NRF51 version: 2024.10 (CB10)
EEPROM: I2C connection [OK].
STORAGE: Storage check [OK].
IMU: BMI088 gyro self-test [OK]
DECK_CORE: Deck 1 test [OK].
SYS: Self test passed!
STAB: Wait for sensor calibration...
SYS: Free heap: 15168 bytes
LHFL: Lighthouse bootloader version: 2
LHFL: Bitstream CRC32: 112BC794 [PASS]
LHFL: Firmware version 6 verified, booting deck!
STAB: Starting stabilizer loop
ESTKALMAN: State out of bounds, resetting
SUP: Can not fly
SUP: Ready to fly
SYS: ----------------------------
SYS: Flapper (Bolt 1.0) is up and running!
SYS: Build 6:28f4efe0d77a (2024.10 +6) CLEAN
SYS: I am 0x20373347344D500E002B002B and I have 1024KB of flash!
CFGBLK: v1, verification [OK]
DECK_INFO: Ignoring all OW decks because of compile flag.
DECK_INFO: CONFIG_DECK_FORCE=bcLedRing:bcFlapperDeck:bcLighthouse4 found
SYS: ----------------------------
SYS: Flapper (Bolt 1.0) is up and running!
SYS: Build 6:28f4efe0d77a (2024.10 +6) CLEAN
SYS: I am 0x20373347344D500E002B002B and I have 1024KB of flash!
CFGBLK: v1, verification [OK]
DECK_INFO: Ignoring all OW decks because of compile flag.
DECK_INFO: CONFIG_DECK_FORCE=bcLedRing:bcFlapperDeck:bcLighthouse4 found
DECK_INFO: compile-time forced driver bcLedRing added
DECK_INFO: compile-time forced driver bcFlapperDeck added
DECK_CORE: 3 deck(s) found
DECK_CORE: Calling INIT on driver bcLedRing for deck 0
DECK_CORE: Calling INIT on driver bcFlapperDeck for deck 1
ver bcLighthouse4 for deck 2
IMU: BMI088: Using SPI interface.
IMU: BMI088 Gyro connection [OK].
IMU: BMI088 Accel connection [OK]
IMU: BMP388 I2C connection [OK]
ESTIMATOR: Using Kalman (2) estimator
CONTROLLER: Controller type forced
CONTROLLER: Using PID (1) controller
PWR_DIST: Using Flapper power distribution | PCB revD (2022) or newer
MTR-DRV: Using brushless motor driver
SYS: About to run tests in system.c.
SYS: NRF51 version: 2024.10 (CB10)
EEPROM: I2C connection [OK].
STORAGE: Storage check [OK].
IMU: BMI088 gyro self-test [OK]
DECK_CORE: Deck 1 test [OK].
SYS: The system resumed after a failed assert [WARNING]
SYS: Assert failed at ../src/drivers/src/uart_syslink.c:549
SYS: cfAssertNormalStart [FAIL]

[matejkarasek]

And here the more common one:

SYS: Build 6:28f4efe0d77a (2024.10 +6) CLEAN
SYS: I am 0x20373347344D500E002A0026 and I have 1024KB of flash!
CFGBLK: v1, verification [OK]
DECK_INFO: Ignoring all OW decks because of compile flag.
DECK_INFO: CONFIG_DECK_FORCE=bcLedRing:bcFlapperDeck:bcLighthouse4 found
DECK_INFO: compile-time forced driver bcLedRing added
DECK_INFO: compile-time forced driver bcFlapperDeck added
DECK_INFO: compile-time forced driver bcLighthouse4 added
DECK_CORE: 3 deck(s) found
DECK_CORE: Calling INIT on driver bcLedRing for deck 0
DECK_CORE: Calling INIT on driver bcFlapperDeck for deck 1
DECK_CORE: Calling INIT on driver bcLighthouse4 for deck 2
IMU: BMI088: Using SPI interface.
IMU: BMI088 Gyro connection [OK].
IMU: BMI088 Accel connection [OK]
IMU: BMP388 I2C connection [OK]
ESTIMATOR: Using Kalman (2) estimator
CONTROLLER: Controller type forced
CONTROLLER: Using PID (1) controller
PWR_DIST: Using Flapper power distribution | PCB revD (2022) or newer
MTR-DRV: Using brushless motor driver
SYS: About to run tests in system.c.
SYS: NRF51 version: 2024.10 (CB10)
EEPROM: I2C connection [OK].
STORAGE: Storage check [OK].
IMU: BMI088 gyro self-test [OK]
DECK_CORE: Deck 1 test [OK].
SYS: Self test passed!
STAB: Wait for sensor calibration...
SYS: Free heap: 15168 bytes
LHFL: Lighthouse bootloader version: 2
STAB: Starting stabilizer loop
ESTKALMAN: State out of bounds, resetting
SUP: Can not fly
SUP: Ready to fly
LHFL: Bitstream CRC32: 112BC794 [PASS]
LHFL: Firmware version 6 verified, booting deck!
SYS: ----------------------------
SYS: Flapper (Bolt 1.0) is up and running!
SYS: Build 6:28f4efe0d77a (2024.10 +6) CLEAN
SYS: I am 0x20373347344D500E00ks because of compile flag.
DECK_INFO: CONFIG_DECK_FORCE=bcLedRing:bcFlapperDeck:bcLighthouse4 found
DECK_INFO: compile-time forced driver bcLedRing added
DECK_INFO: compile-time forced driver bcFlapperDeck added
DECK_INFO: compile-time forced driver bcLighthouse4 added
DECK_CORE: 3 deck(s) found
DECK_CORE: Calling INIT on driver bcLedRing for deck 0
DECK_CORE: Calling INIT on driver bcFlapperDeck for deck 1
DECK_CORE: Calling INIT on driver bcLighthouse4 for deck 2
IMU: BMI088: Using SPI interface.
IMU: BMI088 Gyro connection [OK].
IMU: BMI088 Accel connection [OK]
IMU: BMP388 I2C connection [OK]
ESTIMATOR: Using Kalman (2) estimator
CONTROLLER: Controller type forced
CONTROLLER: Using PID (1) controller
PWR_DIST: Using Flapper power distribution | PCB revD (2022) or newer
MTR-DRV: Using brushless motor driver
SYS: About to run tests in system.c.
SYS: NRF51 version: 2024.10 (CB10)
EEPROM: I2C connection [OK].
STORAGE: Storage check [OK].
IMU: BMI088 gyro self-test [OK]
DECK_CORE: Deck 1 test [OK].
SYS: The system resumed after a failed assert [WARNING]
SYS: Assert failed at ../src/hal/src/radiolink.c:166
SYS: cfAssertNormalStart [FAIL]

[ataffanel]

Thanks a lot @matejkarasek for the logs. We are going to look at it next week.

[matejkarasek]

Thanks @ataffanel !
Some more information on our setup

• disabling OW and forcing LH and other decks was an attempt to see if it reduces the occurrence of these asserts (it does not seem like it)
• the LH deck is connected to the Bolt via an extension cable (all 20 pins), except for these asserts it performs as expected
• the firmware is based on clean 2024.10, the commits on top of that are only related to LED ring effects and platform settings
• we have not been flying with the latest firmware (for a drone show case we prefer not to change things too often), will try and report back, but have been seeing this behavior for quite some time and also with older firmware versions
• some of our drones seem to be more prone to these than others, suggesting this could be caused by minor differences in hardware, like better/worse electrical contact somewhere?

[ArisMorgens]

Hello @matejkarasek!

I tried to reproduce this issue using a Crazyflie 2.1+ with firmware from the latest release (2025.02) and a Lighthouse deck but didn't get any assert fails. I tested it both through the cfclient by manually connecting and disconnecting, and through the cflib. The script that I used connects to the Crazyflie and when it is fully connected (all parameters are updated), it disconnects. This happens for 100 times in a row.

Personally, I only came across this issue only when I tried to implement a custom deck to the Crazyflie, meaning that it might indeed be a hardware issue. Did you experience this when using platforms other than the Flapper?

[matejkarasek]

Hi @ArisMorgens !
Thanks for looking into this. I must say we do not use standard Crazyflies enough to observe this... But the hardware we are using is basically a Bolt and LH deck connected via extension cables. You could simulate this on your end when taking a Bolt with LH deck and flashing it with the clean flapper platform firmware.

If you could share your script, we can try to run it here on the specific hardware setup where we see the assert happening most frequently.

If this is caused by hardware, I am curious about what could be the mechanism behind. As I mentioned, if the system boots up correctly, then there are no more issues with the Lighthouse tracking in flight... If this would be caused by a bad contact somewhere, I would expect this to get worse in flight... The extensions cable will increase slightly the resistance of the individual lines, but could this lead to these asserts? Is there anything in the boot/startup sequence that could be more sensitive to this?

[ArisMorgens]

Hello again!

I have "good" news: I managed to reproduce the issue using the bolt with a Lighthouse deck. I also used flapper firmware 2024.10.02.
I ran my script that continuously connects/disconnects to the drone and after 23 successful connections, an assert failed.

After that, I connected to the cfclient and got the same assert info as you did:
SYS: Assert failed at ../src/hal/src/radiolink.c:166

Now that we can reproduce it, I think the radio experts will take over and investigate the issue.

Here is the simple script that I used.

import time

import cflib
from cflib.crazyflie import Crazyflie
from cflib.crazyflie.swarm import CachedCfFactory
from cflib.crazyflie.syncCrazyflie import SyncCrazyflie
from cflib.utils import uri_helper

URI = uri_helper.uri_from_env(default='radio://0/80/2M/E7E7E7E7E7')


if __name__ == '__main__':
    cflib.crtp.init_drivers()

    factory = CachedCfFactory(rw_cache='./cache')

    for i in range(100):
        try:
            with SyncCrazyflie(URI, cf=Crazyflie(rw_cache='./cache')) as scf:
                print(f'Connected = {scf.is_link_open()}. Fully connected: {scf.is_params_updated()}. ', end='')
                while scf.is_params_updated() is False:
                    print('.', end='')
                    time.sleep(0.1)
                print(f'Fully connected: {scf.is_params_updated()}. Attempt number: {i}  ')
                scf.close_link()
                time.sleep(0.5)
                print(f'Connected = {scf.is_link_open()}')

        except Exception:
            print(f'Something went wrong. Attempt number: {i}')

[matejkarasek]

Hi, thanks a lot for trying it, and great that you managed to reproduce this, hopefully it will give you enough clues to see what is going on...
In the meantime, we will try with your script, also with the latest firmware, and report back... Due to other commitments and holidays, this may only be in August...