ptp4l broken in 6.12.25

[catubc]

Describe the bug

Hello
I've got dozens of rpis on a ptp4l network with a dell workstation master. Seems the rpis with the 6..6.74 kernel synchronize well (the switch to listening mode) but the 6.12.25 kernel RPis aren't doing so. It's the same issue with the previous kernel that was supposed to be fixed in 6.12.10 I believe.

I posted the issue here with a printout and a link to the previous kernel bug.

https://forums.raspberrypi.com/viewtopic.php?t=389111

Steps to reproduce the behaviour

Run ptp4l on a raspbbery pi with 6.12.25 kernel.

Device (s)

Raspberry Pi CM5

System

raspinfo.txt

Logs

The logs are:

cat@raspberrypi:~ $ sudo ethtool -i eth0
driver: macb
version: 6.12.25+rpt-rpi-2712
firmware-version:
expansion-rom-version:
bus-info: 1f00100000.ethernet
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no

cat@raspberrypi:~ $ sudo ptp4l -i eth0 -f /etc/ptp4l.conf -m -l 7
ptp4l[9949.499]: config item (null).assume_two_step is 0
ptp4l[9949.499]: config item (null).check_fup_sync is 0
ptp4l[9949.500]: config item (null).tx_timestamp_timeout is 1
ptp4l[9949.500]: config item (null).hwts_filter is 0
ptp4l[9949.500]: config item (null).clock_servo is 0
ptp4l[9949.500]: config item (null).clock_type is 32768
ptp4l[9949.500]: config item (null).clock_servo is 0
ptp4l[9949.500]: config item (null).clockClass is 248
ptp4l[9949.500]: config item (null).clockAccuracy is 254
ptp4l[9949.500]: config item (null).offsetScaledLogVariance is 65535
ptp4l[9949.500]: config item (null).productDescription is ';;'
ptp4l[9949.500]: config item (null).revisionData is ';;'
ptp4l[9949.500]: config item (null).userDescription is ''
ptp4l[9949.500]: config item (null).manufacturerIdentity is '00:00:00'
ptp4l[9949.500]: config item (null).domainNumber is 0
ptp4l[9949.501]: config item (null).slaveOnly is 1
ptp4l[9949.501]: config item (null).gmCapable is 1
ptp4l[9949.501]: config item (null).gmCapable is 1
ptp4l[9949.501]: config item (null).G.8275.defaultDS.localPriority is 128
ptp4l[9949.501]: config item (null).maxStepsRemoved is 255
ptp4l[9949.501]: config item (null).time_stamping is 1
ptp4l[9949.501]: config item (null).twoStepFlag is 1
ptp4l[9949.501]: config item (null).twoStepFlag is 1
ptp4l[9949.501]: config item (null).time_stamping is 1
ptp4l[9949.501]: config item (null).priority1 is 248
ptp4l[9949.501]: config item (null).priority2 is 248
ptp4l[9949.501]: interface index 2 is up
ptp4l[9949.501]: config item (null).free_running is 0
ptp4l[9949.501]: selected /dev/ptp0 as PTP clock
ptp4l[9949.501]: config item (null).clockIdentity is '000000.0000.000000'
ptp4l[9949.502]: config item (null).uds_address is '/var/run/ptp4l'
ptp4l[9949.502]: section item /var/run/ptp4l.announceReceiptTimeout now 0
ptp4l[9949.502]: section item /var/run/ptp4l.delay_mechanism now 0
ptp4l[9949.502]: section item /var/run/ptp4l.network_transport now 0
ptp4l[9949.502]: section item /var/run/ptp4l.delay_filter_length now 1
ptp4l[9949.502]: config item (null).free_running is 0
ptp4l[9949.502]: config item (null).freq_est_interval is 1
ptp4l[9949.502]: config item (null).write_phase_mode is 0
ptp4l[9949.502]: config item (null).gmCapable is 1
ptp4l[9949.502]: config item (null).kernel_leap is 1
ptp4l[9949.502]: config item (null).utc_offset is 37
ptp4l[9949.502]: config item (null).timeSource is 160
ptp4l[9949.502]: config item (null).pi_proportional_const is 0.000000
ptp4l[9949.502]: config item (null).pi_integral_const is 0.000000
ptp4l[9949.502]: config item (null).pi_proportional_scale is 0.000000
ptp4l[9949.503]: config item (null).pi_proportional_exponent is -0.300000
ptp4l[9949.503]: config item (null).pi_proportional_norm_max is 0.700000
ptp4l[9949.503]: config item (null).pi_integral_scale is 0.000000
ptp4l[9949.503]: config item (null).pi_integral_exponent is 0.400000
ptp4l[9949.503]: config item (null).pi_integral_norm_max is 0.300000
ptp4l[9949.503]: config item (null).step_threshold is 0.000000
ptp4l[9949.503]: config item (null).first_step_threshold is 0.000020
ptp4l[9949.503]: config item (null).max_frequency is 900000000
ptp4l[9949.503]: config item (null).servo_offset_threshold is 0
ptp4l[9949.503]: config item (null).servo_num_offset_values is 10
ptp4l[9949.503]: config item (null).dataset_comparison is 0
ptp4l[9949.503]: config item (null).tsproc_mode is 0
ptp4l[9949.503]: config item (null).delay_filter is 1
ptp4l[9949.503]: config item (null).delay_filter_length is 10
ptp4l[9949.503]: config item (null).initial_delay is 0
ptp4l[9949.503]: config item (null).summary_interval is 0
ptp4l[9949.503]: config item (null).sanity_freq_limit is 200000000
ptp4l[9949.504]: PI servo: sync interval 1.000 kp 0.700 ki 0.300000
ptp4l[9949.504]: config item /var/run/ptp4l.boundary_clock_jbod is 0
ptp4l[9949.504]: config item /var/run/ptp4l.network_transport is 0
ptp4l[9949.504]: config item /var/run/ptp4l.masterOnly is 0
ptp4l[9949.504]: config item /var/run/ptp4l.BMCA is 0
ptp4l[9949.504]: config item /var/run/ptp4l.delayAsymmetry is 0
ptp4l[9949.504]: config item /var/run/ptp4l.follow_up_info is 0
ptp4l[9949.504]: config item /var/run/ptp4l.freq_est_interval is 1
ptp4l[9949.504]: config item /var/run/ptp4l.msg_interval_request is 0
ptp4l[9949.504]: config item /var/run/ptp4l.net_sync_monitor is 0
ptp4l[9949.504]: config item /var/run/ptp4l.path_trace_enabled is 0
ptp4l[9949.504]: config item /var/run/ptp4l.tc_spanning_tree is 0
ptp4l[9949.504]: config item /var/run/ptp4l.ingressLatency is 0
ptp4l[9949.504]: config item /var/run/ptp4l.egressLatency is 0
ptp4l[9949.504]: config item /var/run/ptp4l.delay_mechanism is 0
ptp4l[9949.504]: config item /var/run/ptp4l.hybrid_e2e is 0
ptp4l[9949.505]: config item /var/run/ptp4l.fault_badpeernet_interval is 16
ptp4l[9949.505]: config item /var/run/ptp4l.fault_reset_interval is 4
ptp4l[9949.505]: config item /var/run/ptp4l.tsproc_mode is 0
ptp4l[9949.505]: config item /var/run/ptp4l.delay_filter is 1
ptp4l[9949.505]: config item /var/run/ptp4l.delay_filter_length is 1
ptp4l[9949.505]: config item (null).slave_event_monitor is ''
ptp4l[9949.505]: config item eth0.boundary_clock_jbod is 0
ptp4l[9949.505]: config item eth0.network_transport is 1
ptp4l[9949.505]: config item eth0.masterOnly is 0
ptp4l[9949.505]: config item eth0.BMCA is 0
ptp4l[9949.505]: config item eth0.delayAsymmetry is 0
ptp4l[9949.505]: config item eth0.follow_up_info is 0
ptp4l[9949.505]: config item eth0.freq_est_interval is 1
ptp4l[9949.505]: config item eth0.msg_interval_request is 0
ptp4l[9949.505]: config item eth0.net_sync_monitor is 0
ptp4l[9949.505]: config item eth0.path_trace_enabled is 0
ptp4l[9949.506]: config item eth0.tc_spanning_tree is 0
ptp4l[9949.506]: config item eth0.ingressLatency is 0
ptp4l[9949.506]: config item eth0.egressLatency is 0
ptp4l[9949.506]: config item eth0.delay_mechanism is 1
ptp4l[9949.506]: config item eth0.unicast_master_table is 0
ptp4l[9949.506]: config item eth0.unicast_listen is 0
ptp4l[9949.506]: config item eth0.hybrid_e2e is 0
ptp4l[9949.506]: config item eth0.fault_badpeernet_interval is 16
ptp4l[9949.506]: config item eth0.fault_reset_interval is 4
ptp4l[9949.506]: config item eth0.tsproc_mode is 0
ptp4l[9949.506]: config item eth0.delay_filter is 1
ptp4l[9949.506]: config item eth0.delay_filter_length is 10
ptp4l[9949.506]: config item eth0.logMinDelayReqInterval is 4
ptp4l[9949.506]: config item eth0.logAnnounceInterval is 4
ptp4l[9949.506]: config item eth0.inhibit_announce is 0
ptp4l[9949.506]: config item eth0.ignore_source_id is 0
ptp4l[9949.507]: config item eth0.announceReceiptTimeout is 3
ptp4l[9949.507]: config item eth0.syncReceiptTimeout is 0
ptp4l[9949.507]: config item eth0.transportSpecific is 0
ptp4l[9949.507]: config item eth0.ignore_transport_specific is 0
ptp4l[9949.507]: config item eth0.G.8275.portDS.localPriority is 128
ptp4l[9949.507]: config item eth0.logSyncInterval is 4
ptp4l[9949.507]: config item eth0.operLogSyncInterval is 0
ptp4l[9949.507]: config item eth0.logMinPdelayReqInterval is 0
ptp4l[9949.507]: config item eth0.operLogPdelayReqInterval is 0
ptp4l[9949.507]: config item eth0.neighborPropDelayThresh is 20000000
ptp4l[9949.507]: config item eth0.min_neighbor_prop_delay is -20000000
ptp4l[9949.507]: config item eth0.asCapable is 1
ptp4l[9949.507]: config item eth0.inhibit_delay_req is 0
ptp4l[9949.507]: config item eth0.udp_ttl is 1
ptp4l[9949.508]: config item (null).dscp_event is 0
ptp4l[9949.508]: config item (null).dscp_general is 0
ptp4l[9949.508]: port 1: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[9949.508]: config item /var/run/ptp4l.logMinDelayReqInterval is 4
ptp4l[9949.508]: config item /var/run/ptp4l.logAnnounceInterval is 4
ptp4l[9949.508]: config item /var/run/ptp4l.inhibit_announce is 0
ptp4l[9949.508]: config item /var/run/ptp4l.ignore_source_id is 0
ptp4l[9949.508]: config item /var/run/ptp4l.announceReceiptTimeout is 0
ptp4l[9949.508]: config item /var/run/ptp4l.syncReceiptTimeout is 0
ptp4l[9949.508]: config item /var/run/ptp4l.transportSpecific is 0
ptp4l[9949.508]: config item /var/run/ptp4l.ignore_transport_specific is 0
ptp4l[9949.508]: config item /var/run/ptp4l.G.8275.portDS.localPriority is 128
ptp4l[9949.508]: config item /var/run/ptp4l.logSyncInterval is 4
ptp4l[9949.508]: config item /var/run/ptp4l.operLogSyncInterval is 0
ptp4l[9949.508]: config item /var/run/ptp4l.logMinPdelayReqInterval is 0
ptp4l[9949.508]: config item /var/run/ptp4l.operLogPdelayReqInterval is 0
ptp4l[9949.508]: config item /var/run/ptp4l.neighborPropDelayThresh is 20000000
ptp4l[9949.508]: config item /var/run/ptp4l.min_neighbor_prop_delay is -20000000
ptp4l[9949.508]: config item /var/run/ptp4l.asCapable is 1
ptp4l[9949.508]: config item /var/run/ptp4l.inhibit_delay_req is 0
ptp4l[9949.508]: config item (null).uds_address is '/var/run/ptp4l'
ptp4l[9949.508]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[9949.508]: port 1: received link status notification
ptp4l[9949.508]: interface index 2 is up
ptp4l[9949.609]: port 1: setting asCapable
ptp4l[9952.185]: port 1: new foreign master b49691.fffe.7472d0-1
ptp4l[9969.784]: selected best master clock b49691.fffe.7472d0
ptp4l[9969.785]: port 1: LISTENING to UNCALIBRATED on RS_SLAVE
ptp4l[9978.584]: PI servo: sync interval 8.000 kp 0.087 ki 0.037500
ptp4l[9978.585]: port 1: have FOLLOW_UP 473, expecting FOLLOW_UP but got SYNC 474, dropping
ptp4l[9982.492]: port 1: delay timeout
....

Additional context

Rpi with 6.6.74 kernel works fine. Identical everything else.

[catubc]

Some more logs after running
sudo apt full-upgrade -y

Still same issues:

...
ptp4l[149.024]: config item /var/run/ptp4l.inhibit_delay_req is 0
ptp4l[149.024]: config item (null).uds_address is '/var/run/ptp4l'
ptp4l[149.024]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[149.024]: port 1: received link status notification
ptp4l[149.024]: interface index 2 is up
ptp4l[149.697]: port 1: setting asCapable
ptp4l[149.926]: port 1: new foreign master b49691.fffe.7472d0-1
ptp4l[165.926]: selected best master clock b49691.fffe.7472d0
ptp4l[165.926]: port 1: LISTENING to UNCALIBRATED on RS_SLAVE
ptp4l[169.139]: port 1: delay timeout
ptp4l[173.925]: PI servo: sync interval 8.000 kp 0.087 ki 0.037500
ptp4l[173.925]: port 1: have FOLLOW_UP 10481, expecting FOLLOW_UP but got SYNC 10482, dropping
ptp4l[190.724]: port 1: delay timeout

[pelwell]

Same cause as #6925 (i.e. torvalds/linux@87f7ce2), and the same fix (i.e. torvalds/linux@5ab73b0). It was picked up automatically for backporting because of the Fixes tag, and now appears in 6.12.35 (5f03625). You can install a beta build using sudo rpi-update.

[pelwell]

P.S./N.B. Be careful if you are relying on the initramfs (e.g. for exotic filesystems or drivers) in order to get to your root filing system - rpi-update won't rebuild that for you, you'd be better of waiting for the next kernel package in apt.

[Ve2mrx]

Is this issue present in 6.12.34+rpt as well? I'm having PTP issues since I updated this morning...

[pelwell]

If it was broken in 6.12.25 and fixed in 6.12.35 then I think you can understand why it might not work in 6.12.34.