DC Synchronization Issues: Servo drives exhibit warnings of excessive synchronous cycle errors shortly after startup.

[tianyuZ]

Environment

• OS: Ubuntu 22.04, PREEMPT_RT realtime-kernel
• ROS 2 Version: Humble
• servo motor: Raynen RS3E1
• Master: IGH EtherCAT Master 1.5.3, using e1000e driver
• Network card model: Intel Corporation 82574L Gigabit
• ethercat_driver_ros2 branch: main
• Bind a process to specific CPU cores (CPU core control).

Issues Description

1. When testing five servo motors (configured in CSP mode) of a palletizing machine without using Intel NICs, shortly after sequentially starting the servo motors, the servo drives exhibit warnings of excessive synchronous cycle errors, causing abnormal motor operation. Related error messages are shown in the figure below:

ros_control

kernel log

Both at 100Hz and 1000Hz DC synchronization/control cycle frequencies, servo drives trigger warning errors.

2. When using Intel NICs at 1000Hz DC synchronization/control frequency, all five servo motors operate stably after sequential startup with no warnings.

ros_control

kernel log

However, at 100Hz configuration, synchronous cycle error warnings persist.

Multiple improvement attempts have failed. Suspect potential issues in the DC synchronization implementation within ethercat_driver_ros2

Config files

ethercat_motor_drive/config/controllers.yaml

controller_manager:
  ros__parameters:
    update_rate: 100  # Hz

    joint_state_broadcaster:
      type: joint_state_broadcaster/JointStateBroadcaster

    palletizing_arm_controller:
      type: joint_trajectory_controller/JointTrajectoryController

    velocity_controller:
      type: velocity_controllers/JointGroupVelocityController

    effort_controller:
      type: effort_controllers/JointGroupEffortController

palletizing_arm_controller:
  ros__parameters:
    command_interfaces:
      - position
    state_interfaces:
      - position
    joints:
      - joint_revolute_1
      - joint_revolute_2
      - joint_revolute_3
      - joint_revolute_4
      - joint_revolute_5

velocity_controller:
  ros__parameters:
    joints:
      - joint_revolute_1
      - joint_revolute_2
      - joint_revolute_3
      - joint_revolute_4
      - joint_revolute_5

effort_controller:
  ros__parameters:
    joints:
      - joint_revolute_1
      - joint_revolute_2
      - joint_revolute_3
      - joint_revolute_4
      - joint_revolute_5

EtherCAT Slave configuration

# Configuration file for Raynen RS3E1_5R5L drive
vendor_id: 0x00000922
product_id: 0x00000c01
assign_activate: 0x0300 # DC Synch register
auto_fault_reset: false # true = automatic fault reset, false = fault reset on rising edge command interface "reset_fault"
sdo: # sdo data to be transferred at drive startup
  # - { index: 0x60C2, sub_index: 1, type: int8, value: 10 } # Set interpolation time for cyclic modes to 10 ms
  # - { index: 0x60C2, sub_index: 2, type: int8, value: -3 } # Set base 10-3s
rpdo: # RxPDO = receive PDO 7 Mapping
  - index: 0x1702
    channels:
      - { index: 0x6040, sub_index: 0, type: uint16, default: 0 } # Control word
      - {
          index: 0x607a,
          sub_index: 0,
          type: int32,
          command_interface: position,
          factor: 320421222.9264523077,
          offset: 39533790,
          # offset: 0,
          # default: .nan
          default: 39533790
        } # Target position
      - { index: 0x60ff, sub_index: 0, type: int32, default: 0 } # Target velocity
      - { index: 0x6071, sub_index: 0, type: int16, default: 0 } # Target torque
      - { index: 0x6060, sub_index: 0, type: int8, default: 8 } # Mode of operation
      - { index: 0x60b8, sub_index: 0, type: uint16, default: 0 } # Touch Probe Function
      - { index: 0x607f, sub_index: 0, type: uint32, default: 838860800 } # Max Profile Velocity
tpdo: # TxPDO = transmit PDO 9 Mapping
  - index: 0x1b02
    channels:
      - { index: 0x603f, sub_index: 0, type: uint16 } # Error Code
      - { index: 0x6041, sub_index: 0, type: uint16 } # Status word
      - { 
          index: 0x6064, 
          sub_index: 0, 
          type: int32, 
          state_interface: position,
          factor: 3.120891902436607e-9,
          offset: -0.123380685083629
          # offset: 0
        } # Position actual value
      - { index: 0x6077, sub_index: 0, type: int16, state_interface: effort } # Torque actual value
      - { index: 0x6061, sub_index: 0, type: int8 } # Mode of operation display
      - { index: 0x60b9, sub_index: 0, type: uint16 } # Touch Probe Status
      - { index: 0x60ba, sub_index: 0, type: int32 } # Touch Probe pos1 Pos Value
      - { index: 0x60bc, sub_index: 0, type: int32 } # Touch Probe pos2 Pos Value
      - { index: 0x60fd, sub_index: 0, type: uint32} # Digital Inputs
sm:  # Sync Manager
  - {index: 0, type: output, pdo: ~, watchdog: disable}
  - {index: 1, type: input, pdo: ~, watchdog: disable}
  - {index: 2, type: output, pdo: rpdo, watchdog: enable}
  - {index: 3, type: input, pdo: tpdo, watchdog: disable}

ethercat_motor_drive/description/ros2_control/motor_drive.ros2_control.xacro

<?xml version="1.0"?>
<robot xmlns:xacro="http://www.ros.org/wiki/xacro">

  <xacro:macro name="motor_drive">

    <ros2_control name="motor_drive" type="system">
      <hardware>
        <plugin>ethercat_driver/EthercatDriver</plugin>
        <param name="master_id">0</param>
        <param name="control_frequency">100</param>
      </hardware>

      <joint name="joint_revolute_1">
        <state_interface name="position"/>
        <state_interface name="velocity"/>
        <state_interface name="effort"/>
        <command_interface name="position"/>
        <command_interface name="reset_fault"/>
        <ec_module name="Raynen">
          <plugin>ethercat_generic_plugins/EcCiA402Drive</plugin>
          <param name="alias">0</param>
          <param name="position">0</param>
          <param name="mode_of_operation">8</param>
          <param name="slave_config">$(find ethercat_motor_drive)/config/joint1_raynen_rs3e1_5r5l_3rd_config.yaml</param>
        </ec_module>
      </joint>

      <joint name="joint_revolute_2">
        <state_interface name="position"/>
        <state_interface name="velocity"/>
        <state_interface name="effort"/>
        <command_interface name="position"/>
        <command_interface name="reset_fault"/>
        <ec_module name="Raynen">
          <plugin>ethercat_generic_plugins/EcCiA402Drive</plugin>
          <param name="alias">0</param>
          <param name="position">1</param>
          <param name="mode_of_operation">8</param>
          <param name="slave_config">$(find ethercat_motor_drive)/config/joint2_raynen_rs3e1_7r6m_3rd_config.yaml</param>
        </ec_module>
      </joint>

      <joint name="joint_revolute_3">
        <state_interface name="position"/>
        <state_interface name="velocity"/>
        <state_interface name="effort"/>
        <command_interface name="position"/>
        <command_interface name="reset_fault"/>
        <ec_module name="Raynen">
          <plugin>ethercat_generic_plugins/EcCiA402Drive</plugin>
          <param name="alias">0</param>
          <param name="position">2</param>
          <param name="mode_of_operation">8</param>
          <param name="slave_config">$(find ethercat_motor_drive)/config/joint3_raynen_rs3e1_2r8l_3rd_config.yaml</param>
        </ec_module>
      </joint>

      <joint name="joint_revolute_4">
        <state_interface name="position"/>
        <state_interface name="velocity"/>
        <state_interface name="effort"/>
        <command_interface name="position"/>
        <command_interface name="reset_fault"/>
        <ec_module name="Raynen">
          <plugin>ethercat_generic_plugins/EcCiA402Drive</plugin>
          <param name="alias">0</param>
          <param name="position">3</param>
          <param name="mode_of_operation">8</param>
          <param name="slave_config">$(find ethercat_motor_drive)/config/joint4_raynen_rs3e1_1r6l_3rd_config.yaml</param>
        </ec_module>
      </joint>

      <joint name="joint_revolute_5">
        <state_interface name="position"/>
        <state_interface name="velocity"/>
        <state_interface name="effort"/>
        <command_interface name="position"/>
        <command_interface name="reset_fault"/>
        <ec_module name="Raynen">
          <plugin>ethercat_generic_plugins/EcCiA402Drive</plugin>
          <param name="alias">0</param>
          <param name="position">4</param>
          <param name="mode_of_operation">8</param>
          <param name="slave_config">$(find ethercat_motor_drive)/config/joint5_raynen_rs3e1_1r6l_3rd_config.yaml</param>
        </ec_module>
      </joint>
    </ros2_control>

  </xacro:macro>

</robot>

[467480903]

this error happend with me too , I solve it by this:

from launch import LaunchDescription
from launch.substitutions import Command, FindExecutable, PathJoinSubstitution, LaunchConfiguration
from launch.actions import DeclareLaunchArgument, TimerAction

from launch_ros.actions import Node
from launch_ros.substitutions import FindPackageShare


def generate_launch_description():

    # Declare arguments
    declared_arguments = []
    declared_arguments.append(
        DeclareLaunchArgument(
            'description_file',
            default_value='motor_drive.config.xacro',
            description='URDF/XACRO description file with the axis.',
        )
    )

    description_file = LaunchConfiguration('description_file')

    # Get URDF via xacro
    robot_description_content = Command(
        [
            PathJoinSubstitution([FindExecutable(name="xacro")]),
            " ",
            PathJoinSubstitution(
                [
                    FindPackageShare("ethercat_motor_drive"),
                    "description/config",
                    description_file,
                ]
            ),
        ]
    )
    robot_description = {"robot_description": robot_description_content}

    robot_controllers = PathJoinSubstitution(
        [
            FindPackageShare("ethercat_motor_drive"),
            "config",
            "controllers.yaml",
        ]
    )

    control_node = Node(
        package="controller_manager",
        executable="ros2_control_node",
        parameters=[robot_description, robot_controllers],
        output="both",
    )
    robot_state_pub_node = Node(
        package="robot_state_publisher",
        executable="robot_state_publisher",
        output="both",
        parameters=[robot_description],
    )

    joint_state_broadcaster_spawner = Node(
        package="controller_manager",
        executable="spawner",
        arguments=["joint_state_broadcaster", "-c", "/controller_manager"],
    )

    trajectory_controller_spawner = Node(
        package="controller_manager",
        executable="spawner",
        arguments=["trajectory_controller", "-c", "/controller_manager"],
    )

    # velocity_controller_spawner = Node(
    #     package="controller_manager",
    #     executable="spawner",
    #     arguments=["velocity_controller", "-c", "/controller_manager"],
    # )

    # effort_controller_spawner = Node(
    #     package="controller_manager",
    #     executable="spawner",
    #     arguments=["effort_controller", "-c", "/controller_manager"],
    # )

    nodes = [
        control_node,
        robot_state_pub_node,
        joint_state_broadcaster_spawner,
        # trajectory_controller_spawner,
        TimerAction(
            period=35.0,  
            actions=[trajectory_controller_spawner],
        ),
        # velocity_controller_spawner,
        # effort_controller_spawner,
    ]

    return LaunchDescription(
        declared_arguments +
        nodes)

modify the launch python code , make the trajectory_controller_spawner start later ,

I guess this error comes because the motors are not all enabled , but the ros trajectory controller send something to the motor ,

[467480903]

there is something wrong in your motor config yaml,

this is my yaml:

#零差云控
vendor_id: 0x5a65726f
product_id: 0x00029252
assign_activate: 0x0300   #DC synch register
auto_fault_reset: true
auto_enable_set: false

sdo:  # sdo data to be transferred at drive startup
  - {index: 0x1C33, sub_index: 1, type: uint16, value: 2} # Set 0x1C33:01h = 0x02  DC设置
  # - {index: 0x1C32, sub_index: 1, type: uint16, value: 2}

rpdo: #PxPDO = receive PDO Mapping
  - index: 0x1600
    channels:
      - {index: 0x607a, sub_index: 0, type: int32, command_interface: position, default: .nan, factor: 83442.867648821  , offset: 262144} 
      #target position	
      # - {index: 0x60ff, sub_index: 0, type: int32, default: 0, factor: 83442.867648821}	#target velocity
      # - {index: 0x6071, sub_index: 0, type: int16, default: 0, factor: 1.0}	#target torque
      # - {index: 0x6072, sub_index: 0, type: uint16, default: 500}	# Max torque
      - {index: 0x6040, sub_index: 0, type: uint16, default: 0}	# Control word
      - {index: 0x6060, sub_index: 0, type: uint8, default: 8}	# Control operation
      - {index: 0xf0ff, sub_index: 0, type: uint8}	# Dummy byte
    
tpdo:  #TxPDO = transmit PDO Mapping
  - index: 0x1a00
    channels:
      - {index: 0x6064, sub_index: 0, type: int32, state_interface: position, factor: 0.000011984247763, offset: -3.1415986457136573} #Position actual value	
      # - {index: 0x60F4, sub_index: 0, type: int32}	#Position Following error actual value
      - {index: 0x606c, sub_index: 0, type: int32, state_interface: velocity, factor: 0.000011984247763 }	# Velocity actual value
      - {index: 0x6077, sub_index: 0, type: int16,state_interface: effort, factor: 1.0 }	# Torque actual value
      - {index: 0x6041, sub_index: 0, type: uint16}	# State word
      - {index: 0x603f, sub_index: 0, type: uint16}	# Error code
      - {index: 0x6061, sub_index: 0, type: uint8, state_interface: mode_of_operation}	        # Mode of operation display
      - {index: 0xf0ff, sub_index: 0, type: uint8}	        # dummy byte

I think you sould set the default to .nan