rsi_hw_iface: "1073: Command gear torque An" error on any use

[gavanderhoorn]

Context:

• robot: KR 210
• controller: KR C4 (v8.3.22)
• RSI: 3.3
• afaict payload is configured correctly
• controller in T2

Set up kuka_rsi_hw_interface and the controller as I've done before (followed krl/readme.md as always).

State reporting works fine, RSI connection seems stable (quality: 100, total loss: +2 in 1 or 2 seconds, max contiguous loss: 2).

At the "run rqt_joint_trajectory_controller" section I do just that.

After enabling the rqt controller (push the big red button), robot moves slightly (robot is in 0, -90, 90, 0, 90, 0 pose and apparently rqt_joint_trajectory_controller rounds the values in the sliders to two decimals). Even without touching any of the sliders (so just enabling the rqt controller), the controller will go into a fault mode, always giving me Command gear torque An errors (with n any (or sometimes all) of the joints in the robot).

I've verified the in- and outgoing RSI XML documents and they seem reasonable enough. Maximum correction send right before the controller faults is similar to (A1 angle differs slightly each time):

<Sen Type="ImFree">
    <AK A1="1.544213" A2="0.000000" A3="0.000000" A4="0.000000" A5="0.000000" A6="0.000000"/>
    <IPOC>115859599</IPOC>
</Sen>

Running a simple test script (FollowJointTrajectory action client that sends A1 to +5 degrees in 10 seconds (two point JointTrajectory)) also results in the error, but after somewhat more time.

I must be missing something here, but don't see it (any more).

[gavanderhoorn]

@BrettHemes, @tingelst, @pgorczak, @destogl, @bj0rkedal: did you guys ever run into anything like this?

[SebNag]

Are you using an PREEMPTIVE kernel as recommend in the krl/readme.md ?
I had a similar issue. A real time patch of my OS made the connection stable.

[gavanderhoorn]

No, I'm currently running a -lowlatency kernel with kuka_hardware_interface_node at high priority. I'm aware of what the readme says, but didn't want to compile a kernel yet if a low latency one would do the job. And I'm also slightly confused as to why this is suddenly a problem for us.

Tbh I would've expected the RSI Diagnostics screen to show much worse stats if this is the problem, but I guess we'll have to try anyway.

[pgorczak]

FWIW I had similar stability issues before, that could be mitigated by modifying the RSI/ros-control interface to use relative position commands. In that case, the instability was clearly visible when you plotted the measured joint angles (sudden escalating oscillation).

[gavanderhoorn]

It's kind of hard to know for certain without additional tests -- as we committed the cardinal sin of changing more than a single 'variable' in our attempts to diagnose our issue, but it's very likely that incorrect tool load data was the cause of the excessive torque faults I reported in my OP.

Dropped / missing RSI packets will probably exacerbate this, as the controller will attempt an even larger correction than if everything is contiguous, but the controller is now no longer complaining, so we can move on to other things.

Thanks @SNBasti and @pgorczak for your input.

[rtonnaer]

@gavanderhoorn: I'd like to reopen this issue. In the mean while I have changed the configuration with respect to the first post by Gijs:
Configuration

• I am using a RT PREEMPT kernel: uname -r shows 4.14.3-rt5-custom running on:
• a different systemwith: Ubuntu 16.04.3 LTS with ROS Kinetic. Can this cause extra problems? I am using the indigo-devel branch....
• Using ps ax -o pid,ni,cmd |grep kuka_rsi I have identified this:
  4880 0 /home/rik/workspaces/robotmotion_ws/devel/lib/kuka_rsi_hw_interface/kuka_hardware_interface_node position_trajectory_controller/follow_joint_trajectory:=/joint_trajectory_action position_trajectory_controller/state:=/feedback_states position_trajectory_controller/command:=/joint_path_command __name:=kuka_hardware_interface __log:=/home/rik/.ros/log/9b0f1542-d9b3-11e7-88e7-507b9db1a247/kuka_hardware_interface-2.log
  to be the kuka_rsi_hw_interface process.
• Used sudo chrt -p 99 4880 to provide it with real time priority.
• And checked it with: chrt -p 4880 the result: pid 4880's current scheduling policy: SCHED_RR pid 4880's current scheduling priority: 99
• On the controller the original .src file is running (from this repository) and the robot is switched to auto @ 3 to 5 percent speed (I believe this value does not actually matter. Speed settings come directly from ROS).

Behavior
Using the RRTConnectkConfigDefault planner I am planning some very simple and short motions. When the default velocity scaling is used the robot sounds like driving an old car on a bumpy road. A kind of mechanical vibration. When the velocity and acceleration scaling are decreases (to 0.01) this is less, however the motion is off course very slow.

The RSI Diagnostic Monitor on the KR C4 shows (in my opinion) quite good connection results:
Communication cycles: 52167
Total Loss: 71
Connection quality: 71
Maximum contiguous loss: 2

Additional info
We are using the KR210 with additional axes. It is positioned on a KL2000 linear track and has an external horizontal positioner (KP1). Don`t know if this has any relevance, but it does mean I only use RSI_MOVECORR() for A1 up to A6.

[gavanderhoorn]

Seeing as you are on Kinetic, your description makes me think of moveit/moveit#416.

moveit/moveit#160 could also be an influence here (as in: the fact that it hasn't been merged yet).

When the default velocity scaling is used the robot sounds like driving an old car on a bumpy road. A kind of mechanical vibration.

This should definitely not happen.

To see whether it's the time parameterisation you could try a TOTG planning adapter that would take the place of the IPTP.

To see whether the OMPL changes are of influence we could see whether Indigo causes the same issues. An alternative could be to construct a trajectory manually and then submitting it to the driver (you could still use the TOTG or IPTP in those cases).

[gavanderhoorn]

I re-opened the issue, but I just realised: are you still getting the 1073: Command gear torque An errors, or is this something else?

If the latter, we should probably open a new issue.