Skip to content

[HIL-SERL] possible performance bottleneck in VanillaObservationProcessorStep causes low interaction rates #2173

@jpizarrom

Description

@jpizarrom

System Info

- two cameras 640x480
- lerobot version: 0.3.4
- Platform: Linux-6.11.0-29-generic-x86_64-with-glibc2.39
- Python version: 3.10.17
- Huggingface Hub version: 0.35.1
- Datasets version: 4.1.1
- Numpy version: 2.2.6
- PyTorch version: 2.7.1+cu126
- Is PyTorch built with CUDA support?: True
- Cuda version: 12.6
- GPU model: NVIDIA RTX A5500 Laptop GPU

Information

I was unable to achieve better frame rates when using gym_manipulator/actor due to a possible performance bottleneck in VanillaObservationProcessorStep and ImageCropResizeProcessorStep.

Moving tensors to CUDA within VanillaObservationProcessorStep significantly reduced processing time. This optimization also improved ImageCropResizeProcessorStep performance, resulting in smoother robot arm movement.
this improves the FPS from 5-7 FPS to 13 FPS

This are the times of each step when using cpu

INFO 2025-10-11 11:58:05 pipeline.py:316 Processing time for step 0: 0.0001s, step: AddTeleopActionAsComplimentaryDataStep
INFO 2025-10-11 11:58:05 pipeline.py:316 Processing time for step 1: 0.0000s, step: AddTeleopEventsAsInfoStep
INFO 2025-10-11 11:58:05 pipeline.py:316 Processing time for step 2: 0.0000s, step: InterventionActionProcessorStep
INFO 2025-10-11 11:58:05 pipeline.py:316 Processing time for step 3: 0.0002s, step: MapTensorToDeltaActionDictStep
INFO 2025-10-11 11:58:05 pipeline.py:316 Processing time for step 4: 0.0000s, step: MapDeltaActionToRobotActionStep
INFO 2025-10-11 11:58:05 pipeline.py:316 Processing time for step 5: 0.0003s, step: EEReferenceAndDelta
INFO 2025-10-11 11:58:05 pipeline.py:316 Processing time for step 6: 0.0000s, step: EEBoundsAndSafety
INFO 2025-10-11 11:58:05 pipeline.py:316 Processing time for step 7: 0.0000s, step: GripperVelocityToJoint
INFO 2025-10-11 11:58:05 pipeline.py:316 Processing time for step 8: 0.0002s, step: InverseKinematicsRLStep
INFO 2025-10-11 11:58:05 pipeline.py:316 Processing time for step 9: 0.0001s, step: RobotActionToPolicyActionProcessorStep
INFO 2025-10-11 11:58:05 pipeline.py:316 Processing time for step 0: 0.0700s, step: VanillaObservationProcessorStep
INFO 2025-10-11 11:58:05 pipeline.py:316 Processing time for step 1: 0.0001s, step: JointVelocityProcessorStep
INFO 2025-10-11 11:58:05 pipeline.py:316 Processing time for step 2: 0.0017s, step: MotorCurrentProcessorStep
INFO 2025-10-11 11:58:05 pipeline.py:316 Processing time for step 3: 0.0002s, step: ForwardKinematicsJointsToEEObservation
INFO 2025-10-11 11:58:05 pipeline.py:316 Processing time for step 4: 0.0579s, step: ImageCropResizeProcessorStep
INFO 2025-10-11 11:58:05 pipeline.py:316 Processing time for step 5: 0.0000s, step: TimeLimitProcessorStep
INFO 2025-10-11 11:58:05 pipeline.py:316 Processing time for step 6: 0.0000s, step: GripperPenaltyProcessorStep
INFO 2025-10-11 11:58:05 pipeline.py:316 Processing time for step 7: 0.0001s, step: AddBatchDimensionProcessorStep
INFO 2025-10-11 11:58:05 pipeline.py:316 Processing time for step 8: 0.0002s, step: DeviceProcessorStep
INFO 2025-10-11 11:58:05 ipulator.py:575 Action processor time: 0.0053s, Env step time: 0.0278s, Env processor time: 0.1308s

Expected behavior

Maybe a new config param to select the device of the steps, or use the device from the policy

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions