a confusing result when evaluating

[lrcc275]

I evaluate the pretrained models according to the starter.md
However, the progress bar "Evaluating ........" will stop around 11%
I actually see the metrics like "eval/task/reaching/pos_err/mean" on my terminal, but I notice the "eval/time_outs/sum" is 0.
Does it mean that the robots terminate abnormally？
In the wandb website, there is a video, but the other graphs time_out , pos_error are not drawn correctly
Compared to your summary I think there may be some problem

I run the code with (python scripts/evaluate.py env.sim_device=cuda:0 env.graphics_device_id=0 env.cfg.env.episode_length_s=17.0 env.tasks.reaching.sequence_sampler.file_path=data/tossing.pkl ckpt_path=checkpoints/tossing/ours/model.pt env.tasks.reaching.target_obs_times="[-0.06,-0.04,-0.02,0.0,0.02,0.04,0.06,1.0]" )
using the data and checkpoints downloaded from the websites provided.

I use architecture linux ubuntu22.04 RTX4090 (actually in another machine with RTX3090，I encounter same problem.

[huy-ha]

Does it mean that the robots terminate abnormally？
Yep, it means the episode ended because the robot self-collided or fell down.

This is strange. I've never experienced this myself. Can you pin point which line the code is stuck at? I wonder if it's a Wandb problem, and running with the WANDB_MODE=offline env variable would bypass the issue.

[lrcc275]

Thanks for your reply. This problem still exists after setting wandb_mode=offline.
In fact, the eval function will be executed completely；I needn't interrupt the progress myself.
In eval
"total_steps = self.env.max_episode_length * (self.num_eval_episode_per_env + 2)"
total_steps would be 17 * (18+2) = 3400
however it runs only 330~370 steps(different every time) in "for _ in pbar:......"

[zhiweicai07]

but I notice the "eval/time_outs/sum" is 0.

I also encountered this problem

[xander-2077]

if eval_episodes > self.env.num_envs * self.num_eval_episode_per_env:
  break

In fact, self.env.num_envs * self.num_eval_episode_per_env is equal to 500. You can print the current value of eval_episodes and the code always ends when eval_episodes=500.