[Question] One-time evaluation score is not good indicator, and therefore probably should not be the tuning target?

[CyclonicDyna]

❓ Question

I tried to train the agent with the tuned parameters from the best trial, but strangely, I found out the training cannot be as good as the reported score in the tuning process (actually, much worse). Anyone met this problem?

My guess for this problem is that the one-time evaluation is very stochastic (question: currently, sb3-rl-zoo is using one-time evaluation score as the target, right?). I think It is not a good measurement of how good the agent is.
Probably we should use the mean of multi-time evaluation scores as the tuning target, or using the mean reward from the rollout. What do you think?

(for mountain car)

Thanks for any ideas!

Checklist

• I have checked that there is no similar issue in the repo
• I have read the SB3 documentation
• I have read the RL Zoo README
• If code there is, it is minimal and working
• If code there is, it is formatted using the markdown code blocks for both code and stack traces.

[qgallouedec]

Duplicate #204
Contributions are welcomed 🙂

[araffin]

Related to #286 too.

In the meantime, you can also increase the number of evaluations episodes to reduce the noise (--eval-episodes).

My guess for this problem is that the one-time evaluation is very stochastic (question: currently, sb3-rl-zoo is using one-time evaluation score as the target, right?).

not really, if you are using a pruner, it evaluates the agent periodically on several test episodes, but yes we current test only one seed.
To have hyperparameters that work for many random seeds, you can do a post-processing step (see #151) or run multiple training (#204).

[CyclonicDyna]

Thank you very much @araffin @qgallouedec. I learned a lot from your answers and from the related Q&A : )