[Question] Has Offline RL success on antmaze been reproduced with Minari's antmaze datasets?

[perrin-isir]

Question

I tried to apply Offline RL (in particular IQL) to Minari's antmaze datasets (e.g. medium-play and large-play), but without success so far. IQL succeeds on D4RL antmazes, but with the same hyperparameters (and the same trick that substracts 1 to the reward, see https://github.com/ikostrikov/implicit_q_learning/blob/09d700248117881a75cb21f0adb95c6c8a694cb2/train_offline.py#L68) it fails on Minari's corresponding datasets. Is this a known issue? Has someone been able to solve the Minari antmazes based on the Minari datasets with standard offline RL algorithms?

What I tried so far:

• Following advice from @younik (see [Proposal] Porting Antmazes to Minari. #152 (comment)), I downgraded mujoco (v3.1.6) but it did not improve the results of offline RL. Results are not zero, so the ant is able to walk towards very simple targets in some cases, but in D4RL the results would typically be much better. I also noticed a potential issue with the dataset size, since among the 1000 episodes, many episodes reach the target after 200/300 steps, and the rest is an accumulation of the reward by staying still, so that makes a total of ~250k useful transitions out of the million in the dataset.
• I used minari-dataset-generation-scripts (https://github.com/Farama-Foundation/minari-dataset-generation-scripts) to regenerate the medium-play dataset, but with a slight modification: I truncated episodes after each success, to avoid having lots of transitions only accumulating reward in the dataset. But that didn't improve the performance of IQL.

Has anyone been able to perform Offline RL succcessfully on say, Minari's medium and large antmaze datasets?

Note: with the simpler Minari's D4RL/antmaze/umaze-v1 environment, Offline RL is successful.

[alexdavey]

Hi @perrin-isir,

Thanks for raising this issue and the extensive debugging effort. I personally have not tried to train Offline RL on the AntMaze datasets specifically, and was not aware of this discrepancy with respect to D4RL. Regarding the extra stationary transitions, it was previously my impression that a similar behaviour does appear in some D4RL datasets, however taking a quick look again such I can't find it. If this is the case then the Minari AntMaze datasets may need to be regenerated. I will confirm and then get back to you on that.

[alexdavey]

Hi @perrin-isir,

Thanks for your patience. Here's a script based on CORL that I hacked together to use on Minari's Antmaze environments. It gets a similar score to the original D4RL dataset at 1M timesteps, with the caveat that learning is much slower (and possibly has not yet plateaued).

• The main modification is converting the GoalEnv Dict observation space to a suitable Box one, by concatenating the achieved_goal and direction to the goal (desired_goal - achieved_goal) to the other observations. The other modifications are just converting the Gymnasium environments to a Gym one. These modifications are rather crude, seeding is broken for example, however existing Minari integrations such as d3rlpy didn't work out of the box for me. ¹ I believe this is because the AntMaze environment doesn't include the current xy position by default, while d3rlpy only adds the desired_goal but not the achieved_goal.

• For future dataset versions, it might be useful to directly apply this GoalEnv conversion wrapper before data collection. so that the recovered environments do not require any preprocessing. @younik, do you have any thoughts on this?

• The reason for the slower training is currently unclear. It may be related to the truncation issue you pointed out, but it's something that needs further investigation. I'm planning on creating a set of benchmarks for the new D4RL datasets, since naturally we would like these new datasets to have similar training characteristics compared to the originals, if possible. Hopefully this will weed out any remaining issues, with both the datasets and integrations.

Footnotes

1. There is an existing D3RLPY script for Minari AntMaze, however out of the box I didn't get non-zero returns. ↩

[perrin-isir]

Hi @alexdavey,

Thanks a lot for investigating this isssue.
It is strange, when I run your script ( python iql.py --config medium_play_v2.yaml --env D4RL/antmaze/medium-play-v1 --device cuda), I just get zero rewards across the whole run of 1M steps:

...
Time steps: 295000
---------------------------------------
Evaluation over 100 episodes: 0.000 , D4RL score: 0.000
---------------------------------------
Time steps: 300000
---------------------------------------
Evaluation over 100 episodes: 0.000 , D4RL score: 0.000
...
Time steps: 630000
---------------------------------------
Evaluation over 100 episodes: 0.000 , D4RL score: 0.000
---------------------------------------
Time steps: 635000
---------------------------------------
Evaluation over 100 episodes: 0.000 , D4RL score: 0.000
...

BUT, I had to make a tiny modification to your script to make it work with my config. My environment has gym v0.26.2, and in this version, resets already return the info part of the reset tuple.
So in GoalEnv2Gym I had to change this:

obs, _ = self.env.reset()
return self._transform_obs(obs) 

into this:

obs, info = self.env.reset()
return self._transform_obs(obs), info

(and in eval_actor I had to write state, done = env.reset())

As unlikely as it may seem, I'll try to see if this difference has anything to do with the failed run I get.
Which version of gym are you using? And also, which version of mujoco and python? (In my case: 3.1.6 and 3.10.14; I also notice that I'm using minari v0.5.2, not the latest v0.5.3)

[alexdavey]

Hi @perrin-isir,

That's strange, I re-ran the gist code just after submitting it and it is generating rewards, so I believe it is the correct version. My pip freeze is below (python=3.9, minari=0.5.3, gym=0.23, mujoco=3.1.6). IIRC it is just the default install using the requirements from CORL. I wouldn't expect the change in Minari version to make a difference.

PS: In your eval_actor modification I assume the second value being assigned to done is a typo in the GitHub comment, but if it is also in the code then it might be the issue.

Click to expand `requirements.txt`

absl-py==2.2.2
certifi==2025.4.26
cffi==1.17.1
charset-normalizer==3.4.2
chex==0.1.5
click==8.1.8
cloudpickle==3.1.1
contourpy==1.3.0
cycler==0.12.1
Cython==0.29.37
d4rl @ git+https://github.com/tinkoff-ai/d4rl@db6e4b34bb5ce2a51dd3879177c0a0223208a614
decorator==5.2.1
distrax==0.1.2
dm-env==1.6
dm-tree==0.1.8
dm_control==1.0.30
docker-pycreds==0.4.0
etils==1.5.2
Farama-Notifications==0.0.4
fasteners==0.19
filelock==3.18.0
flax==0.6.1
fonttools==4.58.0
fsspec==2025.5.0
gast==0.6.0
gitdb==4.0.12
GitPython==3.1.44
glfw==2.9.0
gym==0.23.0
gym-notices==0.0.8
gymnasium==1.1.1
gymnasium-robotics==1.3.1
h5py==3.13.0
hf-xet==1.1.2
huggingface-hub==0.32.0
idna==3.10
imageio==2.37.0
importlib_metadata==8.7.0
importlib_resources==6.5.2
jax==0.4.1
jaxlib==0.4.1+cuda11.cudnn86
Jinja2==3.1.6
kiwisolver==1.4.7
labmaze==1.0.6
lxml==5.4.0
markdown-it-py==3.0.0
MarkupSafe==3.0.2
matplotlib==3.9.4
mdurl==0.1.2
minari==0.5.3
mjrl @ git+https://github.com/aravindr93/mjrl@3871d93763d3b49c4741e6daeaebbc605fe140dc
msgpack==1.1.0
mujoco==3.1.6
mujoco-py==2.1.2.14
mypy_extensions==1.1.0
numpy==1.23.1
opt_einsum==3.4.0
optax==0.1.3
packaging==25.0
pathtools==0.1.2
pettingzoo==1.25.0
pillow==11.2.1
promise==2.3
protobuf==3.20.3
psutil==7.0.0
pybullet==3.2.7
pycparser==2.22
pygame==2.1.0
Pygments==2.19.1
PyOpenGL==3.1.9
pyparsing==3.2.3
pyrallis==0.3.1
python-dateutil==2.9.0.post0
PyYAML==6.0.2
requests==2.32.3
rich==14.0.0
scipy==1.13.1
sentry-sdk==2.29.1
setproctitle==1.3.6
shellingham==1.5.4
shortuuid==1.0.13
six==1.17.0
smmap==5.0.2
tensorflow-probability==0.25.0
termcolor==3.1.0
toolz==1.0.0
torch==1.11.0+cu113
tqdm==4.64.0
typer==0.15.4
typing-inspect==0.9.0
typing_extensions==4.13.2
urllib3==2.4.0
wandb==0.12.21
zipp==3.21.0

[perrin-isir]

Haha damn the typo is most definitely the mistake (I cannot retry right now but I will soon and it must be that). Sorry I didn't have much time and wanted to reply before leaving, I shouldn't have :), and I didn't expect I would manage to introduce a bug by just slightly changing 3 lines of code, but I underestimated myself ;)

I will take more time to re-run things, and to test it with my implementation of IQL which is in JAX.

I'll keep you updated.

• FIRST UPDATE: it was indeed the typo, I get the same results now (the slow learning toward 60+ normalized score)

[alexdavey]

Haha it's great to hear that was the issue! :)

[DDVLau]

I've encountered significant performance differences when running the same algorithms on D4RL versus Minari datasets. I attempted to reproduce several Q-ensemble methods (EDAC, SAC-N, BC, TD3-BC...) in previous Mujoco environments (Hopper, Walker2d, HalfCheetah), and comparing:

• medium-v0 in Minari
• medium-v2 in D4RL

I used identical hyperparameters across both datasets. My scripts were adapted from CORL and implemented nearly identical (with minimal differences for gym/gymnasium compatibility) for controlling as many variables as possible.

Results
Similar to @perrin-isir's findings, algorithms show normal learning curves on D4RL datasets but exhibit almost no learning on Minari datasets. D4RL and Minari should be considered two distinct offline datasets at the moment despite their intended similarity. The performance differences are substantial enough that algorithms may need dataset-specific tuning?

I would greatly appreciate it if the Minari team could establish standardised baselines using open-source implementations like d3rlpy or CORL (@alexdavey) to better understand these discrepancies.

---

My environment details are below
Ubuntu 24.04 + Python 3.10.18

• D4RL setup:
  mujoco_py==2.1.2.14 (from d3rlpy)
  gym==0.26.0

• Minari setup:
  mujoco==3.1.6
  gymnasium==1.0.0

[alexdavey]

Hi @DDVLau, thanks for your comment. Establishing baselines is something that we are currently working on.

Apart from the AntMaze environments, whose issues are described in this thread, I'm not aware of issues with learning on the other datasets. For example, I get the following results on MuJoCo using IQL (last scores, single seed):

	Minari simple-v0	Minari medium-v0	D4RL medium-v2 (CORL)
walker2d	4306 (63)	6213 (90)	3720 (81)
hopper	3183 (84)	3043 (80)	2193 (68)
halfcheetah	7765 (49)	15200 (94)	5679 (48)

• The normalised scores are in brackets (to be added in this pr: More MuJoCo datasets minari-dataset-generation-scripts#19).

• For MuJoCo, the D4RL medium datasets are most closely analogous to the simple Minari datasets. The reason for this is that in Minari, the expert policy is generally higher performing than the original D4RL one, and a new simple level of proficiency has been introduced to fill the gap.

My IQL script is also based on CORL¹. If you are experiencing significantly different results to this then please let me know.

As a side note, the Minari MuJoCo datasets require mujoco==3.2.3, although this is unlikely to cause a significant variation in performance.

Footnotes

1. Based on this with some trivial fixes in TransformObservation to address deprecated behaviour. ↩