[Bug Report] Composing RGBImgPartialObsWrapper with ViewSizeWrapper doesn't give the correct observation.

[NishanthVAnand]

Describe the bug
The default view of the agent is 7x7 which can be changed using the ViewSizeWrapper by passing the agent_view_size argument. But, when it is used alongside with the RGBImgPartialObsWrapper, ViewSizeWrapper doesn't have any effect and the observation defaults to 7x7

Code example

import gymnasium as gym
from minigrid.wrappers import *

from PIL import Image

env = gym.make("BabyAI-GoToRedBall-v0")
env = ViewSizeWrapper(env, 5)
env = RGBImgPartialObsWrapper(env)
observation, info = env.reset()
Image.fromarray(observation["image"])

System Info
Describe the characteristic of your environment:

• Minigrid was installed using pip and I am using the latest version (3.0.0).
• Python version: Python 3.9.15

Additional context
Add any other context about the problem here.

Checklist

• I have checked that there is no similar issue in the repo (required)

[pseudo-rnd-thoughts]

Thanks for raising the issue,

Testing with your script

import gymnasium as gym
from minigrid.wrappers import ViewSizeWrapper, RGBImgPartialObsWrapper
import matplotlib.pyplot as plt

env = gym.make("BabyAI-GoToRedBall-v0", render_mode="rgb_array")
obs, _ = env.reset(seed=123)
print(f'base image shape: {obs["image"].shape}') # (7,7,3)

env = RGBImgPartialObsWrapper(env)
obs, _ = env.reset(seed=123)
print(f'RGBImgPartialObsWrapper: {obs["image"].shape}') # (56, 56, 3)

env = ViewSizeWrapper(env, 5)
obs, _ = env.reset(seed=123)
print(f'ViewSizeWrapper: {obs["image"].shape}') # (5, 5, 3)

It seems like the order of the wrappers matters the most. If you change the order then you get the whole grid.

@NishanthVAnand Does this change of order fix the issue?

[NishanthVAnand]

Not quite. The final observation is an RGB image of 5x5 pixels. The expected output is an RGB image corresponding to 5x5 view of the agent. Since tile_size=8, the expected shape of the RGB image is 40x40x3

[pseudo-rnd-thoughts]

@NishanthVAnand Could you provide an example script of what you expect and what you get

[NishanthVAnand]

I have printed obs['image'] at the end of each wrapper application from your code to demonstrate what I mean.

import gymnasium as gym
from minigrid.wrappers import ViewSizeWrapper, RGBImgPartialObsWrapper
import matplotlib.pyplot as plt

env = gym.make("BabyAI-GoToRedBall-v0", render_mode="rgb_array")
obs, _ = env.reset(seed=123)
print(f'base image shape: {obs["image"].shape}') # (7,7,3)

base image shape: (7, 7, 3)

print(obs["image"][0])
array([[2, 5, 0],
       [1, 0, 0],
       [1, 0, 0],
       [1, 0, 0],
       [1, 0, 0],
       [1, 0, 0],
       [6, 1, 0]], dtype=uint8)

env = RGBImgPartialObsWrapper(env)
obs, _ = env.reset(seed=123)
print(f'RGBImgPartialObsWrapper: {obs["image"].shape}') # (56, 56, 3)

RGBImgPartialObsWrapper: (56, 56, 3)

from PIL import Image
Image.fromarray(obs["image"])

env = ViewSizeWrapper(env, 5)
obs, _ = env.reset(seed=123)
print(f'ViewSizeWrapper: {obs["image"].shape}') # (5, 5, 3)

ViewSizeWrapper: (5, 5, 3)

print(obs["image"][0])
array([[1, 0, 0],
       [1, 0, 0],
       [5, 1, 0],
       [7, 0, 0],
       [1, 0, 0]], dtype=uint8)

As you can see, the final output (obs['image']) is no longer an RGB image.

[jkterry1]

@NishanthVAnand would you mind creating a PR for this?