Using the RGBImgObsWrapper

[FZuffer]

Hi,
The default minigrid observation space is of size (7,7,3) and this is an encoding and not the actual RGB image(with pixel values) of the state space. How fair is it to use this default state space to be processed through a CNN as shown in rl-starter-files? Why not use the actual RGB image which could be obtained through the RGBImgObsWrapper instead? Or is there a specific use case to use the wrapper?

[pseudo-rnd-thoughts]

Could you provide an example script for this?

[HarryFindlay03]

Hi, The default minigrid observation space is of size (7,7,3) and this is an encoding and not the actual RGB image(with pixel values) of the state space. How fair is it to use this default state space to be processed through a CNN as shown in rl-starter-files? Why not use the actual RGB image which could be obtained through the RGBImgObsWrapper instead? Or is there a specific use case to use the wrapper?

I understand reply here is very late, but may be useful for people in the future.

I do find it a bit weird that the default observation space image value is some symbolic representation by default and not the pure rgb image. But I am sure there is a valid reason for this. Anyway to your question, yes it is fair to use the CNN in rl-starter-files. I have used it previously and got good results. (~0.9 mean reward, after 1.5e6 steps using PPO, not converged).

If using the RGB image which can be gathered like so

env = ImgObsWrapper(RGBImgObsWrapper(env, tile_size=tile_size))

then I did find the CNN in rl-starter-files was not good enough so I simply used the cnn implementation used in Mnih et al (2015) github link

class MinigridFeaturesExtractor(BaseFeaturesExtractor):

    def __init__(self, observation_space: gym.Space, features_dim: int = 512, normalized_image: bool = False) -> None:
        super().__init__(observation_space, features_dim)

        n_input_channels = observation_space.shape[0]

        # Copying Nature CNN

        self.cnn = nn.Sequential(
            nn.Conv2d(in_channels=n_input_channels, out_channels=32, kernel_size=8, stride=4),
            nn.ReLU(),
            nn.Conv2d(in_channels=32, out_channels=64, kernel_size=4, stride=2),
            nn.ReLU(),
            nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, stride=1),
            nn.ReLU(),
            nn.Flatten(),
        )

        with torch.no_grad():
            n_flatten = self.cnn(torch.as_tensor(observation_space.sample()[None]).float()).shape[1]

        self.linear = nn.Sequential(
            nn.Linear(n_flatten, features_dim),
            nn.ReLU()
        )

    
    def forward(self, observations: torch.Tensor) -> torch.Tensor:
        return self.linear(self.cnn(observations))

I have found this works quite nicely :)

Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G, Petersen S. Human-level control through deep reinforcement learning. nature. 2015 Feb;518(7540):529-33.