Document gSDE inference behavior for PPO, SAC, and A2C

[Copilot]

Fixes #1767

This PR adds documentation clarifying the behavior of gSDE (Generalized State-Dependent Exploration) during inference for models trained with use_sde=True.

Problem

When using models trained with gSDE, users expect that setting deterministic=False in model.predict() would provide stochastic behavior similar to training. However, the automatic noise resetting controlled by sde_sample_freq during training does not occur during inference, resulting in deterministic behavior even when deterministic=False.

Solution

Added informative notes to the documentation of PPO, SAC, and A2C explaining:

1. Expected behavior: gSDE is primarily designed for training, with deterministic behavior recommended for inference
2. Current limitation: model.predict() does not automatically reset noise, leading to deterministic behavior even with deterministic=False
3. Workaround: If stochastic behavior is needed during inference, users must manually call model.policy.reset_noise(env.num_envs) at appropriate intervals

Example of the documented behavior

from stable_baselines3 import PPO

# Training with gSDE
model = PPO("MlpPolicy", "Pendulum-v1", use_sde=True, sde_sample_freq=4)
model.learn(total_timesteps=10000)

vec_env = model.get_env()
obs = vec_env.reset()
# During inference - deterministic behavior (recommended)
action, _ = model.predict(obs, deterministic=True)

# If stochastic behavior is needed during inference
# Manual noise resetting is required
step_count = 0
while True:
    if step_count % 4 == 0:  # Reset every 4 steps (matching sde_sample_freq)
        model.policy.reset_noise(vec_env.num_envs)
    
    action, _ = model.predict(obs, deterministic=False)
    obs, reward, done, info = vec_env.step(action)
    step_count += 1

The documentation changes are minimal and focused, adding clear guidance without modifying any code behavior. This addresses the user's question and provides practical guidance for proper gSDE usage during inference.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[araffin]

@copilot please update the changelog too

[Copilot]

@copilot please update the changelog too

Updated the changelog with documentation entry for gSDE inference behavior clarification (commit 5a3bf4a)