🎥 Here you can see our AI gaming agents in action, demonstrating their gameplay strategies across different games!
| AI Gameplays |
|
| AI Gameplays |
|
| GPT-4o Gameplay | Claude-3.7 Gameplay |
|
|
| Claude-3.7 Gameplay |
|
| Candy Crush Gameplay |
|
The goal of this repo is to provide an easy solution of deploying computer use agents (CUAs) that run on your PC and laptops. As part of LMGames, our current focus is on building local gaming agents.
Current features:
- Gaming agents for Platformer and Atari games.
- Clone this repository:
git clone https://github.com/lmgame-org/GamingAgent.git
cd GamingAgent
- Install dependency:
conda create -n game_cua python==3.10 -y
conda activate game_cua
pip install -e .
Currently we support gaming agents based on the following models:
- OpenAI:
- gpt-4o
- gpt-4o-mini
- o1
- o3-mini (low, medium, high)
- Anthropic:
- claude-3-5-haiku-20241022
- claude-3-5-sonnet-20241022
- claude-3-7-sonnet-20250219 (thinking mode: True or False)
- Gemini:
- gemini-1.5-pro
- gemini-2.0-pro
- gemini-2.0-flash
- gemini-2.0-flash-thinking-exp
- Deepseek:
- chat (V3)
- reasoner (R1)
Set your API keys with:
export OPENAI_API_KEY={YOUR_OPENAI_API_KEY}
export ANTHROPIC_API_KEY={YOUR_ANTHROPIC_API_KEY}
export GEMINI_API_KEY={your_GEMINI_API_KEY}
Install your Super Mario Bros game. In our demo, we adopt SuperMarioBros-C.
Navigate to the repo and follow the installation instructions.
- Once the game is built, download and move the ROM file:
mv path-to-your-ROM-file/"Super Mario Bros. (JU) (PRG0) [!].nes" $YOUR_WORKPLACE/SuperMarioBros-C/build/
- Launch the game with
cd $YOUR_WORKPLACE/SuperMarioBros-C/build
./smbc
- Full screen the game by pressing
F. You should be able to see:
- Open another screen, launch your agent in terminal with
cd $YOUR_WORKPLACE/GamingAgent
python games/superMario/mario_agent.py --api_provider {your_favorite_api_provider} --model_name {official_model_codename}
- Due to concurrency issue, sometimes the agent will temporarily pause your game by pressing
Enter. To avoid the issue, you can launch the agent only after entering the game upon seeing:
--concurrency_interval: Interval in seconds between starting workers.
--api_response_latency_estimate: Estimated API response latency in seconds.
--policy: 'long', 'short', 'alternate' or 'mixed'. In 'long' or 'short' modes only those workers are enabled.
You can implement your own policy in mario_agent.py! Deploying high-concurrency strategy with short-term planning streaming workers vs. low-concurrency strategy with long-term planning workers, or a mix of both.
In our early experiments, 'alternate' policy performs well. Try it yourself and find out which one works better!
Install your Sokoban game. Our implementation is modified from the sokoban.
- Launch the game with
cd $YOUR_WORKPLACE/GamingAgent
python games/sokoban/sokoban.py
You should be able to see the first level:
- Open another terminal screen, launch your agent in terminal with
python games/sokoban/sokoban_agent.py
--api_provider: API provider to use.
--model_name: Model name.
--modality: Modality used, choice of ["text-only", "vision-text"].
--thinking: Whether to use deep thinking.
--starting_level: Starting level for the Sokoban game.
--num_threads: Number of parallel threads to launch. default=10.
num_threads to 1.
2048 is a sliding tile puzzle game where players merge numbered tiles to reach the highest possible value. In our demo, we adopt and modify 2048-Pygame
Run the 2048 game with a defined window size:
python games/game_2048/game_logic.py -wd 600 -ht 600
Use Ctrl to restart the game and the arrow keys to move tiles strategically.
Start the AI agent to play automatically:
python games/2048/2048_agent.py--api_provider: API provider to use.
--model_name: Model name (has to come with vision capability).
Install your Tetris game. In our demo, we adopt Python-Tetris-Game-Pygame.
- Launch the game with
cd $YOUR_WORKPLACE/Python-Tetris-Game-Pygame
python main.py
main.py, line 23: change event time to 500~600ms.
You should be able to see:
-
Adjust Agent's Field of Vision. Either full screen your game or adjust screen region in
/games/tetris/workers.py, line 67 to capture only the gameplay window. For example, inPython-Tetris-Game-Pygamewith MacBook Pro, change the line toregion = (0, 0, screen_width // 32 * 9, screen_height // 32 * 20). -
Open another screen, launch your agent in terminal with
cd $YOUR_WORKPLACE/GamingAgent
python games/tetris/tetris_agent.py
--api_provider: API provider to use.
--model_name: Model name (has to come with vision capability).
--concurrency_interval: Interval in seconds between consecutive workers.
--api_response_latency_estimate: Estimated API response latency in seconds.
--policy: 'fixed', only one policy is supported for now.
Currently we find single-worker agent is able to make meaningful progress in the Tetris game. If the gaming agent spawns multiple independent workers, they don't coordinate well. We will work on improving the agent and gaming policies. We also welcome your thoughts and contributions.
You can freely test the game agent on the online version of Candy Crush.
The example below demonstrates Level 1 gameplay on the online version of Candy Crush.
-
Adjust the agent's field of vision
To enable the agent to reason effectively, you need to crop the Candy Crush board and convert it into text. Adjust the following parameters:--crop_left,--crop_right,--crop_top,--crop_bottomto define the board cropping area from the image.--grid_rows,--grid_colsto specify the board dimensions.
Check the output in./cache/candy_crush/annotated_cropped_image.pngto verify the adjustments.
-
Launch the agent
Open a terminal window and run the following command to start the agent:cd $YOUR_WORKPLACE/GamingAgent python games/candy/candy_agent.py
--api_provider: API provider to use.
--model_name: Model name (has to come with vision capability).
--modality: Modality used, choice of ["text-only", "vision-text"].
--thinking: Whether to use deep thinking.(Special for anthropic models)
The Candy Crush game agent has two workers: one extracts board information from images and converts it into text, and the other plays the game. The AI agent follows a simple prompt to play Candy Crush but performs surprisingly well. Feel free to create and implement your own policy to improve its gameplay.
