Super Mario World TAS Agent

Teaching reinforcement learning agents to speedrun Super Mario World, frame by frame.

We wanted to see whether an agent could play a game the way a tool assisted speedrunner does, reacting frame by frame with an optimal sequence of inputs to finish a level in the least amount of time possible. We chose Super Mario World and compared the performance of three different reinforcement learning algorithms, DQN, PPO and SAC. Group project of five for the Modern Game AI course at Leiden University.

The PPO agent making its way through the first level.

Links · 💻 PPO code · 🤗 Weights · 📄 Report

How it works

All three agents read the game as a stack of grayscale frames through a small convolutional network. DQN puts a value head on top, while PPO and SAC use separate actor and critic heads. Rather than expose the whole controller we gave them a curated set of 42 button combinations, and the reward is forward progress plus a large bonus for finishing the level. To read those values out of the running game we tracked down the SNES memory addresses behind Mario’s position, lives and the end of level flag, using the integration tool that ships with stable-retro, the emulator we ran the game under. The PPO agent, which is the one in the repo, adds a custom multi process vectorised environment that runs many emulator copies in parallel to significantly speed up the search.

Where it got to

It stayed a proof of concept. The agents learn to clear the early sections and make real progress, but only a handful of runs ever reached the end of the level, and the reason they never settle comes down to the reward. Without a time penalty, a fast finish and a slow one earn the same reward, so there is no single best run for the agent to reinforce. Introducing a time penalty causes the agent to learn that dying as fast as possible results in a higher reward than finishing the level slowly. Adding a death penalty on top of the time penalty only shifts the absolute reward, but doesn’t change anything if the agent is never able to reach the finish. We assume that the individual rewards and penalties need to be carefully tuned for the policy to converge to the desired behaviour. Clever reward scheduling or curriculum learning is presumably also beneficial. Having prior knowledge of which buttons are needed to optimally complete a level can also be used to greatly decrease the search space and is how the demo run was achieved. The rest is in the report.