Soul joins you in Minecraft as a friend, on Java and Bedrock. It explores alongside you, helps with tasks, shares discoveries, and remembers your time together when you talk again in the app.
The Problem
Playing with a friend involves more than completing objectives. Players explore independently, share discoveries, and build memories they return to after the game.
Game-playing AI is often evaluated on winning, benchmark performance, or progression. An AI friend must also participate in the shared experience: helping with plans, responding to discoveries, and knowing when to give the other player space.
This introduces several requirements. Soul must make joining a world straightforward, translate informal requests into completed tasks, and handle threats without losing task progress. It must also decide when to initiate an interaction and retain useful memories across the game and the app.
Our Approach
Soul combines an LLM, a low-latency classifier, and deterministic controllers. The LLM handles dialogue and open-ended planning. The classifier interprets requests and selects actions. Code constructs the available actions, executes them, and verifies the results.
Joining your world
You link your Minecraft account once in the Soul app, then connect through a shared server address. A proxy routes your connection to the pod hosting your world, where Soul's bot runs alongside the game server.
World files persist independently of the compute instance. When a world is inactive, its files return to storage. On the next connection, the world is restored into an available slot. The server address and saved world remain the same.
The proxy authenticates the player, queries the Soul server for their world's slot, and forwards the connection. Soul's bot connects to the Paper server within that pod. Geyser supports Bedrock clients; consoles discover the world through an Xbox Live friend session.
Playing on console and mobile
We first implemented Java Edition support, then added Bedrock access for players on phones, tablets, and consoles.
Who decides what
The LLM generates dialogue, sets goals, and plans open-ended tasks. At around 1.7 seconds per call, it is too slow to control time-sensitive actions directly.
The decision model classifies predefined options in ~0.2 seconds and returns confidence scores. Code constructs the options from the current game state, computes counts, and verifies outcomes. The classifier selects only actions whose prerequisites have been checked.
The LLM handles goals, dialogue, and open-ended planning. The classifier selects among predefined options in ~0.2 seconds. Code constructs those options, computes counts, and verifies execution.
One message, two models
A message such as "get me some iron" is sent to the classifier and the LLM in parallel. The classifier identifies and routes the request in ~0.2 seconds, allowing execution to begin before the LLM finishes its reply.
During execution, the classifier selects subsequent actions. The LLM periodically reviews progress, skips responses when the state is unchanged, and comments on relevant events such as finding iron.
The classifier starts the task in 0.2 seconds, before the LLM has replied, then picks each next step. The time axis breaks after 2.4 seconds; later timings are illustrative.
Understanding the ask
Player messages range from resource requests to movement instructions, building tasks, and casual conversation. Code assembles a compact state containing the message, recent chat, inventory, nearby entities and objects, and current task. A single classifier call selects one of 21 routes and its required parameters, such as resource type and quantity.
The router selects the correct route for nearly all beta-player messages evaluated. Low-confidence predictions fall back to phrase matching, providing an alternative path when classification is uncertain.
One classifier call selects a route and its parameters from the assembled game state. Each route maps to an implemented controller. The probabilities shown are illustrative.
Doing the job
Gathering iron from an empty inventory requires a sequence of dependencies: wood, crafting materials, a wooden pickaxe, and then a stone pickaxe. The controller resolves these through a repeated observe–select–execute loop. Code constructs valid actions from the current state, the classifier selects one, and the controller updates the state after execution.
Actions are gated by their prerequisites: mining is unavailable until Soul has a pickaxe. If only one valid action remains, code executes it without a classifier call. Completion is verified against the inventory.
An illustrative iron gathering task from an empty inventory. Outlined cells indicate available actions; filled cells indicate selections. The controller reconstructs the action set after each step.
Staying alive
A deterministic reflex loop checks threats, hunger, and submersion every 750 milliseconds. It handles these conditions directly, without waiting for model inference.
A reflex interrupts the active task and saves a checkpoint. Once the immediate condition is resolved, execution resumes from that checkpoint.
Health and hunger during 90 seconds of an iron gathering task. The reflex loop interrupts execution to handle immediate conditions, then resumes task selection from the saved checkpoint.
Knowing when
Soul can also initiate interactions without a request. Code identifies opportunities grounded in the current game state, such as offering a flower, sharing a discovery, or noting nightfall.
Timing rules suppress these interactions during tasks, for three minutes after task completion, and for four minutes after the previous initiated interaction. Outside these windows, the classifier decides whether to act, and the LLM generates dialogue from the observed state.
Code identifies possible interactions from the game state. Timing rules determine when they are eligible; the classifier decides whether to initiate one. The LLM then generates the dialogue.
Remembering the game
The system records selected events, including chat, deaths, and level-ups, as they occur. Every ten minutes, the LLM summarizes them in Soul's voice. The memory model processes these summaries through the same pipeline used for conversations, retaining useful information and linking it to existing memories.
These memories carry into the app, allowing Soul to refer to shared events or unfinished plans in later conversations.
Events from a 40-minute session, grouped by type. Selected events are recorded immediately, summarized every ten minutes, and processed by the memory model for retention.