Cooperative Multi-Agent RL in Overcooked

I trained two RL agents to cook together in Overcooked, starting with independent PPO and ending with a centralized critic and recipe-aware rewards.

Here are the final policies on three layouts, ordered by how much coordination they require. Below, I trace the path from a simple baseline to the changes that solved each one.

cramped_room

coordination_ring

counter_circuit

Held-out performance: 10.8, 7.3, and 7.9 soups per episode from left to right. The target was seven.

Overcooked is a cooperative cooking game in which players share a kitchen and race to complete orders. Overcooked-AI, introduced by Carroll et al. in 2019, adapts that premise into a two-chef research environment for studying coordination.¹

Each episode reduces the kitchen to one repeated recipe:

1. place three onions in a pot,
2. wait for the soup to cook,
3. plate it,
4. and carry it to a serving counter.

The objective is to complete as many soups as possible within a finite-horizon episode.

The environment includes several kitchen layouts, each requiring a different kind of coordination, from sharing a cramped workspace to dividing work across barriers.

The original Overcooked-AI paper asked whether human data could produce better partners. Its human-aware pipeline first fit a behavior-cloned partner to human-human trajectories, then trained a PPO agent to play with that model. On four of the five layouts, PPO began in self-play before the training partner was gradually replaced by the learned human model. The paper compared these agents with human-data-free self-play and population-based baselines.¹

This experiment stays on the human-data-free side of that comparison: no demonstrations, no pretrained partner, and both agents learning together from scratch. I also impose one algorithm, one fixed reward function, and one hyperparameter set across all three layouts, with no layout-specific tuning. I count a layout as "solved" when the agents average at least 7 soups per episode.

MDP formulation

Each episode is a fully observable cooperative Markov game (equivalently, a team MDP) with horizon $H = 400$:

In the kitchen, those terms are:

• State $\mathcal{S}$: both chefs' positions, orientations, and held objects; the objects on the map; and the contents and cooking timer of every pot.
• Actions $\mathcal{A}_i$: up, down, left, right, stay, or interact. The chefs act at the same time, giving $6^2 = 36$ possible joint actions per step.
• Transition $P$: the game resolves both actions together, including movement, collisions, pickups, and interactions with counters and pots.
• Horizon $H$: in this setup, after 400 steps, the episode ends and the kitchen resets.

There is no fog of war. Each agent receives a 96-dimensional, player-centric view $o_t^i = \phi_i(s_t)$ derived from the full state: held objects, relative positions of relevant stations and ingredients, pot states, and features for the other chef. The coordinate frame differs by player, but both can see the whole task state.

The default environment reward is completely sparse. It pays out only when a finished soup reaches the serving counter:

There is no default reward for picking up an onion, filling a pot, starting it cooking, or plating the soup. This is the unshaped baseline.

Both chefs receive the same reward. Their policies act independently, but training tries to maximize their shared return:

The difficulty comes from coordination and credit assignment, not hidden information. A delivery depends on a long sequence of actions from both chefs, while the final $+20$ gives no indication of which earlier decisions made it happen.

Layouts under test

The layouts share the objective but differ sharply in how much the chefs must coordinate:

Low coordination

cramped_room

Avoid collisions in a shared workspace.

Medium coordination

coordination_ring

Agree on a traffic convention.

High coordination

counter_circuit

Divide roles and hand off ingredients.

The table below summarizes the experiments. Entries such as 3/3 show how many training seeds reached the 7-soup target; the final row instead shows the mean number of soups delivered across 100 held-out episodes:

Preliminary: PPO

Both algorithms in this post are PPO variants that differ only in the critic. PPO is an actor–critic method:² the actor $\pi_\theta(a \mid o)$ maps an observation to a distribution over the six actions, while the critic $V_\omega(o)$ predicts the expected discounted return.

The actor learns from the advantage $\hat{A}_t$: whether an action worked out better or worse than the critic expected. PPO estimates it with GAE($\lambda$),³ beginning with the one-step TD error

and accumulating those errors along the trajectory:

$\lambda$ trades bias against variance: smaller values rely more heavily on the critic, while larger values incorporate more of the sampled trajectory. PPO then clips the policy update so that reusing a rollout for several optimization epochs does not reward moving the new policy arbitrarily far from the policy that collected it. An entropy bonus discourages the policy from becoming deterministic too early. Critic quality matters here because its estimates directly shape the advantages used to train the actor.

Both IPPO and MAPPO use the same PPO update. The actor always evaluates local observations; only critic_obs changes between the two algorithms.

1for _ in range(config.update_epochs):
2    for mb in buffer.get_minibatches(config.num_minibatches):
3        new_log_prob, entropy = actor.evaluate(mb.obs, mb.actions)
4        value = critic(mb.critic_obs)
5
6        ratio = (new_log_prob - mb.log_probs).exp()
7        unclipped = ratio * mb.advantages
8        clipped = torch.clamp(
9            ratio, 1 - config.clip_eps, 1 + config.clip_eps
10        ) * mb.advantages
11        policy_loss = -torch.min(unclipped, clipped).mean()
12
13        v_clipped = mb.values + torch.clamp(
14            value - mb.values, -config.clip_eps, config.clip_eps
15        )
16        value_loss = torch.max(
17            (value - mb.returns) ** 2,
18            (v_clipped - mb.returns) ** 2,
19        ).mean()
20
21        loss = (
22            policy_loss
23            + config.value_coef * value_loss
24            - config.entropy_coef * entropy.mean()
25        )

This shared objective makes the comparison unusually direct: IPPO and MAPPO optimize the same actor loss from the same actions and rewards, but their critics produce different baselines for mb.advantages.

Starting simple: IPPO

Independent PPO (IPPO)⁴ is just that algorithm run per agent, with two twists for the two-chef setting:

• Parameter sharing: the chefs are treated as interchangeable, so they share a single actor $\pi_\theta$; the two agents are maintained as the batch dimension.
• Decentralized critic: each agent estimates its value from only its own 96-dimensional observation, which is the "independent" part. The TD error is therefore computed separately for each agent:

Both chefs receive the same team reward $r_t$, but each value estimate $V_\omega(o^i_t)$ is conditioned on a single agent's observation. The return also depends on the partner's behavior, which changes as the partner learns. As a result, similar observations can lead to very different returns. The critic must average over that variation, increasing the variance of $\delta^i_t$ and of the advantage estimates used to update the policy.

IPPO is an inexpensive baseline and works well when either chef can make progress independently. The three layouts test how its performance changes as the task requires tighter coordination.

IPPO performance across layouts

I ran 3 seeds per layout and measured mean soups over the final 100 updates. Results written as mean ± standard deviation summarize variation across those seeds; the curves show the mean and min–max range.

IPPO soups per episode (mean and min–max band over 3 seeds). Performance converges quickly on cramped_room, improves more slowly on coordination_ring, and remains near zero on counter_circuit.

cramped_room

IPPO solved this layout consistently, achieving 10.5 ± 0.1 soups with all three seeds exceeding the target. Performance converged by approximately update 60 with minimal variation across seeds.

Representative 400-step IPPO trajectory on cramped_room.

coordination_ring

Performance depended on the training budget. After 800 updates, IPPO achieved 3.8 ± 2.4 soups, and none of the three seeds reached the target.

Extending the same runs to 2,000 updates increased performance to 9.5 ± 0.6 soups, with all three seeds reaching the target. The result at 800 updates therefore reflected insufficient training rather than an inability of IPPO to solve the layout. Evaluating the longer training horizon was necessary to distinguish between these explanations.

Representative 400-step stochastic IPPO trajectory on coordination_ring; 4 soups delivered.

counter_circuit

IPPO was unreliable on this layout. After 800 updates, it achieved 0.0 ± 0.0 soups across the three seeds, and the best individual episode produced at most one soup. Two controls tested whether network capacity or training budget accounted for the result:

• Network capacity: doubling the hidden-layer width from 256 to 512 did not improve performance, which remained at 0.0 soups.
• Training budget: after 2,000 updates, one of the three seeds reached 7.9 soups while the other two remained at zero.

IPPO can therefore solve counter_circuit, but its success rate is low and its sample requirements are substantially higher than on the other layouts.

Representative 400-step IPPO trajectory on counter_circuit.

In terms of the training budget needed to reach ≥ 7:

Motivation for a centralized critic

At 800 updates, IPPO is reliable on cramped_room, variable on coordination_ring, and ineffective on counter_circuit. Because the layouts primarily differ in coordination demand, critic conditioning is a natural hypothesis, although this pattern alone does not establish causality.

IPPO estimates the shared return from each agent-centered observation, $V(o_i)$. As the partner's policy changes, similar inputs can lead to different team returns, increasing variance in the TD residuals and advantage estimates. Longer training can compensate by providing more samples, but it does not change what the critic conditions on.

The working hypothesis was that learning gets harder when the value of one chef's action depends strongly on the other chef's position and behavior. Avoiding a collision in cramped_room requires relatively little shared structure; establishing traffic flow around coordination_ring and coordinating handoffs across counter_circuit require more. That suggests testing the critic, though it does not identify the mechanism by itself.

MAPPO isolates this factor by replacing $V(o_i)$ with $V(o_0, o_1)$ while holding the actor, reward function, PPO objective, and training budget fixed. The joint critic is used only during training, so execution remains decentralized. If critic conditioning is the bottleneck, MAPPO should provide little benefit on cramped_room, improve coordination_ring, and help counter_circuit only if its failure is due to value estimation rather than exploration or reward design.

MAPPO formulation

Multi-Agent PPO (MAPPO)⁵ retains the same per-agent actor but trains the critic on both agents' observations, $V(o_0, o_1)$.

The TD residual becomes:

The importance ratio, clipped objective, GAE calculation, and shared actor remain unchanged. There is no longer an $i$ on $\delta_t$: both agents use an advantage computed from the same joint value estimate.

Each $o_i$ already contains features for the other chef, so MAPPO does not reveal hidden state to the critic. Instead, concatenating $(o_0, o_1)$ provides both agent-centered encodings of the global state and makes the joint configuration explicit in the value-function input. Any resulting improvement therefore comes from the critic's representation and the training signal it provides, not from giving the actors additional information.

Only the critic changes; the actor used at execution remains unchanged:

This setup is known as centralized training, decentralized execution (CTDE).⁶ Each actor still acts on its own observation, and the joint critic is discarded after training, so the agents do not need to communicate at inference time.

The excerpts below are lightly condensed from the implementation. In code, the critic distinction is small enough to isolate behind an observation adapter: IPPO passes each local observation through unchanged, while MAPPO rotates the agent axis before concatenation so a shared critic receives a self-first joint view for each chef, $[o_i, o_{-i}]$.

1class LocalAdapter(ObsAdapter):
2    def transform(self, obs: torch.Tensor) -> torch.Tensor:
3        return obs
4
5
6class SelfCenteredJointAdapter(ObsAdapter):
7    def transform(self, obs: torch.Tensor) -> torch.Tensor:
8        # (..., agents, obs_dim) -> (..., agents, agents * obs_dim)
9        n_agents = obs.shape[-2]
10        return torch.cat(
11            [obs.roll(-k, dims=-2) for k in range(n_agents)],
12            dim=-1,
13        )

For two chefs, the first output row is $[o_0, o_1]$ and the second is $[o_1, o_0]$. The actor never sees either concatenation; it continues to consume only $o_i$.

For a clean comparison with IPPO, I kept the actor, optimizer, reward, seeds, and training budget the same. The only change is the critic's input, which grows from 96 to 192 dimensions. If the joint critic helps, it should show up in two ways:

1. sample efficiency on coordination_ring: MAPPO should reach $\ge 7$ in far fewer than IPPO's ~2000 updates.
2. reliability for counter_circuit: MAPPO should clear $\ge 7$ on more than 1/3 seeds.

MAPPO performance across layouts

At the same 800-update budget on coordination_ring, centralizing the critic improves both sample efficiency and consistency:

• MAPPO: 8.0 ± 0.6 soups, 3/3 seeds solved.
• IPPO: 3.8 ± 2.4 soups, 0/3.

MAPPO reaches the target within the original 800-update budget, whereas IPPO requires substantially longer training. The comparison is about how quickly the two methods learn coordinated behavior, not whether IPPO can eventually solve coordination_ring.

coordination_ring: the across-seed MAPPO mean (blue) first crosses the 7-soup target around update 675; IPPO (orange) needs its 2000-update budget and doesn't stay above 7 until ~update 1666.

On coordination_ring, MAPPO learns faster and varies less across seeds, consistent with the chefs finding a stable traffic convention sooner. The mechanism is not fully isolated, however. Each agent's observation already encodes the full state, while the joint critic concatenates both observations and therefore receives a higher-dimensional input. The experiment cannot separate the effects of joint conditioning, added input capacity, and improved credit assignment.

The gain does not carry over to counter_circuit: MAPPO remains at 0.0 after both 800 and 2,000 updates. Whatever the joint critic fixes on the ring, it does not produce a complete collect–cook–plate–deliver sequence on the circuit. The next question is where that sequence breaks down.

Why wasn't counter_circuit learning?

A delivery count of zero tells us only that the full task failed. To find the break, I tracked the stages that precede a delivery: collect three onions, start cooking, and plate the finished soup.

Here, a valid soup contains the required three onions. The environment also permits a pot to start cooking with only one or two onions; those incomplete recipes are referred to as invalid soups below.

Both interventions changed the agents' behavior, but neither produced a valid soup. The aggregate metrics were also suspicious: the dashboard reported roughly ten "soups plated" per episode, yet no soup was ever delivered.

The plating metric counted the wrong thing

The apparent plating count came from Overcooked-AI's generic soup_pickup event. That event also fires when an agent retrieves a soup from a counter, so repeated movement of the same invalid soup was counted as new progress. The metric measured pickups, not completed recipes.

The replacement metric compares held objects across consecutive states. A dish → soup transition means the chef plated directly from a pot; any other new soup pickup came from a counter and is tracked separately.

1desc = self._held_desc(i)       # current (object_name, ingredient_count)
2prev = self._prev_held[i]
3prev_name = prev[0] if prev is not None else None
4
5if desc is not None and desc[0] == "soup" and prev_name != "soup":
6    n_ingredients = min(desc[1], 3)
7    if prev_name == "dish":
8        # This soup was plated from a pot.
9        self._ep_counts[f"pot_soup_pickup_{n_ingredients}"] += 1
10    else:
11        # This soup was merely retrieved from a counter.
12        self._ep_counts["counter_soup_pickup"] += 1
13
14self._prev_held[i] = desc

I therefore inspected the event sequence in 30 failed episodes:

• 268 cooking actions started with one onion in the pot
• 263 one-onion soups were removed from pots
• 0 valid three-onion soups were produced

For comparison, a successful long-run counter_circuit policy started cooking 180 times, always with three onions, and completed 160 deliveries. The failed agents never reached the delivery stage because they never made a valid soup. The breakdown happened when they chose to start cooking an incomplete recipe.

Root cause: the shaped reward favors an invalid recipe

Here's an example trajectory that shows what the counter_circuit policy learned instead.

The learned loop: add one onion $\rightarrow$ start cooking early $\rightarrow$ pick up the invalid soup (+5 reward) $\rightarrow$ dump it because it cannot be delivered $\rightarrow$ repeat.

Two properties of the environment and reward function create this behavior:

1. You can start cooking a pot with any number of onions; one is enough.
2. The default shaping pays +5 for picking up a soup from a pot, and it never checks the recipe.

The flawed bonus exists on every layout, but counter_circuit makes it especially attractive: its central counter turns a valid recipe into repeated trips or a coordinated handoff, while premature cooking offers a short, reliable path to +5. The easier layouts discover valid deliveries early enough for the sparse +20 reward to reinforce the full recipe. This interpretation fits the rollout traces, though the experiments do not isolate geometry as the sole cause.

The policy found a reliable way to earn the specified reward without completing the task. This is an instance of Goodhart's law. More exploration or a centralized critic cannot repair that incentive, and delivery shaping arrives too late. The reward itself has to change.

The fix: recipe-aware reward shaping

Make the shaping care about the recipe, and penalize the offending action directly:

Invalid soup pickups now earn zero reward, and starting an incomplete recipe incurs a penalty. Valid cooking and plating receive bonuses. MAPPO and the training hyperparameters are unchanged.

The critical branch detects the exact step when a pot begins cooking, then checks the ingredient count and current order before assigning reward. The generic soup-pickup bonus is disabled up front, so an invalid soup cannot earn the old +5 later.

1reward_shaping = {
2    "PLACEMENT_IN_POT_REW": 3,
3    "DISH_PICKUP_REWARD": 3,
4    "SOUP_PICKUP_REWARD": 0,
5}
6
7for pos in self.pot_locs:
8    begun = self._pot_begun(pos)
9    if begun and not self._prev_pot_begun[pos]:
10        soup = self.state.objects[pos]
11        n_ingredients = min(len(soup.ingredients), 3)
12
13        if n_ingredients == 3 and self._is_ordered_recipe(soup.ingredients):
14            cook_reward += 5
15        elif n_ingredients < 3:
16            cook_reward -= 3 * n_ingredients + 1
17
18    self._prev_pot_begun[pos] = begun

The delivery term is potential-based rather than a per-step carrying bonus: it pays only for reducing the distance to the serving counter while holding a valid soup. Moving away gives an equal negative reward, and standing still gives zero.

1curr = self._serve_dist(agent)  # None unless holding a valid plated soup
2prev = self._prev_serve_dist[agent]
3
4if curr is not None and prev is not None:
5    delivery_reward[agent] = prev - curr
6
7self._prev_serve_dist[agent] = curr

counter_circuit after the reward change: 1-onion cook-starts (red) collapse to zero while valid cooking, plating, and delivered soups (blue) climb past the 7-soup target and plateau near 8.

Frozen MAPPO policy on counter_circuit: 8 soups delivered in a 400-step stochastic evaluation rollout.

How much reward shaping was actually necessary?

The table above shows the full reward function. For the ablation, I removed three of its terms in turn on counter_circuit: the incomplete-recipe penalty, the valid-cook bonus, and carry shaping.

• Remove the penalty $\rightarrow$ one-onion cooking returns. Removing the +5 invalid-soup bonus is insufficient because a residual shaped path remains: place one onion for +3, then collect the "useful" dish pickup bonus after that pot cooks. Directly penalizing the premature cook action is necessary to eliminate this local optimum.
• Remove the valid-cook bonus or the carry shaping $\rightarrow$ the agents fill both pots to three onions and then stall. Without those terms, the downstream delivery reward is too sparse to reinforce the transition from a full pot to cooking and delivery.

Final results across all three layouts

With MAPPO and the recipe-aware reward function described above, the same hyperparameter set clears the target on all three layouts. The results below use frozen policies evaluated over 100 held-out stochastic episodes each:

Held-out evaluation (no learning): cramped_room 10.8, coordination_ring 7.3, counter_circuit 7.9 soups per episode. All are above the target of 7 (dashed line).

Takeaways

The main learning from these experiments is the value of a good critic in a multi-agent RL setup. We saw that MAPPO improved sample efficiency considerably but still fell short on counter_circuit due to insufficient reward shaping.

On coordination_ring, MAPPO reached the target in 800 updates, while IPPO needed 2,000. Both algorithms used the same actor and reward function. The difference was that the MAPPO critic used both agents' observations to estimate the shared return. This gave the actor a better learning signal when its outcome depended on the other agent's behavior.

A better critic was not enough on counter_circuit. The agents learned to place one onion in the pot, start cooking, pick up a dish, and repeat. This loop earned shaped reward, but it never produced a valid soup. The event trace showed 268 one-onion cook starts and zero valid soups.

The reward ablations showed that both penalties and intermediate rewards were necessary. Without the incomplete-recipe penalty, the one-onion loop returned. Without the valid-cook bonus or carry shaping, the agents filled the pots and stalled. The critic improved how efficiently the agents learned, while reward shaping determined which behavior they learned.

References

1. Micah Carroll, Rohin Shah, Mark K. Ho, et al. "On the Utility of Learning about Humans for Human-AI Coordination." NeurIPS, 2019.

2. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. "Proximal Policy Optimization Algorithms." arXiv:1707.06347, 2017.

3. John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. "High-Dimensional Continuous Control Using Generalized Advantage Estimation." ICLR, 2016.

4. Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, et al. "Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?" arXiv:2011.09533, 2020.

5. Chao Yu, Akash Velu, Eugene Vinitsky, et al. "The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games." NeurIPS, 2022.

6. Ryan Lowe, Yi Wu, Aviv Tamar, et al. "Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments." NeurIPS, 2017.