November 1, 2026

Do LLM Agents Still Need to Be
Told How to Self-Improve?

Shobhita Sundaram

MIT

Cansu Sancaktar

University of Tübingen

Phillip Isola

MIT

Paper Code Data Explorer

Imagine giving an agent a computer, and asking it to improve its ability to play puzzle games. What will it do? Let's say the computer has no pre-installed games; the agent must write its own games if it wants to practice. It must decide what sorts of games to try, which lessons to retain, and how to measure its progress.

These decisions on the design of learning procedures are still usually made by human researchers, even as individual components are increasingly automated by agents. In our new paper, we ask:

Do capable models still need us to specify how they should self-improve?

We study the natural behavior of LLM agents when they are given primitive tools for self-improving via context manipulation, but no explicit recipe for using them.

Teacher-Player setup

Our testbed setting is puzzle games: A player agent plays a turn-based game with limited knowledge of the mechanics and objective. To do well it has to explore efficiently and plan ahead. The player is a coding agent that improves through writing skill files and manipulating its context.

We put a teacher agent into a controlled, sandboxed harness session and give it three things:

  1. Tools: Read/Write (file-editing), Bash (running terminal programs), and Gameplay. The Gameplay tool spins up a fresh player session given a teacher-written game source file and player context, and returns the score and transcript.
  2. Budget: An update budget for revising player text files, and a gameplay budget.
  3. Goal: A main goal to improve the player by optimizing its text, and a generalizability goal that the text should transfer to unseen games.

Then, we let the teacher session run until one of the budgets is exhausted. To evaluate, we measure how well the optimized prompt/skill-files transfer to external, human-written game benchmarks.

Formal Setup

Let \(\pi\) be an LLM with frozen weights \(\theta\). The player \(\pi_{player}\) has context \(c = (c_P, c_L)\), where \(c_P\) is the player prompt and \(c_L\) is auxiliary text files (e.g., skill files). The teacher \(\pi_{teacher}\) is another instance of the same LLM that modifies \(c\). An update is a single edit tool-call to one file in \(c\).

The teacher's "algorithm" is just to act in its sandbox until the budget runs out:

Algorithm 1 Self-organized learning
Input: initial player context \(c_0\), instructions \(I\), budget \(B\)
  1. \(S \gets \text{InitSandbox}(c_0, I, B)\)
  2. \(\mathcal{A}_T \gets \{\text{Read}, \text{Write}, \text{Bash}, \text{Gameplay}\}\)
  3. while \(B > 0\) do
  4. \(a \sim \pi_{teacher}(\cdot \mid S), \; a \in \mathcal{A}_T\) ▷ teacher picks its next action
  5. \(S \gets \text{Execute}(S, a)\) ▷ may consume budget
  6. end while
  7. return \(c_{final}\)

We evaluate a context \(c\) by the player's performance on a held-out test set of games, where each game is played in a fresh player session at a fixed horizon and token budget:

\[ J(c) = \frac{1}{|\mathcal{D}_{test}|} \sum_{g \in \mathcal{D}_{test}} \mathbb{E}_{\pi_{player}(\cdot \mid h_t, c)} \left[ \sum_{t=1}^{T} r_t \right] \]

and measure self-improvement as \(J(c_{final}) - J(c_0) > 0\).


Motivating observation: Comparison to GEPA

Using Claude Opus-5, self-organized learning in a data-free setting achieves comparable performance to a prescribed loop (GEPA) that hill-climbs on 20 real, in-distribution games. This tells us that a strong model composing its own games and learning procedure can roughly match the performance of a text optimizer that uses real data.

Self-organized learning versus GEPA

Self-improving agents use recognizable learning strategies

Across three families of models (Claude, GPT, and GLM), recently released model generations self-improve, while older generations do not.

Newer models that self-improve also demonstrate human-recognizable learning strategies. They align more strongly to behaviors in Zimmerman's cyclical phases model for self-regulated learning, and spend most of their gameplay effort on low-to-medium difficulty games. See the paper for more analysis of teacher behaviors!

Learning strategies of Claude models

Explore teacher sessions

You can explore teacher sessions below. Some interesting moments:

Self-optimized text generalizes

Using Claude Opus-5, we show that the same artifacts generalize across game benchmarks that capture different game horizons, engines, mechanics, and state representations (2D v. 3D). In the paper we also evaluate transfer across models, and how the teacher's initial information affects generalization.

Improvements transfer across benchmarks

Implications

These results reflect a basic trend in deep learning: As models become more capable, they require fewer human inductive biases and less surrounding scaffolding. Put simply, stronger models require weaker harnesses. Here, that scaffolding is the self-improvement loop itself.

The main implications are in two directions:

Many interesting open questions remain:


Cite this work

@misc{sundaram2026selfimprove,
    title={Do LLM Agents Still Need to Be Told How to Self-Improve?},
    author={Shobhita Sundaram and Cansu Sancaktar and Phillip Isola},
    year={2026},
}