LLM::Agent

The engine behind a coding agent: a streaming loop that calls tools through a duck-typed provider, survives backend failure with per-round-trip retry and fallback, writes a durable JSONL transcript you can resume from tomorrow, and compacts the conversation before it outgrows the context window.

It is the machinery, not the product. There is no permission model here (that is MCP::Client::Policy), no UI, no scheduler, and no opinion about what your agent is for.

Synopsis

use LLM::Agent;

my $loop = LLM::Agent::Loop.new(backends => [$primary, $fallback]);
my $run  = $loop.run([$system-message, $user-message]);

# One Supply, every kind of event.
start react whenever $run.events -> $event {
    given $event {
        when LLM::Agent::Event::Token    { print $event.text }
        when LLM::Agent::Event::ToolCall { note "  -> {$event.name}" }
    }
}

# The Promise is KEPT even when the run failed: a failure is data.
my %outcome = await $run.result;
given %outcome<outcome> {
    when 'completed' { say %outcome<final> }
    when 'failed'    { note "gave up: {%outcome<error>}" }
    when 'cancelled' { note 'cancelled' }
}

The modules

use LLM::Agent; loads all seven and is enough for every class. It does not re-export LLM::Agent::Prompt's subs — an is export reaches one scope, not two — so call those fully qualified (LLM::Agent::Prompt::assemble(...)) or use LLM::Agent::Prompt; as well.

A round, step by step

• Compaction check, at the top. Before spending a request, not after: a round that ends with six tool results is exactly the round that blew the budget, so checking on the way in means the next request is the one that fits.

• The round trip. AttemptStarted, a streamed completion, Token events as the text arrives. Retry and fallback happen here, per round trip.

• Limits, before anything is committed. A limit emits LimitReached, appends a system message in LLM::Chat::ToolLoop's exact wording, switches tools off, and gives the model one last round to answer with what it has.

• The assistant turn is committed. AssistantMessage, appended to the conversation and written to the transcript with reasoning and usage as replay-visible extras.

• Tools. One ToolCall per call, one batch through the provider, one ToolResult per result, one role => 'tool' message per result.

• Grants. If the provider can('grants') — a policy does — a changed snapshot is written to the transcript, so a resumed session does not ask the human the same question again.

No tool calls, or tools switched off: RunCompleted, and the run ends.

Failure, retry and fallback

Every failure is classified by LLM::Chat::Retry's classify-error — the same policy LLM::Data::Inference::Task has run in production — into one of three buckets:

max-retries is the number of attempts per backend (Task's semantics, not "extra tries"): 3 means one call and two retries before the chain advances. When the budget runs out, the AttemptFailed says advance, because disposition reports what the loop does rather than what the classifier said in the abstract. When every backend is spent, RunFailed carries the full attempts list.

The timeout is inactivity, not duration

round-trip-timeout (default 120s) measures the gap since the response last did anything — $resp.last-activity-at — not the total time the request has taken. This is a deliberate divergence from LLM::Data::Inference::Task, which bounds total duration: Task generates one JSON document per item and a slow one is a stuck one, while an agent turn legitimately runs for minutes. What is never legitimate is a stream that stops producing tokens and never closes, which is what a dropped connection behind a proxy looks like.

Mid-stream failure: the framing contract

A backend can fail after streaming four hundred tokens. Those tokens were emitted; a Supply has no undo. So the framing events are the contract:

A consumer that ignores this renders a doubled reply the first time a backend 500s halfway through a sentence. Nothing a failed attempt streamed ever reaches the transcript: only committed messages are written.

Cancellation

$run.cancel is idempotent, safe from any thread, and never throws. It asks; the loop winds down and emits RunCancelled.

The two "nothing happens" rows are honest limitations rather than bugs waiting to be fixed. A tool batch has no cancellation to forward, and an fs_write already in flight has already happened; the loop drains the result and emits no ToolResult for it — but it does write one synthetic tool message per abandoned call, because an assistant turn carrying tool_calls that nothing answers is a conversation every provider rejects, and the transcript has to stay resumable. An ask is a leaf: the policy holds a non-reentrant lock and is waiting for a human, and a run cannot dismiss a modal something else owns.

Note also that "the backend was told to cancel" is not "the model stopped generating". Among the LLM::Chat backends only KoboldCpp really aborts upstream; the others stop reading, and the tokens you are no longer being shown are still being billed.

The transcript

One append-only JSONL file, held open on a single flushed handle-mode JSONL::Writer, so every line is durable the moment its method returns.

{"id":"6f1c...","payload":{...},"ts":"2026-08-09T13:10:08.542283Z","type":"message","v":1}

Four types: session-meta (always first, the caller's own hash verbatim), message, grants (the whole snapshot, last one wins) and compaction. Unknown types are preserved on replay and skipped by the readers, so a file written by a newer LLM::Agent still loads.

my $session = LLM::Agent::Session.create(
    path => $path, meta => { agent => 'sadna', cwd => $*CWD.Str },
);

# ... later, in another process ...
my $resumed  = LLM::Agent::Session.load(path => $path);
my @messages = $resumed.messages;     # compaction already applied
my @grants   = $resumed.grants;       # feed straight to a Policy

Replay tolerates a malformed line only as the very last line — which is exactly what a crash mid-write looks like; it is dropped and reported in .warnings. A malformed line anywhere else is fatal, because silently skipping it would hand back a conversation with an invisible hole in it.

What is deliberately not stored: token deltas, attempt telemetry, the contents of permission questions (only the resulting grants), forwarded server logs, and any opinion about where the file should live.

Compaction

my $counter   = LLM::Agent::TokenCount::Usage.new;    # shared with the loop
my $compactor = LLM::Agent::Compactor.new(
    backend        => @backends[0],   # a cheap model is a fine summarizer
    counter        => $counter,
    context-budget => 128_000,
);

The shape is fixed: the sticky prefix (sysprompt and anything else Message.is-sticky agrees with) stays where it is, the last keep-recent turns stay verbatim, and everything between is replaced by one summary message. The recent window is extended backwards while it starts on a tool result, so an assistant turn and the results answering it are never separated — splitting them is a 400 from every provider.

When the summarizer fails, the failure is classified: an abort bucket hard-trims immediately, anything else is retried up to max-attempts (3) with backoff, and a still-unreachable summarizer falls back to a pair-aligned hard trim with fallback => True . That fallback is the whole point of the design — the loop always makes progress, because an agent that stops working when a summarizer is down is worse than an agent that forgot what happened an hour ago.

LLM::Agent::Session replays exactly the transformation the compactor applied, so $session.messages after a resume equals the array the loop was working with when it stopped. t/11 pins that end to end.

Counting tokens

Usage is the default: it remembers the prompt-token count the provider actually billed for the prefix it has already seen and estimates only the tail beyond it. The loop owns one instance and hands the same one to its compactor, so the calibration a run accumulates is not thrown away at the moment it matters most.

The canonical wiring

The whole stack, in the order the pieces have to be built. Note the forward declarations: the MCP client needs the policy's elicit hook at construction, the policy needs the loop's ask shim, and the loop needs the policy — so two of the three have to be named before they exist.

use LLM::Agent;
use MCP::Client;
use MCP::Client::Registry;
use MCP::Client::Policy;

my $loop;                       # forward-declared: the shims come from it
my $policy;                     # forward-declared: the client's hook needs it

my $session = LLM::Agent::Session.create(
    path => $path, meta => { agent => 'myagent', cwd => $*CWD.Str },
);

my $fs = MCP::Client.connect-stdio(
    command   => 'raku-mcp',
    args      => ['--pack=FileSystem', '--root=' ~ $*CWD],
    on-elicit => -> %request { $policy.elicit-hook.(%request) },

    # Server logs become Log events on the run's Supply. THE log-level IS
    # LOAD-BEARING: since the 2026-07-28 revision a modern server sends
    # nothing at all without one, and a silent hook looks exactly like a
    # server that had nothing to say.
    on-log    => -> %params { $loop.log-hook.(%params) },
    log-level => 'info',
);

my $registry = MCP::Client::Registry.new;
$registry.add($fs, prefix => 'fs');

$policy = MCP::Client::Policy.new(
    provider => $registry,
    rules    => [
        |MCP::Client::Policy.default-rules,
        { tool => 'fs_write', decision => 'allow', under => 'scratch' },
    ],
    roots    => { fs => $*CWD.Str },
    grants   => $session.grants,          # last session's "always allow"s
    on-ask   => -> %request { $loop.wrap-ask(&ask-the-human).(%request) },
);

my $counter   = LLM::Agent::TokenCount::Usage.new;
my $compactor = LLM::Agent::Compactor.new(
    backend => @backends[0], :$counter, context-budget => 128_000,
);

$loop = LLM::Agent::Loop.new(
    :@backends, :$counter, :$session, :$compactor, provider => $policy,
);

my $system = LLM::Agent::Prompt::assemble(
    identity => 'You are a coding assistant working in a checked-out repository.',
    sections => [
        LLM::Agent::Prompt::env-block(extra => { cwd => $*CWD.Str }),
        LLM::Agent::Prompt::tool-docs($policy.tools-for-llm),
        LLM::Agent::Prompt::instructions-from-files(['AGENTS.md']),
    ],
);

my $question = LLM::Chat::Conversation::Message.new(
    role => 'user', content => 'Make the tests pass.',
);
$session.append-message($question);      # the app owns the turns it originates

my $run = $loop.run([$system, $question]);

wrap-ask emits AskPending, calls the real asker (which still blocks, and still holds the policy's lock), emits AskAnswered, and returns the answer untouched. What does not work is on-ask => $loop.wrap-ask(&ask) with $loop still undefined: that calls a method on a type object at construction time and dies there.

The event taxonomy

A ? marks a key that is absent from .to-hash when it was not supplied. Exactly one terminal event is emitted per run and the Supply is then done; it is never quit, because a failure is data, not an exception thrown at whoever happened to be tapping.

The Supply is a Supplier::Preserving, so a consumer that taps late still sees the whole run from the beginning — but the buffer is delivered to the first tap only. Tap once and fan out yourself.

Requirements

LLM::Chat 0.8.0+ (backends, messages, and the shared retry policy), MCP::Client 0.2.0+ (only for the tool-provider duck type and the integration tests), JSONL 0.1.3+ (the flushed writer the transcript depends on), JSON::Fast and UUID::V4.

Testing

prove6 -Ilib -It/lib -I../LLM-Chat/lib -I../MCP-Client/lib \
       -I../JSONL/lib -I../Template-Jinja2/lib t/

The suite needs no network, no model and no API key: t/lib/AgentTestKit provides a ScriptedBackend that replays a list of steps — including mid-stream failures, stalls, per-attempt usage and structured error classes — and a duck-typed ScriptedProvider that never throws. t/11 runs the loop over a real MCP::Client::Policy, a real transcript on a temporary file and a real compactor, and checks that a resumed session replays exactly the conversation the loop ended with.

Author

Matt Doughty

License

Artistic-2.0