Revision history for LLM::Chat
0.10.3 2026-10-25T13:25:17Z
- [2026-10-25] A fresh `zef install` failed in t/30 because the
child process it starts could not see the dependencies zef had
staged but not yet installed: it died with "Could not find
Digest::SHA256::Native". zef tests a distribution from a staging
repository and installs nothing until every test has passed, and
the test handed its child only the FileSystem module
repositories. It passed wherever the dependencies were already
installed, so installing LLM::Chat failed only on a machine that
did not have them yet. t/29, t/30 and t/35 now hand each child
the test's whole module search path, every repository by its
path-spec, the staging repository included. The library itself
was unaffected.
0.10.2 2026-10-24T18:26:08+01:00
- [2026-10-24] Installs on Rakudo 2026.07 work again. The test suite
failed there, so installing 0.10.1 on 2026.07 stopped at the test
step: one test pinned the exception class behind Rakudo's message
stub, which is X::AdHoc before 2026.08 and X::StubCode since. It
now accepts either; the library itself was unaffected.
- [2026-10-24] A finished Response could briefly look failed.
`.is-done` turned True one step before `.is-success` (and, on a
failure, before `.is-cancelled` and `.err`), so a consumer polling
`.is-done` from another thread could see a completed generation
as one that had not succeeded. The terminal state is now published
all at once: when `.is-done` is True, `.is-success`,
`.is-cancelled`, `.err` and `.msg` are already final.
0.10.1 2026-10-24T17:46:46+01:00
- [2026-10-24] The HTTP-level tests run against `raw-server`, the
fixture server in LLM::Chat::Test::Kit: an HTTP/1.1 server on
IO::Socket::Async that answers each request with what a responder
returns (a body, chunks, a Supply of chunks, a delay) and
finishes writing every response before it closes anything. Cro's
own server can lose the end of a response on its Connection:
close path when the machine is busy, which would fail the suite
intermittently on a loaded machine. Installing no longer pulls
in Cro::HTTP::Server or Cro::HTTP::Router for the tests.
- [2026-10-24] Requests no longer leave their connection open. Each
one used a fresh Cro client, persistent by default, so its
connection stayed cached in a client that was then thrown away:
a socket left open per request for the life of the process,
which on a 256-descriptor limit (the macOS default) exhausted
file descriptors over a long session or a busy batch. Every
client the backends make (completions, streams, server probes,
KoboldCpp's abort and length check, OpenRouter's generation
lookup) is now non-persistent: it asks the server to close the
connection once the response is done. The test suite also
passes, with nothing but TAP in its output, under that limit.
- [2026-10-24] A transport, hook or tool error whose exception has
no message no longer kills the process. Cro raises one when a
server closes the connection before sending response headers
(Cro::HTTP::Client::PipelineClosedBeforeHeaders). Such an
exception's message is the base Exception's stub, which throws
as soon as it is used, and the error handlers interpolated it
into their log lines — so each threw again from inside its
CATCH, on the request's worker thread, where that is "Unhandled
exception in code scheduled on thread N" and the program exits.
Such a request now fails like any other, with the exception's
class name as its error text; error-class 'unknown'. The side
calls are covered too: a `_reconcile-finish-reason` hook or
OpenRouter's generation lookup that dies that way is logged and
shrugged off like any other failure of theirs, where it used to
fail the completion with "Stub code executed"; so is a
Response's aborter, where `cancel` used to throw at its caller;
the server probes (`probe-server-limits`, KoboldCpp's length
check) end their `note` with "(no message)" rather than nothing;
and ToolLoop quits with the class name when a backend call dies
that way, and hands it back as the tool result's content when
`execute-tools` does. The new LLM::Chat::ExceptionText exports
the two subs behind the fix, `exception-message` and
`exception-text`, which read any exception's message without
ever throwing.
- [2026-09-28] Requires JSON::Fast 0.21.0 or later (zef:timo). 0.19
(cpan:TIMOTIMO) fails its own tests under the RakuAST compiler
frontend that Rakudo 2026.09 makes the default, so installing this
distribution stopped at that dependency.
- [2026-09-25] Thinking, per request. `reasoning_effort` now takes
'none' and 'minimal' beside 'low', 'medium' and 'high', and
OpenAICommon sends it as the top-level OpenAI field (it used to
reach OpenRouter only): llama.cpp's server hands it to the chat
template, where 'none' turns thinking off and a Mistral Small 4
template takes 'high'. A new `reasoning_budget_tokens` setting
caps the thinking per request (llama.cpp's
`reasoning_budget_tokens`, which overrides `--reasoning-budget`
and forces the end-of-thinking tag once spent; KoboldCpp gets it
as `thinking_budget_tokens`). Both go out only when set.
OpenRouter keeps its `reasoning: {effort}` object and drops the
top-level fields.
- [2026-09-25] Thinking is captured, never mixed in. A blocking
reply's `reasoning_content` (llama.cpp, KoboldCpp) now lands on
`reasoning-text` as `reasoning` (OpenRouter) and the streamed
deltas already did; the content stays the answer alone.
`usage.completion_tokens_details.reasoning_tokens` is lifted onto
a new `reasoning-tokens` (a subset of `completion-tokens`,
undefined when unreported).
- [2026-09-25] `id_slot`: a new setting that pins a request to one
llama.cpp server slot, so requests sharing a long prefix reuse
that slot's KV cache dependably. OpenAICommon sends it only when
set; KoboldCpp and OpenRouter never send it.
- [2026-09-25] `probe-server-limits`: ask the server for its context
size. OpenAICommon reads llama.cpp's `GET /props`
(`default_generation_settings.n_ctx`, plus `total_slots`, or the
length of `GET /slots` when `/props` has no slot count);
KoboldCpp reads `GET /api/extra/true_max_context_length`;
OpenRouter and the base class ask nothing. The answer always has
`source` and either `context_length` (and `slots` with its
`slots_source`, or a `slots_note`) or a `note`; it never throws
and each request is bounded by the new `server-probe-timeout`
(3 s). New shared internals for subclasses: `_server-root`,
`_probe-json` and `_probe-json-array`.
- [2026-09-25] Mock: `fake-reasoning` (a thinking trace on every
response), `server-limits` (what `probe-server-limits` answers)
and a `reasoning` usage key.
- [2026-09-23] Streams keep their usage when they fail. Both
streaming paths quit the Response the moment they read a
`length`, `content_filter` or unknown finish reason — and the
text path settled on `stop` the same way — but the usage frame
that stream_options.include_usage asks for comes AFTER the
finish chunk, so a failed stream, and every text stream, lost
its token counts. A finish reason is now recorded where it
arrives and acted on where the stream settles (`[DONE]`, the
body closing, or a new `stream-finish-grace` timer); later
chunks are read for usage and model only, never for content,
reasoning or tool-call deltas. The outcomes are unchanged — the
same success/failure split, error class and error text ("Hit
max tokens", ...) — except that a failure now carries its usage,
and `[DONE]` after a failure reason still fails. A server that
sends its finish chunk and then neither `[DONE]` nor a close is
settled after `stream-finish-grace` seconds (default 10; Inf
waits for the body) and its connection dropped, rather than
holding the Response for the body's own, unbounded, timeout. A
cancel in that gap stays a plain cancel. The blocking text path
now lifts usage and records the finish reason too; it read the
text and nothing else.
- [2026-09-23] New `_reconcile-finish-reason` hook on OpenAICommon:
it sees every finish reason once, on all four paths (on a stream
after the usage frame), and returns the reason the Response acts
on. The Response now has `provider-finish-reason` (the wire
value, verbatim) beside `finish-reason` (the decided one), and
`finish-reason-evidence` (a detached Hash of how a backend
decided, when it checked).
- [2026-09-23] KoboldCpp: truthful stop reasons. KoboldCpp reports
every stop that is not an EOS token as `length` (koboldcpp.py:
`"length" if (sr!=1) else "stop"`), so a generation that ended
on a custom stop sequence was indistinguishable from a truncated
one and failed the stream — commonly after a tool result, with
output "" or "\n". On a `length` finish the backend now asks
`GET /api/extra/perf` and finishes it as `stop` when
`stop_reason` is 2 (CUSTOM_STOPPER in KoboldCpp's expose.h) and
`last_token_count` equals the usage `completion_tokens` — or,
when there is no usage (KoboldCpp's tool-call streams never send
it), is at most 90% of max_tokens. Anything else, including a
perf request that fails or outlasts
`verify-length-finish-timeout` (default 2s), keeps `length`. The
wire value stays on `provider-finish-reason` and the numbers on
`finish-reason-evidence`. Off with `verify-length-finish =>
False`.
- [2026-09-23] Opt-in message normalization for strict chat
templates: `message-normalization => 'alternate-roles'` on every
OpenAI-compatible backend (default 'none'). Mistral-family
templates, which llama.cpp and KoboldCpp --jinja run as-is,
reject a conversation whose roles do not alternate ("After the
optional system message, conversation roles must alternate user
and assistant roles except for tool calls and results").
Adjacent plain messages of one role merge, joined by "\n\n";
the leading system messages become one; a later system message
becomes user text under a `[System message]` marker line
(`folded-system-marker`) and merges into the user turn beside
it, or stands as its own user turn. Tool calls and tool results
are never merged or crossed. Those templates skip tool turns
before checking, so a user message directly after tool results
reads as the user speaking twice: it gets one assistant turn in
front of it, whose text is `tool-result-acknowledgement`
(default 'Understood.'; '' inserts none). Nothing is reordered,
only empty system messages are dropped. The result is a pure
function of the message list and prefix-stable — appending
messages only ever grows the last normalized message, and a
user message after tool results adds the acknowledgement and
itself after them — so repeated prefixes stay byte-identical
for prompt caches. Applies to both chat paths (inside
`wire-message-hashes`, after `system-message-mode`) and to the
messages both text paths hand their template (new
`normalized-messages`).
- [2026-09-23] The text paths no longer hand their template the
caller's own message array. ChatML, Gemma2, Llama3, Llama4 and
MistralV7 `pop` the last message when rendering a continuation,
so `text-completion(@messages, True)` removed the caller's last
message — from a worker thread, while the request was in flight.
- [2026-09-23] Request bodies are now serialized deterministically,
and a JSON Schema's property order is the caller's to choose.
Every completion path handed Cro the body as a Hash, and Cro
wrote its members in whatever order the Hash iterated — which
differs between two hashes holding the same keys and between
two runs of the same program. That included the schema of a
structured-output request. llama.cpp and KoboldCpp compile a
`response_format` / `json_schema` schema into a grammar that
makes the model write an object's fields in the order the
schema lists them, so the model wrote its fields in a random
order per request, and an application could not ask for, say,
the evidence before the conclusion it supports. Random bytes
also defeated provider prompt-prefix caching and made a logged
request impossible to replay exactly.
All four paths (chat and text, blocking and streaming) on
OpenAICommon, OpenRouter and KoboldCpp now post bytes from the
new overridable `_encode-request-body`, with
`Content-Type: application/json` as before. The rule: arrays
keep their order; every object's members are sorted by code
point; a schema's `properties` map follows the sibling
`required` array (in array order), then the optional
properties sorted — at every depth, including nested objects,
`items`, `$defs`, `anyOf` / `oneOf` / `allOf` branches, tool
`parameters` and KoboldCpp's native `json_schema`. So the order
of `required` is the order the model writes the fields. Names
inside property and `$defs`-style maps are never read as
keywords, and data is never reordered as schema: the values of
`const` / `default` / `enum` / `examples`, and the whole
`messages` member, are plain sorted data. The same Hash is now
the same bytes on every call and in every process, and parses
to the same data as before.
The serializer is exported as `LLM::Chat::WireJSON`
(`to-wire-json`, with `:data-members` and `:!schema-order`) for
applications that build prompt text or schemas themselves.
Numbers, booleans, enums and other non-string scalars are still
written by JSON::Fast; strings are escaped in a single linear
pass. Two wire defects of the old path go with it: JSON::Fast
escapes by splicing, which is quadratic in the number of
characters needing an escape (a 1 MB transcript full of
newlines took seconds to serialize), and a combining mark right
after an escape sequence normalized into the escape's letter —
a newline followed by a combining acute went out as `\ń`, an
invalid escape the server rejects. Such a mark is now written
as a `\u` escape.
With LLM_CHAT_DEBUG set, the chat paths now log REQUEST BODY as
the exact compact bytes posted rather than a pretty-printed
re-rendering (new `LLM::Chat::Debug.log-bytes`, which decodes
only when logging is enabled).
Two adjacent Hash-order defects: a streamed tool call whose
`arguments` arrived as an object is now stored as sorted JSON,
so the string echoed back into the next request is stable; and
`ToolLoop`'s identical-call limit now recognises two calls with
equal object arguments built in different orders as identical.
- [2026-09-15] Add opt-in OpenAICommon request-stream-usage for
stream_options.include_usage, preserving other stream options and
blocking request bodies. Both chat and text streaming now retain
usage-only terminal chunks in Response token counters.
- [2026-09-14] OpenAI-compatible HTTP failures now preserve bounded
provider error evidence on the Response before the generation is
quit. Both 4xx and 5xx Cro HTTP errors read at most 16 KiB of the
response body, never headers or request content, and expose
`Response.provider-error` as detached data with status, retained
byte counts, truncation state, content type, bounded text, and
OpenAI-style `error.message` / `type` / `code` when present. The
Response error string is augmented only with those safe structured
error fields, so existing attempt-failure diagnostics can show a
provider reason such as "System message must be at the beginning"
without enabling plaintext debug logs.
`OpenAICommon` also gains `system-message-mode`, default
`preserve`. Set it to `leading` for OpenAI-compatible servers that
reject later system messages: `wire-message-hashes` merges all
system-message content, in original relative order, into one
leading system message separated by a documented stable delimiter
while preserving non-system message order and tool-call fields.
- [2026-09-02] Cancelling a generation on an OpenAI-compatible
backend now actually stops it. `cancel` only ever quit the local
Supplier: the consumer stopped seeing tokens while the backend's
react loop went on draining the body to the end, so the server —
llama.cpp's, OpenRouter, any openai-compat gateway — generated
the whole reply for a request that had been abandoned, and billed
for it. A caller that cancels freely (a chat UI where every
re-roll supersedes the last request) accumulated abandoned
generations on exactly the slow, expensive models where it hurt.
The cancel was cosmetic; only the consumer stopped looking.
There is no abort message in the OAI protocol — a client that no
longer wants a completion hangs up — so all four completion paths
now hand the Response the means to hang up: `Response.cancel`
quits the local supply first, synchronously and unconditionally,
and then drops the HTTP connection via `Cro::HTTP::Response.cancel`,
which quits the body stream and closes the socket rather than
returning a quietly-drained connection to Cro's pool. That is the
same mechanism Cro uses for its own body / total request
timeouts, so a deadline and a cancel now terminate the upstream
generation the same way. Transport teardown is best-effort and
swallows its own failures — a dead endpoint must never make a
cancel throw on the thread that asked for it.
Two edges came with it. A cancel raised while the server is still
prefilling arrives BEFORE there is a response to close, and Cro's
client exposes no handle on a request in flight; the Response
records the request and aborts the moment the transport exists,
so the abort lands on the first response byte instead of being
lost. That window is the common case, not the corner one. And a
cancelled Response is no longer classified: Cro reports the
torn-down body as a `body`-phase timeout, indistinguishable from
a genuinely stalled one, and filing it as `error-class 'timeout'`
would have told a fallback policy to retry the request the user
just abandoned. `.error-class` now stays undefined on a cancel.
New on `Response`: `_attach-aborter` (backends register transport
teardown) and `cancel-requested`. The aborter runs at most once
and is dropped when the Response reaches any terminal state, so a
cancel that loses the race to `[DONE]` is a genuine no-op and a
completed Response keeps `.is-success`.
`KoboldCpp.cancel` is unchanged and still POSTs
`/api/extra/abort`; it now rides along on the connection close
too. The two cover different windows — the POST stops a
generation still in prefill, the close is instant once a body
exists — so both stay.
- MAINT: [2026-08-27] moved the shared HTTP-fixture test support from
C into C
(C). A C directory is its own
CompUnit repo that starts cold on a fresh C, and
parallel test files pulling the fixture's dependency chain into
that cold store can race and fail the install outright. C
is staged and precompiled before the test phase runs, so
shipping test support inside the dist's own namespace dissolves
the race. This is the fleet-wide convention going forward.
0.10.0 2026-08-21T22:05:38+01:00
- [2026-08-21] Require Template::Jinja2 0.3.0. The byte-identical
rendering of upstream GLM and Kimi chat templates leans on its Python
method-call dispatch and Jinja2-faithful filters; an older renderer
would produce silently different prompts.
- [2026-08-20] The OpenRouter backend marks `cache_control`
breakpoints on outgoing chat bodies, so providers that only
cache a prompt prefix when the request says where it ends
(Anthropic, Qwen, Gemini) can actually cache one. Markers go on
the head message and on the last two `user` / `assistant` turns
— the trailing pair is what makes the marker written one round
ago still cover what the next request shares with it — with
tool results and tool-calling assistants stepped over, and
never more than OpenRouter's ceiling of four. Implicit-caching
routes ignore the annotation, so the new `cache-breakpoints`
attribute defaults to On across a mixed fleet; set it to False
to send the plain string shape. Conversation state is untouched
either way: the rewrite happens on the serialized body, so
`Message.to-hash` and `Message.get-checksum` read the same
after a request as before it.
Placing them needed a seam, so `OpenAICommon` gained one:
`_finalize-request-body` is called on all four completion paths
with the assembled body, after messages / prompt / tools /
stream are attached and before the POST. The default returns
its argument, so nothing changes for backends that don't
override it.
- [2026-08-20] New `Response.cached-prompt-tokens`: the slice of
the prompt a provider served out of its own cache rather than
processing fresh, which is what tells a caller whether the
input it is paying full price for is actually being reused.
Lifted from the OAI-spec `usage.prompt_tokens_details.
cached_tokens` on any OpenAI-compatible backend, and on
OpenRouter also from the `/generation` poll — the only place
the streaming path can pick it up, since that backend
deliberately asks for no inline usage frame. A value the
response body reported always wins over the poll's. Undefined
means the provider said nothing about caching, which is not the
same as nothing having been cached, so a reported zero stays a
reported zero.
- [2026-08-19] A streamed response now always reaches a terminal
state. Both streaming paths route every ending through one
place, so `[DONE]`, a `stop` / `tool_calls` finish reason
followed by the body closing, and a body that closes with
neither all settle the Response — the last of those as a
failure with error-class `response`, since a reply cut off in
transit cannot be told from a reply that ended. Previously that
third case left the Response neither done nor quit: a consumer
polling `.is-done` waited out its own timeout, minutes at a
time, for a stream that had been over from the start. Providers
that close on the finish_reason chunk without ever sending
`[DONE]` are common, and they now complete promptly and
successfully instead. The `length` / `content_filter` /
unknown-reason failures are unchanged.
Fixes a hook-ordering bug on the way: text-completion-stream's
`[DONE]` arm completed the supply BEFORE `_on-stream-complete`,
so a subclass's post-stream metadata landed after consumers had
been told the response was finished. Both paths now fire the
hook first, as the hook has always documented.
- [2026-08-19] New `Response.dropped-frames`: streamed `data:`
lines that fail to parse are still skipped — providers
interleave junk into healthy streams — but they are now counted,
and a stream that dropped frames while assembling tool calls
FAILS instead of succeeding. A hole in prose costs a few words
and is visible to whoever reads the reply; a hole in a tool
call's `arguments` is spliced out of a JSON string that is then
concatenated blind, and what comes out can be valid JSON saying
something the model never asked for. Nothing downstream can
detect that, so it no longer reaches `is-success`. Prose-only
streams keep their success and expose the counter. SSE
heartbeats and empty `data:` lines are not counted.
- [2026-08-19] Streaming tool-call deltas that carry no `index`
are assembled correctly. The OAI spec puts an `index` on every
fragment and it is still authoritative when present, but some
providers omit it, and the old fallback used the number of calls
seen so far — which sent every argument fragment to a fresh
call. One tool call arrived shattered into N, each holding an
unparseable slice of the JSON. The fallback now reads `id`,
which the spec sends only on a call's first fragment: an `id`
starts a new call, an arguments-only fragment continues the one
in flight.
- [2026-08-19] `LLM::Chat::Template::Jinja2.render` presents
tool-call `function.arguments` to the template as a decoded
object when they are a JSON object, matching what HuggingFace
chat templates are written against (transformers passes a dict).
GLM-family templates iterate `arguments.items()` and rendered
nothing at all against the JSON string the OpenAI wire format
uses — a silently truncated prompt, and for a token counter an
undercount of the largest part of it. The messages themselves
are untouched: the render works on a copy, so `to-hash` and
every request body built from a conversation still carry the
string. Arguments that are not a JSON object — empty, a fragment
cut short mid-stream, an array, a bare scalar, or a Hash a
caller supplied directly — are passed through exactly as they
are.
- [2026-08-13] `LLM::Chat::TokenCounter.get-request-count` counts a
complete selected-model request: optional runtime context head, stored
conversation, optional runtime context tail, and tool catalogue rendered
through the model's actual template, then tokenized without a second
special-token pass. This is the exact-counting seam for models such as
Kimi K2.7 whose tool declarations live in the chat template. Existing
`get-conversation-count` remains conversation-only and unchanged.
Kimi's published Jinja fallback serializes ordinary OpenAI tool
declarations with compact `tojson`; `tools_ts_str` is an optional
preformatted override, so no model-specific duplicate renderer is used.
Runtime context may be supplied as its natural string halves or as
prebuilt system messages. Rendered prompts opt into declared special
tokens explicitly, which is required by deny-by-default tiktoken
backends and harmless for Hugging Face JSON tokenizers.
- [2026-08-13] Add `LLM::Chat::Template::DeepSeekV4`, a byte-compatible
Raku port of the full prompt-encoding half of DeepSeek's pinned DSV4
reference implementation. It supports chat/thinking modes, reasoning
effort, context and thinking removal, developer/latest-reminder turns,
quick tasks, response formats, OpenAI-format tools/calls, DSML argument
encoding, and merged/call-ordered tool results. `encode-json` preserves
JSON object order for differential goldens; `encode-messages` accepts
ordinary associative message structures; `render` adapts existing
`Conversation::Message` objects. Completion parsing is intentionally not
included. The four official Python outputs are pinned as offline golden
tests with DeepSeek's MIT attribution.
- [2026-08-12] A Cro timeout is now classified BY THE PHASE it
expired in. `X::Cro::HTTP::Client::Timeout` carries a `phase` of
`connection`, `headers` or `body`, and OpenAICommon's classifier
used to file all three as error-class `timeout` — the ADVANCE
bucket. A connection-phase timeout is Cro's 30-second connect
budget running out: nothing was sent, the endpoint said nothing,
and the only evidence is a network that was unwell. That is now
error-class `connection`, the RETRY-SAME bucket. Headers- and
body-phase timeouts keep `timeout`: a backend that accepted the
connection and then would not answer has said something about
itself, and the next backend answers sooner than a backoff ends.
The lumping was fatal on a ONE-BACKEND chain, where `advance`
means the chain is exhausted: a single flaky connect ended the
run having made exactly one call. `classify-error`'s own bucket
table is unchanged — the split is at the source, which is where
the information is.
The classification test no longer re-implements the classifier
it is testing; it calls the real private method through
`^find_private_method`, so the two cannot drift.
- [2026-08-12] `LLM::Chat::Template::Jinja2.render` takes an
optional `:@tools` — the tool declarations the request will
carry, in the provider's own shape — and passes them to the
template as its `tools` variable. Real HuggingFace chat
templates render a tools block when they are given one, so a
render meant to match what the server sees needs them.
Passed ONLY when non-empty: `tools => []` and no tools at all
both render exactly as they did before the parameter existed
(pinned by a test, because a template asking
`{% if tools is defined %}` would otherwise take a different
branch for a caller who named nothing). Deliberately NOT wired
into `LLM::Chat::TokenCounter`: the declarations are a
per-request constant that provider-reported usage has already
been charged for, and counting them per message would bill them
twice.
0.8.1 2026-08-09T20:57:18+01:00
- META fix: Response::OpenRouter and Response::OpenRouter::Stream were
missing from provides and listed under DEPENDS instead. On a machine
without install history, zef resolved those module names to the fossil
LLM::Chat::Backend 0.2.1 dist, whose exact Digest::SHA256::Native 1.0.0
pin only builds with nmake — breaking every fresh Windows install
(caught by LLM-Agent's Windows CI lane). No code change; 0.8.0 as
published carries the broken META and should not be depended on.
0.8.0 2026-08-09T19:41:06+01:00
- [2026-08-09] New C and
C: the retry/fallback policy that
has run inside C since 0.5, factored
out so a second executor can share it instead of copying it.
C exports five pure subs — C
(the abort / retry-same / advance bucket table),
C (C, with the
anti-thundering-herd jitter suppressible via C<:jitter(0e0)> for
deterministic tests), C (chunked sleep that
consults a cancel hook before EVERY chunk including the first,
and returns a Bool rather than throwing, because only the caller
knows which exception type its layer promises),
C and C (which duck-types the
response with C<.?> and therefore does not import
C).
C provides
C and its C /
C subclasses plus C (deliberately NOT an
C, so a user cancel can never be swallowed by a
dead-letter handler), and is now the canonical home of the
C advice contract. Those four are plain global
classes rather than Ced ones on purpose: C
on a nested-name class also exports its LEAF name, so two
modules declaring an C cannot be
imported into the same scope at all — which would have made the
shared hierarchy unusable next to a subclassing one.
Deliberately NOT shared: the
retry loop itself, the abort bucket's C message, hook
payloads and hook shielding — those are shaped by the layer that
owns them. B: this release is purely
additive, every previous test file is untouched, and
C 0.9.0 migrates onto it (its exception
types become subclasses of these, so both hierarchies keep
matching).
- [2026-08-08] Fix: a streamed completion could be KILLED MID-
GENERATION by non-ASCII output. Both SSE loops in OpenAICommon
(C and C) did
C<$buffer ~= $data.decode('utf-8')> per body-byte-stream chunk,
but TCP splits a body wherever it likes — including through the
middle of a multi-byte UTF-8 sequence. C<.decode> on the half
that ends mid-sequence throws "Malformed termination of UTF-8
string", which landed in the streaming CATCH and quit the
Response with error class 'unknown', discarding everything
generated so far. Buffering text could not fix it: the throw
happened before the text reached the buffer. The bytes now go
through an incremental C (new shared internal
C<_stream-decoder>, C<:translate-nl(False)> so SSE framing still
sees the line endings that are actually on the wire), which
keeps an unfinished sequence to itself until the bytes
completing it arrive. Framing, heartbeat skipping, C<[DONE]>
handling, malformed-chunk tolerance and reasoning/content
accumulation are untouched. Emoji, CJK and curly quotes are
ordinary in model output, so this was an intermittent
production crash for anything but plain ASCII.
- [2026-08-08] Fix: non-ASCII in a response body could come back as
mojibake (café -> café). Cro's C returns the
LIST C<('utf-8', 'latin-1')> whenever the Content-Type names no
charset, and C loops that list
with no C — so the latin-1 attempt, which cannot fail
whatever the bytes are, always overwrote the correct utf-8
decode. Many OpenAI-compatible servers answer
C with no charset parameter, and
model output is full of non-ASCII (names, curly quotes, emoji,
CJK). OpenAICommon now decodes response bytes itself via new
internals C<_blob-text> / C<_body-text> / C<_decode-json-body>:
utf-8 first (JSON is UTF-8 by definition, RFC 8259 §8.1), with
latin-1 kept only as a fallback for when the utf-8 decode
actually throws — a proxy's non-JSON error page, say. Affected
the 4xx error-body read that feeds LLM::Chat::Debug, which is
where an operator reads a rejection reason.
- [2026-08-08] The blocking C / C
paths and OpenRouter's C metadata lookup now parse
their JSON from the raw bytes instead of C.
Cro's response body-parser selector reaches the correct parser
only while the server labels the body C; a
server that answers JSON as C fell through to
TextFallback and produced a mojibaked Str, which the completion
path then indexed as a Hash and SILENTLY emitted as empty
content. Well-behaved servers see no change. Transport failures
still propagate as transport exceptions, so error classification
is unchanged.
0.7.1 2026-07-27T15:27:39+01:00
- [2026-07-27] LLM::Chat::Backend::Mock gains
C<&.finish-reason-producer>, an optional
C<(Int $call-index --> Str)> callback mirroring
C<&.error-producer>'s per-call indexing. A defined Str return is
stamped on the Response via C<_set-finish-reason> before the
emission task starts, while the call still SUCCEEDS — the exact
shape of a real blocking completion that ran out of completion
budget (HTTP 200, well-formed but partial body, finish_reason
'length'). Undefined returns leave C unset, so
existing mocks are byte-for-byte unchanged. A scripted error
wins over a scripted finish reason (a failed call never produced
a body). Wired into the non-streaming paths only —
C, and C by delegation —
because the streaming paths already model finish reasons the way
the real transports do. Lets downstream consumers
(LLM::Data::Inference's truncation policy) test truncation
handling without a live model.
- [2026-07-27] Fix: OpenAICommon's C acted
on a terminal C without recording it, so a stream
cut off by the token budget quit with error class 'response' but
an UNDEFINED C<.finish-reason> — consumers could not tell budget
exhaustion apart from a content filter or an unknown reason.
It now calls C<_set-finish-reason> before dispatching on the
reason, matching C.
0.7.0 2026-07-21T03:09:53+01:00
- [2026-07-03] Structured outputs: Backend::Settings gains
C (a plain-Hash JSON Schema, default undefined) and
C. OpenAI-compatible backends (incl.
OpenRouter) send it as C — best-effort,
provider-dependent enforcement; KoboldCpp sends the native
C generate field, which it compiles to a grammar
for sampler-level enforcement. Unset schema changes no request
bytes. Callers should keep parse validation as the backstop for
providers that ignore response_format.
0.6.0 2026-06-22T23:33:11+01:00
- Add LLM::Chat::ToolLoop, a generic streaming tool-call
round loop. It sends OpenAI-format tools to any chat backend,
executes returned tool calls through a caller-supplied
callback, appends assistant/tool turns, and continues until a
final answer or guardrail limit is reached.
- OpenAI-compatible streaming now assembles `delta.tool_calls`
chunks and treats `finish_reason: tool_calls` as successful.
This enables KoboldCpp streamed tool calling.
- Conversation messages can now serialize assistant `tool_calls`
and tool-result `tool_call_id` fields for follow-up chat
completion requests.
- LLM::Chat::Backend::KoboldCpp.cancel is now non-blocking and
failure-proof: the local response stream closes FIRST,
synchronously, then the upstream POST /api/extra/abort fires on
a worker thread, best-effort, with network failures swallowed.
The old ordering awaited the abort before closing locally, so a
busy KoboldCpp froze the caller for the full HTTP round-trip
(interactive cancels hung the UI) and an unreachable one made
cancel throw without ever closing the stream. cancel now
returns the background Promise so callers/tests can await the
abort attempt. Also fixes the abort URL: stripping the /v1
suffix used to leave its leading slash behind, producing
//api/extra/abort.
0.5.1 2026-04-29T23:47:12+01:00
- Bump Github Actions to use node 24+
- LLM::Chat::Backend::OpenAICommon gains a symmetric
`_on-blocking-complete` hook on the non-streaming path,
mirroring the existing `_on-stream-complete` contract: fires
after the response body has been parsed and `_lift-usage`
has lifted OAI/provider usage fields, before
`$response.done`. Default implementation is a no-op.
Subclasses use it to attach post-call metadata that isn't
in the body itself — symmetric with what was already
possible on streams.
- LLM::Chat::Backend::OpenRouter wires the new hook through
to the same `/generation?id=...` lookup the streaming path
already uses, so blocking callers (e.g. App::Storygen, which
calls `chat-completion` rather than `chat-completion-stream`)
now see `.cost` populated by the time `$response.done` fires.
Pre-fix this was the silent regression from dropping
`usage: { include: true }` — `Response::OpenRouter.cost`
stayed Nil on every blocking call.
- LLM::Chat::Backend::OpenRouter refactor: the lookup logic
is now in a private `!fetch-generation-metadata` helper that
both `_on-stream-complete` and `_on-blocking-complete`
delegate to. No behaviour change on the streaming path.
- Tests — t/12-openrouter-backend.rakutest gains a subtest
covering the defensive guards on both completion hooks
(no-op when generation-id is undefined; no crash when the
response isn't OR-augmented). Plan goes 11 → 12; total
LLM::Chat tests 130.
0.5.0 2026-04-27T22:52:25+01:00
- LLM::Chat::Backend::OpenRouter request shape now mirrors
SillyTavern's wire bytes verbatim. Removed two body fields
that were causing OpenRouter's upstream router to hold 200 OK
headers indefinitely against some providers (~80% header-phase
timeouts in App::Cantina vs ~0% in SillyTavern on the same
models / keys / network):
* `usage: { include: true }` — no longer sent.
* `stream_options: { include_usage: true }` — no longer sent
(also removed from OpenAICommon.chat-completion-stream so
all OAI-compatible streams now match this shape).
* `reasoning: { effort, enabled }` — `enabled` key dropped;
we now send only `{ effort }` when reasoning_effort is
configured, matching ST.
Added on every request, also matching ST:
* `include_reasoning: Bool` — Boolean parity with ST's flag.
* `top_k` — plumbed from Settings into the OAI body.
`repetition_penalty` is now omitted when at the default 1.0
(was previously sent unconditionally).
- Cost telemetry that the inline `usage: { include: true }`
block used to carry now arrives via a one-shot post-stream
GET against `/generation?id=...` after `[DONE]`. Lookup is
async and best-effort; on failure $resp.cost stays Nil rather
than escalating. Lifts cost, provider-name, and (when not
already populated from the stream) prompt/completion tokens.
Latency: ~50–200ms after .is-done becomes True before .cost
is readable. New hook _on-stream-complete on OpenAICommon so
future provider subclasses can do the same kind of
post-stream metadata fetch.
- LLM::Chat::Backend::OpenAICommon stream parser now buffers
bytes across body-byte-stream emissions and splits on the
SSE `\n\n` event delimiter before parsing, instead of
decoding+parsing each TCP chunk independently. Pre-fix, a
`data: {...}` JSON object split across two TCP packets would
crash from-json on the truncated half and terminate the
stream as 'unknown' error class. Both chat-completion-stream
and text-completion-stream got the fix. Heartbeat / SSE
comment lines (`: OPENROUTER PROCESSING`) are dropped per
spec — never produce a chunk.
- LLM::Chat::Backend::OpenAICommon.!classify-exception no
longer string-matches "timeout" / "timed out" in the default
arm; only X::Cro::HTTP::Client::Timeout maps to error-class
'timeout' now. Substring matching was masking unrelated
errors (JSON parse failures, stream-cancel messages) as
header timeouts. Connection-error pattern (refused / reset /
DNS / unreachable) stayed — those don't have a typed
exception class to discriminate on.
- LLM::Chat::Debug log format gains elapsed-ms timestamps for
streaming requests: HEADERS RECEIVED, FIRST BODY BYTE, and
EXCEPTION lines all carry "+Nms" relative to the call start
so latency can be diagnosed without external instrumentation.
Existing log labels unchanged.
0.3.0 2026-04-23T15:56:28+01:00
- LLM::Chat::Backend::Response gains structured error metadata:
$.error-status (Int HTTP code) and $.error-class (Str —
'http' / 'timeout' / 'connection' / 'response' / 'unknown').
Populated via _set-error-info(:$class, :$status) alongside
the existing .quit path. Lets consumers branch on error kind
without regex-parsing raw messages — used by the
LLM::Data::Inference::Task model-fallback policy.
- LLM::Chat::Backend::OpenAICommon CATCH blocks classify Cro
exceptions (X::Cro::HTTP::Error — picks up status off
.response.status; X::Cro::HTTP::Client::Timeout) plus
heuristic socket-error detection into the Response's error
fields before quitting. Finish-reason quits (length /
content_filter / unknown) are tagged error-class => 'response'
so the fallback layer advances on them.
- LLM::Chat::Backend::Mock gains &.error-producer — an optional
(Int $call-index --> Hash) callback that scripts per-call
failures for fallback / retry tests. Returning a hash like
{ class => 'http', status => 500, message => 'x' } fails
that call without consuming a slot from @.responses. Also
exposes $.call-index (monotonic per-backend call count,
bumped on every completion regardless of outcome).
0.2.6 2026-04-13T17:03:42+01:00
- LLM::Chat::Backend::Mock: new test-only backend that returns
canned responses in order. Supports streaming (default splits
on whitespace, configurable via :token-splitter), optional delay
between tokens via :stream-delay, and an :initial-delay (10ms
default) so consumers can attach taps before tokens flow.
:fail-on-empty makes the backend die when the response queue
is exhausted instead of repeating the last entry. Useful for
exercising error paths in downstream consumers.
Recording: every completion call is logged to @.recorded-calls
as a hash with kind / messages / tools / response / at. Tests
can assert on what reached the backend, not just what came
back — catches prompt-assembly and template-substitution
regressions. clear-recorded-calls resets the log between
test phases.
0.2.5 2026-04-09T12:30:48+01:00
- Optional :@tools parameter on chat-completion and chat-completion-stream
- Response.tool-calls and has-tool-calls for detecting LLM tool call requests
- Response.finish-reason field
0.2.4 2026-04-09T05:15:41+01:00
- CI: exclude Windows (Tokenizers Rust FFI build not yet supported)
0.2.3 2026-04-09T05:09:48+01:00
- Add GitHub Actions CI workflow with Rust toolchain for Tokenizers
- Add dist.ini for mi6 (UploadToZef, ReadmeFromPod, Badges)
- Add docs/Readme.rakudoc
0.2.2 2026-04-09T04:59:41+01:00
- Add stub LLM::Chat to make mi6 stop renaming the module.
0.2.1 2026-04-09T04:41:36+01:00
- Add LLM::Chat::Template::Jinja2 for HuggingFace chat template support
- from-tokenizer-config class method loads templates from tokenizer_config.json
- Supports bos_token, eos_token passthrough
- Continuation mode maps to add_generation_prompt=false
0.2.0
- Previous releases