Revision history for LLM::Chat
0.10.0 2026-08-21T22:05:38+01:00
- [2026-08-21] Require Template::Jinja2 0.3.0. The byte-identical
rendering of upstream GLM and Kimi chat templates leans on its Python
method-call dispatch and Jinja2-faithful filters; an older renderer
would produce silently different prompts.
- [2026-08-20] The OpenRouter backend marks `cache_control`
breakpoints on outgoing chat bodies, so providers that only
cache a prompt prefix when the request says where it ends
(Anthropic, Qwen, Gemini) can actually cache one. Markers go on
the head message and on the last two `user` / `assistant` turns
— the trailing pair is what makes the marker written one round
ago still cover what the next request shares with it — with
tool results and tool-calling assistants stepped over, and
never more than OpenRouter's ceiling of four. Implicit-caching
routes ignore the annotation, so the new `cache-breakpoints`
attribute defaults to On across a mixed fleet; set it to False
to send the plain string shape. Conversation state is untouched
either way: the rewrite happens on the serialized body, so
`Message.to-hash` and `Message.get-checksum` read the same
after a request as before it.
Placing them needed a seam, so `OpenAICommon` gained one:
`_finalize-request-body` is called on all four completion paths
with the assembled body, after messages / prompt / tools /
stream are attached and before the POST. The default returns
its argument, so nothing changes for backends that don't
override it.
- [2026-08-20] New `Response.cached-prompt-tokens`: the slice of
the prompt a provider served out of its own cache rather than
processing fresh, which is what tells a caller whether the
input it is paying full price for is actually being reused.
Lifted from the OAI-spec `usage.prompt_tokens_details.
cached_tokens` on any OpenAI-compatible backend, and on
OpenRouter also from the `/generation` poll — the only place
the streaming path can pick it up, since that backend
deliberately asks for no inline usage frame. A value the
response body reported always wins over the poll's. Undefined
means the provider said nothing about caching, which is not the
same as nothing having been cached, so a reported zero stays a
reported zero.
- [2026-08-19] A streamed response now always reaches a terminal
state. Both streaming paths route every ending through one
place, so `[DONE]`, a `stop` / `tool_calls` finish reason
followed by the body closing, and a body that closes with
neither all settle the Response — the last of those as a
failure with error-class `response`, since a reply cut off in
transit cannot be told from a reply that ended. Previously that
third case left the Response neither done nor quit: a consumer
polling `.is-done` waited out its own timeout, minutes at a
time, for a stream that had been over from the start. Providers
that close on the finish_reason chunk without ever sending
`[DONE]` are common, and they now complete promptly and
successfully instead. The `length` / `content_filter` /
unknown-reason failures are unchanged.
Fixes a hook-ordering bug on the way: text-completion-stream's
`[DONE]` arm completed the supply BEFORE `_on-stream-complete`,
so a subclass's post-stream metadata landed after consumers had
been told the response was finished. Both paths now fire the
hook first, as the hook has always documented.
- [2026-08-19] New `Response.dropped-frames`: streamed `data:`
lines that fail to parse are still skipped — providers
interleave junk into healthy streams — but they are now counted,
and a stream that dropped frames while assembling tool calls
FAILS instead of succeeding. A hole in prose costs a few words
and is visible to whoever reads the reply; a hole in a tool
call's `arguments` is spliced out of a JSON string that is then
concatenated blind, and what comes out can be valid JSON saying
something the model never asked for. Nothing downstream can
detect that, so it no longer reaches `is-success`. Prose-only
streams keep their success and expose the counter. SSE
heartbeats and empty `data:` lines are not counted.
- [2026-08-19] Streaming tool-call deltas that carry no `index`
are assembled correctly. The OAI spec puts an `index` on every
fragment and it is still authoritative when present, but some
providers omit it, and the old fallback used the number of calls
seen so far — which sent every argument fragment to a fresh
call. One tool call arrived shattered into N, each holding an
unparseable slice of the JSON. The fallback now reads `id`,
which the spec sends only on a call's first fragment: an `id`
starts a new call, an arguments-only fragment continues the one
in flight.
- [2026-08-19] `LLM::Chat::Template::Jinja2.render` presents
tool-call `function.arguments` to the template as a decoded
object when they are a JSON object, matching what HuggingFace
chat templates are written against (transformers passes a dict).
GLM-family templates iterate `arguments.items()` and rendered
nothing at all against the JSON string the OpenAI wire format
uses — a silently truncated prompt, and for a token counter an
undercount of the largest part of it. The messages themselves
are untouched: the render works on a copy, so `to-hash` and
every request body built from a conversation still carry the
string. Arguments that are not a JSON object — empty, a fragment
cut short mid-stream, an array, a bare scalar, or a Hash a
caller supplied directly — are passed through exactly as they
are.
- [2026-08-13] `LLM::Chat::TokenCounter.get-request-count` counts a
complete selected-model request: optional runtime context head, stored
conversation, optional runtime context tail, and tool catalogue rendered
through the model's actual template, then tokenized without a second
special-token pass. This is the exact-counting seam for models such as
Kimi K2.7 whose tool declarations live in the chat template. Existing
`get-conversation-count` remains conversation-only and unchanged.
Kimi's published Jinja fallback serializes ordinary OpenAI tool
declarations with compact `tojson`; `tools_ts_str` is an optional
preformatted override, so no model-specific duplicate renderer is used.
Runtime context may be supplied as its natural string halves or as
prebuilt system messages. Rendered prompts opt into declared special
tokens explicitly, which is required by deny-by-default tiktoken
backends and harmless for Hugging Face JSON tokenizers.
- [2026-08-13] Add `LLM::Chat::Template::DeepSeekV4`, a byte-compatible
Raku port of the full prompt-encoding half of DeepSeek's pinned DSV4
reference implementation. It supports chat/thinking modes, reasoning
effort, context and thinking removal, developer/latest-reminder turns,
quick tasks, response formats, OpenAI-format tools/calls, DSML argument
encoding, and merged/call-ordered tool results. `encode-json` preserves
JSON object order for differential goldens; `encode-messages` accepts
ordinary associative message structures; `render` adapts existing
`Conversation::Message` objects. Completion parsing is intentionally not
included. The four official Python outputs are pinned as offline golden
tests with DeepSeek's MIT attribution.
- [2026-08-12] A Cro timeout is now classified BY THE PHASE it
expired in. `X::Cro::HTTP::Client::Timeout` carries a `phase` of
`connection`, `headers` or `body`, and OpenAICommon's classifier
used to file all three as error-class `timeout` — the ADVANCE
bucket. A connection-phase timeout is Cro's 30-second connect
budget running out: nothing was sent, the endpoint said nothing,
and the only evidence is a network that was unwell. That is now
error-class `connection`, the RETRY-SAME bucket. Headers- and
body-phase timeouts keep `timeout`: a backend that accepted the
connection and then would not answer has said something about
itself, and the next backend answers sooner than a backoff ends.
The lumping was fatal on a ONE-BACKEND chain, where `advance`
means the chain is exhausted: a single flaky connect ended the
run having made exactly one call. `classify-error`'s own bucket
table is unchanged — the split is at the source, which is where
the information is.
The classification test no longer re-implements the classifier
it is testing; it calls the real private method through
`^find_private_method`, so the two cannot drift.
- [2026-08-12] `LLM::Chat::Template::Jinja2.render` takes an
optional `:@tools` — the tool declarations the request will
carry, in the provider's own shape — and passes them to the
template as its `tools` variable. Real HuggingFace chat
templates render a tools block when they are given one, so a
render meant to match what the server sees needs them.
Passed ONLY when non-empty: `tools => []` and no tools at all
both render exactly as they did before the parameter existed
(pinned by a test, because a template asking
`{% if tools is defined %}` would otherwise take a different
branch for a caller who named nothing). Deliberately NOT wired
into `LLM::Chat::TokenCounter`: the declarations are a
per-request constant that provider-reported usage has already
been charged for, and counting them per message would bill them
twice.
0.8.1 2026-08-09T20:57:18+01:00
- META fix: Response::OpenRouter and Response::OpenRouter::Stream were
missing from provides and listed under DEPENDS instead. On a machine
without install history, zef resolved those module names to the fossil
LLM::Chat::Backend 0.2.1 dist, whose exact Digest::SHA256::Native 1.0.0
pin only builds with nmake — breaking every fresh Windows install
(caught by LLM-Agent's Windows CI lane). No code change; 0.8.0 as
published carries the broken META and should not be depended on.
0.8.0 2026-08-09T19:41:06+01:00
- [2026-08-09] New C and
C: the retry/fallback policy that
has run inside C since 0.5, factored
out so a second executor can share it instead of copying it.
C exports five pure subs — C
(the abort / retry-same / advance bucket table),
C (C, with the
anti-thundering-herd jitter suppressible via C<:jitter(0e0)> for
deterministic tests), C (chunked sleep that
consults a cancel hook before EVERY chunk including the first,
and returns a Bool rather than throwing, because only the caller
knows which exception type its layer promises),
C and C (which duck-types the
response with C<.?> and therefore does not import
C).
C provides
C and its C /
C subclasses plus C (deliberately NOT an
C, so a user cancel can never be swallowed by a
dead-letter handler), and is now the canonical home of the
C advice contract. Those four are plain global
classes rather than Ced ones on purpose: C
on a nested-name class also exports its LEAF name, so two
modules declaring an C cannot be
imported into the same scope at all — which would have made the
shared hierarchy unusable next to a subclassing one.
Deliberately NOT shared: the
retry loop itself, the abort bucket's C message, hook
payloads and hook shielding — those are shaped by the layer that
owns them. B: this release is purely
additive, every previous test file is untouched, and
C 0.9.0 migrates onto it (its exception
types become subclasses of these, so both hierarchies keep
matching).
- [2026-08-08] Fix: a streamed completion could be KILLED MID-
GENERATION by non-ASCII output. Both SSE loops in OpenAICommon
(C and C) did
C<$buffer ~= $data.decode('utf-8')> per body-byte-stream chunk,
but TCP splits a body wherever it likes — including through the
middle of a multi-byte UTF-8 sequence. C<.decode> on the half
that ends mid-sequence throws "Malformed termination of UTF-8
string", which landed in the streaming CATCH and quit the
Response with error class 'unknown', discarding everything
generated so far. Buffering text could not fix it: the throw
happened before the text reached the buffer. The bytes now go
through an incremental C (new shared internal
C<_stream-decoder>, C<:translate-nl(False)> so SSE framing still
sees the line endings that are actually on the wire), which
keeps an unfinished sequence to itself until the bytes
completing it arrive. Framing, heartbeat skipping, C<[DONE]>
handling, malformed-chunk tolerance and reasoning/content
accumulation are untouched. Emoji, CJK and curly quotes are
ordinary in model output, so this was an intermittent
production crash for anything but plain ASCII.
- [2026-08-08] Fix: non-ASCII in a response body could come back as
mojibake (café -> café). Cro's C returns the
LIST C<('utf-8', 'latin-1')> whenever the Content-Type names no
charset, and C loops that list
with no C — so the latin-1 attempt, which cannot fail
whatever the bytes are, always overwrote the correct utf-8
decode. Many OpenAI-compatible servers answer
C with no charset parameter, and
model output is full of non-ASCII (names, curly quotes, emoji,
CJK). OpenAICommon now decodes response bytes itself via new
internals C<_blob-text> / C<_body-text> / C<_decode-json-body>:
utf-8 first (JSON is UTF-8 by definition, RFC 8259 §8.1), with
latin-1 kept only as a fallback for when the utf-8 decode
actually throws — a proxy's non-JSON error page, say. Affected
the 4xx error-body read that feeds LLM::Chat::Debug, which is
where an operator reads a rejection reason.
- [2026-08-08] The blocking C / C
paths and OpenRouter's C metadata lookup now parse
their JSON from the raw bytes instead of C.
Cro's response body-parser selector reaches the correct parser
only while the server labels the body C; a
server that answers JSON as C fell through to
TextFallback and produced a mojibaked Str, which the completion
path then indexed as a Hash and SILENTLY emitted as empty
content. Well-behaved servers see no change. Transport failures
still propagate as transport exceptions, so error classification
is unchanged.
0.7.1 2026-07-27T15:27:39+01:00
- [2026-07-27] LLM::Chat::Backend::Mock gains
C<&.finish-reason-producer>, an optional
C<(Int $call-index --> Str)> callback mirroring
C<&.error-producer>'s per-call indexing. A defined Str return is
stamped on the Response via C<_set-finish-reason> before the
emission task starts, while the call still SUCCEEDS — the exact
shape of a real blocking completion that ran out of completion
budget (HTTP 200, well-formed but partial body, finish_reason
'length'). Undefined returns leave C unset, so
existing mocks are byte-for-byte unchanged. A scripted error
wins over a scripted finish reason (a failed call never produced
a body). Wired into the non-streaming paths only —
C, and C by delegation —
because the streaming paths already model finish reasons the way
the real transports do. Lets downstream consumers
(LLM::Data::Inference's truncation policy) test truncation
handling without a live model.
- [2026-07-27] Fix: OpenAICommon's C acted
on a terminal C without recording it, so a stream
cut off by the token budget quit with error class 'response' but
an UNDEFINED C<.finish-reason> — consumers could not tell budget
exhaustion apart from a content filter or an unknown reason.
It now calls C<_set-finish-reason> before dispatching on the
reason, matching C.
0.7.0 2026-07-21T03:09:53+01:00
- [2026-07-03] Structured outputs: Backend::Settings gains
C (a plain-Hash JSON Schema, default undefined) and
C. OpenAI-compatible backends (incl.
OpenRouter) send it as C — best-effort,
provider-dependent enforcement; KoboldCpp sends the native
C generate field, which it compiles to a grammar
for sampler-level enforcement. Unset schema changes no request
bytes. Callers should keep parse validation as the backstop for
providers that ignore response_format.
0.6.0 2026-06-22T23:33:11+01:00
- Add LLM::Chat::ToolLoop, a generic streaming tool-call
round loop. It sends OpenAI-format tools to any chat backend,
executes returned tool calls through a caller-supplied
callback, appends assistant/tool turns, and continues until a
final answer or guardrail limit is reached.
- OpenAI-compatible streaming now assembles `delta.tool_calls`
chunks and treats `finish_reason: tool_calls` as successful.
This enables KoboldCpp streamed tool calling.
- Conversation messages can now serialize assistant `tool_calls`
and tool-result `tool_call_id` fields for follow-up chat
completion requests.
- LLM::Chat::Backend::KoboldCpp.cancel is now non-blocking and
failure-proof: the local response stream closes FIRST,
synchronously, then the upstream POST /api/extra/abort fires on
a worker thread, best-effort, with network failures swallowed.
The old ordering awaited the abort before closing locally, so a
busy KoboldCpp froze the caller for the full HTTP round-trip
(interactive cancels hung the UI) and an unreachable one made
cancel throw without ever closing the stream. cancel now
returns the background Promise so callers/tests can await the
abort attempt. Also fixes the abort URL: stripping the /v1
suffix used to leave its leading slash behind, producing
//api/extra/abort.
0.5.1 2026-04-29T23:47:12+01:00
- Bump Github Actions to use node 24+
- LLM::Chat::Backend::OpenAICommon gains a symmetric
`_on-blocking-complete` hook on the non-streaming path,
mirroring the existing `_on-stream-complete` contract: fires
after the response body has been parsed and `_lift-usage`
has lifted OAI/provider usage fields, before
`$response.done`. Default implementation is a no-op.
Subclasses use it to attach post-call metadata that isn't
in the body itself — symmetric with what was already
possible on streams.
- LLM::Chat::Backend::OpenRouter wires the new hook through
to the same `/generation?id=...` lookup the streaming path
already uses, so blocking callers (e.g. App::Storygen, which
calls `chat-completion` rather than `chat-completion-stream`)
now see `.cost` populated by the time `$response.done` fires.
Pre-fix this was the silent regression from dropping
`usage: { include: true }` — `Response::OpenRouter.cost`
stayed Nil on every blocking call.
- LLM::Chat::Backend::OpenRouter refactor: the lookup logic
is now in a private `!fetch-generation-metadata` helper that
both `_on-stream-complete` and `_on-blocking-complete`
delegate to. No behaviour change on the streaming path.
- Tests — t/12-openrouter-backend.rakutest gains a subtest
covering the defensive guards on both completion hooks
(no-op when generation-id is undefined; no crash when the
response isn't OR-augmented). Plan goes 11 → 12; total
LLM::Chat tests 130.
0.5.0 2026-04-27T22:52:25+01:00
- LLM::Chat::Backend::OpenRouter request shape now mirrors
SillyTavern's wire bytes verbatim. Removed two body fields
that were causing OpenRouter's upstream router to hold 200 OK
headers indefinitely against some providers (~80% header-phase
timeouts in App::Cantina vs ~0% in SillyTavern on the same
models / keys / network):
* `usage: { include: true }` — no longer sent.
* `stream_options: { include_usage: true }` — no longer sent
(also removed from OpenAICommon.chat-completion-stream so
all OAI-compatible streams now match this shape).
* `reasoning: { effort, enabled }` — `enabled` key dropped;
we now send only `{ effort }` when reasoning_effort is
configured, matching ST.
Added on every request, also matching ST:
* `include_reasoning: Bool` — Boolean parity with ST's flag.
* `top_k` — plumbed from Settings into the OAI body.
`repetition_penalty` is now omitted when at the default 1.0
(was previously sent unconditionally).
- Cost telemetry that the inline `usage: { include: true }`
block used to carry now arrives via a one-shot post-stream
GET against `/generation?id=...` after `[DONE]`. Lookup is
async and best-effort; on failure $resp.cost stays Nil rather
than escalating. Lifts cost, provider-name, and (when not
already populated from the stream) prompt/completion tokens.
Latency: ~50–200ms after .is-done becomes True before .cost
is readable. New hook _on-stream-complete on OpenAICommon so
future provider subclasses can do the same kind of
post-stream metadata fetch.
- LLM::Chat::Backend::OpenAICommon stream parser now buffers
bytes across body-byte-stream emissions and splits on the
SSE `\n\n` event delimiter before parsing, instead of
decoding+parsing each TCP chunk independently. Pre-fix, a
`data: {...}` JSON object split across two TCP packets would
crash from-json on the truncated half and terminate the
stream as 'unknown' error class. Both chat-completion-stream
and text-completion-stream got the fix. Heartbeat / SSE
comment lines (`: OPENROUTER PROCESSING`) are dropped per
spec — never produce a chunk.
- LLM::Chat::Backend::OpenAICommon.!classify-exception no
longer string-matches "timeout" / "timed out" in the default
arm; only X::Cro::HTTP::Client::Timeout maps to error-class
'timeout' now. Substring matching was masking unrelated
errors (JSON parse failures, stream-cancel messages) as
header timeouts. Connection-error pattern (refused / reset /
DNS / unreachable) stayed — those don't have a typed
exception class to discriminate on.
- LLM::Chat::Debug log format gains elapsed-ms timestamps for
streaming requests: HEADERS RECEIVED, FIRST BODY BYTE, and
EXCEPTION lines all carry "+Nms" relative to the call start
so latency can be diagnosed without external instrumentation.
Existing log labels unchanged.
0.3.0 2026-04-23T15:56:28+01:00
- LLM::Chat::Backend::Response gains structured error metadata:
$.error-status (Int HTTP code) and $.error-class (Str —
'http' / 'timeout' / 'connection' / 'response' / 'unknown').
Populated via _set-error-info(:$class, :$status) alongside
the existing .quit path. Lets consumers branch on error kind
without regex-parsing raw messages — used by the
LLM::Data::Inference::Task model-fallback policy.
- LLM::Chat::Backend::OpenAICommon CATCH blocks classify Cro
exceptions (X::Cro::HTTP::Error — picks up status off
.response.status; X::Cro::HTTP::Client::Timeout) plus
heuristic socket-error detection into the Response's error
fields before quitting. Finish-reason quits (length /
content_filter / unknown) are tagged error-class => 'response'
so the fallback layer advances on them.
- LLM::Chat::Backend::Mock gains &.error-producer — an optional
(Int $call-index --> Hash) callback that scripts per-call
failures for fallback / retry tests. Returning a hash like
{ class => 'http', status => 500, message => 'x' } fails
that call without consuming a slot from @.responses. Also
exposes $.call-index (monotonic per-backend call count,
bumped on every completion regardless of outcome).
0.2.6 2026-04-13T17:03:42+01:00
- LLM::Chat::Backend::Mock: new test-only backend that returns
canned responses in order. Supports streaming (default splits
on whitespace, configurable via :token-splitter), optional delay
between tokens via :stream-delay, and an :initial-delay (10ms
default) so consumers can attach taps before tokens flow.
:fail-on-empty makes the backend die when the response queue
is exhausted instead of repeating the last entry. Useful for
exercising error paths in downstream consumers.
Recording: every completion call is logged to @.recorded-calls
as a hash with kind / messages / tools / response / at. Tests
can assert on what reached the backend, not just what came
back — catches prompt-assembly and template-substitution
regressions. clear-recorded-calls resets the log between
test phases.
0.2.5 2026-04-09T12:30:48+01:00
- Optional :@tools parameter on chat-completion and chat-completion-stream
- Response.tool-calls and has-tool-calls for detecting LLM tool call requests
- Response.finish-reason field
0.2.4 2026-04-09T05:15:41+01:00
- CI: exclude Windows (Tokenizers Rust FFI build not yet supported)
0.2.3 2026-04-09T05:09:48+01:00
- Add GitHub Actions CI workflow with Rust toolchain for Tokenizers
- Add dist.ini for mi6 (UploadToZef, ReadmeFromPod, Badges)
- Add docs/Readme.rakudoc
0.2.2 2026-04-09T04:59:41+01:00
- Add stub LLM::Chat to make mi6 stop renaming the module.
0.2.1 2026-04-09T04:41:36+01:00
- Add LLM::Chat::Template::Jinja2 for HuggingFace chat template support
- from-tokenizer-config class method loads templates from tokenizer_config.json
- Supports bos_token, eos_token passthrough
- Continuation mode maps to add_generation_prompt=false
0.2.0
- Previous releases