Revision history for LLM::Chat
0.8.1 2026-08-09T20:57:18+01:00
- META fix: Response::OpenRouter and Response::OpenRouter::Stream were
missing from provides and listed under DEPENDS instead. On a machine
without install history, zef resolved those module names to the fossil
LLM::Chat::Backend 0.2.1 dist, whose exact Digest::SHA256::Native 1.0.0
pin only builds with nmake — breaking every fresh Windows install
(caught by LLM-Agent's Windows CI lane). No code change; 0.8.0 as
published carries the broken META and should not be depended on.
0.8.0 2026-08-09T19:41:06+01:00
- [2026-08-09] New C and
C: the retry/fallback policy that
has run inside C since 0.5, factored
out so a second executor can share it instead of copying it.
C exports five pure subs — C
(the abort / retry-same / advance bucket table),
C (C, with the
anti-thundering-herd jitter suppressible via C<:jitter(0e0)> for
deterministic tests), C (chunked sleep that
consults a cancel hook before EVERY chunk including the first,
and returns a Bool rather than throwing, because only the caller
knows which exception type its layer promises),
C and C (which duck-types the
response with C<.?> and therefore does not import
C).
C provides
C and its C /
C subclasses plus C (deliberately NOT an
C, so a user cancel can never be swallowed by a
dead-letter handler), and is now the canonical home of the
C advice contract. Those four are plain global
classes rather than Ced ones on purpose: C
on a nested-name class also exports its LEAF name, so two
modules declaring an C cannot be
imported into the same scope at all — which would have made the
shared hierarchy unusable next to a subclassing one.
Deliberately NOT shared: the
retry loop itself, the abort bucket's C message, hook
payloads and hook shielding — those are shaped by the layer that
owns them. B: this release is purely
additive, every previous test file is untouched, and
C 0.9.0 migrates onto it (its exception
types become subclasses of these, so both hierarchies keep
matching).
- [2026-08-08] Fix: a streamed completion could be KILLED MID-
GENERATION by non-ASCII output. Both SSE loops in OpenAICommon
(C and C) did
C<$buffer ~= $data.decode('utf-8')> per body-byte-stream chunk,
but TCP splits a body wherever it likes — including through the
middle of a multi-byte UTF-8 sequence. C<.decode> on the half
that ends mid-sequence throws "Malformed termination of UTF-8
string", which landed in the streaming CATCH and quit the
Response with error class 'unknown', discarding everything
generated so far. Buffering text could not fix it: the throw
happened before the text reached the buffer. The bytes now go
through an incremental C (new shared internal
C<_stream-decoder>, C<:translate-nl(False)> so SSE framing still
sees the line endings that are actually on the wire), which
keeps an unfinished sequence to itself until the bytes
completing it arrive. Framing, heartbeat skipping, C<[DONE]>
handling, malformed-chunk tolerance and reasoning/content
accumulation are untouched. Emoji, CJK and curly quotes are
ordinary in model output, so this was an intermittent
production crash for anything but plain ASCII.
- [2026-08-08] Fix: non-ASCII in a response body could come back as
mojibake (café -> café). Cro's C returns the
LIST C<('utf-8', 'latin-1')> whenever the Content-Type names no
charset, and C loops that list
with no C — so the latin-1 attempt, which cannot fail
whatever the bytes are, always overwrote the correct utf-8
decode. Many OpenAI-compatible servers answer
C with no charset parameter, and
model output is full of non-ASCII (names, curly quotes, emoji,
CJK). OpenAICommon now decodes response bytes itself via new
internals C<_blob-text> / C<_body-text> / C<_decode-json-body>:
utf-8 first (JSON is UTF-8 by definition, RFC 8259 §8.1), with
latin-1 kept only as a fallback for when the utf-8 decode
actually throws — a proxy's non-JSON error page, say. Affected
the 4xx error-body read that feeds LLM::Chat::Debug, which is
where an operator reads a rejection reason.
- [2026-08-08] The blocking C / C
paths and OpenRouter's C metadata lookup now parse
their JSON from the raw bytes instead of C.
Cro's response body-parser selector reaches the correct parser
only while the server labels the body C; a
server that answers JSON as C fell through to
TextFallback and produced a mojibaked Str, which the completion
path then indexed as a Hash and SILENTLY emitted as empty
content. Well-behaved servers see no change. Transport failures
still propagate as transport exceptions, so error classification
is unchanged.
0.7.1 2026-07-27T15:27:39+01:00
- [2026-07-27] LLM::Chat::Backend::Mock gains
C<&.finish-reason-producer>, an optional
C<(Int $call-index --> Str)> callback mirroring
C<&.error-producer>'s per-call indexing. A defined Str return is
stamped on the Response via C<_set-finish-reason> before the
emission task starts, while the call still SUCCEEDS — the exact
shape of a real blocking completion that ran out of completion
budget (HTTP 200, well-formed but partial body, finish_reason
'length'). Undefined returns leave C unset, so
existing mocks are byte-for-byte unchanged. A scripted error
wins over a scripted finish reason (a failed call never produced
a body). Wired into the non-streaming paths only —
C, and C by delegation —
because the streaming paths already model finish reasons the way
the real transports do. Lets downstream consumers
(LLM::Data::Inference's truncation policy) test truncation
handling without a live model.
- [2026-07-27] Fix: OpenAICommon's C acted
on a terminal C without recording it, so a stream
cut off by the token budget quit with error class 'response' but
an UNDEFINED C<.finish-reason> — consumers could not tell budget
exhaustion apart from a content filter or an unknown reason.
It now calls C<_set-finish-reason> before dispatching on the
reason, matching C.
0.7.0 2026-07-21T03:09:53+01:00
- [2026-07-03] Structured outputs: Backend::Settings gains
C (a plain-Hash JSON Schema, default undefined) and
C. OpenAI-compatible backends (incl.
OpenRouter) send it as C — best-effort,
provider-dependent enforcement; KoboldCpp sends the native
C generate field, which it compiles to a grammar
for sampler-level enforcement. Unset schema changes no request
bytes. Callers should keep parse validation as the backstop for
providers that ignore response_format.
0.6.0 2026-06-22T23:33:11+01:00
- Add LLM::Chat::ToolLoop, a generic streaming tool-call
round loop. It sends OpenAI-format tools to any chat backend,
executes returned tool calls through a caller-supplied
callback, appends assistant/tool turns, and continues until a
final answer or guardrail limit is reached.
- OpenAI-compatible streaming now assembles `delta.tool_calls`
chunks and treats `finish_reason: tool_calls` as successful.
This enables KoboldCpp streamed tool calling.
- Conversation messages can now serialize assistant `tool_calls`
and tool-result `tool_call_id` fields for follow-up chat
completion requests.
- LLM::Chat::Backend::KoboldCpp.cancel is now non-blocking and
failure-proof: the local response stream closes FIRST,
synchronously, then the upstream POST /api/extra/abort fires on
a worker thread, best-effort, with network failures swallowed.
The old ordering awaited the abort before closing locally, so a
busy KoboldCpp froze the caller for the full HTTP round-trip
(interactive cancels hung the UI) and an unreachable one made
cancel throw without ever closing the stream. cancel now
returns the background Promise so callers/tests can await the
abort attempt. Also fixes the abort URL: stripping the /v1
suffix used to leave its leading slash behind, producing
//api/extra/abort.
0.5.1 2026-04-29T23:47:12+01:00
- Bump Github Actions to use node 24+
- LLM::Chat::Backend::OpenAICommon gains a symmetric
`_on-blocking-complete` hook on the non-streaming path,
mirroring the existing `_on-stream-complete` contract: fires
after the response body has been parsed and `_lift-usage`
has lifted OAI/provider usage fields, before
`$response.done`. Default implementation is a no-op.
Subclasses use it to attach post-call metadata that isn't
in the body itself — symmetric with what was already
possible on streams.
- LLM::Chat::Backend::OpenRouter wires the new hook through
to the same `/generation?id=...` lookup the streaming path
already uses, so blocking callers (e.g. App::Storygen, which
calls `chat-completion` rather than `chat-completion-stream`)
now see `.cost` populated by the time `$response.done` fires.
Pre-fix this was the silent regression from dropping
`usage: { include: true }` — `Response::OpenRouter.cost`
stayed Nil on every blocking call.
- LLM::Chat::Backend::OpenRouter refactor: the lookup logic
is now in a private `!fetch-generation-metadata` helper that
both `_on-stream-complete` and `_on-blocking-complete`
delegate to. No behaviour change on the streaming path.
- Tests — t/12-openrouter-backend.rakutest gains a subtest
covering the defensive guards on both completion hooks
(no-op when generation-id is undefined; no crash when the
response isn't OR-augmented). Plan goes 11 → 12; total
LLM::Chat tests 130.
0.5.0 2026-04-27T22:52:25+01:00
- LLM::Chat::Backend::OpenRouter request shape now mirrors
SillyTavern's wire bytes verbatim. Removed two body fields
that were causing OpenRouter's upstream router to hold 200 OK
headers indefinitely against some providers (~80% header-phase
timeouts in App::Cantina vs ~0% in SillyTavern on the same
models / keys / network):
* `usage: { include: true }` — no longer sent.
* `stream_options: { include_usage: true }` — no longer sent
(also removed from OpenAICommon.chat-completion-stream so
all OAI-compatible streams now match this shape).
* `reasoning: { effort, enabled }` — `enabled` key dropped;
we now send only `{ effort }` when reasoning_effort is
configured, matching ST.
Added on every request, also matching ST:
* `include_reasoning: Bool` — Boolean parity with ST's flag.
* `top_k` — plumbed from Settings into the OAI body.
`repetition_penalty` is now omitted when at the default 1.0
(was previously sent unconditionally).
- Cost telemetry that the inline `usage: { include: true }`
block used to carry now arrives via a one-shot post-stream
GET against `/generation?id=...` after `[DONE]`. Lookup is
async and best-effort; on failure $resp.cost stays Nil rather
than escalating. Lifts cost, provider-name, and (when not
already populated from the stream) prompt/completion tokens.
Latency: ~50–200ms after .is-done becomes True before .cost
is readable. New hook _on-stream-complete on OpenAICommon so
future provider subclasses can do the same kind of
post-stream metadata fetch.
- LLM::Chat::Backend::OpenAICommon stream parser now buffers
bytes across body-byte-stream emissions and splits on the
SSE `\n\n` event delimiter before parsing, instead of
decoding+parsing each TCP chunk independently. Pre-fix, a
`data: {...}` JSON object split across two TCP packets would
crash from-json on the truncated half and terminate the
stream as 'unknown' error class. Both chat-completion-stream
and text-completion-stream got the fix. Heartbeat / SSE
comment lines (`: OPENROUTER PROCESSING`) are dropped per
spec — never produce a chunk.
- LLM::Chat::Backend::OpenAICommon.!classify-exception no
longer string-matches "timeout" / "timed out" in the default
arm; only X::Cro::HTTP::Client::Timeout maps to error-class
'timeout' now. Substring matching was masking unrelated
errors (JSON parse failures, stream-cancel messages) as
header timeouts. Connection-error pattern (refused / reset /
DNS / unreachable) stayed — those don't have a typed
exception class to discriminate on.
- LLM::Chat::Debug log format gains elapsed-ms timestamps for
streaming requests: HEADERS RECEIVED, FIRST BODY BYTE, and
EXCEPTION lines all carry "+Nms" relative to the call start
so latency can be diagnosed without external instrumentation.
Existing log labels unchanged.
0.3.0 2026-04-23T15:56:28+01:00
- LLM::Chat::Backend::Response gains structured error metadata:
$.error-status (Int HTTP code) and $.error-class (Str —
'http' / 'timeout' / 'connection' / 'response' / 'unknown').
Populated via _set-error-info(:$class, :$status) alongside
the existing .quit path. Lets consumers branch on error kind
without regex-parsing raw messages — used by the
LLM::Data::Inference::Task model-fallback policy.
- LLM::Chat::Backend::OpenAICommon CATCH blocks classify Cro
exceptions (X::Cro::HTTP::Error — picks up status off
.response.status; X::Cro::HTTP::Client::Timeout) plus
heuristic socket-error detection into the Response's error
fields before quitting. Finish-reason quits (length /
content_filter / unknown) are tagged error-class => 'response'
so the fallback layer advances on them.
- LLM::Chat::Backend::Mock gains &.error-producer — an optional
(Int $call-index --> Hash) callback that scripts per-call
failures for fallback / retry tests. Returning a hash like
{ class => 'http', status => 500, message => 'x' } fails
that call without consuming a slot from @.responses. Also
exposes $.call-index (monotonic per-backend call count,
bumped on every completion regardless of outcome).
0.2.6 2026-04-13T17:03:42+01:00
- LLM::Chat::Backend::Mock: new test-only backend that returns
canned responses in order. Supports streaming (default splits
on whitespace, configurable via :token-splitter), optional delay
between tokens via :stream-delay, and an :initial-delay (10ms
default) so consumers can attach taps before tokens flow.
:fail-on-empty makes the backend die when the response queue
is exhausted instead of repeating the last entry. Useful for
exercising error paths in downstream consumers.
Recording: every completion call is logged to @.recorded-calls
as a hash with kind / messages / tools / response / at. Tests
can assert on what reached the backend, not just what came
back — catches prompt-assembly and template-substitution
regressions. clear-recorded-calls resets the log between
test phases.
0.2.5 2026-04-09T12:30:48+01:00
- Optional :@tools parameter on chat-completion and chat-completion-stream
- Response.tool-calls and has-tool-calls for detecting LLM tool call requests
- Response.finish-reason field
0.2.4 2026-04-09T05:15:41+01:00
- CI: exclude Windows (Tokenizers Rust FFI build not yet supported)
0.2.3 2026-04-09T05:09:48+01:00
- Add GitHub Actions CI workflow with Rust toolchain for Tokenizers
- Add dist.ini for mi6 (UploadToZef, ReadmeFromPod, Badges)
- Add docs/Readme.rakudoc
0.2.2 2026-04-09T04:59:41+01:00
- Add stub LLM::Chat to make mi6 stop renaming the module.
0.2.1 2026-04-09T04:41:36+01:00
- Add LLM::Chat::Template::Jinja2 for HuggingFace chat template support
- from-tokenizer-config class method loads templates from tokenizer_config.json
- Supports bos_token, eos_token passthrough
- Continuation mode maps to add_generation_prompt=false
0.2.0
- Previous releases