Revision history for LLM::Data::Inference
0.7.0 2026-07-27T15:25:50+01:00
- [2026-07-25] Cancellation and deadline aborts now route through
C instead of C. Closing the
Response only quits the local supplier; the upstream stop signal
lives in the backend override, so against KoboldCpp a cancelled
Task now fires C and the server actually
stops generating. Previously every Task-mediated job (reconcile,
narrative-state, downstream-projection, image prompt-gen,
scene-shakeup, relationship-mode in App::Cantina) looked
cancelled to the caller while the backend drained the full
completion. Backends with no upstream abort endpoint degrade to
exactly the previous local close, and a backend whose C
throws falls back to the local close so the poll loop can never
spin on a live response. Both the C branch and the
C<$.timeout> branch are covered.
- [2026-07-27] Typed truncation. A blocking completion cut off by
C comes back as an HTTP 200 SUCCESS with a partial
body and C 'length' — the "advance on
finish-reason quit" rule in the docs was only ever true of the
STREAMING path. Task therefore handed half-finished output to
the parser, watched it fail, and (with C<:parse-retries> set)
re-rolled the identical over-budget request until the budget
was gone. Task and JSONTask now take
C<:truncation-policy('fail' | 'accept' | '')>: 'fail' records
the truncation and advances to the next backend at once WITHOUT
consuming a parse re-roll (C is per-backend
C, so only a different backend can change it);
'accept' is the historical behaviour; the default C<''> derives
'fail' when a parser is installed and 'accept' when one is not,
so JSON tasks fail fast while prose tasks (continuations, style
passes) keep their partial output bit-for-bit as before. The
check runs BEFORE the empty-body rule, so a reasoning model that
burns its whole budget inside C<< >> is diagnosed as
truncation rather than "empty response body". A chain that
truncates throws the new
C — a SUBCLASS of
C, so DLQ routing, C<:on-exhausted> hooks and every
existing C handler keep working unchanged — and
its summary names the levers (raise C, add a
larger-budget fallback, or shrink the request). Failed responses
also keep any text they emitted before quitting, so partial
streamed output is capturable instead of dropped. Requires
LLM::Chat 0.7.1 for the Mock C knob the
tests drive this with.
- [2026-07-27] Add C<:retry-feedback> to Task and JSONTask (default
False). A parse re-roll used to be blind — identical prompt,
identical model, no word about what was wrong with the answer
just rejected. That is survivable for malformed JSON on a
stochastic sampler, but hopeless for a SEMANTIC rejection: a
verbatim-quote validator that says "quote 2 does not appear in
the passage" says exactly the same thing on every re-roll, so
the item burned its whole budget failing identically and then
dead-lettered something the model could have fixed on the second
try. With the flag set, an attempt that follows a parser or
validator failure on the SAME backend carries one extra C
turn: "Your previous response was rejected: . Respond again with the same JSON contract,
corrected." Each re-roll REPLACES the previous feedback instead
of accumulating a transcript of complaints (the message list is
re-derived from the pristine prompt every attempt), and the
pending reason lives per backend, so truncation advances,
backend advances and network retry-sames all send the prompt
they always did — a model is only ever told about output it
actually produced. The rejected text itself is deliberately not
echoed back (it is already the model's own last turn, and this
ecosystem's validator diagnostics are the part it cannot infer);
the canned wording names a JSON contract, which is why prose
Tasks are expected to leave the flag alone. Telemetry is
unchanged — the extra turn shows up only as a larger
C figure (~100 tokens) on the retry row.
0.5.1 2026-07-21T03:15:12+01:00
- [2026-07-19] Add C<:on-exhausted> to Task: an optional hook that
fires exactly once, on the execute thread, immediately before
C is thrown (never on
success, never on the Cancelled path). The payload carries the
same C records and C string the thrown
exception carries, plus C 'exhausted'> and a
C count — letting callers (e.g. a pipeline DLQ) get
the rich attempts/raw-text payload without re-parsing the
caught exception. Shielded exactly like C: a
throwing hook is caught and noted, never suppressing or
altering the Exhausted throw. JSONTask now forwards
C<:on-exhausted> to its inner Task, same as its other hooks.
- [2026-07-04] Cooperative cancellation: Task and JSONTask accept
C<:is-cancelled>, polled before every round-trip, on each
in-flight completion poll (~10 ms cadence — the pending response
is aborted via Response.cancel, same as the timeout path), and
between chunked backoff sleeps. When the hook reports True the
Task throws the new typed X::LLM::Data::Inference::Cancelled,
which carries the same C<.attempts> records as Exhausted but is
deliberately NOT its subclass — a user cancel must never be
swallowed by exhaustion handlers. A cancelled Task no longer
burns the remaining retry chain (previously up to
max-retries x backends round-trips plus full backoff sleeps
after the caller had already walked away).
- [2026-07-03] Add C<:parse-retries> to Task and JSONTask: an
opt-in same-backend re-roll budget consumed only by parser
failures (default 0 keeps the historical advance-immediately
behaviour). Interactive callers on stochastic samplers re-roll
malformed JSON instead of exhausting the chain on one bad shot.
- [2026-07-03] Chain exhaustion now throws the typed
X::LLM::Data::Inference::Exhausted (new
LLM::Data::Inference::Exceptions module) whose C<.attempts>
records every failure — including the RAW model output for each
failed parse — while C<.message> keeps the exact historical
summary string for back-compat.
- [2026-07-03] Harden JSONTask extraction: strip blocks and
markdown code-fence markers, then pick the longest complete
top-level JSON structure via a string-aware balanced-bracket
scan that actually parses (reasoning preambles with stray
braces, plan-out-loud draft objects, and fenced answers all
extract correctly). Hopeless responses fall back to the legacy
slice so established error messages are preserved.
0.3.0 2026-04-29T23:49:54+01:00
- Bump Github Actions to use node 24+
- BREAKING (telemetry payload): the on-call-complete hook payload
key C<:provider-id> is removed and replaced with C<:generation-id>
(clearer name — the value was always OpenRouter's gen-XXXX id,
not a provider name). New keys C<:provider-name> and C<:is-byok>
are added alongside, lifted off
C when present. Reads
now use C<.?> so non-OR Response subclasses (which have no such
accessors) don't blow up — they simply omit the keys.
- Sink contract is otherwise unchanged: every key remains
presence-gated (absent when the provider didn't supply it).
Migrating sinks: rename C<%payload> →
C<%payload>; optionally consume the new
C / C keys for richer routing visibility.
0.2.0 2026-04-23T15:57:28+01:00
- Added auth specifiers to META6.json dependencies
- Task + JSONTask accept an ordered :@backends fallback chain in
addition to the legacy single :$backend. The retry loop
classifies failures into three buckets: abort (HTTP 400 / 401 /
402 / 403 / 404 — config/account errors where retrying any
model in the chain won't help), retry-same (connection errors /
5xx / unclassifiable — likely transient), and advance (timeout /
429 / empty body / parser failure / other 4xx / content-filter-
style finish-reason quits — model-specific pathology). Each
backend gets up to $.max-retries HTTP attempts (initial +
retries-same, exponential backoff with jitter capped at 30 s);
an advance-class error on any attempt short-circuits the budget
and moves on. Abort-class errors re-raise immediately without
trying the rest of the chain. See classify-error and the module
Pod for the full rule table.
- Task.classify-error(:$error-class, :$error-status,
:$parser-failed) exposed as a public method for testability and
for consumers that want to implement the same policy outside
the retry loop.
- Telemetry hook payload adds :backend-index, :model-name,
:error-class, :error-status, so sinks can identify which model
in the chain served each call and what failed. Existing keys
(:attempt, :success, :error, :latency-ms, :prompt-tokens,
:completion-tokens, :total-tokens, :cost, :model-used,
:provider-id, :finish-reason, :stage) are unchanged.
- Single-:backend constructor shape is preserved; a one-element
chain behaves exactly like the pre-fallback Task on the
retry-same path (connection / 5xx still get $.max-retries
same-model attempts with exponential backoff).
BACKWARD COMPATIBILITY — one behavioural break:
Advance-class errors (timeout / 429 / parser failure / empty body
/ content-filter-style finish quits / other 4xx) used to retry
the same model up to $.max-retries times. They now advance to
the next backend in the chain, or die immediately with
"all backend(s) exhausted" if the Task only has one. Retry-same
errors (connection drop / 5xx) are unchanged — still retried on
the same backend up to $.max-retries times before advancing.
Consequence for single-backend callers: a Task that previously
survived a stochastic parser failure via retry now dies on the
first parse error. Three mitigations:
- Preferred: pass :@backends with a fallback model. A chain
of [primary, primary] is also legal and preserves the exact
old "try the same model twice" behaviour on advance-class
failures while keeping retry-same semantics intact.
- Build the retry loop at the application layer if the same
model genuinely recovers for your workload.
- Accept the fail — in practice the old behaviour rarely
recovered on parser failures (malformed JSON tended to
repeat), which is the motivation for the change.
0.1.3 2026-04-07T20:03:42+01:00
- Updated readme examples to use generic routing terms
0.1.2 2026-04-07T20:00:40+01:00
- Removed Windows from CI (upstream Digest::SHA256::Native does not build on Windows)
0.1.1 2026-04-07T19:52:19+01:00
- Added Windows CI support via MSVC
0.1.0 2026-04-07T18:52:19+01:00
- Initial release
- Task: blocking LLM calls with configurable parser and retry
- JSONTask: JSON extraction from LLM responses with key validation
- Router: query-based routing using Roaring::Tags
- PromptBuilder: mustache-style template rendering