Revision history for ONNX::Native
0.2.0 2026-08-11T14:58:54+01:00
- CI: fix the macOS bundle lanes dying at the shim step with
"cuda_def[@]: unbound variable" — macOS runners resolve
`shell: bash` to /bin/bash 3.2, where expanding an empty
array under `set -u` is an error (fixed in bash 4.4, so the
Linux lanes never saw it). Now uses the portable
${arr[@]+"${arr[@]}"} idiom.
- lib/ONNX/Native/FFI.rakumod: convert the `$shim-lib` library-
path binding from `constant` to a state-cached sub
(`sub shim-lib { state $r = _resolve-shim(); $r }`).
`constant X = _resolve-shim()` ran at compile time and baked
the resolved path into the precompiled bytecode — and Rakudo
doesn't track `resources/BINARY_TAG` as a precomp dependency.
A BINARY_TAG bump (which moves staged libs to a new versioned
directory and may GC the previous one) would leave the
precomp pointing at the old path, producing "Cannot locate
native library" errors on freshly installed packages until
the user manually ran `rm -rf ~/.raku/precomp/`. Deferring
resolution to first sub-call means each process picks up the
current tag, regardless of when the precomp was built.
NativeCall accepts a Callable for `is native()` and invokes
it lazily on first use of each bound sub. All
`is native($shim-lib)` callsites updated to
`is native(&shim-lib)` to match.
- Same fix applied across all three native libs in this release
window (Notcurses::Native, Vips::Native, ONNX::Native), so
consumers upgrading any of them past the constant-baking
precomp bug get the fix consistently.
- `Build.rakumod` now garbage-collects sibling staged dirs for
older BINARY_TAGs after each successful install. Without this,
every release accumulated another `binaries-onnxruntime-*` dir
under `~/.local/share/ONNX-Native/`, where stale Raku precomp
could load the older libs alongside the new ones — cf. the
Vips::Native r7→r8 incident that revealed this class of bug.
Set `ONNX_NATIVE_KEEP_OLD_STAGES=1` to opt out (e.g. when
intentionally pinning multiple versions for testing).
- Fix silent shape truncation for rank > 16 models. The Raku
introspection path (`Session.input-info` / `output-info` and
`Tensor.shape`) was hardcoded to a 16-element shape buffer
and read past it via `(^$rank).map`, producing garbage trailing
dims for any model deeper than 16D. The shim already documents
a re-call protocol for cap-exceeded shapes; the Raku side now
honours it via a shared `fetch-shape-with-cap` helper that
auto-grows the buffer (one extra ORT call worst case, common
case unchanged). Covered by t/04-shape-edges.rakutest using a
synthetic rank-17 Identity model fixture.
- Allow rank-0 (scalar) tensors in `onnx_shim_create_tensor`.
The shim used to reject `shape == NULL` unconditionally, which
blocked the legitimate "scalar OrtValue" case (output-as-pooled
classifier head, etc.). Now `shape == NULL` is accepted iff
`rank == 0`; for `rank > 0`, shape is still required to avoid
ORT dereferencing NULL. `Tensor.from-blob(:shape([]))` now
works end-to-end.
- Add `SIZE_MAX / elem_size` overflow guard in
`onnx_shim_tensor_data`'s byte-length computation. Output
tensors materialised by ORT have no a-priori bound, so a
malformed model could in principle wrap size_t. Defense in
depth — Raku-side input validation already covers input
tensors but doesn't run on outputs.
- Add `onnx_shim_runtime_version_string` shim export +
`runtime-version()` / `api-version()` Raku-side helpers.
`api-version()` is the ORT_API_VERSION the shim was compiled
against; `runtime-version()` is whatever libonnxruntime
actually got loaded (e.g. "1.20.1"). Surfacing both turns
"requested API version not available" load failures into
diagnosable "shim X, runtime Y" messages.
- Fix stale doc comment for `onnx_shim_free_name` in
`onnx_native_shim.h`. Said "Free via ORT's default allocator"
but the implementation malloc-copies before returning, so the
free is a plain `free()` (and has been since 0.1.0). Comment
now matches the implementation.
- Atomic load/store on `g_ort` cache pointer in the shim. The
lazy init pattern was technically a data race under C11 even
though concurrent first-callers compute the same pointer
(TSan flags it; pointer-write-as-atomic on x86/x64/arm64 is
true in practice but not formally guaranteed). Now uses
`__atomic_load_n` / `__atomic_store_n` on GCC / Clang and
`_InterlockedCompareExchangePointer` /
`_InterlockedExchangePointer` on MSVC, with an
acquire-on-load / release-on-store ordering. Behaviour is
unchanged for the common case (single-threaded init); the
change closes the race for Raku apps that drive inference
from multiple Promises.
- Document provider flags semantics in `docs/Readme.rakudoc`
(Provider enum § "Provider flags") and reference from
`Native.rakumod`. The shim's `flags` argument is per-provider:
CoreML COREMLFlags bitfield, CUDA / DML `device_id`, ignored
for CPU. Currently the Raku surface hardcodes flags=0 and
this Pod6 paragraph sets the contract for the future
`:%provider-options` API.
- Bump `BINARY_TAG` to `binaries-onnxruntime-1.20.0-r4` in
`BINARY_TAG` and `resources/BINARY_TAG`. Triggers
`.github/workflows/build-binaries.yml` to rebuild the
per-platform tarballs (picking up the shim's atomic-init,
scalar-tensor, runtime-version-string changes). New
`resources/checksums.txt` will land alongside CI's release
output.
- CUDA GPU support on x86_64 Linux + x86_64 Windows added. Set
`ONNX_NATIVE_WITH_CUDA=1` at install time and Build.rakumod
switches the upstream URL to Microsoft's `*-gpu` prebuilt
(which bundles libcudnn / libcublas / libcudart — end user
only needs the NVIDIA driver, not a toolkit install),
stages it under a `-gpu` suffixed dir so CPU and GPU
bundles coexist, and compiles the shim with
`-DONNX_SHIM_WITH_CUDA=1`. Refused cleanly on macOS,
aarch64 Linux, and Windows ARM64 — Microsoft doesn't
publish CUDA prebuilts for those targets and the platforms
themselves either lack drivers (Win-on-ARM) or live in a
different ecosystem (Jetson / JetPack on aarch64-Linux).
- `Session.new` now takes `:cuda-device-id` and
`:coreml-flags` named args (default 0). Each routes to the
matching provider's `flags` channel via
`onnx_shim_enable_provider`. The previous hardcoded 0
becomes the default, so existing calls keep their
behaviour. CUDA device selection finally works for users
with multi-GPU boxes; CoreML callers can now opt into
`COREML_FLAG_USE_CPU_ONLY` / `..._ANE` etc. without editing
the source. DirectML's `device_id` isn't surfaced (DML
itself is deferred, separate NuGet sourcing).
- `.github/workflows/build-binaries.yml` matrix gains
`linux-x86_64-glibc-gpu` + `windows-x86_64-gpu` entries.
They download the matching `*-gpu` upstream prebuilt and
produce parallel artifacts named
`onnxruntime-{linux-x86_64-glibc,windows-x86_64}-gpu.{tar.gz,zip}`
that Build.rakumod fetches when WITH_CUDA is set. CI
verifies the GPU bundle compiles + links + exports the
expected symbols; it does NOT run inference (no free GPU
runners on GHA).
- New `RELEASING.md` documents the binary-release flow and
the manual GPU smoke checklist that gates each binary
release. New `xt/05-gpu-smoke.rakutest` runs that smoke
via prove6 (gated on `ONNX_NATIVE_TEST_GPU=1`).
- Honest CUDA section in `docs/Readme.rakudoc` (was
promising "Picks the GPU variant" before any of the
plumbing existed). New "Windows — DirectML" section
explaining why DML is deferred. `:cuda-device-id` /
`:coreml-flags` documented under `Session.new`.
0.1.1 2026-04-24T05:59:58+01:00
- Add windows-arm64 to the build-binaries.yml matrix.
Microsoft publishes onnxruntime-win-arm64-1.20.0.zip, and
the CRoaring / Tokenizers convention is to build for every
platform upstream supports. Uses the `windows-11-arm`
native ARM runner (falls back to cross-compile from
`windows-2022` with `amd64_arm64` if the runner isn't
accessible to the repo). %PLATFORM-SLUGS /
%UPSTREAM-SLUGS extended with the arm64 mapping.
- Linux shim link: reorder source file before -lonnxruntime.
Ubuntu's GCC defaults to --as-needed, which means a library
specified before any reference to its symbols gets dropped
from DT_NEEDED. With -lonnxruntime first and the source file
last, the link succeeded with undefined references to
OrtGetApiBase, which then failed at first dlopen. Sources
first is the universally-correct Unix linker order; this
just makes us correct.
- Linux shim link: switch from -Wl,--enable-new-dtags
(DT_RUNPATH) to -Wl,--disable-new-dtags (DT_RPATH). DT_RPATH
is searched BEFORE LD_LIBRARY_PATH by the Linux loader, so
a user with an older libonnxruntime in their LD_LIBRARY_PATH
(conda, vcpkg, distro package, IDE integration) can no
longer hijack our staged one. Shim is ABI-pinned against a
specific ORT_API_VERSION and must load the matching
libonnxruntime. macOS is already immune (@rpath/@loader_path
names bypass DYLD_LIBRARY_PATH) and Windows is fixed via
SetDllDirectoryW. Users who genuinely want to override
should use the explicit ONNX_NATIVE_LIB_DIR env var.
- New .github/workflows/glibc-fallback.yml: runs `zef install .`
inside an ubuntu:20.04 container (glibc 2.31, below MIN-GLIBC
2.35) and asserts (a) BINARY_ONLY=1 fails cleanly naming
glibc, and (b) the default path shows the fallback message
and attempts the system-libonnxruntime path. Matches
CRoaring's glibc-fallback.yml structure.
- MIN-GLIBC bumped from 2.31 to 2.35 (ubuntu-22.04 baseline)
to match CRoaring / Tokenizers convention: if a user's host
glibc is below the CI build-host's glibc, trust the
fallback path rather than a prebuilt that *might* work.
ubuntu:18.04 can't run actions/checkout (Node 20 requires
glibc 2.28+) so the old plan of testing at 2.27 wasn't
viable anyway.
- Windows DLL search: FFI.rakumod now steers the Win32
loader to our staged onnxruntime.dll via PATH prepend +
SetDllDirectoryW (kernel32) before the shim's LoadLibrary
resolves. Without this, Windows resolves the shim's
import-lib'd `onnxruntime.dll` using the process-exe's
directory (raku.exe), which on GitHub's windows-latest
runner picks up a 1.17.1 DLL installed system-wide — our
shim is compiled against ORT_API_VERSION=20, that older
DLL only supports versions 1 and 17, and every call died
with "The requested API version [20] is not available".
Matches Vips-Native's existing fix for the same class of
Windows loader problem.
- Windows shim: add ONNX_SHIM_EXPORT macro (__declspec(dllexport)
on Windows, visibility("default") elsewhere) on every shim
function definition. MSVC doesn't export DLL symbols by
default, so without this the shim loaded on Windows but every
`is native` binding failed with "Cannot locate symbol".
- Linux shim: compile with -Wl,--no-undefined and
-Wl,--enable-new-dtags. GCC's default for -shared silently
allows unresolved references, so a missing -lonnxruntime
produced a shim that only failed at first dlopen with
"undefined symbol: OrtGetApiBase". --no-undefined turns that
into a loud link-time error; --enable-new-dtags emits
RUNPATH (overridable via LD_LIBRARY_PATH) rather than legacy
RPATH.
- macOS shim: explicit -Wl,-undefined,error for
defense-in-depth (already the default but pinning it
prevents future regressions via env-level flags).
- Build.rakumod now runs platform-native linkage diagnostics
after a successful shim compile (`ldd` + `readelf -d` on
Linux, `otool -L` + `nm -g` on macOS, `dumpbin /dependents`
+ `/exports` on Windows) so CI logs show the NEEDED /
RUNPATH / exports directly.
0.1.0 2026-04-24T04:58:47+01:00
- Initial release.
- Minimal Raku NativeCall wrapper around ONNX Runtime's C API,
scoped to the inference surface SillyTavern's expressions
classifier exercises: env / session lifecycle, input/output
introspection, FLOAT32 / INT32 / INT64 tensor I/O, Run.
- C shim (src/onnx_native_shim.c, ~22 exported symbols) caches
the OrtApi* on first use and exposes flat extern-C wrappers,
avoiding per-call function-pointer dispatch on the Raku side.
- Class-based public API: `ONNX::Native::Session`,
`ONNX::Native::Tensor`, `ONNX::Native::TensorInfo`, enum
`DType`, enum `Provider`, exceptions under
`X::ONNX::Native::*`.
- Tensor inputs from Raku Blob (zero-copy via internal anchor),
flat Num / Int lists, or pre-packed bytes. Output tensors
decode to Num / Int lists or Blobs.
- Execution providers: CPU (all platforms) + CoreML baked into
Microsoft's macOS prebuilts. CUDA opt-in on Linux via
ONNX_NATIVE_WITH_CUDA=1. DirectML deferred.
- Build.rakumod downloads a per-platform tarball from this
repo's GitHub Release (SHA256-verified against bundled
resources/checksums.txt), stages it under
$XDG_DATA_HOME/ONNX-Native//. Auto-fallbacks to
Microsoft's upstream ORT release (plus local shim compile)
when our Release has no artefact yet, and to system
libonnxruntime (brew / apt -dev package) when both fail.
- Windows path handling in the shim: UTF-8 char* at the Raku
boundary, widened via MultiByteToWideChar before calling
ORT's wchar_t-taking CreateSession. MSVC `cl` invocation
wired up in Build.rakumod's local-compile fallback path.
- Env knobs: ONNX_NATIVE_LIB_DIR, ONNX_NATIVE_BINARY_URL,
ONNX_NATIVE_BINARY_ONLY, ONNX_NATIVE_PREFER_SYSTEM,
ONNX_NATIVE_WITH_CUDA, ONNX_NATIVE_DATA_DIR,
ONNX_NATIVE_CACHE_DIR.
- New CI workflows: .github/workflows/build-binaries.yml
publishes prebuilt per-platform tarballs (macOS arm64/x86_64,
Linux x64/aarch64 glibc, Windows x64) on manual dispatch or
binaries-* tag push. .github/workflows/test.yml runs
`zef install .` + prove6 on Ubuntu / macOS / Windows for
every push and PR.
- Tests: t/01-ffi (shim smoke), t/02-session (public Session
API + error paths), t/03-inference (end-to-end round-trip
against a committed 124-byte add.onnx fixture). Fixture is
produced by a pure-Raku protobuf emitter
(t/fixtures/generate-add-onnx.raku) so regeneration has no
Python dependency.