Property::Text::Controls

A local Raku library and CLI for reviewing selected invisible Unicode display controls in multilingual property text. It creates a minimal occurrence ledger and checks structural scope pairing within explicit review blocks. An optional annotated view makes selected controls visible without silently deleting them.

Useful when text copied from a brochure, email or listing export looks different from its stored order, contains invisible word breaks, or leaves directional formatting open. It is not a translation checker, normalizer, spell checker or maliciousness detector.

Install

Requires Raku 6.d or newer and zef:

zef install 'Property::Text::Controls:auth<zef:novasia-estate>'
property-text-controls examples.txt
property-text-controls --html examples.txt > review.html
property-text-controls --annotated examples.txt > visible.txt

Exit codes: 0 no selected controls; 1 controls present or scope findings; 2 invalid/unreadable/oversized input. A status-1 result still produces a report. Command-line usage errors follow Raku MAIN's usage handling. JSON is the default. --html and --annotated are mutually exclusive.

Library

use Property::Text::Controls;

my %report = audit-text("Synthetic price: \x[202E]12345\x[202C] THB");
say %report<status>;        # controls_present
say %report<control_count>; # 2
say annotate-text("Reference\x[200B]A-101");
# Reference[U+200B ZERO WIDTH SPACE]A-101

Exports:

• audit-text(Str:D --> Hash:D) — bounded, deterministic occurrence/scope report; no source words or filenames.
• annotate-text(Str:D --> Str:D) — explicit opt-in rendering of selected codepoints. Includes original text in Raku's normalized representation. Keep it private when the input is private.
• report-html(%report --> Str:D) — escaped HTML review report, including safe handling of caller-created report text. No JavaScript or network resources.

Selected set

Directional marks: U+061C, U+200E, U+200F. Embedding/override controls: U+202A–U+202E. Isolates: U+2066–U+2069. Break-related controls: U+00AD, U+200B, U+2060, U+FEFF.

These characters can be intentional and useful. Every occurrence is an observation, not an accusation. ZWJ, ZWNJ, ordinary RTL letters, combining accents, emoji and nonbreaking spaces are outside the selected set. This deliberately avoids flagging every multilingual/emoji string as problematic. Other invisible characters and Unicode confusables are not covered.

Coordinates and normalization

Raku normally processes normalized graphemes (NFG). index is a zero-based Raku grapheme index; line and column are one-based review-block and grapheme positions. They are neither byte offsets, JavaScript UTF-16 offsets nor original codepoint offsets. An e plus combining accent can occupy one grapheme; an emoji sequence can too. CRLF is one grapheme and one boundary.

Review blocks end at LF, CR, CRLF, U+2028 or U+2029. Scope checks reset at every block boundary. This is an explicit editorial review contract, not the complete Unicode Bidirectional Algorithm. The original byte sequence cannot be reconstructed from normalized text; keep the source file separately if exact bytes matter.

Pairing rules

Embeddings/overrides push a format scope. PDF closes only a format at the top of the stack and cannot cross an isolate boundary. Isolates push an isolate scope. PDI closes the nearest isolate together with its nested format scopes. Orphan PDF/PDI produce findings. Scopes left open at a block boundary or EOF produce unclosed_scope findings pointing to the opening position.

The report records maximum observed scope depth but does not simulate Unicode's rendering, overflow counters, paragraph-level direction resolution or visual order. A structurally balanced report never establishes that text will look the same in all applications.

Outcomes and limits

no_selected_controls, controls_present, and scope_issues describe this bounded review only. Input is limited to 1 MiB of UTF-8 text and at most 10,000 selected-control occurrences. The CLI checks file size before decoding. The library checks normalized text's encoded size. The optional annotation function applies the text-size bound; audit applies both bounds. Reports never include text snippets. They reveal positions/counts/control types, which can themselves be sensitive in some contexts.

No input is fetched or sent anywhere. There is no telemetry. The package does not promise XSS prevention, identity protection, complete Unicode security analysis or legal document authentication.

Reproduce checks

From the unpacked distribution:

raku -Ilib t/contracts.rakutest
raku -Ilib bin/property-text-controls examples/plain.txt
raku -Ilib bin/property-text-controls --html examples/controls.txt > review.html

The behavioral suite covers legitimate multilingual text, combining accents, emoji, CRLF, directional scope nesting, mismatched closes, block boundaries, privacy, escaped output and resource bounds. All fixtures are synthetic. The distribution includes docs/report-contract.md, docs/review-workflow.md and CONTRIBUTING.md. The native distribution contains full source and tests; a separate GitHub repository is not required to use it.

Developed by NovAsia Estate for practical work with multilingual property content. MIT licensed. No NovAsia account is needed.