Raku Land - TIKTOKEN

tiktoken

ZEF

Tokenizers
6 days ago
3

Raku bindings for HuggingFace Tokenizers and tiktoken

ai llm tiktoken tokenizers

1⁠–1 of 1

Tags

abnf ² acme ⁴ actions ² activerecord ³ admin ³ aes ² agent ² ai ²⁷ algorithm ⁴ ansi ¹¹ apache ² api ⁵¹ app ⁷ ar ² archive ⁹ arguments ² array ¹¹ art ¹¹ ascii ² ast ² async ²² audio ²⁵ auth ³ authoring ² automation ⁴ aws ⁶ base ⁴ base64 ⁴ bdd ⁴ bench ⁴ bidimensional ² binary ⁴ binomial ³ bio ² bitcoin ³ blake ² blob ² blockchain ³ blowfish ² bnf ³ bot ⁴ browser ³ bson ³ build ³ builder ³ bulgarian ³ c ⁶ cache ⁵ cairo ⁵ camellia ² case5 ² cclog ³ censorship ⁵ chacha20 ² channel ² chart ⁶ charts ² chatgpt ⁷ chemistry ³ chess ³ chinese ⁴ cipher ² classification ³ classifier ³ cldr ³ cli ⁸⁷ client ¹⁹ cmac ² color ¹⁸ colour ³ columns ² compile ² compress ¹³ compunit ⁵ concurrency ³ concurrent ¹⁰ config ¹⁷ constants ³ content ² conversion ² convert ⁴ country ³ coverage ⁵ cpan5 ⁸⁹ cpu ² cro ⁸ cron ³ crypt ⁷ crypto ⁸ css ⁹ csv ⁶ data ³⁸ dataframe ⁶ dataset ¹¹ dataslice ⁶ datetime ⁴⁷ db ³⁸ dbdish ² debug ⁸ decoding ² dependency ² des ² des3 ² desktop ⁵ dev ² devel ²⁵ device ² devtools ² diff ⁹ digest ⁴ dist ⁷ dns ⁶ doc ¹⁵ docker ⁵ dsl ⁵ dump ⁵ ec2 ³ ecosystem ¹³ emacs ² email ⁷ encode ⁴ encoding ⁵ endian ² engine ² english ⁴ entity ¹⁰ env ² error ³ escape ³ evaluation ³ event ⁵ examples ² exceptions ² factorial ² fastcgi ⁵ fcgi ⁴ ffi ² fft ³ file ²⁴ files ³ filter ² finance ² find ⁴ form ² format ⁶ formatting ⁶ fortune ² framework ⁴ french ² fun ⁶ functional ⁵ fuzz ² fuzzy ² game ¹³ gamma ³ gcrypt ² gdk3 ³ gemini ⁴ generate ¹³ generative ¹⁵ geo ¹¹ geometry ⁴ german ² getopt ⁴ gio ⁴ git ¹⁵ glib ⁵ gmac ² gnome ¹⁹ gobject ⁴ google ³ gost28147 ² grammar ²⁸ graphics ¹² gtk ² gui ¹² hash ²⁴ heap ² help ² helpers ² hex ³ highlight ³ hmac ² html ³³ http ³⁶ i18n ¹⁴ idea ² image ²³ immutable ⁵ import ² index ³ inference ² inline ² input ³ install ³ interactive ⁵ international ⁵ interpreter ³⁰ introspect ⁶ io ¹⁵ ip ³ irc ¹¹ it ² japanese ³ javascript ⁸ job ³ json ³⁹ kind ⁴ klingon ² l10n ²⁰ language ¹⁷ latex ² layout ⁵ lazy ² length ² levenshtein ³ lexer ² libgcrypt ² linux ⁷ list ⁵ llm ³⁴ locale ⁷ log ³¹ mac ⁴ macos ⁴ mail ² main ⁵ man ⁵ map ⁵ mapping ⁵ markdown ¹⁶ math ⁸⁶ matrix ⁷ maybe ³ mcp ⁹ md4 ³ measure ⁶ media ² messagepack ³ messaging ⁴ meta ¹³ metadata ⁹ metamodel ² midi ³ ml ⁸ moarvm ⁹ mod ² model ¹¹ moderation ³ module ¹⁷ monad ¹⁶ monitor ² month ³ name ² names ³ native ⁹⁷ naughty ¹⁵ navigation ³ ndjson ² net ²³ nlp ¹³ nonce ² notification ⁶ now ³ number ⁶ numbers ² numeric ³ oas ⁴ obfuscate ² olson ⁴ onnx ³ oop ³⁵ openai ⁷ openapi ⁴ openrouter ² operator ⁵ output ³ pack ³ package ³ packaging ³ pandas ³ parallel ² parse ⁶⁵ partition ² passphrase ² passwd ² password ⁸ pastebin ³ path ⁷ paths ³ pdf ²¹ performance ⁶ perl ⁴⁴ permissions ² persistent ² physics ¹⁴ pipeline ¹⁵ placeholder ² playlist ² plot ⁶ plugin ³ pod ²⁸ pointer ² portuguese ³ posix ³ preferences ³ prime ² primes ³ print ³ priority ² proc ³ progress ² promise ² prompt ⁵ proxy ² psgi ³ python ⁵ query ⁷ queue ⁷ radix ² raku ¹⁸ rakuast ⁴ rakudo ⁴ rakudoc ⁵ random ¹⁰ raspberrypi ⁴ rc4 ² readline ⁴ redis ² regex ⁵ regression ³ relationship ⁵ render ⁵ repl ⁵ repository ⁹ result ² rfc ² rgb ³ ripemd160 ² roff ³ root ² router ² routing ⁵ ruby ² run ³ russian ⁴ salsa20 ² scheduler ² schema ⁴ science ² script ⁴ scrypt ³ search ¹³ secp256k1 ³ security ⁴ seed ² semantic ⁵ sequence ³ sequencer ² serialisation ⁵ serialize ¹⁰ series ⁶ serpent ² server ¹² set ² sha ² sha1 ⁵ sha256 ² sha3 ² sha384 ² shake ² simdjson ² size ² slang ¹⁹ sleep ³ smtp ² sockets ² sort ⁵ span ⁴ spec ³ spreadsheet ⁵ sqlcipher ² state ³ static ³ stats ¹¹ stderr ⁴ stdout ⁴ stem ⁵ stemmer ³ stemming ³ stream ⁴ strict ³ strings ⁴ struct ² supply ² swagger ⁴ syntax ² system ¹⁰ table ⁹ tag ⁵ tempfile ² template ²¹ term ¹⁰ termbox ² test ⁴⁴ text ⁷¹ textual ² tiger ² timeline ² timer ³ timezone ² timezones ³ tls ⁷ todo ² token ² toml ³ tool ⁸ toolchain ⁴ tools ⁹ transcription ⁵ transformer ² translation ⁷ tree ⁷ trie ⁷ tui ⁶ twofish ² type ¹¹ types ² typos ² ui ² uid ² ukrainian ³ units ⁸ unix ¹⁴ unpack ² url ⁸ us ² use ² user ⁶ useragent ² util ¹²⁷ uuencode ² validation ⁶ vector ³ video ³ web ⁷² whitespace ² worker ² workflow ¹⁸ wrapper ³ xml ²³ yaml ⁷ zef ³ zip ²

Email • Git • IRC • Status Built 4 mins ago