Raku Land - TOKENIZERS

tokenizers

ZEF

Tokenizers
11 hours ago
0

Wrapper module for Huggingface Tokenizers

ai llm tokenizers

1⁠–1 of 1

Tags

acme ⁴ actions ² admin ² aes ² ai ¹⁷ algorithm ⁴ ansi ⁸ apache ² api ⁴⁰ app ⁶ ar ² archive ⁸ arguments ² array ¹⁰ art ⁹ ascii ² async ²⁰ audio ²³ auth ³ authoring ² automation ³ aws ⁶ base ⁴ base64 ³ bdd ² bench ⁴ bidimensional ² binary ⁴ binomial ³ bio ² bitcoin ³ blake ² blob ² blockchain ³ blowfish ² bot ⁴ browser ² build ³ builder ³ bulgarian ³ c ⁵ cache ⁵ cairo ⁵ camellia ² case5 ² cclog ³ censorship ⁵ chacha20 ² chart ⁶ charts ² chatgpt ⁷ chemistry ³ chess ³ cipher ² cldr ³ cli ⁷³ client ¹⁶ cmac ² color ¹⁴ colour ² columns ² compile ² compress ¹² compunit ⁴ concurrent ¹⁰ config ¹⁶ constants ³ conversion ² convert ⁴ country ³ coverage ⁵ cpan5 ⁹¹ cpu ² cro ⁷ cron ³ crypt ⁷ crypto ⁸ css ⁸ csv ⁵ data ³⁰ dataframe ⁶ dataset ¹¹ dataslice ⁶ datetime ⁴⁴ db ³² dbdish ² debug ⁸ decoding ² dependency ² des ² des3 ² desktop ⁵ dev ² devel ²⁴ device ² diff ⁶ digest ⁴ dist ⁷ dns ⁶ doc ¹³ docker ⁵ dsl ⁴ dump ² ec2 ³ ecosystem ¹³ email ⁶ encode ⁴ encoding ³ endian ² engine ² english ⁴ entity ⁸ error ³ escape ³ evaluation ³ exceptions ² factorial ² fastcgi ⁴ fcgi ⁴ fft ³ file ²¹ files ³ find ³ format ⁴ formatting ⁶ fortune ² french ² fun ⁶ functional ⁵ fuzz ² fuzzy ² game ¹² gamma ³ gcrypt ² gdk3 ³ gemini ³ generate ¹¹ generative ¹⁰ geo ¹⁰ geometry ⁴ german ² getopt ⁴ gio ⁴ git ¹⁴ glib ⁵ gmac ² gnome ¹⁸ gobject ⁴ google ³ gost28147 ² grammar ²² graphics ¹¹ gtk ² gui ¹² hash ²⁰ heap ² help ² helpers ² hex ² highlight ³ hmac ² html ²³ http ³³ i18n ¹⁴ idea ² image ¹⁸ immutable ⁵ import ² index ² inline ² input ³ install ³ interactive ⁵ international ⁵ interpreter ²⁴ introspect ⁴ io ¹⁵ ip ³ irc ⁷ japanese ³ javascript ⁸ json ³¹ kind ⁴ klingon ² l10n ¹⁷ language ¹⁸ layout ² lazy ² levenshtein ³ libgcrypt ² linux ⁷ list ⁵ llm ¹¹ locale ⁷ log ²⁷ mac ⁴ macos ³ main ⁵ man ⁵ map ⁴ mapping ⁵ markdown ¹² math ⁸² matrix ⁶ md4 ³ measure ⁶ media ² messagepack ³ messaging ⁴ meta ¹¹ metadata ⁸ metamodel ² midi ³ moarvm ⁸ model ⁸ moderation ³ module ¹⁷ monad ¹² monitor ² month ³ name ³ names ³ native ⁹² naughty ¹³ navigation ³ net ²³ nlp ¹³ nonce ² notification ⁶ number ⁶ numbers ² numeric ³ obfuscate ² olson ⁴ oop ³³ openai ⁷ operator ⁵ output ³ pack ³ package ³ packaging ² pandas ³ parallel ² parse ⁵³ partition ² passphrase ² passwd ² password ⁸ pastebin ³ path ⁶ paths ² pdf ²¹ performance ² perl ⁴⁴ permissions ² physics ¹⁴ pipeline ⁹ placeholder ² playlist ² plot ⁶ plugin ³ pod ²⁵ pointer ² portuguese ³ posix ³ preferences ³ prime ² primes ³ print ² proc ³ promise ² prompt ⁴ proxy ² psgi ² python ⁵ query ⁴ queue ⁵ radix ² raku ¹⁴ rakudo ² rakudoc ⁴ random ⁹ raspberrypi ⁴ rc4 ² readline ⁴ redis ² regex ⁵ regression ³ relationship ⁵ render ² repl ⁵ repository ⁷ rfc ² rgb ³ ripemd160 ² roff ³ root ² router ² run ³ russian ⁴ salsa20 ² schema ⁴ science ² script ⁴ scrypt ³ search ¹³ secp256k1 ³ security ⁴ seed ² sequence ³ sequencer ² serialisation ⁵ serialize ⁹ series ⁶ serpent ² server ⁹ sha ² sha1 ⁴ sha3 ² shake ² simdjson ² size ² slang ¹⁶ sleep ² sockets ² sort ⁵ span ⁴ spreadsheet ⁵ state ³ static ² stats ¹¹ stderr ² stdout ² stem ⁵ stemmer ³ stemming ³ stream ³ strict ³ strings ⁴ struct ² supply ² system ¹⁰ table ⁸ tag ⁵ template ¹⁷ term ⁷ test ³⁸ text ⁶⁷ textual ² tiger ² timeline ² timer ³ timezones ³ tls ⁷ todo ² token ² toml ³ tool ⁶ toolchain ⁴ transcription ⁵ translation ⁷ tree ³ trie ⁷ tui ² twofish ² type ¹⁰ types ² typos ² ukrainian ³ units ⁸ unix ¹⁴ unpack ² url ⁶ us ² use ² user ⁶ util ¹²⁵ uuencode ² validation ² vector ³ video ² web ⁵³ whitespace ² workflow ¹³ wrapper ³ xml ²² yaml ⁷ zef ³ zip ²

Email • Git • IRC • Status Built 4 mins ago