Named metrics are intentionally kept separate. RYTT encoded UTF-8 bytes are not model-token counts.
| Corpus | Source chars | RYTT tokens | Source bytes | Encoded bytes | cl100k_base | Exact |
|---|---|---|---|---|---|---|
| prose | 113 | 98 | 113 | 274 | n/a | yes |
| code | 75 | 69 | 75 | 174 | n/a | yes |
| mixed_casing | 48 | 42 | 48 | 117 | n/a | yes |
| punctuation | 50 | 45 | 54 | 99 | n/a | yes |
| whitespace | 58 | 49 | 58 | 123 | n/a | yes |
| unicode_passthrough | 26 | 26 | 50 | 61 | n/a | yes |
| ligature_rich | 65 | 29 | 65 | 73 | n/a | yes |