map/symbol.src:12 says "Bare 1-9 choose from a prefix search; keep digit
aliases in matching slots", map/README:55 repeats it, and both
tests/engine_test.c:1001 and tests/mkemoji_test.py were written to it.
None of it happened. dictprefix ends in below(), which returns a node's
own words first and then its children in rune order (dict.c:29-39), and
under `^` the punctuation sorts ahead of the digits:
^ gave ⁽ ⁾ ⁺ ⁻ ⁰ ¹ ² ³ ⁴ ⁵ ...
_ gave ₍ ₎ ₊ ₋ ₀ ₁ ₂ ₃ ₄ ₅ ...
< gave ← ♥ 🫰🫶 ≤ ≠
so the file's own aliases picked the wrong character every time:
before after
^ then 1 ⁽ ¹
^ then 2 ⁾ ²
_ then 1 ₍ ₁
< then 3 🫰 ♥
mkemoji has no bare `^` row to emit, because no source row claims `^` as
an alias -- the prefix exists in the trie only as the parent of `^1`..`^9`
and `^(`..`^n`, and a parent has no words of its own. Giving the nine
superscripts `^` as a second alias, the nine subscripts `_`, and ←≤♥≠ `<`,
makes build() group them in source order and emit three rows:
< ← ≤ ♥ ≠
^ ¹ ² ³ ⁴ ⁵ ⁶ ⁷ ⁸ ⁹
_ ₁ ₂ ₃ ₄ ₅ ₆ ₇ ₈ ₉
which is what engine_test.c:1001's fixture has said all along, and the
first thing below() now returns. Nothing is lost: `^0`, `^(`, `^i` and
their kind still answer their own key, `0` is not a selection key so it
still extends the query, and addkouho drops the duplicate when a child
repeats what the parent already offered. Three rows on 18953.
tests/mkemoji_test.py was asking the wrong question. It checked that the
first nine `^` keys in the file are `^1`..`^9` -- true before and after,
and decided nothing, because the file's order is not the trie's. It now
checks the row the engine actually reads, and fails without this change
with a bare KeyError on `^`.
What this does not cover: `+1` and `-1`. They are not slot aliases, they
are words -- plus one, minus one -- and ➕ and ➖ are the right first
answers to `+` and `-`. 👍 is slot 3 of `+` and reachable there.
Every Korean keyboard's 한자 key answers a lone consonant with the KS X
1001 symbol palette, and has since 한글 워드프로세서: ㅁ for ※ ○ △ ㈜, ㄴ
for the brackets, ㄹ for the units, ㅇ for the circled numbers. strans
sends that key to the same search as a syllable -- Khanja is Ctrl+H at
strans.c:830, and startsearch seeds the query with whatever ko.c left
pending -- but every one of hanja.dict's 187286 readings is a syllable, so
the popup came up with a query in it and nothing to pick:
ㅁ: 0 candidates
ㄴ: 0 candidates
ㄹ: 0 candidates
한: 99 candidates 韓 漢 寒 限 閑 恨 旱 汗 翰 邯 罕 悍 澣 閒 瀚
libhangul ships that palette beside the Hanja table already imported here:
data/hanja/mssymbol.txt, same commit, same author, same BSD-3 terms, same
key:value:comment format -- and keyed by the compatibility jamo ko.c
already holds, U+3141 for ㅁ. So the engine does not change at all; the
same dictlookup on the same trie now finds something:
ㅁ: 75 candidates ＃ ＆ ＊ ＠ § ※ ☆ ★ ○ ● ◎ ◇ ◆ □ ■ △ ▲ ▽ ▼
ㄴ: 23 candidates ＂ （ ） ［ ］ ｛ ｝ ‘ ’ “ ” 〔 〕 〈 〉 《 》 「 」
ㄹ: 94 candidates ＄ ％ ￦ Ｆ ′ ″ ℃ Å ￠ ￡ ￥ ¤ ℉ ‰ € ㎕ ㎖ ㎗ ℓ
한: 99 candidates 韓 漢 寒 限 閑 恨 旱 汗 翰 邯 罕 悍 澣 閒 瀚
Both scripts widen by one rule -- a syllable reading gives Hanja, a jamo
reading gives a symbol -- and hanja.src regenerates byte for byte as it
was, because upstream's own non-syllable readings are words like ㄱ자집
whose values were never Hanja and still fall out. 985 of mssymbol.txt's
987 rows survive: its ideographic space and its soft hyphen do not, since
a candidate the popup cannot draw is not a candidate, and the row format
separates candidates with a space besides.
The two keyspaces cannot collide -- one is syllables, one is single jamo --
so the 187286 existing rows are unchanged, byte for byte, and 18 rows join
them. mkhanja takes a source list as mkemoji already does, and keeps each
upstream header, which is why the licence text now appears twice.
89 unit, check-live, check-stress and valgrind all clean. The five new
assertions were checked by breaking the change five ways: dropping
mssymbol.src from SOURCES, letting issymbol keep a formatting character,
letting a jamo reading keep Hanja, widening isjamo to the vowels, and
making mkhanja reject jamo readings. Each fails only the tests that exist
for it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The import kept only readings of one syllable, so the Hanja search could
convert 한 but never 한자, 학교, or 대한민국 — the conversion every other
Korean input method offers. libhangul's table has 187k readings; both
scripts now keep them all, and the search finds a word as readily as a
syllable. The daemon pays for it: 24 MB instead of 12, and 170 ms to
start instead of 30.
Δ Γ Λ Ω Φ Ψ Σ Θ answered only to De, Ga, La, Om, Ph, Ps, Si, Th, while
their small letters answered to delta, gamma and the rest; they answer to
the names too now. mkemoji and cldr2emoji write UTF-8 whatever the
locale, as the other generators already did; verify-map calls python3 the
one way; the tests' include path drops a directory nothing includes
through; and bench.sh says which binary is missing instead of blaming the
daemon ten seconds later.
An SKK dictionary's ";;" header carries its license notice, which
LICENSES/README.md relies on for kanji.dict, and skk2ktrans dropped
every ";;" line: a re-import as map/README describes lost the grant.
The leading comment block is kept now. README's regenerate-and-verify
recipe omitted mktelex.py although verify-map checks its output; and
the Dockerfile set a shell variable named egrep in plan9's config that
9c never reads, calling egrep by name.
II, PP, SS, sq, mul, vv, xx, oo, dn, inf, and deg were the only way to
∫ ∏ ∑ √ × ✓ ✗ ● ↓ ∞ °; integral, prod, sum, sqrt, times, check, cross,
circle, down, infinity, and degree find them as well.
A query typed in the Hiragana mode is hiragana; CLDR's Japanese
keywords are mostly katakana (スマイル, ハート), so the one could not
find the other. mkemoji now writes each alias that has kana in both
scripts.
emoji.src was a seed of thirty emoji. cldr2emoji now generates it from
Unicode's emoji-test.txt (Emoji 17.0) and CLDR 48.2.0's annotations: the
1914 fully-qualified emoji without their skin-tone variants, each with
its names and keywords in the three languages, so that a search finds
what fcitx5's emoji picker finds. The data is under the Unicode License,
added to LICENSES.
mkemoji wrote a row for every prefix of every alias, so that a query
matched as it was typed; the trie is a prefix index already, and with a
real emoji list those rows would be four times the aliases themselves.
Now emoji.dict has one row per alias, trienode() names a key's node, and
dictlookup walks the entries at and below it, the key's own first, up to
Maxkouho. Trie children are appended rather than pushed, so the walk
keeps the file's order and a bare digit still picks the superscript or
subscript it always did.
The hand-written symbol rows move to symbol.src; emoji.src is left to
the emoji. mkemoji reads both by default, or the files it is given.
hira.map and kata.map knew 152 keys of the 320 every Japanese IME
accepts: no nn, no small kana by x or l (xtu, xya, la, ...), no sya, tya,
zya, jya, dya, cya, tsa, thi, dhi, twu, kye, ye, fya, wha, and zi gave ぢ.
Both are now Mozc's default table in full, kata.map in Katakana, so that
they agree (kata.map alone had di → ディ and a lone v). With Mozc, wi
and we are うぃ and うぇ (wyi and wye the old kana), the v row is ゔ, nn
is ん, and [ ] ~ and the z-prefixed symbols type 「 」 〜 ・ … ← ↓ ↑ →.
n' is a row now, not a case in transjp. A row whose value is っ keeps
its consonant pending only when it is a doubled consonant or tch: xtu
and ltu are っ itself, so a lone っ can finally be typed.
32 hid the tail of common readings: きょう has 41 kanji, 구 has 352
hanja. 128 covers every kanji entry and the hanja dictionary now keeps
that many per reading.
Hiragana spelled out 42 doubled syllables (kka, ssha, ttsu, ...) and so
knew none it did not list: matcha, baggu and beddo came out as ma-tc-ha,
ba-gg-u. Katakana already used the engine's small-tsu rule for a few
doubles. Both maps now list every doubled consonant but n, plus tch, and
hiragana gains di, che and fa/fi/fe/fo like katakana.
The map is the syllable grammar: a key sequence stays pending only
while it is a prefix of some entry, so a rime the table lacked split
the syllable and a trailing tone key landed on the wrong letter (vowis
gave vơí, rooif rôì, thaays thâý), ua+tone was a valueless prefix that
committed the raw keys (cuar gave cuar, muaf muaf), uyê and ươ typed as
uwow took no coda (nguyễn, được), and ưa toned the a (cửa gave cưả).
Add âu ây êu ôi ơi eo ia ưi ưu uê ươu uây oeo uya uyu, the uyê, uwow,
uê and oe codas, and tone ưa on ư.
transvi fixes:
1. qu-/gi- onset tone placement. The u after q, and the i after g when a
vowel follows, are onset glides rather than the rime nucleus, so the
tone must skip them: qua -> quá (was qúa), gia -> giá. The onset was
previously passed straight through to the app, so transvi never saw it
and toned the glide. Keep the onset in the preedit by adding qu-/gi-
clusters to telex.map (mktelex.py onsets(), appended additively to the
curated map), and add onsetglide() so transvi skips the glide. gi- with
no following vowel keeps i as the nucleus (gì, gìn).
2. A tone key on a vowel-less preedit (e.g. "đ" from dd) now commits the
preedit and lets the tone key pass through (eat=0), matching the engine
commit-on-passthrough invariant, instead of eating it into the commit.
Verified against the running engine: qua/quan/quay/quê/quên/quyển,
gia/già/giàu/giữ/giúp/giống, gì/gìn, dd+s; unchanged mua->mùa, của, lúa;
all non-qu/gi words byte-identical to before.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
