data(hanja): a lone consonant is a reading too
Every Korean keyboard's 한자 key answers a lone consonant with the KS X 1001 symbol palette, and has since 한글 워드프로세서: ㅁ for ※ ○ △ ㈜, ㄴ for the brackets, ㄹ for the units, ㅇ for the circled numbers. strans sends that key to the same search as a syllable -- Khanja is Ctrl+H at strans.c:830, and startsearch seeds the query with whatever ko.c left pending -- but every one of hanja.dict's 187286 readings is a syllable, so the popup came up with a query in it and nothing to pick: ㅁ: 0 candidates ㄴ: 0 candidates ㄹ: 0 candidates 한: 99 candidates 韓 漢 寒 限 閑 恨 旱 汗 翰 邯 罕 悍 澣 閒 瀚 libhangul ships that palette beside the Hanja table already imported here: data/hanja/mssymbol.txt, same commit, same author, same BSD-3 terms, same key:value:comment format -- and keyed by the compatibility jamo ko.c already holds, U+3141 for ㅁ. So the engine does not change at all; the same dictlookup on the same trie now finds something: ㅁ: 75 candidates # & * @ § ※ ☆ ★ ○ ● ◎ ◇ ◆ □ ■ △ ▲ ▽ ▼ ㄴ: 23 candidates " ( ) [ ] { } ‘ ’ “ ” 〔 〕 〈 〉 《 》 「 」 ㄹ: 94 candidates $ % ₩ F ′ ″ ℃ Å ¢ £ ¥ ¤ ℉ ‰ € ㎕ ㎖ ㎗ ℓ 한: 99 candidates 韓 漢 寒 限 閑 恨 旱 汗 翰 邯 罕 悍 澣 閒 瀚 Both scripts widen by one rule -- a syllable reading gives Hanja, a jamo reading gives a symbol -- and hanja.src regenerates byte for byte as it was, because upstream's own non-syllable readings are words like ㄱ자집 whose values were never Hanja and still fall out. 985 of mssymbol.txt's 987 rows survive: its ideographic space and its soft hyphen do not, since a candidate the popup cannot draw is not a candidate, and the row format separates candidates with a space besides. The two keyspaces cannot collide -- one is syllables, one is single jamo -- so the 187286 existing rows are unchanged, byte for byte, and 18 rows join them. mkhanja takes a source list as mkemoji already does, and keeps each upstream header, which is why the licence text now appears twice. 89 unit, check-live, check-stress and valgrind all clean. The five new assertions were checked by breaking the change five ways: dropping mssymbol.src from SOURCES, letting issymbol keep a formatting character, letting a jamo reading keep Hanja, widening isjamo to the vowels, and making mkhanja reject jamo readings. Each fails only the tests that exist for it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
49
map/README
49
map/README
@@ -1,42 +1,53 @@
|
||||
# Dictionary data
|
||||
|
||||
## Korean Hanja data
|
||||
## Korean Hanja and symbol data
|
||||
|
||||
`hanja.src` is the tracked, reviewable source for Korean Hanja conversion.
|
||||
Every data row is Hanja, a tab, and its modern Hangul reading, a character
|
||||
or a word:
|
||||
`hanja.src` and `mssymbol.src` are the tracked, reviewable sources for what
|
||||
the Hanja search converts. Every data row is a result, a tab, and its
|
||||
Hangul reading. A Hanja reads as a syllable or a word; a symbol reads as
|
||||
the lone consonant a Korean keyboard's 한자 key offers it under:
|
||||
|
||||
```
|
||||
漢 한
|
||||
漢字 한자
|
||||
※ ㅁ
|
||||
```
|
||||
|
||||
`mkhanja` validates that contract and groups rows by their reading to
|
||||
produce the runtime dictionary. Candidate order follows source order. The
|
||||
source keeps all retained pairs for review; the generated dictionary stores
|
||||
the first 128 candidates per reading because that is the engine's lookup
|
||||
limit.
|
||||
`mkhanja` validates that contract and groups the rows of both sources by
|
||||
their reading to produce the runtime dictionary. The two keyspaces do not
|
||||
meet: a reading is either syllables or one jamo. Candidate order follows
|
||||
source order. The sources keep all retained pairs for review; the
|
||||
generated dictionary stores the first 128 candidates per reading because
|
||||
that is the engine's lookup limit.
|
||||
|
||||
The source is derived from libhangul release tag `libhangul-0.2.0`. The
|
||||
The sources are derived from libhangul release tag `libhangul-0.2.0`. The
|
||||
annotated tag object is `20afc38922e3595ee3ed5b186f2ea05afe663763`, its
|
||||
peeled commit is `41c702f5d3581325b646ef6249f1f641b0427ae0`, and
|
||||
`data/hanja/hanja.txt` has blob ID
|
||||
`199cfd70c4b306257ac2e018714a1285f7cf0ed3`. Import and regenerate it from
|
||||
the repository root with:
|
||||
`data/hanja/hanja.txt` and `data/hanja/mssymbol.txt` have blob IDs
|
||||
`199cfd70c4b306257ac2e018714a1285f7cf0ed3` and
|
||||
`31c4e63d74293b6a759d4a2005cd1e7333746631`. Import and regenerate them
|
||||
from the repository root with:
|
||||
|
||||
```sh
|
||||
curl -L https://raw.githubusercontent.com/libhangul/libhangul/41c702f5d3581325b646ef6249f1f641b0427ae0/data/hanja/hanja.txt -o upstream
|
||||
printf '%s %s\n' dd44dcc856cf542b1022d0f39c2e9b9f8805fdcc5923be80f04849ed97ce0996 upstream | sha256sum -c -
|
||||
map/libhangul2hanja upstream >map/hanja.src
|
||||
curl -L https://raw.githubusercontent.com/libhangul/libhangul/41c702f5d3581325b646ef6249f1f641b0427ae0/data/hanja/mssymbol.txt -o upstream
|
||||
printf '%s %s\n' b685a4ebe2716b25eb29c42f6da6493716ecd78b604b44a33420987d21948e99 upstream | sha256sum -c -
|
||||
map/libhangul2hanja upstream >map/mssymbol.src
|
||||
map/mkhanja >map/hanja.dict
|
||||
```
|
||||
|
||||
`libhangul2hanja` retains only rows whose reading is modern Hangul syllables
|
||||
throughout and whose value is Hanja the popup can draw: U+3400–U+4DBF,
|
||||
U+4E00–U+9FFF, or U+F900–U+FAFF. Thus jamo readings, mixed values, and
|
||||
supplementary-plane ideographs are omitted. The import preserves the upstream license header and row order.
|
||||
The retained data is BSD 3-Clause licensed by Choe Hwanjin; the complete text
|
||||
is in `LICENSES/BSD-3-Clause-libhangul-hanja.txt`.
|
||||
`libhangul2hanja` keeps a syllable reading only when its value is Hanja the
|
||||
popup can draw: U+3400–U+4DBF, U+4E00–U+9FFF, or U+F900–U+FAFF. Thus mixed
|
||||
values and supplementary-plane ideographs are omitted. It keeps a jamo
|
||||
reading, U+3131–U+314E, only when its value is one rune the popup can draw
|
||||
and a candidate row can carry — so `mssymbol.txt`'s ideographic space and
|
||||
soft hyphen are omitted, leaving 985 of its 987 rows, and a Hanja under a
|
||||
jamo reading is omitted as noise. The import preserves each upstream
|
||||
license header and row order. The retained data is BSD 3-Clause licensed
|
||||
by Choe Hwanjin; the complete text is in
|
||||
`LICENSES/BSD-3-Clause-libhangul-hanja.txt`.
|
||||
|
||||
## Emoji and symbol data
|
||||
|
||||
|
||||
@@ -24,6 +24,32 @@
|
||||
;; CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
|
||||
;; ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
|
||||
;; POSSIBILITY OF SUCH DAMAGE.
|
||||
;; Copyright (c) 2005-2014 Choe Hwanjin
|
||||
;; All rights reserved.
|
||||
;;
|
||||
;; Redistribution and use in source and binary forms, with or without
|
||||
;; modification, are permitted provided that the following conditions are met:
|
||||
;;
|
||||
;; 1. Redistributions of source code must retain the above copyright notice,
|
||||
;; this list of conditions and the following disclaimer.
|
||||
;; 2. Redistributions in binary form must reproduce the above copyright notice,
|
||||
;; this list of conditions and the following disclaimer in the documentation
|
||||
;; and/or other materials provided with the distribution.
|
||||
;; 3. Neither the name of the author nor the names of its contributors
|
||||
;; may be used to endorse or promote products derived from this software
|
||||
;; without specific prior written permission.
|
||||
;;
|
||||
;; THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
|
||||
;; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
||||
;; IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
|
||||
;; ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
|
||||
;; LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
|
||||
;; CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
|
||||
;; SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
|
||||
;; INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
|
||||
;; CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
|
||||
;; ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
|
||||
;; POSSIBILITY OF SUCH DAMAGE.
|
||||
|
||||
가 可 家 加 歌 價 街 假 佳 暇 架 价 賈 伽 柯 迦 軻 嘉 駕 嫁 稼 哥 呵 苛 袈 訶 枷 珂 茄 痂 跏 斝 葭 舸 笳 哿 坷 檟 謌 珈 砢 佉 呿 榎 仮 傢 咖 瘕 耞 宊 椵 叚 価 猳 髂 瘸 鴐 䴥 拁 㤎 榢 毠 溊 牁 豭 槚 诃 贾 轲 镓 驾 鲄 婽 尜 岢 幏 彁 徦 愘 戓 戨 斚 樖 泇 渮 滒 炣 牫 牱 犌 玍 癿 笴 糘 胢 腵 蚵 貑 跒 酠 鉫 鉲 鎵 鎶 魺 鴚 麚 㗎 㚙 㞹 㢦 㤉 㧝 㪃 㪼 㱒 㹢 䂟 䈔 䑝 䔅 䕒 䪪 䯊 䶗 嘏
|
||||
가가 假家 可呵 可嘉 家家 呵呵
|
||||
@@ -187311,3 +187337,21 @@
|
||||
힐책 詰責
|
||||
힐척 詰斥
|
||||
힐항 詰抗
|
||||
ㄱ ! ' , . / : ; ? ^ _ ` |  ̄ 、 。 · ‥ … ¨ 〃 ― ∥ \ ∼ ´ ~ ˇ ˘ ˝ ˚ ˙ ¸ ˛ ¡ ¿ ː
|
||||
ㄲ Æ Ð Ħ IJ Ŀ Ł Ø Œ Þ Ŧ Ŋ æ đ ð ħ ı ij ĸ ŀ ł ø œ ß þ ŧ ŋ ʼn
|
||||
ㄴ " ( ) [ ] { } ‘ ’ “ ” 〔 〕 〈 〉 《 》 「 」 『 』 【 】
|
||||
ㄷ + - < = > ± × ÷ ≠ ≤ ≥ ∞ ∴ ♂ ♀ ∠ ⊥ ⌒ ∂ ∇ ≡ ≒ ≪ ≫ √ ∽ ∝ ∵ ∫ ∬ ∈ ∋ ⊆ ⊇ ⊂ ⊃ ∪ ∩ ∧ ∨ ¬ ⇒ ⇔ ∀ ∃ ∮ ∑ ∏
|
||||
ㄸ ぁ あ ぃ い ぅ う ぇ え ぉ お か が き ぎ く ぐ け げ こ ご さ ざ し じ す ず せ ぜ そ ぞ た だ ち ぢ っ つ づ て で と ど な に ぬ ね の は ば ぱ ひ び ぴ ふ ぶ ぷ へ べ ぺ ほ ぼ ぽ ま み む め も ゃ や ゅ ゆ ょ よ ら り る れ ろ ゎ わ ゐ ゑ を ん
|
||||
ㄹ $ % ₩ F ′ ″ ℃ Å ¢ £ ¥ ¤ ℉ ‰ € ㎕ ㎖ ㎗ ℓ ㎘ ㏄ ㎣ ㎤ ㎥ ㎦ ㎙ ㎚ ㎛ ㎜ ㎝ ㎞ ㎟ ㎠ ㎡ ㎢ ㏊ ㎍ ㎎ ㎏ ㏏ ㎈ ㎉ ㏈ ㎧ ㎨ ㎰ ㎱ ㎲ ㎳ ㎴ ㎵ ㎶ ㎷ ㎸ ㎹ ㎀ ㎁ ㎂ ㎃ ㎄ ㎺ ㎻ ㎼ ㎽ ㎾ ㎿ ㎐ ㎑ ㎒ ㎓ ㎔ Ω ㏀ ㏁ ㎊ ㎋ ㎌ ㏖ ㏅ ㎭ ㎮ ㎯ ㏛ ㎩ ㎪ ㎫ ㎬ ㏝ ㏐ ㏓ ㏃ ㏉ ㏜ ㏆
|
||||
ㅁ # & * @ § ※ ☆ ★ ○ ● ◎ ◇ ◆ □ ■ △ ▲ ▽ ▼ → ← ↑ ↓ ↔ 〓 ◁ ◀ ▷ ▶ ♤ ♠ ♡ ♥ ♧ ♣ ⊙ ◈ ▣ ◐ ◑ ▒ ▤ ▥ ▨ ▧ ▦ ▩ ♨ ☏ ☎ ☜ ☞ ¶ † ‡ ↕ ↗ ↙ ↖ ↘ ♭ ♩ ♪ ♬ ㉿ ㈜ № ㏇ ™ ㏂ ㏘ ℡ ® ª º
|
||||
ㅂ ─ │ ┌ ┐ ┘ └ ├ ┬ ┤ ┴ ┼ ━ ┃ ┏ ┓ ┛ ┗ ┣ ┳ ┫ ┻ ╋ ┠ ┯ ┨ ┷ ┿ ┝ ┰ ┥ ┸ ╂ ┒ ┑ ┚ ┙ ┖ ┕ ┎ ┍ ┞ ┟ ┡ ┢ ┦ ┧ ┩ ┪ ┭ ┮ ┱ ┲ ┵ ┶ ┹ ┺ ┽ ┾ ╀ ╁ ╃ ╄ ╅ ╆ ╇ ╈ ╉ ╊
|
||||
ㅃ ァ ア ィ イ ゥ ウ ェ エ ォ オ カ ガ キ ギ ク グ ケ ゲ コ ゴ サ ザ シ ジ ス ズ セ ゼ ソ ゾ タ ダ チ ヂ ッ ツ ヅ テ デ ト ド ナ ニ ヌ ネ ノ ハ バ パ ヒ ビ ピ フ ブ プ ヘ ベ ペ ホ ボ ポ マ ミ ム メ モ ャ ヤ ュ ユ ョ ヨ ラ リ ル レ ロ ヮ ワ ヰ ヱ ヲ ン ヴ ヵ ヶ
|
||||
ㅅ ㉠ ㉡ ㉢ ㉣ ㉤ ㉥ ㉦ ㉧ ㉨ ㉩ ㉪ ㉫ ㉬ ㉭ ㉮ ㉯ ㉰ ㉱ ㉲ ㉳ ㉴ ㉵ ㉶ ㉷ ㉸ ㉹ ㉺ ㉻ ㈀ ㈁ ㈂ ㈃ ㈄ ㈅ ㈆ ㈇ ㈈ ㈉ ㈊ ㈋ ㈌ ㈍ ㈎ ㈏ ㈐ ㈑ ㈒ ㈓ ㈔ ㈕ ㈖ ㈗ ㈘ ㈙ ㈚ ㈛
|
||||
ㅆ А Б В Г Д Е Ё Ж З И Й К Л М Н О П Р С Т У Ф Х Ц Ч Ш Щ Ъ Ы Ь Э Ю Я а б в г д е ё ж з и й к л м н о п р с т у ф х ц ч ш щ ъ ы ь э ю я
|
||||
ㅇ ⓐ ⓑ ⓒ ⓓ ⓔ ⓕ ⓖ ⓗ ⓘ ⓙ ⓚ ⓛ ⓜ ⓝ ⓞ ⓟ ⓠ ⓡ ⓢ ⓣ ⓤ ⓥ ⓦ ⓧ ⓨ ⓩ ① ② ③ ④ ⑤ ⑥ ⑦ ⑧ ⑨ ⑩ ⑪ ⑫ ⑬ ⑭ ⑮ ⒜ ⒝ ⒞ ⒟ ⒠ ⒡ ⒢ ⒣ ⒤ ⒥ ⒦ ⒧ ⒨ ⒩ ⒪ ⒫ ⒬ ⒭ ⒮ ⒯ ⒰ ⒱ ⒲ ⒳ ⒴ ⒵ ⑴ ⑵ ⑶ ⑷ ⑸ ⑹ ⑺ ⑻ ⑼ ⑽ ⑾ ⑿ ⒀ ⒁ ⒂
|
||||
ㅈ 0 1 2 3 4 5 6 7 8 9 ⅰ ⅱ ⅲ ⅳ ⅴ ⅵ ⅶ ⅷ ⅸ ⅹ Ⅰ Ⅱ Ⅲ Ⅳ Ⅴ Ⅵ Ⅶ Ⅷ Ⅸ Ⅹ
|
||||
ㅊ ½ ⅓ ⅔ ¼ ¾ ⅛ ⅜ ⅝ ⅞ ¹ ² ³ ⁴ ⁿ ₁ ₂ ₃ ₄
|
||||
ㅋ ㄱ ㄲ ㄳ ㄴ ㄵ ㄶ ㄷ ㄸ ㄹ ㄺ ㄻ ㄼ ㄽ ㄾ ㄿ ㅀ ㅁ ㅂ ㅃ ㅄ ㅅ ㅆ ㅇ ㅈ ㅉ ㅊ ㅋ ㅌ ㅍ ㅎ ㅏ ㅐ ㅑ ㅒ ㅓ ㅔ ㅕ ㅖ ㅗ ㅘ ㅙ ㅚ ㅛ ㅜ ㅝ ㅞ ㅟ ㅠ ㅡ ㅢ ㅣ
|
||||
ㅌ ㅥ ㅦ ㅧ ㅨ ㅩ ㅪ ㅫ ㅬ ㅭ ㅮ ㅯ ㅰ ㅱ ㅲ ㅳ ㅴ ㅵ ㅶ ㅷ ㅸ ㅹ ㅺ ㅻ ㅼ ㅽ ㅾ ㅿ ㆀ ㆁ ㆂ ㆃ ㆄ ㆅ ㆆ ㆇ ㆈ ㆉ ㆊ ㆋ ㆌ ㆍ ㆎ
|
||||
ㅍ A B C D E F G H I J K L M N O P Q R S T U V W X Y Z a b c d e f g h i j k l m n o p q r s t u v w x y z
|
||||
ㅎ Α Β Γ Δ Ε Ζ Η Θ Ι Κ Λ Μ Ν Ξ Ο Π Ρ Σ Τ Υ Φ Χ Ψ Ω α β γ δ ε ζ η θ ι κ λ μ ν ξ ο π ρ σ τ υ φ χ ψ ω
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Extract Hanja readings from libhangul data."""
|
||||
"""Extract Hanja and symbol readings from libhangul data."""
|
||||
|
||||
import sys
|
||||
import unicodedata
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
@@ -9,12 +10,31 @@ def ishangul(s):
|
||||
return s != "" and all(0xAC00 <= ord(c) <= 0xD7A3 for c in s)
|
||||
|
||||
|
||||
def isjamo(s):
|
||||
return len(s) == 1 and 0x3131 <= ord(s) <= 0x314E
|
||||
|
||||
|
||||
def ishanja(s):
|
||||
return s != "" and all(0x3400 <= ord(c) <= 0x4DBF
|
||||
or 0x4E00 <= ord(c) <= 0x9FFF
|
||||
or 0xF900 <= ord(c) <= 0xFAFF for c in s)
|
||||
|
||||
|
||||
def issymbol(s):
|
||||
"""One rune the popup can draw and a candidate row can carry: not a
|
||||
space, which the row separates candidates with, and not a formatting
|
||||
character, which would leave a blank candidate to pick."""
|
||||
return (len(s) == 1 and not s.isspace() and not ishanja(s)
|
||||
and not unicodedata.category(s).startswith("C"))
|
||||
|
||||
|
||||
def keeps(reading, value):
|
||||
"""A syllable reading gives Hanja; a jamo reading gives a symbol."""
|
||||
if ishangul(reading):
|
||||
return ishanja(value)
|
||||
return isjamo(reading) and issymbol(value)
|
||||
|
||||
|
||||
def extract(src, name):
|
||||
comments = []
|
||||
entries = []
|
||||
@@ -32,16 +52,16 @@ def extract(src, name):
|
||||
fields = line.split(":")
|
||||
if len(fields) != 3:
|
||||
raise ValueError(f"{name}:{lineno}: need key:value:comment")
|
||||
reading, hanja, _ = fields
|
||||
if not ishangul(reading) or not ishanja(hanja):
|
||||
reading, value, _ = fields
|
||||
if not keeps(reading, value):
|
||||
continue
|
||||
pair = (hanja, reading)
|
||||
pair = (value, reading)
|
||||
if pair in seen:
|
||||
raise ValueError(f"{name}:{lineno}: duplicate Hanja reading")
|
||||
raise ValueError(f"{name}:{lineno}: duplicate reading")
|
||||
seen.add(pair)
|
||||
entries.append(pair)
|
||||
if not entries:
|
||||
raise ValueError(f"{name}: no Hanja readings")
|
||||
raise ValueError(f"{name}: no readings")
|
||||
return comments, entries
|
||||
|
||||
|
||||
@@ -63,8 +83,8 @@ def main():
|
||||
print(line)
|
||||
if comments:
|
||||
print()
|
||||
for hanja, reading in entries:
|
||||
print(f"{hanja}\t{reading}")
|
||||
for value, reading in entries:
|
||||
print(f"{value}\t{reading}")
|
||||
except (OSError, UnicodeError, ValueError) as error:
|
||||
print(error, file=sys.stderr)
|
||||
return 1
|
||||
|
||||
63
map/mkhanja
63
map/mkhanja
@@ -1,11 +1,13 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Write hanja.dict from a Hanja-per-row UTF-8 source."""
|
||||
"""Write hanja.dict from result-per-row UTF-8 sources."""
|
||||
|
||||
import sys
|
||||
import unicodedata
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
SOURCE = Path(__file__).with_name("hanja.src")
|
||||
SOURCES = [Path(__file__).with_name(name)
|
||||
for name in ("hanja.src", "mssymbol.src")]
|
||||
MAXCANDIDATES = 128
|
||||
|
||||
|
||||
@@ -13,17 +15,29 @@ def ishangul(s):
|
||||
return s != "" and all(0xAC00 <= ord(c) <= 0xD7A3 for c in s)
|
||||
|
||||
|
||||
def isjamo(s):
|
||||
return len(s) == 1 and 0x3131 <= ord(s) <= 0x314E
|
||||
|
||||
|
||||
def ishanja(s):
|
||||
return s != "" and all(0x3400 <= ord(c) <= 0x4DBF
|
||||
or 0x4E00 <= ord(c) <= 0x9FFF
|
||||
or 0xF900 <= ord(c) <= 0xFAFF for c in s)
|
||||
|
||||
|
||||
def read(path):
|
||||
def issymbol(s):
|
||||
"""One rune the popup can draw and a candidate row can carry: not a
|
||||
space, which the row separates candidates with, and not a formatting
|
||||
character, which would leave a blank candidate to pick."""
|
||||
return (len(s) == 1 and not s.isspace() and not ishanja(s)
|
||||
and not unicodedata.category(s).startswith("C"))
|
||||
|
||||
|
||||
def read(path, table, seen):
|
||||
"""Adds path's rows to table, keyed by reading, and returns its header."""
|
||||
comments = []
|
||||
table = {}
|
||||
seen = set()
|
||||
leading = True
|
||||
found = False
|
||||
with path.open(encoding="utf-8") as src:
|
||||
for lineno, raw in enumerate(src, 1):
|
||||
line = raw.rstrip("\r\n")
|
||||
@@ -36,31 +50,36 @@ def read(path):
|
||||
leading = False
|
||||
fields = line.split("\t")
|
||||
if len(fields) != 2:
|
||||
raise ValueError(f"{path}:{lineno}: need Hanja<TAB>reading")
|
||||
hanja, reading = fields
|
||||
if not ishanja(hanja):
|
||||
raise ValueError(f"{path}:{lineno}: need BMP Hanja")
|
||||
if not ishangul(reading):
|
||||
raise ValueError(f"{path}:{lineno}: need Hangul syllables")
|
||||
pair = (hanja, reading)
|
||||
raise ValueError(f"{path}:{lineno}: need result<TAB>reading")
|
||||
value, reading = fields
|
||||
if ishangul(reading):
|
||||
if not ishanja(value):
|
||||
raise ValueError(f"{path}:{lineno}: need BMP Hanja")
|
||||
elif isjamo(reading):
|
||||
if not issymbol(value):
|
||||
raise ValueError(f"{path}:{lineno}: need one symbol rune")
|
||||
else:
|
||||
raise ValueError(f"{path}:{lineno}: need a syllable or jamo "
|
||||
"reading")
|
||||
pair = (value, reading)
|
||||
if pair in seen:
|
||||
raise ValueError(f"{path}:{lineno}: duplicate Hanja reading")
|
||||
raise ValueError(f"{path}:{lineno}: duplicate reading")
|
||||
seen.add(pair)
|
||||
table.setdefault(reading, []).append(hanja)
|
||||
if not seen:
|
||||
raise ValueError(f"{path}: no Hanja readings")
|
||||
return comments, table
|
||||
found = True
|
||||
table.setdefault(reading, []).append(value)
|
||||
if not found:
|
||||
raise ValueError(f"{path}: no readings")
|
||||
return comments
|
||||
|
||||
|
||||
def main():
|
||||
sys.stdout.reconfigure(encoding="utf-8")
|
||||
sys.stderr.reconfigure(encoding="utf-8")
|
||||
if len(sys.argv) > 2:
|
||||
print(f"usage: {sys.argv[0]} [hanja.src]", file=sys.stderr)
|
||||
return 2
|
||||
path = Path(sys.argv[1]) if len(sys.argv) == 2 else SOURCE
|
||||
paths = [Path(arg) for arg in sys.argv[1:]] or SOURCES
|
||||
table = {}
|
||||
seen = set()
|
||||
try:
|
||||
comments, table = read(path)
|
||||
comments = [line for path in paths for line in read(path, table, seen)]
|
||||
for line in comments:
|
||||
print(line)
|
||||
if comments:
|
||||
|
||||
1012
map/mssymbol.src
Normal file
1012
map/mssymbol.src
Normal file
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user