Hojun-Cho/strans
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

data: simplify and validate Japanese imports

Browse Source
This commit is contained in:
Hojun-Cho
2026-08-12 17:32:28 +09:00
parent a8a6c3b455
commit 8d68e00f2b
11 changed files with 93 additions and 189 deletions
Download Patch File Download Diff File Expand all files Collapse all files

21
map/README
View File

@@ -20,23 +20,22 @@ in source order, duplicate candidates were removed, and the empty
From the repository root, fetch a known upstream revision and convert it with:
```
map/grabskkdicts map/skkdicts REVISION
git clone https://github.com/skk-dev/dict.git map/skkdicts
git -C map/skkdicts checkout --detach REVISION
map/skk2ktrans map/skkdicts/SKK-JISYO.M >map/kanji.dict.new
python3 map/verifymap.py map/kanji.dict.new
```
Use a full skk-dev/dict commit ID for `REVISION` and record it when replacing
the bundled data. Omitting `REVISION` intentionally fetches the upstream
default branch and is not reproducible. `grabskkdicts` refuses to overwrite an
existing destination.
the bundled data. Git is needed only to fetch upstream data; it is not part of
the normal build image.
`skk2ktrans` accepts one or more EUC-JP SKK files (or standard input), writes
UTF-8 tab-separated rows, and merges input in command-line and source order.
It strips SKK annotations, deduplicates candidates, and omits Lisp expressions,
numeric conversion entries, bracket forms, and candidates containing
ASCII whitespace because those forms cannot be consumed as literal candidates
by `dict.c`.
It strips annotations, deduplicates candidates, and omits expressions and
candidates containing whitespace. Escaped candidate delimiters are rejected;
rewrite or omit those entries before import.
`verifymap.py` checks UTF-8, row structure, unique keys, the 64-rune key limit,
canonical candidate spacing, and duplicate dictionary candidates. Pass it the
exact `.map` and `.dict` files installed by the build.
`verifymap.py` checks UTF-8, row structure, unique keys, 64-rune keys and
values, canonical candidate spacing, and duplicate dictionary candidates.
Pass it the exact `.map` and `.dict` files installed by the build.