data: validate and repair Japanese dictionaries

map/README

@@ -1,12 +1,42 @@
-#
-# The following are a set of tools to obtain and process dictionaries from the SKK project in order to use them with ktrans(1).
-#
-# grabskkdicts pulls the skk kana-kanji conversion dictionaries from the skk-dev/dict repo.
-# skk2ktrans takes an skk dictionary and converts it into a kanji jisho suitable to be used with ktrans(1).
-#
-# You can fetch and convert all the dictionaries by running this file.
-#
+# Japanese dictionary data
 
-grabskkdicts
-for(d in skkdicts/SKK-JISYO.*)
-	<$d skk2ktrans >$d.jisho
+`kanji.dict` is the historical dictionary bundled with strans.  Its own header
+identifies it as the SKK Medium dictionary, version 8.1 of May 24, 1995,
+rearranged for Plan 9 ktrans by Kenji Okamoto on February 17, 2000.  It was
+already present when this repository was created in commit
+`dcd1147638f908eca7e4b7616815e215022cc99d` on December 23, 2025.  No original
+SKK input file or upstream revision was committed, so the current file cannot
+honestly be regenerated byte-for-byte from repository artifacts.  The license
+notice in its header permits redistribution and modification under GPL version
+2 or later.
+
+The repository copy keeps that historical candidate order.  Duplicate keys
+were merged at their first occurrence, later unseen candidates were appended
+in source order, duplicate candidates were removed, and the empty
+`ようたつ` row was removed.
+
+## Importing a current SKK dictionary
+
+From the repository root, fetch a known upstream revision and convert it with:
+
+```
+map/grabskkdicts map/skkdicts REVISION
+map/skk2ktrans map/skkdicts/SKK-JISYO.M >map/kanji.dict.new
+python3 map/verifymap.py map/kanji.dict.new
+```
+
+Use a full skk-dev/dict commit ID for `REVISION` and record it when replacing
+the bundled data.  Omitting `REVISION` intentionally fetches the upstream
+default branch and is not reproducible.  `grabskkdicts` refuses to overwrite an
+existing destination.
+
+`skk2ktrans` accepts one or more EUC-JP SKK files (or standard input), writes
+UTF-8 tab-separated rows, and merges input in command-line and source order.
+It strips SKK annotations, deduplicates candidates, and omits Lisp expressions,
+numeric conversion entries, bracket forms, and candidates containing
+ASCII whitespace because those forms cannot be consumed as literal candidates
+by `dict.c`.
+
+`verifymap.py` checks UTF-8, row structure, unique keys, the 64-rune key limit,
+canonical candidate spacing, and duplicate dictionary candidates.  Pass it the
+exact `.map` and `.dict` files installed by the build.