Generate compact classic NPC mapping from ProjectEQ data

docs/development.md

@@ -18,6 +18,25 @@ cargo test
 cargo build --release
 ```
 
+## Refreshing the PEQ NPC gender index
+
+The checked-in `generated/peq-npc-gender-index.json` is generated data, not a
+runtime database dependency. Regenerate it from an explicit ProjectEQ dump
+with the tool below; it reads either `create_tables_content.sql` directly or
+the `peq-dump/create_tables_content.sql` member of the supplied zip archive.
+`unzip` is available in `shell.nix` for the latter.
+
+```bash
+cargo run --bin generate_peq_npc_index -- \
+  --dump "$HOME/Downloads/peq-1759046415.zip" \
+  --output generated/peq-npc-gender-index.json
+```
+
+The tool prints its input identity, complete conflict/neuter statistics, and
+output size. Run it twice and compare the output hashes before updating the
+checked-in artifact. The source archive and any temporary database files must
+not be committed or included in release archives.
+
 The parser tests cover the observed log formats, including NPC dialogue,
 channel-chat exclusion, zone events, normalization, persistent/random speaker
 voice policies, log truncation, and first-run log discovery.

docs/tasks/eqemu-npc-voice-mapping.md

@@ -0,0 +1,436 @@
+# EQEmu-informed NPC voice mapping
+
+## Outcome
+
+Use the EQEmu/ProjectEQ NPC database to narrow the Kokoro voice pool for a
+known EverQuest NPC. The first implementation uses the NPC's recorded gender;
+race, body type, class, and appearance can refine voice character later.
+
+This feature must remain advisory:
+
+- Mudmouth still owns the final voice assignment.
+- A user's saved or pinned voice always wins.
+- Unknown, neuter, and ambiguous NPCs retain today's full-pool behavior.
+- Existing `speakers.json` files remain valid and are never rewritten merely
+  because the bundled data changes.
+- Mudmouth remains fully usable offline.
+
+The release should contain a compact generated lookup, not the ProjectEQ dump,
+a database server, generated audio, or voice model assets.
+
+## Product decisions
+
+### Separate traits from choices
+
+The built-in EQEmu index contains NPC traits, initially only `masculine` or
+`feminine`. It does **not** prescribe an exact Kokoro voice. Mudmouth randomly
+chooses from the matching voices available on the user's Kokoro server and
+persists that choice as it does today.
+
+An exact voice ID is a preference, not an EQEmu fact. Exact choices belong in:
+
+1. the user's local `speakers.json`; or
+2. an optional, separately downloaded community/Legends mapping payload.
+
+This division keeps generated facts reproducible while preserving subjective
+casting choices.
+
+### Keep the built-in release small
+
+Generate a minified, sorted data artifact containing only:
+
+```text
+(display-zone, normalized NPC name) -> gender
+```
+
+Embed it in the executable with `include_bytes!` and deserialize it once at
+startup. `serde_json` is already a dependency, so a first version should not
+add a compression, database, or perfect-hash dependency. The existing `.zip`
+and `.tar.gz` release formats will compress the repeated zone and name text.
+
+Before merging, measure both the raw executable and final release archive
+before and after the data is embedded. Optimize the encoding only if the
+archive delta is material. A compact binary format is a later optimization,
+not an MVP requirement.
+
+The ProjectEQ dump and the generator's temporary MariaDB directory are build
+inputs and must never enter a release archive.
+
+### Preserve user control
+
+The voice manager remains the correction mechanism for incorrect database
+traits, unavailable community voices, and personal taste:
+
+- **Pin** selects an exact voice and is always authoritative.
+- **Reroll persistent random** chooses from the currently resolved pool.
+- **Random every line** chooses from the currently resolved pool for each
+  utterance.
+- An already persisted random voice remains unchanged after an index or
+  mapping-payload update.
+
+The current speaker store is keyed only by normalized NPC name. Do not migrate
+it or silently create zone-qualified speaker records in this feature. Zone is
+used to improve the initial lookup, while the existing stable name-to-voice
+behavior remains intact. Zone-qualified personal overrides can be considered
+separately if playtesting demonstrates a real collision problem.
+
+## Runtime resolution policy
+
+Resolve an NPC in this order:
+
+1. If the speaker already has a persisted exact voice (`Pinned` or
+   `PersistentRandom`), return it unchanged.
+2. If an enabled supplemental mapping has an exact voice for the current
+   `(zone, NPC)` and that voice is in the live curated Kokoro catalog, use and
+   persist it.
+3. If the built-in EQEmu index identifies the current `(zone, NPC)` as
+   feminine, choose from available `af_*` and `bf_*` voices.
+4. If it identifies the NPC as masculine, choose from available `am_*` and
+   `bm_*` voices.
+5. If no zone is known, the lookup is missing, the source records conflict,
+   the NPC is neuter, or the resolved gender pool is empty, choose from the
+   complete available curated pool.
+
+For `RandomEachUtterance`, perform steps 2–5 for every line, but do not turn a
+supplemental recommendation into a local pin. For a manual reroll, perform
+steps 3–5 and persist the new result. A manual pin never gets replaced.
+
+Supplemental mappings are deliberately checked before EQEmu traits so a
+custom Legends NPC can receive a curated exact voice. The initial supplement
+loader should reject or ignore entries already covered by the bundled EQEmu
+index. This makes the first downloadable payload additive rather than a way to
+override known data. A later explicit “community recommendations” mode may
+allow preferences for known NPCs, but it must remain opt-in and below local
+user choices.
+
+## Lookup identity and normalization
+
+Mudmouth knows:
+
+- the speaker name from a `Name says, '…'` log line; and
+- the current display-zone name from `You have entered …`.
+
+The canonical lookup is therefore:
+
+```text
+(current display-zone name, normalized NPC name)
+```
+
+Use the display name exactly as observed in the log. The generator translates
+EQEmu short zone names through `zone.long_name`.
+
+Extract the name-normalization portion of `events::speaker_key` into a shared
+function:
+
+```text
+lowercase
+replace each non-ASCII-alphanumeric run with "_"
+trim leading/trailing "_"
+collapse repeated "_"
+```
+
+`speaker_key` should continue to add the `npc:` prefix for `speakers.json`.
+Both the generator and runtime lookup must call the shared normalization
+function so their behavior cannot drift.
+
+## PEQ dump findings
+
+These figures were measured from the supplied `peq-1759046415.zip` dump. Treat
+the dump as an input to the generator, not application data.
+
+- The required relationship is `spawn2.zone` -> `spawn2.spawngroupID` ->
+  `spawnentry.npcID` -> `npc_types.id`. `spawngroup` is not required.
+- `spawn2.zone` contains a short zone name, while `zone.long_name` contains
+  the display label Mudmouth records.
+- There are 559 distinct display labels for 482 short names. Some labels
+  deliberately combine multiple short zones, such as the seven Muramite
+  Proving Grounds zones. Union all candidates for a display label before
+  deciding whether their genders agree.
+- Gender values in this dump are `0` masculine, `1` feminine, and `2` neuter.
+  Decode them explicitly; the field is not a boolean.
+- After joining the spawn tables, translating zones, and applying Mudmouth
+  normalization, there are 37,262 `(display-zone, speaker)` pairs: 10,046
+  masculine-only, 3,749 feminine-only, 22,692 neuter-only, and 775 with
+  conflicting genders.
+- The v1 gender index therefore has 13,795 eligible entries. Neuter,
+  conflicting, and missing entries deliberately use the full pool.
+- A pair does not need to identify one `npc_types` row. Respawn variants are
+  eligible when every matching row agrees on the same masculine or feminine
+  gender.
+
+The temporary MariaDB data directory used for this measurement was about
+765 MiB. It is suitable for generation, not distribution.
+
+## Generator design
+
+The generator should:
+
+1. Accept an explicit PEQ dump path and output path.
+2. Read raw candidate rows from `npc_types`, `spawnentry`, `spawn2`, and
+   `zone`. It may invoke the `mariadb` client supplied by `shell.nix`; no
+   MariaDB client library should become a Mudmouth runtime dependency.
+3. Normalize NPC names in Rust using the shared runtime function.
+4. Group by `(zone.long_name, normalized_name)` after all short-name rows have
+   been combined.
+5. Emit a row only if every candidate is gender `0` or every candidate is
+   gender `1`.
+6. Decode the database number into a typed `Gender` value rather than carrying
+   magic numbers into runtime code.
+7. Sort records deterministically and write minified output.
+8. Print source identity, accepted/conflicting/neuter counts, and output byte
+   size so refreshes are auditable.
+
+Check the generated artifact into the repository. Runtime and release builds
+must not need MariaDB or the PEQ dump.
+
+The following query is useful for validating results:
+
+```sql
+SELECT
+  z.long_name AS zone,
+  TRIM(BOTH '_' FROM REGEXP_REPLACE(LOWER(n.name), '[^a-z0-9]+', '_'))
+    AS normalized_name,
+  MIN(n.gender) AS gender
+FROM spawn2 AS s
+JOIN spawnentry AS se ON se.spawngroupID = s.spawngroupID
+JOIN npc_types AS n ON n.id = se.npcID
+JOIN zone AS z ON z.short_name = s.zone
+WHERE s.zone <> '' AND n.name <> ''
+GROUP BY z.long_name, normalized_name
+HAVING COUNT(DISTINCT n.gender) = 1 AND MIN(n.gender) IN (0, 1);
+```
+
+Do not use this SQL normalization as the compatibility contract; it is only an
+independent check of the Rust generator.
+
+## Loading the supplied dump
+
+`shell.nix` provides MariaDB and `unzip`. Start `nix-shell`, then use an
+isolated server so the process cannot affect a system MariaDB instance:
+
+```bash
+peq_dir="$(mktemp -d /tmp/mudmouth-peq.XXXXXX)"
+mariadb-install-db --no-defaults --datadir="$peq_dir/data" \
+  --auth-root-authentication-method=normal --skip-test-db
+mariadbd --no-defaults --datadir="$peq_dir/data" \
+  --socket="$peq_dir/mariadb.sock" --pid-file="$peq_dir/mariadbd.pid" \
+  --skip-networking --log-error="$peq_dir/mariadbd.log" &
+
+until mariadb-admin --no-defaults --socket="$peq_dir/mariadb.sock" \
+  -u root ping --silent; do sleep 1; done
+mariadb --no-defaults --socket="$peq_dir/mariadb.sock" -u root \
+  -e 'CREATE DATABASE peq_content DEFAULT CHARACTER SET latin1;'
+unzip -p ~/Downloads/peq-1759046415.zip peq-dump/create_tables_content.sql \
+  | mariadb --no-defaults --socket="$peq_dir/mariadb.sock" -u root peq_content
+```
+
+Only `create_tables_content.sql` is necessary. Shut the isolated server down
+with:
+
+```bash
+mariadb-admin --no-defaults --socket="$peq_dir/mariadb.sock" -u root shutdown
+```
+
+Remove only the known temporary directory after the generator is finished.
+
+## Supplemental Legends/community payload
+
+Support a small, optional data file in Mudmouth's application-data directory,
+separate from `speakers.json`. It contains exact voice recommendations for
+NPCs absent from the bundled EQEmu index:
+
+```json
+{
+  "schema_version": 1,
+  "pack": "eq-legends-community",
+  "game_data_version": "2026-07",
+  "mappings": [
+    {
+      "zone": "Example Zone",
+      "npc": "example_npc",
+      "voice": "bm_george"
+    }
+  ]
+}
+```
+
+Payload rules:
+
+- Store only metadata, normalized identifiers, and Kokoro voice IDs—no audio.
+- Keep it minified for download; compression at transport time is sufficient.
+- Validate `schema_version`, non-empty fields, duplicate keys, and voice IDs.
+- Ignore a recommendation whose voice is unavailable and fall back normally.
+- Ignore entries already known to the bundled index in the first version.
+- A malformed optional payload should produce a visible warning and leave the
+  built-in resolver usable.
+- Loading a new payload must not alter an existing speaker assignment.
+- Do not add automatic networking or updates to the MVP. The user downloads
+  and installs the payload explicitly.
+
+This schema is also the bridge to the future collaborative website. The site
+can collect zone-qualified, selectively shared pins and publish a reviewed,
+versioned supplement without increasing the normal Mudmouth release size.
+Before supporting uploads, add an explicit export preview and consent step;
+never upload `speakers.json` wholesale.
+
+## Agent-sized implementation tasks
+
+Each task should fit in one focused change and leave the branch buildable.
+Tasks 1–4 form the core feature. Tasks 5–7 can proceed after the resolver
+contract in task 3 is stable.
+
+### Task 1 — Shared NPC identity normalization
+
+**Scope:** `src/events.rs` and focused tests.
+
+- Extract `normalize_npc_name(&str) -> String`.
+- Keep `speaker_key` output byte-for-byte compatible by adding `npc:` to the
+  shared result.
+- Add fixtures for punctuation, repeated separators, capitalization, and
+  non-ASCII characters.
+
+**Done when:** all old parser tests pass and generator/runtime callers can use
+the same public normalization function.
+
+### Task 2 — Reproducible PEQ index generator
+
+**Scope:** a non-runtime generator binary or tool, generator documentation,
+and a tiny checked-in fixture dump/export.
+
+- Implement the join-input extraction, grouping, explicit gender decoding,
+  conflict filtering, deterministic sorting, and statistics.
+- Test combined short zones that share one `long_name`, same-gender duplicate
+  NPC rows, conflicts, neuter rows, and malformed input.
+- Generate the full minified index and record its source dump identifier and
+  counts in generated metadata or adjacent documentation.
+
+**Done when:** two runs from the same dump are byte-identical and reproduce
+13,795 accepted rows for the researched dump.
+
+### Task 3 — Built-in index and pool resolver
+
+**Scope:** new `npc_voices` module, generated artifact inclusion, and unit
+tests; no speaker-store behavior changes yet.
+
+- Define typed `Gender` and lookup identity types.
+- Load the embedded index once and expose a pure lookup function.
+- Filter the live curated Kokoro catalog into feminine, masculine, and full
+  pools without duplicating the allowlist.
+- Fall back to the full pool when zone/context/data/pool is missing.
+
+**Done when:** tests cover feminine, masculine, neuter-by-omission,
+conflicting-by-omission, missing-zone, unknown NPC, and an empty gender pool.
+
+### Task 4 — Integrate resolution with speaker persistence and the TUI
+
+**Scope:** `src/speakers.rs`, `src/main.rs`, `src/tui.rs`, and regression
+tests.
+
+- Return saved persistent and pinned voices before consulting new data.
+- Use the resolved pool for first assignment, random-each-utterance, and
+  reroll.
+- Keep legacy `speakers.json` deserialization and keys unchanged.
+- Make the manual voice manager's chosen voice authoritative regardless of
+  the EQEmu gender.
+
+**Done when:** tests prove saved/pinned choices survive refreshes and restarts,
+manual reroll uses the resolved pool, random-each-line uses the resolved pool,
+and legacy JSON loads without migration.
+
+### Task 5 — Optional Legends supplement loader
+
+**Scope:** supplemental schema/parser, application-data loading, resolver
+composition, sample payload, and tests.
+
+- Load the versioned file without adding runtime networking.
+- Validate and deduplicate records.
+- Admit only NPC identities absent from the bundled EQEmu index.
+- Resolve an available exact voice before the EQEmu/full-pool fallback.
+- Report malformed files and unavailable voices without stopping Mudmouth.
+
+**Done when:** an unknown Legends NPC can receive a downloaded exact mapping,
+while known EQEmu NPCs and all existing local assignments remain unchanged.
+
+### Task 6 — Size and packaging guardrail
+
+**Scope:** `scripts/package-release.sh` or a companion reporting script and
+release documentation.
+
+- Confirm the raw PEQ dump, generator inputs, and fixtures are absent from
+  archives.
+- Report generated index size, executable size, and `.zip`/`.tar.gz` sizes.
+- Record the before/after archive delta in the change description.
+- Add a generous regression threshold only after a real baseline exists.
+
+**Done when:** Linux and Windows archive contents are inspected and the
+feature's distribution cost is explicit.
+
+### Task 7 — Playtest handoff
+
+**Scope:** a short manual checklist and optional diagnostic activity messages.
+
+- Show enough non-sensitive resolver information to distinguish saved,
+  supplement, EQEmu-feminine, EQEmu-masculine, and full-pool decisions.
+- Provide known test NPCs/zones from the generated fixture or index.
+- Do not add automated game-driving or audio snapshot tests.
+
+**Done when:** the user can verify the scenarios below during normal play and
+report the NPC, zone, expected class, actual voice, and result.
+
+## Playtest checklist
+
+The user will perform final in-game verification:
+
+1. Enter a zone with a known feminine NPC and confirm the first assignment is
+   an available `af_*` or `bf_*` voice.
+2. Repeat with a known masculine NPC and confirm `am_*` or `bm_*`.
+3. Speak with a neuter or conflicting NPC and confirm it can use the full
+   curated pool.
+4. Speak with an EQ Legends NPC absent from PEQ and confirm normal full-pool
+   fallback without a supplement.
+5. Install a supplement containing that NPC and, before it has a saved local
+   assignment, confirm the recommended voice is used.
+6. Pin a deliberately cross-gender voice, restart Mudmouth, and confirm the
+   pin remains unchanged.
+7. Reroll a known NPC and confirm the new voice stays in its resolved pool.
+8. Enable random-each-line and confirm every selected voice stays in the
+   resolved pool.
+9. Update or replace the index/supplement and confirm previously saved voices
+   do not change.
+10. Visit two zones containing the same display name and note whether the
+    current global speaker identity is confusing enough to justify a future
+    zone-specific override feature.
+
+## Acceptance criteria
+
+- A known `(display zone, normalized NPC name)` with unanimous feminine or
+  masculine PEQ records receives a voice from the matching available pool.
+- Unknown, neuter, conflicting, missing-zone, and empty-subpool cases use the
+  complete available curated pool.
+- Existing persistent voices and pins are never replaced by generated data or
+  supplemental payloads.
+- Pin, reroll, and random-each-line remain available in the voice manager.
+- Legacy `speakers.json` files load without migration.
+- An optional small Legends payload can map NPCs absent from EQEmu without
+  adding audio assets or networking to the main release.
+- The ProjectEQ dump and database tooling are not present in release archives.
+- Automated tests pass, archive size impact is reported, and final behavioral
+  verification is performed through the user's playtest checklist.
+
+## Later roadmap: collaborative voice choices
+
+After the local supplement format has proven stable:
+
+1. Add an export preview for selected pinned mappings only, including zone,
+   normalized NPC name, voice ID, schema version, and game-data version.
+2. Build a website/API that accepts individual submissions, tracks votes and
+   provenance, and moderates invalid zone/NPC/voice combinations.
+3. Generate a reviewed Legends-only supplement from accepted records.
+4. Let users explicitly download/import a versioned payload with checksum and
+   source information.
+5. Consider an opt-in recommendation pack for known EQEmu NPCs. Local pins and
+   existing assignments must remain higher precedence.
+
+Keeping collection, publication, and download outside the executable prevents
+the collaborative roadmap from increasing the normal build size or making
+Mudmouth dependent on the website.