# EQEmu-informed NPC voice mapping ## Outcome Use the EQEmu/ProjectEQ NPC database to narrow the Kokoro voice pool for a known EverQuest NPC. The first implementation uses the NPC's recorded gender; race, body type, class, and appearance can refine voice character later. This feature must remain advisory: - Mudmouth still owns the final voice assignment. - A user's saved or pinned voice always wins. - Unknown, neuter, and ambiguous NPCs retain today's full-pool behavior. - Existing `speakers.json` files remain valid and are never rewritten merely because the bundled data changes. - Mudmouth remains fully usable offline. The release should contain a compact generated lookup, not the ProjectEQ dump, a database server, generated audio, or voice model assets. ## Product decisions ### Separate traits from choices The built-in EQEmu index contains NPC traits, initially only `masculine` or `feminine`. It does **not** prescribe an exact Kokoro voice. Mudmouth randomly chooses from the matching voices available on the user's Kokoro server and persists that choice as it does today. An exact voice ID is a preference, not an EQEmu fact. Exact choices belong in: 1. the user's local `speakers.json`; or 2. an optional, separately downloaded community/Legends mapping payload. This division keeps generated facts reproducible while preserving subjective casting choices. ### Keep the built-in release small Generate a minified, sorted data artifact containing only: ```text (display-zone, normalized NPC name) -> gender ``` Embed it in the executable with `include_bytes!` and deserialize it once at startup. `serde_json` is already a dependency, so a first version should not add a compression, database, or perfect-hash dependency. The existing `.zip` and `.tar.gz` release formats will compress the repeated zone and name text. Before merging, measure both the raw executable and final release archive before and after the data is embedded. Optimize the encoding only if the archive delta is material. A compact binary format is a later optimization, not an MVP requirement. The ProjectEQ dump and the generator's temporary MariaDB directory are build inputs and must never enter a release archive. ### Preserve user control The voice manager remains the correction mechanism for incorrect database traits, unavailable community voices, and personal taste: - **Pin** selects an exact voice and is always authoritative. - **Reroll persistent random** chooses from the currently resolved pool. - **Random every line** chooses from the currently resolved pool for each utterance. - An already persisted random voice remains unchanged after an index or mapping-payload update. The current speaker store is keyed only by normalized NPC name. Do not migrate it or silently create zone-qualified speaker records in this feature. Zone is used to improve the initial lookup, while the existing stable name-to-voice behavior remains intact. Zone-qualified personal overrides can be considered separately if playtesting demonstrates a real collision problem. ## Runtime resolution policy Resolve an NPC in this order: 1. If the speaker already has a persisted exact voice (`Pinned` or `PersistentRandom`), return it unchanged. 2. If an enabled supplemental mapping has an exact voice for the current `(zone, NPC)` and that voice is in the live curated Kokoro catalog, use and persist it. 3. If the built-in EQEmu index identifies the current `(zone, NPC)` as feminine, choose from available `af_*` and `bf_*` voices. 4. If it identifies the NPC as masculine, choose from available `am_*` and `bm_*` voices. 5. If no zone is known, the lookup is missing, the source records conflict, the NPC is neuter, or the resolved gender pool is empty, choose from the complete available curated pool. For `RandomEachUtterance`, perform steps 2–5 for every line, but do not turn a supplemental recommendation into a local pin. For a manual reroll, perform steps 3–5 and persist the new result. A manual pin never gets replaced. Supplemental mappings are deliberately checked before EQEmu traits so a custom Legends NPC can receive a curated exact voice. The initial supplement loader should reject or ignore entries already covered by the bundled EQEmu index. This makes the first downloadable payload additive rather than a way to override known data. A later explicit “community recommendations” mode may allow preferences for known NPCs, but it must remain opt-in and below local user choices. ## Lookup identity and normalization Mudmouth knows: - the speaker name from a `Name says, '…'` log line; and - the current display-zone name from `You have entered …`. The canonical lookup is therefore: ```text (current display-zone name, normalized NPC name) ``` Use the display name exactly as observed in the log. The generator translates EQEmu short zone names through `zone.long_name`. Extract the name-normalization portion of `events::speaker_key` into a shared function: ```text lowercase replace each non-ASCII-alphanumeric run with "_" trim leading/trailing "_" collapse repeated "_" ``` `speaker_key` should continue to add the `npc:` prefix for `speakers.json`. Both the generator and runtime lookup must call the shared normalization function so their behavior cannot drift. ## PEQ dump findings These figures were measured from the supplied `peq-1759046415.zip` dump. Treat the dump as an input to the generator, not application data. - The required relationship is `spawn2.zone` -> `spawn2.spawngroupID` -> `spawnentry.npcID` -> `npc_types.id`. `spawngroup` is not required. - `spawn2.zone` contains a short zone name, while `zone.long_name` contains the display label Mudmouth records. - There are 559 distinct display labels for 482 short names. Some labels deliberately combine multiple short zones, such as the seven Muramite Proving Grounds zones. Union all candidates for a display label before deciding whether their genders agree. - Gender values in this dump are `0` masculine, `1` feminine, and `2` neuter. Decode them explicitly; the field is not a boolean. - After joining the spawn tables, translating zones, and applying Mudmouth normalization, there are 37,262 `(display-zone, speaker)` pairs: 10,046 masculine-only, 3,749 feminine-only, 22,692 neuter-only, and 775 with conflicting genders. - The v1 gender index therefore has 13,795 eligible entries. Neuter, conflicting, and missing entries deliberately use the full pool. - A pair does not need to identify one `npc_types` row. Respawn variants are eligible when every matching row agrees on the same masculine or feminine gender. The temporary MariaDB data directory used for this measurement was about 765 MiB. It is suitable for generation, not distribution. ## Generator design The generator should: 1. Accept an explicit PEQ dump path and output path. 2. Read raw candidate rows from `npc_types`, `spawnentry`, `spawn2`, and `zone`. It may invoke the `mariadb` client supplied by `shell.nix`; no MariaDB client library should become a Mudmouth runtime dependency. 3. Normalize NPC names in Rust using the shared runtime function. 4. Group by `(zone.long_name, normalized_name)` after all short-name rows have been combined. 5. Emit a row only if every candidate is gender `0` or every candidate is gender `1`. 6. Decode the database number into a typed `Gender` value rather than carrying magic numbers into runtime code. 7. Sort records deterministically and write minified output. 8. Print source identity, accepted/conflicting/neuter counts, and output byte size so refreshes are auditable. Check the generated artifact into the repository. Runtime and release builds must not need MariaDB or the PEQ dump. The following query is useful for validating results: ```sql SELECT z.long_name AS zone, TRIM(BOTH '_' FROM REGEXP_REPLACE(LOWER(n.name), '[^a-z0-9]+', '_')) AS normalized_name, MIN(n.gender) AS gender FROM spawn2 AS s JOIN spawnentry AS se ON se.spawngroupID = s.spawngroupID JOIN npc_types AS n ON n.id = se.npcID JOIN zone AS z ON z.short_name = s.zone WHERE s.zone <> '' AND n.name <> '' GROUP BY z.long_name, normalized_name HAVING COUNT(DISTINCT n.gender) = 1 AND MIN(n.gender) IN (0, 1); ``` Do not use this SQL normalization as the compatibility contract; it is only an independent check of the Rust generator. ## Loading the supplied dump `shell.nix` provides MariaDB and `unzip`. Start `nix-shell`, then use an isolated server so the process cannot affect a system MariaDB instance: ```bash peq_dir="$(mktemp -d /tmp/mudmouth-peq.XXXXXX)" mariadb-install-db --no-defaults --datadir="$peq_dir/data" \ --auth-root-authentication-method=normal --skip-test-db mariadbd --no-defaults --datadir="$peq_dir/data" \ --socket="$peq_dir/mariadb.sock" --pid-file="$peq_dir/mariadbd.pid" \ --skip-networking --log-error="$peq_dir/mariadbd.log" & until mariadb-admin --no-defaults --socket="$peq_dir/mariadb.sock" \ -u root ping --silent; do sleep 1; done mariadb --no-defaults --socket="$peq_dir/mariadb.sock" -u root \ -e 'CREATE DATABASE peq_content DEFAULT CHARACTER SET latin1;' unzip -p ~/Downloads/peq-1759046415.zip peq-dump/create_tables_content.sql \ | mariadb --no-defaults --socket="$peq_dir/mariadb.sock" -u root peq_content ``` Only `create_tables_content.sql` is necessary. Shut the isolated server down with: ```bash mariadb-admin --no-defaults --socket="$peq_dir/mariadb.sock" -u root shutdown ``` Remove only the known temporary directory after the generator is finished. ## Supplemental Legends/community payload Support a small, optional data file in Mudmouth's application-data directory, separate from `speakers.json`. It contains exact voice recommendations for NPCs absent from the bundled EQEmu index: ```json { "schema_version": 1, "pack": "eq-legends-community", "game_data_version": "2026-07", "mappings": [ { "zone": "Example Zone", "npc": "example_npc", "voice": "bm_george" } ] } ``` Payload rules: - Store only metadata, normalized identifiers, and Kokoro voice IDs—no audio. - Keep it minified for download; compression at transport time is sufficient. - Validate `schema_version`, non-empty fields, duplicate keys, and voice IDs. - Ignore a recommendation whose voice is unavailable and fall back normally. - Ignore entries already known to the bundled index in the first version. - A malformed optional payload should produce a visible warning and leave the built-in resolver usable. - Loading a new payload must not alter an existing speaker assignment. - Do not add automatic networking or updates to the MVP. The user downloads and installs the payload explicitly. This schema is also the bridge to the future collaborative website. The site can collect zone-qualified, selectively shared pins and publish a reviewed, versioned supplement without increasing the normal Mudmouth release size. Before supporting uploads, add an explicit export preview and consent step; never upload `speakers.json` wholesale. ## Agent-sized implementation tasks Each task should fit in one focused change and leave the branch buildable. Tasks 1–4 form the core feature. Tasks 5–7 can proceed after the resolver contract in task 3 is stable. ### Task 1 — Shared NPC identity normalization **Scope:** `src/events.rs` and focused tests. - Extract `normalize_npc_name(&str) -> String`. - Keep `speaker_key` output byte-for-byte compatible by adding `npc:` to the shared result. - Add fixtures for punctuation, repeated separators, capitalization, and non-ASCII characters. **Done when:** all old parser tests pass and generator/runtime callers can use the same public normalization function. ### Task 2 — Reproducible PEQ index generator **Scope:** a non-runtime generator binary or tool, generator documentation, and a tiny checked-in fixture dump/export. - Implement the join-input extraction, grouping, explicit gender decoding, conflict filtering, deterministic sorting, and statistics. - Test combined short zones that share one `long_name`, same-gender duplicate NPC rows, conflicts, neuter rows, and malformed input. - Generate the full minified index and record its source dump identifier and counts in generated metadata or adjacent documentation. **Done when:** two runs from the same dump are byte-identical and reproduce 13,795 accepted rows for the researched dump. ### Task 3 — Built-in index and pool resolver **Scope:** new `npc_voices` module, generated artifact inclusion, and unit tests; no speaker-store behavior changes yet. - Define typed `Gender` and lookup identity types. - Load the embedded index once and expose a pure lookup function. - Filter the live curated Kokoro catalog into feminine, masculine, and full pools without duplicating the allowlist. - Fall back to the full pool when zone/context/data/pool is missing. **Done when:** tests cover feminine, masculine, neuter-by-omission, conflicting-by-omission, missing-zone, unknown NPC, and an empty gender pool. ### Task 4 — Integrate resolution with speaker persistence and the TUI **Scope:** `src/speakers.rs`, `src/main.rs`, `src/tui.rs`, and regression tests. - Return saved persistent and pinned voices before consulting new data. - Use the resolved pool for first assignment, random-each-utterance, and reroll. - Keep legacy `speakers.json` deserialization and keys unchanged. - Make the manual voice manager's chosen voice authoritative regardless of the EQEmu gender. **Done when:** tests prove saved/pinned choices survive refreshes and restarts, manual reroll uses the resolved pool, random-each-line uses the resolved pool, and legacy JSON loads without migration. ### Task 5 — Optional Legends supplement loader **Scope:** supplemental schema/parser, application-data loading, resolver composition, sample payload, and tests. - Load the versioned file without adding runtime networking. - Validate and deduplicate records. - Admit only NPC identities absent from the bundled EQEmu index. - Resolve an available exact voice before the EQEmu/full-pool fallback. - Report malformed files and unavailable voices without stopping Mudmouth. **Done when:** an unknown Legends NPC can receive a downloaded exact mapping, while known EQEmu NPCs and all existing local assignments remain unchanged. ### Task 6 — Size and packaging guardrail **Scope:** `scripts/package-release.sh` or a companion reporting script and release documentation. - Confirm the raw PEQ dump, generator inputs, and fixtures are absent from archives. - Report generated index size, executable size, and `.zip`/`.tar.gz` sizes. - Record the before/after archive delta in the change description. - Add a generous regression threshold only after a real baseline exists. **Done when:** Linux and Windows archive contents are inspected and the feature's distribution cost is explicit. ### Task 7 — Playtest handoff **Scope:** a short manual checklist and optional diagnostic activity messages. - Show enough non-sensitive resolver information to distinguish saved, supplement, EQEmu-feminine, EQEmu-masculine, and full-pool decisions. - Provide known test NPCs/zones from the generated fixture or index. - Do not add automated game-driving or audio snapshot tests. **Done when:** the user can verify the scenarios below during normal play and report the NPC, zone, expected class, actual voice, and result. ## Playtest checklist The user will perform final in-game verification: 1. Enter a zone with a known feminine NPC and confirm the first assignment is an available `af_*` or `bf_*` voice. 2. Repeat with a known masculine NPC and confirm `am_*` or `bm_*`. 3. Speak with a neuter or conflicting NPC and confirm it can use the full curated pool. 4. Speak with an EQ Legends NPC absent from PEQ and confirm normal full-pool fallback without a supplement. 5. Install a supplement containing that NPC and, before it has a saved local assignment, confirm the recommended voice is used. 6. Pin a deliberately cross-gender voice, restart Mudmouth, and confirm the pin remains unchanged. 7. Reroll a known NPC and confirm the new voice stays in its resolved pool. 8. Enable random-each-line and confirm every selected voice stays in the resolved pool. 9. Update or replace the index/supplement and confirm previously saved voices do not change. 10. Visit two zones containing the same display name and note whether the current global speaker identity is confusing enough to justify a future zone-specific override feature. ## Acceptance criteria - A known `(display zone, normalized NPC name)` with unanimous feminine or masculine PEQ records receives a voice from the matching available pool. - Unknown, neuter, conflicting, missing-zone, and empty-subpool cases use the complete available curated pool. - Existing persistent voices and pins are never replaced by generated data or supplemental payloads. - Pin, reroll, and random-each-line remain available in the voice manager. - Legacy `speakers.json` files load without migration. - An optional small Legends payload can map NPCs absent from EQEmu without adding audio assets or networking to the main release. - The ProjectEQ dump and database tooling are not present in release archives. - Automated tests pass, archive size impact is reported, and final behavioral verification is performed through the user's playtest checklist. ## Later roadmap: collaborative voice choices After the local supplement format has proven stable: 1. Add an export preview for selected pinned mappings only, including zone, normalized NPC name, voice ID, schema version, and game-data version. 2. Build a website/API that accepts individual submissions, tracks votes and provenance, and moderates invalid zone/NPC/voice combinations. 3. Generate a reviewed Legends-only supplement from accepted records. 4. Let users explicitly download/import a versioned payload with checksum and source information. 5. Consider an opt-in recommendation pack for known EQEmu NPCs. Local pins and existing assignments must remain higher precedence. Keeping collection, publication, and download outside the executable prevents the collaborative roadmap from increasing the normal build size or making Mudmouth dependent on the website.