mudmouth/docs/tasks/eqemu-npc-voice-mapping.md

EQEmu-informed NPC voice mapping

Outcome

Use the EQEmu/ProjectEQ NPC database to narrow the Kokoro voice pool for a known EverQuest NPC. The first implementation uses the NPC's recorded gender; race, body type, class, and appearance can refine voice character later.

This feature must remain advisory:

• Mudmouth still owns the final voice assignment.
• A user's saved or pinned voice always wins.
• Unknown, neuter, and ambiguous NPCs retain today's full-pool behavior.
• Existing speakers.json files remain valid and are never rewritten merely because the bundled data changes.
• Mudmouth remains fully usable offline.

The release should contain a compact generated lookup, not the ProjectEQ dump, a database server, generated audio, or voice model assets.

Product decisions

Separate traits from choices

The built-in EQEmu index contains NPC traits, initially only masculine or feminine. It does not prescribe an exact Kokoro voice. Mudmouth randomly chooses from the matching voices available on the user's Kokoro server and persists that choice as it does today.

An exact voice ID is a preference, not an EQEmu fact. Exact choices belong in:

1. the user's local speakers.json; or
2. an optional, separately downloaded community/Legends mapping payload.

This division keeps generated facts reproducible while preserving subjective casting choices.

Keep the built-in release small

Generate a minified, sorted data artifact containing only:

(display-zone, normalized NPC name) -> gender

Embed it in the executable with include_bytes! and deserialize it once at startup. serde_json is already a dependency, so a first version should not add a compression, database, or perfect-hash dependency. The existing .zip and .tar.gz release formats will compress the repeated zone and name text.

Before merging, measure both the raw executable and final release archive before and after the data is embedded. Optimize the encoding only if the archive delta is material. A compact binary format is a later optimization, not an MVP requirement.

The ProjectEQ dump and the generator's temporary MariaDB directory are build inputs and must never enter a release archive.

Preserve user control

The voice manager remains the correction mechanism for incorrect database traits, unavailable community voices, and personal taste:

• Pin selects an exact voice and is always authoritative.
• Reroll persistent random chooses from the currently resolved pool.
• Random every line chooses from the currently resolved pool for each utterance.
• An already persisted random voice remains unchanged after an index or mapping-payload update.

The current speaker store is keyed only by normalized NPC name. Do not migrate it or silently create zone-qualified speaker records in this feature. Zone is used to improve the initial lookup, while the existing stable name-to-voice behavior remains intact. Zone-qualified personal overrides can be considered separately if playtesting demonstrates a real collision problem.

Runtime resolution policy

Resolve an NPC in this order:

1. If the speaker already has a persisted exact voice (Pinned or PersistentRandom), return it unchanged.
2. If an enabled supplemental mapping has an exact voice for the current (zone, NPC) and that voice is in the live curated Kokoro catalog, use and persist it.
3. If the built-in EQEmu index identifies the current (zone, NPC) as feminine, choose from available af_* and bf_* voices.
4. If it identifies the NPC as masculine, choose from available am_* and bm_* voices.
5. If no zone is known, the lookup is missing, the source records conflict, the NPC is neuter, or the resolved gender pool is empty, choose from the complete available curated pool.

For RandomEachUtterance, perform steps 2–5 for every line, but do not turn a supplemental recommendation into a local pin. For a manual reroll, perform steps 3–5 and persist the new result. A manual pin never gets replaced.

Supplemental mappings are deliberately checked before EQEmu traits so a custom Legends NPC can receive a curated exact voice. The initial supplement loader should reject or ignore entries already covered by the bundled EQEmu index. This makes the first downloadable payload additive rather than a way to override known data. A later explicit “community recommendations” mode may allow preferences for known NPCs, but it must remain opt-in and below local user choices.

Lookup identity and normalization

Mudmouth knows:

• the speaker name from a Name says, '…' log line; and
• the current display-zone name from You have entered ….

The canonical lookup is therefore:

(current display-zone name, normalized NPC name)

Use the display name exactly as observed in the log. The generator translates EQEmu short zone names through zone.long_name.

Extract the name-normalization portion of events::speaker_key into a shared function:

lowercase
replace each non-ASCII-alphanumeric run with "_"
trim leading/trailing "_"
collapse repeated "_"

speaker_key should continue to add the npc: prefix for speakers.json. Both the generator and runtime lookup must call the shared normalization function so their behavior cannot drift.

PEQ dump findings

These figures were measured from the supplied peq-1759046415.zip dump. Treat the dump as an input to the generator, not application data.

• The required relationship is spawn2.zone -> spawn2.spawngroupID -> spawnentry.npcID -> npc_types.id. spawngroup is not required.
• spawn2.zone contains a short zone name, while zone.long_name contains the display label Mudmouth records.
• There are 559 distinct display labels for 482 short names. Some labels deliberately combine multiple short zones, such as the seven Muramite Proving Grounds zones. Union all candidates for a display label before deciding whether their genders agree.
• Gender values in this dump are 0 masculine, 1 feminine, and 2 neuter. Decode them explicitly; the field is not a boolean.
• After joining the spawn tables, translating zones, and applying Mudmouth normalization, there are 37,262 (display-zone, speaker) pairs: 10,046 masculine-only, 3,749 feminine-only, 22,692 neuter-only, and 775 with conflicting genders.
• The v1 gender index therefore has 13,795 eligible entries. Neuter, conflicting, and missing entries deliberately use the full pool.
• A pair does not need to identify one npc_types row. Respawn variants are eligible when every matching row agrees on the same masculine or feminine gender.

The temporary MariaDB data directory used for this measurement was about 765 MiB. It is suitable for generation, not distribution.

Generator design

The generator should:

1. Accept an explicit PEQ dump path and output path.
2. Read raw candidate rows from npc_types, spawnentry, spawn2, and zone. It may invoke the mariadb client supplied by shell.nix; no MariaDB client library should become a Mudmouth runtime dependency.
3. Normalize NPC names in Rust using the shared runtime function.
4. Group by (zone.long_name, normalized_name) after all short-name rows have been combined.
5. Emit a row only if every candidate is gender 0 or every candidate is gender 1.
6. Decode the database number into a typed Gender value rather than carrying magic numbers into runtime code.
7. Sort records deterministically and write minified output.
8. Print source identity, accepted/conflicting/neuter counts, and output byte size so refreshes are auditable.

Check the generated artifact into the repository. Runtime and release builds must not need MariaDB or the PEQ dump.

The following query is useful for validating results:

SELECT
  z.long_name AS zone,
  TRIM(BOTH '_' FROM REGEXP_REPLACE(LOWER(n.name), '[^a-z0-9]+', '_'))
    AS normalized_name,
  MIN(n.gender) AS gender
FROM spawn2 AS s
JOIN spawnentry AS se ON se.spawngroupID = s.spawngroupID
JOIN npc_types AS n ON n.id = se.npcID
JOIN zone AS z ON z.short_name = s.zone
WHERE s.zone <> '' AND n.name <> ''
GROUP BY z.long_name, normalized_name
HAVING COUNT(DISTINCT n.gender) = 1 AND MIN(n.gender) IN (0, 1);

Do not use this SQL normalization as the compatibility contract; it is only an independent check of the Rust generator.

Loading the supplied dump

shell.nix provides MariaDB and unzip. Start nix-shell, then use an isolated server so the process cannot affect a system MariaDB instance:

peq_dir="$(mktemp -d /tmp/mudmouth-peq.XXXXXX)"
mariadb-install-db --no-defaults --datadir="$peq_dir/data" \
  --auth-root-authentication-method=normal --skip-test-db
mariadbd --no-defaults --datadir="$peq_dir/data" \
  --socket="$peq_dir/mariadb.sock" --pid-file="$peq_dir/mariadbd.pid" \
  --skip-networking --log-error="$peq_dir/mariadbd.log" &

until mariadb-admin --no-defaults --socket="$peq_dir/mariadb.sock" \
  -u root ping --silent; do sleep 1; done
mariadb --no-defaults --socket="$peq_dir/mariadb.sock" -u root \
  -e 'CREATE DATABASE peq_content DEFAULT CHARACTER SET latin1;'
unzip -p ~/Downloads/peq-1759046415.zip peq-dump/create_tables_content.sql \
  | mariadb --no-defaults --socket="$peq_dir/mariadb.sock" -u root peq_content

Only create_tables_content.sql is necessary. Shut the isolated server down with:

mariadb-admin --no-defaults --socket="$peq_dir/mariadb.sock" -u root shutdown

Remove only the known temporary directory after the generator is finished.

Supplemental Legends/community payload

Support a small, optional data file in Mudmouth's application-data directory, separate from speakers.json. It contains exact voice recommendations for NPCs absent from the bundled EQEmu index:

{
  "schema_version": 1,
  "pack": "eq-legends-community",
  "game_data_version": "2026-07",
  "mappings": [
    {
      "zone": "Example Zone",
      "npc": "example_npc",
      "voice": "bm_george"
    }
  ]
}

Payload rules:

• Store only metadata, normalized identifiers, and Kokoro voice IDs—no audio.
• Keep it minified for download; compression at transport time is sufficient.
• Validate schema_version, non-empty fields, duplicate keys, and voice IDs.
• Ignore a recommendation whose voice is unavailable and fall back normally.
• Ignore entries already known to the bundled index in the first version.
• A malformed optional payload should produce a visible warning and leave the built-in resolver usable.
• Loading a new payload must not alter an existing speaker assignment.
• Do not add automatic networking or updates to the MVP. The user downloads and installs the payload explicitly.

This schema is also the bridge to the future collaborative website. The site can collect zone-qualified, selectively shared pins and publish a reviewed, versioned supplement without increasing the normal Mudmouth release size. Before supporting uploads, add an explicit export preview and consent step; never upload speakers.json wholesale.

Agent-sized implementation tasks

Each task should fit in one focused change and leave the branch buildable. Tasks 1–4 form the core feature. Tasks 5–7 can proceed after the resolver contract in task 3 is stable.

Task 1 — Shared NPC identity normalization

Scope: src/events.rs and focused tests.

• Extract normalize_npc_name(&str) -> String.
• Keep speaker_key output byte-for-byte compatible by adding npc: to the shared result.
• Add fixtures for punctuation, repeated separators, capitalization, and non-ASCII characters.

Done when: all old parser tests pass and generator/runtime callers can use the same public normalization function.

Task 2 — Reproducible PEQ index generator

Scope: a non-runtime generator binary or tool, generator documentation, and a tiny checked-in fixture dump/export.

• Implement the join-input extraction, grouping, explicit gender decoding, conflict filtering, deterministic sorting, and statistics.
• Test combined short zones that share one long_name, same-gender duplicate NPC rows, conflicts, neuter rows, and malformed input.
• Generate the full minified index and record its source dump identifier and counts in generated metadata or adjacent documentation.

Done when: two runs from the same dump are byte-identical and reproduce 13,795 accepted rows for the researched dump.

Task 3 — Built-in index and pool resolver

Scope: new npc_voices module, generated artifact inclusion, and unit tests; no speaker-store behavior changes yet.

• Define typed Gender and lookup identity types.
• Load the embedded index once and expose a pure lookup function.
• Filter the live curated Kokoro catalog into feminine, masculine, and full pools without duplicating the allowlist.
• Fall back to the full pool when zone/context/data/pool is missing.

Done when: tests cover feminine, masculine, neuter-by-omission, conflicting-by-omission, missing-zone, unknown NPC, and an empty gender pool.

Task 4 — Integrate resolution with speaker persistence and the TUI

Scope: src/speakers.rs, src/main.rs, src/tui.rs, and regression tests.

• Return saved persistent and pinned voices before consulting new data.
• Use the resolved pool for first assignment, random-each-utterance, and reroll.
• Keep legacy speakers.json deserialization and keys unchanged.
• Make the manual voice manager's chosen voice authoritative regardless of the EQEmu gender.

Done when: tests prove saved/pinned choices survive refreshes and restarts, manual reroll uses the resolved pool, random-each-line uses the resolved pool, and legacy JSON loads without migration.

Task 5 — Optional Legends supplement loader

Scope: supplemental schema/parser, application-data loading, resolver composition, sample payload, and tests.

• Load the versioned file without adding runtime networking.
• Validate and deduplicate records.
• Admit only NPC identities absent from the bundled EQEmu index.
• Resolve an available exact voice before the EQEmu/full-pool fallback.
• Report malformed files and unavailable voices without stopping Mudmouth.

Done when: an unknown Legends NPC can receive a downloaded exact mapping, while known EQEmu NPCs and all existing local assignments remain unchanged.

Task 6 — Size and packaging guardrail

Scope: scripts/package-release.sh or a companion reporting script and release documentation.

• Confirm the raw PEQ dump, generator inputs, and fixtures are absent from archives.
• Report generated index size, executable size, and .zip/.tar.gz sizes.
• Record the before/after archive delta in the change description.
• Add a generous regression threshold only after a real baseline exists.

Done when: Linux and Windows archive contents are inspected and the feature's distribution cost is explicit.

Task 7 — Playtest handoff

Scope: a short manual checklist and optional diagnostic activity messages.

• Show enough non-sensitive resolver information to distinguish saved, supplement, EQEmu-feminine, EQEmu-masculine, and full-pool decisions.
• Provide known test NPCs/zones from the generated fixture or index.
• Do not add automated game-driving or audio snapshot tests.

Done when: the user can verify the scenarios below during normal play and report the NPC, zone, expected class, actual voice, and result.

Playtest checklist

The user will perform final in-game verification:

1. Enter a zone with a known feminine NPC and confirm the first assignment is an available af_* or bf_* voice.
2. Repeat with a known masculine NPC and confirm am_* or bm_*.
3. Speak with a neuter or conflicting NPC and confirm it can use the full curated pool.
4. Speak with an EQ Legends NPC absent from PEQ and confirm normal full-pool fallback without a supplement.
5. Install a supplement containing that NPC and, before it has a saved local assignment, confirm the recommended voice is used.
6. Pin a deliberately cross-gender voice, restart Mudmouth, and confirm the pin remains unchanged.
7. Reroll a known NPC and confirm the new voice stays in its resolved pool.
8. Enable random-each-line and confirm every selected voice stays in the resolved pool.
9. Update or replace the index/supplement and confirm previously saved voices do not change.
10. Visit two zones containing the same display name and note whether the current global speaker identity is confusing enough to justify a future zone-specific override feature.

Acceptance criteria

• A known (display zone, normalized NPC name) with unanimous feminine or masculine PEQ records receives a voice from the matching available pool.
• Unknown, neuter, conflicting, missing-zone, and empty-subpool cases use the complete available curated pool.
• Existing persistent voices and pins are never replaced by generated data or supplemental payloads.
• Pin, reroll, and random-each-line remain available in the voice manager.
• Legacy speakers.json files load without migration.
• An optional small Legends payload can map NPCs absent from EQEmu without adding audio assets or networking to the main release.
• The ProjectEQ dump and database tooling are not present in release archives.
• Automated tests pass, archive size impact is reported, and final behavioral verification is performed through the user's playtest checklist.

Later roadmap: collaborative voice choices

After the local supplement format has proven stable:

1. Add an export preview for selected pinned mappings only, including zone, normalized NPC name, voice ID, schema version, and game-data version.
2. Build a website/API that accepts individual submissions, tracks votes and provenance, and moderates invalid zone/NPC/voice combinations.
3. Generate a reviewed Legends-only supplement from accepted records.
4. Let users explicitly download/import a versioned payload with checksum and source information.
5. Consider an opt-in recommendation pack for known EQEmu NPCs. Local pins and existing assignments must remain higher precedence.

Keeping collection, publication, and download outside the executable prevents the collaborative roadmap from increasing the normal build size or making Mudmouth dependent on the website.