# EQEmu-informed NPC voice mapping

## Outcome

Use the EQEmu/ProjectEQ NPC database to narrow the Kokoro voice pool for a
known EverQuest NPC. The first implementation uses the NPC's recorded gender;
race, body type, class, and appearance can refine voice character later.

This feature must remain advisory:

- Mudmouth still owns the final voice assignment.
- A user's saved or pinned voice always wins.
- Unknown, neuter, and ambiguous NPCs retain today's full-pool behavior.
- Existing `speakers.json` files remain valid and are never rewritten merely
  because the bundled data changes.
- Mudmouth remains fully usable offline.

The release should contain a compact generated lookup, not the ProjectEQ dump,
a database server, generated audio, or voice model assets.

## Product decisions

### Separate traits from choices

The built-in EQEmu index contains NPC traits, initially only `masculine` or
`feminine`. It does **not** prescribe an exact Kokoro voice. Mudmouth randomly
chooses from the matching voices available on the user's Kokoro server and
persists that choice as it does today.

An exact voice ID is a preference, not an EQEmu fact. Exact choices belong in:

1. the user's local `speakers.json`; or
2. an optional, separately downloaded community/Legends mapping payload.

This division keeps generated facts reproducible while preserving subjective
casting choices.

### Keep the built-in release small

Generate a minified, sorted data artifact containing only:

```text
(display-zone, normalized NPC name) -> gender
```

Embed it in the executable with `include_bytes!` and deserialize it once at
startup. `serde_json` is already a dependency, so a first version should not
add a compression, database, or perfect-hash dependency. The existing `.zip`
and `.tar.gz` release formats will compress the repeated zone and name text.

Before merging, measure both the raw executable and final release archive
before and after the data is embedded. Optimize the encoding only if the
archive delta is material. A compact binary format is a later optimization,
not an MVP requirement.

The ProjectEQ dump and the generator's temporary MariaDB directory are build
inputs and must never enter a release archive.

### Preserve user control

The voice manager remains the correction mechanism for incorrect database
traits, unavailable community voices, and personal taste:

- **Pin** selects an exact voice and is always authoritative.
- **Reroll persistent random** chooses from the currently resolved pool.
- **Random every line** chooses from the currently resolved pool for each
  utterance.
- An already persisted random voice remains unchanged after an index or
  mapping-payload update.

The current speaker store is keyed only by normalized NPC name. Do not migrate
it or silently create zone-qualified speaker records in this feature. Zone is
used to improve the initial lookup, while the existing stable name-to-voice
behavior remains intact. Zone-qualified personal overrides can be considered
separately if playtesting demonstrates a real collision problem.

## Runtime resolution policy

Resolve an NPC in this order:

1. If the speaker already has a persisted exact voice (`Pinned` or
   `PersistentRandom`), return it unchanged.
2. If an enabled supplemental mapping has an exact voice for the current
   `(zone, NPC)` and that voice is in the live curated Kokoro catalog, use and
   persist it.
3. If the built-in EQEmu index identifies the current `(zone, NPC)` as
   feminine, choose from available `af_*` and `bf_*` voices.
4. If it identifies the NPC as masculine, choose from available `am_*` and
   `bm_*` voices.
5. If no zone is known, the lookup is missing, the source records conflict,
   the NPC is neuter, or the resolved gender pool is empty, choose from the
   complete available curated pool.

For `RandomEachUtterance`, perform steps 2–5 for every line, but do not turn a
supplemental recommendation into a local pin. For a manual reroll, perform
steps 3–5 and persist the new result. A manual pin never gets replaced.

Supplemental mappings are deliberately checked before EQEmu traits so a
custom Legends NPC can receive a curated exact voice. The initial supplement
loader should reject or ignore entries already covered by the bundled EQEmu
index. This makes the first downloadable payload additive rather than a way to
override known data. A later explicit “community recommendations” mode may
allow preferences for known NPCs, but it must remain opt-in and below local
user choices.

## Lookup identity and normalization

Mudmouth knows:

- the speaker name from a `Name says, '…'` log line; and
- the current display-zone name from `You have entered …`.

The canonical lookup is therefore:

```text
(current display-zone name, normalized NPC name)
```

Use the display name exactly as observed in the log. The generator translates
EQEmu short zone names through `zone.long_name`.

Extract the name-normalization portion of `events::speaker_key` into a shared
function:

```text
lowercase
replace each non-ASCII-alphanumeric run with "_"
trim leading/trailing "_"
collapse repeated "_"
```

`speaker_key` should continue to add the `npc:` prefix for `speakers.json`.
Both the generator and runtime lookup must call the shared normalization
function so their behavior cannot drift.

## PEQ dump findings

These figures were measured from the supplied `peq-1759046415.zip` dump. Treat
the dump as an input to the generator, not application data.

- The required relationship is `spawn2.zone` -> `spawn2.spawngroupID` ->
  `spawnentry.npcID` -> `npc_types.id`. `spawngroup` is not required.
- `spawn2.zone` contains a short zone name, while `zone.long_name` contains
  the display label Mudmouth records.
- There are 559 distinct display labels for 482 short names. Some labels
  deliberately combine multiple short zones, such as the seven Muramite
  Proving Grounds zones. Union all candidates for a display label before
  deciding whether their genders agree.
- Gender values in this dump are `0` masculine, `1` feminine, and `2` neuter.
  Decode them explicitly; the field is not a boolean.
- After joining the spawn tables, translating zones, and applying Mudmouth
  normalization, there are 37,262 `(display-zone, speaker)` pairs: 10,046
  masculine-only, 3,749 feminine-only, 22,692 neuter-only, and 775 with
  conflicting genders.
- The v1 gender index therefore has 13,795 eligible entries. Neuter,
  conflicting, and missing entries deliberately use the full pool.
- A pair does not need to identify one `npc_types` row. Respawn variants are
  eligible when every matching row agrees on the same masculine or feminine
  gender.

The temporary MariaDB data directory used for this measurement was about
765 MiB. It is suitable for generation, not distribution.

## Generator design

The generator should:

1. Accept an explicit PEQ dump path and output path.
2. Read raw candidate rows from `npc_types`, `spawnentry`, `spawn2`, and
   `zone`. It may invoke the `mariadb` client supplied by `shell.nix`; no
   MariaDB client library should become a Mudmouth runtime dependency.
3. Normalize NPC names in Rust using the shared runtime function.
4. Group by `(zone.long_name, normalized_name)` after all short-name rows have
   been combined.
5. Emit a row only if every candidate is gender `0` or every candidate is
   gender `1`.
6. Decode the database number into a typed `Gender` value rather than carrying
   magic numbers into runtime code.
7. Sort records deterministically and write minified output.
8. Print source identity, accepted/conflicting/neuter counts, and output byte
   size so refreshes are auditable.

Check the generated artifact into the repository. Runtime and release builds
must not need MariaDB or the PEQ dump.

The following query is useful for validating results:

```sql
SELECT
  z.long_name AS zone,
  TRIM(BOTH '_' FROM REGEXP_REPLACE(LOWER(n.name), '[^a-z0-9]+', '_'))
    AS normalized_name,
  MIN(n.gender) AS gender
FROM spawn2 AS s
JOIN spawnentry AS se ON se.spawngroupID = s.spawngroupID
JOIN npc_types AS n ON n.id = se.npcID
JOIN zone AS z ON z.short_name = s.zone
WHERE s.zone <> '' AND n.name <> ''
GROUP BY z.long_name, normalized_name
HAVING COUNT(DISTINCT n.gender) = 1 AND MIN(n.gender) IN (0, 1);
```

Do not use this SQL normalization as the compatibility contract; it is only an
independent check of the Rust generator.

## Loading the supplied dump

`shell.nix` provides MariaDB and `unzip`. Start `nix-shell`, then use an
isolated server so the process cannot affect a system MariaDB instance:

```bash
peq_dir="$(mktemp -d /tmp/mudmouth-peq.XXXXXX)"
mariadb-install-db --no-defaults --datadir="$peq_dir/data" \
  --auth-root-authentication-method=normal --skip-test-db
mariadbd --no-defaults --datadir="$peq_dir/data" \
  --socket="$peq_dir/mariadb.sock" --pid-file="$peq_dir/mariadbd.pid" \
  --skip-networking --log-error="$peq_dir/mariadbd.log" &

until mariadb-admin --no-defaults --socket="$peq_dir/mariadb.sock" \
  -u root ping --silent; do sleep 1; done
mariadb --no-defaults --socket="$peq_dir/mariadb.sock" -u root \
  -e 'CREATE DATABASE peq_content DEFAULT CHARACTER SET latin1;'
unzip -p ~/Downloads/peq-1759046415.zip peq-dump/create_tables_content.sql \
  | mariadb --no-defaults --socket="$peq_dir/mariadb.sock" -u root peq_content
```

Only `create_tables_content.sql` is necessary. Shut the isolated server down
with:

```bash
mariadb-admin --no-defaults --socket="$peq_dir/mariadb.sock" -u root shutdown
```

Remove only the known temporary directory after the generator is finished.

## Supplemental Legends/community payload

Support a small, optional data file in Mudmouth's application-data directory,
separate from `speakers.json`. It contains exact voice recommendations for
NPCs absent from the bundled EQEmu index:

```json
{
  "schema_version": 1,
  "pack": "eq-legends-community",
  "game_data_version": "2026-07",
  "mappings": [
    {
      "zone": "Example Zone",
      "npc": "example_npc",
      "voice": "bm_george"
    }
  ]
}
```

Payload rules:

- Store only metadata, normalized identifiers, and Kokoro voice IDs—no audio.
- Keep it minified for download; compression at transport time is sufficient.
- Validate `schema_version`, non-empty fields, duplicate keys, and voice IDs.
- Ignore a recommendation whose voice is unavailable and fall back normally.
- Ignore entries already known to the bundled index in the first version.
- A malformed optional payload should produce a visible warning and leave the
  built-in resolver usable.
- Loading a new payload must not alter an existing speaker assignment.
- Do not add automatic networking or updates to the MVP. The user downloads
  and installs the payload explicitly.

This schema is also the bridge to the future collaborative website. The site
can collect zone-qualified, selectively shared pins and publish a reviewed,
versioned supplement without increasing the normal Mudmouth release size.
Before supporting uploads, add an explicit export preview and consent step;
never upload `speakers.json` wholesale.

## Agent-sized implementation tasks

Each task should fit in one focused change and leave the branch buildable.
Tasks 1–4 form the core feature. Tasks 5–7 can proceed after the resolver
contract in task 3 is stable.

### Task 1 — Shared NPC identity normalization

**Scope:** `src/events.rs` and focused tests.

- Extract `normalize_npc_name(&str) -> String`.
- Keep `speaker_key` output byte-for-byte compatible by adding `npc:` to the
  shared result.
- Add fixtures for punctuation, repeated separators, capitalization, and
  non-ASCII characters.

**Done when:** all old parser tests pass and generator/runtime callers can use
the same public normalization function.

### Task 2 — Reproducible PEQ index generator

**Scope:** a non-runtime generator binary or tool, generator documentation,
and a tiny checked-in fixture dump/export.

- Implement the join-input extraction, grouping, explicit gender decoding,
  conflict filtering, deterministic sorting, and statistics.
- Test combined short zones that share one `long_name`, same-gender duplicate
  NPC rows, conflicts, neuter rows, and malformed input.
- Generate the full minified index and record its source dump identifier and
  counts in generated metadata or adjacent documentation.

**Done when:** two runs from the same dump are byte-identical and reproduce
13,795 accepted rows for the researched dump.

### Task 3 — Built-in index and pool resolver

**Scope:** new `npc_voices` module, generated artifact inclusion, and unit
tests; no speaker-store behavior changes yet.

- Define typed `Gender` and lookup identity types.
- Load the embedded index once and expose a pure lookup function.
- Filter the live curated Kokoro catalog into feminine, masculine, and full
  pools without duplicating the allowlist.
- Fall back to the full pool when zone/context/data/pool is missing.

**Done when:** tests cover feminine, masculine, neuter-by-omission,
conflicting-by-omission, missing-zone, unknown NPC, and an empty gender pool.

### Task 4 — Integrate resolution with speaker persistence and the TUI

**Scope:** `src/speakers.rs`, `src/main.rs`, `src/tui.rs`, and regression
tests.

- Return saved persistent and pinned voices before consulting new data.
- Use the resolved pool for first assignment, random-each-utterance, and
  reroll.
- Keep legacy `speakers.json` deserialization and keys unchanged.
- Make the manual voice manager's chosen voice authoritative regardless of
  the EQEmu gender.

**Done when:** tests prove saved/pinned choices survive refreshes and restarts,
manual reroll uses the resolved pool, random-each-line uses the resolved pool,
and legacy JSON loads without migration.

### Task 5 — Optional Legends supplement loader

**Scope:** supplemental schema/parser, application-data loading, resolver
composition, sample payload, and tests.

- Load the versioned file without adding runtime networking.
- Validate and deduplicate records.
- Admit only NPC identities absent from the bundled EQEmu index.
- Resolve an available exact voice before the EQEmu/full-pool fallback.
- Report malformed files and unavailable voices without stopping Mudmouth.

**Done when:** an unknown Legends NPC can receive a downloaded exact mapping,
while known EQEmu NPCs and all existing local assignments remain unchanged.

### Task 6 — Size and packaging guardrail

**Scope:** `scripts/package-release.sh` or a companion reporting script and
release documentation.

- Confirm the raw PEQ dump, generator inputs, and fixtures are absent from
  archives.
- Report generated index size, executable size, and `.zip`/`.tar.gz` sizes.
- Record the before/after archive delta in the change description.
- Add a generous regression threshold only after a real baseline exists.

**Done when:** Linux and Windows archive contents are inspected and the
feature's distribution cost is explicit.

### Task 7 — Playtest handoff

**Scope:** a short manual checklist and optional diagnostic activity messages.

- Show enough non-sensitive resolver information to distinguish saved,
  supplement, EQEmu-feminine, EQEmu-masculine, and full-pool decisions.
- Provide known test NPCs/zones from the generated fixture or index.
- Do not add automated game-driving or audio snapshot tests.

**Done when:** the user can verify the scenarios below during normal play and
report the NPC, zone, expected class, actual voice, and result.

## Playtest checklist

The user will perform final in-game verification:

1. Enter a zone with a known feminine NPC and confirm the first assignment is
   an available `af_*` or `bf_*` voice.
2. Repeat with a known masculine NPC and confirm `am_*` or `bm_*`.
3. Speak with a neuter or conflicting NPC and confirm it can use the full
   curated pool.
4. Speak with an EQ Legends NPC absent from PEQ and confirm normal full-pool
   fallback without a supplement.
5. Install a supplement containing that NPC and, before it has a saved local
   assignment, confirm the recommended voice is used.
6. Pin a deliberately cross-gender voice, restart Mudmouth, and confirm the
   pin remains unchanged.
7. Reroll a known NPC and confirm the new voice stays in its resolved pool.
8. Enable random-each-line and confirm every selected voice stays in the
   resolved pool.
9. Update or replace the index/supplement and confirm previously saved voices
   do not change.
10. Visit two zones containing the same display name and note whether the
    current global speaker identity is confusing enough to justify a future
    zone-specific override feature.

## Acceptance criteria

- A known `(display zone, normalized NPC name)` with unanimous feminine or
  masculine PEQ records receives a voice from the matching available pool.
- Unknown, neuter, conflicting, missing-zone, and empty-subpool cases use the
  complete available curated pool.
- Existing persistent voices and pins are never replaced by generated data or
  supplemental payloads.
- Pin, reroll, and random-each-line remain available in the voice manager.
- Legacy `speakers.json` files load without migration.
- An optional small Legends payload can map NPCs absent from EQEmu without
  adding audio assets or networking to the main release.
- The ProjectEQ dump and database tooling are not present in release archives.
- Automated tests pass, archive size impact is reported, and final behavioral
  verification is performed through the user's playtest checklist.

## Later roadmap: collaborative voice choices

After the local supplement format has proven stable:

1. Add an export preview for selected pinned mappings only, including zone,
   normalized NPC name, voice ID, schema version, and game-data version.
2. Build a website/API that accepts individual submissions, tracks votes and
   provenance, and moderates invalid zone/NPC/voice combinations.
3. Generate a reviewed Legends-only supplement from accepted records.
4. Let users explicitly download/import a versioned payload with checksum and
   source information.
5. Consider an opt-in recommendation pack for known EQEmu NPCs. Local pins and
   existing assignments must remain higher precedence.

Keeping collection, publication, and download outside the executable prevents
the collaborative roadmap from increasing the normal build size or making
Mudmouth dependent on the website.
