Add a size and packaging guardrail

docs/development.md

@@ -18,6 +18,25 @@ cargo test
 cargo build --release
 ```
 
+## Refreshing the PEQ NPC gender index
+
+The checked-in `generated/peq-npc-gender-index.json` is generated data, not a
+runtime database dependency. Regenerate it from an explicit ProjectEQ dump
+with the tool below; it reads either `create_tables_content.sql` directly or
+the `peq-dump/create_tables_content.sql` member of the supplied zip archive.
+`unzip` is available in `shell.nix` for the latter.
+
+```bash
+cargo run --bin generate_peq_npc_index -- \
+  --dump "$HOME/Downloads/peq-1759046415.zip" \
+  --output generated/peq-npc-gender-index.json
+```
+
+The tool prints its input identity, complete conflict/neuter statistics, and
+output size. Run it twice and compare the output hashes before updating the
+checked-in artifact. The source archive and any temporary database files must
+not be committed or included in release archives.
+
 The parser tests cover the observed log formats, including NPC dialogue,
 channel-chat exclusion, zone events, normalization, persistent/random speaker
 voice policies, log truncation, and first-run log discovery.
@@ -57,5 +76,35 @@ nix-build release.nix -A windows -o result-windows
 ```
 
 The packager writes platform-qualified archives and `SHA256SUMS.txt` to
-`dist/`. `scripts/publish-release.sh` uploads those exact assets as a Gitea
-pre-release after the worktree is committed and clean.
+`dist/`. It also writes `SIZE-REPORT.txt`, recording the generated PEQ index,
+the two executables, and both final archives in bytes. Before writing the
+checksums, the packager verifies that each archive contains exactly its binary,
+`README.md`, `LICENSE`, and `config.toml`. This excludes PEQ dumps, SQL/fixture
+inputs, MariaDB data, the generator, and all source files from releases.
+
+Keep the size report with the release/change description. When changing the
+embedded index encoding, compare its archive sizes against the previous report
+and record the Linux `.tar.gz` and Windows `.zip` deltas. Do not add an archive
+size threshold until a stable release baseline has been established.
+
+### PEQ index size audit — 2026-07-29
+
+The following release builds compare pre-PEQ commit `561ace9` with the current
+core mapping implementation (tasks 1–4). The after size includes the embedded
+index and its resolver code.
+
+| Artifact | Before | After | Delta |
+| --- | ---: | ---: | ---: |
+| Generated index | 0 B | 719,199 B | +719,199 B |
+| Linux executable | 6,772,856 B | 7,535,632 B | +762,776 B |
+| Linux `.tar.gz` | 2,871,605 B | 3,010,723 B | +139,118 B |
+| Windows executable | 6,577,369 B | 7,332,144 B | +754,775 B |
+| Windows `.zip` | 2,805,692 B | 2,944,841 B | +139,149 B |
+
+Both archives were inspected after packaging and contained only the platform
+binary, `README.md`, `LICENSE`, and `config.toml`. The compressed distribution
+cost is about 139 KB per platform; retain the JSON encoding until a future
+measurement establishes a material reason to optimize it.
+
+`scripts/publish-release.sh` uploads the archive assets and checksums as a
+Gitea pre-release after the worktree is committed and clean.

docs/tasks/eqemu-npc-voice-mapping.md

@@ -0,0 +1,436 @@
+# EQEmu-informed NPC voice mapping
+
+## Outcome
+
+Use the EQEmu/ProjectEQ NPC database to narrow the Kokoro voice pool for a
+known EverQuest NPC. The first implementation uses the NPC's recorded gender;
+race, body type, class, and appearance can refine voice character later.
+
+This feature must remain advisory:
+
+- Mudmouth still owns the final voice assignment.
+- A user's saved or pinned voice always wins.
+- Unknown, neuter, and ambiguous NPCs retain today's full-pool behavior.
+- Existing `speakers.json` files remain valid and are never rewritten merely
+  because the bundled data changes.
+- Mudmouth remains fully usable offline.
+
+The release should contain a compact generated lookup, not the ProjectEQ dump,
+a database server, generated audio, or voice model assets.
+
+## Product decisions
+
+### Separate traits from choices
+
+The built-in EQEmu index contains NPC traits, initially only `masculine` or
+`feminine`. It does **not** prescribe an exact Kokoro voice. Mudmouth randomly
+chooses from the matching voices available on the user's Kokoro server and
+persists that choice as it does today.
+
+An exact voice ID is a preference, not an EQEmu fact. Exact choices belong in:
+
+1. the user's local `speakers.json`; or
+2. an optional, separately downloaded community/Legends mapping payload.
+
+This division keeps generated facts reproducible while preserving subjective
+casting choices.
+
+### Keep the built-in release small
+
+Generate a minified, sorted data artifact containing only:
+
+```text
+(display-zone, normalized NPC name) -> gender
+```
+
+Embed it in the executable with `include_bytes!` and deserialize it once at
+startup. `serde_json` is already a dependency, so a first version should not
+add a compression, database, or perfect-hash dependency. The existing `.zip`
+and `.tar.gz` release formats will compress the repeated zone and name text.
+
+Before merging, measure both the raw executable and final release archive
+before and after the data is embedded. Optimize the encoding only if the
+archive delta is material. A compact binary format is a later optimization,
+not an MVP requirement.
+
+The ProjectEQ dump and the generator's temporary MariaDB directory are build
+inputs and must never enter a release archive.
+
+### Preserve user control
+
+The voice manager remains the correction mechanism for incorrect database
+traits, unavailable community voices, and personal taste:
+
+- **Pin** selects an exact voice and is always authoritative.
+- **Reroll persistent random** chooses from the currently resolved pool.
+- **Random every line** chooses from the currently resolved pool for each
+  utterance.
+- An already persisted random voice remains unchanged after an index or
+  mapping-payload update.
+
+The current speaker store is keyed only by normalized NPC name. Do not migrate
+it or silently create zone-qualified speaker records in this feature. Zone is
+used to improve the initial lookup, while the existing stable name-to-voice
+behavior remains intact. Zone-qualified personal overrides can be considered
+separately if playtesting demonstrates a real collision problem.
+
+## Runtime resolution policy
+
+Resolve an NPC in this order:
+
+1. If the speaker already has a persisted exact voice (`Pinned` or
+   `PersistentRandom`), return it unchanged.
+2. If an enabled supplemental mapping has an exact voice for the current
+   `(zone, NPC)` and that voice is in the live curated Kokoro catalog, use and
+   persist it.
+3. If the built-in EQEmu index identifies the current `(zone, NPC)` as
+   feminine, choose from available `af_*` and `bf_*` voices.
+4. If it identifies the NPC as masculine, choose from available `am_*` and
+   `bm_*` voices.
+5. If no zone is known, the lookup is missing, the source records conflict,
+   the NPC is neuter, or the resolved gender pool is empty, choose from the
+   complete available curated pool.
+
+For `RandomEachUtterance`, perform steps 2–5 for every line, but do not turn a
+supplemental recommendation into a local pin. For a manual reroll, perform
+steps 3–5 and persist the new result. A manual pin never gets replaced.
+
+Supplemental mappings are deliberately checked before EQEmu traits so a
+custom Legends NPC can receive a curated exact voice. The initial supplement
+loader should reject or ignore entries already covered by the bundled EQEmu
+index. This makes the first downloadable payload additive rather than a way to
+override known data. A later explicit “community recommendations” mode may
+allow preferences for known NPCs, but it must remain opt-in and below local
+user choices.
+
+## Lookup identity and normalization
+
+Mudmouth knows:
+
+- the speaker name from a `Name says, '…'` log line; and
+- the current display-zone name from `You have entered …`.
+
+The canonical lookup is therefore:
+
+```text
+(current display-zone name, normalized NPC name)
+```
+
+Use the display name exactly as observed in the log. The generator translates
+EQEmu short zone names through `zone.long_name`.
+
+Extract the name-normalization portion of `events::speaker_key` into a shared
+function:
+
+```text
+lowercase
+replace each non-ASCII-alphanumeric run with "_"
+trim leading/trailing "_"
+collapse repeated "_"
+```
+
+`speaker_key` should continue to add the `npc:` prefix for `speakers.json`.
+Both the generator and runtime lookup must call the shared normalization
+function so their behavior cannot drift.
+
+## PEQ dump findings
+
+These figures were measured from the supplied `peq-1759046415.zip` dump. Treat
+the dump as an input to the generator, not application data.
+
+- The required relationship is `spawn2.zone` -> `spawn2.spawngroupID` ->
+  `spawnentry.npcID` -> `npc_types.id`. `spawngroup` is not required.
+- `spawn2.zone` contains a short zone name, while `zone.long_name` contains
+  the display label Mudmouth records.
+- There are 559 distinct display labels for 482 short names. Some labels
+  deliberately combine multiple short zones, such as the seven Muramite
+  Proving Grounds zones. Union all candidates for a display label before
+  deciding whether their genders agree.
+- Gender values in this dump are `0` masculine, `1` feminine, and `2` neuter.
+  Decode them explicitly; the field is not a boolean.
+- After joining the spawn tables, translating zones, and applying Mudmouth
+  normalization, there are 37,262 `(display-zone, speaker)` pairs: 10,046
+  masculine-only, 3,749 feminine-only, 22,692 neuter-only, and 775 with
+  conflicting genders.
+- The v1 gender index therefore has 13,795 eligible entries. Neuter,
+  conflicting, and missing entries deliberately use the full pool.
+- A pair does not need to identify one `npc_types` row. Respawn variants are
+  eligible when every matching row agrees on the same masculine or feminine
+  gender.
+
+The temporary MariaDB data directory used for this measurement was about
+765 MiB. It is suitable for generation, not distribution.
+
+## Generator design
+
+The generator should:
+
+1. Accept an explicit PEQ dump path and output path.
+2. Read raw candidate rows from `npc_types`, `spawnentry`, `spawn2`, and
+   `zone`. It may invoke the `mariadb` client supplied by `shell.nix`; no
+   MariaDB client library should become a Mudmouth runtime dependency.
+3. Normalize NPC names in Rust using the shared runtime function.
+4. Group by `(zone.long_name, normalized_name)` after all short-name rows have
+   been combined.
+5. Emit a row only if every candidate is gender `0` or every candidate is
+   gender `1`.
+6. Decode the database number into a typed `Gender` value rather than carrying
+   magic numbers into runtime code.
+7. Sort records deterministically and write minified output.
+8. Print source identity, accepted/conflicting/neuter counts, and output byte
+   size so refreshes are auditable.
+
+Check the generated artifact into the repository. Runtime and release builds
+must not need MariaDB or the PEQ dump.
+
+The following query is useful for validating results:
+
+```sql
+SELECT
+  z.long_name AS zone,
+  TRIM(BOTH '_' FROM REGEXP_REPLACE(LOWER(n.name), '[^a-z0-9]+', '_'))
+    AS normalized_name,
+  MIN(n.gender) AS gender
+FROM spawn2 AS s
+JOIN spawnentry AS se ON se.spawngroupID = s.spawngroupID
+JOIN npc_types AS n ON n.id = se.npcID
+JOIN zone AS z ON z.short_name = s.zone
+WHERE s.zone <> '' AND n.name <> ''
+GROUP BY z.long_name, normalized_name
+HAVING COUNT(DISTINCT n.gender) = 1 AND MIN(n.gender) IN (0, 1);
+```
+
+Do not use this SQL normalization as the compatibility contract; it is only an
+independent check of the Rust generator.
+
+## Loading the supplied dump
+
+`shell.nix` provides MariaDB and `unzip`. Start `nix-shell`, then use an
+isolated server so the process cannot affect a system MariaDB instance:
+
+```bash
+peq_dir="$(mktemp -d /tmp/mudmouth-peq.XXXXXX)"
+mariadb-install-db --no-defaults --datadir="$peq_dir/data" \
+  --auth-root-authentication-method=normal --skip-test-db
+mariadbd --no-defaults --datadir="$peq_dir/data" \
+  --socket="$peq_dir/mariadb.sock" --pid-file="$peq_dir/mariadbd.pid" \
+  --skip-networking --log-error="$peq_dir/mariadbd.log" &
+
+until mariadb-admin --no-defaults --socket="$peq_dir/mariadb.sock" \
+  -u root ping --silent; do sleep 1; done
+mariadb --no-defaults --socket="$peq_dir/mariadb.sock" -u root \
+  -e 'CREATE DATABASE peq_content DEFAULT CHARACTER SET latin1;'
+unzip -p ~/Downloads/peq-1759046415.zip peq-dump/create_tables_content.sql \
+  | mariadb --no-defaults --socket="$peq_dir/mariadb.sock" -u root peq_content
+```
+
+Only `create_tables_content.sql` is necessary. Shut the isolated server down
+with:
+
+```bash
+mariadb-admin --no-defaults --socket="$peq_dir/mariadb.sock" -u root shutdown
+```
+
+Remove only the known temporary directory after the generator is finished.
+
+## Supplemental Legends/community payload
+
+Support a small, optional data file in Mudmouth's application-data directory,
+separate from `speakers.json`. It contains exact voice recommendations for
+NPCs absent from the bundled EQEmu index:
+
+```json
+{
+  "schema_version": 1,
+  "pack": "eq-legends-community",
+  "game_data_version": "2026-07",
+  "mappings": [
+    {
+      "zone": "Example Zone",
+      "npc": "example_npc",
+      "voice": "bm_george"
+    }
+  ]
+}
+```
+
+Payload rules:
+
+- Store only metadata, normalized identifiers, and Kokoro voice IDs—no audio.
+- Keep it minified for download; compression at transport time is sufficient.
+- Validate `schema_version`, non-empty fields, duplicate keys, and voice IDs.
+- Ignore a recommendation whose voice is unavailable and fall back normally.
+- Ignore entries already known to the bundled index in the first version.
+- A malformed optional payload should produce a visible warning and leave the
+  built-in resolver usable.
+- Loading a new payload must not alter an existing speaker assignment.
+- Do not add automatic networking or updates to the MVP. The user downloads
+  and installs the payload explicitly.
+
+This schema is also the bridge to the future collaborative website. The site
+can collect zone-qualified, selectively shared pins and publish a reviewed,
+versioned supplement without increasing the normal Mudmouth release size.
+Before supporting uploads, add an explicit export preview and consent step;
+never upload `speakers.json` wholesale.
+
+## Agent-sized implementation tasks
+
+Each task should fit in one focused change and leave the branch buildable.
+Tasks 1–4 form the core feature. Tasks 5–7 can proceed after the resolver
+contract in task 3 is stable.
+
+### Task 1 — Shared NPC identity normalization
+
+**Scope:** `src/events.rs` and focused tests.
+
+- Extract `normalize_npc_name(&str) -> String`.
+- Keep `speaker_key` output byte-for-byte compatible by adding `npc:` to the
+  shared result.
+- Add fixtures for punctuation, repeated separators, capitalization, and
+  non-ASCII characters.
+
+**Done when:** all old parser tests pass and generator/runtime callers can use
+the same public normalization function.
+
+### Task 2 — Reproducible PEQ index generator
+
+**Scope:** a non-runtime generator binary or tool, generator documentation,
+and a tiny checked-in fixture dump/export.
+
+- Implement the join-input extraction, grouping, explicit gender decoding,
+  conflict filtering, deterministic sorting, and statistics.
+- Test combined short zones that share one `long_name`, same-gender duplicate
+  NPC rows, conflicts, neuter rows, and malformed input.
+- Generate the full minified index and record its source dump identifier and
+  counts in generated metadata or adjacent documentation.
+
+**Done when:** two runs from the same dump are byte-identical and reproduce
+13,795 accepted rows for the researched dump.
+
+### Task 3 — Built-in index and pool resolver
+
+**Scope:** new `npc_voices` module, generated artifact inclusion, and unit
+tests; no speaker-store behavior changes yet.
+
+- Define typed `Gender` and lookup identity types.
+- Load the embedded index once and expose a pure lookup function.
+- Filter the live curated Kokoro catalog into feminine, masculine, and full
+  pools without duplicating the allowlist.
+- Fall back to the full pool when zone/context/data/pool is missing.
+
+**Done when:** tests cover feminine, masculine, neuter-by-omission,
+conflicting-by-omission, missing-zone, unknown NPC, and an empty gender pool.
+
+### Task 4 — Integrate resolution with speaker persistence and the TUI
+
+**Scope:** `src/speakers.rs`, `src/main.rs`, `src/tui.rs`, and regression
+tests.
+
+- Return saved persistent and pinned voices before consulting new data.
+- Use the resolved pool for first assignment, random-each-utterance, and
+  reroll.
+- Keep legacy `speakers.json` deserialization and keys unchanged.
+- Make the manual voice manager's chosen voice authoritative regardless of
+  the EQEmu gender.
+
+**Done when:** tests prove saved/pinned choices survive refreshes and restarts,
+manual reroll uses the resolved pool, random-each-line uses the resolved pool,
+and legacy JSON loads without migration.
+
+### Task 5 — Optional Legends supplement loader
+
+**Scope:** supplemental schema/parser, application-data loading, resolver
+composition, sample payload, and tests.
+
+- Load the versioned file without adding runtime networking.
+- Validate and deduplicate records.
+- Admit only NPC identities absent from the bundled EQEmu index.
+- Resolve an available exact voice before the EQEmu/full-pool fallback.
+- Report malformed files and unavailable voices without stopping Mudmouth.
+
+**Done when:** an unknown Legends NPC can receive a downloaded exact mapping,
+while known EQEmu NPCs and all existing local assignments remain unchanged.
+
+### Task 6 — Size and packaging guardrail
+
+**Scope:** `scripts/package-release.sh` or a companion reporting script and
+release documentation.
+
+- Confirm the raw PEQ dump, generator inputs, and fixtures are absent from
+  archives.
+- Report generated index size, executable size, and `.zip`/`.tar.gz` sizes.
+- Record the before/after archive delta in the change description.
+- Add a generous regression threshold only after a real baseline exists.
+
+**Done when:** Linux and Windows archive contents are inspected and the
+feature's distribution cost is explicit.
+
+### Task 7 — Playtest handoff
+
+**Scope:** a short manual checklist and optional diagnostic activity messages.
+
+- Show enough non-sensitive resolver information to distinguish saved,
+  supplement, EQEmu-feminine, EQEmu-masculine, and full-pool decisions.
+- Provide known test NPCs/zones from the generated fixture or index.
+- Do not add automated game-driving or audio snapshot tests.
+
+**Done when:** the user can verify the scenarios below during normal play and
+report the NPC, zone, expected class, actual voice, and result.
+
+## Playtest checklist
+
+The user will perform final in-game verification:
+
+1. Enter a zone with a known feminine NPC and confirm the first assignment is
+   an available `af_*` or `bf_*` voice.
+2. Repeat with a known masculine NPC and confirm `am_*` or `bm_*`.
+3. Speak with a neuter or conflicting NPC and confirm it can use the full
+   curated pool.
+4. Speak with an EQ Legends NPC absent from PEQ and confirm normal full-pool
+   fallback without a supplement.
+5. Install a supplement containing that NPC and, before it has a saved local
+   assignment, confirm the recommended voice is used.
+6. Pin a deliberately cross-gender voice, restart Mudmouth, and confirm the
+   pin remains unchanged.
+7. Reroll a known NPC and confirm the new voice stays in its resolved pool.
+8. Enable random-each-line and confirm every selected voice stays in the
+   resolved pool.
+9. Update or replace the index/supplement and confirm previously saved voices
+   do not change.
+10. Visit two zones containing the same display name and note whether the
+    current global speaker identity is confusing enough to justify a future
+    zone-specific override feature.
+
+## Acceptance criteria
+
+- A known `(display zone, normalized NPC name)` with unanimous feminine or
+  masculine PEQ records receives a voice from the matching available pool.
+- Unknown, neuter, conflicting, missing-zone, and empty-subpool cases use the
+  complete available curated pool.
+- Existing persistent voices and pins are never replaced by generated data or
+  supplemental payloads.
+- Pin, reroll, and random-each-line remain available in the voice manager.
+- Legacy `speakers.json` files load without migration.
+- An optional small Legends payload can map NPCs absent from EQEmu without
+  adding audio assets or networking to the main release.
+- The ProjectEQ dump and database tooling are not present in release archives.
+- Automated tests pass, archive size impact is reported, and final behavioral
+  verification is performed through the user's playtest checklist.
+
+## Later roadmap: collaborative voice choices
+
+After the local supplement format has proven stable:
+
+1. Add an export preview for selected pinned mappings only, including zone,
+   normalized NPC name, voice ID, schema version, and game-data version.
+2. Build a website/API that accepts individual submissions, tracks votes and
+   provenance, and moderates invalid zone/NPC/voice combinations.
+3. Generate a reviewed Legends-only supplement from accepted records.
+4. Let users explicitly download/import a versioned payload with checksum and
+   source information.
+5. Consider an opt-in recommendation pack for known EQEmu NPCs. Local pins and
+   existing assignments must remain higher precedence.
+
+Keeping collection, publication, and download outside the executable prevents
+the collaborative roadmap from increasing the normal build size or making
+Mudmouth dependent on the website.

scripts/package-release.sh

@@ -6,6 +6,35 @@ usage() {
   exit 2
 }
 
+file_size() {
+  wc -c < "$1" | tr -d '[:space:]'
+}
+
+expected_archive_contents() {
+  local root=$1 executable=$2
+  printf '%s/\n' "$root"
+  printf '%s/%s\n' "$root" "$executable"
+  printf '%s/%s\n' "$root" README.md
+  printf '%s/%s\n' "$root" LICENSE
+  printf '%s/%s\n' "$root" config.toml
+}
+
+verify_archive_contents() {
+  local archive=$1 root=$2 executable=$3 contents
+  case "$archive" in
+    *.tar.gz) contents=$(tar -tzf "$archive") ;;
+    *.zip) contents=$(zipinfo -1 "$archive") ;;
+    *) echo "Unsupported release archive: $archive" >&2; exit 1 ;;
+  esac
+
+  if ! diff -u \
+    <(expected_archive_contents "$root" "$executable" | sort) \
+    <(printf '%s\n' "$contents" | sort); then
+    echo "Release archive contains unexpected or missing files: $archive" >&2
+    exit 1
+  fi
+}
+
 linux_binary=
 windows_binary=
 while (($#)); do
@@ -22,6 +51,9 @@ done
 version=$(awk -F '"' '$1 ~ /^version = / { print $2; exit }' Cargo.toml)
 [[ -n "$version" ]] || { echo "Could not read the package version." >&2; exit 1; }
 
+index=generated/peq-npc-gender-index.json
+[[ -f "$index" ]] || { echo "Missing generated PEQ index: $index" >&2; exit 1; }
+
 dist=dist
 stage=$(mktemp -d)
 repository_root=$PWD
@@ -49,6 +81,25 @@ cp README.md LICENSE config.toml "$stage/$windows_root/"
   zip -q -r "$repository_root/$dist/${windows_root}.zip" "$windows_root"
 )
 
+linux_archive="$dist/mudmouth-v${version}-x86_64-unknown-linux-gnu.tar.gz"
+windows_archive="$dist/${windows_root}.zip"
+verify_archive_contents "$linux_archive" "mudmouth-v${version}-x86_64-unknown-linux-gnu" mudmouth
+verify_archive_contents "$windows_archive" "$windows_root" mudmouth.exe
+
+size_report="$dist/SIZE-REPORT.txt"
+{
+  echo "Mudmouth release size report"
+  echo
+  printf 'Generated PEQ NPC index: %s bytes\n' "$(file_size "$index")"
+  printf 'Linux executable: %s bytes\n' "$(file_size "$linux_binary")"
+  printf 'Linux .tar.gz: %s bytes\n' "$(file_size "$linux_archive")"
+  printf 'Windows executable: %s bytes\n' "$(file_size "$windows_binary")"
+  printf 'Windows .zip: %s bytes\n' "$(file_size "$windows_archive")"
+  echo
+  echo "Archive contents: verified against the release allowlist (binary, README, LICENSE, config.toml)."
+  echo "PEQ dumps, generator inputs, fixtures, database files, and source code are absent."
+} > "$size_report"
+
 (
   cd "$dist"
   sha256sum -- *.tar.gz *.zip > SHA256SUMS.txt
@@ -56,3 +107,5 @@ cp README.md LICENSE config.toml "$stage/$windows_root/"
 
 printf 'Created release assets in %s:\n' "$dist"
 cat "$dist/SHA256SUMS.txt"
+printf '\n'
+cat "$size_report"

shell.nix

@@ -8,6 +8,10 @@ pkgs.mkShell {
     clippy
     pkg-config
     alsa-lib
+    # The ProjectEQ dump is a MariaDB SQL dump.  This provides mariadbd,
+    # mariadb-install-db, and the mariadb/mysql client aliases.
+    mariadb
+    unzip
   ];
 
   PKG_CONFIG_PATH = "${pkgs.alsa-lib.dev}/lib/pkgconfig";

src/bin/generate_peq_npc_index.rs

@@ -0,0 +1,420 @@
+//! Generate Mudmouth's compact NPC gender lookup from a ProjectEQ SQL dump.
+//!
+//! This is deliberately a build-time tool. It reads only the four tables used
+//! by the lookup and does not add a database dependency to the application.
+
+use anyhow::{Context, Result, bail};
+use clap::Parser;
+use serde::Serialize;
+use std::{
+    collections::{BTreeMap, BTreeSet, HashMap},
+    fs::{self, File},
+    io::{BufRead, BufReader},
+    path::{Path, PathBuf},
+    process::{Command, Stdio},
+};
+
+#[allow(dead_code)]
+#[path = "../events.rs"]
+mod events;
+
+#[derive(Debug, Parser)]
+#[command(about = "Generate a minified NPC gender index from a PEQ SQL dump")]
+struct Cli {
+    /// PEQ create_tables_content.sql file, or a .zip containing that file.
+    #[arg(long)]
+    dump: PathBuf,
+
+    /// Destination for the generated minified JSON index.
+    #[arg(long)]
+    output: PathBuf,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Serialize)]
+#[serde(rename_all = "lowercase")]
+enum Gender {
+    Masculine,
+    Feminine,
+    Neuter,
+}
+
+impl Gender {
+    fn from_database(value: &str) -> Result<Self> {
+        match value {
+            "0" => Ok(Self::Masculine),
+            "1" => Ok(Self::Feminine),
+            "2" => Ok(Self::Neuter),
+            _ => bail!("invalid npc_types.gender {value:?}; expected 0, 1, or 2"),
+        }
+    }
+}
+
+#[derive(Debug, Default)]
+struct PeqRows {
+    zones: HashMap<String, BTreeSet<String>>,
+    spawn_entries: HashMap<u32, BTreeSet<u32>>,
+    spawns: Vec<(String, u32)>,
+    npcs: HashMap<u32, (String, Gender)>,
+    seen_tables: BTreeSet<Table>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
+enum Table {
+    NpcTypes,
+    SpawnEntry,
+    Spawn2,
+    Zone,
+}
+
+impl Table {
+    fn from_insert(line: &str) -> Option<Self> {
+        [
+            ("INSERT INTO `npc_types` VALUES", Self::NpcTypes),
+            ("INSERT INTO `spawnentry` VALUES", Self::SpawnEntry),
+            ("INSERT INTO `spawn2` VALUES", Self::Spawn2),
+            ("INSERT INTO `zone` VALUES", Self::Zone),
+        ]
+        .into_iter()
+        .find_map(|(prefix, table)| line.starts_with(prefix).then_some(table))
+    }
+}
+
+#[derive(Debug, Serialize)]
+struct GeneratedIndex {
+    schema_version: u8,
+    source: String,
+    stats: Statistics,
+    entries: Vec<IndexEntry>,
+}
+
+#[derive(Debug, Default, Serialize, PartialEq, Eq)]
+struct Statistics {
+    pairs: usize,
+    masculine_only: usize,
+    feminine_only: usize,
+    neuter_only: usize,
+    conflicting: usize,
+    accepted: usize,
+}
+
+#[derive(Debug, Serialize, PartialEq, Eq)]
+struct IndexEntry(String, String, Gender);
+
+fn main() -> Result<()> {
+    let cli = Cli::parse();
+    let rows = read_dump(&cli.dump)?;
+    let index = generate_index(&rows, source_identity(&cli.dump)?);
+    let bytes = serde_json::to_vec(&index).context("could not serialize PEQ index")?;
+    fs::write(&cli.output, &bytes)
+        .with_context(|| format!("could not write {}", cli.output.display()))?;
+
+    println!("source: {}", index.source);
+    println!(
+        "pairs: {} (masculine: {}, feminine: {}, neuter: {}, conflicting: {}, accepted: {})",
+        index.stats.pairs,
+        index.stats.masculine_only,
+        index.stats.feminine_only,
+        index.stats.neuter_only,
+        index.stats.conflicting,
+        index.stats.accepted,
+    );
+    println!("output: {} bytes ({})", bytes.len(), cli.output.display());
+    Ok(())
+}
+
+fn source_identity(path: &Path) -> Result<String> {
+    path.file_name()
+        .and_then(|name| name.to_str())
+        .map(str::to_owned)
+        .context("dump path must have a UTF-8 file name")
+}
+
+fn read_dump(path: &Path) -> Result<PeqRows> {
+    if path.extension().is_some_and(|extension| extension == "zip") {
+        let mut child = Command::new("unzip")
+            .args([
+                "-p",
+                &path.to_string_lossy(),
+                "peq-dump/create_tables_content.sql",
+            ])
+            .stdout(Stdio::piped())
+            .spawn()
+            .context("could not run unzip; install it or provide the extracted SQL dump")?;
+        let stdout = child
+            .stdout
+            .take()
+            .context("unzip did not provide stdout")?;
+        let rows = parse_sql(BufReader::new(stdout));
+        let status = child.wait().context("could not wait for unzip")?;
+        if !status.success() {
+            bail!("unzip failed while reading {}", path.display());
+        }
+        rows
+    } else {
+        let file =
+            File::open(path).with_context(|| format!("could not read {}", path.display()))?;
+        parse_sql(BufReader::new(file))
+    }
+}
+
+fn parse_sql<R: BufRead>(reader: R) -> Result<PeqRows> {
+    let mut rows = PeqRows::default();
+    let mut current_table = None;
+
+    for (line_number, line) in reader.lines().enumerate() {
+        let line = line.with_context(|| format!("could not read SQL line {}", line_number + 1))?;
+        if let Some(table) = Table::from_insert(&line) {
+            current_table = Some(table);
+            rows.seen_tables.insert(table);
+        }
+
+        if let Some(table) = current_table {
+            for fields in sql_tuples(&line).with_context(|| {
+                format!("malformed {table:?} row at SQL line {}", line_number + 1)
+            })? {
+                add_row(&mut rows, table, &fields).with_context(|| {
+                    format!("invalid {table:?} row at SQL line {}", line_number + 1)
+                })?;
+            }
+            if line.trim_end().ends_with(';') {
+                current_table = None;
+            }
+        }
+    }
+
+    for table in [
+        Table::NpcTypes,
+        Table::SpawnEntry,
+        Table::Spawn2,
+        Table::Zone,
+    ] {
+        if !rows.seen_tables.contains(&table) {
+            bail!("dump is missing INSERT rows for {table:?}");
+        }
+    }
+    Ok(rows)
+}
+
+fn add_row(rows: &mut PeqRows, table: Table, fields: &[String]) -> Result<()> {
+    match table {
+        Table::NpcTypes => {
+            let id = field_u32(fields, 0, "id")?;
+            let name = field_string(fields, 1, "name")?;
+            let gender = Gender::from_database(field(fields, 9, "gender")?)?;
+            if rows.npcs.insert(id, (name, gender)).is_some() {
+                bail!("duplicate npc_types.id {id}");
+            }
+        }
+        Table::SpawnEntry => {
+            let group = field_u32(fields, 0, "spawngroupID")?;
+            let npc = field_u32(fields, 1, "npcID")?;
+            rows.spawn_entries.entry(group).or_default().insert(npc);
+        }
+        Table::Spawn2 => {
+            let group = field_u32(fields, 1, "spawngroupID")?;
+            let zone = field_string(fields, 2, "zone")?;
+            if !zone.is_empty() {
+                rows.spawns.push((zone, group));
+            }
+        }
+        Table::Zone => {
+            let short_name = field_string(fields, 3, "short_name")?;
+            let long_name = field_string(fields, 4, "long_name")?;
+            if !short_name.is_empty() {
+                rows.zones.entry(short_name).or_default().insert(long_name);
+            }
+        }
+    }
+    Ok(())
+}
+
+fn field<'a>(fields: &'a [String], index: usize, name: &str) -> Result<&'a str> {
+    fields
+        .get(index)
+        .map(String::as_str)
+        .with_context(|| format!("missing {name} column"))
+}
+
+fn field_u32(fields: &[String], index: usize, name: &str) -> Result<u32> {
+    field(fields, index, name)?
+        .parse()
+        .with_context(|| format!("{name} is not an unsigned integer"))
+}
+
+fn field_string(fields: &[String], index: usize, name: &str) -> Result<String> {
+    let value = field(fields, index, name)?;
+    if value == "NULL" {
+        bail!("{name} may not be NULL");
+    }
+    Ok(value.to_owned())
+}
+
+fn generate_index(rows: &PeqRows, source: String) -> GeneratedIndex {
+    let mut candidates = BTreeMap::<(String, String), BTreeSet<Gender>>::new();
+    for (short_zone, group) in &rows.spawns {
+        let Some(long_zones) = rows.zones.get(short_zone) else {
+            continue;
+        };
+        let Some(npc_ids) = rows.spawn_entries.get(group) else {
+            continue;
+        };
+        for npc_id in npc_ids {
+            let Some((name, gender)) = rows.npcs.get(npc_id) else {
+                continue;
+            };
+            if name.is_empty() {
+                continue;
+            }
+            let normalized_name = events::normalize_npc_name(name);
+            if normalized_name.is_empty() {
+                continue;
+            }
+            for long_zone in long_zones {
+                candidates
+                    .entry((long_zone.clone(), normalized_name.clone()))
+                    .or_default()
+                    .insert(*gender);
+            }
+        }
+    }
+
+    let mut stats = Statistics {
+        pairs: candidates.len(),
+        ..Default::default()
+    };
+    let mut entries = Vec::new();
+    for ((zone, name), genders) in candidates {
+        match genders
+            .iter()
+            .next()
+            .copied()
+            .filter(|_| genders.len() == 1)
+        {
+            Some(Gender::Masculine) => {
+                stats.masculine_only += 1;
+                stats.accepted += 1;
+                entries.push(IndexEntry(zone, name, Gender::Masculine));
+            }
+            Some(Gender::Feminine) => {
+                stats.feminine_only += 1;
+                stats.accepted += 1;
+                entries.push(IndexEntry(zone, name, Gender::Feminine));
+            }
+            Some(Gender::Neuter) => stats.neuter_only += 1,
+            None => stats.conflicting += 1,
+        }
+    }
+
+    GeneratedIndex {
+        schema_version: 1,
+        source,
+        stats,
+        entries,
+    }
+}
+
+/// Parses the SQL tuple syntax emitted by `mariadb-dump`.
+fn sql_tuples(line: &str) -> Result<Vec<Vec<String>>> {
+    let mut tuples = Vec::new();
+    let mut fields = Vec::new();
+    let mut field = String::new();
+    let mut in_tuple = false;
+    let mut in_string = false;
+    let mut escaped = false;
+
+    for character in line.chars() {
+        if !in_tuple {
+            if character == '(' {
+                in_tuple = true;
+                fields.clear();
+                field.clear();
+            }
+            continue;
+        }
+
+        if escaped {
+            field.push(match character {
+                '0' => '\0',
+                'b' => '\u{0008}',
+                'n' => '\n',
+                'r' => '\r',
+                't' => '\t',
+                'Z' => '\u{001a}',
+                other => other,
+            });
+            escaped = false;
+        } else if in_string && character == '\\' {
+            escaped = true;
+        } else if character == '\'' {
+            in_string = !in_string;
+        } else if !in_string && character == ',' {
+            fields.push(std::mem::take(&mut field));
+        } else if !in_string && character == ')' {
+            fields.push(std::mem::take(&mut field));
+            tuples.push(std::mem::take(&mut fields));
+            in_tuple = false;
+        } else {
+            field.push(character);
+        }
+    }
+
+    if in_tuple || in_string || escaped {
+        bail!("unterminated SQL tuple");
+    }
+    Ok(tuples)
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    const FIXTURE: &str = include_str!("../../tests/fixtures/peq-small.sql");
+
+    #[test]
+    fn groups_short_zones_and_filters_ambiguous_gender() {
+        let rows = parse_sql(BufReader::new(FIXTURE.as_bytes())).unwrap();
+        let index = generate_index(&rows, "fixture.sql".into());
+
+        assert_eq!(
+            index.entries,
+            vec![
+                IndexEntry(
+                    "Shared Zone".into(),
+                    "captain_bob".into(),
+                    Gender::Masculine
+                ),
+                IndexEntry("Shared Zone".into(), "lady_ada".into(), Gender::Feminine),
+            ]
+        );
+        assert_eq!(
+            index.stats,
+            Statistics {
+                pairs: 4,
+                masculine_only: 1,
+                feminine_only: 1,
+                neuter_only: 1,
+                conflicting: 1,
+                accepted: 2,
+            }
+        );
+    }
+
+    #[test]
+    fn output_is_minified_and_deterministic() {
+        let rows = parse_sql(BufReader::new(FIXTURE.as_bytes())).unwrap();
+        let first = serde_json::to_vec(&generate_index(&rows, "fixture.sql".into())).unwrap();
+        let second = serde_json::to_vec(&generate_index(&rows, "fixture.sql".into())).unwrap();
+
+        assert_eq!(first, second);
+        assert!(!first.contains(&b'\n'));
+    }
+
+    #[test]
+    fn rejects_malformed_rows_and_unknown_gender_values() {
+        let malformed = "INSERT INTO `npc_types` VALUES\n(1,'too short');\n";
+        assert!(parse_sql(BufReader::new(malformed.as_bytes())).is_err());
+
+        let invalid_gender = "INSERT INTO `npc_types` VALUES\n(1,'Bad','','',0,0,0,0,0,9);\n";
+        assert!(parse_sql(BufReader::new(invalid_gender.as_bytes())).is_err());
+    }
+}

src/events.rs

@@ -75,7 +75,11 @@ fn parse_npc_says(message: &str) -> Option<(String, String)> {
     Some((speaker.to_string(), text.to_string()))
 }
 
-pub fn speaker_key(name: &str) -> String {
+/// Normalizes an EverQuest NPC name for use in stable lookup identities.
+///
+/// Non-ASCII and punctuation characters are separators, and repeated
+/// separators collapse to a single underscore.
+pub fn normalize_npc_name(name: &str) -> String {
     let normalized = name
         .chars()
         .flat_map(char::to_lowercase)
@@ -88,12 +92,15 @@ pub fn speaker_key(name: &str) -> String {
         })
         .collect::<String>();
     let normalized = normalized.trim_matches('_');
-    let compact = normalized
+    normalized
         .split('_')
         .filter(|part| !part.is_empty())
         .collect::<Vec<_>>()
-        .join("_");
-    format!("npc:{compact}")
+        .join("_")
+}
+
+pub fn speaker_key(name: &str) -> String {
+    format!("npc:{}", normalize_npc_name(name))
 }
 
 #[cfg(test)]
@@ -140,4 +147,19 @@ mod tests {
         assert_eq!(speaker_key("A lizardman healer"), "npc:a_lizardman_healer");
         assert_eq!(speaker_key("Zok  Caropni!"), "npc:zok_caropni");
     }
+
+    #[test]
+    fn normalizes_npc_names_for_shared_lookup() {
+        let fixtures = [
+            ("Zok Caropni!", "zok_caropni"),
+            ("a---lizardman__healer", "a_lizardman_healer"),
+            ("MiXeD CaSe NPC", "mixed_case_npc"),
+            ("F\u{00e9}lix the C\u{00e1}t", "f_lix_the_c_t"),
+        ];
+
+        for (name, expected) in fixtures {
+            assert_eq!(normalize_npc_name(name), expected, "{name}");
+            assert_eq!(speaker_key(name), format!("npc:{expected}"), "{name}");
+        }
+    }
 }

src/main.rs

@@ -2,6 +2,7 @@ mod config;
 mod eq_discovery;
 mod events;
 mod kokoro;
+mod npc_voices;
 mod speakers;
 mod tui;
 mod workers;
@@ -65,6 +66,7 @@ async fn main() -> Result<()> {
         config.kokoro_url = url;
     }
 
+    let _ = npc_voices::builtin_index();
     let speakers = speakers::SpeakerStore::load()?;
     let (worker_sender, worker_receiver) = mpsc::channel::<WorkerEvent>();
     let audio_sender = spawn_audio_worker(config.volume, worker_sender.clone());
@@ -166,6 +168,7 @@ fn process_worker_event(
                     speaker.clone(),
                     &app.voices,
                     app.current_zone.clone(),
+                    npc_voices::builtin_index(),
                 ) {
                     app.speakers.save()?;
                     if synthesis_sender

src/npc_voices.rs

@@ -0,0 +1,235 @@
+//! Built-in EQEmu NPC gender lookup and Kokoro voice-pool resolution.
+//!
+//! This module intentionally only describes a suitable *pool* of voices. The
+//! speaker store remains responsible for selecting and persisting an exact
+//! voice.
+
+use crate::events::normalize_npc_name;
+use serde::Deserialize;
+use std::{collections::BTreeMap, sync::LazyLock};
+
+const BUILTIN_INDEX_BYTES: &[u8] = include_bytes!(concat!(
+    env!("CARGO_MANIFEST_DIR"),
+    "/generated/peq-npc-gender-index.json"
+));
+
+/// The two gender traits that can narrow an NPC's initial voice pool.
+#[derive(Debug, Clone, Copy, Deserialize, PartialEq, Eq)]
+#[serde(rename_all = "lowercase")]
+pub enum Gender {
+    Masculine,
+    Feminine,
+}
+
+/// A zone-qualified NPC identity used by the generated EQEmu lookup.
+///
+/// `zone` is deliberately preserved exactly as it appears in the EverQuest
+/// log. NPC names use the same normalization as speaker-store keys.
+#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord)]
+pub struct LookupIdentity {
+    zone: String,
+    npc: String,
+}
+
+impl LookupIdentity {
+    pub fn new(zone: impl Into<String>, npc: &str) -> Option<Self> {
+        let zone = zone.into();
+        let npc = normalize_npc_name(npc);
+        if zone.is_empty() || npc.is_empty() {
+            return None;
+        }
+        Some(Self { zone, npc })
+    }
+
+    pub fn from_observed(zone: Option<&str>, npc: &str) -> Option<Self> {
+        Self::new(zone?, npc)
+    }
+}
+
+/// A deserialized generated EQEmu lookup.
+#[derive(Debug, Default)]
+pub struct NpcVoiceIndex {
+    entries: BTreeMap<LookupIdentity, Gender>,
+}
+
+impl NpcVoiceIndex {
+    pub fn lookup(&self, identity: &LookupIdentity) -> Option<Gender> {
+        self.entries.get(identity).copied()
+    }
+
+    /// Resolves the appropriate live voice pool for an NPC.
+    ///
+    /// An unknown identity, missing zone, or empty gender-specific pool all
+    /// deliberately fall back to the complete live curated catalog.
+    pub fn resolve_pool(
+        &self,
+        identity: Option<&LookupIdentity>,
+        live_curated_catalog: &[String],
+    ) -> Vec<String> {
+        VoicePools::from_live_curated_catalog(live_curated_catalog)
+            .for_gender(identity.and_then(|identity| self.lookup(identity)))
+            .to_vec()
+    }
+
+    fn from_bytes(bytes: &[u8]) -> Result<Self, serde_json::Error> {
+        let generated: GeneratedIndex = serde_json::from_slice(bytes)?;
+        assert_eq!(
+            generated.schema_version, 1,
+            "unsupported bundled PEQ index schema"
+        );
+
+        let entries = generated
+            .entries
+            .into_iter()
+            .map(|GeneratedEntry(zone, npc, gender)| {
+                (
+                    LookupIdentity::new(zone, &npc)
+                        .expect("bundled PEQ index contains an empty lookup identity"),
+                    gender,
+                )
+            })
+            .collect();
+        Ok(Self { entries })
+    }
+}
+
+#[derive(Debug, Deserialize)]
+struct GeneratedIndex {
+    schema_version: u8,
+    entries: Vec<GeneratedEntry>,
+}
+
+#[derive(Debug, Deserialize)]
+struct GeneratedEntry(String, String, Gender);
+
+static BUILTIN_INDEX: LazyLock<NpcVoiceIndex> = LazyLock::new(|| {
+    NpcVoiceIndex::from_bytes(BUILTIN_INDEX_BYTES)
+        .expect("bundled PEQ index must be valid generated JSON")
+});
+
+/// Returns the process-wide, once-deserialized built-in EQEmu lookup.
+pub fn builtin_index() -> &'static NpcVoiceIndex {
+    &BUILTIN_INDEX
+}
+
+/// Separates the already-curated live Kokoro catalog into usable pools.
+///
+/// The Kokoro client owns the curated allowlist. Keeping it out of this module
+/// ensures this resolver cannot drift from the catalog shown to the user.
+#[derive(Debug, Clone, Default, PartialEq, Eq)]
+pub struct VoicePools {
+    full: Vec<String>,
+    feminine: Vec<String>,
+    masculine: Vec<String>,
+}
+
+impl VoicePools {
+    pub fn from_live_curated_catalog(live_curated_catalog: &[String]) -> Self {
+        let mut pools = Self {
+            full: live_curated_catalog.to_vec(),
+            ..Self::default()
+        };
+        for voice in live_curated_catalog {
+            if voice.starts_with("af_") || voice.starts_with("bf_") {
+                pools.feminine.push(voice.clone());
+            } else if voice.starts_with("am_") || voice.starts_with("bm_") {
+                pools.masculine.push(voice.clone());
+            }
+        }
+        pools
+    }
+
+    pub fn for_gender(&self, gender: Option<Gender>) -> &[String] {
+        match gender {
+            Some(Gender::Feminine) if !self.feminine.is_empty() => &self.feminine,
+            Some(Gender::Masculine) if !self.masculine.is_empty() => &self.masculine,
+            _ => &self.full,
+        }
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    fn test_index(entries: &[(&str, &str, Gender)]) -> NpcVoiceIndex {
+        NpcVoiceIndex {
+            entries: entries
+                .iter()
+                .map(|(zone, npc, gender)| (LookupIdentity::new(*zone, npc).unwrap(), *gender))
+                .collect(),
+        }
+    }
+
+    fn catalog() -> Vec<String> {
+        ["af_heart", "bf_alice", "am_adam", "bm_george"]
+            .into_iter()
+            .map(str::to_owned)
+            .collect()
+    }
+
+    #[test]
+    fn feminine_and_masculine_npcs_use_matching_pools() {
+        let index = test_index(&[
+            ("Test Zone", "Lady Ada", Gender::Feminine),
+            ("Test Zone", "Captain Bob", Gender::Masculine),
+        ]);
+        let voices = catalog();
+
+        let lady = LookupIdentity::new("Test Zone", "Lady Ada").unwrap();
+        assert_eq!(
+            index.resolve_pool(Some(&lady), &voices),
+            vec!["af_heart", "bf_alice"]
+        );
+
+        let captain = LookupIdentity::new("Test Zone", "Captain Bob").unwrap();
+        assert_eq!(
+            index.resolve_pool(Some(&captain), &voices),
+            vec!["am_adam", "bm_george"]
+        );
+    }
+
+    #[test]
+    fn omitted_neuter_and_conflicting_entries_use_the_full_pool() {
+        let index = test_index(&[]);
+        let voices = catalog();
+
+        for npc in ["Neuter Construct", "Conflicting Guard"] {
+            let identity = LookupIdentity::new("Test Zone", npc).unwrap();
+            assert_eq!(index.resolve_pool(Some(&identity), &voices), voices);
+        }
+    }
+
+    #[test]
+    fn missing_zone_and_unknown_npc_use_the_full_pool() {
+        let index = test_index(&[("Test Zone", "Lady Ada", Gender::Feminine)]);
+        let voices = catalog();
+
+        assert_eq!(index.resolve_pool(None, &voices), voices);
+        let unknown = LookupIdentity::new("Test Zone", "Unknown NPC").unwrap();
+        assert_eq!(index.resolve_pool(Some(&unknown), &voices), voices);
+        assert!(LookupIdentity::from_observed(None, "Lady Ada").is_none());
+    }
+
+    #[test]
+    fn an_empty_gender_pool_falls_back_to_the_full_pool() {
+        let index = test_index(&[("Test Zone", "Lady Ada", Gender::Feminine)]);
+        let voices = vec!["am_adam".into()];
+        let lady = LookupIdentity::new("Test Zone", "Lady Ada").unwrap();
+
+        assert_eq!(index.resolve_pool(Some(&lady), &voices), voices);
+    }
+
+    #[test]
+    fn embedded_index_is_available_for_known_npcs() {
+        let identity = LookupIdentity::new("West Commonlands", "Bealya Tanilsuia").unwrap();
+        assert_eq!(builtin_index().lookup(&identity), Some(Gender::Feminine));
+    }
+
+    #[test]
+    fn identity_normalizes_npc_names_but_preserves_display_zone() {
+        let identity = LookupIdentity::new("A Zone", "Zok  Caropni!").unwrap();
+        assert_eq!(identity.zone, "A Zone");
+        assert_eq!(identity.npc, "zok_caropni");
+    }
+}

src/speakers.rs

@@ -1,4 +1,7 @@
-use crate::config::{Config, atomic_write};
+use crate::{
+    config::{Config, atomic_write},
+    npc_voices::{LookupIdentity, NpcVoiceIndex},
+};
 use anyhow::{Context, Result};
 use chrono::{DateTime, Local};
 use rand::prelude::IndexedRandom;
@@ -61,26 +64,39 @@ impl SpeakerStore {
         name: String,
         voices: &[String],
         zone: Option<String>,
+        index: &NpcVoiceIndex,
     ) -> Option<String> {
         let now = Local::now();
-        let speaker = self.speakers.entry(key).or_insert_with(|| {
-            let voice = choose_voice(voices).unwrap_or_else(|| "af_heart".into());
+        if let Some(speaker) = self.speakers.get_mut(&key) {
+            speaker.last_seen = now;
+            speaker.last_zone = zone;
+            return match &speaker.mode {
+                VoiceMode::PersistentRandom { voice } | VoiceMode::Pinned { voice } => {
+                    Some(voice.clone())
+                }
+                VoiceMode::RandomEachUtterance => choose_voice(&resolved_pool(
+                    index,
+                    &name,
+                    speaker.last_zone.as_deref(),
+                    voices,
+                )),
+            };
+        }
+
+        let voice = choose_voice(&resolved_pool(index, &name, zone.as_deref(), voices))
+            .unwrap_or_else(|| "af_heart".into());
+        self.speakers.insert(
+            key,
             Speaker {
                 name,
-                mode: VoiceMode::PersistentRandom { voice },
+                mode: VoiceMode::PersistentRandom {
+                    voice: voice.clone(),
+                },
                 last_seen: now,
-                last_zone: zone.clone(),
-            }
-        });
-
-        speaker.last_seen = now;
-        speaker.last_zone = zone;
-        match &speaker.mode {
-            VoiceMode::PersistentRandom { voice } | VoiceMode::Pinned { voice } => {
-                Some(voice.clone())
-            }
-            VoiceMode::RandomEachUtterance => choose_voice(voices),
-        }
+                last_zone: zone,
+            },
+        );
+        Some(voice)
     }
 
     pub fn entries(&self) -> impl Iterator<Item = (&String, &Speaker)> {
@@ -103,13 +119,30 @@ impl SpeakerStore {
         }
     }
 
-    pub fn reset_persistent_random(&mut self, key: &str, voices: &[String]) {
-        if let (Some(speaker), Some(voice)) = (self.speakers.get_mut(key), choose_voice(voices)) {
+    pub fn reset_persistent_random(&mut self, key: &str, voices: &[String], index: &NpcVoiceIndex) {
+        if let Some(speaker) = self.speakers.get_mut(key)
+            && let Some(voice) = choose_voice(&resolved_pool(
+                index,
+                &speaker.name,
+                speaker.last_zone.as_deref(),
+                voices,
+            ))
+        {
             speaker.mode = VoiceMode::PersistentRandom { voice };
         }
     }
 }
 
+fn resolved_pool(
+    index: &NpcVoiceIndex,
+    name: &str,
+    zone: Option<&str>,
+    voices: &[String],
+) -> Vec<String> {
+    let identity = LookupIdentity::from_observed(zone, name);
+    index.resolve_pool(identity.as_ref(), voices)
+}
+
 fn choose_voice(voices: &[String]) -> Option<String> {
     voices.choose(&mut rand::rng()).cloned()
 }
@@ -117,25 +150,147 @@ fn choose_voice(voices: &[String]) -> Option<String> {
 #[cfg(test)]
 mod tests {
     use super::*;
+    use crate::npc_voices::builtin_index;
 
-    #[test]
-    fn random_assignment_is_persisted() {
-        let mut store = SpeakerStore::default();
-        let voices = vec!["af_heart".into(), "am_adam".into()];
-        let first = store.resolve_voice("npc:turga".into(), "Turga".into(), &voices, None);
-        let next = store.resolve_voice("npc:turga".into(), "Turga".into(), &voices, None);
-        assert_eq!(first, next);
+    const FEMALE_NPC: &str = "Bealya Tanilsuia";
+    const FEMALE_ZONE: &str = "West Commonlands";
+
+    fn catalog() -> Vec<String> {
+        ["af_heart", "bf_alice", "am_adam", "bm_george"]
+            .into_iter()
+            .map(str::to_owned)
+            .collect()
+    }
+
+    fn resolve(
+        store: &mut SpeakerStore,
+        key: &str,
+        name: &str,
+        voices: &[String],
+        zone: Option<&str>,
+    ) -> Option<String> {
+        store.resolve_voice(
+            key.into(),
+            name.into(),
+            voices,
+            zone.map(str::to_owned),
+            builtin_index(),
+        )
+    }
+
+    fn is_feminine(voice: &str) -> bool {
+        voice.starts_with("af_") || voice.starts_with("bf_")
     }
 
     #[test]
-    fn random_each_utterance_remains_configurable() {
+    fn saved_persistent_and_pinned_voices_survive_catalog_refreshes_and_restarts() {
         let mut store = SpeakerStore::default();
-        let voices = vec!["af_heart".into()];
-        store.resolve_voice("npc:turga".into(), "Turga".into(), &voices, None);
-        store.set_random_each_utterance("npc:turga");
+        let voices = catalog();
+        let persistent = resolve(
+            &mut store,
+            "npc:bealya_tanilsuia",
+            FEMALE_NPC,
+            &voices,
+            Some(FEMALE_ZONE),
+        )
+        .unwrap();
+        assert!(is_feminine(&persistent));
+
+        store.pin("npc:bealya_tanilsuia", "am_adam".into());
+        let serialized = serde_json::to_string(&store).unwrap();
+        let mut reloaded: SpeakerStore = serde_json::from_str(&serialized).unwrap();
+        let refreshed_catalog = vec!["af_heart".into()];
+
         assert_eq!(
-            store.get("npc:turga").unwrap().mode,
+            resolve(
+                &mut reloaded,
+                "npc:bealya_tanilsuia",
+                FEMALE_NPC,
+                &refreshed_catalog,
+                Some(FEMALE_ZONE),
+            ),
+            Some("am_adam".into()),
+        );
+
+        let mut persistent_store = SpeakerStore::default();
+        persistent_store.speakers.insert(
+            "npc:bealya_tanilsuia".into(),
+            Speaker {
+                name: FEMALE_NPC.into(),
+                mode: VoiceMode::PersistentRandom {
+                    voice: persistent.clone(),
+                },
+                last_seen: Local::now(),
+                last_zone: Some(FEMALE_ZONE.into()),
+            },
+        );
+        let serialized = serde_json::to_string(&persistent_store).unwrap();
+        let mut reloaded: SpeakerStore = serde_json::from_str(&serialized).unwrap();
+        assert_eq!(
+            resolve(
+                &mut reloaded,
+                "npc:bealya_tanilsuia",
+                FEMALE_NPC,
+                &refreshed_catalog,
+                Some(FEMALE_ZONE),
+            ),
+            Some(persistent),
+        );
+    }
+
+    #[test]
+    fn manual_reroll_and_random_each_utterance_use_the_resolved_pool() {
+        let mut store = SpeakerStore::default();
+        let voices = catalog();
+        resolve(
+            &mut store,
+            "npc:bealya_tanilsuia",
+            FEMALE_NPC,
+            &voices,
+            Some(FEMALE_ZONE),
+        );
+        store.reset_persistent_random("npc:bealya_tanilsuia", &voices, builtin_index());
+        assert!(matches!(
+            &store.get("npc:bealya_tanilsuia").unwrap().mode,
+            VoiceMode::PersistentRandom { voice } if is_feminine(voice)
+        ));
+
+        store.set_random_each_utterance("npc:bealya_tanilsuia");
+        assert_eq!(
+            store.get("npc:bealya_tanilsuia").unwrap().mode,
             VoiceMode::RandomEachUtterance
         );
+        for _ in 0..8 {
+            assert!(is_feminine(
+                &resolve(
+                    &mut store,
+                    "npc:bealya_tanilsuia",
+                    FEMALE_NPC,
+                    &voices,
+                    Some(FEMALE_ZONE),
+                )
+                .unwrap()
+            ));
+        }
+    }
+
+    #[test]
+    fn legacy_speaker_json_loads_without_migration() {
+        let legacy = r#"{
+            "speakers": {
+                "npc:turga": {
+                    "name": "Turga",
+                    "mode": {"mode": "pinned", "voice": "am_adam"},
+                    "last_seen": "2026-07-17T20:10:29+00:00",
+                    "last_zone": null
+                }
+            }
+        }"#;
+        let mut store: SpeakerStore = serde_json::from_str(legacy).unwrap();
+
+        assert_eq!(
+            resolve(&mut store, "npc:turga", "Turga", &["af_heart".into()], None,),
+            Some("am_adam".into()),
+        );
     }
 }

src/tui.rs

@@ -1,6 +1,7 @@
 use crate::{
     config::Config,
     eq_discovery::{CharacterLog, discover_character_logs},
+    npc_voices::builtin_index,
     speakers::SpeakerStore,
     workers::{KokoroStatus, WorkerEvent},
 };
@@ -344,7 +345,8 @@ impl App {
             }
             KeyCode::Char('x') => {
                 if let Some(key) = keys.get(self.selected_speaker) {
-                    self.speakers.reset_persistent_random(key, &self.voices);
+                    self.speakers
+                        .reset_persistent_random(key, &self.voices, builtin_index());
                     return Some(UiAction::ConfigChanged);
                 }
             }

tests/fixtures/peq-small.sql

@@ -0,0 +1,14 @@
+-- Minimal mariadb-dump-shaped PEQ fixture for the NPC gender index generator.
+INSERT INTO `npc_types` VALUES
+(1,'Captain Bob','','',0,0,0,0,0,0),
+(2,'CAPTAIN---BOB','','',0,0,0,0,0,0),
+(3,'Lady Ada','','',0,0,0,0,0,1),
+(4,'Neutral Guard','','',0,0,0,0,0,2),
+(5,'Conflict','','',0,0,0,0,0,0),
+(6,'Conflict!','','',0,0,0,0,0,1);
+INSERT INTO `spawnentry` VALUES
+(10,1),(11,2),(12,3),(13,4),(14,5),(15,6);
+INSERT INTO `spawn2` VALUES
+(1,10,'north'),(2,11,'south'),(3,12,'north'),(4,13,'north'),(5,14,'north'),(6,15,'south');
+INSERT INTO `zone` VALUES
+(1,1,0,'north','Shared Zone'),(2,2,0,'south','Shared Zone');