pleb/mudmouth
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity

Generate compact classic NPC mapping from ProjectEQ data

Browse Source
This commit is contained in:
pleb
2026-07-29 10:40:45 -07:00
parent 561ace9443
commit b4e622c79b
7 changed files with 920 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/development.md
+19
View File
@@ -18,6 +18,25 @@ cargo test
cargo build --release
```
## Refreshing the PEQ NPC gender index
The checked-in `generated/peq-npc-gender-index.json` is generated data, not a
runtime database dependency. Regenerate it from an explicit ProjectEQ dump
with the tool below; it reads either `create_tables_content.sql` directly or
the `peq-dump/create_tables_content.sql` member of the supplied zip archive.
`unzip` is available in `shell.nix` for the latter.
```bash
cargo run --bin generate_peq_npc_index -- \
--dump "$HOME/Downloads/peq-1759046415.zip" \
--output generated/peq-npc-gender-index.json
```
The tool prints its input identity, complete conflict/neuter statistics, and
output size. Run it twice and compare the output hashes before updating the
checked-in artifact. The source archive and any temporary database files must
not be committed or included in release archives.
The parser tests cover the observed log formats, including NPC dialogue,
channel-chat exclusion, zone events, normalization, persistent/random speaker
voice policies, log truncation, and first-run log discovery.
docs/tasks/eqemu-npc-voice-mapping.md
+436
View File
@@ -0,0 +1,436 @@
# EQEmu-informed NPC voice mapping
## Outcome
Use the EQEmu/ProjectEQ NPC database to narrow the Kokoro voice pool for a
known EverQuest NPC. The first implementation uses the NPC's recorded gender;
race, body type, class, and appearance can refine voice character later.
This feature must remain advisory:
- Mudmouth still owns the final voice assignment.
- A user's saved or pinned voice always wins.
- Unknown, neuter, and ambiguous NPCs retain today's full-pool behavior.
- Existing `speakers.json` files remain valid and are never rewritten merely
because the bundled data changes.
- Mudmouth remains fully usable offline.
The release should contain a compact generated lookup, not the ProjectEQ dump,
a database server, generated audio, or voice model assets.
## Product decisions
### Separate traits from choices
The built-in EQEmu index contains NPC traits, initially only `masculine` or
`feminine`. It does **not** prescribe an exact Kokoro voice. Mudmouth randomly
chooses from the matching voices available on the user's Kokoro server and
persists that choice as it does today.
An exact voice ID is a preference, not an EQEmu fact. Exact choices belong in:
1. the user's local `speakers.json`; or
2. an optional, separately downloaded community/Legends mapping payload.
This division keeps generated facts reproducible while preserving subjective
casting choices.
### Keep the built-in release small
Generate a minified, sorted data artifact containing only:
```text
(display-zone, normalized NPC name) -> gender
```
Embed it in the executable with `include_bytes!` and deserialize it once at
startup. `serde_json` is already a dependency, so a first version should not
add a compression, database, or perfect-hash dependency. The existing `.zip`
and `.tar.gz` release formats will compress the repeated zone and name text.
Before merging, measure both the raw executable and final release archive
before and after the data is embedded. Optimize the encoding only if the
archive delta is material. A compact binary format is a later optimization,
not an MVP requirement.
The ProjectEQ dump and the generator's temporary MariaDB directory are build
inputs and must never enter a release archive.
### Preserve user control
The voice manager remains the correction mechanism for incorrect database
traits, unavailable community voices, and personal taste:
- **Pin** selects an exact voice and is always authoritative.
- **Reroll persistent random** chooses from the currently resolved pool.
- **Random every line** chooses from the currently resolved pool for each
utterance.
- An already persisted random voice remains unchanged after an index or
mapping-payload update.
The current speaker store is keyed only by normalized NPC name. Do not migrate
it or silently create zone-qualified speaker records in this feature. Zone is
used to improve the initial lookup, while the existing stable name-to-voice
behavior remains intact. Zone-qualified personal overrides can be considered
separately if playtesting demonstrates a real collision problem.
## Runtime resolution policy
Resolve an NPC in this order:
1. If the speaker already has a persisted exact voice (`Pinned` or
`PersistentRandom`), return it unchanged.
2. If an enabled supplemental mapping has an exact voice for the current
`(zone, NPC)` and that voice is in the live curated Kokoro catalog, use and
persist it.
3. If the built-in EQEmu index identifies the current `(zone, NPC)` as
feminine, choose from available `af_*` and `bf_*` voices.
4. If it identifies the NPC as masculine, choose from available `am_*` and
`bm_*` voices.
5. If no zone is known, the lookup is missing, the source records conflict,
the NPC is neuter, or the resolved gender pool is empty, choose from the
complete available curated pool.
For `RandomEachUtterance`, perform steps 2–5 for every line, but do not turn a
supplemental recommendation into a local pin. For a manual reroll, perform
steps 3–5 and persist the new result. A manual pin never gets replaced.
Supplemental mappings are deliberately checked before EQEmu traits so a
custom Legends NPC can receive a curated exact voice. The initial supplement
loader should reject or ignore entries already covered by the bundled EQEmu
index. This makes the first downloadable payload additive rather than a way to
override known data. A later explicit “community recommendations” mode may
allow preferences for known NPCs, but it must remain opt-in and below local
user choices.
## Lookup identity and normalization
Mudmouth knows:
- the speaker name from a `Name says, '…'` log line; and
- the current display-zone name from `You have entered …`.
The canonical lookup is therefore:
```text
(current display-zone name, normalized NPC name)
```
Use the display name exactly as observed in the log. The generator translates
EQEmu short zone names through `zone.long_name`.
Extract the name-normalization portion of `events::speaker_key` into a shared
function:
```text
lowercase
replace each non-ASCII-alphanumeric run with "_"
trim leading/trailing "_"
collapse repeated "_"
```
`speaker_key` should continue to add the `npc:` prefix for `speakers.json`.
Both the generator and runtime lookup must call the shared normalization
function so their behavior cannot drift.
## PEQ dump findings
These figures were measured from the supplied `peq-1759046415.zip` dump. Treat
the dump as an input to the generator, not application data.
- The required relationship is `spawn2.zone` -> `spawn2.spawngroupID` ->
`spawnentry.npcID` -> `npc_types.id`. `spawngroup` is not required.
- `spawn2.zone` contains a short zone name, while `zone.long_name` contains
the display label Mudmouth records.
- There are 559 distinct display labels for 482 short names. Some labels
deliberately combine multiple short zones, such as the seven Muramite
Proving Grounds zones. Union all candidates for a display label before
deciding whether their genders agree.
- Gender values in this dump are `0` masculine, `1` feminine, and `2` neuter.
Decode them explicitly; the field is not a boolean.
- After joining the spawn tables, translating zones, and applying Mudmouth
normalization, there are 37,262 `(display-zone, speaker)` pairs: 10,046
masculine-only, 3,749 feminine-only, 22,692 neuter-only, and 775 with
conflicting genders.
- The v1 gender index therefore has 13,795 eligible entries. Neuter,
conflicting, and missing entries deliberately use the full pool.
- A pair does not need to identify one `npc_types` row. Respawn variants are
eligible when every matching row agrees on the same masculine or feminine
gender.
The temporary MariaDB data directory used for this measurement was about
765 MiB. It is suitable for generation, not distribution.
## Generator design
The generator should:
1. Accept an explicit PEQ dump path and output path.
2. Read raw candidate rows from `npc_types`, `spawnentry`, `spawn2`, and
`zone`. It may invoke the `mariadb` client supplied by `shell.nix`; no
MariaDB client library should become a Mudmouth runtime dependency.
3. Normalize NPC names in Rust using the shared runtime function.
4. Group by `(zone.long_name, normalized_name)` after all short-name rows have
been combined.
5. Emit a row only if every candidate is gender `0` or every candidate is
gender `1`.
6. Decode the database number into a typed `Gender` value rather than carrying
magic numbers into runtime code.
7. Sort records deterministically and write minified output.
8. Print source identity, accepted/conflicting/neuter counts, and output byte
size so refreshes are auditable.
Check the generated artifact into the repository. Runtime and release builds
must not need MariaDB or the PEQ dump.
The following query is useful for validating results:
```sql
SELECT
z.long_name AS zone,
TRIM(BOTH '_' FROM REGEXP_REPLACE(LOWER(n.name), '[^a-z0-9]+', '_'))
AS normalized_name,
MIN(n.gender) AS gender
FROM spawn2 AS s
JOIN spawnentry AS se ON se.spawngroupID = s.spawngroupID
JOIN npc_types AS n ON n.id = se.npcID
JOIN zone AS z ON z.short_name = s.zone
WHERE s.zone <> '' AND n.name <> ''
GROUP BY z.long_name, normalized_name
HAVING COUNT(DISTINCT n.gender) = 1 AND MIN(n.gender) IN (0, 1);
```
Do not use this SQL normalization as the compatibility contract; it is only an
independent check of the Rust generator.
## Loading the supplied dump
`shell.nix` provides MariaDB and `unzip`. Start `nix-shell`, then use an
isolated server so the process cannot affect a system MariaDB instance:
```bash
peq_dir="$(mktemp -d /tmp/mudmouth-peq.XXXXXX)"
mariadb-install-db --no-defaults --datadir="$peq_dir/data" \
--auth-root-authentication-method=normal --skip-test-db
mariadbd --no-defaults --datadir="$peq_dir/data" \
--socket="$peq_dir/mariadb.sock" --pid-file="$peq_dir/mariadbd.pid" \
--skip-networking --log-error="$peq_dir/mariadbd.log" &
until mariadb-admin --no-defaults --socket="$peq_dir/mariadb.sock" \
-u root ping --silent; do sleep 1; done
mariadb --no-defaults --socket="$peq_dir/mariadb.sock" -u root \
-e 'CREATE DATABASE peq_content DEFAULT CHARACTER SET latin1;'
unzip -p ~/Downloads/peq-1759046415.zip peq-dump/create_tables_content.sql \
| mariadb --no-defaults --socket="$peq_dir/mariadb.sock" -u root peq_content
```
Only `create_tables_content.sql` is necessary. Shut the isolated server down
with:
```bash
mariadb-admin --no-defaults --socket="$peq_dir/mariadb.sock" -u root shutdown
```
Remove only the known temporary directory after the generator is finished.
## Supplemental Legends/community payload
Support a small, optional data file in Mudmouth's application-data directory,
separate from `speakers.json`. It contains exact voice recommendations for
NPCs absent from the bundled EQEmu index:
```json
{
"schema_version": 1,
"pack": "eq-legends-community",
"game_data_version": "2026-07",
"mappings": [
{
"zone": "Example Zone",
"npc": "example_npc",
"voice": "bm_george"
}
]
}
```
Payload rules:
- Store only metadata, normalized identifiers, and Kokoro voice IDs—no audio.
- Keep it minified for download; compression at transport time is sufficient.
- Validate `schema_version`, non-empty fields, duplicate keys, and voice IDs.
- Ignore a recommendation whose voice is unavailable and fall back normally.
- Ignore entries already known to the bundled index in the first version.
- A malformed optional payload should produce a visible warning and leave the
built-in resolver usable.
- Loading a new payload must not alter an existing speaker assignment.
- Do not add automatic networking or updates to the MVP. The user downloads
and installs the payload explicitly.
This schema is also the bridge to the future collaborative website. The site
can collect zone-qualified, selectively shared pins and publish a reviewed,
versioned supplement without increasing the normal Mudmouth release size.
Before supporting uploads, add an explicit export preview and consent step;
never upload `speakers.json` wholesale.
## Agent-sized implementation tasks
Each task should fit in one focused change and leave the branch buildable.
Tasks 1–4 form the core feature. Tasks 5–7 can proceed after the resolver
contract in task 3 is stable.
### Task 1 — Shared NPC identity normalization
**Scope:** `src/events.rs` and focused tests.
- Extract `normalize_npc_name(&str) -> String`.
- Keep `speaker_key` output byte-for-byte compatible by adding `npc:` to the
shared result.
- Add fixtures for punctuation, repeated separators, capitalization, and
non-ASCII characters.
**Done when:** all old parser tests pass and generator/runtime callers can use
the same public normalization function.
### Task 2 — Reproducible PEQ index generator
**Scope:** a non-runtime generator binary or tool, generator documentation,
and a tiny checked-in fixture dump/export.
- Implement the join-input extraction, grouping, explicit gender decoding,
conflict filtering, deterministic sorting, and statistics.
- Test combined short zones that share one `long_name`, same-gender duplicate
NPC rows, conflicts, neuter rows, and malformed input.
- Generate the full minified index and record its source dump identifier and
counts in generated metadata or adjacent documentation.
**Done when:** two runs from the same dump are byte-identical and reproduce
13,795 accepted rows for the researched dump.
### Task 3 — Built-in index and pool resolver
**Scope:** new `npc_voices` module, generated artifact inclusion, and unit
tests; no speaker-store behavior changes yet.
- Define typed `Gender` and lookup identity types.
- Load the embedded index once and expose a pure lookup function.
- Filter the live curated Kokoro catalog into feminine, masculine, and full
pools without duplicating the allowlist.
- Fall back to the full pool when zone/context/data/pool is missing.
**Done when:** tests cover feminine, masculine, neuter-by-omission,
conflicting-by-omission, missing-zone, unknown NPC, and an empty gender pool.
### Task 4 — Integrate resolution with speaker persistence and the TUI
**Scope:** `src/speakers.rs`, `src/main.rs`, `src/tui.rs`, and regression
tests.
- Return saved persistent and pinned voices before consulting new data.
- Use the resolved pool for first assignment, random-each-utterance, and
reroll.
- Keep legacy `speakers.json` deserialization and keys unchanged.
- Make the manual voice manager's chosen voice authoritative regardless of
the EQEmu gender.
**Done when:** tests prove saved/pinned choices survive refreshes and restarts,
manual reroll uses the resolved pool, random-each-line uses the resolved pool,
and legacy JSON loads without migration.
### Task 5 — Optional Legends supplement loader
**Scope:** supplemental schema/parser, application-data loading, resolver
composition, sample payload, and tests.
- Load the versioned file without adding runtime networking.
- Validate and deduplicate records.
- Admit only NPC identities absent from the bundled EQEmu index.
- Resolve an available exact voice before the EQEmu/full-pool fallback.
- Report malformed files and unavailable voices without stopping Mudmouth.
**Done when:** an unknown Legends NPC can receive a downloaded exact mapping,
while known EQEmu NPCs and all existing local assignments remain unchanged.
### Task 6 — Size and packaging guardrail
**Scope:** `scripts/package-release.sh` or a companion reporting script and
release documentation.
- Confirm the raw PEQ dump, generator inputs, and fixtures are absent from
archives.
- Report generated index size, executable size, and `.zip`/`.tar.gz` sizes.
- Record the before/after archive delta in the change description.
- Add a generous regression threshold only after a real baseline exists.
**Done when:** Linux and Windows archive contents are inspected and the
feature's distribution cost is explicit.
### Task 7 — Playtest handoff
**Scope:** a short manual checklist and optional diagnostic activity messages.
- Show enough non-sensitive resolver information to distinguish saved,
supplement, EQEmu-feminine, EQEmu-masculine, and full-pool decisions.
- Provide known test NPCs/zones from the generated fixture or index.
- Do not add automated game-driving or audio snapshot tests.
**Done when:** the user can verify the scenarios below during normal play and
report the NPC, zone, expected class, actual voice, and result.
## Playtest checklist
The user will perform final in-game verification:
1. Enter a zone with a known feminine NPC and confirm the first assignment is
an available `af_*` or `bf_*` voice.
2. Repeat with a known masculine NPC and confirm `am_*` or `bm_*`.
3. Speak with a neuter or conflicting NPC and confirm it can use the full
curated pool.
4. Speak with an EQ Legends NPC absent from PEQ and confirm normal full-pool
fallback without a supplement.
5. Install a supplement containing that NPC and, before it has a saved local
assignment, confirm the recommended voice is used.
6. Pin a deliberately cross-gender voice, restart Mudmouth, and confirm the
pin remains unchanged.
7. Reroll a known NPC and confirm the new voice stays in its resolved pool.
8. Enable random-each-line and confirm every selected voice stays in the
resolved pool.
9. Update or replace the index/supplement and confirm previously saved voices
do not change.
10. Visit two zones containing the same display name and note whether the
current global speaker identity is confusing enough to justify a future
zone-specific override feature.
## Acceptance criteria
- A known `(display zone, normalized NPC name)` with unanimous feminine or
masculine PEQ records receives a voice from the matching available pool.
- Unknown, neuter, conflicting, missing-zone, and empty-subpool cases use the
complete available curated pool.
- Existing persistent voices and pins are never replaced by generated data or
supplemental payloads.
- Pin, reroll, and random-each-line remain available in the voice manager.
- Legacy `speakers.json` files load without migration.
- An optional small Legends payload can map NPCs absent from EQEmu without
adding audio assets or networking to the main release.
- The ProjectEQ dump and database tooling are not present in release archives.
- Automated tests pass, archive size impact is reported, and final behavioral
verification is performed through the user's playtest checklist.
## Later roadmap: collaborative voice choices
After the local supplement format has proven stable:
1. Add an export preview for selected pinned mappings only, including zone,
normalized NPC name, voice ID, schema version, and game-data version.
2. Build a website/API that accepts individual submissions, tracks votes and
provenance, and moderates invalid zone/NPC/voice combinations.
3. Generate a reviewed Legends-only supplement from accepted records.
4. Let users explicitly download/import a versioned payload with checksum and
source information.
5. Consider an opt-in recommendation pack for known EQEmu NPCs. Local pins and
existing assignments must remain higher precedence.
Keeping collection, publication, and download outside the executable prevents
the collaborative roadmap from increasing the normal build size or making
Mudmouth dependent on the website.
generated/peq-npc-gender-index.json
+1
View File
File diff suppressed because one or more lines are too long
shell.nix
+4
View File
@@ -8,6 +8,10 @@ pkgs.mkShell {
clippy
pkg-config
alsa-lib
# The ProjectEQ dump is a MariaDB SQL dump. This provides mariadbd,
# mariadb-install-db, and the mariadb/mysql client aliases.
mariadb
unzip
];
PKG_CONFIG_PATH = "${pkgs.alsa-lib.dev}/lib/pkgconfig";
src/bin/generate_peq_npc_index.rs
+420
View File
@@ -0,0 +1,420 @@
//! Generate Mudmouth's compact NPC gender lookup from a ProjectEQ SQL dump.
//!
//! This is deliberately a build-time tool. It reads only the four tables used
//! by the lookup and does not add a database dependency to the application.
use anyhow::{Context, Result, bail};
use clap::Parser;
use serde::Serialize;
use std::{
collections::{BTreeMap, BTreeSet, HashMap},
fs::{self, File},
io::{BufRead, BufReader},
path::{Path, PathBuf},
process::{Command, Stdio},
};
#[allow(dead_code)]
#[path = "../events.rs"]
mod events;
#[derive(Debug, Parser)]
#[command(about = "Generate a minified NPC gender index from a PEQ SQL dump")]
struct Cli {
/// PEQ create_tables_content.sql file, or a .zip containing that file.
#[arg(long)]
dump: PathBuf,
/// Destination for the generated minified JSON index.
#[arg(long)]
output: PathBuf,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Serialize)]
#[serde(rename_all = "lowercase")]
enum Gender {
Masculine,
Feminine,
Neuter,
}
impl Gender {
fn from_database(value: &str) -> Result<Self> {
match value {
"0" => Ok(Self::Masculine),
"1" => Ok(Self::Feminine),
"2" => Ok(Self::Neuter),
_ => bail!("invalid npc_types.gender {value:?}; expected 0, 1, or 2"),
}
}
}
#[derive(Debug, Default)]
struct PeqRows {
zones: HashMap<String, BTreeSet<String>>,
spawn_entries: HashMap<u32, BTreeSet<u32>>,
spawns: Vec<(String, u32)>,
npcs: HashMap<u32, (String, Gender)>,
seen_tables: BTreeSet<Table>,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
enum Table {
NpcTypes,
SpawnEntry,
Spawn2,
Zone,
}
impl Table {
fn from_insert(line: &str) -> Option<Self> {
[
("INSERT INTO `npc_types` VALUES", Self::NpcTypes),
("INSERT INTO `spawnentry` VALUES", Self::SpawnEntry),
("INSERT INTO `spawn2` VALUES", Self::Spawn2),
("INSERT INTO `zone` VALUES", Self::Zone),
]
.into_iter()
.find_map(|(prefix, table)| line.starts_with(prefix).then_some(table))
}
}
#[derive(Debug, Serialize)]
struct GeneratedIndex {
schema_version: u8,
source: String,
stats: Statistics,
entries: Vec<IndexEntry>,
}
#[derive(Debug, Default, Serialize, PartialEq, Eq)]
struct Statistics {
pairs: usize,
masculine_only: usize,
feminine_only: usize,
neuter_only: usize,
conflicting: usize,
accepted: usize,
}
#[derive(Debug, Serialize, PartialEq, Eq)]
struct IndexEntry(String, String, Gender);
fn main() -> Result<()> {
let cli = Cli::parse();
let rows = read_dump(&cli.dump)?;
let index = generate_index(&rows, source_identity(&cli.dump)?);
let bytes = serde_json::to_vec(&index).context("could not serialize PEQ index")?;
fs::write(&cli.output, &bytes)
.with_context(|| format!("could not write {}", cli.output.display()))?;
println!("source: {}", index.source);
println!(
"pairs: {} (masculine: {}, feminine: {}, neuter: {}, conflicting: {}, accepted: {})",
index.stats.pairs,
index.stats.masculine_only,
index.stats.feminine_only,
index.stats.neuter_only,
index.stats.conflicting,
index.stats.accepted,
);
println!("output: {} bytes ({})", bytes.len(), cli.output.display());
Ok(())
}
fn source_identity(path: &Path) -> Result<String> {
path.file_name()
.and_then(|name| name.to_str())
.map(str::to_owned)
.context("dump path must have a UTF-8 file name")
}
fn read_dump(path: &Path) -> Result<PeqRows> {
if path.extension().is_some_and(|extension| extension == "zip") {
let mut child = Command::new("unzip")
.args([
"-p",
&path.to_string_lossy(),
"peq-dump/create_tables_content.sql",
])
.stdout(Stdio::piped())
.spawn()
.context("could not run unzip; install it or provide the extracted SQL dump")?;
let stdout = child
.stdout
.take()
.context("unzip did not provide stdout")?;
let rows = parse_sql(BufReader::new(stdout));
let status = child.wait().context("could not wait for unzip")?;
if !status.success() {
bail!("unzip failed while reading {}", path.display());
}
rows
} else {
let file =
File::open(path).with_context(|| format!("could not read {}", path.display()))?;
parse_sql(BufReader::new(file))
}
}
fn parse_sql<R: BufRead>(reader: R) -> Result<PeqRows> {
let mut rows = PeqRows::default();
let mut current_table = None;
for (line_number, line) in reader.lines().enumerate() {
let line = line.with_context(|| format!("could not read SQL line {}", line_number + 1))?;
if let Some(table) = Table::from_insert(&line) {
current_table = Some(table);
rows.seen_tables.insert(table);
}
if let Some(table) = current_table {
for fields in sql_tuples(&line).with_context(|| {
format!("malformed {table:?} row at SQL line {}", line_number + 1)
})? {
add_row(&mut rows, table, &fields).with_context(|| {
format!("invalid {table:?} row at SQL line {}", line_number + 1)
})?;
}
if line.trim_end().ends_with(';') {
current_table = None;
}
}
}
for table in [
Table::NpcTypes,
Table::SpawnEntry,
Table::Spawn2,
Table::Zone,
] {
if !rows.seen_tables.contains(&table) {
bail!("dump is missing INSERT rows for {table:?}");
}
}
Ok(rows)
}
fn add_row(rows: &mut PeqRows, table: Table, fields: &[String]) -> Result<()> {
match table {
Table::NpcTypes => {
let id = field_u32(fields, 0, "id")?;
let name = field_string(fields, 1, "name")?;
let gender = Gender::from_database(field(fields, 9, "gender")?)?;
if rows.npcs.insert(id, (name, gender)).is_some() {
bail!("duplicate npc_types.id {id}");
}
}
Table::SpawnEntry => {
let group = field_u32(fields, 0, "spawngroupID")?;
let npc = field_u32(fields, 1, "npcID")?;
rows.spawn_entries.entry(group).or_default().insert(npc);
}
Table::Spawn2 => {
let group = field_u32(fields, 1, "spawngroupID")?;
let zone = field_string(fields, 2, "zone")?;
if !zone.is_empty() {
rows.spawns.push((zone, group));
}
}
Table::Zone => {
let short_name = field_string(fields, 3, "short_name")?;
let long_name = field_string(fields, 4, "long_name")?;
if !short_name.is_empty() {
rows.zones.entry(short_name).or_default().insert(long_name);
}
}
}
Ok(())
}
fn field<'a>(fields: &'a [String], index: usize, name: &str) -> Result<&'a str> {
fields
.get(index)
.map(String::as_str)
.with_context(|| format!("missing {name} column"))
}
fn field_u32(fields: &[String], index: usize, name: &str) -> Result<u32> {
field(fields, index, name)?
.parse()
.with_context(|| format!("{name} is not an unsigned integer"))
}
fn field_string(fields: &[String], index: usize, name: &str) -> Result<String> {
let value = field(fields, index, name)?;
if value == "NULL" {
bail!("{name} may not be NULL");
}
Ok(value.to_owned())
}
fn generate_index(rows: &PeqRows, source: String) -> GeneratedIndex {
let mut candidates = BTreeMap::<(String, String), BTreeSet<Gender>>::new();
for (short_zone, group) in &rows.spawns {
let Some(long_zones) = rows.zones.get(short_zone) else {
continue;
};
let Some(npc_ids) = rows.spawn_entries.get(group) else {
continue;
};
for npc_id in npc_ids {
let Some((name, gender)) = rows.npcs.get(npc_id) else {
continue;
};
if name.is_empty() {
continue;
}
let normalized_name = events::normalize_npc_name(name);
if normalized_name.is_empty() {
continue;
}
for long_zone in long_zones {
candidates
.entry((long_zone.clone(), normalized_name.clone()))
.or_default()
.insert(*gender);
}
}
}
let mut stats = Statistics {
pairs: candidates.len(),
..Default::default()
};
let mut entries = Vec::new();
for ((zone, name), genders) in candidates {
match genders
.iter()
.next()
.copied()
.filter(|_| genders.len() == 1)
{
Some(Gender::Masculine) => {
stats.masculine_only += 1;
stats.accepted += 1;
entries.push(IndexEntry(zone, name, Gender::Masculine));
}
Some(Gender::Feminine) => {
stats.feminine_only += 1;
stats.accepted += 1;
entries.push(IndexEntry(zone, name, Gender::Feminine));
}
Some(Gender::Neuter) => stats.neuter_only += 1,
None => stats.conflicting += 1,
}
}
GeneratedIndex {
schema_version: 1,
source,
stats,
entries,
}
}
/// Parses the SQL tuple syntax emitted by `mariadb-dump`.
fn sql_tuples(line: &str) -> Result<Vec<Vec<String>>> {
let mut tuples = Vec::new();
let mut fields = Vec::new();
let mut field = String::new();
let mut in_tuple = false;
let mut in_string = false;
let mut escaped = false;
for character in line.chars() {
if !in_tuple {
if character == '(' {
in_tuple = true;
fields.clear();
field.clear();
}
continue;
}
if escaped {
field.push(match character {
'0' => '\0',
'b' => '\u{0008}',
'n' => '\n',
'r' => '\r',
't' => '\t',
'Z' => '\u{001a}',
other => other,
});
escaped = false;
} else if in_string && character == '\\' {
escaped = true;
} else if character == '\'' {
in_string = !in_string;
} else if !in_string && character == ',' {
fields.push(std::mem::take(&mut field));
} else if !in_string && character == ')' {
fields.push(std::mem::take(&mut field));
tuples.push(std::mem::take(&mut fields));
in_tuple = false;
} else {
field.push(character);
}
}
if in_tuple || in_string || escaped {
bail!("unterminated SQL tuple");
}
Ok(tuples)
}
#[cfg(test)]
mod tests {
use super::*;
const FIXTURE: &str = include_str!("../../tests/fixtures/peq-small.sql");
#[test]
fn groups_short_zones_and_filters_ambiguous_gender() {
let rows = parse_sql(BufReader::new(FIXTURE.as_bytes())).unwrap();
let index = generate_index(&rows, "fixture.sql".into());
assert_eq!(
index.entries,
vec![
IndexEntry(
"Shared Zone".into(),
"captain_bob".into(),
Gender::Masculine
),
IndexEntry("Shared Zone".into(), "lady_ada".into(), Gender::Feminine),
]
);
assert_eq!(
index.stats,
Statistics {
pairs: 4,
masculine_only: 1,
feminine_only: 1,
neuter_only: 1,
conflicting: 1,
accepted: 2,
}
);
}
#[test]
fn output_is_minified_and_deterministic() {
let rows = parse_sql(BufReader::new(FIXTURE.as_bytes())).unwrap();
let first = serde_json::to_vec(&generate_index(&rows, "fixture.sql".into())).unwrap();
let second = serde_json::to_vec(&generate_index(&rows, "fixture.sql".into())).unwrap();
assert_eq!(first, second);
assert!(!first.contains(&b'\n'));
}
#[test]
fn rejects_malformed_rows_and_unknown_gender_values() {
let malformed = "INSERT INTO `npc_types` VALUES\n(1,'too short');\n";
assert!(parse_sql(BufReader::new(malformed.as_bytes())).is_err());
let invalid_gender = "INSERT INTO `npc_types` VALUES\n(1,'Bad','','',0,0,0,0,0,9);\n";
assert!(parse_sql(BufReader::new(invalid_gender.as_bytes())).is_err());
}
}
src/events.rs
+26 -4
View File
@@ -75,7 +75,11 @@ fn parse_npc_says(message: &str) -> Option<(String, String)> {
Some((speaker.to_string(), text.to_string()))
}
pub fn speaker_key(name: &str) -> String {
/// Normalizes an EverQuest NPC name for use in stable lookup identities.
///
/// Non-ASCII and punctuation characters are separators, and repeated
/// separators collapse to a single underscore.
pub fn normalize_npc_name(name: &str) -> String {
let normalized = name
.chars()
.flat_map(char::to_lowercase)
@@ -88,12 +92,15 @@ pub fn speaker_key(name: &str) -> String {
})
.collect::<String>();
let normalized = normalized.trim_matches('_');
let compact = normalized
normalized
.split('_')
.filter(|part| !part.is_empty())
.collect::<Vec<_>>()
.join("_");
format!("npc:{compact}")
.join("_")
}
pub fn speaker_key(name: &str) -> String {
format!("npc:{}", normalize_npc_name(name))
}
#[cfg(test)]
@@ -140,4 +147,19 @@ mod tests {
assert_eq!(speaker_key("A lizardman healer"), "npc:a_lizardman_healer");
assert_eq!(speaker_key("Zok Caropni!"), "npc:zok_caropni");
}
#[test]
fn normalizes_npc_names_for_shared_lookup() {
let fixtures = [
("Zok Caropni!", "zok_caropni"),
("a---lizardman__healer", "a_lizardman_healer"),
("MiXeD CaSe NPC", "mixed_case_npc"),
("F\u{00e9}lix the C\u{00e1}t", "f_lix_the_c_t"),
];
for (name, expected) in fixtures {
assert_eq!(normalize_npc_name(name), expected, "{name}");
assert_eq!(speaker_key(name), format!("npc:{expected}"), "{name}");
}
}
}
tests/fixtures/peq-small.sql
Vendored
+14
View File
@@ -0,0 +1,14 @@
-- Minimal mariadb-dump-shaped PEQ fixture for the NPC gender index generator.
INSERT INTO `npc_types` VALUES
(1,'Captain Bob','','',0,0,0,0,0,0),
(2,'CAPTAIN---BOB','','',0,0,0,0,0,0),
(3,'Lady Ada','','',0,0,0,0,0,1),
(4,'Neutral Guard','','',0,0,0,0,0,2),
(5,'Conflict','','',0,0,0,0,0,0),
(6,'Conflict!','','',0,0,0,0,0,1);
INSERT INTO `spawnentry` VALUES
(10,1),(11,2),(12,3),(13,4),(14,5),(15,6);
INSERT INTO `spawn2` VALUES
(1,10,'north'),(2,11,'south'),(3,12,'north'),(4,13,'north'),(5,14,'north'),(6,15,'south');
INSERT INTO `zone` VALUES
(1,1,0,'north','Shared Zone'),(2,2,0,'south','Shared Zone');