Record the search index rebuild schedule in the plan

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/plan.md

@@ -288,7 +288,10 @@ the fork, so do it first.
    step 7); the server's `.gitmodules` points at it over SSH.
 3. Bring this repo's flake up to date (it's still on `nixos-25.05`) only if the
    exporter gets packaged here.
-4. Ask the user to add a sops secret `plebleader.env` for incineroar, holding
+4. **Done 2026-11-09** (`systems/brenise.dev/incineroar/plebleader.env` in
+   nix-pleb, dotenv format, encrypted from this repo's `.env` without
+   decrypting anything; wired up on the `plebleader` branch).
+   Ask the user to add a sops secret `plebleader.env` for incineroar, holding
    `BL_CLIENT_ID` and `BL_CLIENT_SECRET`. These are the same OAuth app as
    plebsaber.stream; check that its registered callback URLs include the one
    used for seeding. Agents never decrypt it.
@@ -730,6 +733,65 @@ profile and scores, and a replay opens in ArcViewer.
 
 ### Phase 5: Convert ScoreSaber scores and merge
 
+**Built and checked on a scratch copy 2026-11-09; committed, deploy
+pending.** Server `50f7df6` (`Tools/ArchiveImporter/ScoreMerge.cs`, rules in
+the fork's `MIRROR.md`), website `5522f245` (SS badge, `/watch/` for `.dat`),
+exporter `ssMapHash` in this repo. All pushed. Result on the local archive: 3,710 leaderboards with one score
+each (3,566 BL + 141 SS-only BL leaderboards + 3 DLC levels BL lacks). Winners: 3,564 BL main,
+1 BL legacy, 139 SS legacy, 6 SS main; 2,022 SS scores were the same play as
+a BL score and dropped; 20 BL scores got an improvement recomputed against an
+earlier SS play. An unchanged rerun writes 0 rows. What differs from the
+steps below:
+
+- Step 1: **no new columns.** Provenance goes in `Platform`
+  (`scoresaber,<SS player id>`), and `Id` ≥ 1e9 marks SS scores. Columns
+  would have needed a models-fork change, a migration, and edits to the ~90
+  score projections before the website saw them; `Platform` already reaches
+  every score response.
+- Step 2: OST levels are `ost_<LevelId>` on SS and `<levelid>` on BL; 5 of 8
+  resolve that way. Me Porto Bonito, Nuthin' But A "G" Thang and Stressed Out
+  (3 scores) aren't on BL (404), so the importer builds a song and leaderboard
+  from the SS item (`<LevelId><diff><mode>` ids, no ratings, accuracy 0).
+  The exporter now maps `ost_` hashes too (`scoresaber.ts` `ssMapHash`).
+  HMD and controller come from SS `deviceHmd`/`deviceControllerLeft`. SS
+  replays are served from `storage/ssreplays` → `archive/scoresaber`, so the
+  URL is `/cdn/ssreplays/<player>/replays/<id>.dat`.
+- **Pp isn't upstream's.** Neither the fork's `rebalanced-pp` formula
+  (median 0.94x, range 0.69–1.16x) nor upstream `master`'s (0.81x)
+  reproduces the pp on archived BL scores; live BL's code is private. So the
+  fold compares every candidate by **locally computed** pp, which keeps the
+  order right, and BL winners keep upstream's pp. The 70 SS winners on
+  ranked maps show local pp, and improvement pp deltas are local on both
+  sides. Matching pp first sent two BL scores (KillerToy, Make Me) to worse
+  SS plays.
+- Step 3: same-play duplicates (equal base score, ≤ 2 min apart; observed
+  −30 s…+9 s) keep the BL copy. Each upstream `scoreImprovement` is turned
+  back into the best it replaced and added as a candidate, so upstream's
+  improvement stays exactly when that rebuilt best is also the fold's previous
+  best. Older archived BL bests are candidates too.
+- Step 5: the badge sits next to the rank in both `ScoreRank.svelte` (rank 0
+  already renders "-"). `/watch/` takes the map hash from the score
+  (`/score/{id}` → `song.hash`) for `.dat` replays, from `?scoreId=` or
+  1e9 + the id in the filename. Expanding an SS row shows its leaderboard and
+  no stats tab; the page logs a JSON error for the 404 score stats, harmless.
+- Check: The Goddess Figure (Normal) is the case where a 2022 legacy SS
+  play beats the BL score; it shows with the SS badge, and the BL Hard score
+  there now reads "+76,821 vs 4 years ago" (vs the legacy SS play). An SS
+  replay redirects to ArcViewer with the `.dat` and local zip and loads the map;
+  watching it play still needs a real browser.
+
+Deployed 2026-11-09 (nix-pleb `a41f3c2`); the first sync imported 3,710
+scores with the same split by source as the local check. The profile's score
+search then returned 500, for any search: the Lucene song and player indexes
+were written next to the DLL in the read-only `/nix/store` (phase 2 step 6
+had planned to move them). Server `2631c46` adds `SearchIndexPath`
+(MirrorMode, set to `/var/lib/plebleader/search`). Server `81f12ee` then
+moved the rebuild into the importer: it replaces the indexes at the end of a
+run that added scores (so at most once a day, right after the sync), or when
+there are none. The API builds them only when missing; upstream appended every
+song again every 6 h. nix-pleb `bcdd591`, `a313f5b` bump both. The next `plebleader-sync`
+run re-imports with the merge.
+
 ScoreSaber scores become BeatLeader `Score` rows, and each leaderboard keeps
 exactly one score. BeatLeader already shows how a score compares with the
 previous best (`ScoreImprovement`), so after the merge that comparison covers
@@ -800,6 +862,35 @@ the BL score, and confirm the improvement display and the ArcViewer replay.
 
 ### Phase 6: Ongoing sync
 
+**Built 2026-11-09; deployed on incineroar the same day (daily timer at
+17:00, first run clean; see phase 7).** `node scripts/archive/cli.ts sync
+[--ratings] [--no-ss-replays]` (this repo `d182fd8`), packaged as the
+flake's `plebleader-archive`. Tested against the local archive: the token
+refresh rotated the chain (now valid until about 2026-11-23), and with the
+newest BL and SS score files set aside, the run fetched exactly those two
+back, byte-identical, and re-snapshotted the BL leaderboard. What differs
+from the steps below:
+
+- Main account only; both legacy accounts are frozen. The incremental
+  window starts a day before the newest archived score (BL `timepost`, SS
+  `timeSet`), computed from the archive rather than `state.json`. Replays and
+  score stats are retried for every score in the window, not only new ones.
+- New BL hashes get a refreshed leaderboard snapshot; new SS-only hashes get
+  one if they have none. Map zips cover every hash in the window.
+- Ratings drift: Sundays by the machine's clock, or `--ratings`.
+- Each run writes a `sync` section to `state.json` (new score ids, maps
+  without a zip). `exportScores` no longer resets `newestTimepost` to 0 on an
+  empty window.
+- Step 8: the importer runs in full after every sync (about 10 s), as a
+  second `ExecStart` of `plebleader-sync`; no `--since` flag.
+- **Importer fix** (server `dd94772`): the archive keeps an old best after
+  its improvement is synced. The importer now takes only the newest score
+  per leaderboard and removes rows superseded since the last import. Before
+  this, the second score would have broken the
+  `(PlayerId, LeaderboardId, ValidContexts)` unique index. Tested on a
+  scratch copy with a fake improved score: added, old one removed, and back
+  again once the fake was deleted.
+
 The `sync` subcommand runs once a day from a systemd timer on incineroar:
 `OnCalendar=*-*-* 17:00:00 America/Los_Angeles` with `Persistent=true`, so a
 run missed while the machine was off happens at the next boot. By 5 pm the
@@ -840,7 +931,40 @@ the local zip.
 
 ### Phase 7: NixOS deployment on incineroar
 
-**Built 2026-11-09, not switched yet.** nix-pleb branch `plebleader`
+**Done 2026-11-09: merged into nix-pleb `main` (fast-forward to `3b3eb47`),
+switched on incineroar, first sync clean, and BL `.bsor` replays play in
+ArcViewer.** Issues hit after the switch:
+
+- **Missing `/run/secrets/plebleader.env`.** The file was encrypted only to
+  litten's age key (the repo `.sops.yaml` rule lists only litten), so
+  incineroar's host key couldn't decrypt it. Fixed by adding incineroar's host
+  key and `age1kuxd…` with `sops rotate --add-age`. Any new incineroar sops
+  file needs the same three recipients as `sops.yaml`.
+- **First sync (13:48):** the token refreshed, with the chain valid to
+  2026-11-23 (each daily refresh moves that forward), 0 new scores, and the
+  importer loaded 3565 scores and 2552 songs in 11 s. The token now lives only
+  in `/var/lib/plebleader/auth/`.
+- **Website "TypeError: Cannot read properties of undefined (reading
+  'id')".** `utils/format.js` takes the first host label as the leaderboard
+  context when the host has more than 2 labels, so the context became
+  `incineroar` and the navbar's context lookup found nothing. The phase 4 dev
+  check missed it because a numeric last label (`127.0.0.1`) raises that limit
+  to 4. Website fix `e5f6279a`: non-BeatLeader hosts always use `general`.
+- **ArcViewer "Download failed! HTTP/1.1 403 Forbidden"** for
+  `/cdn/replays/…bsor`, and `/cdn/maps/…` redirected to BeatSaver even for
+  archived zips. Caddy had been running since 2026-11-07. The switch added it
+  to group `plebleader` but only reloaded it, so the process still had just
+  `caddy` (`grep Groups /proc/<pid>/status`). Fixed with
+  `doas systemctl restart caddy`. Any later change to Caddy's groups needs a
+  restart too.
+
+Phase 6 added a second commit on the
+branch: the `plebleader.env` sops secret, `plebleader-sync` (sync, then the
+importer) and its timer, and lock bumps for `plebsaber-stream` (`d182fd8`)
+and `plebleader-server` (`dd94772`). The `plebsaber-stream` bump also
+rebuilds beldum's site, with no change to the site code.
+
+nix-pleb branch `plebleader`
 (`2ada4b7`, in the worktree `~/ops/nix-pleb-plebleader`, based on `main`
 `981f9e7`) adds the inputs `plebleader-server` (server `0481759`) and
 `plebleader-website` (`8d540642`), both following `nixpkgs-unstable`, plus to
@@ -879,11 +1003,19 @@ To deploy (the user does this):
    this is a copy):
    `doas cp -a ~/ops/beatsaber/plebsaber.stream/.data/plebleader/archive /var/lib/plebleader/`,
    then `doas chown -R plebleader:plebleader /var/lib/plebleader/archive` and
-   `doas chmod -R g+rX,o-rwx /var/lib/plebleader/archive`. Keep the token chain
-   (`.data/plebleader/auth`) where it is until phase 6 moves the sync.
-3. `doas systemctl start plebleader-import`, then open
-   `https://incineroar.brenise.dev:4442/u/pleb` and a replay.
-4. Afterwards, remove the worktree with `git worktree remove ../nix-pleb-plebleader`.
+   `doas chmod -R g+rX,o-rwx /var/lib/plebleader/archive`.
+3. Move the sync's token chain. **Before 2026-11-23**, or it expires and
+   needs `auth login` again:
+   `doas install -d -o plebleader -g plebleader -m 0700 /var/lib/plebleader/auth`
+   and
+   `doas install -o plebleader -g plebleader -m 0600 ~/ops/beatsaber/plebsaber.stream/.data/plebleader/auth/beatleader_tokens.json /var/lib/plebleader/auth/`,
+   then delete the local copy. From then on, never run `sync` or
+   `auth refresh` locally: the refresh token rotates, and whichever side
+   refreshes second breaks.
+4. `doas systemctl start plebleader-sync` (sync, then import), then open
+   `https://incineroar.brenise.dev:4442/u/pleb` and a replay. Check
+   `journalctl -u plebleader-sync` and `systemctl list-timers plebleader-sync`.
+5. Afterwards, remove the worktree with `git worktree remove ../nix-pleb-plebleader`.
 
 - Packages: `buildDotnetModule` for the fork (generate `deps.json` with
   `nix build .#…passthru.fetch-deps`), `buildNpmPackage` for the website (start
@@ -899,6 +1031,42 @@ To deploy (the user does this):
   `nix build .#nixosConfigurations.incineroar.config.system.build.toplevel --no-link`.
   The user does the switch.
 
+### Phase 8: Self-hosted BeatLeader web replay viewer
+
+Upstream, the score row's replay icon opens BeatLeader's web viewer
+(`BL_REPLAYS_URL`, replay.beatleader.com), and turning on ArcViewer in the
+score settings adds a separate `altReplay` icon (`Song/Icons.svelte`,
+`webPlayerLink(…, 'arcviewer', …)`). Phase 4 pointed `BL_REPLAYS_URL` at
+`/watch/`, so both icons open ArcViewer now. The user wants the upstream
+behaviour back: the replay icon opens the BeatLeader viewer, and ArcViewer
+stays a separate icon.
+
+Source: `~/src/BeatLeader/BeatSaber-Web-Replays` (webpack 5, npm). Steps:
+
+1. Fork it to gitea `pleb/BeatSaber-Web-Replays` with a `pleb/main` branch,
+   and change as little as possible.
+2. Resolve `getApiUrl()` (`src/utils.js`, `src/open-replay-decoder.js`) to
+   `/cors/blapi` on non-BeatLeader hosts, the same rule as the website's
+   `IS_MIRROR`. Check how it fetches the replay file and the score (`link=`,
+   `scoreId=`) against the mirror's `/cdn/replays/` and `/cors/blapi/score/{id}`.
+3. Maps: `zip-loader.js` tries `cdn.songs.beatleader.xyz`, then
+   `/cors/beat-saver2/api/maps/…` (a Netlify proxy), then `r2cdn`/`cdn.beatsaver.com`.
+   Try `/cdn/maps/<hash>.zip` first, which already falls back to r2cdn in Caddy.
+   Either proxy `/cors/beat-saver2/` to `api.beatsaver.com` in Caddy or skip it.
+4. Drop or stub analytics (googletagmanager), and check its other runtime
+   fetches (fonts, `cdn.jsdelivr.net`). Discord and Patreon are plain links.
+5. Add a flake with `buildNpmPackage` that outputs the built static files
+   (`index.html`, `assets`, `build`, `vendor`, as in its `predeploy` script),
+   and add it as a nix-pleb input.
+6. Caddy: serve it at `/replay/` on the plebleader vhost.
+7. Website: in mirror mode set `BL_REPLAYS_URL` to `/replay/` instead of
+   `/watch/`. Leave the explicit ArcViewer links (`altReplay`, `ReplayButton`
+   with `webPlayer == 'arcviewer'`) on `/watch/`.
+
+**Check:** on `/u/pleb` the replay icon plays a score in the BeatLeader
+viewer from local files, and with ArcViewer turned on in settings, the second
+icon plays it in ArcViewer.
+
 ## Risks and fallbacks
 
 - **The server doesn't build or run on Linux, or EF/SQLite can't translate

scripts/archive/cli.ts

@@ -21,7 +21,7 @@ import * as path from 'node:path';
 import { ARCHIVE_DIR, BL_API, DEFAULT_MAPS_SINCE, LEGACY_PLAYER_ID, MAIN_PLAYER_ID, PLAYER_IDS, TOKENS_FILE, USER_AGENT, loadDotEnv, parseSince } from './config.ts';
 import { getAccessToken, login, readTokens, refresh } from './auth.ts';
 import { archivedScores, blHashes, exportLeaderboards, exportProfile, exportReplays, exportScoreStats, exportScores, newestScoreTime } from './beatleader.ts';
-import { archivedSSScores, exportSSPlayer, exportSSReplays, exportSSScores, newestSSScoreTime, ssHashes } from './scoresaber.ts';
+import { archivedSSScores, exportSSPlayer, exportSSReplays, exportSSScores, newestSSScoreTime, ssHashes, ssMapHash } from './scoresaber.ts';
 import { exportMaps } from './beatsaver.ts';
 import { archivePath, exists, listFiles, readState, updateState } from './store.ts';
 
@@ -83,7 +83,7 @@ function playedHashes(since: number): string[] {
 function mapHashes(mapsSince: number): string[] {
 	const legacyWithReplay = archivedSSScores(LEGACY_PLAYER_ID, 0)
 		.filter((it) => exists(archivePath('scoresaber', LEGACY_PLAYER_ID, 'replays', `${it.score.id}.dat`)))
-		.map((it) => it.leaderboard.songHash.toLowerCase());
+		.map(ssMapHash);
 	return [...new Set([...playedHashes(mapsSince), ...legacyWithReplay])].sort();
 }
 
@@ -137,7 +137,7 @@ async function sync(opts: { ratings: boolean; ssReplays: boolean }): Promise<num
 	await exportSSPlayer(MAIN_PLAYER_ID);
 	const ssSince = Math.max(0, newestSSScoreTime(MAIN_PLAYER_ID) - SYNC_OVERLAP_S);
 	const newSS = await exportSSScores(MAIN_PLAYER_ID, ssSince);
-	const newSSHashes = [...new Set(newSS.map((it) => it.leaderboard.songHash.toLowerCase()))].sort();
+	const newSSHashes = [...new Set(newSS.map(ssMapHash))].sort();
 	await exportLeaderboards(newSSHashes.filter((h) => !newBLHashes.includes(h)));
 	if (opts.ssReplays) await exportSSReplays(MAIN_PLAYER_ID, ssSince);
 

scripts/archive/scoresaber.ts

@@ -102,6 +102,14 @@ function seedFromScraper(playerId: string, items: SSItem[]): void {
 	if (copied) console.log(`  copied ${copied} replays from ${dir}`);
 }
 
+/**
+ * The map hash BeatLeader and the archive use for a ScoreSaber item: lowercase, and for official
+ * levels the level id without ScoreSaber's "ost_" prefix (BeatLeader: "beatsaber", SS: "ost_BeatSaber").
+ */
+export function ssMapHash(it: SSItem): string {
+	return it.leaderboard.songHash.toLowerCase().replace(/^ost_/, '');
+}
+
 export function ssHashes(playerId: string, sinceUnix = 0): string[] {
-	return [...new Set(archivedSSScores(playerId, sinceUnix).map((it) => it.leaderboard.songHash.toLowerCase()))].sort();
+	return [...new Set(archivedSSScores(playerId, sinceUnix).map(ssMapHash))].sort();
 }