Record the search index rebuild schedule in the plan

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/plan.md

@@ -288,7 +288,10 @@ the fork, so do it first.
    step 7); the server's `.gitmodules` points at it over SSH.
 3. Bring this repo's flake up to date (it's still on `nixos-25.05`) only if the
    exporter gets packaged here.
-4. Ask the user to add a sops secret `plebleader.env` for incineroar, holding
+4. **Done 2026-11-09** (`systems/brenise.dev/incineroar/plebleader.env` in
+   nix-pleb, dotenv format, encrypted from this repo's `.env` without
+   decrypting anything; wired up on the `plebleader` branch).
+   Ask the user to add a sops secret `plebleader.env` for incineroar, holding
    `BL_CLIENT_ID` and `BL_CLIENT_SECRET`. These are the same OAuth app as
    plebsaber.stream; check that its registered callback URLs include the one
    used for seeding. Agents never decrypt it.
@@ -543,6 +546,54 @@ loads, and `GET /player/76561199407393962` returns 404 rather than 500.
 
 ### Phase 3: Importer
 
+**Importer done 2026-11-09** (`pleb/main` at `e3f991c`, pushed to Gitea). Usage
+and the full file → table mapping are in the fork's `MIRROR.md`. A full import
+takes about 11 s, and an unchanged rerun writes 0 rows. What differs from the
+steps below:
+
+- The archived responses are deserialized straight into the entity types,
+  because their property names match. Only scalar columns are copied onto the
+  tracked rows; keys, foreign keys and navigations are set explicitly. Leaderboard
+  snapshots are the main source for `Song`/`DifficultyDescription`. A score's
+  embedded leaderboard is a fallback only, and no score needed it.
+- One-to-one children (`ScoreImprovement`, `ReplayOffsets`, `ModifiersRating`,
+  `DifficultyStatistics`, `ModifiersMap`, stats, settings) get local ids, so
+  they can't collide with upstream ids. Scores, difficulties, songs,
+  leaderboards, mappers and history rows keep upstream ids.
+- `PlayerScoreStatsHistory` lives in **`StorageContext`**, so that database
+  isn't empty. The exporter had saved only 50 days of `/history` (the API
+  default); it now asks for `count=100000` (1,138 days for pleb). The triangle
+  needs those monthly points.
+- No legacy `Player` row. An `AccountLink` (`PCOculusID` = legacy id) plus a
+  `// pleb:` patch in `PlayerIdToMain` resolves `/player/76561197962107242`
+  to pleb. Upstream only checks Steam-range ids against Oculus links. The
+  legacy account's 3 BL scores are skipped until phase 5.
+- `StoragePath/replays` is a symlink to the archive's `beatleader/replays`.
+  Score stats are copied to `StoragePath/scorestats/<scoreId>.json`.
+- Clans aren't mirrored (`ClanOrder` is blank).
+
+Imported: 2,552 songs, 10,104 difficulties, 10,098 leaderboards, 628 mappers,
+3,565 scores, 3,568 score stats, 1,138 history rows. API check against the
+archived upstream responses, on a scratch DB with `ASPNETCORE_ENVIRONMENT=Mirror`:
+
+- `/player/{id}?stats=true` matches. The only differences are row ids,
+  clans, and `linkedIds`, which now lists the alt.
+- `/player/{id}/scores?includeIO=true`: 3,565 total. Scores, improvements and
+  offsets match, and unranked difficulties now carry stars and the skill
+  triangle ratings.
+- `/accgraph` and `/history/triangle` match. `/score/statistic/{id}` redirects
+  to the local file.
+- `/leaderboard/{id}` and `/leaderboard/scores/{id}` first returned 500.
+  About 10 projections order `Player.Clans` by an expression over the outer
+  row's `ClanOrder`, which needs SQL `APPLY`, and SQLite has none. **Fixed**
+  in `e3f991c`: `Utils/MirrorQueryInterceptor.cs`, an EF query interceptor
+  registered only in MirrorMode, rewrites any ordering of `Player.Clans` to
+  ordering by clan id. One file instead of patching every call site; the
+  mirror has no clans, so the order doesn't matter. The leaderboard,
+  clan-ranking and context variants now return 200.
+
+The check below (stock website) is still to do in phase 4.
+
 `Tools/ArchiveImporter` lives in the fork, references the main project, and
 uses its `AppContext`/`StorageContext`. Run it as a one-shot that is idempotent
 and upserts by upstream id.
@@ -563,7 +614,7 @@ and upserts by upstream id.
   an `AccountLink` or alt pointing to the main player, so `PlayerIdToMain`
   resolves `/u/76561197962107242` to `pleb`. Its 3 BL scores go through the
   phase 5 merge like any other candidate.
-- `StorageContext` (attempts) stays empty, because attempts aren't exported.
+- `StorageContext` holds `PlayerScoreStatsHistory`; its attempts tables stay empty, because attempts aren't exported.
 - `scorestats/*.json` → the `scorestats` storage container, using the filename
   the server expects (check `ScoreController.GetStatistic`).
 
@@ -577,6 +628,65 @@ on the profile.
 
 ### Phase 4: Website, viewer, Caddy
 
+**Steps 1–2 done and both checks passed 2026-11-08**, on a dev stand-in for
+the vhost: the importer into `.data/plebleader/dev/db/`, the API on 8042
+(`PublicBaseUrl=http://127.0.0.1:8890`), and an ad-hoc `nix shell
+nixpkgs#caddy` on `http://127.0.0.1:8890` with the routes below
+(`.data/plebleader/dev/{run-api.sh,Caddyfile}`). Use `127.0.0.1`, not
+`localhost`: the website sends `localhost` to the real upstream API. Commits:
+server `7cc964a`, website `f5194ee3`, `9ebe84ea`, `8d540642`, all local
+(not pushed yet). Step 3 (nix-pleb) is still open.
+
+- **Phase 3 check passed.** `/u/pleb`: 11,608.35pp, #3,861 global, US #1,643,
+  672 ranked and 3,565 total scores, unranked difficulties with stars (8.33★
+  on the Phase 1 test map), the skill triangle with monthly history. The
+  leaderboard page and `/score/{id}` (accuracy graphs from local score stats)
+  work too.
+- **Server fix:** the profile page calls `GET /v2/player/{id}/hydrate`, which
+  the live API has and the public source doesn't (404, "Player not found").
+  `Controllers/MirrorHydrateController.cs` rebuilds it from
+  `PlayerController.Get` plus the `profile` object that the website's
+  `process-hydrate.js` reads.
+- **Server run note:** `dotnet build` doesn't copy `wwwroot/` into `bin/`.
+  Without it `PreviewController`'s constructor throws on `/images/Star.png`,
+  and any controller that depends on it (`/user`, `/user/config`) returns 500.
+  The dev run passes `--webroot <src>/wwwroot`; the Nix package uses `dotnet
+  publish`, which copies it.
+- **Website, as built:** `IS_MIRROR` in `api-queue.js` is true when the API
+  resolves to `/cors/blapi/`. In that case `BL_REPLAYS_URL` is `/watch/` and
+  score stats come from `/cdn/scorestats/`. Every hardcoded
+  `allpoland.github.io/ArcViewer/` link points at `/watch/` (except the clan
+  page's embed). `public/watch/index.html` takes BL replay-viewer or ArcViewer
+  parameters (`link`, `replayURL`, `scoreId`, `id`, optional `hash`) and
+  redirects to `/arcviewer/?replayURL=…&url=/cdn/maps/<hash>.zip&noProxy=true`.
+  It reads the hash from the replay filename
+  (`<score>-<player>-<diff>-<mode>-<HASH>.bsor`); for `scoreId` it first asks
+  `/cors/blapi/score/{id}` for the replay URL. Phase 5 can pass `hash=` for
+  `.dat` replays. The analyzer icons are off by default (an upstream Patreon
+  service). Clans, events, the socket page and the star-rating graph are still
+  visible; they just show nothing.
+- **ArcViewer facts:** without `noProxy=true` it routes every non-whitelisted
+  URL through `cors.bsmg.dev`, which can't reach a LAN host. When both `id`
+  and `url` are given, `id` wins, so never send both. With no map parameter
+  it looks up the replay's own hash on BeatSaver. The `deploy` build ships
+  uncompressed `Build/` files, so no `Content-Encoding` headers are needed.
+  Its own calls to `api.beatleader.com` (player card, leaderboard info) fail
+  from this origin, so the player card stays blank; fixing that needs a Unity
+  build, so it's left as is.
+- **Map zips:** BeatSaver's CDN serves `https://r2cdn.beatsaver.com/<hash>.zip`
+  with `Access-Control-Allow-Origin: *` (it rejects HEAD, use GET). Caddy
+  redirects to it when the archive has no zip, so the website always links
+  the local path and the ~630 older maps outside `--maps-since` still play.
+- **Nix:** the website fork's `flake.nix` builds `packages.default` (yarn
+  v1 offline, ~16 s; deps hash `sha256-ThGvMJfU…`). Its `postPatch` turns the
+  themes' Google Fonts `@import url(...)` into `@import (css) url(...)`,
+  because less would fetch them at build time. ArcViewer `deploy` at
+  `c776256497b66f7c91a74162cfcd943b0f45ee2e` is
+  `sha256-4A+aVnFrKyWQjFnH2N/CwoVIlwrcgV2fYYrtaOiodo0=` for `fetchFromGitHub`.
+
+Step 3 was done together with phase 7 (see there): the vhost below is in
+nix-pleb, waiting for the switch.
+
 1. In the website fork, point `BL_REPLAYS_URL` (and the "watch replay" links)
    at `/arcviewer/?replayURL=<encoded local replay url>&url=<encoded /cdn/maps/<hash>.zip>`.
    Fall back to `id=<bsr>` only for hashes that have no local zip. Hide or stub
@@ -597,26 +707,91 @@ on the profile.
        @plebleader_allowed remote_ip 127.0.0.1/32 ::1/128 192.168.1.36/32 192.168.254.21/32
        handle @plebleader_allowed {
          handle_path /cors/blapi/* { reverse_proxy http://127.0.0.1:8042 }
-         handle_path /cdn/maps/* { root * /var/lib/plebleader/archive/beatsaver/zips; file_server }
+         handle_path /cdn/maps/* {
+           root * /var/lib/plebleader/archive/beatsaver/zips
+           @nozip not file
+           redir @nozip https://r2cdn.beatsaver.com{path} 302
+           file_server
+         }
          handle_path /cdn/* { root * /var/lib/plebleader/storage; file_server }
          handle_path /arcviewer/* { root * ${arcviewer}; file_server }
          handle_path /cors/cdn-assets-bl/* { reverse_proxy https://cdn.assets.beatleader.com { header_up Host {upstream_hostport} } }
          handle_path /cors/cdnbeatsaver/* { reverse_proxy https://cdn.beatsaver.com { header_up Host {upstream_hostport} } }
-         handle { root * ${plebleader-website}; try_files {path} /index.html; file_server }
+         # {path}/index.html serves /watch/; without it the SPA's 404 answers
+         handle { root * ${plebleader-website}; try_files {path} {path}/index.html /index.html; file_server }
        }
        handle { respond "Forbidden" 403 }
      '';
    };
    ```
 
-   ArcViewer's WebGL build may need `Content-Encoding` headers for `.br`/`.gz`
+   Checked: the `deploy` branch ships uncompressed `Build/` files, so no
-   files in `Build/`. Check which ones the `deploy` branch ships.
+   `Content-Encoding` headers are needed.
 
 **Check:** in Chromium, `https://incineroar.brenise.dev:4442/u/pleb` shows the
 profile and scores, and a replay opens in ArcViewer.
 
 ### Phase 5: Convert ScoreSaber scores and merge
 
+**Built and checked on a scratch copy 2026-11-09; committed, deploy
+pending.** Server `50f7df6` (`Tools/ArchiveImporter/ScoreMerge.cs`, rules in
+the fork's `MIRROR.md`), website `5522f245` (SS badge, `/watch/` for `.dat`),
+exporter `ssMapHash` in this repo. All pushed. Result on the local archive: 3,710 leaderboards with one score
+each (3,566 BL + 141 SS-only BL leaderboards + 3 DLC levels BL lacks). Winners: 3,564 BL main,
+1 BL legacy, 139 SS legacy, 6 SS main; 2,022 SS scores were the same play as
+a BL score and dropped; 20 BL scores got an improvement recomputed against an
+earlier SS play. An unchanged rerun writes 0 rows. What differs from the
+steps below:
+
+- Step 1: **no new columns.** Provenance goes in `Platform`
+  (`scoresaber,<SS player id>`), and `Id` ≥ 1e9 marks SS scores. Columns
+  would have needed a models-fork change, a migration, and edits to the ~90
+  score projections before the website saw them; `Platform` already reaches
+  every score response.
+- Step 2: OST levels are `ost_<LevelId>` on SS and `<levelid>` on BL; 5 of 8
+  resolve that way. Me Porto Bonito, Nuthin' But A "G" Thang and Stressed Out
+  (3 scores) aren't on BL (404), so the importer builds a song and leaderboard
+  from the SS item (`<LevelId><diff><mode>` ids, no ratings, accuracy 0).
+  The exporter now maps `ost_` hashes too (`scoresaber.ts` `ssMapHash`).
+  HMD and controller come from SS `deviceHmd`/`deviceControllerLeft`. SS
+  replays are served from `storage/ssreplays` → `archive/scoresaber`, so the
+  URL is `/cdn/ssreplays/<player>/replays/<id>.dat`.
+- **Pp isn't upstream's.** Neither the fork's `rebalanced-pp` formula
+  (median 0.94x, range 0.69–1.16x) nor upstream `master`'s (0.81x)
+  reproduces the pp on archived BL scores; live BL's code is private. So the
+  fold compares every candidate by **locally computed** pp, which keeps the
+  order right, and BL winners keep upstream's pp. The 70 SS winners on
+  ranked maps show local pp, and improvement pp deltas are local on both
+  sides. Matching pp first sent two BL scores (KillerToy, Make Me) to worse
+  SS plays.
+- Step 3: same-play duplicates (equal base score, ≤ 2 min apart; observed
+  −30 s…+9 s) keep the BL copy. Each upstream `scoreImprovement` is turned
+  back into the best it replaced and added as a candidate, so upstream's
+  improvement stays exactly when that rebuilt best is also the fold's previous
+  best. Older archived BL bests are candidates too.
+- Step 5: the badge sits next to the rank in both `ScoreRank.svelte` (rank 0
+  already renders "-"). `/watch/` takes the map hash from the score
+  (`/score/{id}` → `song.hash`) for `.dat` replays, from `?scoreId=` or
+  1e9 + the id in the filename. Expanding an SS row shows its leaderboard and
+  no stats tab; the page logs a JSON error for the 404 score stats, harmless.
+- Check: The Goddess Figure (Normal) is the case where a 2022 legacy SS
+  play beats the BL score; it shows with the SS badge, and the BL Hard score
+  there now reads "+76,821 vs 4 years ago" (vs the legacy SS play). An SS
+  replay redirects to ArcViewer with the `.dat` and local zip and loads the map;
+  watching it play still needs a real browser.
+
+Deployed 2026-11-09 (nix-pleb `a41f3c2`); the first sync imported 3,710
+scores with the same split by source as the local check. The profile's score
+search then returned 500, for any search: the Lucene song and player indexes
+were written next to the DLL in the read-only `/nix/store` (phase 2 step 6
+had planned to move them). Server `2631c46` adds `SearchIndexPath`
+(MirrorMode, set to `/var/lib/plebleader/search`). Server `81f12ee` then
+moved the rebuild into the importer: it replaces the indexes at the end of a
+run that added scores (so at most once a day, right after the sync), or when
+there are none. The API builds them only when missing; upstream appended every
+song again every 6 h. nix-pleb `bcdd591`, `a313f5b` bump both. The next `plebleader-sync`
+run re-imports with the merge.
+
 ScoreSaber scores become BeatLeader `Score` rows, and each leaderboard keeps
 exactly one score. BeatLeader already shows how a score compares with the
 previous best (`ScoreImprovement`), so after the merge that comparison covers
@@ -687,6 +862,35 @@ the BL score, and confirm the improvement display and the ArcViewer replay.
 
 ### Phase 6: Ongoing sync
 
+**Built 2026-11-09; deployed on incineroar the same day (daily timer at
+17:00, first run clean; see phase 7).** `node scripts/archive/cli.ts sync
+[--ratings] [--no-ss-replays]` (this repo `d182fd8`), packaged as the
+flake's `plebleader-archive`. Tested against the local archive: the token
+refresh rotated the chain (now valid until about 2026-11-23), and with the
+newest BL and SS score files set aside, the run fetched exactly those two
+back, byte-identical, and re-snapshotted the BL leaderboard. What differs
+from the steps below:
+
+- Main account only; both legacy accounts are frozen. The incremental
+  window starts a day before the newest archived score (BL `timepost`, SS
+  `timeSet`), computed from the archive rather than `state.json`. Replays and
+  score stats are retried for every score in the window, not only new ones.
+- New BL hashes get a refreshed leaderboard snapshot; new SS-only hashes get
+  one if they have none. Map zips cover every hash in the window.
+- Ratings drift: Sundays by the machine's clock, or `--ratings`.
+- Each run writes a `sync` section to `state.json` (new score ids, maps
+  without a zip). `exportScores` no longer resets `newestTimepost` to 0 on an
+  empty window.
+- Step 8: the importer runs in full after every sync (about 10 s), as a
+  second `ExecStart` of `plebleader-sync`; no `--since` flag.
+- **Importer fix** (server `dd94772`): the archive keeps an old best after
+  its improvement is synced. The importer now takes only the newest score
+  per leaderboard and removes rows superseded since the last import. Before
+  this, the second score would have broken the
+  `(PlayerId, LeaderboardId, ValidContexts)` unique index. Tested on a
+  scratch copy with a fake improved score: added, old one removed, and back
+  again once the fake was deleted.
+
 The `sync` subcommand runs once a day from a systemd timer on incineroar:
 `OnCalendar=*-*-* 17:00:00 America/Los_Angeles` with `Persistent=true`, so a
 run missed while the machine was off happens at the next boot. By 5 pm the
@@ -727,6 +931,92 @@ the local zip.
 
 ### Phase 7: NixOS deployment on incineroar
 
+**Done 2026-11-09: merged into nix-pleb `main` (fast-forward to `3b3eb47`),
+switched on incineroar, first sync clean, and BL `.bsor` replays play in
+ArcViewer.** Issues hit after the switch:
+
+- **Missing `/run/secrets/plebleader.env`.** The file was encrypted only to
+  litten's age key (the repo `.sops.yaml` rule lists only litten), so
+  incineroar's host key couldn't decrypt it. Fixed by adding incineroar's host
+  key and `age1kuxd…` with `sops rotate --add-age`. Any new incineroar sops
+  file needs the same three recipients as `sops.yaml`.
+- **First sync (13:48):** the token refreshed, with the chain valid to
+  2026-11-23 (each daily refresh moves that forward), 0 new scores, and the
+  importer loaded 3565 scores and 2552 songs in 11 s. The token now lives only
+  in `/var/lib/plebleader/auth/`.
+- **Website "TypeError: Cannot read properties of undefined (reading
+  'id')".** `utils/format.js` takes the first host label as the leaderboard
+  context when the host has more than 2 labels, so the context became
+  `incineroar` and the navbar's context lookup found nothing. The phase 4 dev
+  check missed it because a numeric last label (`127.0.0.1`) raises that limit
+  to 4. Website fix `e5f6279a`: non-BeatLeader hosts always use `general`.
+- **ArcViewer "Download failed! HTTP/1.1 403 Forbidden"** for
+  `/cdn/replays/…bsor`, and `/cdn/maps/…` redirected to BeatSaver even for
+  archived zips. Caddy had been running since 2026-11-07. The switch added it
+  to group `plebleader` but only reloaded it, so the process still had just
+  `caddy` (`grep Groups /proc/<pid>/status`). Fixed with
+  `doas systemctl restart caddy`. Any later change to Caddy's groups needs a
+  restart too.
+
+Phase 6 added a second commit on the
+branch: the `plebleader.env` sops secret, `plebleader-sync` (sync, then the
+importer) and its timer, and lock bumps for `plebsaber-stream` (`d182fd8`)
+and `plebleader-server` (`dd94772`). The `plebsaber-stream` bump also
+rebuilds beldum's site, with no change to the site code.
+
+nix-pleb branch `plebleader`
+(`2ada4b7`, in the worktree `~/ops/nix-pleb-plebleader`, based on `main`
+`981f9e7`) adds the inputs `plebleader-server` (server `0481759`) and
+`plebleader-website` (`8d540642`), both following `nixpkgs-unstable`, plus to
+`incineroar/workstation.nix`: user and group `plebleader` (with `caddy` in
+the group), `plebleader-api`, a one-shot `plebleader-import`, ArcViewer
+`fetchFromGitHub` and the 4442 vhost. The toplevel builds, and the system's
+Caddy adapts the generated Caddyfile. The sync unit and timer and the sops
+secret wait for phase 6's `sync` command. What differs from the bullets
+below:
+
+- Server fork: the private upstream submodules (`beatleader-server-private`,
+  `bs-simulator`) are removed from `.gitmodules`, because a
+  submodule-aware fetch fails on them. The flake sets `self.submodules`; the
+  nix-pleb input uses `?ref=pleb/main&submodules=1` over SSH (the models fork's
+  scp-style URL fetches fine). The package's `version` is a fixed `0.1.0`:
+  `buildDotnetModule` hands it to MSBuild, and a git short rev made restore
+  fail with no message (`NU1105` only in `project.assets.json`). 336 NuGet
+  packages in `deps.json`. The package was tested by hand (importer and API
+  on a scratch DB, all endpoints 200, `wwwroot` included).
+- Website: the yarn v1 flake from phase 4, not `buildNpmPackage`.
+- Units: `WorkingDirectory=/var/lib/plebleader/app` (data protection keys go
+  to `../keys`), `StateDirectory` for `app`, `db`, `storage`, `maps`,
+  `StateDirectoryMode=0750` and `UMask=0027`, so Caddy reads through the group
+  and nothing is world-readable. Paths and `PublicBaseUrl` come from the
+  fork's `appsettings.Mirror.json`, so the units only set
+  `ASPNETCORE_ENVIRONMENT=Mirror`, `ASPNETCORE_URLS` and `HOME`.
+
+To deploy (the user does this):
+
+1. Merge `plebleader` into nix-pleb `main`. The main checkout has uncommitted
+   `flake.lock` changes, so commit or stash those first, or cherry-pick the
+   commit and rerun `nix flake lock` if `flake.lock` conflicts. Then
+   `sudo --preserve-env=SSH_AUTH_SOCK nixos-rebuild switch --flake .#incineroar`.
+   `plebleader-api` starts on empty databases.
+2. Move the archive in (21 GB; `/home` and `/` are separate filesystems, so
+   this is a copy):
+   `doas cp -a ~/ops/beatsaber/plebsaber.stream/.data/plebleader/archive /var/lib/plebleader/`,
+   then `doas chown -R plebleader:plebleader /var/lib/plebleader/archive` and
+   `doas chmod -R g+rX,o-rwx /var/lib/plebleader/archive`.
+3. Move the sync's token chain. **Before 2026-11-23**, or it expires and
+   needs `auth login` again:
+   `doas install -d -o plebleader -g plebleader -m 0700 /var/lib/plebleader/auth`
+   and
+   `doas install -o plebleader -g plebleader -m 0600 ~/ops/beatsaber/plebsaber.stream/.data/plebleader/auth/beatleader_tokens.json /var/lib/plebleader/auth/`,
+   then delete the local copy. From then on, never run `sync` or
+   `auth refresh` locally: the refresh token rotates, and whichever side
+   refreshes second breaks.
+4. `doas systemctl start plebleader-sync` (sync, then import), then open
+   `https://incineroar.brenise.dev:4442/u/pleb` and a replay. Check
+   `journalctl -u plebleader-sync` and `systemctl list-timers plebleader-sync`.
+5. Afterwards, remove the worktree with `git worktree remove ../nix-pleb-plebleader`.
+
 - Packages: `buildDotnetModule` for the fork (generate `deps.json` with
   `nix build .#…passthru.fetch-deps`), `buildNpmPackage` for the website (start
   with `npmDepsHash = lib.fakeHash`, then pin the `got:` value), and
@@ -741,6 +1031,42 @@ the local zip.
   `nix build .#nixosConfigurations.incineroar.config.system.build.toplevel --no-link`.
   The user does the switch.
 
+### Phase 8: Self-hosted BeatLeader web replay viewer
+
+Upstream, the score row's replay icon opens BeatLeader's web viewer
+(`BL_REPLAYS_URL`, replay.beatleader.com), and turning on ArcViewer in the
+score settings adds a separate `altReplay` icon (`Song/Icons.svelte`,
+`webPlayerLink(…, 'arcviewer', …)`). Phase 4 pointed `BL_REPLAYS_URL` at
+`/watch/`, so both icons open ArcViewer now. The user wants the upstream
+behaviour back: the replay icon opens the BeatLeader viewer, and ArcViewer
+stays a separate icon.
+
+Source: `~/src/BeatLeader/BeatSaber-Web-Replays` (webpack 5, npm). Steps:
+
+1. Fork it to gitea `pleb/BeatSaber-Web-Replays` with a `pleb/main` branch,
+   and change as little as possible.
+2. Resolve `getApiUrl()` (`src/utils.js`, `src/open-replay-decoder.js`) to
+   `/cors/blapi` on non-BeatLeader hosts, the same rule as the website's
+   `IS_MIRROR`. Check how it fetches the replay file and the score (`link=`,
+   `scoreId=`) against the mirror's `/cdn/replays/` and `/cors/blapi/score/{id}`.
+3. Maps: `zip-loader.js` tries `cdn.songs.beatleader.xyz`, then
+   `/cors/beat-saver2/api/maps/…` (a Netlify proxy), then `r2cdn`/`cdn.beatsaver.com`.
+   Try `/cdn/maps/<hash>.zip` first, which already falls back to r2cdn in Caddy.
+   Either proxy `/cors/beat-saver2/` to `api.beatsaver.com` in Caddy or skip it.
+4. Drop or stub analytics (googletagmanager), and check its other runtime
+   fetches (fonts, `cdn.jsdelivr.net`). Discord and Patreon are plain links.
+5. Add a flake with `buildNpmPackage` that outputs the built static files
+   (`index.html`, `assets`, `build`, `vendor`, as in its `predeploy` script),
+   and add it as a nix-pleb input.
+6. Caddy: serve it at `/replay/` on the plebleader vhost.
+7. Website: in mirror mode set `BL_REPLAYS_URL` to `/replay/` instead of
+   `/watch/`. Leave the explicit ArcViewer links (`altReplay`, `ReplayButton`
+   with `webPlayer == 'arcviewer'`) on `/watch/`.
+
+**Check:** on `/u/pleb` the replay icon plays a score in the BeatLeader
+viewer from local files, and with ArcViewer turned on in settings, the second
+icon plays it in ArcViewer.
+
 ## Risks and fallbacks
 
 - **The server doesn't build or run on Linux, or EF/SQLite can't translate

flake.nix

@@ -44,12 +44,35 @@
           mainProgram = "plebsaber-stream";
         };
       };
+    # The plebleader archive CLI (scripts/archive, docs/plan.md): plain
+    # TypeScript with only Node built-ins, run through Node's type stripping.
+    mkArchiveCli = pkgs:
+      pkgs.stdenvNoCC.mkDerivation {
+        pname = "plebleader-archive";
+        version = "0.1.0";
+        src = ./scripts/archive;
+        nativeBuildInputs = [ pkgs.makeWrapper ];
+        installPhase = ''
+          runHook preInstall
+          mkdir -p $out/lib/plebleader-archive
+          cp *.ts $out/lib/plebleader-archive/
+          makeWrapper ${pkgs.nodejs_22}/bin/node $out/bin/plebleader-archive \
+            --add-flags "--experimental-strip-types --disable-warning=ExperimentalWarning" \
+            --add-flags "$out/lib/plebleader-archive/cli.ts"
+          runHook postInstall
+        '';
+        meta = {
+          description = "Archive exporter and daily sync for the personal BeatLeader mirror";
+          mainProgram = "plebleader-archive";
+        };
+      };
   in rec {
     packages = forAllSystems (system: let
       pkgs = import nixpkgs { inherit system; };
       pkg = mkPackage pkgs;
     in {
       plebsaber-stream = pkg;
+      plebleader-archive = mkArchiveCli pkgs;
       default = pkg;
     });
 

scripts/archive/beatleader.ts

@@ -3,11 +3,11 @@
 import * as path from 'node:path';
 import { BL_API } from './config.ts';
 import { get, limiters } from './http.ts';
-import { archivePath, eachWithProgress, exists, listFiles, readJson, save, saveSnapshotIfChanged, today, updateState } from './store.ts';
+import { archivePath, eachWithProgress, exists, listFiles, readJson, readState, save, saveSnapshotIfChanged, today, updateState } from './store.ts';
 
 const bl = (...p: string[]) => archivePath('beatleader', ...p);
 
-type BLScore = { id: number; timepost: number; replay?: string | null; leaderboard: { song: { hash: string } } };
+export type BLScore = { id: number; playerId: string; timepost: number; replay?: string | null; leaderboard: { song: { hash: string } } };
 
 async function blGet(pathAndQuery: string, accept?: number[]) {
 	return get(`${BL_API}${pathAndQuery}`, { limiter: limiters.beatleader, accept });
@@ -26,7 +26,9 @@ export async function exportProfile(playerId: string): Promise<void> {
 	const dir = (kind: string) => bl('players', playerId, kind);
 	const endpoints: [string, string][] = [
 		['profile', `/player/${playerId}?stats=true`],
-		['history', `/player/${playerId}/history`],
+		// count: days of history; the default is 50, which loses the triangle's
+		// monthly points. Upstream keeps one row per day since the first score.
+		['history', `/player/${playerId}/history?count=100000`],
 		['triangle', `/player/${playerId}/history/triangle`],
 		['accgraph', `/player/${playerId}/accgraph`]
 	];
@@ -37,12 +39,12 @@ export async function exportProfile(playerId: string): Promise<void> {
 	}
 }
 
-/** Page through best scores newest-first from `sinceUnix`, saving each score object. */
+/** Page through best scores newest-first from `sinceUnix`, saving each score object. Returns the new ones. */
-export async function exportScores(playerId: string, sinceUnix: number): Promise<void> {
+export async function exportScores(playerId: string, sinceUnix: number): Promise<BLScore[]> {
 	console.log(`BeatLeader scores for ${playerId} since ${new Date(sinceUnix * 1000).toISOString()}`);
-	let added = 0;
+	const added: BLScore[] = [];
 	let seen = 0;
-	let newest = 0;
+	let newest = Number(readState()[`beatleader:${playerId}`]?.newestTimepost ?? 0);
 	for (let page = 1; ; page++) {
 		const qs = `count=100&page=${page}&sortBy=date&order=desc&includeIO=true&time_from=${sinceUnix}`;
 		const res = await blGet(`/player/${playerId}/scores?${qs}`);
@@ -54,12 +56,18 @@ export async function exportScores(playerId: string, sinceUnix: number): Promise
 			if (exists(file)) continue;
 			// The page is parsed and each element re-serialized: same content, no reshaping.
 			save(file, JSON.stringify(score));
-			added++;
+			added.push(score);
 		}
-		console.log(`  page ${page}: ${seen}/${json.metadata.total} seen, ${added} new`);
+		console.log(`  page ${page}: ${seen}/${json.metadata.total} seen, ${added.length} new`);
 		if (json.data.length < 100 || seen >= json.metadata.total) break;
 	}
 	updateState(`beatleader:${playerId}`, { scoresSince: sinceUnix, newestTimepost: newest, lastScoresRun: new Date().toISOString() });
+	return added;
+}
+
+/** Newest archived score time for a player (unix seconds), 0 when there is none. */
+export function newestScoreTime(playerId: string): number {
+	return archivedScores().reduce((max, s) => (s.playerId === playerId ? Math.max(max, s.timepost) : max), 0);
 }
 
 export async function exportReplays(scores: BLScore[]): Promise<void> {

scripts/archive/cli.ts

@@ -9,6 +9,10 @@
 //                                   plus all legacy ScoreSaber maps)
 //   export all         [--since D]  bl, ss, maps (not ss-replays)
 //   report             [--since D]  counts and gaps in the archive
+//   sync    [--ratings] [--no-ss-replays]
+//                                   daily incremental run (phase 6): token refresh, new BL and SS
+//                                   scores with their replays, stats, leaderboards and maps.
+//                                   --ratings re-snapshots every played hash (default on Sundays)
 //
 // --since: YYYY-MM-DD, unix seconds, or "all" (default: all). --maps-since defaults to the two-year
 // window (DEFAULT_MAPS_SINCE). Every command skips files that already exist, so reruns only fill gaps.
@@ -16,10 +20,10 @@ import * as fs from 'node:fs';
 import * as path from 'node:path';
 import { ARCHIVE_DIR, BL_API, DEFAULT_MAPS_SINCE, LEGACY_PLAYER_ID, MAIN_PLAYER_ID, PLAYER_IDS, TOKENS_FILE, USER_AGENT, loadDotEnv, parseSince } from './config.ts';
 import { getAccessToken, login, readTokens, refresh } from './auth.ts';
-import { archivedScores, blHashes, exportLeaderboards, exportProfile, exportReplays, exportScoreStats, exportScores } from './beatleader.ts';
+import { archivedScores, blHashes, exportLeaderboards, exportProfile, exportReplays, exportScoreStats, exportScores, newestScoreTime } from './beatleader.ts';
-import { archivedSSScores, exportSSPlayer, exportSSReplays, exportSSScores, ssHashes } from './scoresaber.ts';
+import { archivedSSScores, exportSSPlayer, exportSSReplays, exportSSScores, newestSSScoreTime, ssHashes, ssMapHash } from './scoresaber.ts';
 import { exportMaps } from './beatsaver.ts';
-import { archivePath, exists, listFiles, readState } from './store.ts';
+import { archivePath, exists, listFiles, readState, updateState } from './store.ts';
 
 function fmt(ms: number): string {
 	const d = new Date(ms);
@@ -79,7 +83,7 @@ function playedHashes(since: number): string[] {
 function mapHashes(mapsSince: number): string[] {
 	const legacyWithReplay = archivedSSScores(LEGACY_PLAYER_ID, 0)
 		.filter((it) => exists(archivePath('scoresaber', LEGACY_PLAYER_ID, 'replays', `${it.score.id}.dat`)))
-		.map((it) => it.leaderboard.songHash.toLowerCase());
+		.map(ssMapHash);
 	return [...new Set([...playedHashes(mapsSince), ...legacyWithReplay])].sort();
 }
 
@@ -106,6 +110,64 @@ async function exportSS(since: number): Promise<void> {
 	await exportLeaderboards([...new Set(PLAYER_IDS.flatMap((id) => ssHashes(id, since)))].sort());
 }
 
+/** Overlap of each incremental window, so scores posted late or with a skewed clock aren't missed. */
+const SYNC_OVERLAP_S = 24 * 3600;
+
+/**
+ * Phase 6: the daily run. Main account only; the legacy accounts are frozen. Every step skips
+ * what's already archived, so a rerun (or a run after a missed day) just fills the gap.
+ */
+async function sync(opts: { ratings: boolean; ssReplays: boolean }): Promise<number> {
+	const started = new Date();
+	// Token upkeep first: persist the rotated refresh token before anything else, and fail loudly.
+	const tokens = await refresh();
+	console.log(`Token refreshed; refresh chain valid until about ${fmt(tokens.refresh_expires_at)}\n`);
+
+	await exportProfile(MAIN_PLAYER_ID);
+	const blSince = Math.max(0, newestScoreTime(MAIN_PLAYER_ID) - SYNC_OVERLAP_S);
+	const newBL = await exportScores(MAIN_PLAYER_ID, blSince);
+	// The whole window, not just the new scores, so a replay or stats file that failed yesterday is retried.
+	const blWindow = archivedScores(blSince).filter((s) => s.playerId === MAIN_PLAYER_ID);
+	await exportReplays(blWindow);
+	await exportScoreStats(blWindow);
+	// A new score can change a leaderboard (status, ratings), so re-snapshot its hash.
+	const newBLHashes = [...new Set(newBL.map((s) => s.leaderboard.song.hash.toLowerCase()))].sort();
+	await exportLeaderboards(newBLHashes, true);
+
+	await exportSSPlayer(MAIN_PLAYER_ID);
+	const ssSince = Math.max(0, newestSSScoreTime(MAIN_PLAYER_ID) - SYNC_OVERLAP_S);
+	const newSS = await exportSSScores(MAIN_PLAYER_ID, ssSince);
+	const newSSHashes = [...new Set(newSS.map(ssMapHash))].sort();
+	await exportLeaderboards(newSSHashes.filter((h) => !newBLHashes.includes(h)));
+	if (opts.ssReplays) await exportSSReplays(MAIN_PLAYER_ID, ssSince);
+
+	// Maps for everything played in either window; existing zips are skipped.
+	const windowHashes = [...new Set([...blWindow.map((s) => s.leaderboard.song.hash.toLowerCase()), ...ssHashes(MAIN_PLAYER_ID, ssSince)])].sort();
+	await exportMaps(windowHashes);
+	const noZip = windowHashes.filter((h) => !exists(archivePath('beatsaver', 'zips', `${h}.zip`)));
+
+	if (opts.ratings) {
+		// Upstream rebalances ratings; a new dated snapshot is written only when a hash changed.
+		await exportLeaderboards(playedHashes(0), true);
+	}
+
+	const summary = {
+		lastRun: started.toISOString(),
+		finished: new Date().toISOString(),
+		newBeatLeaderScores: newBL.map((s) => s.id),
+		newScoreSaberScores: newSS.map((it) => it.score.id),
+		ratingsRefreshed: opts.ratings,
+		mapsWithoutZip: noZip
+	};
+	updateState('sync', summary);
+	console.log('');
+	console.log(`Sync done: ${newBL.length} new BeatLeader scores, ${newSS.length} new ScoreSaber scores${opts.ratings ? ', ratings refreshed' : ''}.`);
+	if (noZip.length) console.log(`Maps without a zip (see beatsaver/missing.json; the game's CustomLevels are the manual fallback): ${noZip.join(' ')}`);
+	const failures = readState().failures;
+	if (failures && Object.keys(failures).length) console.warn(`Recorded failures (retried on the next run): ${JSON.stringify(failures)}`);
+	return 0;
+}
+
 function report(since: number, mapsSince: number): void {
 	const count = (...p: string[]) => listFiles(archivePath(...p)).length;
 	const day = (t: number) => (t ? new Date(t * 1000).toISOString().slice(0, 10) : 'all time');
@@ -218,12 +280,17 @@ async function main(): Promise<number> {
 				return 0;
 		}
 	}
+	if (cmd === 'sync') {
+		// Sundays in the machine's timezone (America/Los_Angeles on incineroar).
+		const ratings = args.includes('--ratings') || new Date().getDay() === 0;
+		return sync({ ratings, ssReplays: !args.includes('--no-ss-replays') });
+	}
 	if (cmd === 'report') {
 		report(since, mapsSince);
 		return 0;
 	}
 
-	console.error('Usage: node scripts/archive/cli.ts <auth login|refresh|status | export bl|ss|ss-replays|maps|all | report> [--since YYYY-MM-DD|all] [--maps-since YYYY-MM-DD|all]');
+	console.error('Usage: node scripts/archive/cli.ts <auth login|refresh|status | export bl|ss|ss-replays|maps|all | report | sync [--ratings] [--no-ss-replays]> [--since YYYY-MM-DD|all] [--maps-since YYYY-MM-DD|all]');
 	return 2;
 }
 

scripts/archive/scoresaber.ts

@@ -26,10 +26,10 @@ export async function exportSSPlayer(playerId: string): Promise<void> {
 	console.log(`ScoreSaber player ${playerId}: ${saveSnapshotIfChanged(ss(playerId, 'player'), r.body) ? 'saved' : 'unchanged'}`);
 }
 
-/** Page `sort=recent` until scores are older than `sinceUnix`. */
+/** Page `sort=recent` until scores are older than `sinceUnix`. Returns the new items. */
-export async function exportSSScores(playerId: string, sinceUnix: number): Promise<void> {
+export async function exportSSScores(playerId: string, sinceUnix: number): Promise<SSItem[]> {
 	console.log(`ScoreSaber scores for ${playerId} since ${new Date(sinceUnix * 1000).toISOString()}`);
-	let added = 0;
+	const added: SSItem[] = [];
 	let seen = 0;
 	for (let page = 1; ; page++) {
 		const r = await get(`${SS_API}/player/${playerId}/scores?limit=100&page=${page}&sort=recent`, { limiter: limiters.scoresaber });
@@ -44,12 +44,18 @@ export async function exportSSScores(playerId: string, sinceUnix: number): Promi
 			const file = ss(playerId, 'scores', `${item.score.id}.json`);
 			if (exists(file)) continue;
 			save(file, JSON.stringify(item));
-			added++;
+			added.push(item);
 		}
-		console.log(`  page ${page}: ${seen} in window, ${added} new (total on server ${json.metadata.total})`);
+		console.log(`  page ${page}: ${seen} in window, ${added.length} new (total on server ${json.metadata.total})`);
 		if (reachedCutoff || json.playerScores.length < 100) break;
 	}
 	updateState(`scoresaber:${playerId}`, { scoresSince: sinceUnix, lastScoresRun: new Date().toISOString() });
+	return added;
+}
+
+/** Newest archived ScoreSaber score time for a player (unix seconds), 0 when there is none. */
+export function newestSSScoreTime(playerId: string): number {
+	return archivedSSScores(playerId).reduce((max, it) => Math.max(max, Math.floor(Date.parse(it.score.timeSet) / 1000)), 0);
 }
 
 /** Download replays for archived scores with hasReplay, oldest first, ~10.5 s apart. */
@@ -96,6 +102,14 @@ function seedFromScraper(playerId: string, items: SSItem[]): void {
 	if (copied) console.log(`  copied ${copied} replays from ${dir}`);
 }
 
+/**
+ * The map hash BeatLeader and the archive use for a ScoreSaber item: lowercase, and for official
+ * levels the level id without ScoreSaber's "ost_" prefix (BeatLeader: "beatsaber", SS: "ost_BeatSaber").
+ */
+export function ssMapHash(it: SSItem): string {
+	return it.leaderboard.songHash.toLowerCase().replace(/^ost_/, '');
+}
+
 export function ssHashes(playerId: string, sinceUnix = 0): string[] {
-	return [...new Set(archivedSSScores(playerId, sinceUnix).map((it) => it.leaderboard.songHash.toLowerCase()))].sort();
+	return [...new Set(archivedSSScores(playerId, sinceUnix).map(ssMapHash))].sort();
 }