From d75448af66d10bda219bea1cb01d0e3dd7d12dec Mon Sep 17 00:00:00 2001 From: pleb Date: Sun, 8 Nov 2026 19:03:13 -0800 Subject: [PATCH] Plan on-demand map unpacking instead of unpacking every zip upfront Nothing on the mirror's read path reads unpacked maps, and ArcViewer loads the zips directly, so phase 1 step 9 is dropped and a deferred phase 2 step 9 describes a small on-demand adapter. Co-Authored-By: Claude Opus 5.5 --- docs/plan.md | 48 ++++++++++++++++++++++++++++++++++++------------ 1 file changed, 36 insertions(+), 12 deletions(-) diff --git a/docs/plan.md b/docs/plan.md index 2dea312..e16a229 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -18,7 +18,7 @@ Written 2026-11-08. The numbers under "Facts" were checked live on that date. | Source of truth | A raw, append-only **archive** of unmodified API responses and replay files. The SQLite database can always be rebuilt from it, and so can any future replacement system. | | Exporter and sync | TypeScript in this repo, building on `src/lib/server/beatleader.ts`, `beatleaderAuth.ts`, `scoresaber.ts` and `beatsaver.ts`. It writes only to the archive. | | BeatLeader auth | OAuth 2.0 (`profile offline_access`). The sync owns its own chain of refresh tokens (`scripts/archive/cli.ts auth login`, seeded 2026-11-08). Unranked ratings do **not** need it: they come from the anonymous `/leaderboards/hash/{hash}` (see Facts). | -| Map files | Archive the BeatSaver zip of every map I've played, using `beatsaver.ts`. Unpack them into the server's `MapsPath`, so the server and ArcViewer never need BeatSaver. | +| Map files | Archive the BeatSaver zip of every map I've played, using `beatsaver.ts`. Don't unpack them ahead of time: ArcViewer loads the zip directly, and the server unpacks on demand only if a mirror code path ever needs map files (phase 2, step 9). Neither ever needs BeatSaver. | | Sync schedule | Daily at 17:00 America/Los_Angeles, incineroar's timezone. | | Access | Only incineroar and snorlax, through Caddy's `remote_ip` allowlist. | | Map data scope | Only maps I've played: BeatLeader leaderboards and ratings, plus the BeatSaver metadata and zip. | @@ -256,7 +256,7 @@ gitea pleb/beatleader-website fork, branch `pleb/main` state.json sync cursors (last timepost per source/feed) db/app.db, db/storage.db SQLite, rebuilt from the archive storage/ S3 replacement (containers as dirs); replays/ and ssreplays/ symlink into archive/ - maps// MapsPath: archive zips unpacked (server + /cdn/maps) + maps// MapsPath: unpack cache, filled on demand from archive zips (phase 2, step 9) auth/beatleader_tokens.json the sync's own OAuth token chain (rotates on refresh) secrets → EnvironmentFile from sops BL_CLIENT_ID, BL_CLIENT_SECRET ``` @@ -373,10 +373,10 @@ Steps: `CustomWIPLevels`, and copy those whose hash is in `missing.json` to `game-levels//`. Whatever is still missing stays listed; those replays just can't render the map. -9. **Unpack for the server.** Extract each zip, or copy each recovered folder, - to `maps//`. Then the server's `MapDownloader` never goes to the - network, and ArcViewer can load `url=/cdn/maps/.zip` (serve the zips) - instead of `id=` from BeatSaver. +9. **Unpack for the server: dropped 2026-11-08**, replaced by on-demand + unpacking (phase 2, step 9). ArcViewer loads `url=/cdn/maps/.zip` + (Caddy serves the archive's zips) instead of `id=` from BeatSaver, so + it never needs unpacked folders. 10. **Profile and history.** Save `/player/{id}?stats=true`, `/history`, `/history/triangle`, `/accgraph`, and `/user` (strip anything secret). 11. **ScoreSaber main.** Page through all scores (22 pages), then download the @@ -416,9 +416,8 @@ Played hashes 2560 Archive size 21.78 GB ``` -Steps 8 (game-folder recovery), 9 (unpack into `maps/`) and 13 (local mod -replays) are still open. Step 9 is needed before phase 4; it can happen when -the archive moves to `/var/lib/plebleader`. +Steps 8 (game-folder recovery) and 13 (local mod replays) are still open. +Step 9 was dropped in favour of on-demand unpacking. The 29 missing hashes are all "hash unknown to BeatSaver", and none belongs to a legacy score with a replay (all 53 of those maps are archived). The 8 hashes @@ -515,6 +514,30 @@ small and in its own commit, so upstream rebases stay manageable. `MapsPath`, `MirrorMode`). Use development OpenIddict certs, or generate a self-signed pfx into the state dir. +9. **On-demand map unpacking (deferred; do it only when needed).** + `MapDownloader.Map(hash)` (in the `MapPostprocessor` submodule) returns + `MapsPath//` when it exists and otherwise downloads from BeatSaver. + Add a step before the download: if `/.zip` exists + (the archive's `beatsaver/zips/`), extract it to a temp folder under + `MapsPath`, check it contains `Info.dat`, rename it to `/` and return + that. Accept upper- and lowercase hashes, like `Map` does. In `MirrorMode`, + never fall through to BeatSaver: return null instead. That makes + `MapsPath` a cache that can be deleted at any time. + - **Technical debt: low.** It's about 20 lines and changes no + interfaces. The catch is where it lives: `MapDownloader` is in + `MapPostprocessor`, so editing it means a fourth fork. Its methods aren't + virtual, so they can't be overridden. Without a fork, add a small + `Utils/LocalMapZips.EnsureUnpacked(hash)` in the server and call it + before `_downloader.Map(...)` at the three call sites + (`ReplayController`, `EarthDayController`, `MapRefresh`). That's a few + extra lines per rebase and no submodule fork, so prefer it. + - **Why deferred:** no read path of the mirror calls `MapDownloader` today. + Its callers are `MapRefresh` (off in `MirrorMode`), the Earth Day event + controller, and replay upload and recalculation in `ReplayController`. + It becomes necessary if the mirror ever recalculates score stats + (for example the optional `.dat` → `.bsor` conversion in phase 5) or + accepts replay uploads. + **Check:** the server starts on an empty SQLite file, `/swagger/index.html` loads, and `GET /player/76561199407393962` returns 404 rather than 500. @@ -532,8 +555,9 @@ and upserts by upstream id. - `leaderboards//latest` updates the difficulty ratings, so maps you played but have no best score for still get them. - `beatsaver/maps/.json` fills in `Song` fields that BeatLeader leaves - out (BeatSaver key, description, tags, upload date). The server reads the - map files themselves from `maps//`, filled in phase 1, step 9. + out (BeatSaver key, description, tags, upload date). The server + doesn't read map files on any mirror path; if one appears, phase 2, step 9 + unpacks them on demand. - `players//profile/latest` → `Player`, `PlayerScoreStats`, `ProfileSettings`, plus history rows from `/history`. Insert the legacy id as an `AccountLink` or alt pointing to the main player, so `PlayerIdToMain` @@ -673,7 +697,7 @@ day's play is usually over. JSON, the replay, the score stats, and a leaderboard refresh for that hash. 2. **Attempts:** skipped (see Decisions). 3. **New maps:** for every hash not in the archive yet, do phase 1, steps 6–9: - BeatLeader leaderboard, BeatSaver JSON and zip, then unpack into `maps/`. + BeatLeader leaderboard, then the BeatSaver JSON and zip (nothing to unpack). The game's own folders are only a manual fallback, since sync can't mount Windows. List unresolved hashes in the run summary. 4. **Profile:** one dated snapshot per run, plus `/history`.