pleb/plebsaber.stream
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Plan on-demand map unpacking instead of unpacking every zip upfront

Browse Source 
Nothing on the mirror's read path reads unpacked maps, and ArcViewer loads
the zips directly, so phase 1 step 9 is dropped and a deferred phase 2
step 9 describes a small on-demand adapter.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
pleb
2026-11-08 19:03:13 -08:00
co-authored by Claude Opus 5.5
parent 48dc0db04b
commit d75448af66
1 changed files with 36 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plan.md
+36 -12
View File
@@ -18,7 +18,7 @@ Written 2026-11-08. The numbers under "Facts" were checked live on that date.
| Source of truth | A raw, append-only **archive** of unmodified API responses and replay files. The SQLite database can always be rebuilt from it, and so can any future replacement system. |
| Exporter and sync | TypeScript in this repo, building on `src/lib/server/beatleader.ts`, `beatleaderAuth.ts`, `scoresaber.ts` and `beatsaver.ts`. It writes only to the archive. |
| BeatLeader auth | OAuth 2.0 (`profile offline_access`). The sync owns its own chain of refresh tokens (`scripts/archive/cli.ts auth login`, seeded 2026-11-08). Unranked ratings do **not** need it: they come from the anonymous `/leaderboards/hash/{hash}` (see Facts). |
| Map files | Archive the BeatSaver zip of every map I've played, using `beatsaver.ts`. Unpack them into the server's `MapsPath`, so the server and ArcViewer never need BeatSaver. |
| Map files | Archive the BeatSaver zip of every map I've played, using `beatsaver.ts`. Don't unpack them ahead of time: ArcViewer loads the zip directly, and the server unpacks on demand only if a mirror code path ever needs map files (phase 2, step 9). Neither ever needs BeatSaver. |
| Sync schedule | Daily at 17:00 America/Los_Angeles, incineroar's timezone. |
| Access | Only incineroar and snorlax, through Caddy's `remote_ip` allowlist. |
| Map data scope | Only maps I've played: BeatLeader leaderboards and ratings, plus the BeatSaver metadata and zip. |
@@ -256,7 +256,7 @@ gitea pleb/beatleader-website fork, branch `pleb/main`
state.json sync cursors (last timepost per source/feed)
db/app.db, db/storage.db SQLite, rebuilt from the archive
storage/ S3 replacement (containers as dirs); replays/ and ssreplays/ symlink into archive/
maps/<hash>/ MapsPath: archive zips unpacked (server + /cdn/maps)
maps/<hash>/ MapsPath: unpack cache, filled on demand from archive zips (phase 2, step 9)
auth/beatleader_tokens.json the sync's own OAuth token chain (rotates on refresh)
secrets → EnvironmentFile from sops BL_CLIENT_ID, BL_CLIENT_SECRET
```
@@ -373,10 +373,10 @@ Steps:
`CustomWIPLevels`, and copy those whose hash is in `missing.json` to
`game-levels/<hash>/`. Whatever is still missing stays listed; those replays
just can't render the map.
9. **Unpack for the server.** Extract each zip, or copy each recovered folder,
to `maps/<hash>/`. Then the server's `MapDownloader` never goes to the
network, and ArcViewer can load `url=/cdn/maps/<hash>.zip` (serve the zips)
instead of `id=<bsr>` from BeatSaver.
9. **Unpack for the server: dropped 2026-11-08**, replaced by on-demand
unpacking (phase 2, step 9). ArcViewer loads `url=/cdn/maps/<hash>.zip`
(Caddy serves the archive's zips) instead of `id=<bsr>` from BeatSaver, so
it never needs unpacked folders.
10. **Profile and history.** Save `/player/{id}?stats=true`, `/history`,
`/history/triangle`, `/accgraph`, and `/user` (strip anything secret).
11. **ScoreSaber main.** Page through all scores (22 pages), then download the
@@ -416,9 +416,8 @@ Played hashes 2560
Archive size 21.78 GB
```
Steps 8 (game-folder recovery), 9 (unpack into `maps/`) and 13 (local mod
replays) are still open. Step 9 is needed before phase 4; it can happen when
the archive moves to `/var/lib/plebleader`.
Steps 8 (game-folder recovery) and 13 (local mod replays) are still open.
Step 9 was dropped in favour of on-demand unpacking.
The 29 missing hashes are all "hash unknown to BeatSaver", and none belongs to
a legacy score with a replay (all 53 of those maps are archived). The 8 hashes
@@ -515,6 +514,30 @@ small and in its own commit, so upstream rebases stay manageable.
`MapsPath`, `MirrorMode`). Use development OpenIddict certs, or generate a
self-signed pfx into the state dir.
9. **On-demand map unpacking (deferred; do it only when needed).**
`MapDownloader.Map(hash)` (in the `MapPostprocessor` submodule) returns
`MapsPath/<hash>/` when it exists and otherwise downloads from BeatSaver.
Add a step before the download: if `<MapZipsPath>/<hash>.zip` exists
(the archive's `beatsaver/zips/`), extract it to a temp folder under
`MapsPath`, check it contains `Info.dat`, rename it to `<hash>/` and return
that. Accept upper- and lowercase hashes, like `Map` does. In `MirrorMode`,
never fall through to BeatSaver: return null instead. That makes
`MapsPath` a cache that can be deleted at any time.
- **Technical debt: low.** It's about 20 lines and changes no
interfaces. The catch is where it lives: `MapDownloader` is in
`MapPostprocessor`, so editing it means a fourth fork. Its methods aren't
virtual, so they can't be overridden. Without a fork, add a small
`Utils/LocalMapZips.EnsureUnpacked(hash)` in the server and call it
before `_downloader.Map(...)` at the three call sites
(`ReplayController`, `EarthDayController`, `MapRefresh`). That's a few
extra lines per rebase and no submodule fork, so prefer it.
- **Why deferred:** no read path of the mirror calls `MapDownloader` today.
Its callers are `MapRefresh` (off in `MirrorMode`), the Earth Day event
controller, and replay upload and recalculation in `ReplayController`.
It becomes necessary if the mirror ever recalculates score stats
(for example the optional `.dat` → `.bsor` conversion in phase 5) or
accepts replay uploads.
**Check:** the server starts on an empty SQLite file, `/swagger/index.html`
loads, and `GET /player/76561199407393962` returns 404 rather than 500.
@@ -532,8 +555,9 @@ and upserts by upstream id.
- `leaderboards/<hash>/latest` updates the difficulty ratings, so maps you
played but have no best score for still get them.
- `beatsaver/maps/<hash>.json` fills in `Song` fields that BeatLeader leaves
out (BeatSaver key, description, tags, upload date). The server reads the
map files themselves from `maps/<hash>/`, filled in phase 1, step 9.
out (BeatSaver key, description, tags, upload date). The server
doesn't read map files on any mirror path; if one appears, phase 2, step 9
unpacks them on demand.
- `players/<id>/profile/latest` → `Player`, `PlayerScoreStats`,
`ProfileSettings`, plus history rows from `/history`. Insert the legacy id as
an `AccountLink` or alt pointing to the main player, so `PlayerIdToMain`
@@ -673,7 +697,7 @@ day's play is usually over.
JSON, the replay, the score stats, and a leaderboard refresh for that hash.
2. **Attempts:** skipped (see Decisions).
3. **New maps:** for every hash not in the archive yet, do phase 1, steps 6–9:
BeatLeader leaderboard, BeatSaver JSON and zip, then unpack into `maps/`.
BeatLeader leaderboard, then the BeatSaver JSON and zip (nothing to unpack).
The game's own folders are only a manual fallback, since sync can't mount
Windows. List unresolved hashes in the run summary.
4. **Profile:** one dated snapshot per run, plus `/history`.