pleb/plebsaber.stream
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add archive exporter for the personal BeatLeader mirror

Browse Source 
scripts/archive is a CLI that exports BeatLeader and ScoreSaber scores,
replays, score stats, leaderboard snapshots and BeatSaver map zips into an
append-only archive. Anchor the .gitignore 'archive' rule to the repo root
so it no longer hides scripts/archive. Record phase 1 results in the plan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
pleb
2026-11-08 18:30:02 -08:00
co-authored by Claude Opus 5.5
parent 257df6f1d8
commit a759308711
10 changed files with 1093 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plan.md
+95 -31
View File
@@ -17,7 +17,7 @@ Written 2026-11-08. The numbers under "Facts" were checked live on that date.
| Replay viewer | Self-host ArcViewer's prebuilt WebGL build (the `deploy` branch) for both BeatLeader (`.bsor`) and ScoreSaber (`.dat`) replays. No Unity build. |
| Source of truth | A raw, append-only **archive** of unmodified API responses and replay files. The SQLite database can always be rebuilt from it, and so can any future replacement system. |
| Exporter and sync | TypeScript in this repo, building on `src/lib/server/beatleader.ts`, `beatleaderAuth.ts`, `scoresaber.ts` and `beatsaver.ts`. It writes only to the archive. |
| BeatLeader auth | OAuth 2.0 (`profile offline_access`), the same flow as plebsaber.stream's login, which already unlocks unranked ratings. The sync owns its own chain of refresh tokens. |
| BeatLeader auth | OAuth 2.0 (`profile offline_access`). The sync owns its own chain of refresh tokens (`scripts/archive/cli.ts auth login`, seeded 2026-11-08). Unranked ratings do **not** need it: they come from the anonymous `/leaderboards/hash/{hash}` (see Facts). |
| Map files | Archive the BeatSaver zip of every map I've played, using `beatsaver.ts`. Unpack them into the server's `MapsPath`, so the server and ArcViewer never need BeatSaver. |
| Sync schedule | Daily at 17:00 America/Los_Angeles, incineroar's timezone. |
| Access | Only incineroar and snorlax, through Caddy's `remote_ip` allowlist. |
@@ -49,23 +49,32 @@ player/hash/difficulty/characteristic, so it can recover some scores whose
server copy has `hasReplay: false`. `~/ops/beatsaber/backups` deliberately
leaves these folders out.
### Unranked ratings need an authenticated request
### Unranked ratings and auth (corrected 2026-11-08)
- Every difficulty object has `stars`, `passRating`, `accRating`, `techRating`
(the skill triangle), `predictedAcc`, plus `modifiersRating` with
`ss*`/`fs*`/`sf*`/`bfs*`/`bsf*` variants of each rating.
- For **unranked** maps the server fills these in only when the *requesting*
user has `profileSettings.showAllRatings == true`. Mine does. An anonymous
request for the same score returns `stars/pass/acc/techRating: null`; only
`predictedAcc` comes through. See
`Controllers/PlayerScoresController.cs` (`showRatings = currentProfileSettings?.ShowAllRatings`).
- **The solution is OAuth 2.0, as in plebsaber.stream.** Logging in through
`/auth/beatleader/login` requests `scp:profile scp:offline_access`. Requests
sent with `Authorization: Bearer <access_token>` (`createBeatLeaderAPI(fetch,
token)`) come back with unranked ratings filled in. The code is in
`src/lib/server/beatleaderAuth.ts`: `exchangeCodeForTokens`,
`refreshAccessToken`, `getValidAccessToken`, and the token store
`beatleader_tokens.json` in the data dir.
- On **score lists** (`/player/{id}/scores`) and `/leaderboard/{id}`, unranked
ratings are filled in only when the *requesting* user has
`profileSettings.showAllRatings == true`. That check uses
`HttpContext.CurrentUserID`, which reads only the beatleader.com **cookie
session** (with an `Issued` claim). An OAuth bearer token does not count:
tested, bearer and anonymous both return `stars/pass/acc/techRating: null`.
- **`GET /leaderboards/hash/{hash}` returns full ratings for unranked
difficulties, anonymously**, including all `modifiersRating` variants.
Tested on an unranked ExpertPlus (`9db01c9b…`): stars 8.33, pass 3.92, acc
9.75, tech 7.70. Phase 1, step 6 already fetches this for every played hash,
so the importer takes difficulty ratings from these snapshots, not from the
nulls embedded in the scores.
- Bearer tokens only work on `/oauth2/identity`, clan management and
ReeSabers presets (`CurrentOauthUserID`). `GET /user` with a bearer token is
401. The exporter doesn't actually need auth; the token chain is kept for
identity checks and in case a cookie-gated endpoint turns out to matter.
- The sync's chain lives in `scripts/archive/auth.ts`: login via the loopback
redirect `http://localhost:5173/auth/beatleader/callback` (registered on the
OAuth app, as is `https://plebsaber.stream/auth/beatleader/callback`), token
store `<PLEBLEADER_DIR>/auth/beatleader_tokens.json` (default
`.data/plebleader/`, mode 0600). `auth status` checks identity and ratings.
- Token lifetimes: access tokens last 3,600 s and refresh tokens 14 days. Each
refresh **rotates** the refresh token and invalidates the old one. So the
sync job needs its **own** refresh token, separate from the
@@ -111,8 +120,12 @@ ScoreSaber (public, no auth): `GET https://scoresaber.com/api/player/{id}/full`,
`score.hasReplay`), and `GET /api/v2/scores/{scoreId}/replay` (`.dat`). The
ScoreSaber API doesn't send CORS headers, so it only works server-side.
`~/ops/beatsaber/ss-scraper/download_replays.py` already handles pacing
(30 s minimum between replays), 429 backoff, and resuming. At that pace the
1,034 main-account replays take about 9 hours.
(30 s minimum between replays), 429 backoff, and resuming. That pacing is
outdated: as of 2026-11-08 ScoreSaber sends `x-ratelimit-{limit,remaining,reset}-{short,medium,long}`
headers. The replay endpoint allows 20 (short), 60/min (medium) and
**360/hour** (long, the one that binds); player endpoints allow 25/100/400.
`scripts/archive` paces replays at 10.5 s and pauses whenever a window's
remaining count hits 0, so 1,034 replays take about 3 h instead of 9.
### Upstream server: what the fork has to change
@@ -192,7 +205,8 @@ avoids CORS and mixed-content problems.
- Disk: `/` has 479 GB free and `/home` 684 GB. The archive should be 10–25 GB:
about 4.6 GB of best-score replays, about 1.4 GB of ScoreSaber replays, and BeatSaver zips for
roughly 2,000–3,000 played maps (ScoreSaber alone has 1,474 unique hashes),
at a few MB each.
at a few MB each. Actual (2026-11-08): 21 GB with 1,897 zips at about 8 MB
each. Fetching the remaining ~630 older maps later adds about 5 GB.
- Allowed clients: incineroar (`127.0.0.1/32`, `::1/128`, `192.168.1.36/32`) and
snorlax (`192.168.254.21/32`, the address the amelia fallback vhost already
allows).
@@ -265,7 +279,9 @@ the fork, so do it first.
plebsaber.stream; check that its registered callback URLs include the one
used for seeding. Agents never decrypt it.
For local runs before deployment, use the gitignored `.env` in this repo.
5. **Seed the sync's token chain.** Add a small `scripts/archive` command
5. **Done 2026-11-08** (`scripts/archive/cli.ts auth login`; refresh rotation
tested; chain expires about 2026-11-22 unless refreshed).
**Seed the sync's token chain.** Add a small `scripts/archive` command
(`auth login`) that runs the same authorization-code flow as
`/auth/beatleader/login` with `scope=profile offline_access`. It catches the
callback on a loopback port, or reuses a plebsaber.stream callback URL the
@@ -275,6 +291,26 @@ the fork, so do it first.
### Phase 1: Full export (one-time backup)
**Scope (decided 2026-11-08):** all player data for both accounts, all
history: profiles, BL scores, replays, score stats, SS scores and replays,
plus BL leaderboard JSON (ratings) for every played hash. Only **BeatSaver
map files** are limited to maps played in the last two years
(`--maps-since 2024-11-08`, `DEFAULT_MAPS_SINCE` in `scripts/archive/config.ts`).
**Exception:** legacy ScoreSaber scores that have an archived replay get their
map fetched whatever its age, so those replays render (58 replays, 53 maps;
`mapHashes()` in `cli.ts`). Legacy scores without a replay don't get a map. Older map zips come later with `export maps --maps-since all`; existing files
are skipped, so that only fetches what's missing. Replays of older scores
won't render a map in ArcViewer until then. SS main has 1,034 replays (about
3 h at 10.5 s each). The 58 SS legacy replays are copied from ss-scraper's
download instead of being fetched again.
Commands: `node scripts/archive/cli.ts export bl|ss|ss-replays|maps|all
[--since …]` and `report`. The archive lives in `.data/plebleader/archive/`
(set `PLEBLEADER_DIR` to change the root) until `/var/lib/plebleader` exists
on incineroar; then move it. Logs go in `.data/plebleader/logs/`.
Profile, history, triangle and accgraph are stored as dated snapshots in
`players/<id>/<kind>/<YYYY-MM-DD>.json`, written only when changed.
Build `scripts/archive/` as a small CLI (`export bl`, `export ss`,
`export maps`, `import-local-mods`, `sync`). Rules:
@@ -283,14 +319,15 @@ Build `scripts/archive/` as a small CLI (`export bl`, `export ss`,
`state.json`, and write to temp files before renaming.
- Respect rate limits. Use about 4 requests per second to BeatLeader and honor
`Retry-After` (the existing `requestWith429Retry` handles 429s). For
ScoreSaber replays, keep ss-scraper's 30 s pacing.
ScoreSaber replays, stay under the 360/hour limit (10.5 s pacing) and honor
the `x-ratelimit-*` headers.
- Send `User-Agent: plebsaber-archive/0.1`. Don't put an email address in it.
Steps:
1. **Auth check.** With the bearer token, `GET /user` must return `player.id ==
76561199407393962`, and an unranked score's `difficulty.stars` must not be
null. If either fails, stop.
1. **Auth check.** `node scripts/archive/cli.ts auth status`: `/oauth2/identity`
must return `76561199407393962`, and `/leaderboards/hash/{hash}` must return
non-null `stars` for an unranked difficulty. If either fails, stop.
2. **BeatLeader scores.** Page through `/player/{id}/scores` with
`sortBy=date&order=desc&count=100&includeIO=true` until it's exhausted
(about 36 pages). Do the same for the legacy id (3 scores).
@@ -301,8 +338,9 @@ Steps:
6. **BeatLeader map data.** Collect the set of played hashes: every unique
`song.hash` in BL scores, plus `leaderboard.songHash` in both
ScoreSaber accounts. Normalize them to lowercase. For each, call
`GET /leaderboards/hash/{hash}` with the bearer token. That gives ratings
and the triangle for every difficulty, ranked or not.
`GET /leaderboards/hash/{hash}` (no auth needed). That gives ratings and
the triangle for every difficulty, ranked or not. This is the **only**
source of unranked ratings; the scores from step 2 have them nulled.
7. **Map files from BeatSaver.** For each played hash, call
`BeatSaverAPI.getMapByHash(hash)` and save the response to
`beatsaver/maps/<hash>.json`. Then download the zip for the `versions[]`
@@ -311,11 +349,16 @@ Steps:
checks it's a valid zip containing `Info.dat`, then renames it. Go about 2
requests per second and honor 429s. Write each hash that can't be resolved
(deleted map, or an old version gone from the CDN) to `missing.json`.
8. **Recover missing maps from the game.** With the Windows partition mounted
(`mountwin`), hash every folder in the BSManager instance's
`Beat Saber_Data/CustomLevels` and `CustomWIPLevels`. Copy the folders whose
hash is in `missing.json` to `game-levels/<hash>/`. Whatever is still
missing stays listed; those replays can't render the map.
8. **Recover missing maps from the game: optional, low priority.** Decided
2026-11-08: maps gone from BeatSaver are accepted as missing. They were most
likely updated or removed for a reason. The user's online backups and local
map archives from before 2026 don't have them. Older cold storage might, so
that's a "someday" check, not a task. The current BSManager instance's
`CustomLevels` may hold a few recent ones and is cheap to try: with
`mountwin`, hash every folder in `Beat Saber_Data/CustomLevels` and
`CustomWIPLevels`, and copy those whose hash is in `missing.json` to
`game-levels/<hash>/`. Whatever is still missing stays listed; those replays
just can't render the map.
9. **Unpack for the server.** Extract each zip, or copy each recovered folder,
to `maps/<hash>/`. Then the server's `MapDownloader` never goes to the
network, and ArcViewer can load `url=/cdn/maps/<hash>.zip` (serve the zips)
@@ -343,6 +386,26 @@ leaderboard snapshot, plus a BeatSaver zip or recovered game folder, unless
it's listed in `missing.json`. Write a `report` subcommand that prints counts
and what's missing, and paste its output into this file.
**Report, 2026-11-08** (ScoreSaber main replays still downloading):
```
Window: scores since all time, BeatSaver maps since 2024-11-08
BeatLeader scores 3568 in window (3568 archived total, both accounts)
BeatLeader replays 3568/3568
BeatLeader score stats 3568/3568
ScoreSaber 76561199407393962: 2107 scores, replays 435/1034
ScoreSaber 76561197962107242: 199 scores, replays 58/58
Played hashes 2560
BL leaderboard snapshot 2552/2560
BeatSaver zip 1897/1926 in maps window (missing.json lists 29)
Archive size 21.15 GB
```
The 29 missing hashes are all "hash unknown to BeatSaver", and none belongs to
a legacy score with a replay (all 53 of those maps are archived). The 8 hashes
without a BL leaderboard are ScoreSaber-only maps BeatLeader never saw. Zips
average about 8 MB (15 GB for 1,897), more than the estimate below.
### Phase 2: Server fork on SQLite
Work on the `pleb/main` branch of `pleb/beatleader-server`. Keep each change
@@ -607,8 +670,9 @@ the local zip.
far. Sync stops, a failing unit makes that visible, and `auth login`
re-seeds the chain.
- **Maps disappear from BeatSaver.** That's why zips are archived as soon as a
hash first appears. Old versions that are already gone come from the game's
own folders, or stay listed in `missing.json`.
hash first appears. Old versions that are already gone stay listed in
`missing.json`, which is acceptable (decided 2026-11-08). Cold storage might
hold some; check someday, without much hope.
- **Upstream API changes.** The archive stores raw responses, so a change
breaks the importer, not the data. Version the importer's mapping.
- **Upstream code moves fast** (the website is updated daily). Pin fork