Compare commits
4
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
  plebandClaude Opus 5.5
|
d75448af66 | ||
|
plebandClaude Opus 5.5
|
48dc0db04b | ||
|
plebandClaude Opus 5.5
|
a759308711 | ||
|
plebandClaude Opus 5.5
|
257df6f1d8 |
+1
-1
@@ -24,4 +24,4 @@ vite.config.ts.timestamp-*
|
|||||||
.data
|
.data
|
||||||
|
|
||||||
.env
|
.env
|
||||||
archive
|
/archive
|
||||||
|
|||||||
@@ -17,3 +17,7 @@
|
|||||||
|
|
||||||
- Dont' use `cd`, prefer `pwd`
|
- Dont' use `cd`, prefer `pwd`
|
||||||
- Dont' use `rg`, prefer `grep -r`
|
- Dont' use `rg`, prefer `grep -r`
|
||||||
|
|
||||||
|
## Personal BeatLeader mirror
|
||||||
|
|
||||||
|
- Active project: a self-hosted BeatLeader fork on incineroar holding only pleb's data, plus ScoreSaber history and ArcViewer. Read [docs/plan.md](docs/plan.md) first; it records the decisions, the verified API facts, and the phased steps.
|
||||||
|
|||||||
+793
@@ -0,0 +1,793 @@
|
|||||||
|
# Plan: a personal BeatLeader on incineroar
|
||||||
|
|
||||||
|
Goal: run my own BeatLeader instance on incineroar that holds only my profile
|
||||||
|
(`pleb`) and play history, with ScoreSaber history merged in, and keep it in
|
||||||
|
sync with upstream BeatLeader. The minimum viable version stays a fork of
|
||||||
|
upstream rather than a rewrite. Replacing BeatLeader with my own system is a
|
||||||
|
later project. This plan makes sure the data I collect is ready for that.
|
||||||
|
|
||||||
|
Written 2026-11-08. The numbers under "Facts" were checked live on that date.
|
||||||
|
|
||||||
|
## Decisions
|
||||||
|
|
||||||
|
| Topic | Decision |
|
||||||
|
|---|---|
|
||||||
|
| Server | Fork `BeatLeader/beatleader-server` (ASP.NET Core, net10.0). Swap SQL Server for SQLite and Cloudflare R2 for the local filesystem. Turn off every background job that recalculates ranks or pp. |
|
||||||
|
| Frontend | Fork `BeatLeader/beatleader-website` (Svelte 5 + rollup). Change as little as possible. |
|
||||||
|
| Replay viewer | Self-host ArcViewer's prebuilt WebGL build (the `deploy` branch) for both BeatLeader (`.bsor`) and ScoreSaber (`.dat`) replays. No Unity build. |
|
||||||
|
| Source of truth | A raw, append-only **archive** of unmodified API responses and replay files. The SQLite database can always be rebuilt from it, and so can any future replacement system. |
|
||||||
|
| Exporter and sync | TypeScript in this repo, building on `src/lib/server/beatleader.ts`, `beatleaderAuth.ts`, `scoresaber.ts` and `beatsaver.ts`. It writes only to the archive. |
|
||||||
|
| BeatLeader auth | OAuth 2.0 (`profile offline_access`). The sync owns its own chain of refresh tokens (`scripts/archive/cli.ts auth login`, seeded 2026-11-08). Unranked ratings do **not** need it: they come from the anonymous `/leaderboards/hash/{hash}` (see Facts). |
|
||||||
|
| Map files | Archive the BeatSaver zip of every map I've played, using `beatsaver.ts`. Don't unpack them ahead of time: ArcViewer loads the zip directly, and the server unpacks on demand only if a mirror code path ever needs map files (phase 2, step 9). Neither ever needs BeatSaver. |
|
||||||
|
| Sync schedule | Daily at 17:00 America/Los_Angeles, incineroar's timezone. |
|
||||||
|
| Access | Only incineroar and snorlax, through Caddy's `remote_ip` allowlist. |
|
||||||
|
| Map data scope | Only maps I've played: BeatLeader leaderboards and ratings, plus the BeatSaver metadata and zip. |
|
||||||
|
| Importer | C# inside the server fork, so it shares the EF models. It reads the archive and upserts into SQLite. |
|
||||||
|
| Upstream values | Keep them as they are: pp, ranks, star ratings, and the skill triangle. The local instance has one player, so recalculating would turn every rank into #1. |
|
||||||
|
| Attempts | **Skipped.** No failed or quit attempt replays, and no attempt records (`/scoresstats`). Leaving them out is less work: it's one endpoint the exporter simply doesn't call. Best scores, with their `scoreImprovement`, are the play history. |
|
||||||
|
| ScoreSaber merge | **Convert ScoreSaber scores into BeatLeader `Score` rows.** Each leaderboard keeps one score, chosen across BL, SS main and SS legacy by BeatLeader's own rule. BeatLeader's "improvement over previous best" display then covers ScoreSaber plays too. No separate table. |
|
||||||
|
| Identity | One local player: id `76561199407393962`, name `pleb`. The legacy Steam id `76561197962107242` (`i5u1d0r`) becomes a linked alt of that player. |
|
||||||
|
| Port | `https://incineroar.brenise.dev:4442`. Caddy uses 4430–4441 on incineroar, so 4442 is the next free one (4444 is free too). The API listens on `127.0.0.1:8042`, which is unused. |
|
||||||
|
|
||||||
|
## Facts (checked 2026-11-08)
|
||||||
|
|
||||||
|
### My accounts
|
||||||
|
|
||||||
|
| Source | Id | Scores | Replays |
|
||||||
|
|---|---|---|---|
|
||||||
|
| BeatLeader `pleb` | 76561199407393962 | 3,565 best scores (2,492 unranked); 3,559 play count | one `.bsor` per score on `cdn.replays.beatleader.com`, about 1.3 MB each |
|
||||||
|
| BeatLeader attempts (**not exported**) | same | 16,518 attempt records (`/player/{id}/scoresstats`), including fails and quits | non-finished attempts are at `/otherreplays/<name>.bsor`, auth-gated |
|
||||||
|
| BeatLeader legacy | 76561197962107242 (`i5u1d0r`) | 3 | |
|
||||||
|
| ScoreSaber main | 76561199407393962 (`pleb`) | 2,107 (814 ranked), 1,474 unique maps, first on 2023-11-17 | **1,034** have `hasReplay` |
|
||||||
|
| ScoreSaber legacy | 76561197962107242 (`i5u1d0r`), inactive | 199 (161 ranked), first seen 2022-05-07 | 58 have replays, **already downloaded** to `~/ops/beatsaber/ss-scraper/replays/76561197962107242/` (`.dat` plus a `.json` with metadata each) |
|
||||||
|
|
||||||
|
The local mods keep replays too, as a fallback source. The Windows BSManager
|
||||||
|
instance (`~/Windows/Users/pleb/BSManager/BSInstances/<ver>/`, mounted with
|
||||||
|
`mountwin` on incineroar) has `UserData/BeatLeader/Replays/` and
|
||||||
|
`UserData/ScoreSaber/Replays/`. ScoreSaber keeps one file per
|
||||||
|
player/hash/difficulty/characteristic, so it can recover some scores whose
|
||||||
|
server copy has `hasReplay: false`. `~/ops/beatsaber/backups` deliberately
|
||||||
|
leaves these folders out.
|
||||||
|
|
||||||
|
### Unranked ratings and auth (corrected 2026-11-08)
|
||||||
|
|
||||||
|
- Every difficulty object has `stars`, `passRating`, `accRating`, `techRating`
|
||||||
|
(the skill triangle), `predictedAcc`, plus `modifiersRating` with
|
||||||
|
`ss*`/`fs*`/`sf*`/`bfs*`/`bsf*` variants of each rating.
|
||||||
|
- On **score lists** (`/player/{id}/scores`) and `/leaderboard/{id}`, unranked
|
||||||
|
ratings are filled in only when the *requesting* user has
|
||||||
|
`profileSettings.showAllRatings == true`. That check uses
|
||||||
|
`HttpContext.CurrentUserID`, which reads only the beatleader.com **cookie
|
||||||
|
session** (with an `Issued` claim). An OAuth bearer token does not count:
|
||||||
|
tested, bearer and anonymous both return `stars/pass/acc/techRating: null`.
|
||||||
|
- **`GET /leaderboards/hash/{hash}` returns full ratings for unranked
|
||||||
|
difficulties, anonymously**, including all `modifiersRating` variants.
|
||||||
|
Tested on an unranked ExpertPlus (`9db01c9b…`): stars 8.33, pass 3.92, acc
|
||||||
|
9.75, tech 7.70. Phase 1, step 6 already fetches this for every played hash,
|
||||||
|
so the importer takes difficulty ratings from these snapshots, not from the
|
||||||
|
nulls embedded in the scores.
|
||||||
|
- Bearer tokens only work on `/oauth2/identity`, clan management and
|
||||||
|
ReeSabers presets (`CurrentOauthUserID`). `GET /user` with a bearer token is
|
||||||
|
401. The exporter doesn't actually need auth; the token chain is kept for
|
||||||
|
identity checks and in case a cookie-gated endpoint turns out to matter.
|
||||||
|
- The sync's chain lives in `scripts/archive/auth.ts`: login via the loopback
|
||||||
|
redirect `http://localhost:5173/auth/beatleader/callback` (registered on the
|
||||||
|
OAuth app, as is `https://plebsaber.stream/auth/beatleader/callback`), token
|
||||||
|
store `<PLEBLEADER_DIR>/auth/beatleader_tokens.json` (default
|
||||||
|
`.data/plebleader/`, mode 0600). `auth status` checks identity and ratings.
|
||||||
|
- Token lifetimes: access tokens last 3,600 s and refresh tokens 14 days. Each
|
||||||
|
refresh **rotates** the refresh token and invalidates the old one. So the
|
||||||
|
sync job needs its **own** refresh token, separate from the
|
||||||
|
plebsaber.stream site's, or the two invalidate each other. It persists every
|
||||||
|
new refresh token atomically before using the new access token. The daily
|
||||||
|
sync stays well within the 14 days. If the chain breaks (no run for 14 days,
|
||||||
|
or a failed write), log in once more through plebsaber.stream's
|
||||||
|
`/auth/beatleader/login` and seed the new token.
|
||||||
|
|
||||||
|
### BeatLeader endpoints the exporter uses
|
||||||
|
|
||||||
|
Base `https://api.beatleader.com`. Rate limit: 50 GETs per 10 s overall,
|
||||||
|
10 per 10 s on `/players`, 8 per second on `/maps`. Swagger is at `/swagger/index.html`.
|
||||||
|
|
||||||
|
| Data | Endpoint | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| identity and settings | `GET /user` | confirms the token works (`player.id`) |
|
||||||
|
| profile | `GET /player/{id}?stats=true` | `scoreStats`, `history`, pinned scores, badges |
|
||||||
|
| best scores | `GET /player/{id}/scores?count=100&page=N&sortBy=date&order=desc&includeIO=true` | `count` ≤ 100. Each score includes `leaderboard.song` and `leaderboard.difficulty` (with ratings), `replay`, `offsets`, `scoreImprovement`. Incremental runs use `time_from=<unix>`. |
|
||||||
|
| replay stats | `GET /score/statistic/{scoreId}` | the accuracy graph, hit tracker and so on, from the server's `scorestats` storage |
|
||||||
|
| map data | `GET /leaderboards/hash/{hash}` | every difficulty of a song, with ratings when the request is authenticated |
|
||||||
|
| rank history | `GET /player/{id}/history`, `/history/triangle`, `/accgraph` | |
|
||||||
|
| live feed (optional) | `wss://sockets.api.beatleader.com/scores` | all scores globally. Filter on my id for near-real-time sync. |
|
||||||
|
|
||||||
|
BeatSaver (public, no auth): `BeatSaverAPI.getMapByHash(hash)` in
|
||||||
|
`src/lib/server/beatsaver.ts` calls `GET https://api.beatsaver.com/maps/hash/{hash}`.
|
||||||
|
Each entry in `versions[]` carries its own `hash` and `downloadURL` (the zip).
|
||||||
|
Each map version has its own hash, and scores refer to a version hash, so
|
||||||
|
download the version whose `hash` matches. Old versions may only be on the CDN
|
||||||
|
or nowhere at all. When a version or map is gone from BeatSaver, take it from
|
||||||
|
the game's own folders on the Windows side instead: `Beat Saber_Data/CustomLevels`
|
||||||
|
and `CustomWIPLevels` in the BSManager instance. The folders there are
|
||||||
|
unpacked maps, so compute their hash the way the game does: SHA-1 over
|
||||||
|
`Info.dat` plus each difficulty file, in order.
|
||||||
|
|
||||||
|
The server's `MapsPath` (`MapPostprocessor/MapDownloader.cs`) wants
|
||||||
|
`<MapsPath>/<hash>/` folders containing an unpacked map with `Info.dat`. It
|
||||||
|
checks the lowercase and uppercase hash. Only when a folder is missing does it
|
||||||
|
fetch `beatsaver.com/api/maps/hash/{hash}` and unpack the zip.
|
||||||
|
|
||||||
|
ScoreSaber (public, no auth): `GET https://scoresaber.com/api/player/{id}/full`,
|
||||||
|
`GET /api/player/{id}/scores?limit=100&page=N&sort=recent` (each item has
|
||||||
|
`score.hasReplay`), and `GET /api/v2/scores/{scoreId}/replay` (`.dat`). The
|
||||||
|
ScoreSaber API doesn't send CORS headers, so it only works server-side.
|
||||||
|
`~/ops/beatsaber/ss-scraper/download_replays.py` already handles pacing
|
||||||
|
(30 s minimum between replays), 429 backoff, and resuming. That pacing is
|
||||||
|
outdated: as of 2026-11-08 ScoreSaber sends `x-ratelimit-{limit,remaining,reset}-{short,medium,long}`
|
||||||
|
headers. The replay endpoint allows 20 (short), 60/min (medium) and
|
||||||
|
**360/hour** (long, the one that binds); player endpoints allow 25/100/400.
|
||||||
|
`scripts/archive` paces replays at 10.5 s and pauses whenever a window's
|
||||||
|
remaining count hits 0, so 1,034 replays take about 3 h instead of 9.
|
||||||
|
|
||||||
|
### Upstream server: what the fork has to change
|
||||||
|
|
||||||
|
These checkouts are in `~/src`. Run `git pull` before relying on them.
|
||||||
|
|
||||||
|
- `~/src/BeatLeader/beatleader-server` (master was at 2026-07-19 `1474312`; the
|
||||||
|
live branch `rebalanced-pp` was updated 2026-10-25). Its public submodules are
|
||||||
|
initialized: `beatleader-server-models`, `beatleader-parser`, `ReplayDecoder`,
|
||||||
|
`MapPostprocessor`, `RatingAPI`. Two submodules are **private and
|
||||||
|
unreachable**: `beatleader-server-private` and `bs-simulator`. Upstream
|
||||||
|
already copes with that. `Startup.cs` falls back to `ReplayRecalculatorStub`,
|
||||||
|
and nothing in the public code imports a namespace that only the private
|
||||||
|
repos define. **Verified 2026-11-08:** it builds on Linux with nixpkgs
|
||||||
|
`dotnet-sdk_10` (10.0.403) with both private folders empty: 0 errors,
|
||||||
|
about 2,300 warnings, about 1 minute.
|
||||||
|
- Database: `UseSqlServer` for two contexts (`Startup.cs:380-386`):
|
||||||
|
`AppContext` (`DefaultConnection`) and `StorageContext`
|
||||||
|
(`StorageConnection`, which holds `PlayerLeaderboardStats`). There are 570
|
||||||
|
SQL Server migrations.
|
||||||
|
- `Z.EntityFramework.Extensions.EFCore` is commercial, with a monthly trial.
|
||||||
|
Call sites: about 92 `BulkSaveChanges`, 56 `BulkUpdate`, 19 `BulkDelete`,
|
||||||
|
3 `BulkInsert`.
|
||||||
|
- Raw SQL in 5 places: `DBCC CHECKIDENT` (`Services/DailyRefresh.cs`),
|
||||||
|
`TABLESAMPLE` (`ScoreController.GetRandomScore`), `TRUNCATE` and admin
|
||||||
|
statements (`Controllers/Admin/AdminController.cs`).
|
||||||
|
- Storage: all 31 helpers in `Utils/S3Helper.cs` go to Cloudflare R2
|
||||||
|
(`S3AccountID.r2.cloudflarestorage.com`) and return
|
||||||
|
`https://cdn.{container}.beatleader.com/{file}`. Containers: `replays`,
|
||||||
|
`otherreplays`, `assets`, `previews`, `scorestats`, `playlists`, `unicode`,
|
||||||
|
`configs`, `replayedvalues`, `reepresets`, `songs`, `attempts`. Nothing calls
|
||||||
|
S3 outside this file.
|
||||||
|
- Background jobs (`Startup.cs:455-478`): `EveryMinuteRefresh`,
|
||||||
|
`MinuteRefresh`, `HourlyRefresh`, `DailyRefresh`, `HistoryService`,
|
||||||
|
`BotService`, `RankingService`, `SongService`, `RichPresenceService` and
|
||||||
|
`MapRefresh` run only when `ServicesHost=YES`. These always run:
|
||||||
|
`ConstantsService`, `RefreshTaskService`, `ClanTaskService`,
|
||||||
|
`ClanMessageService`, `ClanCallbackService`, `SearchService` (Lucene),
|
||||||
|
`OauthService`, `LeaderboardPlayerStatsService`.
|
||||||
|
- Config keys, all from `IConfiguration`: `DefaultConnection`,
|
||||||
|
`StorageConnection`, `S3*`, `MapsPath` (default `/home/maps`), `SteamKey`,
|
||||||
|
`SteamApi`, `BeatSaverId/Secret`, `Openiddict*` (production expects
|
||||||
|
`../keys/openiddict_certificate*.pfx`; development uses dev certs), plus
|
||||||
|
optional OAuth secrets for Patreon, Twitch, Discord and others. Those
|
||||||
|
providers register only when their keys are set.
|
||||||
|
- The star-rating models are public: `RatingAPI/` (C# with ONNX models) and
|
||||||
|
`beatleader-analyzer`. We don't need them for the minimum version, because
|
||||||
|
we keep upstream's ratings. They are what would let a later replacement rate
|
||||||
|
maps itself.
|
||||||
|
|
||||||
|
### Upstream website
|
||||||
|
|
||||||
|
`~/src/BeatLeader/beatleader-website` (2026-11-07). In
|
||||||
|
`src/network/queues/beatleader/api-queue.js`, on any host it doesn't recognize
|
||||||
|
the API base is **`/cors/blapi/`**, which is relative to the page. So our host
|
||||||
|
only needs Caddy to proxy `/cors/blapi/*` to the API, with no code change. It
|
||||||
|
rewrites assets to `/cors/cdn-assets-bl/` and `/cors/cdnbeatsaver/` in the same
|
||||||
|
way. `BL_REPLAYS_URL` and `BL_ANALYZER_URL` fall back to the public
|
||||||
|
`replay.beatleader.com` and need a patch. Build with `npm run build` (rollup)
|
||||||
|
into `public/`.
|
||||||
|
|
||||||
|
### ArcViewer
|
||||||
|
|
||||||
|
`~/src/AllPoland/ArcViewer` (Unity 6000.0.67f1). The `deploy` branch on GitHub
|
||||||
|
holds a ready WebGL site (`index.html`, `Build/`, `TemplateData/`). It loads
|
||||||
|
replays from `?replayURL=<url>` and recognizes ScoreSaber replays by their
|
||||||
|
magic header (`ScoreSaberDecoder.IsScoreSaberReplay`, including the legacy
|
||||||
|
format), so it works for both `.bsor` and `.dat`. The map comes from `id=<bsr>`
|
||||||
|
(BeatSaver) or `url=<zip>`. Serving it from the same origin as the replays
|
||||||
|
avoids CORS and mixed-content problems.
|
||||||
|
|
||||||
|
### incineroar
|
||||||
|
|
||||||
|
- Caddy vhosts are in `~/ops/nix-pleb/systems/brenise.dev/incineroar/workstation.nix`.
|
||||||
|
They follow the pattern `"${host_fqdn}:44xx"` with `${tlsConfig}` (Namecheap
|
||||||
|
DNS-01) and a `remote_ip` allowlist (`127.0.0.1/32 ::1/128 192.168.1.34/32
|
||||||
|
192.168.1.35/32 192.168.1.36/32 10.19.21.34/32`). The host firewall is off;
|
||||||
|
Caddy does the gating.
|
||||||
|
- Disk: `/` has 479 GB free and `/home` 684 GB. The archive should be 10–25 GB:
|
||||||
|
about 4.6 GB of best-score replays, about 1.4 GB of ScoreSaber replays, and BeatSaver zips for
|
||||||
|
roughly 2,000–3,000 played maps (ScoreSaber alone has 1,474 unique hashes),
|
||||||
|
at a few MB each. Actual (2026-11-08): 21 GB with 1,897 zips at about 8 MB
|
||||||
|
each. Fetching the remaining ~630 older maps later adds about 5 GB.
|
||||||
|
- Allowed clients: incineroar (`127.0.0.1/32`, `::1/128`, `192.168.1.36/32`) and
|
||||||
|
snorlax (`192.168.254.21/32`, the address the amelia fallback vhost already
|
||||||
|
allows).
|
||||||
|
- Timezone: `America/Los_Angeles`.
|
||||||
|
- nix-pleb rule: put host app config in an existing `<system>/<purpose>.nix`
|
||||||
|
(here `workstation.nix`), not a new module file. Never use `nixos-rebuild
|
||||||
|
switch` without asking; use `build`. Never decrypt sops secrets: ask the user
|
||||||
|
to add the values.
|
||||||
|
|
||||||
|
## Layout
|
||||||
|
|
||||||
|
```
|
||||||
|
plebsaber.stream/ (this repo)
|
||||||
|
docs/plan.md this file
|
||||||
|
scripts/archive/ exporter and sync CLI (TypeScript, node ≥ 22)
|
||||||
|
src/lib/server/beatleader.ts extend: bearer auth from the sync's token store, paging, time_from, raw passthrough
|
||||||
|
src/lib/server/scoresaber.ts extend: player scores, replay download
|
||||||
|
|
||||||
|
gitea pleb/beatleader-server fork, branch `pleb/main` (based on upstream `rebalanced-pp`)
|
||||||
|
MIRROR.md what MirrorMode changes and how to run it
|
||||||
|
appsettings.Mirror.json incineroar values (ASPNETCORE_ENVIRONMENT=Mirror)
|
||||||
|
Tools/ArchiveImporter/ C# console app sharing AppContext/StorageContext
|
||||||
|
gitea pleb/beatleader-server-models fork of the models submodule, branch `pleb/main`
|
||||||
|
gitea pleb/beatleader-website fork, branch `pleb/main`
|
||||||
|
|
||||||
|
/var/lib/plebleader/ on incineroar, user `plebleader`
|
||||||
|
archive/ source of truth, append-only
|
||||||
|
beatleader/
|
||||||
|
user.json latest /user (no secrets)
|
||||||
|
players/<id>/profile/<YYYY-MM-DD>.json
|
||||||
|
players/<id>/history.json, triangle.json, accgraph.json
|
||||||
|
scores/<scoreId>.json one ScoreResponse each, unmodified
|
||||||
|
scorestats/<scoreId>.json
|
||||||
|
leaderboards/<hash>/<YYYY-MM-DD>.json new snapshot only when the content changed
|
||||||
|
replays/<original filename>.bsor
|
||||||
|
scoresaber/<playerId>/
|
||||||
|
player/<YYYY-MM-DD>.json
|
||||||
|
scores/<scoreId>.json the full item (score plus leaderboard)
|
||||||
|
replays/<scoreId>.dat
|
||||||
|
beatsaver/
|
||||||
|
maps/<hash>.json getMapByHash response, unmodified
|
||||||
|
zips/<hash>.zip the map version matching that hash
|
||||||
|
missing.json hashes BeatSaver no longer serves, with their source if recovered
|
||||||
|
game-levels/<hash>/ maps recovered from the Windows CustomLevels when BeatSaver lacks them
|
||||||
|
local-mods/{beatleader,scoresaber}/... copied from the Windows UserData
|
||||||
|
state.json sync cursors (last timepost per source/feed)
|
||||||
|
db/app.db, db/storage.db SQLite, rebuilt from the archive
|
||||||
|
storage/ S3 replacement (containers as dirs); replays/ and ssreplays/ symlink into archive/
|
||||||
|
maps/<hash>/ MapsPath: unpack cache, filled on demand from archive zips (phase 2, step 9)
|
||||||
|
auth/beatleader_tokens.json the sync's own OAuth token chain (rotates on refresh)
|
||||||
|
secrets → EnvironmentFile from sops BL_CLIENT_ID, BL_CLIENT_SECRET
|
||||||
|
```
|
||||||
|
|
||||||
|
## Phases
|
||||||
|
|
||||||
|
Each phase ends with a check. Phase 1 protects the data and doesn't depend on
|
||||||
|
the fork, so do it first.
|
||||||
|
|
||||||
|
### Phase 0: Prep
|
||||||
|
|
||||||
|
1. **Done 2026-11-08** (all were already current; upstream server `master`
|
||||||
|
`1474312`, `rebalanced-pp` `52a50ed`, website `2b3ef891`, ArcViewer `deploy`
|
||||||
|
`c776256`, ScoreSaber website `836d789`). Update the checkouts:
|
||||||
|
`git -C ~/src/BeatLeader/beatleader-server pull --recurse-submodules`, and
|
||||||
|
the same for `beatleader-website`, `~/src/AllPoland/ArcViewer` (also
|
||||||
|
`git fetch origin deploy`), and `~/src/ScoreSaber/website`. The ScoreSaber
|
||||||
|
website is public now; use it to check the score and replay API shapes.
|
||||||
|
2. **Done 2026-11-08.** Create the Gitea forks `pleb/beatleader-server` and `pleb/beatleader-website`
|
||||||
|
with the `gitea-api` skill (mirror or push). Keep `upstream` as a remote so
|
||||||
|
you can rebase later.
|
||||||
|
Created as private, non-mirror imports through `POST /repos/migrate` (the
|
||||||
|
Gitea token lacks `write:user`, which `POST /user/repos` needs), with
|
||||||
|
default branch `pleb/main`. The `~/src` checkouts keep `origin` = GitHub and
|
||||||
|
gained a `gitea` remote; `pleb/main` is checked out there. The server's
|
||||||
|
`pleb/main` starts from `rebalanced-pp`, which is `master` plus 4 commits
|
||||||
|
and what the live API most likely runs, so the entities match the archived
|
||||||
|
responses. `pleb/beatleader-server-models` was forked too (see phase 2,
|
||||||
|
step 7); the server's `.gitmodules` points at it over SSH.
|
||||||
|
3. Bring this repo's flake up to date (it's still on `nixos-25.05`) only if the
|
||||||
|
exporter gets packaged here.
|
||||||
|
4. Ask the user to add a sops secret `plebleader.env` for incineroar, holding
|
||||||
|
`BL_CLIENT_ID` and `BL_CLIENT_SECRET`. These are the same OAuth app as
|
||||||
|
plebsaber.stream; check that its registered callback URLs include the one
|
||||||
|
used for seeding. Agents never decrypt it.
|
||||||
|
For local runs before deployment, use the gitignored `.env` in this repo.
|
||||||
|
5. **Done 2026-11-08** (`scripts/archive/cli.ts auth login`; refresh rotation
|
||||||
|
tested; chain expires about 2026-11-22 unless refreshed).
|
||||||
|
**Seed the sync's token chain.** Add a small `scripts/archive` command
|
||||||
|
(`auth login`) that runs the same authorization-code flow as
|
||||||
|
`/auth/beatleader/login` with `scope=profile offline_access`. It catches the
|
||||||
|
callback on a loopback port, or reuses a plebsaber.stream callback URL the
|
||||||
|
OAuth app already allows. It writes `auth/beatleader_tokens.json`. Don't
|
||||||
|
copy the plebsaber.stream site's own refresh token, because rotation would
|
||||||
|
break whichever side refreshes second.
|
||||||
|
|
||||||
|
### Phase 1: Full export (one-time backup)
|
||||||
|
|
||||||
|
**Scope (decided 2026-11-08):** all player data for both accounts, all
|
||||||
|
history: profiles, BL scores, replays, score stats, SS scores and replays,
|
||||||
|
plus BL leaderboard JSON (ratings) for every played hash. Only **BeatSaver
|
||||||
|
map files** are limited to maps played in the last two years
|
||||||
|
(`--maps-since 2024-11-08`, `DEFAULT_MAPS_SINCE` in `scripts/archive/config.ts`).
|
||||||
|
**Exception:** legacy ScoreSaber scores that have an archived replay get their
|
||||||
|
map fetched whatever its age, so those replays render (58 replays, 53 maps;
|
||||||
|
`mapHashes()` in `cli.ts`). Legacy scores without a replay don't get a map. Older map zips come later with `export maps --maps-since all`; existing files
|
||||||
|
are skipped, so that only fetches what's missing. Replays of older scores
|
||||||
|
won't render a map in ArcViewer until then. SS main has 1,034 replays (about
|
||||||
|
3 h at 10.5 s each). The 58 SS legacy replays are copied from ss-scraper's
|
||||||
|
download instead of being fetched again.
|
||||||
|
|
||||||
|
Commands: `node scripts/archive/cli.ts export bl|ss|ss-replays|maps|all
|
||||||
|
[--since …]` and `report`. The archive lives in `.data/plebleader/archive/`
|
||||||
|
(set `PLEBLEADER_DIR` to change the root) until `/var/lib/plebleader` exists
|
||||||
|
on incineroar; then move it. Logs go in `.data/plebleader/logs/`.
|
||||||
|
Profile, history, triangle and accgraph are stored as dated snapshots in
|
||||||
|
`players/<id>/<kind>/<YYYY-MM-DD>.json`, written only when changed.
|
||||||
|
|
||||||
|
Build `scripts/archive/` as a small CLI (`export bl`, `export ss`,
|
||||||
|
`export maps`, `import-local-mods`, `sync`). Rules:
|
||||||
|
|
||||||
|
- Store responses byte-for-byte. Never reshape them on the way in.
|
||||||
|
- Make every step resumable: skip files that already exist, keep cursors in
|
||||||
|
`state.json`, and write to temp files before renaming.
|
||||||
|
- Respect rate limits. Use about 4 requests per second to BeatLeader and honor
|
||||||
|
`Retry-After` (the existing `requestWith429Retry` handles 429s). For
|
||||||
|
ScoreSaber replays, stay under the 360/hour limit (10.5 s pacing) and honor
|
||||||
|
the `x-ratelimit-*` headers.
|
||||||
|
- Send `User-Agent: plebsaber-archive/0.1`. Don't put an email address in it.
|
||||||
|
|
||||||
|
Steps:
|
||||||
|
|
||||||
|
1. **Auth check.** `node scripts/archive/cli.ts auth status`: `/oauth2/identity`
|
||||||
|
must return `76561199407393962`, and `/leaderboards/hash/{hash}` must return
|
||||||
|
non-null `stars` for an unranked difficulty. If either fails, stop.
|
||||||
|
2. **BeatLeader scores.** Page through `/player/{id}/scores` with
|
||||||
|
`sortBy=date&order=desc&count=100&includeIO=true` until it's exhausted
|
||||||
|
(about 36 pages). Do the same for the legacy id (3 scores).
|
||||||
|
3. **Attempts: skipped.** Don't call `/scoresstats` or `/otherreplays`.
|
||||||
|
4. **Replays.** Fetch every `scores/*.replay` from the CDN (no auth, CORS `*`).
|
||||||
|
Keep the original filenames.
|
||||||
|
5. **Score stats.** `GET /score/statistic/{id}` for each best score.
|
||||||
|
6. **BeatLeader map data.** Collect the set of played hashes: every unique
|
||||||
|
`song.hash` in BL scores, plus `leaderboard.songHash` in both
|
||||||
|
ScoreSaber accounts. Normalize them to lowercase. For each, call
|
||||||
|
`GET /leaderboards/hash/{hash}` (no auth needed). That gives ratings and
|
||||||
|
the triangle for every difficulty, ranked or not. This is the **only**
|
||||||
|
source of unranked ratings; the scores from step 2 have them nulled.
|
||||||
|
7. **Map files from BeatSaver.** For each played hash, call
|
||||||
|
`BeatSaverAPI.getMapByHash(hash)` and save the response to
|
||||||
|
`beatsaver/maps/<hash>.json`. Then download the zip for the `versions[]`
|
||||||
|
entry whose `hash` matches, to `beatsaver/zips/<hash>.zip`. Extend
|
||||||
|
`beatsaver.ts` with a `downloadMapZip(version)` that streams to a temp file,
|
||||||
|
checks it's a valid zip containing `Info.dat`, then renames it. Go about 2
|
||||||
|
requests per second and honor 429s. Write each hash that can't be resolved
|
||||||
|
(deleted map, or an old version gone from the CDN) to `missing.json`.
|
||||||
|
8. **Recover missing maps from the game: optional, low priority.** Decided
|
||||||
|
2026-11-08: maps gone from BeatSaver are accepted as missing. They were most
|
||||||
|
likely updated or removed for a reason. The user's online backups and local
|
||||||
|
map archives from before 2026 don't have them. Older cold storage might, so
|
||||||
|
that's a "someday" check, not a task. The current BSManager instance's
|
||||||
|
`CustomLevels` may hold a few recent ones and is cheap to try: with
|
||||||
|
`mountwin`, hash every folder in `Beat Saber_Data/CustomLevels` and
|
||||||
|
`CustomWIPLevels`, and copy those whose hash is in `missing.json` to
|
||||||
|
`game-levels/<hash>/`. Whatever is still missing stays listed; those replays
|
||||||
|
just can't render the map.
|
||||||
|
9. **Unpack for the server: dropped 2026-11-08**, replaced by on-demand
|
||||||
|
unpacking (phase 2, step 9). ArcViewer loads `url=/cdn/maps/<hash>.zip`
|
||||||
|
(Caddy serves the archive's zips) instead of `id=<bsr>` from BeatSaver, so
|
||||||
|
it never needs unpacked folders.
|
||||||
|
10. **Profile and history.** Save `/player/{id}?stats=true`, `/history`,
|
||||||
|
`/history/triangle`, `/accgraph`, and `/user` (strip anything secret).
|
||||||
|
11. **ScoreSaber main.** Page through all scores (22 pages), then download the
|
||||||
|
1,034 replays. Either port the `download_replays.py` logic to TypeScript or
|
||||||
|
run it as-is with `-o archive/scoresaber`, and normalize the layout.
|
||||||
|
12. **ScoreSaber legacy.** Copy the 58 replays and the metadata from
|
||||||
|
`~/ops/beatsaber/ss-scraper/replays/76561197962107242/` into
|
||||||
|
`archive/scoresaber/76561197962107242/` (the filenames start with the
|
||||||
|
scoreId). Fetch the 199 score items again so the archive has complete,
|
||||||
|
unmodified responses.
|
||||||
|
13. **Local mod replays.** With the Windows partition mounted (`mountwin`, then
|
||||||
|
`umount ~/Windows` afterwards), copy `UserData/BeatLeader/Replays` and
|
||||||
|
`UserData/ScoreSaber/Replays` into `archive/local-mods/`. Afterwards, match
|
||||||
|
the ScoreSaber files against scores that have `hasReplay: false` by
|
||||||
|
player/hash/difficulty/characteristic.
|
||||||
|
|
||||||
|
**Check:** the counts match the live API: 3,565 BL scores, 3 legacy,
|
||||||
|
2,107 plus 199 SS scores, and replay files for every score with a
|
||||||
|
replay URL or `hasReplay`. Every played hash should have a BeatLeader
|
||||||
|
leaderboard snapshot, plus a BeatSaver zip or recovered game folder, unless
|
||||||
|
it's listed in `missing.json`. Write a `report` subcommand that prints counts
|
||||||
|
and what's missing, and paste its output into this file.
|
||||||
|
|
||||||
|
**Report, 2026-11-08 (phase 1 complete).** The ScoreSaber replays finished
|
||||||
|
with zero 429s.
|
||||||
|
|
||||||
|
```
|
||||||
|
Window: scores since all time, BeatSaver maps since 2024-11-08
|
||||||
|
BeatLeader scores 3568 in window (3568 archived total, both accounts)
|
||||||
|
BeatLeader replays 3568/3568
|
||||||
|
BeatLeader score stats 3568/3568
|
||||||
|
ScoreSaber 76561199407393962: 2107 scores, replays 1034/1034
|
||||||
|
ScoreSaber 76561197962107242: 199 scores, replays 58/58
|
||||||
|
Played hashes 2560
|
||||||
|
BL leaderboard snapshot 2552/2560
|
||||||
|
BeatSaver zip 1897/1926 in maps window (missing.json lists 29)
|
||||||
|
Archive size 21.78 GB
|
||||||
|
```
|
||||||
|
|
||||||
|
Steps 8 (game-folder recovery) and 13 (local mod replays) are still open.
|
||||||
|
Step 9 was dropped in favour of on-demand unpacking.
|
||||||
|
|
||||||
|
The 29 missing hashes are all "hash unknown to BeatSaver", and none belongs to
|
||||||
|
a legacy score with a replay (all 53 of those maps are archived). The 8 hashes
|
||||||
|
without a BL leaderboard are ScoreSaber-only maps BeatLeader never saw. Zips
|
||||||
|
average about 8 MB (15 GB for 1,897), more than the estimate below.
|
||||||
|
|
||||||
|
### Phase 2: Server fork on SQLite
|
||||||
|
|
||||||
|
**Done 2026-11-08** (`pleb/main` at `1741ce9`, models fork at `8f233e8`).
|
||||||
|
What differs from the steps below:
|
||||||
|
|
||||||
|
- Step 2: `Microsoft.Data.SqlClient` stays as a direct package, because one
|
||||||
|
controller catches `SqlException`. One controller had an unused
|
||||||
|
`using BeatLeader_Server.Migrations;` that broke the build once the old
|
||||||
|
migrations were gone. The fresh migrations create 155 tables and 217
|
||||||
|
indexes. WAL and migrations are applied at startup by `Utils/MirrorSetup.cs`;
|
||||||
|
`busy_timeout` comes from `Default Timeout=30` in the connection string.
|
||||||
|
- Step 3: `Utils/BulkCompat.cs` sits in the global namespace, like Z's own
|
||||||
|
extensions, because some callers have no EF `using`. 49 of the `BulkUpdate`
|
||||||
|
calls pass `ColumnInputExpression`, often on detached stubs that carry only
|
||||||
|
the key and those columns, so the shim marks only the listed columns
|
||||||
|
modified and detaches the stubs after saving. A plain `UpdateRange` would
|
||||||
|
have wiped every other column. **Not exercised at runtime yet**; the
|
||||||
|
importer will be the first real user.
|
||||||
|
- Step 4: the two index-defragmentation scripts became `PRAGMA optimize` and
|
||||||
|
`REINDEX; ANALYZE;`.
|
||||||
|
- Step 5: `GetS3Client` returns a `LocalStorageClient` (an `AmazonS3Client`
|
||||||
|
subclass) when `StoragePath` is set; the six S3 primitives branch on it.
|
||||||
|
Upstream behaviour is unchanged without `StoragePath`.
|
||||||
|
- Step 6: `MirrorMode` also skips `RefreshTaskService` and `MapRefresh`, and
|
||||||
|
the external OAuth providers. Two surprises showed up in testing:
|
||||||
|
`GetLazy` fetched unknown players from Steam (it now returns null, so 404),
|
||||||
|
and the BeatSaver login is registered unconditionally and validated on
|
||||||
|
every request (it gets placeholder credentials, otherwise every request
|
||||||
|
was a 500).
|
||||||
|
- Step 7: instead of patching about 35 `showRatings` decisions, the four
|
||||||
|
`HideRatings()` leaves return early when `MirrorSettings.ShowAllRatings`
|
||||||
|
(in the models fork) is set; `MirrorSetup` sets it. Sorting and filtering by
|
||||||
|
stars on unranked maps still follow the requesting user's settings; patch
|
||||||
|
`Utils/{Map,Leaderboard}ListUtils/Sort.cs` if that matters.
|
||||||
|
- Step 8: `appsettings.Mirror.json` loads with `ASPNETCORE_ENVIRONMENT=Mirror`;
|
||||||
|
`MIRROR.md` in the fork documents it. OpenIddict uses ephemeral keys.
|
||||||
|
Run the DLL with `--contentRoot <bin>` from a working directory in the state
|
||||||
|
dir, because data protection keys go to `../keys/`. `dotnet run` ignores
|
||||||
|
`ASPNETCORE_URLS` in favour of `launchSettings.json`.
|
||||||
|
|
||||||
|
Check result: on empty files, `/swagger/index.html` 200,
|
||||||
|
`/player/76561199407393962` 404, `/players` and `/leaderboards` 200 with empty
|
||||||
|
pages, no `fail:` lines in the log.
|
||||||
|
|
||||||
|
Work on the `pleb/main` branch of `pleb/beatleader-server`. Keep each change
|
||||||
|
small and in its own commit, so upstream rebases stay manageable.
|
||||||
|
|
||||||
|
1. **Build on Linux** before changing anything:
|
||||||
|
`nix shell nixpkgs#dotnet-sdk_10 -c dotnet build BeatLeader-Server.csproj`
|
||||||
|
(nixpkgs has SDK 10.0.403). Record which errors come from the missing
|
||||||
|
private submodules versus the platform. Delete or exclude the empty
|
||||||
|
`beatleader-server-private` and `bs-simulator` folders, and stub whatever
|
||||||
|
they turn out to supply.
|
||||||
|
2. **SQLite.** Replace `Microsoft.EntityFrameworkCore.SqlServer` with
|
||||||
|
`...Sqlite`, and point both `AddDbContextFactory` calls at `UseSqlite`
|
||||||
|
(`Data Source=/var/lib/plebleader/db/app.db` and `storage.db`). On
|
||||||
|
connection open, set `PRAGMA journal_mode=WAL; busy_timeout=5000;`. Move the
|
||||||
|
570 SQL Server migrations aside and generate one fresh migration per context
|
||||||
|
(`dotnet ef migrations add InitialSqlite --context AppContext`, and the same
|
||||||
|
for `StorageContext`), so later schema changes still work.
|
||||||
|
3. **Drop Z.EntityFramework.Extensions.** Add `Utils/BulkCompat.cs` with
|
||||||
|
extension methods of the same names and shapes: `BulkSaveChanges` →
|
||||||
|
`SaveChanges`, `BulkUpdate` → `UpdateRange` + `SaveChanges`, `BulkDelete` →
|
||||||
|
`RemoveRange` + `SaveChanges`, `BulkInsert` → `AddRange` + `SaveChanges`.
|
||||||
|
With one player, performance doesn't matter, and about 170 call sites keep
|
||||||
|
compiling unchanged.
|
||||||
|
4. **Raw SQL.** `DBCC CHECKIDENT` → `DELETE FROM sqlite_sequence WHERE name IN (...)`.
|
||||||
|
`TABLESAMPLE` → `ORDER BY RANDOM() LIMIT 1`. `TRUNCATE` → `DELETE FROM`.
|
||||||
|
5. **Storage.** Rewrite the bodies of the 31 `S3Helper` methods to read and
|
||||||
|
write `${StoragePath}/{container}/{filename}`, and return
|
||||||
|
`${PublicBaseUrl}/cdn/{container}/{filename}`. `GetS3Client` can return a
|
||||||
|
client that's never used, so the method signatures don't change.
|
||||||
|
6. **Background jobs.** Leave `ServicesHost` unset. Add a config flag
|
||||||
|
(`MirrorMode=true`) that also skips registering `RankingService`, the clan
|
||||||
|
services, `OauthService` and `LeaderboardPlayerStatsService`. Keep
|
||||||
|
`SearchService` and `ConstantsService` if they start cleanly, and point
|
||||||
|
Lucene's index directory at the state dir.
|
||||||
|
7. **Auth.** For the minimum version the site is read-only behind Caddy's IP
|
||||||
|
allowlist. No login is needed: the website shows a public profile. If the
|
||||||
|
ratings in API responses still depend on `ShowAllRatings` for the
|
||||||
|
*requesting* user, then in `MirrorMode` treat requests as coming from
|
||||||
|
`76561199407393962`, or just store ratings so they always show. That means
|
||||||
|
patching the `showRatings` checks, which all read
|
||||||
|
`ProfileSettings.ShowAllRatings`, to `true`.
|
||||||
|
8. **Configuration.** Put an `appsettings.Production.json` template in the fork
|
||||||
|
and the real values in the NixOS unit's environment (`ASPNETCORE_URLS=http://127.0.0.1:8042`,
|
||||||
|
`ConnectionStrings`/`DefaultConnection`, `StoragePath`, `PublicBaseUrl`,
|
||||||
|
`MapsPath`, `MirrorMode`). Use development OpenIddict certs, or generate a
|
||||||
|
self-signed pfx into the state dir.
|
||||||
|
|
||||||
|
9. **On-demand map unpacking (deferred; do it only when needed).**
|
||||||
|
`MapDownloader.Map(hash)` (in the `MapPostprocessor` submodule) returns
|
||||||
|
`MapsPath/<hash>/` when it exists and otherwise downloads from BeatSaver.
|
||||||
|
Add a step before the download: if `<MapZipsPath>/<hash>.zip` exists
|
||||||
|
(the archive's `beatsaver/zips/`), extract it to a temp folder under
|
||||||
|
`MapsPath`, check it contains `Info.dat`, rename it to `<hash>/` and return
|
||||||
|
that. Accept upper- and lowercase hashes, like `Map` does. In `MirrorMode`,
|
||||||
|
never fall through to BeatSaver: return null instead. That makes
|
||||||
|
`MapsPath` a cache that can be deleted at any time.
|
||||||
|
- **Technical debt: low.** It's about 20 lines and changes no
|
||||||
|
interfaces. The catch is where it lives: `MapDownloader` is in
|
||||||
|
`MapPostprocessor`, so editing it means a fourth fork. Its methods aren't
|
||||||
|
virtual, so they can't be overridden. Without a fork, add a small
|
||||||
|
`Utils/LocalMapZips.EnsureUnpacked(hash)` in the server and call it
|
||||||
|
before `_downloader.Map(...)` at the three call sites
|
||||||
|
(`ReplayController`, `EarthDayController`, `MapRefresh`). That's a few
|
||||||
|
extra lines per rebase and no submodule fork, so prefer it.
|
||||||
|
- **Why deferred:** no read path of the mirror calls `MapDownloader` today.
|
||||||
|
Its callers are `MapRefresh` (off in `MirrorMode`), the Earth Day event
|
||||||
|
controller, and replay upload and recalculation in `ReplayController`.
|
||||||
|
It becomes necessary if the mirror ever recalculates score stats
|
||||||
|
(for example the optional `.dat` → `.bsor` conversion in phase 5) or
|
||||||
|
accepts replay uploads.
|
||||||
|
|
||||||
|
**Check:** the server starts on an empty SQLite file, `/swagger/index.html`
|
||||||
|
loads, and `GET /player/76561199407393962` returns 404 rather than 500.
|
||||||
|
|
||||||
|
### Phase 3: Importer
|
||||||
|
|
||||||
|
`Tools/ArchiveImporter` lives in the fork, references the main project, and
|
||||||
|
uses its `AppContext`/`StorageContext`. Run it as a one-shot that is idempotent
|
||||||
|
and upserts by upstream id.
|
||||||
|
|
||||||
|
- `beatleader/scores/*.json` (ScoreResponse) → `Song`,
|
||||||
|
`DifficultyDescription` (keep the upstream ratings and `ModifiersRating` as
|
||||||
|
they are), `Leaderboard`, `Score` (with upstream `Id`, `Rank`, `Pp`,
|
||||||
|
`Weight`, and `Timepost`), `ScoreImprovement`, `ReplayOffsets`. Rewrite
|
||||||
|
`Replay` to the local `/cdn/replays/<file>`.
|
||||||
|
- `leaderboards/<hash>/latest` updates the difficulty ratings, so maps you
|
||||||
|
played but have no best score for still get them.
|
||||||
|
- `beatsaver/maps/<hash>.json` fills in `Song` fields that BeatLeader leaves
|
||||||
|
out (BeatSaver key, description, tags, upload date). The server
|
||||||
|
doesn't read map files on any mirror path; if one appears, phase 2, step 9
|
||||||
|
unpacks them on demand.
|
||||||
|
- `players/<id>/profile/latest` → `Player`, `PlayerScoreStats`,
|
||||||
|
`ProfileSettings`, plus history rows from `/history`. Insert the legacy id as
|
||||||
|
an `AccountLink` or alt pointing to the main player, so `PlayerIdToMain`
|
||||||
|
resolves `/u/76561197962107242` to `pleb`. Its 3 BL scores go through the
|
||||||
|
phase 5 merge like any other candidate.
|
||||||
|
- `StorageContext` (attempts) stays empty, because attempts aren't exported.
|
||||||
|
- `scorestats/*.json` → the `scorestats` storage container, using the filename
|
||||||
|
the server expects (check `ScoreController.GetStatistic`).
|
||||||
|
|
||||||
|
The response DTOs aren't the entities, and this mapping is the largest single
|
||||||
|
piece of work. Use `beatleader-server-models/Models/*` (entities) and
|
||||||
|
`ResponseUtils.cs` (entity → DTO) as the reference, working backwards.
|
||||||
|
|
||||||
|
**Check:** the stock website (phase 4) shows my profile with the right score
|
||||||
|
count, the pp and ranks from upstream, unranked star ratings, and the triangle
|
||||||
|
on the profile.
|
||||||
|
|
||||||
|
### Phase 4: Website, viewer, Caddy
|
||||||
|
|
||||||
|
1. In the website fork, point `BL_REPLAYS_URL` (and the "watch replay" links)
|
||||||
|
at `/arcviewer/?replayURL=<encoded local replay url>&url=<encoded /cdn/maps/<hash>.zip>`.
|
||||||
|
Fall back to `id=<bsr>` only for hashes that have no local zip. Hide or stub
|
||||||
|
the features that can't work offline: clans, events, friends, the Socket
|
||||||
|
page, and the stage analyzer and `ppai2` calls in `StarRatingGraph.svelte` /
|
||||||
|
`star-generator.js`. Leave the `/cors/blapi/` resolution as it is.
|
||||||
|
2. Package the ArcViewer `deploy` branch with `fetchFromGitHub` at a pinned
|
||||||
|
revision, as static files.
|
||||||
|
3. Add a Caddy vhost in `incineroar/workstation.nix`:
|
||||||
|
|
||||||
|
```
|
||||||
|
"${host_fqdn}:4442" = {
|
||||||
|
# Personal BeatLeader mirror (plebleader). Website + ArcViewer static,
|
||||||
|
# API on 8042, replays/assets/maps from the state dir.
|
||||||
|
# Only incineroar itself and snorlax (192.168.254.21).
|
||||||
|
extraConfig = ''
|
||||||
|
${tlsConfig}
|
||||||
|
@plebleader_allowed remote_ip 127.0.0.1/32 ::1/128 192.168.1.36/32 192.168.254.21/32
|
||||||
|
handle @plebleader_allowed {
|
||||||
|
handle_path /cors/blapi/* { reverse_proxy http://127.0.0.1:8042 }
|
||||||
|
handle_path /cdn/maps/* { root * /var/lib/plebleader/archive/beatsaver/zips; file_server }
|
||||||
|
handle_path /cdn/* { root * /var/lib/plebleader/storage; file_server }
|
||||||
|
handle_path /arcviewer/* { root * ${arcviewer}; file_server }
|
||||||
|
handle_path /cors/cdn-assets-bl/* { reverse_proxy https://cdn.assets.beatleader.com { header_up Host {upstream_hostport} } }
|
||||||
|
handle_path /cors/cdnbeatsaver/* { reverse_proxy https://cdn.beatsaver.com { header_up Host {upstream_hostport} } }
|
||||||
|
handle { root * ${plebleader-website}; try_files {path} /index.html; file_server }
|
||||||
|
}
|
||||||
|
handle { respond "Forbidden" 403 }
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
ArcViewer's WebGL build may need `Content-Encoding` headers for `.br`/`.gz`
|
||||||
|
files in `Build/`. Check which ones the `deploy` branch ships.
|
||||||
|
|
||||||
|
**Check:** in Chromium, `https://incineroar.brenise.dev:4442/u/pleb` shows the
|
||||||
|
profile and scores, and a replay opens in ArcViewer.
|
||||||
|
|
||||||
|
### Phase 5: Convert ScoreSaber scores and merge
|
||||||
|
|
||||||
|
ScoreSaber scores become BeatLeader `Score` rows, and each leaderboard keeps
|
||||||
|
exactly one score. BeatLeader already shows how a score compares with the
|
||||||
|
previous best (`ScoreImprovement`), so after the merge that comparison covers
|
||||||
|
ScoreSaber plays too. This is importer logic in the fork. The archive keeps
|
||||||
|
the raw ScoreSaber responses as they are.
|
||||||
|
|
||||||
|
1. **Provenance columns.** Add nullable `Source` (`null` = BeatLeader,
|
||||||
|
`"scoresaber"`) and `SourceScoreId` / `SourcePlayerId` to `Score`. This goes
|
||||||
|
into the fresh SQLite migration from phase 2. The website uses them for a
|
||||||
|
small "SS" badge, and the importer uses them to stay idempotent.
|
||||||
|
2. **Convert** each ScoreSaber item (`scoresaber/*/scores/*.json`) into a
|
||||||
|
candidate `Score`:
|
||||||
|
- Leaderboard: `leaderboard.songHash` (lowercased), plus
|
||||||
|
`leaderboard.difficulty.difficulty`. ScoreSaber uses 1/3/5/7/9 for
|
||||||
|
Easy…ExpertPlus, the same numbers as BeatLeader's `difficulty.value`.
|
||||||
|
`gameMode` `SoloStandard` → mode `Standard` (strip `Solo`). Resolve it to
|
||||||
|
the BL leaderboard from `leaderboards/<hash>` (phase 1, step 6). If BL has
|
||||||
|
none, create `Song`/`DifficultyDescription`/`Leaderboard` from the
|
||||||
|
BeatSaver JSON, with no ratings.
|
||||||
|
- `BaseScore`, `ModifiedScore`, `Modifiers` (the same comma-separated codes),
|
||||||
|
`MissedNotes`, `BadCuts`, `FullCombo`, `MaxCombo`, `Timeset`/`Timepost`
|
||||||
|
come from `timeSet`. Set `Platform = "scoresaber"` and `Hmd` = unknown,
|
||||||
|
unless the codes are mapped.
|
||||||
|
- `Accuracy = BaseScore / maxScore`, taking `maxScore` from the **BL**
|
||||||
|
difficulty. ScoreSaber reports `maxScore: 0` on many unranked
|
||||||
|
leaderboards. Check this against how `ReplayUtils` computes BL accuracy.
|
||||||
|
- `Pp`: `ReplayUtils.PpFromScore(score, difficulty)`, using the archived BL
|
||||||
|
ratings, when the BL difficulty is ranked; otherwise 0, as BeatLeader
|
||||||
|
does. ScoreSaber's own pp and rank aren't comparable, so keep them only in
|
||||||
|
the archive. Set `Rank` to 0, and have the website show "—" for
|
||||||
|
ScoreSaber-sourced rows.
|
||||||
|
- `Id = 1_000_000_000 + ssScoreId`. ScoreSaber ids are about 95M and
|
||||||
|
upstream BL ids about 151M, so this can't collide and still fits in an
|
||||||
|
int32.
|
||||||
|
- `Replay = /cdn/ssreplays/<ssScoreId>.dat` when the archive has one (from
|
||||||
|
the server, or recovered from local mod replays); otherwise null. No
|
||||||
|
`ReplayOffsets` and no score stats, because the server can't decode `.dat`.
|
||||||
|
3. **Merge per leaderboard.** The candidates are the upstream BL best, the
|
||||||
|
ScoreSaber main best, the ScoreSaber legacy best, and the legacy BL score.
|
||||||
|
Sort them by time and fold them with BeatLeader's own
|
||||||
|
`ReplayUtils.IsNewScoreBetter(old, new)`: compare by pp when either score
|
||||||
|
has pp, otherwise by modified score, where a no-fail score loses to a clean
|
||||||
|
one. The winner becomes the leaderboard's `Score`.
|
||||||
|
- `ScoreImprovement` on the winner is the difference from the best before it
|
||||||
|
in the fold: score, accuracy, pp, misses, bad cuts, modifiers, timeset. If
|
||||||
|
the winner is the upstream BL score and nothing from ScoreSaber came
|
||||||
|
between it and its previous BL best, keep upstream's `scoreImprovement`
|
||||||
|
as it is.
|
||||||
|
- Candidates that lose stay in the archive only.
|
||||||
|
4. **Player totals.** Keep upstream's `PlayerScoreStats`, pp and rank. They
|
||||||
|
won't count ScoreSaber scores that won. Recomputing the weighted pp total
|
||||||
|
locally is an optional later step; label the profile numbers "BeatLeader,
|
||||||
|
as of <sync time>".
|
||||||
|
5. **Website.** Show the "SS" badge on ScoreSaber-sourced scores, hide the
|
||||||
|
accuracy graph and stats for them (there are no score stats), and send their
|
||||||
|
replay button to `/arcviewer/?replayURL=/cdn/ssreplays/<id>.dat&url=/cdn/maps/<hash>.zip`.
|
||||||
|
Every score shows under the name `pleb`; `i5u1d0r` survives only as
|
||||||
|
`SourcePlayerId`.
|
||||||
|
|
||||||
|
Optional later: convert `.dat` into `.bsor`, using ArcViewer's
|
||||||
|
`ScoreSaberDecoder` as the reference for the format. The server's statistics
|
||||||
|
code could then produce score stats for ScoreSaber-sourced scores too.
|
||||||
|
|
||||||
|
**Check:** the leaderboard count is the union of the BL and SS leaderboards I
|
||||||
|
played, with exactly one score each. The importer prints how many winners came
|
||||||
|
from each source. Spot-check a map where an old legacy ScoreSaber score beats
|
||||||
|
the BL score, and confirm the improvement display and the ArcViewer replay.
|
||||||
|
|
||||||
|
### Phase 6: Ongoing sync
|
||||||
|
|
||||||
|
The `sync` subcommand runs once a day from a systemd timer on incineroar:
|
||||||
|
`OnCalendar=*-*-* 17:00:00 America/Los_Angeles` with `Persistent=true`, so a
|
||||||
|
run missed while the machine was off happens at the next boot. By 5 pm the
|
||||||
|
day's play is usually over.
|
||||||
|
|
||||||
|
1. **BeatLeader scores:** `/player/{id}/scores?sortBy=date&order=desc&time_from=<cursor-1d>`.
|
||||||
|
Overlap a day and skip ids you already have. New score ids → save the score
|
||||||
|
JSON, the replay, the score stats, and a leaderboard refresh for that hash.
|
||||||
|
2. **Attempts:** skipped (see Decisions).
|
||||||
|
3. **New maps:** for every hash not in the archive yet, do phase 1, steps 6–9:
|
||||||
|
BeatLeader leaderboard, then the BeatSaver JSON and zip (nothing to unpack).
|
||||||
|
The game's own folders are only a manual fallback, since sync can't mount
|
||||||
|
Windows. List unresolved hashes in the run summary.
|
||||||
|
4. **Profile:** one dated snapshot per run, plus `/history`.
|
||||||
|
5. **Ratings drift:** on Sundays' run, re-fetch `/leaderboards/hash/{hash}` for
|
||||||
|
every map I've played, and store a new snapshot only when it changed.
|
||||||
|
Upstream rebalances ratings (see the `rebalanced-pp` branch), and keeping
|
||||||
|
the old snapshots preserves the history.
|
||||||
|
6. **ScoreSaber main:** `sort=recent` until reaching a known score id, then
|
||||||
|
download the new replays at the slow pace. A day's worth fits easily in one
|
||||||
|
run. Legacy is frozen; skip it.
|
||||||
|
7. **Token upkeep:** refresh the OAuth token at the start of each run and write
|
||||||
|
the rotated refresh token to `auth/beatleader_tokens.json` (temp file plus
|
||||||
|
rename) before doing anything else. A daily run stays far inside the 14-day
|
||||||
|
window. If the refresh fails, the unit fails loudly and the message says to
|
||||||
|
run `auth login` again. Alloy ships journal lines to Loki, so set up an
|
||||||
|
alert there.
|
||||||
|
8. Once the archive has changed, run the importer incrementally, through a
|
||||||
|
`--since` flag or by watching `state.json`. That includes the phase 5 merge
|
||||||
|
for every leaderboard that got a new BL or SS score.
|
||||||
|
|
||||||
|
Optional later: a manual `sync` run (`systemctl start plebleader-sync`) for
|
||||||
|
when I want results before 5 pm.
|
||||||
|
|
||||||
|
**Check:** play a map, and after the next 17:00 run the score, its replay and
|
||||||
|
the map file show up on the local site, and the replay opens in ArcViewer with
|
||||||
|
the local zip.
|
||||||
|
|
||||||
|
### Phase 7: NixOS deployment on incineroar
|
||||||
|
|
||||||
|
- Packages: `buildDotnetModule` for the fork (generate `deps.json` with
|
||||||
|
`nix build .#…passthru.fetch-deps`), `buildNpmPackage` for the website (start
|
||||||
|
with `npmDepsHash = lib.fakeHash`, then pin the `got:` value), and
|
||||||
|
`fetchFromGitHub` for ArcViewer `deploy`. The forks expose their packages
|
||||||
|
through their own flakes and become flake inputs in nix-pleb, next to the
|
||||||
|
existing `plebsaber-stream` input. Use `git+https://gitea.satstack.dev/pleb/<repo>.git`.
|
||||||
|
- Services: put `plebleader-api`, `plebleader-sync` (service and timer) and the
|
||||||
|
Caddy vhost in `incineroar/workstation.nix`. Don't create a new module file.
|
||||||
|
Use a static user `plebleader`, `StateDirectory=plebleader`, and an
|
||||||
|
`EnvironmentFile` from the sops secret.
|
||||||
|
- Validate with
|
||||||
|
`nix build .#nixosConfigurations.incineroar.config.system.build.toplevel --no-link`.
|
||||||
|
The user does the switch.
|
||||||
|
|
||||||
|
## Risks and fallbacks
|
||||||
|
|
||||||
|
- **The server doesn't build or run on Linux, or EF/SQLite can't translate
|
||||||
|
queries.** SQL Server is case-insensitive and SQLite isn't, and SQLite has no
|
||||||
|
`DateTimeOffset` ordering. Fix it endpoint by endpoint, but only for the
|
||||||
|
endpoints the profile, scores and leaderboard pages call (find them in the
|
||||||
|
network tab). Fallback: skip the fork and write a small read-only API in
|
||||||
|
this repo that serves the archive in the response shapes the website
|
||||||
|
expects. That would also be the first step toward the full replacement.
|
||||||
|
- **The OAuth refresh chain breaks** (more than 14 days without a run, a lost
|
||||||
|
write, or upstream auth changes). The archive keeps everything collected so
|
||||||
|
far. Sync stops, a failing unit makes that visible, and `auth login`
|
||||||
|
re-seeds the chain.
|
||||||
|
- **Maps disappear from BeatSaver.** That's why zips are archived as soon as a
|
||||||
|
hash first appears. Old versions that are already gone stay listed in
|
||||||
|
`missing.json`, which is acceptable (decided 2026-11-08). Cold storage might
|
||||||
|
hold some; check someday, without much hope.
|
||||||
|
- **Upstream API changes.** The archive stores raw responses, so a change
|
||||||
|
breaks the importer, not the data. Version the importer's mapping.
|
||||||
|
- **Upstream code moves fast** (the website is updated daily). Pin fork
|
||||||
|
revisions. Rebase only when there's a reason to.
|
||||||
|
- **Ranks and pp go stale locally.** That's by design: they're upstream's
|
||||||
|
values from the last sync. The UI could show "as of <sync time>".
|
||||||
|
|
||||||
|
## Open questions for the user
|
||||||
|
|
||||||
|
Settled on 2026-11-08: port 4442; access limited to incineroar and snorlax;
|
||||||
|
map data limited to played maps; OAuth for auth; sync daily at 17:00
|
||||||
|
America/Los_Angeles.
|
||||||
|
|
||||||
|
Also settled: skip attempts and their replays entirely. Where BL and SS scores
|
||||||
|
overlap, keep one per leaderboard by converting ScoreSaber scores to
|
||||||
|
BeatLeader scores (phase 5).
|
||||||
|
|
||||||
|
Nothing is open right now.
|
||||||
|
|
||||||
|
## Where the earlier work lives
|
||||||
|
|
||||||
|
- `~/ops/beatsaber/ss-scraper/`: ScoreSaber replay downloader. The legacy
|
||||||
|
account is done; it has no git commits yet.
|
||||||
|
- `~/ops/beatsaber/playlist-tool/src/clients/{beatleader,scoresaber,beatsaver}/`:
|
||||||
|
generated Python OpenAPI clients. Use them as a reference for the response
|
||||||
|
models.
|
||||||
|
- `~/ops/beatsaber/backups/`: UserData backups. They leave the replay
|
||||||
|
folders out.
|
||||||
|
- `~/ops/beatsaber/beatleader/beatleader-website.code-workspace`: points at
|
||||||
|
`~/src/beatleader/...` (lowercase). The checkouts are actually in
|
||||||
|
`~/src/BeatLeader/`, so fix that path.
|
||||||
|
- `~/src/BeatLeader/bl-dev-sandbox` is a BSOR v2 format experiment, not a dev
|
||||||
|
environment. Ignore it.
|
||||||
@@ -0,0 +1,192 @@
|
|||||||
|
// BeatLeader OAuth 2.0 token chain owned by the archive sync.
|
||||||
|
//
|
||||||
|
// This chain is separate from plebsaber.stream's own login: BeatLeader rotates
|
||||||
|
// the refresh token on every refresh and invalidates the old one, so two
|
||||||
|
// consumers sharing one chain would break each other.
|
||||||
|
import * as fs from 'node:fs';
|
||||||
|
import * as http from 'node:http';
|
||||||
|
import { randomUUID } from 'node:crypto';
|
||||||
|
import { BL_API, TOKENS_FILE, USER_AGENT, clientCredentials, writeFileAtomic } from './config.ts';
|
||||||
|
|
||||||
|
const AUTHORIZE_URL = `${BL_API}/oauth2/authorize`;
|
||||||
|
const TOKEN_URL = `${BL_API}/oauth2/token`;
|
||||||
|
const SCOPES = 'profile offline_access';
|
||||||
|
// Registered redirect URIs on the OAuth app (verified 2026-11-08):
|
||||||
|
// https://plebsaber.stream/auth/beatleader/callback
|
||||||
|
// http://localhost:5173/auth/beatleader/callback
|
||||||
|
const LOOPBACK_PORT = 5173;
|
||||||
|
const LOOPBACK_REDIRECT = `http://localhost:${LOOPBACK_PORT}/auth/beatleader/callback`;
|
||||||
|
const REFRESH_TOKEN_LIFETIME_MS = 14 * 24 * 3600 * 1000;
|
||||||
|
|
||||||
|
export type TokenStore = {
|
||||||
|
access_token: string;
|
||||||
|
refresh_token: string;
|
||||||
|
/** epoch ms */
|
||||||
|
access_expires_at: number;
|
||||||
|
/** epoch ms, estimated from BeatLeader's 14-day refresh token lifetime */
|
||||||
|
refresh_expires_at: number;
|
||||||
|
/** epoch ms of the last token grant/refresh */
|
||||||
|
updated_at: number;
|
||||||
|
player_id?: string;
|
||||||
|
player_name?: string;
|
||||||
|
};
|
||||||
|
|
||||||
|
type TokenResponse = { access_token: string; refresh_token?: string; expires_in?: number; token_type?: string };
|
||||||
|
|
||||||
|
export function readTokens(): TokenStore | null {
|
||||||
|
try {
|
||||||
|
return JSON.parse(fs.readFileSync(TOKENS_FILE, 'utf-8')) as TokenStore;
|
||||||
|
} catch {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function saveTokens(res: TokenResponse, previous: TokenStore | null, identity?: { id: string; name?: string }): TokenStore {
|
||||||
|
if (!res.refresh_token) {
|
||||||
|
throw new Error('Token response has no refresh_token; is offline_access granted?');
|
||||||
|
}
|
||||||
|
const now = Date.now();
|
||||||
|
const store: TokenStore = {
|
||||||
|
access_token: res.access_token,
|
||||||
|
refresh_token: res.refresh_token,
|
||||||
|
access_expires_at: now + (res.expires_in ?? 3600) * 1000,
|
||||||
|
refresh_expires_at: now + REFRESH_TOKEN_LIFETIME_MS,
|
||||||
|
updated_at: now,
|
||||||
|
player_id: identity?.id ?? previous?.player_id,
|
||||||
|
player_name: identity?.name ?? previous?.player_name
|
||||||
|
};
|
||||||
|
// Persist the rotated refresh token before anything uses the new access token.
|
||||||
|
writeFileAtomic(TOKENS_FILE, JSON.stringify(store, null, 2) + '\n', 0o600);
|
||||||
|
return store;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function tokenRequest(params: Record<string, string>): Promise<TokenResponse> {
|
||||||
|
const { clientId, clientSecret } = clientCredentials();
|
||||||
|
const res = await fetch(TOKEN_URL, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: { 'Content-Type': 'application/x-www-form-urlencoded', 'User-Agent': USER_AGENT },
|
||||||
|
body: new URLSearchParams({ client_id: clientId, client_secret: clientSecret, ...params })
|
||||||
|
});
|
||||||
|
const text = await res.text();
|
||||||
|
if (!res.ok) throw new Error(`Token request failed: HTTP ${res.status} ${text.slice(0, 300)}`);
|
||||||
|
return JSON.parse(text) as TokenResponse;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function fetchIdentity(accessToken: string): Promise<{ id: string; name?: string }> {
|
||||||
|
const res = await fetch(`${BL_API}/oauth2/identity`, {
|
||||||
|
headers: { Authorization: `Bearer ${accessToken}`, 'User-Agent': USER_AGENT }
|
||||||
|
});
|
||||||
|
if (!res.ok) throw new Error(`Identity request failed: HTTP ${res.status}`);
|
||||||
|
const json = (await res.json()) as { id?: string; name?: string };
|
||||||
|
if (!json.id) throw new Error('Identity response has no id');
|
||||||
|
return { id: json.id, name: json.name };
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Wait for the OAuth redirect on the loopback port, or for a pasted redirect URL on stdin. */
|
||||||
|
function waitForCode(state: string): Promise<string> {
|
||||||
|
return new Promise((resolve, reject) => {
|
||||||
|
let done = false;
|
||||||
|
const finish = (err: Error | null, code?: string) => {
|
||||||
|
if (done) return;
|
||||||
|
done = true;
|
||||||
|
server.close();
|
||||||
|
process.stdin.pause();
|
||||||
|
if (err) reject(err);
|
||||||
|
else resolve(code!);
|
||||||
|
};
|
||||||
|
|
||||||
|
const fromUrl = (raw: string): string => {
|
||||||
|
const url = new URL(raw, LOOPBACK_REDIRECT);
|
||||||
|
const error = url.searchParams.get('error');
|
||||||
|
if (error) throw new Error(`Authorization denied: ${error} ${url.searchParams.get('error_description') ?? ''}`);
|
||||||
|
const code = url.searchParams.get('code');
|
||||||
|
if (!code) throw new Error('Redirect has no code parameter');
|
||||||
|
if (url.searchParams.get('state') !== state) throw new Error('State mismatch; start the login again');
|
||||||
|
return code;
|
||||||
|
};
|
||||||
|
|
||||||
|
const server = http.createServer((req, res) => {
|
||||||
|
if (!req.url?.startsWith('/auth/beatleader/callback')) {
|
||||||
|
res.writeHead(404).end();
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
const code = fromUrl(req.url);
|
||||||
|
res.writeHead(200, { 'Content-Type': 'text/plain; charset=utf-8' });
|
||||||
|
res.end('plebleader archive: BeatLeader authorization received. You can close this tab.\n');
|
||||||
|
finish(null, code);
|
||||||
|
} catch (err) {
|
||||||
|
res.writeHead(400, { 'Content-Type': 'text/plain; charset=utf-8' });
|
||||||
|
res.end(`plebleader archive: ${(err as Error).message}\n`);
|
||||||
|
finish(err as Error);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
server.on('error', (err: NodeJS.ErrnoException) => {
|
||||||
|
if (err.code === 'EADDRINUSE') {
|
||||||
|
console.error(`Port ${LOOPBACK_PORT} is busy (is \`vite dev\` running?). Paste the redirected URL below instead.`);
|
||||||
|
} else {
|
||||||
|
finish(err);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
server.listen(LOOPBACK_PORT, 'localhost');
|
||||||
|
|
||||||
|
// Fallback: paste the full redirected URL (e.g. when the browser is on another machine).
|
||||||
|
process.stdin.setEncoding('utf-8');
|
||||||
|
process.stdin.on('data', (chunk: string) => {
|
||||||
|
const line = chunk.trim();
|
||||||
|
if (!line) return;
|
||||||
|
try {
|
||||||
|
finish(null, fromUrl(line));
|
||||||
|
} catch (err) {
|
||||||
|
console.error((err as Error).message);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
process.stdin.resume();
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function login(): Promise<TokenStore> {
|
||||||
|
const { clientId } = clientCredentials();
|
||||||
|
const state = randomUUID();
|
||||||
|
const url = new URL(AUTHORIZE_URL);
|
||||||
|
url.searchParams.set('client_id', clientId);
|
||||||
|
url.searchParams.set('scope', SCOPES);
|
||||||
|
url.searchParams.set('response_type', 'code');
|
||||||
|
url.searchParams.set('redirect_uri', LOOPBACK_REDIRECT);
|
||||||
|
url.searchParams.set('prompt', 'consent');
|
||||||
|
url.searchParams.set('state', state);
|
||||||
|
|
||||||
|
console.log('Open this URL in a browser on this machine and approve access:\n');
|
||||||
|
console.log(` ${url.toString()}\n`);
|
||||||
|
console.log(`Listening on ${LOOPBACK_REDIRECT}`);
|
||||||
|
console.log('If the browser is on another machine, paste the URL it was redirected to (the page will fail to load) and press Enter.\n');
|
||||||
|
|
||||||
|
const code = await waitForCode(state);
|
||||||
|
const res = await tokenRequest({ grant_type: 'authorization_code', code, redirect_uri: LOOPBACK_REDIRECT });
|
||||||
|
// Save immediately so the refresh token survives even if the identity lookup fails.
|
||||||
|
let store = saveTokens(res, readTokens());
|
||||||
|
const identity = await fetchIdentity(store.access_token);
|
||||||
|
store = { ...store, player_id: identity.id, player_name: identity.name };
|
||||||
|
writeFileAtomic(TOKENS_FILE, JSON.stringify(store, null, 2) + '\n', 0o600);
|
||||||
|
return store;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function refresh(): Promise<TokenStore> {
|
||||||
|
const current = readTokens();
|
||||||
|
if (!current?.refresh_token) {
|
||||||
|
throw new Error(`No refresh token in ${TOKENS_FILE}. Run \`auth login\` first.`);
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
const res = await tokenRequest({ grant_type: 'refresh_token', refresh_token: current.refresh_token });
|
||||||
|
return saveTokens(res, current);
|
||||||
|
} catch (err) {
|
||||||
|
throw new Error(`${(err as Error).message}\nThe refresh chain is broken. Run \`node scripts/archive/cli.ts auth login\` to seed it again.`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** A valid access token, refreshing (and rotating the stored refresh token) when needed. */
|
||||||
|
export async function getAccessToken(minValidityMs = 5 * 60 * 1000): Promise<string> {
|
||||||
|
const current = readTokens();
|
||||||
|
if (current && current.access_expires_at - Date.now() > minValidityMs) return current.access_token;
|
||||||
|
return (await refresh()).access_token;
|
||||||
|
}
|
||||||
@@ -0,0 +1,116 @@
|
|||||||
|
// BeatLeader exporter: scores, replays, score stats, leaderboards per played hash, profile.
|
||||||
|
// Writes only to archive/beatleader/. Everything is resumable: existing files are skipped.
|
||||||
|
import * as path from 'node:path';
|
||||||
|
import { BL_API } from './config.ts';
|
||||||
|
import { get, limiters } from './http.ts';
|
||||||
|
import { archivePath, eachWithProgress, exists, listFiles, readJson, save, saveSnapshotIfChanged, today, updateState } from './store.ts';
|
||||||
|
|
||||||
|
const bl = (...p: string[]) => archivePath('beatleader', ...p);
|
||||||
|
|
||||||
|
type BLScore = { id: number; timepost: number; replay?: string | null; leaderboard: { song: { hash: string } } };
|
||||||
|
|
||||||
|
async function blGet(pathAndQuery: string, accept?: number[]) {
|
||||||
|
return get(`${BL_API}${pathAndQuery}`, { limiter: limiters.beatleader, accept });
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Archived BL scores (parsed), optionally only those at or after `sinceUnix`. */
|
||||||
|
export function archivedScores(sinceUnix = 0): BLScore[] {
|
||||||
|
const dir = bl('scores');
|
||||||
|
return listFiles(dir, '.json')
|
||||||
|
.map((f) => readJson<BLScore>(path.join(dir, f)))
|
||||||
|
.filter((s) => s.timepost >= sinceUnix);
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function exportProfile(playerId: string): Promise<void> {
|
||||||
|
console.log(`BeatLeader profile ${playerId}`);
|
||||||
|
const dir = (kind: string) => bl('players', playerId, kind);
|
||||||
|
const endpoints: [string, string][] = [
|
||||||
|
['profile', `/player/${playerId}?stats=true`],
|
||||||
|
['history', `/player/${playerId}/history`],
|
||||||
|
['triangle', `/player/${playerId}/history/triangle`],
|
||||||
|
['accgraph', `/player/${playerId}/accgraph`]
|
||||||
|
];
|
||||||
|
for (const [kind, url] of endpoints) {
|
||||||
|
const res = await blGet(url);
|
||||||
|
const wrote = saveSnapshotIfChanged(dir(kind), res.body);
|
||||||
|
console.log(` ${kind}: ${wrote ? `saved ${today()}` : 'unchanged'}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Page through best scores newest-first from `sinceUnix`, saving each score object. */
|
||||||
|
export async function exportScores(playerId: string, sinceUnix: number): Promise<void> {
|
||||||
|
console.log(`BeatLeader scores for ${playerId} since ${new Date(sinceUnix * 1000).toISOString()}`);
|
||||||
|
let added = 0;
|
||||||
|
let seen = 0;
|
||||||
|
let newest = 0;
|
||||||
|
for (let page = 1; ; page++) {
|
||||||
|
const qs = `count=100&page=${page}&sortBy=date&order=desc&includeIO=true&time_from=${sinceUnix}`;
|
||||||
|
const res = await blGet(`/player/${playerId}/scores?${qs}`);
|
||||||
|
const json = JSON.parse(res.body.toString('utf-8')) as { metadata: { total: number }; data: BLScore[] };
|
||||||
|
for (const score of json.data) {
|
||||||
|
seen++;
|
||||||
|
newest = Math.max(newest, score.timepost);
|
||||||
|
const file = bl('scores', `${score.id}.json`);
|
||||||
|
if (exists(file)) continue;
|
||||||
|
// The page is parsed and each element re-serialized: same content, no reshaping.
|
||||||
|
save(file, JSON.stringify(score));
|
||||||
|
added++;
|
||||||
|
}
|
||||||
|
console.log(` page ${page}: ${seen}/${json.metadata.total} seen, ${added} new`);
|
||||||
|
if (json.data.length < 100 || seen >= json.metadata.total) break;
|
||||||
|
}
|
||||||
|
updateState(`beatleader:${playerId}`, { scoresSince: sinceUnix, newestTimepost: newest, lastScoresRun: new Date().toISOString() });
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function exportReplays(scores: BLScore[]): Promise<void> {
|
||||||
|
const withReplay = scores.filter((s) => s.replay);
|
||||||
|
console.log(`BeatLeader replays: ${withReplay.length} scores with a replay URL`);
|
||||||
|
const res = await eachWithProgress('replays', withReplay, async (s) => {
|
||||||
|
const name = decodeURIComponent(new URL(s.replay!).pathname.split('/').pop()!);
|
||||||
|
const file = bl('replays', name);
|
||||||
|
if (exists(file)) return 'skipped';
|
||||||
|
const r = await get(s.replay!, { limiter: limiters.blCdn, accept: [404] });
|
||||||
|
if (r.status === 404) return 'missing';
|
||||||
|
save(file, r.body);
|
||||||
|
return 'done';
|
||||||
|
});
|
||||||
|
recordFailures('replays', res.failed.map((f) => f.item.id));
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function exportScoreStats(scores: BLScore[]): Promise<void> {
|
||||||
|
console.log(`BeatLeader score stats: ${scores.length} scores`);
|
||||||
|
const res = await eachWithProgress('scorestats', scores, async (s) => {
|
||||||
|
const file = bl('scorestats', `${s.id}.json`);
|
||||||
|
if (exists(file)) return 'skipped';
|
||||||
|
// 302 to a presigned R2 URL; fetch follows it.
|
||||||
|
const r = await blGet(`/score/statistic/${s.id}`, [404]);
|
||||||
|
if (r.status === 404) return 'missing';
|
||||||
|
save(file, r.body);
|
||||||
|
return 'done';
|
||||||
|
});
|
||||||
|
recordFailures('scorestats', res.failed.map((f) => f.item.id));
|
||||||
|
}
|
||||||
|
|
||||||
|
/** /leaderboards/hash/{hash}: all difficulties with full ratings (also unranked, no auth needed). */
|
||||||
|
export async function exportLeaderboards(hashes: string[], refresh = false): Promise<void> {
|
||||||
|
console.log(`BeatLeader leaderboards: ${hashes.length} hashes${refresh ? ' (refresh)' : ''}`);
|
||||||
|
const res = await eachWithProgress('leaderboards', hashes, async (hash) => {
|
||||||
|
const dir = bl('leaderboards', hash);
|
||||||
|
if (!refresh && listFiles(dir, '.json').length > 0) return 'skipped';
|
||||||
|
const r = await blGet(`/leaderboards/hash/${hash}`, [404]);
|
||||||
|
if (r.status === 404) return 'missing';
|
||||||
|
return saveSnapshotIfChanged(dir, r.body) ? 'done' : 'skipped';
|
||||||
|
});
|
||||||
|
recordFailures('leaderboards', res.failed.map((f) => f.item));
|
||||||
|
}
|
||||||
|
|
||||||
|
function recordFailures(kind: string, ids: (string | number)[]): void {
|
||||||
|
if (ids.length) {
|
||||||
|
updateState('failures', { [`beatleader:${kind}`]: ids });
|
||||||
|
console.warn(` ${ids.length} ${kind} failed; rerun to retry`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export function blHashes(sinceUnix = 0): string[] {
|
||||||
|
return [...new Set(archivedScores(sinceUnix).map((s) => s.leaderboard.song.hash.toLowerCase()))].sort();
|
||||||
|
}
|
||||||
@@ -0,0 +1,89 @@
|
|||||||
|
// BeatSaver exporter: map JSON and the zip of the exact version hash that was played.
|
||||||
|
import * as fs from 'node:fs';
|
||||||
|
import { get, limiters } from './http.ts';
|
||||||
|
import { archivePath, eachWithProgress, exists, readJson, save } from './store.ts';
|
||||||
|
|
||||||
|
const bs = (...p: string[]) => archivePath('beatsaver', ...p);
|
||||||
|
const MISSING_FILE = () => bs('missing.json');
|
||||||
|
|
||||||
|
type BSMap = { id: string; versions: { hash: string; downloadURL: string }[] };
|
||||||
|
type Missing = Record<string, { reason: string; checkedAt: string; recoveredFrom?: string }>;
|
||||||
|
|
||||||
|
function readMissing(): Missing {
|
||||||
|
try {
|
||||||
|
return readJson<Missing>(MISSING_FILE());
|
||||||
|
} catch {
|
||||||
|
return {};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Entry names from a zip's central directory, or null if the zip is truncated or malformed. */
|
||||||
|
export function zipEntryNames(buf: Buffer): string[] | null {
|
||||||
|
if (buf.length < 22 || buf.readUInt32LE(0) !== 0x04034b50) return null;
|
||||||
|
const tailStart = Math.max(0, buf.length - 65_557);
|
||||||
|
const eocd = buf.subarray(tailStart).lastIndexOf(Buffer.from([0x50, 0x4b, 0x05, 0x06]));
|
||||||
|
if (eocd < 0) return null;
|
||||||
|
const e = tailStart + eocd;
|
||||||
|
const count = buf.readUInt16LE(e + 10);
|
||||||
|
let p = buf.readUInt32LE(e + 16);
|
||||||
|
const names: string[] = [];
|
||||||
|
for (let i = 0; i < count; i++) {
|
||||||
|
if (p + 46 > buf.length || buf.readUInt32LE(p) !== 0x02014b50) return null;
|
||||||
|
const nameLen = buf.readUInt16LE(p + 28);
|
||||||
|
const extraLen = buf.readUInt16LE(p + 30);
|
||||||
|
const commentLen = buf.readUInt16LE(p + 32);
|
||||||
|
names.push(buf.toString('utf-8', p + 46, p + 46 + nameLen));
|
||||||
|
p += 46 + nameLen + extraLen + commentLen;
|
||||||
|
}
|
||||||
|
return names;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** A zip is usable if its central directory is intact and lists an Info.dat (any case, any folder). */
|
||||||
|
export function looksLikeMapZip(buf: Buffer): boolean {
|
||||||
|
return zipEntryNames(buf)?.some((n) => /(^|\/)info\.dat$/i.test(n)) ?? false;
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function exportMaps(hashes: string[]): Promise<void> {
|
||||||
|
const missing = readMissing();
|
||||||
|
console.log(`BeatSaver maps: ${hashes.length} hashes`);
|
||||||
|
const res = await eachWithProgress('maps', hashes, async (hash) => {
|
||||||
|
const jsonFile = bs('maps', `${hash}.json`);
|
||||||
|
const zipFile = bs('zips', `${hash}.zip`);
|
||||||
|
if (exists(zipFile) && exists(jsonFile)) return 'skipped';
|
||||||
|
|
||||||
|
let map: BSMap | null = null;
|
||||||
|
if (exists(jsonFile)) {
|
||||||
|
map = readJson<BSMap>(jsonFile);
|
||||||
|
} else {
|
||||||
|
const r = await get(`https://api.beatsaver.com/maps/hash/${hash}`, { limiter: limiters.beatsaver, accept: [404] });
|
||||||
|
if (r.status === 200) {
|
||||||
|
save(jsonFile, r.body);
|
||||||
|
map = JSON.parse(r.body.toString('utf-8')) as BSMap;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (exists(zipFile)) return 'done';
|
||||||
|
|
||||||
|
// Prefer the version entry for this exact hash; old versions often remain on the CDN.
|
||||||
|
const version = map?.versions.find((v) => v.hash.toLowerCase() === hash);
|
||||||
|
const candidates = [version?.downloadURL, `https://r2cdn.beatsaver.com/${hash}.zip`, `https://cdn.beatsaver.com/${hash}.zip`].filter(
|
||||||
|
(u): u is string => Boolean(u)
|
||||||
|
);
|
||||||
|
for (const url of [...new Set(candidates)]) {
|
||||||
|
const r = await get(url, { limiter: limiters.beatsaverCdn, accept: [403, 404] });
|
||||||
|
if (r.status === 200 && looksLikeMapZip(r.body)) {
|
||||||
|
save(zipFile, r.body);
|
||||||
|
delete missing[hash];
|
||||||
|
return 'done';
|
||||||
|
}
|
||||||
|
}
|
||||||
|
missing[hash] = {
|
||||||
|
reason: map ? `map ${map.id} found but version zip unavailable` : 'hash unknown to BeatSaver',
|
||||||
|
checkedAt: new Date().toISOString()
|
||||||
|
};
|
||||||
|
return 'missing';
|
||||||
|
});
|
||||||
|
fs.mkdirSync(bs(), { recursive: true });
|
||||||
|
save(MISSING_FILE(), JSON.stringify(missing, null, 2) + '\n');
|
||||||
|
console.log(` missing.json: ${Object.keys(missing).length} hashes`);
|
||||||
|
if (res.failed.length) console.warn(` ${res.failed.length} failed; rerun to retry`);
|
||||||
|
}
|
||||||
@@ -0,0 +1,236 @@
|
|||||||
|
#!/usr/bin/env node
|
||||||
|
// plebleader archive CLI. Run from the repo root: node scripts/archive/cli.ts <command> [options]
|
||||||
|
//
|
||||||
|
// auth login | refresh | status the sync's own BeatLeader OAuth token chain
|
||||||
|
// export bl [--since D] BL profile, scores, replays, score stats, leaderboards
|
||||||
|
// export ss [--since D] ScoreSaber player snapshot and scores (fast)
|
||||||
|
// export ss-replays [--since D] ScoreSaber replays, ~10.5 s apart (run in background)
|
||||||
|
// export maps [--maps-since D] BeatSaver JSON + zip for played hashes (default: last 2 years,
|
||||||
|
// plus all legacy ScoreSaber maps)
|
||||||
|
// export all [--since D] bl, ss, maps (not ss-replays)
|
||||||
|
// report [--since D] counts and gaps in the archive
|
||||||
|
//
|
||||||
|
// --since: YYYY-MM-DD, unix seconds, or "all" (default: all). --maps-since defaults to the two-year
|
||||||
|
// window (DEFAULT_MAPS_SINCE). Every command skips files that already exist, so reruns only fill gaps.
|
||||||
|
import * as fs from 'node:fs';
|
||||||
|
import * as path from 'node:path';
|
||||||
|
import { ARCHIVE_DIR, BL_API, DEFAULT_MAPS_SINCE, LEGACY_PLAYER_ID, MAIN_PLAYER_ID, PLAYER_IDS, TOKENS_FILE, USER_AGENT, loadDotEnv, parseSince } from './config.ts';
|
||||||
|
import { getAccessToken, login, readTokens, refresh } from './auth.ts';
|
||||||
|
import { archivedScores, blHashes, exportLeaderboards, exportProfile, exportReplays, exportScoreStats, exportScores } from './beatleader.ts';
|
||||||
|
import { archivedSSScores, exportSSPlayer, exportSSReplays, exportSSScores, ssHashes } from './scoresaber.ts';
|
||||||
|
import { exportMaps } from './beatsaver.ts';
|
||||||
|
import { archivePath, exists, listFiles, readState } from './store.ts';
|
||||||
|
|
||||||
|
function fmt(ms: number): string {
|
||||||
|
const d = new Date(ms);
|
||||||
|
const left = ms - Date.now();
|
||||||
|
const hours = Math.round(left / 36e5);
|
||||||
|
return `${d.toISOString()} (${left < 0 ? 'expired' : hours >= 48 ? `${Math.round(hours / 24)} d left` : `${hours} h left`})`;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function blGet<T>(path: string, token: string): Promise<T> {
|
||||||
|
const res = await fetch(`${BL_API}${path}`, {
|
||||||
|
headers: { Authorization: `Bearer ${token}`, 'User-Agent': USER_AGENT }
|
||||||
|
});
|
||||||
|
if (!res.ok) throw new Error(`GET ${path}: HTTP ${res.status}`);
|
||||||
|
return (await res.json()) as T;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Phase 1 step 1. The bearer token must belong to pleb, and an unranked map's ratings must be
|
||||||
|
* readable through /leaderboards/hash/{hash}.
|
||||||
|
*
|
||||||
|
* Verified 2026-11-08: BeatLeader only honors bearer tokens on /oauth2/identity, clan management
|
||||||
|
* and ReeSabers presets. /user and the showAllRatings gate on score lists read the beatleader.com
|
||||||
|
* cookie session instead, so score lists always come back with unranked ratings nulled. The
|
||||||
|
* per-hash leaderboard lookup returns full ratings (incl. modifiersRating) even anonymously.
|
||||||
|
*/
|
||||||
|
async function verify(): Promise<boolean> {
|
||||||
|
const token = await getAccessToken();
|
||||||
|
const identity = await blGet<{ id?: string; name?: string }>('/oauth2/identity', token);
|
||||||
|
const idOk = identity.id === MAIN_PLAYER_ID;
|
||||||
|
console.log(`GET /oauth2/identity: ${identity.id} (${identity.name}) ${idOk ? 'OK' : `MISMATCH, expected ${MAIN_PLAYER_ID}`}`);
|
||||||
|
|
||||||
|
type ScorePage = { data: { leaderboard: { song: { name: string; hash: string }; difficulty: { difficultyName: string; status: number } } }[] };
|
||||||
|
// status 3 = ranked; anything else counts as unranked for this check.
|
||||||
|
const page = await blGet<ScorePage>(`/player/${MAIN_PLAYER_ID}/scores?count=50&page=1&sortBy=date&order=desc&type=unranked`, token);
|
||||||
|
const unranked = page.data.find((s) => s.leaderboard.difficulty.status !== 3);
|
||||||
|
if (!unranked) {
|
||||||
|
console.log('No unranked score found in the latest 50 to check ratings against.');
|
||||||
|
return idOk;
|
||||||
|
}
|
||||||
|
type HashLookup = { leaderboards: { difficulty: { difficultyName: string; stars: number | null; techRating: number | null } }[] };
|
||||||
|
const { song, difficulty } = unranked.leaderboard;
|
||||||
|
const lookup = await blGet<HashLookup>(`/leaderboards/hash/${song.hash}`, token);
|
||||||
|
const d = lookup.leaderboards.find((l) => l.difficulty.difficultyName === difficulty.difficultyName)?.difficulty;
|
||||||
|
const ratingsOk = d?.stars != null;
|
||||||
|
console.log(`Unranked "${song.name}" ${difficulty.difficultyName} via /leaderboards/hash: stars=${d?.stars} tech=${d?.techRating} ${ratingsOk ? 'OK' : 'MISSING'}`);
|
||||||
|
return idOk && ratingsOk;
|
||||||
|
}
|
||||||
|
|
||||||
|
function playedHashes(since: number): string[] {
|
||||||
|
return [...new Set([...blHashes(since), ...PLAYER_IDS.flatMap((id) => ssHashes(id, since))])].sort();
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Hashes to fetch BeatSaver map files for: played within the maps window, plus — regardless of
|
||||||
|
* age — the maps of legacy ScoreSaber scores that have an archived replay, so those replays render.
|
||||||
|
*/
|
||||||
|
function mapHashes(mapsSince: number): string[] {
|
||||||
|
const legacyWithReplay = archivedSSScores(LEGACY_PLAYER_ID, 0)
|
||||||
|
.filter((it) => exists(archivePath('scoresaber', LEGACY_PLAYER_ID, 'replays', `${it.score.id}.dat`)))
|
||||||
|
.map((it) => it.leaderboard.songHash.toLowerCase());
|
||||||
|
return [...new Set([...playedHashes(mapsSince), ...legacyWithReplay])].sort();
|
||||||
|
}
|
||||||
|
|
||||||
|
async function exportBL(since: number): Promise<void> {
|
||||||
|
for (const id of PLAYER_IDS) {
|
||||||
|
await exportProfile(id);
|
||||||
|
await exportScores(id, since);
|
||||||
|
}
|
||||||
|
// The scores dir holds both accounts' scores (keyed by score id).
|
||||||
|
const scores = archivedScores(since);
|
||||||
|
await exportReplays(scores);
|
||||||
|
await exportScoreStats(scores);
|
||||||
|
// BL leaderboard JSON (ratings) for every played hash, not just the maps window: it's small
|
||||||
|
// and the importer needs it for star ratings on old scores.
|
||||||
|
await exportLeaderboards(blHashes(since));
|
||||||
|
}
|
||||||
|
|
||||||
|
async function exportSS(since: number): Promise<void> {
|
||||||
|
for (const id of PLAYER_IDS) {
|
||||||
|
await exportSSPlayer(id);
|
||||||
|
await exportSSScores(id, since);
|
||||||
|
}
|
||||||
|
// Hashes only ScoreSaber knows about still need BL leaderboard data (phase 5 merge).
|
||||||
|
await exportLeaderboards([...new Set(PLAYER_IDS.flatMap((id) => ssHashes(id, since)))].sort());
|
||||||
|
}
|
||||||
|
|
||||||
|
function report(since: number, mapsSince: number): void {
|
||||||
|
const count = (...p: string[]) => listFiles(archivePath(...p)).length;
|
||||||
|
const day = (t: number) => (t ? new Date(t * 1000).toISOString().slice(0, 10) : 'all time');
|
||||||
|
const bl = archivedScores(since);
|
||||||
|
const blWithReplay = bl.filter((s) => s.replay);
|
||||||
|
const blReplayMissing = blWithReplay.filter((s) => !exists(archivePath('beatleader', 'replays', decodeURIComponent(new URL(s.replay!).pathname.split('/').pop()!))));
|
||||||
|
const statsMissing = bl.filter((s) => !exists(archivePath('beatleader', 'scorestats', `${s.id}.json`)));
|
||||||
|
const ss = PLAYER_IDS.map((id) => {
|
||||||
|
const items = archivedSSScores(id, since);
|
||||||
|
const withReplay = items.filter((it) => it.score.hasReplay);
|
||||||
|
const missing = withReplay.filter((it) => !exists(archivePath('scoresaber', id, 'replays', `${it.score.id}.dat`)));
|
||||||
|
return { id, items, withReplay, missing };
|
||||||
|
});
|
||||||
|
const allHashes = playedHashes(since);
|
||||||
|
const noLeaderboard = allHashes.filter((h) => listFiles(archivePath('beatleader', 'leaderboards', h)).length === 0);
|
||||||
|
const hashes = mapHashes(Math.max(since, mapsSince));
|
||||||
|
const noZip = hashes.filter((h) => !exists(archivePath('beatsaver', 'zips', `${h}.zip`)));
|
||||||
|
let missingJson: Record<string, unknown> = {};
|
||||||
|
try {
|
||||||
|
missingJson = JSON.parse(fs.readFileSync(archivePath('beatsaver', 'missing.json'), 'utf-8'));
|
||||||
|
} catch {}
|
||||||
|
const du = (dir: string): number => {
|
||||||
|
let total = 0;
|
||||||
|
const walk = (d: string) => {
|
||||||
|
for (const e of fs.readdirSync(d, { withFileTypes: true })) {
|
||||||
|
const p = path.join(d, e.name);
|
||||||
|
if (e.isDirectory()) walk(p);
|
||||||
|
else total += fs.statSync(p).size;
|
||||||
|
}
|
||||||
|
};
|
||||||
|
if (fs.existsSync(dir)) walk(dir);
|
||||||
|
return total;
|
||||||
|
};
|
||||||
|
|
||||||
|
console.log(`Archive: ${ARCHIVE_DIR}`);
|
||||||
|
console.log(`Window: scores since ${day(since)}, BeatSaver maps since ${day(mapsSince)}`);
|
||||||
|
console.log('');
|
||||||
|
console.log(`BeatLeader scores ${bl.length} in window (${count('beatleader', 'scores')} archived total, both accounts)`);
|
||||||
|
console.log(`BeatLeader replays ${blWithReplay.length - blReplayMissing.length}/${blWithReplay.length}`);
|
||||||
|
console.log(`BeatLeader score stats ${bl.length - statsMissing.length}/${bl.length}`);
|
||||||
|
for (const s of ss) {
|
||||||
|
console.log(`ScoreSaber ${s.id}: ${s.items.length} scores, replays ${s.withReplay.length - s.missing.length}/${s.withReplay.length}`);
|
||||||
|
}
|
||||||
|
console.log(`Played hashes ${allHashes.length}`);
|
||||||
|
console.log(` BL leaderboard snapshot ${allHashes.length - noLeaderboard.length}/${allHashes.length}`);
|
||||||
|
console.log(` BeatSaver zip ${hashes.length - noZip.length}/${hashes.length} in maps window (missing.json lists ${Object.keys(missingJson).length})`);
|
||||||
|
console.log(`Archive size ${(du(ARCHIVE_DIR) / 1e9).toFixed(2)} GB`);
|
||||||
|
const failures = readState().failures;
|
||||||
|
if (failures && Object.keys(failures).length) console.log(`Failures from last runs: ${JSON.stringify(failures)}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
function option(args: string[], name: string): string | undefined {
|
||||||
|
const i = args.indexOf(name);
|
||||||
|
return i >= 0 ? args[i + 1] : args.find((a) => a.startsWith(`${name}=`))?.slice(name.length + 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function main(): Promise<number> {
|
||||||
|
loadDotEnv();
|
||||||
|
const args = process.argv.slice(2);
|
||||||
|
const [cmd, sub] = args;
|
||||||
|
|
||||||
|
if (cmd === 'auth' && sub === 'login') {
|
||||||
|
const store = await login();
|
||||||
|
console.log(`\nLogged in as ${store.player_name} (${store.player_id}). Tokens saved to ${TOKENS_FILE}`);
|
||||||
|
console.log(`Refresh token valid until about ${fmt(store.refresh_expires_at)}`);
|
||||||
|
return (await verify()) ? 0 : 1;
|
||||||
|
}
|
||||||
|
if (cmd === 'auth' && sub === 'refresh') {
|
||||||
|
const store = await refresh();
|
||||||
|
console.log(`Refreshed. Access token until ${fmt(store.access_expires_at)}; refresh until about ${fmt(store.refresh_expires_at)}`);
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
if (cmd === 'auth' && sub === 'status') {
|
||||||
|
const store = readTokens();
|
||||||
|
if (!store) {
|
||||||
|
console.log(`No tokens at ${TOKENS_FILE}. Run \`auth login\`.`);
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
console.log(`Tokens: ${TOKENS_FILE}`);
|
||||||
|
console.log(`Player: ${store.player_name} (${store.player_id})`);
|
||||||
|
console.log(`Access: ${fmt(store.access_expires_at)}`);
|
||||||
|
console.log(`Refresh: ${fmt(store.refresh_expires_at)} (estimated)`);
|
||||||
|
return (await verify()) ? 0 : 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
const since = parseSince(option(args, '--since'));
|
||||||
|
const mapsSince = Math.max(since, parseSince(option(args, '--maps-since'), DEFAULT_MAPS_SINCE));
|
||||||
|
if (cmd === 'export') {
|
||||||
|
console.log(`Archive ${ARCHIVE_DIR}, since ${since ? new Date(since * 1000).toISOString() : 'all time'}\n`);
|
||||||
|
switch (sub) {
|
||||||
|
case 'bl':
|
||||||
|
await exportBL(since);
|
||||||
|
return 0;
|
||||||
|
case 'ss':
|
||||||
|
await exportSS(since);
|
||||||
|
return 0;
|
||||||
|
case 'ss-replays':
|
||||||
|
// Legacy first: its replays are mostly already on disk from ss-scraper.
|
||||||
|
for (const id of [...PLAYER_IDS].reverse()) await exportSSReplays(id, since);
|
||||||
|
return 0;
|
||||||
|
case 'maps':
|
||||||
|
await exportMaps(mapHashes(mapsSince));
|
||||||
|
return 0;
|
||||||
|
case 'all':
|
||||||
|
await exportBL(since);
|
||||||
|
await exportSS(since);
|
||||||
|
await exportMaps(mapHashes(mapsSince));
|
||||||
|
console.log('');
|
||||||
|
report(since, mapsSince);
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (cmd === 'report') {
|
||||||
|
report(since, mapsSince);
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
console.error('Usage: node scripts/archive/cli.ts <auth login|refresh|status | export bl|ss|ss-replays|maps|all | report> [--since YYYY-MM-DD|all] [--maps-since YYYY-MM-DD|all]');
|
||||||
|
return 2;
|
||||||
|
}
|
||||||
|
|
||||||
|
main().then(
|
||||||
|
(code) => process.exit(code),
|
||||||
|
(err) => {
|
||||||
|
console.error(err instanceof Error ? err.message : err);
|
||||||
|
process.exit(1);
|
||||||
|
}
|
||||||
|
);
|
||||||
@@ -0,0 +1,76 @@
|
|||||||
|
// Shared configuration for the plebleader archive CLI.
|
||||||
|
// Runs directly under Node >= 22 (type stripping), no build step.
|
||||||
|
import * as fs from 'node:fs';
|
||||||
|
import * as path from 'node:path';
|
||||||
|
|
||||||
|
export const USER_AGENT = 'plebsaber-archive/0.1';
|
||||||
|
export const BL_API = 'https://api.beatleader.com';
|
||||||
|
export const MAIN_PLAYER_ID = '76561199407393962';
|
||||||
|
export const LEGACY_PLAYER_ID = '76561197962107242';
|
||||||
|
|
||||||
|
export const PLAYER_IDS = [MAIN_PLAYER_ID, LEGACY_PLAYER_ID];
|
||||||
|
|
||||||
|
/** Scores, replays, score stats and profile data: all history by default. */
|
||||||
|
export const DEFAULT_SINCE = 'all';
|
||||||
|
/**
|
||||||
|
* BeatSaver map files (JSON + zip) only for maps played in the last two years. Older maps can
|
||||||
|
* be fetched later with --maps-since all (existing files are skipped).
|
||||||
|
*/
|
||||||
|
export const DEFAULT_MAPS_SINCE = '2024-11-08';
|
||||||
|
|
||||||
|
/** Where ss-scraper already downloaded ScoreSaber replays (<playerId>/<scoreId>_<name>.dat). */
|
||||||
|
export const SS_SCRAPER_REPLAYS = process.env.SS_SCRAPER_REPLAYS ?? path.join(process.env.HOME ?? '', 'ops/beatsaber/ss-scraper/replays');
|
||||||
|
|
||||||
|
/** Parse --since: YYYY-MM-DD (UTC), unix seconds, or "all". */
|
||||||
|
export function parseSince(value: string | undefined, fallback = DEFAULT_SINCE): number {
|
||||||
|
const v = value ?? fallback;
|
||||||
|
if (v === 'all') return 0;
|
||||||
|
if (/^\d+$/.test(v)) return Number(v);
|
||||||
|
const ms = Date.parse(/^\d{4}-\d{2}-\d{2}$/.test(v) ? `${v}T00:00:00Z` : v);
|
||||||
|
if (Number.isNaN(ms)) throw new Error(`Invalid --since: ${v}`);
|
||||||
|
return Math.floor(ms / 1000);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Root of the plebleader state (archive/, auth/, db/, ...).
|
||||||
|
* Production: /var/lib/plebleader. Local default: ./.data/plebleader (gitignored).
|
||||||
|
*/
|
||||||
|
export const ROOT = path.resolve(process.env.PLEBLEADER_DIR ?? path.join(process.cwd(), '.data', 'plebleader'));
|
||||||
|
export const ARCHIVE_DIR = path.join(ROOT, 'archive');
|
||||||
|
export const AUTH_DIR = path.join(ROOT, 'auth');
|
||||||
|
export const TOKENS_FILE = path.join(AUTH_DIR, 'beatleader_tokens.json');
|
||||||
|
|
||||||
|
/** Load KEY=VALUE (optionally `export KEY=VALUE`) lines from .env without overriding the real environment. */
|
||||||
|
export function loadDotEnv(file = path.join(process.cwd(), '.env')): void {
|
||||||
|
if (!fs.existsSync(file)) return;
|
||||||
|
for (const line of fs.readFileSync(file, 'utf-8').split('\n')) {
|
||||||
|
const m = line.match(/^\s*(?:export\s+)?([A-Za-z_][A-Za-z0-9_]*)\s*=\s*(.*?)\s*$/);
|
||||||
|
if (!m) continue;
|
||||||
|
const [, key, raw] = m;
|
||||||
|
const value = raw.replace(/^(['"])(.*)\1$/, '$2');
|
||||||
|
if (process.env[key] === undefined) process.env[key] = value;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export function clientCredentials(): { clientId: string; clientSecret: string } {
|
||||||
|
const clientId = process.env.BL_CLIENT_ID;
|
||||||
|
const clientSecret = process.env.BL_CLIENT_SECRET;
|
||||||
|
if (!clientId || !clientSecret) {
|
||||||
|
throw new Error('BL_CLIENT_ID and BL_CLIENT_SECRET must be set (environment or .env).');
|
||||||
|
}
|
||||||
|
return { clientId, clientSecret };
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Write a file atomically: temp file in the same directory, fsync, rename. */
|
||||||
|
export function writeFileAtomic(file: string, data: string | Uint8Array, mode = 0o644): void {
|
||||||
|
fs.mkdirSync(path.dirname(file), { recursive: true });
|
||||||
|
const tmp = `${file}.tmp-${process.pid}-${Date.now()}`;
|
||||||
|
const fd = fs.openSync(tmp, 'w', mode);
|
||||||
|
try {
|
||||||
|
fs.writeSync(fd, typeof data === 'string' ? Buffer.from(data, 'utf-8') : data);
|
||||||
|
fs.fsyncSync(fd);
|
||||||
|
} finally {
|
||||||
|
fs.closeSync(fd);
|
||||||
|
}
|
||||||
|
fs.renameSync(tmp, file);
|
||||||
|
}
|
||||||
@@ -0,0 +1,107 @@
|
|||||||
|
// Rate-limited GET with 429 / 5xx / network retries. Returns raw bytes so callers can
|
||||||
|
// store responses byte-for-byte.
|
||||||
|
import { USER_AGENT } from './config.ts';
|
||||||
|
|
||||||
|
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
|
||||||
|
|
||||||
|
/** Enforces a minimum interval between request starts. */
|
||||||
|
export class Limiter {
|
||||||
|
private next = 0;
|
||||||
|
private readonly minIntervalMs: number;
|
||||||
|
constructor(minIntervalMs: number) {
|
||||||
|
this.minIntervalMs = minIntervalMs;
|
||||||
|
}
|
||||||
|
async wait(): Promise<void> {
|
||||||
|
const now = Date.now();
|
||||||
|
const at = Math.max(now, this.next);
|
||||||
|
this.next = at + this.minIntervalMs;
|
||||||
|
if (at > now) await sleep(at - now);
|
||||||
|
}
|
||||||
|
/** Push the next slot out, e.g. after a 429. */
|
||||||
|
backoff(ms: number): void {
|
||||||
|
this.next = Math.max(this.next, Date.now() + ms);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export const limiters = {
|
||||||
|
beatleader: new Limiter(250), // ~4 req/s, well under 50 per 10 s
|
||||||
|
blCdn: new Limiter(250),
|
||||||
|
scoresaber: new Limiter(1000),
|
||||||
|
// /api/v2/scores/{id}/replay sends x-ratelimit-limit-long: 360 per hour (checked 2026-11-08;
|
||||||
|
// short 20 and medium 60/min don't bind). 10.5 s ≈ 343/h. get() also pauses at remaining 0.
|
||||||
|
scoresaberReplay: new Limiter(10_500),
|
||||||
|
beatsaver: new Limiter(500), // ~2 req/s
|
||||||
|
beatsaverCdn: new Limiter(250)
|
||||||
|
};
|
||||||
|
|
||||||
|
export type GetResult = { status: number; body: Buffer; headers: Headers; url: string };
|
||||||
|
|
||||||
|
export class HttpError extends Error {
|
||||||
|
readonly status: number;
|
||||||
|
constructor(status: number, message: string) {
|
||||||
|
super(message);
|
||||||
|
this.status = status;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* GET with retries. Resolves with the final response for 2xx and for statuses listed in
|
||||||
|
* `accept` (e.g. 404); throws HttpError for anything else once retries are exhausted.
|
||||||
|
*/
|
||||||
|
export async function get(
|
||||||
|
url: string,
|
||||||
|
opts: { limiter: Limiter; headers?: Record<string, string>; accept?: number[]; retries?: number; timeoutMs?: number }
|
||||||
|
): Promise<GetResult> {
|
||||||
|
const retries = opts.retries ?? 6;
|
||||||
|
let backoffMs = 2000;
|
||||||
|
for (let attempt = 0; ; attempt++) {
|
||||||
|
await opts.limiter.wait();
|
||||||
|
let res: Response;
|
||||||
|
try {
|
||||||
|
res = await fetch(url, {
|
||||||
|
headers: { 'User-Agent': USER_AGENT, ...opts.headers },
|
||||||
|
signal: AbortSignal.timeout(opts.timeoutMs ?? 120_000)
|
||||||
|
});
|
||||||
|
} catch (err) {
|
||||||
|
if (attempt >= retries) throw err;
|
||||||
|
console.warn(` network error on ${shortUrl(url)}: ${(err as Error).message}; retrying in ${backoffMs / 1000}s`);
|
||||||
|
await sleep(backoffMs);
|
||||||
|
backoffMs = Math.min(backoffMs * 2, 300_000);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
honorRateLimitHeaders(res.headers, opts.limiter, url);
|
||||||
|
if (res.ok || opts.accept?.includes(res.status)) {
|
||||||
|
return { status: res.status, body: Buffer.from(await res.arrayBuffer()), headers: res.headers, url: res.url };
|
||||||
|
}
|
||||||
|
const text = (await res.text().catch(() => '')).slice(0, 200);
|
||||||
|
const retryable = res.status === 429 || res.status >= 500;
|
||||||
|
if (!retryable || attempt >= retries) {
|
||||||
|
throw new HttpError(res.status, `GET ${shortUrl(url)}: HTTP ${res.status} ${text}`);
|
||||||
|
}
|
||||||
|
const retryAfter = Number(res.headers.get('retry-after'));
|
||||||
|
const wait = Number.isFinite(retryAfter) && retryAfter > 0 ? retryAfter * 1000 : backoffMs;
|
||||||
|
console.warn(` HTTP ${res.status} on ${shortUrl(url)}; waiting ${Math.round(wait / 1000)}s`);
|
||||||
|
opts.limiter.backoff(wait);
|
||||||
|
backoffMs = Math.min(backoffMs * 2, 300_000);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* ScoreSaber-style headers: x-ratelimit-remaining-{short,medium,long} and
|
||||||
|
* x-ratelimit-reset-{...} (seconds until that window resets). When any window is exhausted,
|
||||||
|
* hold the limiter until it resets instead of running into a 429.
|
||||||
|
*/
|
||||||
|
function honorRateLimitHeaders(headers: Headers, limiter: Limiter, url: string): void {
|
||||||
|
for (const w of ['short', 'medium', 'long']) {
|
||||||
|
const remaining = headers.get(`x-ratelimit-remaining-${w}`);
|
||||||
|
const reset = Number(headers.get(`x-ratelimit-reset-${w}`));
|
||||||
|
if (remaining !== null && Number(remaining) <= 0 && Number.isFinite(reset) && reset > 0) {
|
||||||
|
console.warn(` rate limit window "${w}" exhausted on ${shortUrl(url)}; pausing ${reset}s`);
|
||||||
|
limiter.backoff((reset + 1) * 1000);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function shortUrl(url: string): string {
|
||||||
|
return url.split('?')[0];
|
||||||
|
}
|
||||||
@@ -0,0 +1,101 @@
|
|||||||
|
// ScoreSaber exporter: player snapshot, scores (score + leaderboard items), replays.
|
||||||
|
// Public API, no auth. Replays are paced at ~10.5 s (360/hour limit) and run as their own command.
|
||||||
|
import * as fs from 'node:fs';
|
||||||
|
import * as path from 'node:path';
|
||||||
|
import { SS_SCRAPER_REPLAYS } from './config.ts';
|
||||||
|
import { get, limiters } from './http.ts';
|
||||||
|
import { archivePath, eachWithProgress, exists, listFiles, readJson, save, saveSnapshotIfChanged, updateState } from './store.ts';
|
||||||
|
|
||||||
|
const SS_API = 'https://scoresaber.com/api';
|
||||||
|
const ss = (playerId: string, ...p: string[]) => archivePath('scoresaber', playerId, ...p);
|
||||||
|
|
||||||
|
export type SSItem = {
|
||||||
|
score: { id: number; timeSet: string; hasReplay: boolean };
|
||||||
|
leaderboard: { songHash: string };
|
||||||
|
};
|
||||||
|
|
||||||
|
export function archivedSSScores(playerId: string, sinceUnix = 0): SSItem[] {
|
||||||
|
const dir = ss(playerId, 'scores');
|
||||||
|
return listFiles(dir, '.json')
|
||||||
|
.map((f) => readJson<SSItem>(path.join(dir, f)))
|
||||||
|
.filter((it) => Date.parse(it.score.timeSet) / 1000 >= sinceUnix);
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function exportSSPlayer(playerId: string): Promise<void> {
|
||||||
|
const r = await get(`${SS_API}/player/${playerId}/full`, { limiter: limiters.scoresaber });
|
||||||
|
console.log(`ScoreSaber player ${playerId}: ${saveSnapshotIfChanged(ss(playerId, 'player'), r.body) ? 'saved' : 'unchanged'}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Page `sort=recent` until scores are older than `sinceUnix`. */
|
||||||
|
export async function exportSSScores(playerId: string, sinceUnix: number): Promise<void> {
|
||||||
|
console.log(`ScoreSaber scores for ${playerId} since ${new Date(sinceUnix * 1000).toISOString()}`);
|
||||||
|
let added = 0;
|
||||||
|
let seen = 0;
|
||||||
|
for (let page = 1; ; page++) {
|
||||||
|
const r = await get(`${SS_API}/player/${playerId}/scores?limit=100&page=${page}&sort=recent`, { limiter: limiters.scoresaber });
|
||||||
|
const json = JSON.parse(r.body.toString('utf-8')) as { playerScores: SSItem[]; metadata: { total: number } };
|
||||||
|
let reachedCutoff = false;
|
||||||
|
for (const item of json.playerScores) {
|
||||||
|
if (Date.parse(item.score.timeSet) / 1000 < sinceUnix) {
|
||||||
|
reachedCutoff = true;
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
seen++;
|
||||||
|
const file = ss(playerId, 'scores', `${item.score.id}.json`);
|
||||||
|
if (exists(file)) continue;
|
||||||
|
save(file, JSON.stringify(item));
|
||||||
|
added++;
|
||||||
|
}
|
||||||
|
console.log(` page ${page}: ${seen} in window, ${added} new (total on server ${json.metadata.total})`);
|
||||||
|
if (reachedCutoff || json.playerScores.length < 100) break;
|
||||||
|
}
|
||||||
|
updateState(`scoresaber:${playerId}`, { scoresSince: sinceUnix, lastScoresRun: new Date().toISOString() });
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Download replays for archived scores with hasReplay, oldest first, ~10.5 s apart. */
|
||||||
|
export async function exportSSReplays(playerId: string, sinceUnix: number): Promise<void> {
|
||||||
|
const items = archivedSSScores(playerId, sinceUnix)
|
||||||
|
.filter((it) => it.score.hasReplay)
|
||||||
|
.sort((a, b) => Date.parse(a.score.timeSet) - Date.parse(b.score.timeSet));
|
||||||
|
seedFromScraper(playerId, items);
|
||||||
|
const todo = items.filter((it) => !exists(ss(playerId, 'replays', `${it.score.id}.dat`)));
|
||||||
|
console.log(`ScoreSaber replays for ${playerId}: ${items.length} with hasReplay, ${todo.length} to download (~${Math.ceil((todo.length * 10.5) / 60)} min)`);
|
||||||
|
const res = await eachWithProgress(
|
||||||
|
'ss-replays',
|
||||||
|
todo,
|
||||||
|
async (it) => {
|
||||||
|
const r = await get(`${SS_API}/v2/scores/${it.score.id}/replay`, {
|
||||||
|
limiter: limiters.scoresaberReplay,
|
||||||
|
accept: [404],
|
||||||
|
timeoutMs: 300_000
|
||||||
|
});
|
||||||
|
if (r.status === 404) return 'missing';
|
||||||
|
save(ss(playerId, 'replays', `${it.score.id}.dat`), r.body);
|
||||||
|
return 'done';
|
||||||
|
},
|
||||||
|
10
|
||||||
|
);
|
||||||
|
if (res.missing || res.failed.length) {
|
||||||
|
updateState('failures', { [`scoresaber:${playerId}:replays`]: res.failed.map((f) => f.item.score.id) });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Copy replays ss-scraper already downloaded (<scoreId>_<name>.dat) instead of fetching them again. */
|
||||||
|
function seedFromScraper(playerId: string, items: SSItem[]): void {
|
||||||
|
const dir = path.join(SS_SCRAPER_REPLAYS, playerId);
|
||||||
|
const byId = new Map<string, string>();
|
||||||
|
for (const f of listFiles(dir, '.dat')) byId.set(f.split('_')[0], f);
|
||||||
|
let copied = 0;
|
||||||
|
for (const it of items) {
|
||||||
|
const src = byId.get(String(it.score.id));
|
||||||
|
const dest = ss(playerId, 'replays', `${it.score.id}.dat`);
|
||||||
|
if (!src || exists(dest)) continue;
|
||||||
|
save(dest, fs.readFileSync(path.join(dir, src)));
|
||||||
|
copied++;
|
||||||
|
}
|
||||||
|
if (copied) console.log(` copied ${copied} replays from ${dir}`);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function ssHashes(playerId: string, sinceUnix = 0): string[] {
|
||||||
|
return [...new Set(archivedSSScores(playerId, sinceUnix).map((it) => it.leaderboard.songHash.toLowerCase()))].sort();
|
||||||
|
}
|
||||||
@@ -0,0 +1,80 @@
|
|||||||
|
// Archive filesystem helpers. The archive is append-only: files are written once
|
||||||
|
// (atomically) and never rewritten, except state.json.
|
||||||
|
import * as fs from 'node:fs';
|
||||||
|
import * as path from 'node:path';
|
||||||
|
import { ARCHIVE_DIR, writeFileAtomic } from './config.ts';
|
||||||
|
|
||||||
|
export const archivePath = (...parts: string[]) => path.join(ARCHIVE_DIR, ...parts);
|
||||||
|
|
||||||
|
export const exists = (file: string) => fs.existsSync(file);
|
||||||
|
|
||||||
|
export function save(file: string, data: string | Uint8Array): void {
|
||||||
|
writeFileAtomic(file, data);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Today's date (UTC) as YYYY-MM-DD, used for dated snapshot filenames. */
|
||||||
|
export const today = () => new Date().toISOString().slice(0, 10);
|
||||||
|
|
||||||
|
export function listFiles(dir: string, ext?: string): string[] {
|
||||||
|
try {
|
||||||
|
return fs.readdirSync(dir).filter((f) => !f.includes('.tmp-') && (!ext || f.endsWith(ext)));
|
||||||
|
} catch {
|
||||||
|
return [];
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export function readJson<T>(file: string): T {
|
||||||
|
return JSON.parse(fs.readFileSync(file, 'utf-8')) as T;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Save a dated snapshot in `dir` only when the content differs from the newest existing one.
|
||||||
|
* Returns true if a new snapshot was written.
|
||||||
|
*/
|
||||||
|
export function saveSnapshotIfChanged(dir: string, body: Buffer): boolean {
|
||||||
|
const latest = listFiles(dir, '.json').sort().at(-1);
|
||||||
|
if (latest && fs.readFileSync(path.join(dir, latest)).equals(body)) return false;
|
||||||
|
save(path.join(dir, `${today()}.json`), body);
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
type State = Record<string, Record<string, unknown>>;
|
||||||
|
const STATE_FILE = () => archivePath('state.json');
|
||||||
|
|
||||||
|
export function readState(): State {
|
||||||
|
try {
|
||||||
|
return readJson<State>(STATE_FILE());
|
||||||
|
} catch {
|
||||||
|
return {};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export function updateState(section: string, patch: Record<string, unknown>): void {
|
||||||
|
const state = readState();
|
||||||
|
state[section] = { ...state[section], ...patch };
|
||||||
|
writeFileAtomic(STATE_FILE(), JSON.stringify(state, null, 2) + '\n');
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Run `fn` over items with a progress line every `every` items. Errors are collected, not thrown. */
|
||||||
|
export async function eachWithProgress<T>(
|
||||||
|
label: string,
|
||||||
|
items: T[],
|
||||||
|
fn: (item: T) => Promise<'done' | 'skipped' | 'missing'>,
|
||||||
|
every = 50
|
||||||
|
): Promise<{ done: number; skipped: number; missing: number; failed: { item: T; error: string }[] }> {
|
||||||
|
const out = { done: 0, skipped: 0, missing: 0, failed: [] as { item: T; error: string }[] };
|
||||||
|
let i = 0;
|
||||||
|
for (const item of items) {
|
||||||
|
i++;
|
||||||
|
try {
|
||||||
|
out[await fn(item)]++;
|
||||||
|
} catch (err) {
|
||||||
|
out.failed.push({ item, error: (err as Error).message });
|
||||||
|
console.warn(` ${label}: ${(err as Error).message}`);
|
||||||
|
}
|
||||||
|
if (i % every === 0 || i === items.length) {
|
||||||
|
console.log(` ${label}: ${i}/${items.length} (new ${out.done}, existing ${out.skipped}, missing ${out.missing}, failed ${out.failed.length})`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user