From 257df6f1d8c306fe8bce7a9bce8e503da25b8140 Mon Sep 17 00:00:00 2001 From: pleb Date: Sun, 8 Nov 2026 10:38:16 -0800 Subject: [PATCH] Add plan for a personal BeatLeader mirror on incineroar docs/plan.md covers exporting BeatLeader and ScoreSaber history into a raw archive, a SQLite fork of beatleader-server, merging ScoreSaber scores into BeatLeader scores, ArcViewer replays, and a daily sync. AGENTS.md points new sessions at it. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 4 + docs/plan.md | 644 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 648 insertions(+) create mode 100644 docs/plan.md diff --git a/AGENTS.md b/AGENTS.md index 22c10c7..12e15e7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -17,3 +17,7 @@ - Dont' use `cd`, prefer `pwd` - Dont' use `rg`, prefer `grep -r` + +## Personal BeatLeader mirror + +- Active project: a self-hosted BeatLeader fork on incineroar holding only pleb's data, plus ScoreSaber history and ArcViewer. Read [docs/plan.md](docs/plan.md) first; it records the decisions, the verified API facts, and the phased steps. diff --git a/docs/plan.md b/docs/plan.md new file mode 100644 index 0000000..ae64148 --- /dev/null +++ b/docs/plan.md @@ -0,0 +1,644 @@ +# Plan: a personal BeatLeader on incineroar + +Goal: run my own BeatLeader instance on incineroar that holds only my profile +(`pleb`) and play history, with ScoreSaber history merged in, and keep it in +sync with upstream BeatLeader. The minimum viable version stays a fork of +upstream rather than a rewrite. Replacing BeatLeader with my own system is a +later project. This plan makes sure the data I collect is ready for that. + +Written 2026-11-08. The numbers under "Facts" were checked live on that date. + +## Decisions + +| Topic | Decision | +|---|---| +| Server | Fork `BeatLeader/beatleader-server` (ASP.NET Core, net10.0). Swap SQL Server for SQLite and Cloudflare R2 for the local filesystem. Turn off every background job that recalculates ranks or pp. | +| Frontend | Fork `BeatLeader/beatleader-website` (Svelte 5 + rollup). Change as little as possible. | +| Replay viewer | Self-host ArcViewer's prebuilt WebGL build (the `deploy` branch) for both BeatLeader (`.bsor`) and ScoreSaber (`.dat`) replays. No Unity build. | +| Source of truth | A raw, append-only **archive** of unmodified API responses and replay files. The SQLite database can always be rebuilt from it, and so can any future replacement system. | +| Exporter and sync | TypeScript in this repo, building on `src/lib/server/beatleader.ts`, `beatleaderAuth.ts`, `scoresaber.ts` and `beatsaver.ts`. It writes only to the archive. | +| BeatLeader auth | OAuth 2.0 (`profile offline_access`), the same flow as plebsaber.stream's login, which already unlocks unranked ratings. The sync owns its own chain of refresh tokens. | +| Map files | Archive the BeatSaver zip of every map I've played, using `beatsaver.ts`. Unpack them into the server's `MapsPath`, so the server and ArcViewer never need BeatSaver. | +| Sync schedule | Daily at 17:00 America/Los_Angeles, incineroar's timezone. | +| Access | Only incineroar and snorlax, through Caddy's `remote_ip` allowlist. | +| Map data scope | Only maps I've played: BeatLeader leaderboards and ratings, plus the BeatSaver metadata and zip. | +| Importer | C# inside the server fork, so it shares the EF models. It reads the archive and upserts into SQLite. | +| Upstream values | Keep them as they are: pp, ranks, star ratings, and the skill triangle. The local instance has one player, so recalculating would turn every rank into #1. | +| Attempts | **Skipped.** No failed or quit attempt replays, and no attempt records (`/scoresstats`). Leaving them out is less work: it's one endpoint the exporter simply doesn't call. Best scores, with their `scoreImprovement`, are the play history. | +| ScoreSaber merge | **Convert ScoreSaber scores into BeatLeader `Score` rows.** Each leaderboard keeps one score, chosen across BL, SS main and SS legacy by BeatLeader's own rule. BeatLeader's "improvement over previous best" display then covers ScoreSaber plays too. No separate table. | +| Identity | One local player: id `76561199407393962`, name `pleb`. The legacy Steam id `76561197962107242` (`i5u1d0r`) becomes a linked alt of that player. | +| Port | `https://incineroar.brenise.dev:4442`. Caddy uses 4430–4441 on incineroar, so 4442 is the next free one (4444 is free too). The API listens on `127.0.0.1:8042`, which is unused. | + +## Facts (checked 2026-11-08) + +### My accounts + +| Source | Id | Scores | Replays | +|---|---|---|---| +| BeatLeader `pleb` | 76561199407393962 | 3,565 best scores (2,492 unranked); 3,559 play count | one `.bsor` per score on `cdn.replays.beatleader.com`, about 1.3 MB each | +| BeatLeader attempts (**not exported**) | same | 16,518 attempt records (`/player/{id}/scoresstats`), including fails and quits | non-finished attempts are at `/otherreplays/.bsor`, auth-gated | +| BeatLeader legacy | 76561197962107242 (`i5u1d0r`) | 3 | | +| ScoreSaber main | 76561199407393962 (`pleb`) | 2,107 (814 ranked), 1,474 unique maps, first on 2023-11-17 | **1,034** have `hasReplay` | +| ScoreSaber legacy | 76561197962107242 (`i5u1d0r`), inactive | 199 (161 ranked), first seen 2022-05-07 | 58 have replays, **already downloaded** to `~/ops/beatsaber/ss-scraper/replays/76561197962107242/` (`.dat` plus a `.json` with metadata each) | + +The local mods keep replays too, as a fallback source. The Windows BSManager +instance (`~/Windows/Users/pleb/BSManager/BSInstances//`, mounted with +`mountwin` on incineroar) has `UserData/BeatLeader/Replays/` and +`UserData/ScoreSaber/Replays/`. ScoreSaber keeps one file per +player/hash/difficulty/characteristic, so it can recover some scores whose +server copy has `hasReplay: false`. `~/ops/beatsaber/backups` deliberately +leaves these folders out. + +### Unranked ratings need an authenticated request + +- Every difficulty object has `stars`, `passRating`, `accRating`, `techRating` + (the skill triangle), `predictedAcc`, plus `modifiersRating` with + `ss*`/`fs*`/`sf*`/`bfs*`/`bsf*` variants of each rating. +- For **unranked** maps the server fills these in only when the *requesting* + user has `profileSettings.showAllRatings == true`. Mine does. An anonymous + request for the same score returns `stars/pass/acc/techRating: null`; only + `predictedAcc` comes through. See + `Controllers/PlayerScoresController.cs` (`showRatings = currentProfileSettings?.ShowAllRatings`). +- **The solution is OAuth 2.0, as in plebsaber.stream.** Logging in through + `/auth/beatleader/login` requests `scp:profile scp:offline_access`. Requests + sent with `Authorization: Bearer ` (`createBeatLeaderAPI(fetch, + token)`) come back with unranked ratings filled in. The code is in + `src/lib/server/beatleaderAuth.ts`: `exchangeCodeForTokens`, + `refreshAccessToken`, `getValidAccessToken`, and the token store + `beatleader_tokens.json` in the data dir. +- Token lifetimes: access tokens last 3,600 s and refresh tokens 14 days. Each + refresh **rotates** the refresh token and invalidates the old one. So the + sync job needs its **own** refresh token, separate from the + plebsaber.stream site's, or the two invalidate each other. It persists every + new refresh token atomically before using the new access token. The daily + sync stays well within the 14 days. If the chain breaks (no run for 14 days, + or a failed write), log in once more through plebsaber.stream's + `/auth/beatleader/login` and seed the new token. + +### BeatLeader endpoints the exporter uses + +Base `https://api.beatleader.com`. Rate limit: 50 GETs per 10 s overall, +10 per 10 s on `/players`, 8 per second on `/maps`. Swagger is at `/swagger/index.html`. + +| Data | Endpoint | Notes | +|---|---|---| +| identity and settings | `GET /user` | confirms the token works (`player.id`) | +| profile | `GET /player/{id}?stats=true` | `scoreStats`, `history`, pinned scores, badges | +| best scores | `GET /player/{id}/scores?count=100&page=N&sortBy=date&order=desc&includeIO=true` | `count` ≤ 100. Each score includes `leaderboard.song` and `leaderboard.difficulty` (with ratings), `replay`, `offsets`, `scoreImprovement`. Incremental runs use `time_from=`. | +| replay stats | `GET /score/statistic/{scoreId}` | the accuracy graph, hit tracker and so on, from the server's `scorestats` storage | +| map data | `GET /leaderboards/hash/{hash}` | every difficulty of a song, with ratings when the request is authenticated | +| rank history | `GET /player/{id}/history`, `/history/triangle`, `/accgraph` | | +| live feed (optional) | `wss://sockets.api.beatleader.com/scores` | all scores globally. Filter on my id for near-real-time sync. | + +BeatSaver (public, no auth): `BeatSaverAPI.getMapByHash(hash)` in +`src/lib/server/beatsaver.ts` calls `GET https://api.beatsaver.com/maps/hash/{hash}`. +Each entry in `versions[]` carries its own `hash` and `downloadURL` (the zip). +Each map version has its own hash, and scores refer to a version hash, so +download the version whose `hash` matches. Old versions may only be on the CDN +or nowhere at all. When a version or map is gone from BeatSaver, take it from +the game's own folders on the Windows side instead: `Beat Saber_Data/CustomLevels` +and `CustomWIPLevels` in the BSManager instance. The folders there are +unpacked maps, so compute their hash the way the game does: SHA-1 over +`Info.dat` plus each difficulty file, in order. + +The server's `MapsPath` (`MapPostprocessor/MapDownloader.cs`) wants +`//` folders containing an unpacked map with `Info.dat`. It +checks the lowercase and uppercase hash. Only when a folder is missing does it +fetch `beatsaver.com/api/maps/hash/{hash}` and unpack the zip. + +ScoreSaber (public, no auth): `GET https://scoresaber.com/api/player/{id}/full`, +`GET /api/player/{id}/scores?limit=100&page=N&sort=recent` (each item has +`score.hasReplay`), and `GET /api/v2/scores/{scoreId}/replay` (`.dat`). The +ScoreSaber API doesn't send CORS headers, so it only works server-side. +`~/ops/beatsaber/ss-scraper/download_replays.py` already handles pacing +(30 s minimum between replays), 429 backoff, and resuming. At that pace the +1,034 main-account replays take about 9 hours. + +### Upstream server: what the fork has to change + +These checkouts are in `~/src`. Run `git pull` before relying on them. + +- `~/src/BeatLeader/beatleader-server` (master was at 2026-07-19 `1474312`; the + live branch `rebalanced-pp` was updated 2026-10-25). Its public submodules are + initialized: `beatleader-server-models`, `beatleader-parser`, `ReplayDecoder`, + `MapPostprocessor`, `RatingAPI`. Two submodules are **private and + unreachable**: `beatleader-server-private` and `bs-simulator`. Upstream + already copes with that. `Startup.cs` falls back to `ReplayRecalculatorStub`, + and nothing in the public code imports a namespace that only the private + repos define. **I never compiled it**: an agent build of third-party code was + blocked in that session. Building on Linux is the first task in phase 2. +- Database: `UseSqlServer` for two contexts (`Startup.cs:380-386`): + `AppContext` (`DefaultConnection`) and `StorageContext` + (`StorageConnection`, which holds `PlayerLeaderboardStats`). There are 570 + SQL Server migrations. +- `Z.EntityFramework.Extensions.EFCore` is commercial, with a monthly trial. + Call sites: about 92 `BulkSaveChanges`, 56 `BulkUpdate`, 19 `BulkDelete`, + 3 `BulkInsert`. +- Raw SQL in 5 places: `DBCC CHECKIDENT` (`Services/DailyRefresh.cs`), + `TABLESAMPLE` (`ScoreController.GetRandomScore`), `TRUNCATE` and admin + statements (`Controllers/Admin/AdminController.cs`). +- Storage: all 31 helpers in `Utils/S3Helper.cs` go to Cloudflare R2 + (`S3AccountID.r2.cloudflarestorage.com`) and return + `https://cdn.{container}.beatleader.com/{file}`. Containers: `replays`, + `otherreplays`, `assets`, `previews`, `scorestats`, `playlists`, `unicode`, + `configs`, `replayedvalues`, `reepresets`, `songs`, `attempts`. Nothing calls + S3 outside this file. +- Background jobs (`Startup.cs:455-478`): `EveryMinuteRefresh`, + `MinuteRefresh`, `HourlyRefresh`, `DailyRefresh`, `HistoryService`, + `BotService`, `RankingService`, `SongService`, `RichPresenceService` and + `MapRefresh` run only when `ServicesHost=YES`. These always run: + `ConstantsService`, `RefreshTaskService`, `ClanTaskService`, + `ClanMessageService`, `ClanCallbackService`, `SearchService` (Lucene), + `OauthService`, `LeaderboardPlayerStatsService`. +- Config keys, all from `IConfiguration`: `DefaultConnection`, + `StorageConnection`, `S3*`, `MapsPath` (default `/home/maps`), `SteamKey`, + `SteamApi`, `BeatSaverId/Secret`, `Openiddict*` (production expects + `../keys/openiddict_certificate*.pfx`; development uses dev certs), plus + optional OAuth secrets for Patreon, Twitch, Discord and others. Those + providers register only when their keys are set. +- The star-rating models are public: `RatingAPI/` (C# with ONNX models) and + `beatleader-analyzer`. We don't need them for the minimum version, because + we keep upstream's ratings. They are what would let a later replacement rate + maps itself. + +### Upstream website + +`~/src/BeatLeader/beatleader-website` (2026-11-07). In +`src/network/queues/beatleader/api-queue.js`, on any host it doesn't recognize +the API base is **`/cors/blapi/`**, which is relative to the page. So our host +only needs Caddy to proxy `/cors/blapi/*` to the API, with no code change. It +rewrites assets to `/cors/cdn-assets-bl/` and `/cors/cdnbeatsaver/` in the same +way. `BL_REPLAYS_URL` and `BL_ANALYZER_URL` fall back to the public +`replay.beatleader.com` and need a patch. Build with `npm run build` (rollup) +into `public/`. + +### ArcViewer + +`~/src/AllPoland/ArcViewer` (Unity 6000.0.67f1). The `deploy` branch on GitHub +holds a ready WebGL site (`index.html`, `Build/`, `TemplateData/`). It loads +replays from `?replayURL=` and recognizes ScoreSaber replays by their +magic header (`ScoreSaberDecoder.IsScoreSaberReplay`, including the legacy +format), so it works for both `.bsor` and `.dat`. The map comes from `id=` +(BeatSaver) or `url=`. Serving it from the same origin as the replays +avoids CORS and mixed-content problems. + +### incineroar + +- Caddy vhosts are in `~/ops/nix-pleb/systems/brenise.dev/incineroar/workstation.nix`. + They follow the pattern `"${host_fqdn}:44xx"` with `${tlsConfig}` (Namecheap + DNS-01) and a `remote_ip` allowlist (`127.0.0.1/32 ::1/128 192.168.1.34/32 + 192.168.1.35/32 192.168.1.36/32 10.19.21.34/32`). The host firewall is off; + Caddy does the gating. +- Disk: `/` has 479 GB free and `/home` 684 GB. The archive should be 10–25 GB: + about 4.6 GB of best-score replays, about 1.4 GB of ScoreSaber replays, and BeatSaver zips for + roughly 2,000–3,000 played maps (ScoreSaber alone has 1,474 unique hashes), + at a few MB each. +- Allowed clients: incineroar (`127.0.0.1/32`, `::1/128`, `192.168.1.36/32`) and + snorlax (`192.168.254.21/32`, the address the amelia fallback vhost already + allows). +- Timezone: `America/Los_Angeles`. +- nix-pleb rule: put host app config in an existing `/.nix` + (here `workstation.nix`), not a new module file. Never use `nixos-rebuild + switch` without asking; use `build`. Never decrypt sops secrets: ask the user + to add the values. + +## Layout + +``` +plebsaber.stream/ (this repo) + docs/plan.md this file + scripts/archive/ exporter and sync CLI (TypeScript, node ≥ 22) + src/lib/server/beatleader.ts extend: bearer auth from the sync's token store, paging, time_from, raw passthrough + src/lib/server/scoresaber.ts extend: player scores, replay download + +gitea pleb/beatleader-server fork, branch `pleb/main` + Tools/ArchiveImporter/ C# console app sharing AppContext/StorageContext +gitea pleb/beatleader-website fork, branch `pleb/main` + +/var/lib/plebleader/ on incineroar, user `plebleader` + archive/ source of truth, append-only + beatleader/ + user.json latest /user (no secrets) + players//profile/.json + players//history.json, triangle.json, accgraph.json + scores/.json one ScoreResponse each, unmodified + scorestats/.json + leaderboards//.json new snapshot only when the content changed + replays/.bsor + scoresaber// + player/.json + scores/.json the full item (score plus leaderboard) + replays/.dat + beatsaver/ + maps/.json getMapByHash response, unmodified + zips/.zip the map version matching that hash + missing.json hashes BeatSaver no longer serves, with their source if recovered + game-levels// maps recovered from the Windows CustomLevels when BeatSaver lacks them + local-mods/{beatleader,scoresaber}/... copied from the Windows UserData + state.json sync cursors (last timepost per source/feed) + db/app.db, db/storage.db SQLite, rebuilt from the archive + storage/ S3 replacement (containers as dirs); replays/ and ssreplays/ symlink into archive/ + maps// MapsPath: archive zips unpacked (server + /cdn/maps) + auth/beatleader_tokens.json the sync's own OAuth token chain (rotates on refresh) + secrets → EnvironmentFile from sops BL_CLIENT_ID, BL_CLIENT_SECRET +``` + +## Phases + +Each phase ends with a check. Phase 1 protects the data and doesn't depend on +the fork, so do it first. + +### Phase 0: Prep + +1. Update the checkouts: + `git -C ~/src/BeatLeader/beatleader-server pull --recurse-submodules`, and + the same for `beatleader-website`, `~/src/AllPoland/ArcViewer` (also + `git fetch origin deploy`), and `~/src/ScoreSaber/website`. The ScoreSaber + website is public now; use it to check the score and replay API shapes. +2. Create the Gitea forks `pleb/beatleader-server` and `pleb/beatleader-website` + with the `gitea-api` skill (mirror or push). Keep `upstream` as a remote so + you can rebase later. +3. Bring this repo's flake up to date (it's still on `nixos-25.05`) only if the + exporter gets packaged here. +4. Ask the user to add a sops secret `plebleader.env` for incineroar, holding + `BL_CLIENT_ID` and `BL_CLIENT_SECRET`. These are the same OAuth app as + plebsaber.stream; check that its registered callback URLs include the one + used for seeding. Agents never decrypt it. + For local runs before deployment, use the gitignored `.env` in this repo. +5. **Seed the sync's token chain.** Add a small `scripts/archive` command + (`auth login`) that runs the same authorization-code flow as + `/auth/beatleader/login` with `scope=profile offline_access`. It catches the + callback on a loopback port, or reuses a plebsaber.stream callback URL the + OAuth app already allows. It writes `auth/beatleader_tokens.json`. Don't + copy the plebsaber.stream site's own refresh token, because rotation would + break whichever side refreshes second. + +### Phase 1: Full export (one-time backup) + +Build `scripts/archive/` as a small CLI (`export bl`, `export ss`, +`export maps`, `import-local-mods`, `sync`). Rules: + +- Store responses byte-for-byte. Never reshape them on the way in. +- Make every step resumable: skip files that already exist, keep cursors in + `state.json`, and write to temp files before renaming. +- Respect rate limits. Use about 4 requests per second to BeatLeader and honor + `Retry-After` (the existing `requestWith429Retry` handles 429s). For + ScoreSaber replays, keep ss-scraper's 30 s pacing. +- Send `User-Agent: plebsaber-archive/0.1`. Don't put an email address in it. + +Steps: + +1. **Auth check.** With the bearer token, `GET /user` must return `player.id == + 76561199407393962`, and an unranked score's `difficulty.stars` must not be + null. If either fails, stop. +2. **BeatLeader scores.** Page through `/player/{id}/scores` with + `sortBy=date&order=desc&count=100&includeIO=true` until it's exhausted + (about 36 pages). Do the same for the legacy id (3 scores). +3. **Attempts: skipped.** Don't call `/scoresstats` or `/otherreplays`. +4. **Replays.** Fetch every `scores/*.replay` from the CDN (no auth, CORS `*`). + Keep the original filenames. +5. **Score stats.** `GET /score/statistic/{id}` for each best score. +6. **BeatLeader map data.** Collect the set of played hashes: every unique + `song.hash` in BL scores, plus `leaderboard.songHash` in both + ScoreSaber accounts. Normalize them to lowercase. For each, call + `GET /leaderboards/hash/{hash}` with the bearer token. That gives ratings + and the triangle for every difficulty, ranked or not. +7. **Map files from BeatSaver.** For each played hash, call + `BeatSaverAPI.getMapByHash(hash)` and save the response to + `beatsaver/maps/.json`. Then download the zip for the `versions[]` + entry whose `hash` matches, to `beatsaver/zips/.zip`. Extend + `beatsaver.ts` with a `downloadMapZip(version)` that streams to a temp file, + checks it's a valid zip containing `Info.dat`, then renames it. Go about 2 + requests per second and honor 429s. Write each hash that can't be resolved + (deleted map, or an old version gone from the CDN) to `missing.json`. +8. **Recover missing maps from the game.** With the Windows partition mounted + (`mountwin`), hash every folder in the BSManager instance's + `Beat Saber_Data/CustomLevels` and `CustomWIPLevels`. Copy the folders whose + hash is in `missing.json` to `game-levels//`. Whatever is still + missing stays listed; those replays can't render the map. +9. **Unpack for the server.** Extract each zip, or copy each recovered folder, + to `maps//`. Then the server's `MapDownloader` never goes to the + network, and ArcViewer can load `url=/cdn/maps/.zip` (serve the zips) + instead of `id=` from BeatSaver. +10. **Profile and history.** Save `/player/{id}?stats=true`, `/history`, + `/history/triangle`, `/accgraph`, and `/user` (strip anything secret). +11. **ScoreSaber main.** Page through all scores (22 pages), then download the + 1,034 replays. Either port the `download_replays.py` logic to TypeScript or + run it as-is with `-o archive/scoresaber`, and normalize the layout. +12. **ScoreSaber legacy.** Copy the 58 replays and the metadata from + `~/ops/beatsaber/ss-scraper/replays/76561197962107242/` into + `archive/scoresaber/76561197962107242/` (the filenames start with the + scoreId). Fetch the 199 score items again so the archive has complete, + unmodified responses. +13. **Local mod replays.** With the Windows partition mounted (`mountwin`, then + `umount ~/Windows` afterwards), copy `UserData/BeatLeader/Replays` and + `UserData/ScoreSaber/Replays` into `archive/local-mods/`. Afterwards, match + the ScoreSaber files against scores that have `hasReplay: false` by + player/hash/difficulty/characteristic. + +**Check:** the counts match the live API: 3,565 BL scores, 3 legacy, +2,107 plus 199 SS scores, and replay files for every score with a +replay URL or `hasReplay`. Every played hash should have a BeatLeader +leaderboard snapshot, plus a BeatSaver zip or recovered game folder, unless +it's listed in `missing.json`. Write a `report` subcommand that prints counts +and what's missing, and paste its output into this file. + +### Phase 2: Server fork on SQLite + +Work on the `pleb/main` branch of `pleb/beatleader-server`. Keep each change +small and in its own commit, so upstream rebases stay manageable. + +1. **Build on Linux** before changing anything: + `nix shell nixpkgs#dotnet-sdk_10 -c dotnet build BeatLeader-Server.csproj` + (nixpkgs has SDK 10.0.403). Record which errors come from the missing + private submodules versus the platform. Delete or exclude the empty + `beatleader-server-private` and `bs-simulator` folders, and stub whatever + they turn out to supply. +2. **SQLite.** Replace `Microsoft.EntityFrameworkCore.SqlServer` with + `...Sqlite`, and point both `AddDbContextFactory` calls at `UseSqlite` + (`Data Source=/var/lib/plebleader/db/app.db` and `storage.db`). On + connection open, set `PRAGMA journal_mode=WAL; busy_timeout=5000;`. Move the + 570 SQL Server migrations aside and generate one fresh migration per context + (`dotnet ef migrations add InitialSqlite --context AppContext`, and the same + for `StorageContext`), so later schema changes still work. +3. **Drop Z.EntityFramework.Extensions.** Add `Utils/BulkCompat.cs` with + extension methods of the same names and shapes: `BulkSaveChanges` → + `SaveChanges`, `BulkUpdate` → `UpdateRange` + `SaveChanges`, `BulkDelete` → + `RemoveRange` + `SaveChanges`, `BulkInsert` → `AddRange` + `SaveChanges`. + With one player, performance doesn't matter, and about 170 call sites keep + compiling unchanged. +4. **Raw SQL.** `DBCC CHECKIDENT` → `DELETE FROM sqlite_sequence WHERE name IN (...)`. + `TABLESAMPLE` → `ORDER BY RANDOM() LIMIT 1`. `TRUNCATE` → `DELETE FROM`. +5. **Storage.** Rewrite the bodies of the 31 `S3Helper` methods to read and + write `${StoragePath}/{container}/{filename}`, and return + `${PublicBaseUrl}/cdn/{container}/{filename}`. `GetS3Client` can return a + client that's never used, so the method signatures don't change. +6. **Background jobs.** Leave `ServicesHost` unset. Add a config flag + (`MirrorMode=true`) that also skips registering `RankingService`, the clan + services, `OauthService` and `LeaderboardPlayerStatsService`. Keep + `SearchService` and `ConstantsService` if they start cleanly, and point + Lucene's index directory at the state dir. +7. **Auth.** For the minimum version the site is read-only behind Caddy's IP + allowlist. No login is needed: the website shows a public profile. If the + ratings in API responses still depend on `ShowAllRatings` for the + *requesting* user, then in `MirrorMode` treat requests as coming from + `76561199407393962`, or just store ratings so they always show. That means + patching the `showRatings` checks, which all read + `ProfileSettings.ShowAllRatings`, to `true`. +8. **Configuration.** Put an `appsettings.Production.json` template in the fork + and the real values in the NixOS unit's environment (`ASPNETCORE_URLS=http://127.0.0.1:8042`, + `ConnectionStrings`/`DefaultConnection`, `StoragePath`, `PublicBaseUrl`, + `MapsPath`, `MirrorMode`). Use development OpenIddict certs, or generate a + self-signed pfx into the state dir. + +**Check:** the server starts on an empty SQLite file, `/swagger/index.html` +loads, and `GET /player/76561199407393962` returns 404 rather than 500. + +### Phase 3: Importer + +`Tools/ArchiveImporter` lives in the fork, references the main project, and +uses its `AppContext`/`StorageContext`. Run it as a one-shot that is idempotent +and upserts by upstream id. + +- `beatleader/scores/*.json` (ScoreResponse) → `Song`, + `DifficultyDescription` (keep the upstream ratings and `ModifiersRating` as + they are), `Leaderboard`, `Score` (with upstream `Id`, `Rank`, `Pp`, + `Weight`, and `Timepost`), `ScoreImprovement`, `ReplayOffsets`. Rewrite + `Replay` to the local `/cdn/replays/`. +- `leaderboards//latest` updates the difficulty ratings, so maps you + played but have no best score for still get them. +- `beatsaver/maps/.json` fills in `Song` fields that BeatLeader leaves + out (BeatSaver key, description, tags, upload date). The server reads the + map files themselves from `maps//`, filled in phase 1, step 9. +- `players//profile/latest` → `Player`, `PlayerScoreStats`, + `ProfileSettings`, plus history rows from `/history`. Insert the legacy id as + an `AccountLink` or alt pointing to the main player, so `PlayerIdToMain` + resolves `/u/76561197962107242` to `pleb`. Its 3 BL scores go through the + phase 5 merge like any other candidate. +- `StorageContext` (attempts) stays empty, because attempts aren't exported. +- `scorestats/*.json` → the `scorestats` storage container, using the filename + the server expects (check `ScoreController.GetStatistic`). + +The response DTOs aren't the entities, and this mapping is the largest single +piece of work. Use `beatleader-server-models/Models/*` (entities) and +`ResponseUtils.cs` (entity → DTO) as the reference, working backwards. + +**Check:** the stock website (phase 4) shows my profile with the right score +count, the pp and ranks from upstream, unranked star ratings, and the triangle +on the profile. + +### Phase 4: Website, viewer, Caddy + +1. In the website fork, point `BL_REPLAYS_URL` (and the "watch replay" links) + at `/arcviewer/?replayURL=&url=.zip>`. + Fall back to `id=` only for hashes that have no local zip. Hide or stub + the features that can't work offline: clans, events, friends, the Socket + page, and the stage analyzer and `ppai2` calls in `StarRatingGraph.svelte` / + `star-generator.js`. Leave the `/cors/blapi/` resolution as it is. +2. Package the ArcViewer `deploy` branch with `fetchFromGitHub` at a pinned + revision, as static files. +3. Add a Caddy vhost in `incineroar/workstation.nix`: + + ``` + "${host_fqdn}:4442" = { + # Personal BeatLeader mirror (plebleader). Website + ArcViewer static, + # API on 8042, replays/assets/maps from the state dir. + # Only incineroar itself and snorlax (192.168.254.21). + extraConfig = '' + ${tlsConfig} + @plebleader_allowed remote_ip 127.0.0.1/32 ::1/128 192.168.1.36/32 192.168.254.21/32 + handle @plebleader_allowed { + handle_path /cors/blapi/* { reverse_proxy http://127.0.0.1:8042 } + handle_path /cdn/maps/* { root * /var/lib/plebleader/archive/beatsaver/zips; file_server } + handle_path /cdn/* { root * /var/lib/plebleader/storage; file_server } + handle_path /arcviewer/* { root * ${arcviewer}; file_server } + handle_path /cors/cdn-assets-bl/* { reverse_proxy https://cdn.assets.beatleader.com { header_up Host {upstream_hostport} } } + handle_path /cors/cdnbeatsaver/* { reverse_proxy https://cdn.beatsaver.com { header_up Host {upstream_hostport} } } + handle { root * ${plebleader-website}; try_files {path} /index.html; file_server } + } + handle { respond "Forbidden" 403 } + ''; + }; + ``` + + ArcViewer's WebGL build may need `Content-Encoding` headers for `.br`/`.gz` + files in `Build/`. Check which ones the `deploy` branch ships. + +**Check:** in Chromium, `https://incineroar.brenise.dev:4442/u/pleb` shows the +profile and scores, and a replay opens in ArcViewer. + +### Phase 5: Convert ScoreSaber scores and merge + +ScoreSaber scores become BeatLeader `Score` rows, and each leaderboard keeps +exactly one score. BeatLeader already shows how a score compares with the +previous best (`ScoreImprovement`), so after the merge that comparison covers +ScoreSaber plays too. This is importer logic in the fork. The archive keeps +the raw ScoreSaber responses as they are. + +1. **Provenance columns.** Add nullable `Source` (`null` = BeatLeader, + `"scoresaber"`) and `SourceScoreId` / `SourcePlayerId` to `Score`. This goes + into the fresh SQLite migration from phase 2. The website uses them for a + small "SS" badge, and the importer uses them to stay idempotent. +2. **Convert** each ScoreSaber item (`scoresaber/*/scores/*.json`) into a + candidate `Score`: + - Leaderboard: `leaderboard.songHash` (lowercased), plus + `leaderboard.difficulty.difficulty`. ScoreSaber uses 1/3/5/7/9 for + Easy…ExpertPlus, the same numbers as BeatLeader's `difficulty.value`. + `gameMode` `SoloStandard` → mode `Standard` (strip `Solo`). Resolve it to + the BL leaderboard from `leaderboards/` (phase 1, step 6). If BL has + none, create `Song`/`DifficultyDescription`/`Leaderboard` from the + BeatSaver JSON, with no ratings. + - `BaseScore`, `ModifiedScore`, `Modifiers` (the same comma-separated codes), + `MissedNotes`, `BadCuts`, `FullCombo`, `MaxCombo`, `Timeset`/`Timepost` + come from `timeSet`. Set `Platform = "scoresaber"` and `Hmd` = unknown, + unless the codes are mapped. + - `Accuracy = BaseScore / maxScore`, taking `maxScore` from the **BL** + difficulty. ScoreSaber reports `maxScore: 0` on many unranked + leaderboards. Check this against how `ReplayUtils` computes BL accuracy. + - `Pp`: `ReplayUtils.PpFromScore(score, difficulty)`, using the archived BL + ratings, when the BL difficulty is ranked; otherwise 0, as BeatLeader + does. ScoreSaber's own pp and rank aren't comparable, so keep them only in + the archive. Set `Rank` to 0, and have the website show "—" for + ScoreSaber-sourced rows. + - `Id = 1_000_000_000 + ssScoreId`. ScoreSaber ids are about 95M and + upstream BL ids about 151M, so this can't collide and still fits in an + int32. + - `Replay = /cdn/ssreplays/.dat` when the archive has one (from + the server, or recovered from local mod replays); otherwise null. No + `ReplayOffsets` and no score stats, because the server can't decode `.dat`. +3. **Merge per leaderboard.** The candidates are the upstream BL best, the + ScoreSaber main best, the ScoreSaber legacy best, and the legacy BL score. + Sort them by time and fold them with BeatLeader's own + `ReplayUtils.IsNewScoreBetter(old, new)`: compare by pp when either score + has pp, otherwise by modified score, where a no-fail score loses to a clean + one. The winner becomes the leaderboard's `Score`. + - `ScoreImprovement` on the winner is the difference from the best before it + in the fold: score, accuracy, pp, misses, bad cuts, modifiers, timeset. If + the winner is the upstream BL score and nothing from ScoreSaber came + between it and its previous BL best, keep upstream's `scoreImprovement` + as it is. + - Candidates that lose stay in the archive only. +4. **Player totals.** Keep upstream's `PlayerScoreStats`, pp and rank. They + won't count ScoreSaber scores that won. Recomputing the weighted pp total + locally is an optional later step; label the profile numbers "BeatLeader, + as of ". +5. **Website.** Show the "SS" badge on ScoreSaber-sourced scores, hide the + accuracy graph and stats for them (there are no score stats), and send their + replay button to `/arcviewer/?replayURL=/cdn/ssreplays/.dat&url=/cdn/maps/.zip`. + Every score shows under the name `pleb`; `i5u1d0r` survives only as + `SourcePlayerId`. + +Optional later: convert `.dat` into `.bsor`, using ArcViewer's +`ScoreSaberDecoder` as the reference for the format. The server's statistics +code could then produce score stats for ScoreSaber-sourced scores too. + +**Check:** the leaderboard count is the union of the BL and SS leaderboards I +played, with exactly one score each. The importer prints how many winners came +from each source. Spot-check a map where an old legacy ScoreSaber score beats +the BL score, and confirm the improvement display and the ArcViewer replay. + +### Phase 6: Ongoing sync + +The `sync` subcommand runs once a day from a systemd timer on incineroar: +`OnCalendar=*-*-* 17:00:00 America/Los_Angeles` with `Persistent=true`, so a +run missed while the machine was off happens at the next boot. By 5 pm the +day's play is usually over. + +1. **BeatLeader scores:** `/player/{id}/scores?sortBy=date&order=desc&time_from=`. + Overlap a day and skip ids you already have. New score ids → save the score + JSON, the replay, the score stats, and a leaderboard refresh for that hash. +2. **Attempts:** skipped (see Decisions). +3. **New maps:** for every hash not in the archive yet, do phase 1, steps 6–9: + BeatLeader leaderboard, BeatSaver JSON and zip, then unpack into `maps/`. + The game's own folders are only a manual fallback, since sync can't mount + Windows. List unresolved hashes in the run summary. +4. **Profile:** one dated snapshot per run, plus `/history`. +5. **Ratings drift:** on Sundays' run, re-fetch `/leaderboards/hash/{hash}` for + every map I've played, and store a new snapshot only when it changed. + Upstream rebalances ratings (see the `rebalanced-pp` branch), and keeping + the old snapshots preserves the history. +6. **ScoreSaber main:** `sort=recent` until reaching a known score id, then + download the new replays at the slow pace. A day's worth fits easily in one + run. Legacy is frozen; skip it. +7. **Token upkeep:** refresh the OAuth token at the start of each run and write + the rotated refresh token to `auth/beatleader_tokens.json` (temp file plus + rename) before doing anything else. A daily run stays far inside the 14-day + window. If the refresh fails, the unit fails loudly and the message says to + run `auth login` again. Alloy ships journal lines to Loki, so set up an + alert there. +8. Once the archive has changed, run the importer incrementally, through a + `--since` flag or by watching `state.json`. That includes the phase 5 merge + for every leaderboard that got a new BL or SS score. + +Optional later: a manual `sync` run (`systemctl start plebleader-sync`) for +when I want results before 5 pm. + +**Check:** play a map, and after the next 17:00 run the score, its replay and +the map file show up on the local site, and the replay opens in ArcViewer with +the local zip. + +### Phase 7: NixOS deployment on incineroar + +- Packages: `buildDotnetModule` for the fork (generate `deps.json` with + `nix build .#…passthru.fetch-deps`), `buildNpmPackage` for the website (start + with `npmDepsHash = lib.fakeHash`, then pin the `got:` value), and + `fetchFromGitHub` for ArcViewer `deploy`. The forks expose their packages + through their own flakes and become flake inputs in nix-pleb, next to the + existing `plebsaber-stream` input. Use `git+https://gitea.satstack.dev/pleb/.git`. +- Services: put `plebleader-api`, `plebleader-sync` (service and timer) and the + Caddy vhost in `incineroar/workstation.nix`. Don't create a new module file. + Use a static user `plebleader`, `StateDirectory=plebleader`, and an + `EnvironmentFile` from the sops secret. +- Validate with + `nix build .#nixosConfigurations.incineroar.config.system.build.toplevel --no-link`. + The user does the switch. + +## Risks and fallbacks + +- **The server doesn't build or run on Linux, or EF/SQLite can't translate + queries.** SQL Server is case-insensitive and SQLite isn't, and SQLite has no + `DateTimeOffset` ordering. Fix it endpoint by endpoint, but only for the + endpoints the profile, scores and leaderboard pages call (find them in the + network tab). Fallback: skip the fork and write a small read-only API in + this repo that serves the archive in the response shapes the website + expects. That would also be the first step toward the full replacement. +- **The OAuth refresh chain breaks** (more than 14 days without a run, a lost + write, or upstream auth changes). The archive keeps everything collected so + far. Sync stops, a failing unit makes that visible, and `auth login` + re-seeds the chain. +- **Maps disappear from BeatSaver.** That's why zips are archived as soon as a + hash first appears. Old versions that are already gone come from the game's + own folders, or stay listed in `missing.json`. +- **Upstream API changes.** The archive stores raw responses, so a change + breaks the importer, not the data. Version the importer's mapping. +- **Upstream code moves fast** (the website is updated daily). Pin fork + revisions. Rebase only when there's a reason to. +- **Ranks and pp go stale locally.** That's by design: they're upstream's + values from the last sync. The UI could show "as of ". + +## Open questions for the user + +Settled on 2026-11-08: port 4442; access limited to incineroar and snorlax; +map data limited to played maps; OAuth for auth; sync daily at 17:00 +America/Los_Angeles. + +Also settled: skip attempts and their replays entirely. Where BL and SS scores +overlap, keep one per leaderboard by converting ScoreSaber scores to +BeatLeader scores (phase 5). + +Nothing is open right now. + +## Where the earlier work lives + +- `~/ops/beatsaber/ss-scraper/`: ScoreSaber replay downloader. The legacy + account is done; it has no git commits yet. +- `~/ops/beatsaber/playlist-tool/src/clients/{beatleader,scoresaber,beatsaver}/`: + generated Python OpenAPI clients. Use them as a reference for the response + models. +- `~/ops/beatsaber/backups/`: UserData backups. They leave the replay + folders out. +- `~/ops/beatsaber/beatleader/beatleader-website.code-workspace`: points at + `~/src/beatleader/...` (lowercase). The checkouts are actually in + `~/src/BeatLeader/`, so fix that path. +- `~/src/BeatLeader/bl-dev-sandbox` is a BSOR v2 format experiment, not a dev + environment. Ignore it.