plebandClaude Opus 5.5
d0d746ec7f
ScoreSaber names official levels "ost_<LevelId>" where BeatLeader uses "<levelid>"; ssMapHash applies that for leaderboard snapshots and maps. The plan records the merge results and where it differs from the steps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1112 lines
68 KiB
Markdown
1112 lines
68 KiB
Markdown
# Plan: a personal BeatLeader on incineroar
|
||
|
||
Goal: run my own BeatLeader instance on incineroar that holds only my profile
|
||
(`pleb`) and play history, with ScoreSaber history merged in, and keep it in
|
||
sync with upstream BeatLeader. The minimum viable version stays a fork of
|
||
upstream rather than a rewrite. Replacing BeatLeader with my own system is a
|
||
later project. This plan makes sure the data I collect is ready for that.
|
||
|
||
Written 2026-11-08. The numbers under "Facts" were checked live on that date.
|
||
|
||
## Decisions
|
||
|
||
| Topic | Decision |
|
||
|---|---|
|
||
| Server | Fork `BeatLeader/beatleader-server` (ASP.NET Core, net10.0). Swap SQL Server for SQLite and Cloudflare R2 for the local filesystem. Turn off every background job that recalculates ranks or pp. |
|
||
| Frontend | Fork `BeatLeader/beatleader-website` (Svelte 5 + rollup). Change as little as possible. |
|
||
| Replay viewer | Self-host ArcViewer's prebuilt WebGL build (the `deploy` branch) for both BeatLeader (`.bsor`) and ScoreSaber (`.dat`) replays. No Unity build. |
|
||
| Source of truth | A raw, append-only **archive** of unmodified API responses and replay files. The SQLite database can always be rebuilt from it, and so can any future replacement system. |
|
||
| Exporter and sync | TypeScript in this repo, building on `src/lib/server/beatleader.ts`, `beatleaderAuth.ts`, `scoresaber.ts` and `beatsaver.ts`. It writes only to the archive. |
|
||
| BeatLeader auth | OAuth 2.0 (`profile offline_access`). The sync owns its own chain of refresh tokens (`scripts/archive/cli.ts auth login`, seeded 2026-11-08). Unranked ratings do **not** need it: they come from the anonymous `/leaderboards/hash/{hash}` (see Facts). |
|
||
| Map files | Archive the BeatSaver zip of every map I've played, using `beatsaver.ts`. Don't unpack them ahead of time: ArcViewer loads the zip directly, and the server unpacks on demand only if a mirror code path ever needs map files (phase 2, step 9). Neither ever needs BeatSaver. |
|
||
| Sync schedule | Daily at 17:00 America/Los_Angeles, incineroar's timezone. |
|
||
| Access | Only incineroar and snorlax, through Caddy's `remote_ip` allowlist. |
|
||
| Map data scope | Only maps I've played: BeatLeader leaderboards and ratings, plus the BeatSaver metadata and zip. |
|
||
| Importer | C# inside the server fork, so it shares the EF models. It reads the archive and upserts into SQLite. |
|
||
| Upstream values | Keep them as they are: pp, ranks, star ratings, and the skill triangle. The local instance has one player, so recalculating would turn every rank into #1. |
|
||
| Attempts | **Skipped.** No failed or quit attempt replays, and no attempt records (`/scoresstats`). Leaving them out is less work: it's one endpoint the exporter simply doesn't call. Best scores, with their `scoreImprovement`, are the play history. |
|
||
| ScoreSaber merge | **Convert ScoreSaber scores into BeatLeader `Score` rows.** Each leaderboard keeps one score, chosen across BL, SS main and SS legacy by BeatLeader's own rule. BeatLeader's "improvement over previous best" display then covers ScoreSaber plays too. No separate table. |
|
||
| Identity | One local player: id `76561199407393962`, name `pleb`. The legacy Steam id `76561197962107242` (`i5u1d0r`) becomes a linked alt of that player. |
|
||
| Port | `https://incineroar.brenise.dev:4442`. Caddy uses 4430–4441 on incineroar, so 4442 is the next free one (4444 is free too). The API listens on `127.0.0.1:8042`, which is unused. |
|
||
|
||
## Facts (checked 2026-11-08)
|
||
|
||
### My accounts
|
||
|
||
| Source | Id | Scores | Replays |
|
||
|---|---|---|---|
|
||
| BeatLeader `pleb` | 76561199407393962 | 3,565 best scores (2,492 unranked); 3,559 play count | one `.bsor` per score on `cdn.replays.beatleader.com`, about 1.3 MB each |
|
||
| BeatLeader attempts (**not exported**) | same | 16,518 attempt records (`/player/{id}/scoresstats`), including fails and quits | non-finished attempts are at `/otherreplays/<name>.bsor`, auth-gated |
|
||
| BeatLeader legacy | 76561197962107242 (`i5u1d0r`) | 3 | |
|
||
| ScoreSaber main | 76561199407393962 (`pleb`) | 2,107 (814 ranked), 1,474 unique maps, first on 2023-11-17 | **1,034** have `hasReplay` |
|
||
| ScoreSaber legacy | 76561197962107242 (`i5u1d0r`), inactive | 199 (161 ranked), first seen 2022-05-07 | 58 have replays, **already downloaded** to `~/ops/beatsaber/ss-scraper/replays/76561197962107242/` (`.dat` plus a `.json` with metadata each) |
|
||
|
||
The local mods keep replays too, as a fallback source. The Windows BSManager
|
||
instance (`~/Windows/Users/pleb/BSManager/BSInstances/<ver>/`, mounted with
|
||
`mountwin` on incineroar) has `UserData/BeatLeader/Replays/` and
|
||
`UserData/ScoreSaber/Replays/`. ScoreSaber keeps one file per
|
||
player/hash/difficulty/characteristic, so it can recover some scores whose
|
||
server copy has `hasReplay: false`. `~/ops/beatsaber/backups` deliberately
|
||
leaves these folders out.
|
||
|
||
### Unranked ratings and auth (corrected 2026-11-08)
|
||
|
||
- Every difficulty object has `stars`, `passRating`, `accRating`, `techRating`
|
||
(the skill triangle), `predictedAcc`, plus `modifiersRating` with
|
||
`ss*`/`fs*`/`sf*`/`bfs*`/`bsf*` variants of each rating.
|
||
- On **score lists** (`/player/{id}/scores`) and `/leaderboard/{id}`, unranked
|
||
ratings are filled in only when the *requesting* user has
|
||
`profileSettings.showAllRatings == true`. That check uses
|
||
`HttpContext.CurrentUserID`, which reads only the beatleader.com **cookie
|
||
session** (with an `Issued` claim). An OAuth bearer token does not count:
|
||
tested, bearer and anonymous both return `stars/pass/acc/techRating: null`.
|
||
- **`GET /leaderboards/hash/{hash}` returns full ratings for unranked
|
||
difficulties, anonymously**, including all `modifiersRating` variants.
|
||
Tested on an unranked ExpertPlus (`9db01c9b…`): stars 8.33, pass 3.92, acc
|
||
9.75, tech 7.70. Phase 1, step 6 already fetches this for every played hash,
|
||
so the importer takes difficulty ratings from these snapshots, not from the
|
||
nulls embedded in the scores.
|
||
- Bearer tokens only work on `/oauth2/identity`, clan management and
|
||
ReeSabers presets (`CurrentOauthUserID`). `GET /user` with a bearer token is
|
||
401. The exporter doesn't actually need auth; the token chain is kept for
|
||
identity checks and in case a cookie-gated endpoint turns out to matter.
|
||
- The sync's chain lives in `scripts/archive/auth.ts`: login via the loopback
|
||
redirect `http://localhost:5173/auth/beatleader/callback` (registered on the
|
||
OAuth app, as is `https://plebsaber.stream/auth/beatleader/callback`), token
|
||
store `<PLEBLEADER_DIR>/auth/beatleader_tokens.json` (default
|
||
`.data/plebleader/`, mode 0600). `auth status` checks identity and ratings.
|
||
- Token lifetimes: access tokens last 3,600 s and refresh tokens 14 days. Each
|
||
refresh **rotates** the refresh token and invalidates the old one. So the
|
||
sync job needs its **own** refresh token, separate from the
|
||
plebsaber.stream site's, or the two invalidate each other. It persists every
|
||
new refresh token atomically before using the new access token. The daily
|
||
sync stays well within the 14 days. If the chain breaks (no run for 14 days,
|
||
or a failed write), log in once more through plebsaber.stream's
|
||
`/auth/beatleader/login` and seed the new token.
|
||
|
||
### BeatLeader endpoints the exporter uses
|
||
|
||
Base `https://api.beatleader.com`. Rate limit: 50 GETs per 10 s overall,
|
||
10 per 10 s on `/players`, 8 per second on `/maps`. Swagger is at `/swagger/index.html`.
|
||
|
||
| Data | Endpoint | Notes |
|
||
|---|---|---|
|
||
| identity and settings | `GET /user` | confirms the token works (`player.id`) |
|
||
| profile | `GET /player/{id}?stats=true` | `scoreStats`, `history`, pinned scores, badges |
|
||
| best scores | `GET /player/{id}/scores?count=100&page=N&sortBy=date&order=desc&includeIO=true` | `count` ≤ 100. Each score includes `leaderboard.song` and `leaderboard.difficulty` (with ratings), `replay`, `offsets`, `scoreImprovement`. Incremental runs use `time_from=<unix>`. |
|
||
| replay stats | `GET /score/statistic/{scoreId}` | the accuracy graph, hit tracker and so on, from the server's `scorestats` storage |
|
||
| map data | `GET /leaderboards/hash/{hash}` | every difficulty of a song, with ratings when the request is authenticated |
|
||
| rank history | `GET /player/{id}/history`, `/history/triangle`, `/accgraph` | |
|
||
| live feed (optional) | `wss://sockets.api.beatleader.com/scores` | all scores globally. Filter on my id for near-real-time sync. |
|
||
|
||
BeatSaver (public, no auth): `BeatSaverAPI.getMapByHash(hash)` in
|
||
`src/lib/server/beatsaver.ts` calls `GET https://api.beatsaver.com/maps/hash/{hash}`.
|
||
Each entry in `versions[]` carries its own `hash` and `downloadURL` (the zip).
|
||
Each map version has its own hash, and scores refer to a version hash, so
|
||
download the version whose `hash` matches. Old versions may only be on the CDN
|
||
or nowhere at all. When a version or map is gone from BeatSaver, take it from
|
||
the game's own folders on the Windows side instead: `Beat Saber_Data/CustomLevels`
|
||
and `CustomWIPLevels` in the BSManager instance. The folders there are
|
||
unpacked maps, so compute their hash the way the game does: SHA-1 over
|
||
`Info.dat` plus each difficulty file, in order.
|
||
|
||
The server's `MapsPath` (`MapPostprocessor/MapDownloader.cs`) wants
|
||
`<MapsPath>/<hash>/` folders containing an unpacked map with `Info.dat`. It
|
||
checks the lowercase and uppercase hash. Only when a folder is missing does it
|
||
fetch `beatsaver.com/api/maps/hash/{hash}` and unpack the zip.
|
||
|
||
ScoreSaber (public, no auth): `GET https://scoresaber.com/api/player/{id}/full`,
|
||
`GET /api/player/{id}/scores?limit=100&page=N&sort=recent` (each item has
|
||
`score.hasReplay`), and `GET /api/v2/scores/{scoreId}/replay` (`.dat`). The
|
||
ScoreSaber API doesn't send CORS headers, so it only works server-side.
|
||
`~/ops/beatsaber/ss-scraper/download_replays.py` already handles pacing
|
||
(30 s minimum between replays), 429 backoff, and resuming. That pacing is
|
||
outdated: as of 2026-11-08 ScoreSaber sends `x-ratelimit-{limit,remaining,reset}-{short,medium,long}`
|
||
headers. The replay endpoint allows 20 (short), 60/min (medium) and
|
||
**360/hour** (long, the one that binds); player endpoints allow 25/100/400.
|
||
`scripts/archive` paces replays at 10.5 s and pauses whenever a window's
|
||
remaining count hits 0, so 1,034 replays take about 3 h instead of 9.
|
||
|
||
### Upstream server: what the fork has to change
|
||
|
||
These checkouts are in `~/src`. Run `git pull` before relying on them.
|
||
|
||
- `~/src/BeatLeader/beatleader-server` (master was at 2026-07-19 `1474312`; the
|
||
live branch `rebalanced-pp` was updated 2026-10-25). Its public submodules are
|
||
initialized: `beatleader-server-models`, `beatleader-parser`, `ReplayDecoder`,
|
||
`MapPostprocessor`, `RatingAPI`. Two submodules are **private and
|
||
unreachable**: `beatleader-server-private` and `bs-simulator`. Upstream
|
||
already copes with that. `Startup.cs` falls back to `ReplayRecalculatorStub`,
|
||
and nothing in the public code imports a namespace that only the private
|
||
repos define. **Verified 2026-11-08:** it builds on Linux with nixpkgs
|
||
`dotnet-sdk_10` (10.0.403) with both private folders empty: 0 errors,
|
||
about 2,300 warnings, about 1 minute.
|
||
- Database: `UseSqlServer` for two contexts (`Startup.cs:380-386`):
|
||
`AppContext` (`DefaultConnection`) and `StorageContext`
|
||
(`StorageConnection`, which holds `PlayerLeaderboardStats`). There are 570
|
||
SQL Server migrations.
|
||
- `Z.EntityFramework.Extensions.EFCore` is commercial, with a monthly trial.
|
||
Call sites: about 92 `BulkSaveChanges`, 56 `BulkUpdate`, 19 `BulkDelete`,
|
||
3 `BulkInsert`.
|
||
- Raw SQL in 5 places: `DBCC CHECKIDENT` (`Services/DailyRefresh.cs`),
|
||
`TABLESAMPLE` (`ScoreController.GetRandomScore`), `TRUNCATE` and admin
|
||
statements (`Controllers/Admin/AdminController.cs`).
|
||
- Storage: all 31 helpers in `Utils/S3Helper.cs` go to Cloudflare R2
|
||
(`S3AccountID.r2.cloudflarestorage.com`) and return
|
||
`https://cdn.{container}.beatleader.com/{file}`. Containers: `replays`,
|
||
`otherreplays`, `assets`, `previews`, `scorestats`, `playlists`, `unicode`,
|
||
`configs`, `replayedvalues`, `reepresets`, `songs`, `attempts`. Nothing calls
|
||
S3 outside this file.
|
||
- Background jobs (`Startup.cs:455-478`): `EveryMinuteRefresh`,
|
||
`MinuteRefresh`, `HourlyRefresh`, `DailyRefresh`, `HistoryService`,
|
||
`BotService`, `RankingService`, `SongService`, `RichPresenceService` and
|
||
`MapRefresh` run only when `ServicesHost=YES`. These always run:
|
||
`ConstantsService`, `RefreshTaskService`, `ClanTaskService`,
|
||
`ClanMessageService`, `ClanCallbackService`, `SearchService` (Lucene),
|
||
`OauthService`, `LeaderboardPlayerStatsService`.
|
||
- Config keys, all from `IConfiguration`: `DefaultConnection`,
|
||
`StorageConnection`, `S3*`, `MapsPath` (default `/home/maps`), `SteamKey`,
|
||
`SteamApi`, `BeatSaverId/Secret`, `Openiddict*` (production expects
|
||
`../keys/openiddict_certificate*.pfx`; development uses dev certs), plus
|
||
optional OAuth secrets for Patreon, Twitch, Discord and others. Those
|
||
providers register only when their keys are set.
|
||
- The star-rating models are public: `RatingAPI/` (C# with ONNX models) and
|
||
`beatleader-analyzer`. We don't need them for the minimum version, because
|
||
we keep upstream's ratings. They are what would let a later replacement rate
|
||
maps itself.
|
||
|
||
### Upstream website
|
||
|
||
`~/src/BeatLeader/beatleader-website` (2026-11-07). In
|
||
`src/network/queues/beatleader/api-queue.js`, on any host it doesn't recognize
|
||
the API base is **`/cors/blapi/`**, which is relative to the page. So our host
|
||
only needs Caddy to proxy `/cors/blapi/*` to the API, with no code change. It
|
||
rewrites assets to `/cors/cdn-assets-bl/` and `/cors/cdnbeatsaver/` in the same
|
||
way. `BL_REPLAYS_URL` and `BL_ANALYZER_URL` fall back to the public
|
||
`replay.beatleader.com` and need a patch. Build with `npm run build` (rollup)
|
||
into `public/`.
|
||
|
||
### ArcViewer
|
||
|
||
`~/src/AllPoland/ArcViewer` (Unity 6000.0.67f1). The `deploy` branch on GitHub
|
||
holds a ready WebGL site (`index.html`, `Build/`, `TemplateData/`). It loads
|
||
replays from `?replayURL=<url>` and recognizes ScoreSaber replays by their
|
||
magic header (`ScoreSaberDecoder.IsScoreSaberReplay`, including the legacy
|
||
format), so it works for both `.bsor` and `.dat`. The map comes from `id=<bsr>`
|
||
(BeatSaver) or `url=<zip>`. Serving it from the same origin as the replays
|
||
avoids CORS and mixed-content problems.
|
||
|
||
### incineroar
|
||
|
||
- Caddy vhosts are in `~/ops/nix-pleb/systems/brenise.dev/incineroar/workstation.nix`.
|
||
They follow the pattern `"${host_fqdn}:44xx"` with `${tlsConfig}` (Namecheap
|
||
DNS-01) and a `remote_ip` allowlist (`127.0.0.1/32 ::1/128 192.168.1.34/32
|
||
192.168.1.35/32 192.168.1.36/32 10.19.21.34/32`). The host firewall is off;
|
||
Caddy does the gating.
|
||
- Disk: `/` has 479 GB free and `/home` 684 GB. The archive should be 10–25 GB:
|
||
about 4.6 GB of best-score replays, about 1.4 GB of ScoreSaber replays, and BeatSaver zips for
|
||
roughly 2,000–3,000 played maps (ScoreSaber alone has 1,474 unique hashes),
|
||
at a few MB each. Actual (2026-11-08): 21 GB with 1,897 zips at about 8 MB
|
||
each. Fetching the remaining ~630 older maps later adds about 5 GB.
|
||
- Allowed clients: incineroar (`127.0.0.1/32`, `::1/128`, `192.168.1.36/32`) and
|
||
snorlax (`192.168.254.21/32`, the address the amelia fallback vhost already
|
||
allows).
|
||
- Timezone: `America/Los_Angeles`.
|
||
- nix-pleb rule: put host app config in an existing `<system>/<purpose>.nix`
|
||
(here `workstation.nix`), not a new module file. Never use `nixos-rebuild
|
||
switch` without asking; use `build`. Never decrypt sops secrets: ask the user
|
||
to add the values.
|
||
|
||
## Layout
|
||
|
||
```
|
||
plebsaber.stream/ (this repo)
|
||
docs/plan.md this file
|
||
scripts/archive/ exporter and sync CLI (TypeScript, node ≥ 22)
|
||
src/lib/server/beatleader.ts extend: bearer auth from the sync's token store, paging, time_from, raw passthrough
|
||
src/lib/server/scoresaber.ts extend: player scores, replay download
|
||
|
||
gitea pleb/beatleader-server fork, branch `pleb/main` (based on upstream `rebalanced-pp`)
|
||
MIRROR.md what MirrorMode changes and how to run it
|
||
appsettings.Mirror.json incineroar values (ASPNETCORE_ENVIRONMENT=Mirror)
|
||
Tools/ArchiveImporter/ C# console app sharing AppContext/StorageContext
|
||
gitea pleb/beatleader-server-models fork of the models submodule, branch `pleb/main`
|
||
gitea pleb/beatleader-website fork, branch `pleb/main`
|
||
|
||
/var/lib/plebleader/ on incineroar, user `plebleader`
|
||
archive/ source of truth, append-only
|
||
beatleader/
|
||
user.json latest /user (no secrets)
|
||
players/<id>/profile/<YYYY-MM-DD>.json
|
||
players/<id>/history.json, triangle.json, accgraph.json
|
||
scores/<scoreId>.json one ScoreResponse each, unmodified
|
||
scorestats/<scoreId>.json
|
||
leaderboards/<hash>/<YYYY-MM-DD>.json new snapshot only when the content changed
|
||
replays/<original filename>.bsor
|
||
scoresaber/<playerId>/
|
||
player/<YYYY-MM-DD>.json
|
||
scores/<scoreId>.json the full item (score plus leaderboard)
|
||
replays/<scoreId>.dat
|
||
beatsaver/
|
||
maps/<hash>.json getMapByHash response, unmodified
|
||
zips/<hash>.zip the map version matching that hash
|
||
missing.json hashes BeatSaver no longer serves, with their source if recovered
|
||
game-levels/<hash>/ maps recovered from the Windows CustomLevels when BeatSaver lacks them
|
||
local-mods/{beatleader,scoresaber}/... copied from the Windows UserData
|
||
state.json sync cursors (last timepost per source/feed)
|
||
db/app.db, db/storage.db SQLite, rebuilt from the archive
|
||
storage/ S3 replacement (containers as dirs); replays/ and ssreplays/ symlink into archive/
|
||
maps/<hash>/ MapsPath: unpack cache, filled on demand from archive zips (phase 2, step 9)
|
||
auth/beatleader_tokens.json the sync's own OAuth token chain (rotates on refresh)
|
||
secrets → EnvironmentFile from sops BL_CLIENT_ID, BL_CLIENT_SECRET
|
||
```
|
||
|
||
## Phases
|
||
|
||
Each phase ends with a check. Phase 1 protects the data and doesn't depend on
|
||
the fork, so do it first.
|
||
|
||
### Phase 0: Prep
|
||
|
||
1. **Done 2026-11-08** (all were already current; upstream server `master`
|
||
`1474312`, `rebalanced-pp` `52a50ed`, website `2b3ef891`, ArcViewer `deploy`
|
||
`c776256`, ScoreSaber website `836d789`). Update the checkouts:
|
||
`git -C ~/src/BeatLeader/beatleader-server pull --recurse-submodules`, and
|
||
the same for `beatleader-website`, `~/src/AllPoland/ArcViewer` (also
|
||
`git fetch origin deploy`), and `~/src/ScoreSaber/website`. The ScoreSaber
|
||
website is public now; use it to check the score and replay API shapes.
|
||
2. **Done 2026-11-08.** Create the Gitea forks `pleb/beatleader-server` and `pleb/beatleader-website`
|
||
with the `gitea-api` skill (mirror or push). Keep `upstream` as a remote so
|
||
you can rebase later.
|
||
Created as private, non-mirror imports through `POST /repos/migrate` (the
|
||
Gitea token lacks `write:user`, which `POST /user/repos` needs), with
|
||
default branch `pleb/main`. The `~/src` checkouts keep `origin` = GitHub and
|
||
gained a `gitea` remote; `pleb/main` is checked out there. The server's
|
||
`pleb/main` starts from `rebalanced-pp`, which is `master` plus 4 commits
|
||
and what the live API most likely runs, so the entities match the archived
|
||
responses. `pleb/beatleader-server-models` was forked too (see phase 2,
|
||
step 7); the server's `.gitmodules` points at it over SSH.
|
||
3. Bring this repo's flake up to date (it's still on `nixos-25.05`) only if the
|
||
exporter gets packaged here.
|
||
4. **Done 2026-11-09** (`systems/brenise.dev/incineroar/plebleader.env` in
|
||
nix-pleb, dotenv format, encrypted from this repo's `.env` without
|
||
decrypting anything; wired up on the `plebleader` branch).
|
||
Ask the user to add a sops secret `plebleader.env` for incineroar, holding
|
||
`BL_CLIENT_ID` and `BL_CLIENT_SECRET`. These are the same OAuth app as
|
||
plebsaber.stream; check that its registered callback URLs include the one
|
||
used for seeding. Agents never decrypt it.
|
||
For local runs before deployment, use the gitignored `.env` in this repo.
|
||
5. **Done 2026-11-08** (`scripts/archive/cli.ts auth login`; refresh rotation
|
||
tested; chain expires about 2026-11-22 unless refreshed).
|
||
**Seed the sync's token chain.** Add a small `scripts/archive` command
|
||
(`auth login`) that runs the same authorization-code flow as
|
||
`/auth/beatleader/login` with `scope=profile offline_access`. It catches the
|
||
callback on a loopback port, or reuses a plebsaber.stream callback URL the
|
||
OAuth app already allows. It writes `auth/beatleader_tokens.json`. Don't
|
||
copy the plebsaber.stream site's own refresh token, because rotation would
|
||
break whichever side refreshes second.
|
||
|
||
### Phase 1: Full export (one-time backup)
|
||
|
||
**Scope (decided 2026-11-08):** all player data for both accounts, all
|
||
history: profiles, BL scores, replays, score stats, SS scores and replays,
|
||
plus BL leaderboard JSON (ratings) for every played hash. Only **BeatSaver
|
||
map files** are limited to maps played in the last two years
|
||
(`--maps-since 2024-11-08`, `DEFAULT_MAPS_SINCE` in `scripts/archive/config.ts`).
|
||
**Exception:** legacy ScoreSaber scores that have an archived replay get their
|
||
map fetched whatever its age, so those replays render (58 replays, 53 maps;
|
||
`mapHashes()` in `cli.ts`). Legacy scores without a replay don't get a map. Older map zips come later with `export maps --maps-since all`; existing files
|
||
are skipped, so that only fetches what's missing. Replays of older scores
|
||
won't render a map in ArcViewer until then. SS main has 1,034 replays (about
|
||
3 h at 10.5 s each). The 58 SS legacy replays are copied from ss-scraper's
|
||
download instead of being fetched again.
|
||
|
||
Commands: `node scripts/archive/cli.ts export bl|ss|ss-replays|maps|all
|
||
[--since …]` and `report`. The archive lives in `.data/plebleader/archive/`
|
||
(set `PLEBLEADER_DIR` to change the root) until `/var/lib/plebleader` exists
|
||
on incineroar; then move it. Logs go in `.data/plebleader/logs/`.
|
||
Profile, history, triangle and accgraph are stored as dated snapshots in
|
||
`players/<id>/<kind>/<YYYY-MM-DD>.json`, written only when changed.
|
||
|
||
Build `scripts/archive/` as a small CLI (`export bl`, `export ss`,
|
||
`export maps`, `import-local-mods`, `sync`). Rules:
|
||
|
||
- Store responses byte-for-byte. Never reshape them on the way in.
|
||
- Make every step resumable: skip files that already exist, keep cursors in
|
||
`state.json`, and write to temp files before renaming.
|
||
- Respect rate limits. Use about 4 requests per second to BeatLeader and honor
|
||
`Retry-After` (the existing `requestWith429Retry` handles 429s). For
|
||
ScoreSaber replays, stay under the 360/hour limit (10.5 s pacing) and honor
|
||
the `x-ratelimit-*` headers.
|
||
- Send `User-Agent: plebsaber-archive/0.1`. Don't put an email address in it.
|
||
|
||
Steps:
|
||
|
||
1. **Auth check.** `node scripts/archive/cli.ts auth status`: `/oauth2/identity`
|
||
must return `76561199407393962`, and `/leaderboards/hash/{hash}` must return
|
||
non-null `stars` for an unranked difficulty. If either fails, stop.
|
||
2. **BeatLeader scores.** Page through `/player/{id}/scores` with
|
||
`sortBy=date&order=desc&count=100&includeIO=true` until it's exhausted
|
||
(about 36 pages). Do the same for the legacy id (3 scores).
|
||
3. **Attempts: skipped.** Don't call `/scoresstats` or `/otherreplays`.
|
||
4. **Replays.** Fetch every `scores/*.replay` from the CDN (no auth, CORS `*`).
|
||
Keep the original filenames.
|
||
5. **Score stats.** `GET /score/statistic/{id}` for each best score.
|
||
6. **BeatLeader map data.** Collect the set of played hashes: every unique
|
||
`song.hash` in BL scores, plus `leaderboard.songHash` in both
|
||
ScoreSaber accounts. Normalize them to lowercase. For each, call
|
||
`GET /leaderboards/hash/{hash}` (no auth needed). That gives ratings and
|
||
the triangle for every difficulty, ranked or not. This is the **only**
|
||
source of unranked ratings; the scores from step 2 have them nulled.
|
||
7. **Map files from BeatSaver.** For each played hash, call
|
||
`BeatSaverAPI.getMapByHash(hash)` and save the response to
|
||
`beatsaver/maps/<hash>.json`. Then download the zip for the `versions[]`
|
||
entry whose `hash` matches, to `beatsaver/zips/<hash>.zip`. Extend
|
||
`beatsaver.ts` with a `downloadMapZip(version)` that streams to a temp file,
|
||
checks it's a valid zip containing `Info.dat`, then renames it. Go about 2
|
||
requests per second and honor 429s. Write each hash that can't be resolved
|
||
(deleted map, or an old version gone from the CDN) to `missing.json`.
|
||
8. **Recover missing maps from the game: optional, low priority.** Decided
|
||
2026-11-08: maps gone from BeatSaver are accepted as missing. They were most
|
||
likely updated or removed for a reason. The user's online backups and local
|
||
map archives from before 2026 don't have them. Older cold storage might, so
|
||
that's a "someday" check, not a task. The current BSManager instance's
|
||
`CustomLevels` may hold a few recent ones and is cheap to try: with
|
||
`mountwin`, hash every folder in `Beat Saber_Data/CustomLevels` and
|
||
`CustomWIPLevels`, and copy those whose hash is in `missing.json` to
|
||
`game-levels/<hash>/`. Whatever is still missing stays listed; those replays
|
||
just can't render the map.
|
||
9. **Unpack for the server: dropped 2026-11-08**, replaced by on-demand
|
||
unpacking (phase 2, step 9). ArcViewer loads `url=/cdn/maps/<hash>.zip`
|
||
(Caddy serves the archive's zips) instead of `id=<bsr>` from BeatSaver, so
|
||
it never needs unpacked folders.
|
||
10. **Profile and history.** Save `/player/{id}?stats=true`, `/history`,
|
||
`/history/triangle`, `/accgraph`, and `/user` (strip anything secret).
|
||
11. **ScoreSaber main.** Page through all scores (22 pages), then download the
|
||
1,034 replays. Either port the `download_replays.py` logic to TypeScript or
|
||
run it as-is with `-o archive/scoresaber`, and normalize the layout.
|
||
12. **ScoreSaber legacy.** Copy the 58 replays and the metadata from
|
||
`~/ops/beatsaber/ss-scraper/replays/76561197962107242/` into
|
||
`archive/scoresaber/76561197962107242/` (the filenames start with the
|
||
scoreId). Fetch the 199 score items again so the archive has complete,
|
||
unmodified responses.
|
||
13. **Local mod replays.** With the Windows partition mounted (`mountwin`, then
|
||
`umount ~/Windows` afterwards), copy `UserData/BeatLeader/Replays` and
|
||
`UserData/ScoreSaber/Replays` into `archive/local-mods/`. Afterwards, match
|
||
the ScoreSaber files against scores that have `hasReplay: false` by
|
||
player/hash/difficulty/characteristic.
|
||
|
||
**Check:** the counts match the live API: 3,565 BL scores, 3 legacy,
|
||
2,107 plus 199 SS scores, and replay files for every score with a
|
||
replay URL or `hasReplay`. Every played hash should have a BeatLeader
|
||
leaderboard snapshot, plus a BeatSaver zip or recovered game folder, unless
|
||
it's listed in `missing.json`. Write a `report` subcommand that prints counts
|
||
and what's missing, and paste its output into this file.
|
||
|
||
**Report, 2026-11-08 (phase 1 complete).** The ScoreSaber replays finished
|
||
with zero 429s.
|
||
|
||
```
|
||
Window: scores since all time, BeatSaver maps since 2024-11-08
|
||
BeatLeader scores 3568 in window (3568 archived total, both accounts)
|
||
BeatLeader replays 3568/3568
|
||
BeatLeader score stats 3568/3568
|
||
ScoreSaber 76561199407393962: 2107 scores, replays 1034/1034
|
||
ScoreSaber 76561197962107242: 199 scores, replays 58/58
|
||
Played hashes 2560
|
||
BL leaderboard snapshot 2552/2560
|
||
BeatSaver zip 1897/1926 in maps window (missing.json lists 29)
|
||
Archive size 21.78 GB
|
||
```
|
||
|
||
Steps 8 (game-folder recovery) and 13 (local mod replays) are still open.
|
||
Step 9 was dropped in favour of on-demand unpacking.
|
||
|
||
The 29 missing hashes are all "hash unknown to BeatSaver", and none belongs to
|
||
a legacy score with a replay (all 53 of those maps are archived). The 8 hashes
|
||
without a BL leaderboard are ScoreSaber-only maps BeatLeader never saw. Zips
|
||
average about 8 MB (15 GB for 1,897), more than the estimate below.
|
||
|
||
### Phase 2: Server fork on SQLite
|
||
|
||
**Done 2026-11-08** (`pleb/main` at `1741ce9`, models fork at `8f233e8`).
|
||
What differs from the steps below:
|
||
|
||
- Step 2: `Microsoft.Data.SqlClient` stays as a direct package, because one
|
||
controller catches `SqlException`. One controller had an unused
|
||
`using BeatLeader_Server.Migrations;` that broke the build once the old
|
||
migrations were gone. The fresh migrations create 155 tables and 217
|
||
indexes. WAL and migrations are applied at startup by `Utils/MirrorSetup.cs`;
|
||
`busy_timeout` comes from `Default Timeout=30` in the connection string.
|
||
- Step 3: `Utils/BulkCompat.cs` sits in the global namespace, like Z's own
|
||
extensions, because some callers have no EF `using`. 49 of the `BulkUpdate`
|
||
calls pass `ColumnInputExpression`, often on detached stubs that carry only
|
||
the key and those columns, so the shim marks only the listed columns
|
||
modified and detaches the stubs after saving. A plain `UpdateRange` would
|
||
have wiped every other column. **Not exercised at runtime yet**; the
|
||
importer will be the first real user.
|
||
- Step 4: the two index-defragmentation scripts became `PRAGMA optimize` and
|
||
`REINDEX; ANALYZE;`.
|
||
- Step 5: `GetS3Client` returns a `LocalStorageClient` (an `AmazonS3Client`
|
||
subclass) when `StoragePath` is set; the six S3 primitives branch on it.
|
||
Upstream behaviour is unchanged without `StoragePath`.
|
||
- Step 6: `MirrorMode` also skips `RefreshTaskService` and `MapRefresh`, and
|
||
the external OAuth providers. Two surprises showed up in testing:
|
||
`GetLazy` fetched unknown players from Steam (it now returns null, so 404),
|
||
and the BeatSaver login is registered unconditionally and validated on
|
||
every request (it gets placeholder credentials, otherwise every request
|
||
was a 500).
|
||
- Step 7: instead of patching about 35 `showRatings` decisions, the four
|
||
`HideRatings()` leaves return early when `MirrorSettings.ShowAllRatings`
|
||
(in the models fork) is set; `MirrorSetup` sets it. Sorting and filtering by
|
||
stars on unranked maps still follow the requesting user's settings; patch
|
||
`Utils/{Map,Leaderboard}ListUtils/Sort.cs` if that matters.
|
||
- Step 8: `appsettings.Mirror.json` loads with `ASPNETCORE_ENVIRONMENT=Mirror`;
|
||
`MIRROR.md` in the fork documents it. OpenIddict uses ephemeral keys.
|
||
Run the DLL with `--contentRoot <bin>` from a working directory in the state
|
||
dir, because data protection keys go to `../keys/`. `dotnet run` ignores
|
||
`ASPNETCORE_URLS` in favour of `launchSettings.json`.
|
||
|
||
Check result: on empty files, `/swagger/index.html` 200,
|
||
`/player/76561199407393962` 404, `/players` and `/leaderboards` 200 with empty
|
||
pages, no `fail:` lines in the log.
|
||
|
||
Work on the `pleb/main` branch of `pleb/beatleader-server`. Keep each change
|
||
small and in its own commit, so upstream rebases stay manageable.
|
||
|
||
1. **Build on Linux** before changing anything:
|
||
`nix shell nixpkgs#dotnet-sdk_10 -c dotnet build BeatLeader-Server.csproj`
|
||
(nixpkgs has SDK 10.0.403). Record which errors come from the missing
|
||
private submodules versus the platform. Delete or exclude the empty
|
||
`beatleader-server-private` and `bs-simulator` folders, and stub whatever
|
||
they turn out to supply.
|
||
2. **SQLite.** Replace `Microsoft.EntityFrameworkCore.SqlServer` with
|
||
`...Sqlite`, and point both `AddDbContextFactory` calls at `UseSqlite`
|
||
(`Data Source=/var/lib/plebleader/db/app.db` and `storage.db`). On
|
||
connection open, set `PRAGMA journal_mode=WAL; busy_timeout=5000;`. Move the
|
||
570 SQL Server migrations aside and generate one fresh migration per context
|
||
(`dotnet ef migrations add InitialSqlite --context AppContext`, and the same
|
||
for `StorageContext`), so later schema changes still work.
|
||
3. **Drop Z.EntityFramework.Extensions.** Add `Utils/BulkCompat.cs` with
|
||
extension methods of the same names and shapes: `BulkSaveChanges` →
|
||
`SaveChanges`, `BulkUpdate` → `UpdateRange` + `SaveChanges`, `BulkDelete` →
|
||
`RemoveRange` + `SaveChanges`, `BulkInsert` → `AddRange` + `SaveChanges`.
|
||
With one player, performance doesn't matter, and about 170 call sites keep
|
||
compiling unchanged.
|
||
4. **Raw SQL.** `DBCC CHECKIDENT` → `DELETE FROM sqlite_sequence WHERE name IN (...)`.
|
||
`TABLESAMPLE` → `ORDER BY RANDOM() LIMIT 1`. `TRUNCATE` → `DELETE FROM`.
|
||
5. **Storage.** Rewrite the bodies of the 31 `S3Helper` methods to read and
|
||
write `${StoragePath}/{container}/{filename}`, and return
|
||
`${PublicBaseUrl}/cdn/{container}/{filename}`. `GetS3Client` can return a
|
||
client that's never used, so the method signatures don't change.
|
||
6. **Background jobs.** Leave `ServicesHost` unset. Add a config flag
|
||
(`MirrorMode=true`) that also skips registering `RankingService`, the clan
|
||
services, `OauthService` and `LeaderboardPlayerStatsService`. Keep
|
||
`SearchService` and `ConstantsService` if they start cleanly, and point
|
||
Lucene's index directory at the state dir.
|
||
7. **Auth.** For the minimum version the site is read-only behind Caddy's IP
|
||
allowlist. No login is needed: the website shows a public profile. If the
|
||
ratings in API responses still depend on `ShowAllRatings` for the
|
||
*requesting* user, then in `MirrorMode` treat requests as coming from
|
||
`76561199407393962`, or just store ratings so they always show. That means
|
||
patching the `showRatings` checks, which all read
|
||
`ProfileSettings.ShowAllRatings`, to `true`.
|
||
8. **Configuration.** Put an `appsettings.Production.json` template in the fork
|
||
and the real values in the NixOS unit's environment (`ASPNETCORE_URLS=http://127.0.0.1:8042`,
|
||
`ConnectionStrings`/`DefaultConnection`, `StoragePath`, `PublicBaseUrl`,
|
||
`MapsPath`, `MirrorMode`). Use development OpenIddict certs, or generate a
|
||
self-signed pfx into the state dir.
|
||
|
||
9. **On-demand map unpacking (deferred; do it only when needed).**
|
||
`MapDownloader.Map(hash)` (in the `MapPostprocessor` submodule) returns
|
||
`MapsPath/<hash>/` when it exists and otherwise downloads from BeatSaver.
|
||
Add a step before the download: if `<MapZipsPath>/<hash>.zip` exists
|
||
(the archive's `beatsaver/zips/`), extract it to a temp folder under
|
||
`MapsPath`, check it contains `Info.dat`, rename it to `<hash>/` and return
|
||
that. Accept upper- and lowercase hashes, like `Map` does. In `MirrorMode`,
|
||
never fall through to BeatSaver: return null instead. That makes
|
||
`MapsPath` a cache that can be deleted at any time.
|
||
- **Technical debt: low.** It's about 20 lines and changes no
|
||
interfaces. The catch is where it lives: `MapDownloader` is in
|
||
`MapPostprocessor`, so editing it means a fourth fork. Its methods aren't
|
||
virtual, so they can't be overridden. Without a fork, add a small
|
||
`Utils/LocalMapZips.EnsureUnpacked(hash)` in the server and call it
|
||
before `_downloader.Map(...)` at the three call sites
|
||
(`ReplayController`, `EarthDayController`, `MapRefresh`). That's a few
|
||
extra lines per rebase and no submodule fork, so prefer it.
|
||
- **Why deferred:** no read path of the mirror calls `MapDownloader` today.
|
||
Its callers are `MapRefresh` (off in `MirrorMode`), the Earth Day event
|
||
controller, and replay upload and recalculation in `ReplayController`.
|
||
It becomes necessary if the mirror ever recalculates score stats
|
||
(for example the optional `.dat` → `.bsor` conversion in phase 5) or
|
||
accepts replay uploads.
|
||
|
||
**Check:** the server starts on an empty SQLite file, `/swagger/index.html`
|
||
loads, and `GET /player/76561199407393962` returns 404 rather than 500.
|
||
|
||
### Phase 3: Importer
|
||
|
||
**Importer done 2026-11-09** (`pleb/main` at `e3f991c`, pushed to Gitea). Usage
|
||
and the full file → table mapping are in the fork's `MIRROR.md`. A full import
|
||
takes about 11 s, and an unchanged rerun writes 0 rows. What differs from the
|
||
steps below:
|
||
|
||
- The archived responses are deserialized straight into the entity types,
|
||
because their property names match. Only scalar columns are copied onto the
|
||
tracked rows; keys, foreign keys and navigations are set explicitly. Leaderboard
|
||
snapshots are the main source for `Song`/`DifficultyDescription`. A score's
|
||
embedded leaderboard is a fallback only, and no score needed it.
|
||
- One-to-one children (`ScoreImprovement`, `ReplayOffsets`, `ModifiersRating`,
|
||
`DifficultyStatistics`, `ModifiersMap`, stats, settings) get local ids, so
|
||
they can't collide with upstream ids. Scores, difficulties, songs,
|
||
leaderboards, mappers and history rows keep upstream ids.
|
||
- `PlayerScoreStatsHistory` lives in **`StorageContext`**, so that database
|
||
isn't empty. The exporter had saved only 50 days of `/history` (the API
|
||
default); it now asks for `count=100000` (1,138 days for pleb). The triangle
|
||
needs those monthly points.
|
||
- No legacy `Player` row. An `AccountLink` (`PCOculusID` = legacy id) plus a
|
||
`// pleb:` patch in `PlayerIdToMain` resolves `/player/76561197962107242`
|
||
to pleb. Upstream only checks Steam-range ids against Oculus links. The
|
||
legacy account's 3 BL scores are skipped until phase 5.
|
||
- `StoragePath/replays` is a symlink to the archive's `beatleader/replays`.
|
||
Score stats are copied to `StoragePath/scorestats/<scoreId>.json`.
|
||
- Clans aren't mirrored (`ClanOrder` is blank).
|
||
|
||
Imported: 2,552 songs, 10,104 difficulties, 10,098 leaderboards, 628 mappers,
|
||
3,565 scores, 3,568 score stats, 1,138 history rows. API check against the
|
||
archived upstream responses, on a scratch DB with `ASPNETCORE_ENVIRONMENT=Mirror`:
|
||
|
||
- `/player/{id}?stats=true` matches. The only differences are row ids,
|
||
clans, and `linkedIds`, which now lists the alt.
|
||
- `/player/{id}/scores?includeIO=true`: 3,565 total. Scores, improvements and
|
||
offsets match, and unranked difficulties now carry stars and the skill
|
||
triangle ratings.
|
||
- `/accgraph` and `/history/triangle` match. `/score/statistic/{id}` redirects
|
||
to the local file.
|
||
- `/leaderboard/{id}` and `/leaderboard/scores/{id}` first returned 500.
|
||
About 10 projections order `Player.Clans` by an expression over the outer
|
||
row's `ClanOrder`, which needs SQL `APPLY`, and SQLite has none. **Fixed**
|
||
in `e3f991c`: `Utils/MirrorQueryInterceptor.cs`, an EF query interceptor
|
||
registered only in MirrorMode, rewrites any ordering of `Player.Clans` to
|
||
ordering by clan id. One file instead of patching every call site; the
|
||
mirror has no clans, so the order doesn't matter. The leaderboard,
|
||
clan-ranking and context variants now return 200.
|
||
|
||
The check below (stock website) is still to do in phase 4.
|
||
|
||
`Tools/ArchiveImporter` lives in the fork, references the main project, and
|
||
uses its `AppContext`/`StorageContext`. Run it as a one-shot that is idempotent
|
||
and upserts by upstream id.
|
||
|
||
- `beatleader/scores/*.json` (ScoreResponse) → `Song`,
|
||
`DifficultyDescription` (keep the upstream ratings and `ModifiersRating` as
|
||
they are), `Leaderboard`, `Score` (with upstream `Id`, `Rank`, `Pp`,
|
||
`Weight`, and `Timepost`), `ScoreImprovement`, `ReplayOffsets`. Rewrite
|
||
`Replay` to the local `/cdn/replays/<file>`.
|
||
- `leaderboards/<hash>/latest` updates the difficulty ratings, so maps you
|
||
played but have no best score for still get them.
|
||
- `beatsaver/maps/<hash>.json` fills in `Song` fields that BeatLeader leaves
|
||
out (BeatSaver key, description, tags, upload date). The server
|
||
doesn't read map files on any mirror path; if one appears, phase 2, step 9
|
||
unpacks them on demand.
|
||
- `players/<id>/profile/latest` → `Player`, `PlayerScoreStats`,
|
||
`ProfileSettings`, plus history rows from `/history`. Insert the legacy id as
|
||
an `AccountLink` or alt pointing to the main player, so `PlayerIdToMain`
|
||
resolves `/u/76561197962107242` to `pleb`. Its 3 BL scores go through the
|
||
phase 5 merge like any other candidate.
|
||
- `StorageContext` holds `PlayerScoreStatsHistory`; its attempts tables stay empty, because attempts aren't exported.
|
||
- `scorestats/*.json` → the `scorestats` storage container, using the filename
|
||
the server expects (check `ScoreController.GetStatistic`).
|
||
|
||
The response DTOs aren't the entities, and this mapping is the largest single
|
||
piece of work. Use `beatleader-server-models/Models/*` (entities) and
|
||
`ResponseUtils.cs` (entity → DTO) as the reference, working backwards.
|
||
|
||
**Check:** the stock website (phase 4) shows my profile with the right score
|
||
count, the pp and ranks from upstream, unranked star ratings, and the triangle
|
||
on the profile.
|
||
|
||
### Phase 4: Website, viewer, Caddy
|
||
|
||
**Steps 1–2 done and both checks passed 2026-11-08**, on a dev stand-in for
|
||
the vhost: the importer into `.data/plebleader/dev/db/`, the API on 8042
|
||
(`PublicBaseUrl=http://127.0.0.1:8890`), and an ad-hoc `nix shell
|
||
nixpkgs#caddy` on `http://127.0.0.1:8890` with the routes below
|
||
(`.data/plebleader/dev/{run-api.sh,Caddyfile}`). Use `127.0.0.1`, not
|
||
`localhost`: the website sends `localhost` to the real upstream API. Commits:
|
||
server `7cc964a`, website `f5194ee3`, `9ebe84ea`, `8d540642`, all local
|
||
(not pushed yet). Step 3 (nix-pleb) is still open.
|
||
|
||
- **Phase 3 check passed.** `/u/pleb`: 11,608.35pp, #3,861 global, US #1,643,
|
||
672 ranked and 3,565 total scores, unranked difficulties with stars (8.33★
|
||
on the Phase 1 test map), the skill triangle with monthly history. The
|
||
leaderboard page and `/score/{id}` (accuracy graphs from local score stats)
|
||
work too.
|
||
- **Server fix:** the profile page calls `GET /v2/player/{id}/hydrate`, which
|
||
the live API has and the public source doesn't (404, "Player not found").
|
||
`Controllers/MirrorHydrateController.cs` rebuilds it from
|
||
`PlayerController.Get` plus the `profile` object that the website's
|
||
`process-hydrate.js` reads.
|
||
- **Server run note:** `dotnet build` doesn't copy `wwwroot/` into `bin/`.
|
||
Without it `PreviewController`'s constructor throws on `/images/Star.png`,
|
||
and any controller that depends on it (`/user`, `/user/config`) returns 500.
|
||
The dev run passes `--webroot <src>/wwwroot`; the Nix package uses `dotnet
|
||
publish`, which copies it.
|
||
- **Website, as built:** `IS_MIRROR` in `api-queue.js` is true when the API
|
||
resolves to `/cors/blapi/`. In that case `BL_REPLAYS_URL` is `/watch/` and
|
||
score stats come from `/cdn/scorestats/`. Every hardcoded
|
||
`allpoland.github.io/ArcViewer/` link points at `/watch/` (except the clan
|
||
page's embed). `public/watch/index.html` takes BL replay-viewer or ArcViewer
|
||
parameters (`link`, `replayURL`, `scoreId`, `id`, optional `hash`) and
|
||
redirects to `/arcviewer/?replayURL=…&url=/cdn/maps/<hash>.zip&noProxy=true`.
|
||
It reads the hash from the replay filename
|
||
(`<score>-<player>-<diff>-<mode>-<HASH>.bsor`); for `scoreId` it first asks
|
||
`/cors/blapi/score/{id}` for the replay URL. Phase 5 can pass `hash=` for
|
||
`.dat` replays. The analyzer icons are off by default (an upstream Patreon
|
||
service). Clans, events, the socket page and the star-rating graph are still
|
||
visible; they just show nothing.
|
||
- **ArcViewer facts:** without `noProxy=true` it routes every non-whitelisted
|
||
URL through `cors.bsmg.dev`, which can't reach a LAN host. When both `id`
|
||
and `url` are given, `id` wins, so never send both. With no map parameter
|
||
it looks up the replay's own hash on BeatSaver. The `deploy` build ships
|
||
uncompressed `Build/` files, so no `Content-Encoding` headers are needed.
|
||
Its own calls to `api.beatleader.com` (player card, leaderboard info) fail
|
||
from this origin, so the player card stays blank; fixing that needs a Unity
|
||
build, so it's left as is.
|
||
- **Map zips:** BeatSaver's CDN serves `https://r2cdn.beatsaver.com/<hash>.zip`
|
||
with `Access-Control-Allow-Origin: *` (it rejects HEAD, use GET). Caddy
|
||
redirects to it when the archive has no zip, so the website always links
|
||
the local path and the ~630 older maps outside `--maps-since` still play.
|
||
- **Nix:** the website fork's `flake.nix` builds `packages.default` (yarn
|
||
v1 offline, ~16 s; deps hash `sha256-ThGvMJfU…`). Its `postPatch` turns the
|
||
themes' Google Fonts `@import url(...)` into `@import (css) url(...)`,
|
||
because less would fetch them at build time. ArcViewer `deploy` at
|
||
`c776256497b66f7c91a74162cfcd943b0f45ee2e` is
|
||
`sha256-4A+aVnFrKyWQjFnH2N/CwoVIlwrcgV2fYYrtaOiodo0=` for `fetchFromGitHub`.
|
||
|
||
Step 3 was done together with phase 7 (see there): the vhost below is in
|
||
nix-pleb, waiting for the switch.
|
||
|
||
1. In the website fork, point `BL_REPLAYS_URL` (and the "watch replay" links)
|
||
at `/arcviewer/?replayURL=<encoded local replay url>&url=<encoded /cdn/maps/<hash>.zip>`.
|
||
Fall back to `id=<bsr>` only for hashes that have no local zip. Hide or stub
|
||
the features that can't work offline: clans, events, friends, the Socket
|
||
page, and the stage analyzer and `ppai2` calls in `StarRatingGraph.svelte` /
|
||
`star-generator.js`. Leave the `/cors/blapi/` resolution as it is.
|
||
2. Package the ArcViewer `deploy` branch with `fetchFromGitHub` at a pinned
|
||
revision, as static files.
|
||
3. Add a Caddy vhost in `incineroar/workstation.nix`:
|
||
|
||
```
|
||
"${host_fqdn}:4442" = {
|
||
# Personal BeatLeader mirror (plebleader). Website + ArcViewer static,
|
||
# API on 8042, replays/assets/maps from the state dir.
|
||
# Only incineroar itself and snorlax (192.168.254.21).
|
||
extraConfig = ''
|
||
${tlsConfig}
|
||
@plebleader_allowed remote_ip 127.0.0.1/32 ::1/128 192.168.1.36/32 192.168.254.21/32
|
||
handle @plebleader_allowed {
|
||
handle_path /cors/blapi/* { reverse_proxy http://127.0.0.1:8042 }
|
||
handle_path /cdn/maps/* {
|
||
root * /var/lib/plebleader/archive/beatsaver/zips
|
||
@nozip not file
|
||
redir @nozip https://r2cdn.beatsaver.com{path} 302
|
||
file_server
|
||
}
|
||
handle_path /cdn/* { root * /var/lib/plebleader/storage; file_server }
|
||
handle_path /arcviewer/* { root * ${arcviewer}; file_server }
|
||
handle_path /cors/cdn-assets-bl/* { reverse_proxy https://cdn.assets.beatleader.com { header_up Host {upstream_hostport} } }
|
||
handle_path /cors/cdnbeatsaver/* { reverse_proxy https://cdn.beatsaver.com { header_up Host {upstream_hostport} } }
|
||
# {path}/index.html serves /watch/; without it the SPA's 404 answers
|
||
handle { root * ${plebleader-website}; try_files {path} {path}/index.html /index.html; file_server }
|
||
}
|
||
handle { respond "Forbidden" 403 }
|
||
'';
|
||
};
|
||
```
|
||
|
||
Checked: the `deploy` branch ships uncompressed `Build/` files, so no
|
||
`Content-Encoding` headers are needed.
|
||
|
||
**Check:** in Chromium, `https://incineroar.brenise.dev:4442/u/pleb` shows the
|
||
profile and scores, and a replay opens in ArcViewer.
|
||
|
||
### Phase 5: Convert ScoreSaber scores and merge
|
||
|
||
**Built and checked on a scratch copy 2026-11-09; committed, deploy
|
||
pending.** Server `50f7df6` (`Tools/ArchiveImporter/ScoreMerge.cs`, rules in
|
||
the fork's `MIRROR.md`), website `5522f245` (SS badge, `/watch/` for `.dat`),
|
||
exporter `ssMapHash` in this repo. All pushed. Result on the local archive: 3,710 leaderboards with one score
|
||
each (3,566 BL + 141 SS-only BL leaderboards + 3 DLC levels BL lacks). Winners: 3,564 BL main,
|
||
1 BL legacy, 139 SS legacy, 6 SS main; 2,022 SS scores were the same play as
|
||
a BL score and dropped; 20 BL scores got an improvement recomputed against an
|
||
earlier SS play. An unchanged rerun writes 0 rows. What differs from the
|
||
steps below:
|
||
|
||
- Step 1: **no new columns.** Provenance goes in `Platform`
|
||
(`scoresaber,<SS player id>`), and `Id` ≥ 1e9 marks SS scores. Columns
|
||
would have needed a models-fork change, a migration, and edits to the ~90
|
||
score projections before the website saw them; `Platform` already reaches
|
||
every score response.
|
||
- Step 2: OST levels are `ost_<LevelId>` on SS and `<levelid>` on BL; 5 of 8
|
||
resolve that way. Me Porto Bonito, Nuthin' But A "G" Thang and Stressed Out
|
||
(3 scores) aren't on BL (404), so the importer builds a song and leaderboard
|
||
from the SS item (`<LevelId><diff><mode>` ids, no ratings, accuracy 0).
|
||
The exporter now maps `ost_` hashes too (`scoresaber.ts` `ssMapHash`).
|
||
HMD and controller come from SS `deviceHmd`/`deviceControllerLeft`. SS
|
||
replays are served from `storage/ssreplays` → `archive/scoresaber`, so the
|
||
URL is `/cdn/ssreplays/<player>/replays/<id>.dat`.
|
||
- **Pp isn't upstream's.** Neither the fork's `rebalanced-pp` formula
|
||
(median 0.94x, range 0.69–1.16x) nor upstream `master`'s (0.81x)
|
||
reproduces the pp on archived BL scores; live BL's code is private. So the
|
||
fold compares every candidate by **locally computed** pp, which keeps the
|
||
order right, and BL winners keep upstream's pp. The 70 SS winners on
|
||
ranked maps show local pp, and improvement pp deltas are local on both
|
||
sides. Matching pp first sent two BL scores (KillerToy, Make Me) to worse
|
||
SS plays.
|
||
- Step 3: same-play duplicates (equal base score, ≤ 2 min apart; observed
|
||
−30 s…+9 s) keep the BL copy. Each upstream `scoreImprovement` is turned
|
||
back into the best it replaced and added as a candidate, so upstream's
|
||
improvement stays exactly when that rebuilt best is also the fold's previous
|
||
best. Older archived BL bests are candidates too.
|
||
- Step 5: the badge sits next to the rank in both `ScoreRank.svelte` (rank 0
|
||
already renders "-"). `/watch/` takes the map hash from the score
|
||
(`/score/{id}` → `song.hash`) for `.dat` replays, from `?scoreId=` or
|
||
1e9 + the id in the filename. Expanding an SS row shows its leaderboard and
|
||
no stats tab; the page logs a JSON error for the 404 score stats, harmless.
|
||
- Check: The Goddess Figure (Normal) is the case where a 2022 legacy SS
|
||
play beats the BL score; it shows with the SS badge, and the BL Hard score
|
||
there now reads "+76,821 vs 4 years ago" (vs the legacy SS play). An SS
|
||
replay redirects to ArcViewer with the `.dat` and local zip and loads the map;
|
||
watching it play still needs a real browser.
|
||
|
||
To deploy: nix-pleb bumps `plebleader-server`, `plebleader-website` and
|
||
`plebsaber-stream` (sync CLI); switch. The next `plebleader-sync`
|
||
run re-imports with the merge.
|
||
|
||
ScoreSaber scores become BeatLeader `Score` rows, and each leaderboard keeps
|
||
exactly one score. BeatLeader already shows how a score compares with the
|
||
previous best (`ScoreImprovement`), so after the merge that comparison covers
|
||
ScoreSaber plays too. This is importer logic in the fork. The archive keeps
|
||
the raw ScoreSaber responses as they are.
|
||
|
||
1. **Provenance columns.** Add nullable `Source` (`null` = BeatLeader,
|
||
`"scoresaber"`) and `SourceScoreId` / `SourcePlayerId` to `Score`. This goes
|
||
into the fresh SQLite migration from phase 2. The website uses them for a
|
||
small "SS" badge, and the importer uses them to stay idempotent.
|
||
2. **Convert** each ScoreSaber item (`scoresaber/*/scores/*.json`) into a
|
||
candidate `Score`:
|
||
- Leaderboard: `leaderboard.songHash` (lowercased), plus
|
||
`leaderboard.difficulty.difficulty`. ScoreSaber uses 1/3/5/7/9 for
|
||
Easy…ExpertPlus, the same numbers as BeatLeader's `difficulty.value`.
|
||
`gameMode` `SoloStandard` → mode `Standard` (strip `Solo`). Resolve it to
|
||
the BL leaderboard from `leaderboards/<hash>` (phase 1, step 6). If BL has
|
||
none, create `Song`/`DifficultyDescription`/`Leaderboard` from the
|
||
BeatSaver JSON, with no ratings.
|
||
- `BaseScore`, `ModifiedScore`, `Modifiers` (the same comma-separated codes),
|
||
`MissedNotes`, `BadCuts`, `FullCombo`, `MaxCombo`, `Timeset`/`Timepost`
|
||
come from `timeSet`. Set `Platform = "scoresaber"` and `Hmd` = unknown,
|
||
unless the codes are mapped.
|
||
- `Accuracy = BaseScore / maxScore`, taking `maxScore` from the **BL**
|
||
difficulty. ScoreSaber reports `maxScore: 0` on many unranked
|
||
leaderboards. Check this against how `ReplayUtils` computes BL accuracy.
|
||
- `Pp`: `ReplayUtils.PpFromScore(score, difficulty)`, using the archived BL
|
||
ratings, when the BL difficulty is ranked; otherwise 0, as BeatLeader
|
||
does. ScoreSaber's own pp and rank aren't comparable, so keep them only in
|
||
the archive. Set `Rank` to 0, and have the website show "—" for
|
||
ScoreSaber-sourced rows.
|
||
- `Id = 1_000_000_000 + ssScoreId`. ScoreSaber ids are about 95M and
|
||
upstream BL ids about 151M, so this can't collide and still fits in an
|
||
int32.
|
||
- `Replay = /cdn/ssreplays/<ssScoreId>.dat` when the archive has one (from
|
||
the server, or recovered from local mod replays); otherwise null. No
|
||
`ReplayOffsets` and no score stats, because the server can't decode `.dat`.
|
||
3. **Merge per leaderboard.** The candidates are the upstream BL best, the
|
||
ScoreSaber main best, the ScoreSaber legacy best, and the legacy BL score.
|
||
Sort them by time and fold them with BeatLeader's own
|
||
`ReplayUtils.IsNewScoreBetter(old, new)`: compare by pp when either score
|
||
has pp, otherwise by modified score, where a no-fail score loses to a clean
|
||
one. The winner becomes the leaderboard's `Score`.
|
||
- `ScoreImprovement` on the winner is the difference from the best before it
|
||
in the fold: score, accuracy, pp, misses, bad cuts, modifiers, timeset. If
|
||
the winner is the upstream BL score and nothing from ScoreSaber came
|
||
between it and its previous BL best, keep upstream's `scoreImprovement`
|
||
as it is.
|
||
- Candidates that lose stay in the archive only.
|
||
4. **Player totals.** Keep upstream's `PlayerScoreStats`, pp and rank. They
|
||
won't count ScoreSaber scores that won. Recomputing the weighted pp total
|
||
locally is an optional later step; label the profile numbers "BeatLeader,
|
||
as of <sync time>".
|
||
5. **Website.** Show the "SS" badge on ScoreSaber-sourced scores, hide the
|
||
accuracy graph and stats for them (there are no score stats), and send their
|
||
replay button to `/arcviewer/?replayURL=/cdn/ssreplays/<id>.dat&url=/cdn/maps/<hash>.zip`.
|
||
Every score shows under the name `pleb`; `i5u1d0r` survives only as
|
||
`SourcePlayerId`.
|
||
|
||
Optional later: convert `.dat` into `.bsor`, using ArcViewer's
|
||
`ScoreSaberDecoder` as the reference for the format. The server's statistics
|
||
code could then produce score stats for ScoreSaber-sourced scores too.
|
||
|
||
**Check:** the leaderboard count is the union of the BL and SS leaderboards I
|
||
played, with exactly one score each. The importer prints how many winners came
|
||
from each source. Spot-check a map where an old legacy ScoreSaber score beats
|
||
the BL score, and confirm the improvement display and the ArcViewer replay.
|
||
|
||
### Phase 6: Ongoing sync
|
||
|
||
**Built 2026-11-09; deployed on incineroar the same day (daily timer at
|
||
17:00, first run clean; see phase 7).** `node scripts/archive/cli.ts sync
|
||
[--ratings] [--no-ss-replays]` (this repo `d182fd8`), packaged as the
|
||
flake's `plebleader-archive`. Tested against the local archive: the token
|
||
refresh rotated the chain (now valid until about 2026-11-23), and with the
|
||
newest BL and SS score files set aside, the run fetched exactly those two
|
||
back, byte-identical, and re-snapshotted the BL leaderboard. What differs
|
||
from the steps below:
|
||
|
||
- Main account only; both legacy accounts are frozen. The incremental
|
||
window starts a day before the newest archived score (BL `timepost`, SS
|
||
`timeSet`), computed from the archive rather than `state.json`. Replays and
|
||
score stats are retried for every score in the window, not only new ones.
|
||
- New BL hashes get a refreshed leaderboard snapshot; new SS-only hashes get
|
||
one if they have none. Map zips cover every hash in the window.
|
||
- Ratings drift: Sundays by the machine's clock, or `--ratings`.
|
||
- Each run writes a `sync` section to `state.json` (new score ids, maps
|
||
without a zip). `exportScores` no longer resets `newestTimepost` to 0 on an
|
||
empty window.
|
||
- Step 8: the importer runs in full after every sync (about 10 s), as a
|
||
second `ExecStart` of `plebleader-sync`; no `--since` flag.
|
||
- **Importer fix** (server `dd94772`): the archive keeps an old best after
|
||
its improvement is synced. The importer now takes only the newest score
|
||
per leaderboard and removes rows superseded since the last import. Before
|
||
this, the second score would have broken the
|
||
`(PlayerId, LeaderboardId, ValidContexts)` unique index. Tested on a
|
||
scratch copy with a fake improved score: added, old one removed, and back
|
||
again once the fake was deleted.
|
||
|
||
The `sync` subcommand runs once a day from a systemd timer on incineroar:
|
||
`OnCalendar=*-*-* 17:00:00 America/Los_Angeles` with `Persistent=true`, so a
|
||
run missed while the machine was off happens at the next boot. By 5 pm the
|
||
day's play is usually over.
|
||
|
||
1. **BeatLeader scores:** `/player/{id}/scores?sortBy=date&order=desc&time_from=<cursor-1d>`.
|
||
Overlap a day and skip ids you already have. New score ids → save the score
|
||
JSON, the replay, the score stats, and a leaderboard refresh for that hash.
|
||
2. **Attempts:** skipped (see Decisions).
|
||
3. **New maps:** for every hash not in the archive yet, do phase 1, steps 6–9:
|
||
BeatLeader leaderboard, then the BeatSaver JSON and zip (nothing to unpack).
|
||
The game's own folders are only a manual fallback, since sync can't mount
|
||
Windows. List unresolved hashes in the run summary.
|
||
4. **Profile:** one dated snapshot per run, plus `/history`.
|
||
5. **Ratings drift:** on Sundays' run, re-fetch `/leaderboards/hash/{hash}` for
|
||
every map I've played, and store a new snapshot only when it changed.
|
||
Upstream rebalances ratings (see the `rebalanced-pp` branch), and keeping
|
||
the old snapshots preserves the history.
|
||
6. **ScoreSaber main:** `sort=recent` until reaching a known score id, then
|
||
download the new replays at the slow pace. A day's worth fits easily in one
|
||
run. Legacy is frozen; skip it.
|
||
7. **Token upkeep:** refresh the OAuth token at the start of each run and write
|
||
the rotated refresh token to `auth/beatleader_tokens.json` (temp file plus
|
||
rename) before doing anything else. A daily run stays far inside the 14-day
|
||
window. If the refresh fails, the unit fails loudly and the message says to
|
||
run `auth login` again. Alloy ships journal lines to Loki, so set up an
|
||
alert there.
|
||
8. Once the archive has changed, run the importer incrementally, through a
|
||
`--since` flag or by watching `state.json`. That includes the phase 5 merge
|
||
for every leaderboard that got a new BL or SS score.
|
||
|
||
Optional later: a manual `sync` run (`systemctl start plebleader-sync`) for
|
||
when I want results before 5 pm.
|
||
|
||
**Check:** play a map, and after the next 17:00 run the score, its replay and
|
||
the map file show up on the local site, and the replay opens in ArcViewer with
|
||
the local zip.
|
||
|
||
### Phase 7: NixOS deployment on incineroar
|
||
|
||
**Done 2026-11-09: merged into nix-pleb `main` (fast-forward to `3b3eb47`),
|
||
switched on incineroar, first sync clean, and BL `.bsor` replays play in
|
||
ArcViewer.** Issues hit after the switch:
|
||
|
||
- **Missing `/run/secrets/plebleader.env`.** The file was encrypted only to
|
||
litten's age key (the repo `.sops.yaml` rule lists only litten), so
|
||
incineroar's host key couldn't decrypt it. Fixed by adding incineroar's host
|
||
key and `age1kuxd…` with `sops rotate --add-age`. Any new incineroar sops
|
||
file needs the same three recipients as `sops.yaml`.
|
||
- **First sync (13:48):** the token refreshed, with the chain valid to
|
||
2026-11-23 (each daily refresh moves that forward), 0 new scores, and the
|
||
importer loaded 3565 scores and 2552 songs in 11 s. The token now lives only
|
||
in `/var/lib/plebleader/auth/`.
|
||
- **Website "TypeError: Cannot read properties of undefined (reading
|
||
'id')".** `utils/format.js` takes the first host label as the leaderboard
|
||
context when the host has more than 2 labels, so the context became
|
||
`incineroar` and the navbar's context lookup found nothing. The phase 4 dev
|
||
check missed it because a numeric last label (`127.0.0.1`) raises that limit
|
||
to 4. Website fix `e5f6279a`: non-BeatLeader hosts always use `general`.
|
||
- **ArcViewer "Download failed! HTTP/1.1 403 Forbidden"** for
|
||
`/cdn/replays/…bsor`, and `/cdn/maps/…` redirected to BeatSaver even for
|
||
archived zips. Caddy had been running since 2026-11-07. The switch added it
|
||
to group `plebleader` but only reloaded it, so the process still had just
|
||
`caddy` (`grep Groups /proc/<pid>/status`). Fixed with
|
||
`doas systemctl restart caddy`. Any later change to Caddy's groups needs a
|
||
restart too.
|
||
|
||
Phase 6 added a second commit on the
|
||
branch: the `plebleader.env` sops secret, `plebleader-sync` (sync, then the
|
||
importer) and its timer, and lock bumps for `plebsaber-stream` (`d182fd8`)
|
||
and `plebleader-server` (`dd94772`). The `plebsaber-stream` bump also
|
||
rebuilds beldum's site, with no change to the site code.
|
||
|
||
nix-pleb branch `plebleader`
|
||
(`2ada4b7`, in the worktree `~/ops/nix-pleb-plebleader`, based on `main`
|
||
`981f9e7`) adds the inputs `plebleader-server` (server `0481759`) and
|
||
`plebleader-website` (`8d540642`), both following `nixpkgs-unstable`, plus to
|
||
`incineroar/workstation.nix`: user and group `plebleader` (with `caddy` in
|
||
the group), `plebleader-api`, a one-shot `plebleader-import`, ArcViewer
|
||
`fetchFromGitHub` and the 4442 vhost. The toplevel builds, and the system's
|
||
Caddy adapts the generated Caddyfile. The sync unit and timer and the sops
|
||
secret wait for phase 6's `sync` command. What differs from the bullets
|
||
below:
|
||
|
||
- Server fork: the private upstream submodules (`beatleader-server-private`,
|
||
`bs-simulator`) are removed from `.gitmodules`, because a
|
||
submodule-aware fetch fails on them. The flake sets `self.submodules`; the
|
||
nix-pleb input uses `?ref=pleb/main&submodules=1` over SSH (the models fork's
|
||
scp-style URL fetches fine). The package's `version` is a fixed `0.1.0`:
|
||
`buildDotnetModule` hands it to MSBuild, and a git short rev made restore
|
||
fail with no message (`NU1105` only in `project.assets.json`). 336 NuGet
|
||
packages in `deps.json`. The package was tested by hand (importer and API
|
||
on a scratch DB, all endpoints 200, `wwwroot` included).
|
||
- Website: the yarn v1 flake from phase 4, not `buildNpmPackage`.
|
||
- Units: `WorkingDirectory=/var/lib/plebleader/app` (data protection keys go
|
||
to `../keys`), `StateDirectory` for `app`, `db`, `storage`, `maps`,
|
||
`StateDirectoryMode=0750` and `UMask=0027`, so Caddy reads through the group
|
||
and nothing is world-readable. Paths and `PublicBaseUrl` come from the
|
||
fork's `appsettings.Mirror.json`, so the units only set
|
||
`ASPNETCORE_ENVIRONMENT=Mirror`, `ASPNETCORE_URLS` and `HOME`.
|
||
|
||
To deploy (the user does this):
|
||
|
||
1. Merge `plebleader` into nix-pleb `main`. The main checkout has uncommitted
|
||
`flake.lock` changes, so commit or stash those first, or cherry-pick the
|
||
commit and rerun `nix flake lock` if `flake.lock` conflicts. Then
|
||
`sudo --preserve-env=SSH_AUTH_SOCK nixos-rebuild switch --flake .#incineroar`.
|
||
`plebleader-api` starts on empty databases.
|
||
2. Move the archive in (21 GB; `/home` and `/` are separate filesystems, so
|
||
this is a copy):
|
||
`doas cp -a ~/ops/beatsaber/plebsaber.stream/.data/plebleader/archive /var/lib/plebleader/`,
|
||
then `doas chown -R plebleader:plebleader /var/lib/plebleader/archive` and
|
||
`doas chmod -R g+rX,o-rwx /var/lib/plebleader/archive`.
|
||
3. Move the sync's token chain. **Before 2026-11-23**, or it expires and
|
||
needs `auth login` again:
|
||
`doas install -d -o plebleader -g plebleader -m 0700 /var/lib/plebleader/auth`
|
||
and
|
||
`doas install -o plebleader -g plebleader -m 0600 ~/ops/beatsaber/plebsaber.stream/.data/plebleader/auth/beatleader_tokens.json /var/lib/plebleader/auth/`,
|
||
then delete the local copy. From then on, never run `sync` or
|
||
`auth refresh` locally: the refresh token rotates, and whichever side
|
||
refreshes second breaks.
|
||
4. `doas systemctl start plebleader-sync` (sync, then import), then open
|
||
`https://incineroar.brenise.dev:4442/u/pleb` and a replay. Check
|
||
`journalctl -u plebleader-sync` and `systemctl list-timers plebleader-sync`.
|
||
5. Afterwards, remove the worktree with `git worktree remove ../nix-pleb-plebleader`.
|
||
|
||
- Packages: `buildDotnetModule` for the fork (generate `deps.json` with
|
||
`nix build .#…passthru.fetch-deps`), `buildNpmPackage` for the website (start
|
||
with `npmDepsHash = lib.fakeHash`, then pin the `got:` value), and
|
||
`fetchFromGitHub` for ArcViewer `deploy`. The forks expose their packages
|
||
through their own flakes and become flake inputs in nix-pleb, next to the
|
||
existing `plebsaber-stream` input. Use `git+https://gitea.satstack.dev/pleb/<repo>.git`.
|
||
- Services: put `plebleader-api`, `plebleader-sync` (service and timer) and the
|
||
Caddy vhost in `incineroar/workstation.nix`. Don't create a new module file.
|
||
Use a static user `plebleader`, `StateDirectory=plebleader`, and an
|
||
`EnvironmentFile` from the sops secret.
|
||
- Validate with
|
||
`nix build .#nixosConfigurations.incineroar.config.system.build.toplevel --no-link`.
|
||
The user does the switch.
|
||
|
||
### Phase 8: Self-hosted BeatLeader web replay viewer
|
||
|
||
Upstream, the score row's replay icon opens BeatLeader's web viewer
|
||
(`BL_REPLAYS_URL`, replay.beatleader.com), and turning on ArcViewer in the
|
||
score settings adds a separate `altReplay` icon (`Song/Icons.svelte`,
|
||
`webPlayerLink(…, 'arcviewer', …)`). Phase 4 pointed `BL_REPLAYS_URL` at
|
||
`/watch/`, so both icons open ArcViewer now. The user wants the upstream
|
||
behaviour back: the replay icon opens the BeatLeader viewer, and ArcViewer
|
||
stays a separate icon.
|
||
|
||
Source: `~/src/BeatLeader/BeatSaber-Web-Replays` (webpack 5, npm). Steps:
|
||
|
||
1. Fork it to gitea `pleb/BeatSaber-Web-Replays` with a `pleb/main` branch,
|
||
and change as little as possible.
|
||
2. Resolve `getApiUrl()` (`src/utils.js`, `src/open-replay-decoder.js`) to
|
||
`/cors/blapi` on non-BeatLeader hosts, the same rule as the website's
|
||
`IS_MIRROR`. Check how it fetches the replay file and the score (`link=`,
|
||
`scoreId=`) against the mirror's `/cdn/replays/` and `/cors/blapi/score/{id}`.
|
||
3. Maps: `zip-loader.js` tries `cdn.songs.beatleader.xyz`, then
|
||
`/cors/beat-saver2/api/maps/…` (a Netlify proxy), then `r2cdn`/`cdn.beatsaver.com`.
|
||
Try `/cdn/maps/<hash>.zip` first, which already falls back to r2cdn in Caddy.
|
||
Either proxy `/cors/beat-saver2/` to `api.beatsaver.com` in Caddy or skip it.
|
||
4. Drop or stub analytics (googletagmanager), and check its other runtime
|
||
fetches (fonts, `cdn.jsdelivr.net`). Discord and Patreon are plain links.
|
||
5. Add a flake with `buildNpmPackage` that outputs the built static files
|
||
(`index.html`, `assets`, `build`, `vendor`, as in its `predeploy` script),
|
||
and add it as a nix-pleb input.
|
||
6. Caddy: serve it at `/replay/` on the plebleader vhost.
|
||
7. Website: in mirror mode set `BL_REPLAYS_URL` to `/replay/` instead of
|
||
`/watch/`. Leave the explicit ArcViewer links (`altReplay`, `ReplayButton`
|
||
with `webPlayer == 'arcviewer'`) on `/watch/`.
|
||
|
||
**Check:** on `/u/pleb` the replay icon plays a score in the BeatLeader
|
||
viewer from local files, and with ArcViewer turned on in settings, the second
|
||
icon plays it in ArcViewer.
|
||
|
||
## Risks and fallbacks
|
||
|
||
- **The server doesn't build or run on Linux, or EF/SQLite can't translate
|
||
queries.** SQL Server is case-insensitive and SQLite isn't, and SQLite has no
|
||
`DateTimeOffset` ordering. Fix it endpoint by endpoint, but only for the
|
||
endpoints the profile, scores and leaderboard pages call (find them in the
|
||
network tab). Fallback: skip the fork and write a small read-only API in
|
||
this repo that serves the archive in the response shapes the website
|
||
expects. That would also be the first step toward the full replacement.
|
||
- **The OAuth refresh chain breaks** (more than 14 days without a run, a lost
|
||
write, or upstream auth changes). The archive keeps everything collected so
|
||
far. Sync stops, a failing unit makes that visible, and `auth login`
|
||
re-seeds the chain.
|
||
- **Maps disappear from BeatSaver.** That's why zips are archived as soon as a
|
||
hash first appears. Old versions that are already gone stay listed in
|
||
`missing.json`, which is acceptable (decided 2026-11-08). Cold storage might
|
||
hold some; check someday, without much hope.
|
||
- **Upstream API changes.** The archive stores raw responses, so a change
|
||
breaks the importer, not the data. Version the importer's mapping.
|
||
- **Upstream code moves fast** (the website is updated daily). Pin fork
|
||
revisions. Rebase only when there's a reason to.
|
||
- **Ranks and pp go stale locally.** That's by design: they're upstream's
|
||
values from the last sync. The UI could show "as of <sync time>".
|
||
|
||
## Open questions for the user
|
||
|
||
Settled on 2026-11-08: port 4442; access limited to incineroar and snorlax;
|
||
map data limited to played maps; OAuth for auth; sync daily at 17:00
|
||
America/Los_Angeles.
|
||
|
||
Also settled: skip attempts and their replays entirely. Where BL and SS scores
|
||
overlap, keep one per leaderboard by converting ScoreSaber scores to
|
||
BeatLeader scores (phase 5).
|
||
|
||
Nothing is open right now.
|
||
|
||
## Where the earlier work lives
|
||
|
||
- `~/ops/beatsaber/ss-scraper/`: ScoreSaber replay downloader. The legacy
|
||
account is done; it has no git commits yet.
|
||
- `~/ops/beatsaber/playlist-tool/src/clients/{beatleader,scoresaber,beatsaver}/`:
|
||
generated Python OpenAPI clients. Use them as a reference for the response
|
||
models.
|
||
- `~/ops/beatsaber/backups/`: UserData backups. They leave the replay
|
||
folders out.
|
||
- `~/ops/beatsaber/beatleader/beatleader-website.code-workspace`: points at
|
||
`~/src/beatleader/...` (lowercase). The checkouts are actually in
|
||
`~/src/BeatLeader/`, so fix that path.
|
||
- `~/src/BeatLeader/bl-dev-sandbox` is a BSOR v2 format experiment, not a dev
|
||
environment. Ignore it.
|