Files

T

plebandClaude Opus 5.5 3a32008ed6 Record the search index rebuild schedule in the plan

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

2026-11-09 15:14:07 -08:00

68 KiB

Raw Blame History

Plan: a personal BeatLeader on incineroar

Goal: run my own BeatLeader instance on incineroar that holds only my profile (pleb) and play history, with ScoreSaber history merged in, and keep it in sync with upstream BeatLeader. The minimum viable version stays a fork of upstream rather than a rewrite. Replacing BeatLeader with my own system is a later project. This plan makes sure the data I collect is ready for that.

Written 2026-11-08. The numbers under "Facts" were checked live on that date.

Decisions

Topic	Decision
Server	Fork `BeatLeader/beatleader-server` (ASP.NET Core, net10.0). Swap SQL Server for SQLite and Cloudflare R2 for the local filesystem. Turn off every background job that recalculates ranks or pp.
Frontend	Fork `BeatLeader/beatleader-website` (Svelte 5 + rollup). Change as little as possible.
Replay viewer	Self-host ArcViewer's prebuilt WebGL build (the `deploy` branch) for both BeatLeader (`.bsor`) and ScoreSaber (`.dat`) replays. No Unity build.
Source of truth	A raw, append-only archive of unmodified API responses and replay files. The SQLite database can always be rebuilt from it, and so can any future replacement system.
Exporter and sync	TypeScript in this repo, building on `src/lib/server/beatleader.ts`, `beatleaderAuth.ts`, `scoresaber.ts` and `beatsaver.ts`. It writes only to the archive.
BeatLeader auth	OAuth 2.0 (`profile offline_access`). The sync owns its own chain of refresh tokens (`scripts/archive/cli.ts auth login`, seeded 2026-11-08). Unranked ratings do not need it: they come from the anonymous `/leaderboards/hash/{hash}` (see Facts).
Map files	Archive the BeatSaver zip of every map I've played, using `beatsaver.ts`. Don't unpack them ahead of time: ArcViewer loads the zip directly, and the server unpacks on demand only if a mirror code path ever needs map files (phase 2, step 9). Neither ever needs BeatSaver.
Sync schedule	Daily at 17:00 America/Los_Angeles, incineroar's timezone.
Access	Only incineroar and snorlax, through Caddy's `remote_ip` allowlist.
Map data scope	Only maps I've played: BeatLeader leaderboards and ratings, plus the BeatSaver metadata and zip.
Importer	C# inside the server fork, so it shares the EF models. It reads the archive and upserts into SQLite.
Upstream values	Keep them as they are: pp, ranks, star ratings, and the skill triangle. The local instance has one player, so recalculating would turn every rank into #1.
Attempts	Skipped. No failed or quit attempt replays, and no attempt records (`/scoresstats`). Leaving them out is less work: it's one endpoint the exporter simply doesn't call. Best scores, with their `scoreImprovement`, are the play history.
ScoreSaber merge	Convert ScoreSaber scores into BeatLeader `Score` rows. Each leaderboard keeps one score, chosen across BL, SS main and SS legacy by BeatLeader's own rule. BeatLeader's "improvement over previous best" display then covers ScoreSaber plays too. No separate table.
Identity	One local player: id `76561199407393962`, name `pleb`. The legacy Steam id `76561197962107242` (`i5u1d0r`) becomes a linked alt of that player.
Port	`https://incineroar.brenise.dev:4442`. Caddy uses 4430–4441 on incineroar, so 4442 is the next free one (4444 is free too). The API listens on `127.0.0.1:8042`, which is unused.

Facts (checked 2026-11-08)

My accounts

Source	Id	Scores	Replays
BeatLeader `pleb`	76561199407393962	3,565 best scores (2,492 unranked); 3,559 play count	one `.bsor` per score on `cdn.replays.beatleader.com`, about 1.3 MB each
BeatLeader attempts (not exported)	same	16,518 attempt records (`/player/{id}/scoresstats`), including fails and quits	non-finished attempts are at `/otherreplays/<name>.bsor`, auth-gated
BeatLeader legacy	76561197962107242 (`i5u1d0r`)	3
ScoreSaber main	76561199407393962 (`pleb`)	2,107 (814 ranked), 1,474 unique maps, first on 2023-11-17	1,034 have `hasReplay`
ScoreSaber legacy	76561197962107242 (`i5u1d0r`), inactive	199 (161 ranked), first seen 2022-05-07	58 have replays, already downloaded to `~/ops/beatsaber/ss-scraper/replays/76561197962107242/` (`.dat` plus a `.json` with metadata each)

The local mods keep replays too, as a fallback source. The Windows BSManager instance (~/Windows/Users/pleb/BSManager/BSInstances/<ver>/, mounted with mountwin on incineroar) has UserData/BeatLeader/Replays/ and UserData/ScoreSaber/Replays/. ScoreSaber keeps one file per player/hash/difficulty/characteristic, so it can recover some scores whose server copy has hasReplay: false. ~/ops/beatsaber/backups deliberately leaves these folders out.

Unranked ratings and auth (corrected 2026-11-08)

Every difficulty object has stars, passRating, accRating, techRating (the skill triangle), predictedAcc, plus modifiersRating with ss*/fs*/sf*/bfs*/bsf* variants of each rating.
On score lists (/player/{id}/scores) and /leaderboard/{id}, unranked ratings are filled in only when the requesting user has profileSettings.showAllRatings == true. That check uses HttpContext.CurrentUserID, which reads only the beatleader.com cookie session (with an Issued claim). An OAuth bearer token does not count: tested, bearer and anonymous both return stars/pass/acc/techRating: null.
GET /leaderboards/hash/{hash} returns full ratings for unranked difficulties, anonymously, including all modifiersRating variants. Tested on an unranked ExpertPlus (9db01c9b…): stars 8.33, pass 3.92, acc 9.75, tech 7.70. Phase 1, step 6 already fetches this for every played hash, so the importer takes difficulty ratings from these snapshots, not from the nulls embedded in the scores.
Bearer tokens only work on /oauth2/identity, clan management and ReeSabers presets (CurrentOauthUserID). GET /user with a bearer token is 401. The exporter doesn't actually need auth; the token chain is kept for identity checks and in case a cookie-gated endpoint turns out to matter.
The sync's chain lives in scripts/archive/auth.ts: login via the loopback redirect http://localhost:5173/auth/beatleader/callback (registered on the OAuth app, as is https://plebsaber.stream/auth/beatleader/callback), token store <PLEBLEADER_DIR>/auth/beatleader_tokens.json (default .data/plebleader/, mode 0600). auth status checks identity and ratings.
Token lifetimes: access tokens last 3,600 s and refresh tokens 14 days. Each refresh rotates the refresh token and invalidates the old one. So the sync job needs its own refresh token, separate from the plebsaber.stream site's, or the two invalidate each other. It persists every new refresh token atomically before using the new access token. The daily sync stays well within the 14 days. If the chain breaks (no run for 14 days, or a failed write), log in once more through plebsaber.stream's /auth/beatleader/login and seed the new token.

BeatLeader endpoints the exporter uses

Base https://api.beatleader.com. Rate limit: 50 GETs per 10 s overall, 10 per 10 s on /players, 8 per second on /maps. Swagger is at /swagger/index.html.

Data	Endpoint	Notes
identity and settings	`GET /user`	confirms the token works (`player.id`)
profile	`GET /player/{id}?stats=true`	`scoreStats`, `history`, pinned scores, badges
best scores	`GET /player/{id}/scores?count=100&page=N&sortBy=date&order=desc&includeIO=true`	`count` ≤ 100. Each score includes `leaderboard.song` and `leaderboard.difficulty` (with ratings), `replay`, `offsets`, `scoreImprovement`. Incremental runs use `time_from=<unix>`.
replay stats	`GET /score/statistic/{scoreId}`	the accuracy graph, hit tracker and so on, from the server's `scorestats` storage
map data	`GET /leaderboards/hash/{hash}`	every difficulty of a song, with ratings when the request is authenticated
rank history	`GET /player/{id}/history`, `/history/triangle`, `/accgraph`
live feed (optional)	`wss://sockets.api.beatleader.com/scores`	all scores globally. Filter on my id for near-real-time sync.

BeatSaver (public, no auth): BeatSaverAPI.getMapByHash(hash) in src/lib/server/beatsaver.ts calls GET https://api.beatsaver.com/maps/hash/{hash}. Each entry in versions[] carries its own hash and downloadURL (the zip). Each map version has its own hash, and scores refer to a version hash, so download the version whose hash matches. Old versions may only be on the CDN or nowhere at all. When a version or map is gone from BeatSaver, take it from the game's own folders on the Windows side instead: Beat Saber_Data/CustomLevels and CustomWIPLevels in the BSManager instance. The folders there are unpacked maps, so compute their hash the way the game does: SHA-1 over Info.dat plus each difficulty file, in order.

The server's MapsPath (MapPostprocessor/MapDownloader.cs) wants <MapsPath>/<hash>/ folders containing an unpacked map with Info.dat. It checks the lowercase and uppercase hash. Only when a folder is missing does it fetch beatsaver.com/api/maps/hash/{hash} and unpack the zip.

ScoreSaber (public, no auth): GET https://scoresaber.com/api/player/{id}/full, GET /api/player/{id}/scores?limit=100&page=N&sort=recent (each item has score.hasReplay), and GET /api/v2/scores/{scoreId}/replay (.dat). The ScoreSaber API doesn't send CORS headers, so it only works server-side. ~/ops/beatsaber/ss-scraper/download_replays.py already handles pacing (30 s minimum between replays), 429 backoff, and resuming. That pacing is outdated: as of 2026-11-08 ScoreSaber sends x-ratelimit-{limit,remaining,reset}-{short,medium,long} headers. The replay endpoint allows 20 (short), 60/min (medium) and 360/hour (long, the one that binds); player endpoints allow 25/100/400. scripts/archive paces replays at 10.5 s and pauses whenever a window's remaining count hits 0, so 1,034 replays take about 3 h instead of 9.

Upstream server: what the fork has to change

These checkouts are in ~/src. Run git pull before relying on them.

~/src/BeatLeader/beatleader-server (master was at 2026-07-19 1474312; the live branch rebalanced-pp was updated 2026-10-25). Its public submodules are initialized: beatleader-server-models, beatleader-parser, ReplayDecoder, MapPostprocessor, RatingAPI. Two submodules are private and unreachable: beatleader-server-private and bs-simulator. Upstream already copes with that. Startup.cs falls back to ReplayRecalculatorStub, and nothing in the public code imports a namespace that only the private repos define. Verified 2026-11-08: it builds on Linux with nixpkgs dotnet-sdk_10 (10.0.403) with both private folders empty: 0 errors, about 2,300 warnings, about 1 minute.
Database: UseSqlServer for two contexts (Startup.cs:380-386): AppContext (DefaultConnection) and StorageContext (StorageConnection, which holds PlayerLeaderboardStats). There are 570 SQL Server migrations.
Z.EntityFramework.Extensions.EFCore is commercial, with a monthly trial. Call sites: about 92 BulkSaveChanges, 56 BulkUpdate, 19 BulkDelete, 3 BulkInsert.
Raw SQL in 5 places: DBCC CHECKIDENT (Services/DailyRefresh.cs), TABLESAMPLE (ScoreController.GetRandomScore), TRUNCATE and admin statements (Controllers/Admin/AdminController.cs).
Storage: all 31 helpers in Utils/S3Helper.cs go to Cloudflare R2 (S3AccountID.r2.cloudflarestorage.com) and return https://cdn.{container}.beatleader.com/{file}. Containers: replays, otherreplays, assets, previews, scorestats, playlists, unicode, configs, replayedvalues, reepresets, songs, attempts. Nothing calls S3 outside this file.
Background jobs (Startup.cs:455-478): EveryMinuteRefresh, MinuteRefresh, HourlyRefresh, DailyRefresh, HistoryService, BotService, RankingService, SongService, RichPresenceService and MapRefresh run only when ServicesHost=YES. These always run: ConstantsService, RefreshTaskService, ClanTaskService, ClanMessageService, ClanCallbackService, SearchService (Lucene), OauthService, LeaderboardPlayerStatsService.
Config keys, all from IConfiguration: DefaultConnection, StorageConnection, S3*, MapsPath (default /home/maps), SteamKey, SteamApi, BeatSaverId/Secret, Openiddict* (production expects ../keys/openiddict_certificate*.pfx; development uses dev certs), plus optional OAuth secrets for Patreon, Twitch, Discord and others. Those providers register only when their keys are set.
The star-rating models are public: RatingAPI/ (C# with ONNX models) and beatleader-analyzer. We don't need them for the minimum version, because we keep upstream's ratings. They are what would let a later replacement rate maps itself.

Upstream website

~/src/BeatLeader/beatleader-website (2026-11-07). In src/network/queues/beatleader/api-queue.js, on any host it doesn't recognize the API base is /cors/blapi/, which is relative to the page. So our host only needs Caddy to proxy /cors/blapi/* to the API, with no code change. It rewrites assets to /cors/cdn-assets-bl/ and /cors/cdnbeatsaver/ in the same way. BL_REPLAYS_URL and BL_ANALYZER_URL fall back to the public replay.beatleader.com and need a patch. Build with npm run build (rollup) into public/.

ArcViewer

~/src/AllPoland/ArcViewer (Unity 6000.0.67f1). The deploy branch on GitHub holds a ready WebGL site (index.html, Build/, TemplateData/). It loads replays from ?replayURL=<url> and recognizes ScoreSaber replays by their magic header (ScoreSaberDecoder.IsScoreSaberReplay, including the legacy format), so it works for both .bsor and .dat. The map comes from id=<bsr> (BeatSaver) or url=<zip>. Serving it from the same origin as the replays avoids CORS and mixed-content problems.

incineroar

Caddy vhosts are in ~/ops/nix-pleb/systems/brenise.dev/incineroar/workstation.nix. They follow the pattern "${host_fqdn}:44xx" with ${tlsConfig} (Namecheap DNS-01) and a remote_ip allowlist (127.0.0.1/32 ::1/128 192.168.1.34/32 192.168.1.35/32 192.168.1.36/32 10.19.21.34/32). The host firewall is off; Caddy does the gating.
Disk: / has 479 GB free and /home 684 GB. The archive should be 10–25 GB: about 4.6 GB of best-score replays, about 1.4 GB of ScoreSaber replays, and BeatSaver zips for roughly 2,000–3,000 played maps (ScoreSaber alone has 1,474 unique hashes), at a few MB each. Actual (2026-11-08): 21 GB with 1,897 zips at about 8 MB each. Fetching the remaining ~630 older maps later adds about 5 GB.
Allowed clients: incineroar (127.0.0.1/32, ::1/128, 192.168.1.36/32) and snorlax (192.168.254.21/32, the address the amelia fallback vhost already allows).
Timezone: America/Los_Angeles.
nix-pleb rule: put host app config in an existing <system>/<purpose>.nix (here workstation.nix), not a new module file. Never use nixos-rebuild switch without asking; use build. Never decrypt sops secrets: ask the user to add the values.

Layout

plebsaber.stream/                      (this repo)
  docs/plan.md                         this file
  scripts/archive/                     exporter and sync CLI (TypeScript, node ≥ 22)
  src/lib/server/beatleader.ts         extend: bearer auth from the sync's token store, paging, time_from, raw passthrough
  src/lib/server/scoresaber.ts         extend: player scores, replay download

gitea pleb/beatleader-server           fork, branch `pleb/main` (based on upstream `rebalanced-pp`)
  MIRROR.md                            what MirrorMode changes and how to run it
  appsettings.Mirror.json              incineroar values (ASPNETCORE_ENVIRONMENT=Mirror)
  Tools/ArchiveImporter/               C# console app sharing AppContext/StorageContext
gitea pleb/beatleader-server-models    fork of the models submodule, branch `pleb/main`
gitea pleb/beatleader-website          fork, branch `pleb/main`

/var/lib/plebleader/                   on incineroar, user `plebleader`
  archive/                             source of truth, append-only
    beatleader/
      user.json                        latest /user (no secrets)
      players/<id>/profile/<YYYY-MM-DD>.json
      players/<id>/history.json, triangle.json, accgraph.json
      scores/<scoreId>.json            one ScoreResponse each, unmodified
      scorestats/<scoreId>.json
      leaderboards/<hash>/<YYYY-MM-DD>.json   new snapshot only when the content changed
      replays/<original filename>.bsor
    scoresaber/<playerId>/
      player/<YYYY-MM-DD>.json
      scores/<scoreId>.json            the full item (score plus leaderboard)
      replays/<scoreId>.dat
    beatsaver/
      maps/<hash>.json                 getMapByHash response, unmodified
      zips/<hash>.zip                  the map version matching that hash
      missing.json                     hashes BeatSaver no longer serves, with their source if recovered
    game-levels/<hash>/                maps recovered from the Windows CustomLevels when BeatSaver lacks them
    local-mods/{beatleader,scoresaber}/...   copied from the Windows UserData
    state.json                         sync cursors (last timepost per source/feed)
  db/app.db, db/storage.db             SQLite, rebuilt from the archive
  storage/                             S3 replacement (containers as dirs); replays/ and ssreplays/ symlink into archive/
  maps/<hash>/                         MapsPath: unpack cache, filled on demand from archive zips (phase 2, step 9)
  auth/beatleader_tokens.json          the sync's own OAuth token chain (rotates on refresh)
  secrets → EnvironmentFile from sops  BL_CLIENT_ID, BL_CLIENT_SECRET

Phases

Each phase ends with a check. Phase 1 protects the data and doesn't depend on the fork, so do it first.

Phase 0: Prep

Done 2026-11-08 (all were already current; upstream server master 1474312, rebalanced-pp 52a50ed, website 2b3ef891, ArcViewer deploy c776256, ScoreSaber website 836d789). Update the checkouts: git -C ~/src/BeatLeader/beatleader-server pull --recurse-submodules, and the same for beatleader-website, ~/src/AllPoland/ArcViewer (also git fetch origin deploy), and ~/src/ScoreSaber/website. The ScoreSaber website is public now; use it to check the score and replay API shapes.
Done 2026-11-08. Create the Gitea forks pleb/beatleader-server and pleb/beatleader-website with the gitea-api skill (mirror or push). Keep upstream as a remote so you can rebase later. Created as private, non-mirror imports through POST /repos/migrate (the Gitea token lacks write:user, which POST /user/repos needs), with default branch pleb/main. The ~/src checkouts keep origin = GitHub and gained a gitea remote; pleb/main is checked out there. The server's pleb/main starts from rebalanced-pp, which is master plus 4 commits and what the live API most likely runs, so the entities match the archived responses. pleb/beatleader-server-models was forked too (see phase 2, step 7); the server's .gitmodules points at it over SSH.
Bring this repo's flake up to date (it's still on nixos-25.05) only if the exporter gets packaged here.
Done 2026-11-09 (systems/brenise.dev/incineroar/plebleader.env in nix-pleb, dotenv format, encrypted from this repo's .env without decrypting anything; wired up on the plebleader branch). Ask the user to add a sops secret plebleader.env for incineroar, holding BL_CLIENT_ID and BL_CLIENT_SECRET. These are the same OAuth app as plebsaber.stream; check that its registered callback URLs include the one used for seeding. Agents never decrypt it. For local runs before deployment, use the gitignored .env in this repo.
Done 2026-11-08 (scripts/archive/cli.ts auth login; refresh rotation tested; chain expires about 2026-11-22 unless refreshed). Seed the sync's token chain. Add a small scripts/archive command (auth login) that runs the same authorization-code flow as /auth/beatleader/login with scope=profile offline_access. It catches the callback on a loopback port, or reuses a plebsaber.stream callback URL the OAuth app already allows. It writes auth/beatleader_tokens.json. Don't copy the plebsaber.stream site's own refresh token, because rotation would break whichever side refreshes second.

Phase 1: Full export (one-time backup)

Scope (decided 2026-11-08): all player data for both accounts, all history: profiles, BL scores, replays, score stats, SS scores and replays, plus BL leaderboard JSON (ratings) for every played hash. Only BeatSaver map files are limited to maps played in the last two years (--maps-since 2024-11-08, DEFAULT_MAPS_SINCE in scripts/archive/config.ts). Exception: legacy ScoreSaber scores that have an archived replay get their map fetched whatever its age, so those replays render (58 replays, 53 maps; mapHashes() in cli.ts). Legacy scores without a replay don't get a map. Older map zips come later with export maps --maps-since all; existing files are skipped, so that only fetches what's missing. Replays of older scores won't render a map in ArcViewer until then. SS main has 1,034 replays (about 3 h at 10.5 s each). The 58 SS legacy replays are copied from ss-scraper's download instead of being fetched again.

Commands: node scripts/archive/cli.ts export bl|ss|ss-replays|maps|all [--since …] and report. The archive lives in .data/plebleader/archive/ (set PLEBLEADER_DIR to change the root) until /var/lib/plebleader exists on incineroar; then move it. Logs go in .data/plebleader/logs/. Profile, history, triangle and accgraph are stored as dated snapshots in players/<id>/<kind>/<YYYY-MM-DD>.json, written only when changed.

Build scripts/archive/ as a small CLI (export bl, export ss, export maps, import-local-mods, sync). Rules:

Store responses byte-for-byte. Never reshape them on the way in.
Make every step resumable: skip files that already exist, keep cursors in state.json, and write to temp files before renaming.
Respect rate limits. Use about 4 requests per second to BeatLeader and honor Retry-After (the existing requestWith429Retry handles 429s). For ScoreSaber replays, stay under the 360/hour limit (10.5 s pacing) and honor the x-ratelimit-* headers.
Send User-Agent: plebsaber-archive/0.1. Don't put an email address in it.

Steps:

Auth check. node scripts/archive/cli.ts auth status: /oauth2/identity must return 76561199407393962, and /leaderboards/hash/{hash} must return non-null stars for an unranked difficulty. If either fails, stop.
BeatLeader scores. Page through /player/{id}/scores with sortBy=date&order=desc&count=100&includeIO=true until it's exhausted (about 36 pages). Do the same for the legacy id (3 scores).
Attempts: skipped. Don't call /scoresstats or /otherreplays.
Replays. Fetch every scores/*.replay from the CDN (no auth, CORS *). Keep the original filenames.
Score stats. GET /score/statistic/{id} for each best score.
BeatLeader map data. Collect the set of played hashes: every unique song.hash in BL scores, plus leaderboard.songHash in both ScoreSaber accounts. Normalize them to lowercase. For each, call GET /leaderboards/hash/{hash} (no auth needed). That gives ratings and the triangle for every difficulty, ranked or not. This is the only source of unranked ratings; the scores from step 2 have them nulled.
Map files from BeatSaver. For each played hash, call BeatSaverAPI.getMapByHash(hash) and save the response to beatsaver/maps/<hash>.json. Then download the zip for the versions[] entry whose hash matches, to beatsaver/zips/<hash>.zip. Extend beatsaver.ts with a downloadMapZip(version) that streams to a temp file, checks it's a valid zip containing Info.dat, then renames it. Go about 2 requests per second and honor 429s. Write each hash that can't be resolved (deleted map, or an old version gone from the CDN) to missing.json.
Recover missing maps from the game: optional, low priority. Decided 2026-11-08: maps gone from BeatSaver are accepted as missing. They were most likely updated or removed for a reason. The user's online backups and local map archives from before 2026 don't have them. Older cold storage might, so that's a "someday" check, not a task. The current BSManager instance's CustomLevels may hold a few recent ones and is cheap to try: with mountwin, hash every folder in Beat Saber_Data/CustomLevels and CustomWIPLevels, and copy those whose hash is in missing.json to game-levels/<hash>/. Whatever is still missing stays listed; those replays just can't render the map.
Unpack for the server: dropped 2026-11-08, replaced by on-demand unpacking (phase 2, step 9). ArcViewer loads url=/cdn/maps/<hash>.zip (Caddy serves the archive's zips) instead of id=<bsr> from BeatSaver, so it never needs unpacked folders.
Profile and history. Save /player/{id}?stats=true, /history, /history/triangle, /accgraph, and /user (strip anything secret).
ScoreSaber main. Page through all scores (22 pages), then download the 1,034 replays. Either port the download_replays.py logic to TypeScript or run it as-is with -o archive/scoresaber, and normalize the layout.
ScoreSaber legacy. Copy the 58 replays and the metadata from ~/ops/beatsaber/ss-scraper/replays/76561197962107242/ into archive/scoresaber/76561197962107242/ (the filenames start with the scoreId). Fetch the 199 score items again so the archive has complete, unmodified responses.
Local mod replays. With the Windows partition mounted (mountwin, then umount ~/Windows afterwards), copy UserData/BeatLeader/Replays and UserData/ScoreSaber/Replays into archive/local-mods/. Afterwards, match the ScoreSaber files against scores that have hasReplay: false by player/hash/difficulty/characteristic.

Check: the counts match the live API: 3,565 BL scores, 3 legacy, 2,107 plus 199 SS scores, and replay files for every score with a replay URL or hasReplay. Every played hash should have a BeatLeader leaderboard snapshot, plus a BeatSaver zip or recovered game folder, unless it's listed in missing.json. Write a report subcommand that prints counts and what's missing, and paste its output into this file.

Report, 2026-11-08 (phase 1 complete). The ScoreSaber replays finished with zero 429s.

Window:  scores since all time, BeatSaver maps since 2024-11-08
BeatLeader scores        3568 in window (3568 archived total, both accounts)
BeatLeader replays       3568/3568
BeatLeader score stats   3568/3568
ScoreSaber 76561199407393962: 2107 scores, replays 1034/1034
ScoreSaber 76561197962107242: 199 scores, replays 58/58
Played hashes            2560
  BL leaderboard snapshot 2552/2560
  BeatSaver zip           1897/1926 in maps window (missing.json lists 29)
Archive size             21.78 GB

Steps 8 (game-folder recovery) and 13 (local mod replays) are still open. Step 9 was dropped in favour of on-demand unpacking.

The 29 missing hashes are all "hash unknown to BeatSaver", and none belongs to a legacy score with a replay (all 53 of those maps are archived). The 8 hashes without a BL leaderboard are ScoreSaber-only maps BeatLeader never saw. Zips average about 8 MB (15 GB for 1,897), more than the estimate below.

Phase 2: Server fork on SQLite

Done 2026-11-08 (pleb/main at 1741ce9, models fork at 8f233e8). What differs from the steps below:

Step 2: Microsoft.Data.SqlClient stays as a direct package, because one controller catches SqlException. One controller had an unused using BeatLeader_Server.Migrations; that broke the build once the old migrations were gone. The fresh migrations create 155 tables and 217 indexes. WAL and migrations are applied at startup by Utils/MirrorSetup.cs; busy_timeout comes from Default Timeout=30 in the connection string.
Step 3: Utils/BulkCompat.cs sits in the global namespace, like Z's own extensions, because some callers have no EF using. 49 of the BulkUpdate calls pass ColumnInputExpression, often on detached stubs that carry only the key and those columns, so the shim marks only the listed columns modified and detaches the stubs after saving. A plain UpdateRange would have wiped every other column. Not exercised at runtime yet; the importer will be the first real user.
Step 4: the two index-defragmentation scripts became PRAGMA optimize and REINDEX; ANALYZE;.
Step 5: GetS3Client returns a LocalStorageClient (an AmazonS3Client subclass) when StoragePath is set; the six S3 primitives branch on it. Upstream behaviour is unchanged without StoragePath.
Step 6: MirrorMode also skips RefreshTaskService and MapRefresh, and the external OAuth providers. Two surprises showed up in testing: GetLazy fetched unknown players from Steam (it now returns null, so 404), and the BeatSaver login is registered unconditionally and validated on every request (it gets placeholder credentials, otherwise every request was a 500).
Step 7: instead of patching about 35 showRatings decisions, the four HideRatings() leaves return early when MirrorSettings.ShowAllRatings (in the models fork) is set; MirrorSetup sets it. Sorting and filtering by stars on unranked maps still follow the requesting user's settings; patch Utils/{Map,Leaderboard}ListUtils/Sort.cs if that matters.
Step 8: appsettings.Mirror.json loads with ASPNETCORE_ENVIRONMENT=Mirror; MIRROR.md in the fork documents it. OpenIddict uses ephemeral keys. Run the DLL with --contentRoot <bin> from a working directory in the state dir, because data protection keys go to ../keys/. dotnet run ignores ASPNETCORE_URLS in favour of launchSettings.json.

Check result: on empty files, /swagger/index.html 200, /player/76561199407393962 404, /players and /leaderboards 200 with empty pages, no fail: lines in the log.

Work on the pleb/main branch of pleb/beatleader-server. Keep each change small and in its own commit, so upstream rebases stay manageable.

Build on Linux before changing anything: nix shell nixpkgs#dotnet-sdk_10 -c dotnet build BeatLeader-Server.csproj (nixpkgs has SDK 10.0.403). Record which errors come from the missing private submodules versus the platform. Delete or exclude the empty beatleader-server-private and bs-simulator folders, and stub whatever they turn out to supply.
SQLite. Replace Microsoft.EntityFrameworkCore.SqlServer with ...Sqlite, and point both AddDbContextFactory calls at UseSqlite (Data Source=/var/lib/plebleader/db/app.db and storage.db). On connection open, set PRAGMA journal_mode=WAL; busy_timeout=5000;. Move the 570 SQL Server migrations aside and generate one fresh migration per context (dotnet ef migrations add InitialSqlite --context AppContext, and the same for StorageContext), so later schema changes still work.
Drop Z.EntityFramework.Extensions. Add Utils/BulkCompat.cs with extension methods of the same names and shapes: BulkSaveChanges → SaveChanges, BulkUpdate → UpdateRange + SaveChanges, BulkDelete → RemoveRange + SaveChanges, BulkInsert → AddRange + SaveChanges. With one player, performance doesn't matter, and about 170 call sites keep compiling unchanged.
Raw SQL. DBCC CHECKIDENT → DELETE FROM sqlite_sequence WHERE name IN (...). TABLESAMPLE → ORDER BY RANDOM() LIMIT 1. TRUNCATE → DELETE FROM.
Storage. Rewrite the bodies of the 31 S3Helper methods to read and write ${StoragePath}/{container}/{filename}, and return ${PublicBaseUrl}/cdn/{container}/{filename}. GetS3Client can return a client that's never used, so the method signatures don't change.
Background jobs. Leave ServicesHost unset. Add a config flag (MirrorMode=true) that also skips registering RankingService, the clan services, OauthService and LeaderboardPlayerStatsService. Keep SearchService and ConstantsService if they start cleanly, and point Lucene's index directory at the state dir.
Auth. For the minimum version the site is read-only behind Caddy's IP allowlist. No login is needed: the website shows a public profile. If the ratings in API responses still depend on ShowAllRatings for the requesting user, then in MirrorMode treat requests as coming from 76561199407393962, or just store ratings so they always show. That means patching the showRatings checks, which all read ProfileSettings.ShowAllRatings, to true.
Configuration. Put an appsettings.Production.json template in the fork and the real values in the NixOS unit's environment (ASPNETCORE_URLS=http://127.0.0.1:8042, ConnectionStrings/DefaultConnection, StoragePath, PublicBaseUrl, MapsPath, MirrorMode). Use development OpenIddict certs, or generate a self-signed pfx into the state dir.
On-demand map unpacking (deferred; do it only when needed). MapDownloader.Map(hash) (in the MapPostprocessor submodule) returns MapsPath/<hash>/ when it exists and otherwise downloads from BeatSaver. Add a step before the download: if <MapZipsPath>/<hash>.zip exists (the archive's beatsaver/zips/), extract it to a temp folder under MapsPath, check it contains Info.dat, rename it to <hash>/ and return that. Accept upper- and lowercase hashes, like Map does. In MirrorMode, never fall through to BeatSaver: return null instead. That makes MapsPath a cache that can be deleted at any time.
- Technical debt: low. It's about 20 lines and changes no interfaces. The catch is where it lives: MapDownloader is in MapPostprocessor, so editing it means a fourth fork. Its methods aren't virtual, so they can't be overridden. Without a fork, add a small Utils/LocalMapZips.EnsureUnpacked(hash) in the server and call it before _downloader.Map(...) at the three call sites (ReplayController, EarthDayController, MapRefresh). That's a few extra lines per rebase and no submodule fork, so prefer it.
- Why deferred: no read path of the mirror calls MapDownloader today. Its callers are MapRefresh (off in MirrorMode), the Earth Day event controller, and replay upload and recalculation in ReplayController. It becomes necessary if the mirror ever recalculates score stats (for example the optional .dat → .bsor conversion in phase 5) or accepts replay uploads.

Check: the server starts on an empty SQLite file, /swagger/index.html loads, and GET /player/76561199407393962 returns 404 rather than 500.

Phase 3: Importer

Importer done 2026-11-09 (pleb/main at e3f991c, pushed to Gitea). Usage and the full file → table mapping are in the fork's MIRROR.md. A full import takes about 11 s, and an unchanged rerun writes 0 rows. What differs from the steps below:

The archived responses are deserialized straight into the entity types, because their property names match. Only scalar columns are copied onto the tracked rows; keys, foreign keys and navigations are set explicitly. Leaderboard snapshots are the main source for Song/DifficultyDescription. A score's embedded leaderboard is a fallback only, and no score needed it.
One-to-one children (ScoreImprovement, ReplayOffsets, ModifiersRating, DifficultyStatistics, ModifiersMap, stats, settings) get local ids, so they can't collide with upstream ids. Scores, difficulties, songs, leaderboards, mappers and history rows keep upstream ids.
PlayerScoreStatsHistory lives in StorageContext, so that database isn't empty. The exporter had saved only 50 days of /history (the API default); it now asks for count=100000 (1,138 days for pleb). The triangle needs those monthly points.
No legacy Player row. An AccountLink (PCOculusID = legacy id) plus a // pleb: patch in PlayerIdToMain resolves /player/76561197962107242 to pleb. Upstream only checks Steam-range ids against Oculus links. The legacy account's 3 BL scores are skipped until phase 5.
StoragePath/replays is a symlink to the archive's beatleader/replays. Score stats are copied to StoragePath/scorestats/<scoreId>.json.
Clans aren't mirrored (ClanOrder is blank).

Imported: 2,552 songs, 10,104 difficulties, 10,098 leaderboards, 628 mappers, 3,565 scores, 3,568 score stats, 1,138 history rows. API check against the archived upstream responses, on a scratch DB with ASPNETCORE_ENVIRONMENT=Mirror:

/player/{id}?stats=true matches. The only differences are row ids, clans, and linkedIds, which now lists the alt.
/player/{id}/scores?includeIO=true: 3,565 total. Scores, improvements and offsets match, and unranked difficulties now carry stars and the skill triangle ratings.
/accgraph and /history/triangle match. /score/statistic/{id} redirects to the local file.
/leaderboard/{id} and /leaderboard/scores/{id} first returned 500. About 10 projections order Player.Clans by an expression over the outer row's ClanOrder, which needs SQL APPLY, and SQLite has none. Fixed in e3f991c: Utils/MirrorQueryInterceptor.cs, an EF query interceptor registered only in MirrorMode, rewrites any ordering of Player.Clans to ordering by clan id. One file instead of patching every call site; the mirror has no clans, so the order doesn't matter. The leaderboard, clan-ranking and context variants now return 200.

The check below (stock website) is still to do in phase 4.

Tools/ArchiveImporter lives in the fork, references the main project, and uses its AppContext/StorageContext. Run it as a one-shot that is idempotent and upserts by upstream id.

beatleader/scores/*.json (ScoreResponse) → Song, DifficultyDescription (keep the upstream ratings and ModifiersRating as they are), Leaderboard, Score (with upstream Id, Rank, Pp, Weight, and Timepost), ScoreImprovement, ReplayOffsets. Rewrite Replay to the local /cdn/replays/<file>.
leaderboards/<hash>/latest updates the difficulty ratings, so maps you played but have no best score for still get them.
beatsaver/maps/<hash>.json fills in Song fields that BeatLeader leaves out (BeatSaver key, description, tags, upload date). The server doesn't read map files on any mirror path; if one appears, phase 2, step 9 unpacks them on demand.
players/<id>/profile/latest → Player, PlayerScoreStats, ProfileSettings, plus history rows from /history. Insert the legacy id as an AccountLink or alt pointing to the main player, so PlayerIdToMain resolves /u/76561197962107242 to pleb. Its 3 BL scores go through the phase 5 merge like any other candidate.
StorageContext holds PlayerScoreStatsHistory; its attempts tables stay empty, because attempts aren't exported.
scorestats/*.json → the scorestats storage container, using the filename the server expects (check ScoreController.GetStatistic).

The response DTOs aren't the entities, and this mapping is the largest single piece of work. Use beatleader-server-models/Models/* (entities) and ResponseUtils.cs (entity → DTO) as the reference, working backwards.

Check: the stock website (phase 4) shows my profile with the right score count, the pp and ranks from upstream, unranked star ratings, and the triangle on the profile.

Phase 4: Website, viewer, Caddy

Steps 1–2 done and both checks passed 2026-11-08, on a dev stand-in for the vhost: the importer into .data/plebleader/dev/db/, the API on 8042 (PublicBaseUrl=http://127.0.0.1:8890), and an ad-hoc nix shell nixpkgs#caddy on http://127.0.0.1:8890 with the routes below (.data/plebleader/dev/{run-api.sh,Caddyfile}). Use 127.0.0.1, not localhost: the website sends localhost to the real upstream API. Commits: server 7cc964a, website f5194ee3, 9ebe84ea, 8d540642, all local (not pushed yet). Step 3 (nix-pleb) is still open.

Phase 3 check passed. /u/pleb: 11,608.35pp, #3,861 global, US #1,643, 672 ranked and 3,565 total scores, unranked difficulties with stars (8.33★ on the Phase 1 test map), the skill triangle with monthly history. The leaderboard page and /score/{id} (accuracy graphs from local score stats) work too.
Server fix: the profile page calls GET /v2/player/{id}/hydrate, which the live API has and the public source doesn't (404, "Player not found"). Controllers/MirrorHydrateController.cs rebuilds it from PlayerController.Get plus the profile object that the website's process-hydrate.js reads.
Server run note: dotnet build doesn't copy wwwroot/ into bin/. Without it PreviewController's constructor throws on /images/Star.png, and any controller that depends on it (/user, /user/config) returns 500. The dev run passes --webroot <src>/wwwroot; the Nix package uses dotnet publish, which copies it.
Website, as built: IS_MIRROR in api-queue.js is true when the API resolves to /cors/blapi/. In that case BL_REPLAYS_URL is /watch/ and score stats come from /cdn/scorestats/. Every hardcoded allpoland.github.io/ArcViewer/ link points at /watch/ (except the clan page's embed). public/watch/index.html takes BL replay-viewer or ArcViewer parameters (link, replayURL, scoreId, id, optional hash) and redirects to /arcviewer/?replayURL=…&url=/cdn/maps/<hash>.zip&noProxy=true. It reads the hash from the replay filename (<score>-<player>-<diff>-<mode>-<HASH>.bsor); for scoreId it first asks /cors/blapi/score/{id} for the replay URL. Phase 5 can pass hash= for .dat replays. The analyzer icons are off by default (an upstream Patreon service). Clans, events, the socket page and the star-rating graph are still visible; they just show nothing.
ArcViewer facts: without noProxy=true it routes every non-whitelisted URL through cors.bsmg.dev, which can't reach a LAN host. When both id and url are given, id wins, so never send both. With no map parameter it looks up the replay's own hash on BeatSaver. The deploy build ships uncompressed Build/ files, so no Content-Encoding headers are needed. Its own calls to api.beatleader.com (player card, leaderboard info) fail from this origin, so the player card stays blank; fixing that needs a Unity build, so it's left as is.
Map zips: BeatSaver's CDN serves https://r2cdn.beatsaver.com/<hash>.zip with Access-Control-Allow-Origin: * (it rejects HEAD, use GET). Caddy redirects to it when the archive has no zip, so the website always links the local path and the ~630 older maps outside --maps-since still play.
Nix: the website fork's flake.nix builds packages.default (yarn v1 offline, ~16 s; deps hash sha256-ThGvMJfU…). Its postPatch turns the themes' Google Fonts @import url(...) into @import (css) url(...), because less would fetch them at build time. ArcViewer deploy at c776256497b66f7c91a74162cfcd943b0f45ee2e is sha256-4A+aVnFrKyWQjFnH2N/CwoVIlwrcgV2fYYrtaOiodo0= for fetchFromGitHub.

Step 3 was done together with phase 7 (see there): the vhost below is in nix-pleb, waiting for the switch.

In the website fork, point BL_REPLAYS_URL (and the "watch replay" links) at /arcviewer/?replayURL=<encoded local replay url>&url=<encoded /cdn/maps/<hash>.zip>. Fall back to id=<bsr> only for hashes that have no local zip. Hide or stub the features that can't work offline: clans, events, friends, the Socket page, and the stage analyzer and ppai2 calls in StarRatingGraph.svelte / star-generator.js. Leave the /cors/blapi/ resolution as it is.
Package the ArcViewer deploy branch with fetchFromGitHub at a pinned revision, as static files.

Add a Caddy vhost in incineroar/workstation.nix:

"${host_fqdn}:4442" = {
  # Personal BeatLeader mirror (plebleader). Website + ArcViewer static,
  # API on 8042, replays/assets/maps from the state dir.
  # Only incineroar itself and snorlax (192.168.254.21).
  extraConfig = ''
    ${tlsConfig}
    @plebleader_allowed remote_ip 127.0.0.1/32 ::1/128 192.168.1.36/32 192.168.254.21/32
    handle @plebleader_allowed {
      handle_path /cors/blapi/* { reverse_proxy http://127.0.0.1:8042 }
      handle_path /cdn/maps/* {
        root * /var/lib/plebleader/archive/beatsaver/zips
        @nozip not file
        redir @nozip https://r2cdn.beatsaver.com{path} 302
        file_server
      }
      handle_path /cdn/* { root * /var/lib/plebleader/storage; file_server }
      handle_path /arcviewer/* { root * ${arcviewer}; file_server }
      handle_path /cors/cdn-assets-bl/* { reverse_proxy https://cdn.assets.beatleader.com { header_up Host {upstream_hostport} } }
      handle_path /cors/cdnbeatsaver/* { reverse_proxy https://cdn.beatsaver.com { header_up Host {upstream_hostport} } }
      # {path}/index.html serves /watch/; without it the SPA's 404 answers
      handle { root * ${plebleader-website}; try_files {path} {path}/index.html /index.html; file_server }
    }
    handle { respond "Forbidden" 403 }
  '';
};

Checked: the deploy branch ships uncompressed Build/ files, so no Content-Encoding headers are needed.

Check: in Chromium, https://incineroar.brenise.dev:4442/u/pleb shows the profile and scores, and a replay opens in ArcViewer.

Phase 5: Convert ScoreSaber scores and merge

Built and checked on a scratch copy 2026-11-09; committed, deploy pending. Server 50f7df6 (Tools/ArchiveImporter/ScoreMerge.cs, rules in the fork's MIRROR.md), website 5522f245 (SS badge, /watch/ for .dat), exporter ssMapHash in this repo. All pushed. Result on the local archive: 3,710 leaderboards with one score each (3,566 BL + 141 SS-only BL leaderboards + 3 DLC levels BL lacks). Winners: 3,564 BL main, 1 BL legacy, 139 SS legacy, 6 SS main; 2,022 SS scores were the same play as a BL score and dropped; 20 BL scores got an improvement recomputed against an earlier SS play. An unchanged rerun writes 0 rows. What differs from the steps below:

Step 1: no new columns. Provenance goes in Platform (scoresaber,<SS player id>), and Id ≥ 1e9 marks SS scores. Columns would have needed a models-fork change, a migration, and edits to the ~90 score projections before the website saw them; Platform already reaches every score response.
Step 2: OST levels are ost_<LevelId> on SS and <levelid> on BL; 5 of 8 resolve that way. Me Porto Bonito, Nuthin' But A "G" Thang and Stressed Out (3 scores) aren't on BL (404), so the importer builds a song and leaderboard from the SS item (<LevelId><diff><mode> ids, no ratings, accuracy 0). The exporter now maps ost_ hashes too (scoresaber.ts ssMapHash). HMD and controller come from SS deviceHmd/deviceControllerLeft. SS replays are served from storage/ssreplays → archive/scoresaber, so the URL is /cdn/ssreplays/<player>/replays/<id>.dat.
Pp isn't upstream's. Neither the fork's rebalanced-pp formula (median 0.94x, range 0.69–1.16x) nor upstream master's (0.81x) reproduces the pp on archived BL scores; live BL's code is private. So the fold compares every candidate by locally computed pp, which keeps the order right, and BL winners keep upstream's pp. The 70 SS winners on ranked maps show local pp, and improvement pp deltas are local on both sides. Matching pp first sent two BL scores (KillerToy, Make Me) to worse SS plays.
Step 3: same-play duplicates (equal base score, ≤ 2 min apart; observed −30 s…+9 s) keep the BL copy. Each upstream scoreImprovement is turned back into the best it replaced and added as a candidate, so upstream's improvement stays exactly when that rebuilt best is also the fold's previous best. Older archived BL bests are candidates too.
Step 5: the badge sits next to the rank in both ScoreRank.svelte (rank 0 already renders "-"). /watch/ takes the map hash from the score (/score/{id} → song.hash) for .dat replays, from ?scoreId= or 1e9 + the id in the filename. Expanding an SS row shows its leaderboard and no stats tab; the page logs a JSON error for the 404 score stats, harmless.
Check: The Goddess Figure (Normal) is the case where a 2022 legacy SS play beats the BL score; it shows with the SS badge, and the BL Hard score there now reads "+76,821 vs 4 years ago" (vs the legacy SS play). An SS replay redirects to ArcViewer with the .dat and local zip and loads the map; watching it play still needs a real browser.

Deployed 2026-11-09 (nix-pleb a41f3c2); the first sync imported 3,710 scores with the same split by source as the local check. The profile's score search then returned 500, for any search: the Lucene song and player indexes were written next to the DLL in the read-only /nix/store (phase 2 step 6 had planned to move them). Server 2631c46 adds SearchIndexPath (MirrorMode, set to /var/lib/plebleader/search). Server 81f12ee then moved the rebuild into the importer: it replaces the indexes at the end of a run that added scores (so at most once a day, right after the sync), or when there are none. The API builds them only when missing; upstream appended every song again every 6 h. nix-pleb bcdd591, a313f5b bump both. The next plebleader-sync run re-imports with the merge.

ScoreSaber scores become BeatLeader Score rows, and each leaderboard keeps exactly one score. BeatLeader already shows how a score compares with the previous best (ScoreImprovement), so after the merge that comparison covers ScoreSaber plays too. This is importer logic in the fork. The archive keeps the raw ScoreSaber responses as they are.

Provenance columns. Add nullable Source (null = BeatLeader, "scoresaber") and SourceScoreId / SourcePlayerId to Score. This goes into the fresh SQLite migration from phase 2. The website uses them for a small "SS" badge, and the importer uses them to stay idempotent.
Convert each ScoreSaber item (scoresaber/*/scores/*.json) into a candidate Score:
- Leaderboard: leaderboard.songHash (lowercased), plus leaderboard.difficulty.difficulty. ScoreSaber uses 1/3/5/7/9 for Easy…ExpertPlus, the same numbers as BeatLeader's difficulty.value. gameMode SoloStandard → mode Standard (strip Solo). Resolve it to the BL leaderboard from leaderboards/<hash> (phase 1, step 6). If BL has none, create Song/DifficultyDescription/Leaderboard from the BeatSaver JSON, with no ratings.
- BaseScore, ModifiedScore, Modifiers (the same comma-separated codes), MissedNotes, BadCuts, FullCombo, MaxCombo, Timeset/Timepost come from timeSet. Set Platform = "scoresaber" and Hmd = unknown, unless the codes are mapped.
- Accuracy = BaseScore / maxScore, taking maxScore from the BL difficulty. ScoreSaber reports maxScore: 0 on many unranked leaderboards. Check this against how ReplayUtils computes BL accuracy.
- Pp: ReplayUtils.PpFromScore(score, difficulty), using the archived BL ratings, when the BL difficulty is ranked; otherwise 0, as BeatLeader does. ScoreSaber's own pp and rank aren't comparable, so keep them only in the archive. Set Rank to 0, and have the website show "—" for ScoreSaber-sourced rows.
- Id = 1_000_000_000 + ssScoreId. ScoreSaber ids are about 95M and upstream BL ids about 151M, so this can't collide and still fits in an int32.
- Replay = /cdn/ssreplays/<ssScoreId>.dat when the archive has one (from the server, or recovered from local mod replays); otherwise null. No ReplayOffsets and no score stats, because the server can't decode .dat.
Merge per leaderboard. The candidates are the upstream BL best, the ScoreSaber main best, the ScoreSaber legacy best, and the legacy BL score. Sort them by time and fold them with BeatLeader's own ReplayUtils.IsNewScoreBetter(old, new): compare by pp when either score has pp, otherwise by modified score, where a no-fail score loses to a clean one. The winner becomes the leaderboard's Score.
- ScoreImprovement on the winner is the difference from the best before it in the fold: score, accuracy, pp, misses, bad cuts, modifiers, timeset. If the winner is the upstream BL score and nothing from ScoreSaber came between it and its previous BL best, keep upstream's scoreImprovement as it is.
- Candidates that lose stay in the archive only.
Player totals. Keep upstream's PlayerScoreStats, pp and rank. They won't count ScoreSaber scores that won. Recomputing the weighted pp total locally is an optional later step; label the profile numbers "BeatLeader, as of ".
Website. Show the "SS" badge on ScoreSaber-sourced scores, hide the accuracy graph and stats for them (there are no score stats), and send their replay button to /arcviewer/?replayURL=/cdn/ssreplays/<id>.dat&url=/cdn/maps/<hash>.zip. Every score shows under the name pleb; i5u1d0r survives only as SourcePlayerId.

Optional later: convert .dat into .bsor, using ArcViewer's ScoreSaberDecoder as the reference for the format. The server's statistics code could then produce score stats for ScoreSaber-sourced scores too.

Check: the leaderboard count is the union of the BL and SS leaderboards I played, with exactly one score each. The importer prints how many winners came from each source. Spot-check a map where an old legacy ScoreSaber score beats the BL score, and confirm the improvement display and the ArcViewer replay.

Phase 6: Ongoing sync

Built 2026-11-09; deployed on incineroar the same day (daily timer at 17:00, first run clean; see phase 7). node scripts/archive/cli.ts sync [--ratings] [--no-ss-replays] (this repo d182fd8), packaged as the flake's plebleader-archive. Tested against the local archive: the token refresh rotated the chain (now valid until about 2026-11-23), and with the newest BL and SS score files set aside, the run fetched exactly those two back, byte-identical, and re-snapshotted the BL leaderboard. What differs from the steps below:

Main account only; both legacy accounts are frozen. The incremental window starts a day before the newest archived score (BL timepost, SS timeSet), computed from the archive rather than state.json. Replays and score stats are retried for every score in the window, not only new ones.
New BL hashes get a refreshed leaderboard snapshot; new SS-only hashes get one if they have none. Map zips cover every hash in the window.
Ratings drift: Sundays by the machine's clock, or --ratings.
Each run writes a sync section to state.json (new score ids, maps without a zip). exportScores no longer resets newestTimepost to 0 on an empty window.
Step 8: the importer runs in full after every sync (about 10 s), as a second ExecStart of plebleader-sync; no --since flag.
Importer fix (server dd94772): the archive keeps an old best after its improvement is synced. The importer now takes only the newest score per leaderboard and removes rows superseded since the last import. Before this, the second score would have broken the (PlayerId, LeaderboardId, ValidContexts) unique index. Tested on a scratch copy with a fake improved score: added, old one removed, and back again once the fake was deleted.

The sync subcommand runs once a day from a systemd timer on incineroar: OnCalendar=*-*-* 17:00:00 America/Los_Angeles with Persistent=true, so a run missed while the machine was off happens at the next boot. By 5 pm the day's play is usually over.

BeatLeader scores: /player/{id}/scores?sortBy=date&order=desc&time_from=<cursor-1d>. Overlap a day and skip ids you already have. New score ids → save the score JSON, the replay, the score stats, and a leaderboard refresh for that hash.
Attempts: skipped (see Decisions).
New maps: for every hash not in the archive yet, do phase 1, steps 6–9: BeatLeader leaderboard, then the BeatSaver JSON and zip (nothing to unpack). The game's own folders are only a manual fallback, since sync can't mount Windows. List unresolved hashes in the run summary.
Profile: one dated snapshot per run, plus /history.
Ratings drift: on Sundays' run, re-fetch /leaderboards/hash/{hash} for every map I've played, and store a new snapshot only when it changed. Upstream rebalances ratings (see the rebalanced-pp branch), and keeping the old snapshots preserves the history.
ScoreSaber main: sort=recent until reaching a known score id, then download the new replays at the slow pace. A day's worth fits easily in one run. Legacy is frozen; skip it.
Token upkeep: refresh the OAuth token at the start of each run and write the rotated refresh token to auth/beatleader_tokens.json (temp file plus rename) before doing anything else. A daily run stays far inside the 14-day window. If the refresh fails, the unit fails loudly and the message says to run auth login again. Alloy ships journal lines to Loki, so set up an alert there.
Once the archive has changed, run the importer incrementally, through a --since flag or by watching state.json. That includes the phase 5 merge for every leaderboard that got a new BL or SS score.

Optional later: a manual sync run (systemctl start plebleader-sync) for when I want results before 5 pm.

Check: play a map, and after the next 17:00 run the score, its replay and the map file show up on the local site, and the replay opens in ArcViewer with the local zip.

Phase 7: NixOS deployment on incineroar

Done 2026-11-09: merged into nix-pleb main (fast-forward to 3b3eb47), switched on incineroar, first sync clean, and BL .bsor replays play in ArcViewer. Issues hit after the switch:

Missing /run/secrets/plebleader.env. The file was encrypted only to litten's age key (the repo .sops.yaml rule lists only litten), so incineroar's host key couldn't decrypt it. Fixed by adding incineroar's host key and age1kuxd… with sops rotate --add-age. Any new incineroar sops file needs the same three recipients as sops.yaml.
First sync (13:48): the token refreshed, with the chain valid to 2026-11-23 (each daily refresh moves that forward), 0 new scores, and the importer loaded 3565 scores and 2552 songs in 11 s. The token now lives only in /var/lib/plebleader/auth/.
Website "TypeError: Cannot read properties of undefined (reading 'id')". utils/format.js takes the first host label as the leaderboard context when the host has more than 2 labels, so the context became incineroar and the navbar's context lookup found nothing. The phase 4 dev check missed it because a numeric last label (127.0.0.1) raises that limit to 4. Website fix e5f6279a: non-BeatLeader hosts always use general.
ArcViewer "Download failed! HTTP/1.1 403 Forbidden" for /cdn/replays/…bsor, and /cdn/maps/… redirected to BeatSaver even for archived zips. Caddy had been running since 2026-11-07. The switch added it to group plebleader but only reloaded it, so the process still had just caddy (grep Groups /proc/<pid>/status). Fixed with doas systemctl restart caddy. Any later change to Caddy's groups needs a restart too.

Phase 6 added a second commit on the branch: the plebleader.env sops secret, plebleader-sync (sync, then the importer) and its timer, and lock bumps for plebsaber-stream (d182fd8) and plebleader-server (dd94772). The plebsaber-stream bump also rebuilds beldum's site, with no change to the site code.

nix-pleb branch plebleader (2ada4b7, in the worktree ~/ops/nix-pleb-plebleader, based on main 981f9e7) adds the inputs plebleader-server (server 0481759) and plebleader-website (8d540642), both following nixpkgs-unstable, plus to incineroar/workstation.nix: user and group plebleader (with caddy in the group), plebleader-api, a one-shot plebleader-import, ArcViewer fetchFromGitHub and the 4442 vhost. The toplevel builds, and the system's Caddy adapts the generated Caddyfile. The sync unit and timer and the sops secret wait for phase 6's sync command. What differs from the bullets below:

Server fork: the private upstream submodules (beatleader-server-private, bs-simulator) are removed from .gitmodules, because a submodule-aware fetch fails on them. The flake sets self.submodules; the nix-pleb input uses ?ref=pleb/main&submodules=1 over SSH (the models fork's scp-style URL fetches fine). The package's version is a fixed 0.1.0: buildDotnetModule hands it to MSBuild, and a git short rev made restore fail with no message (NU1105 only in project.assets.json). 336 NuGet packages in deps.json. The package was tested by hand (importer and API on a scratch DB, all endpoints 200, wwwroot included).
Website: the yarn v1 flake from phase 4, not buildNpmPackage.
Units: WorkingDirectory=/var/lib/plebleader/app (data protection keys go to ../keys), StateDirectory for app, db, storage, maps, StateDirectoryMode=0750 and UMask=0027, so Caddy reads through the group and nothing is world-readable. Paths and PublicBaseUrl come from the fork's appsettings.Mirror.json, so the units only set ASPNETCORE_ENVIRONMENT=Mirror, ASPNETCORE_URLS and HOME.

To deploy (the user does this):

Merge plebleader into nix-pleb main. The main checkout has uncommitted flake.lock changes, so commit or stash those first, or cherry-pick the commit and rerun nix flake lock if flake.lock conflicts. Then sudo --preserve-env=SSH_AUTH_SOCK nixos-rebuild switch --flake .#incineroar. plebleader-api starts on empty databases.
Move the archive in (21 GB; /home and / are separate filesystems, so this is a copy): doas cp -a ~/ops/beatsaber/plebsaber.stream/.data/plebleader/archive /var/lib/plebleader/, then doas chown -R plebleader:plebleader /var/lib/plebleader/archive and doas chmod -R g+rX,o-rwx /var/lib/plebleader/archive.
Move the sync's token chain. Before 2026-11-23, or it expires and needs auth login again: doas install -d -o plebleader -g plebleader -m 0700 /var/lib/plebleader/auth and doas install -o plebleader -g plebleader -m 0600 ~/ops/beatsaber/plebsaber.stream/.data/plebleader/auth/beatleader_tokens.json /var/lib/plebleader/auth/, then delete the local copy. From then on, never run sync or auth refresh locally: the refresh token rotates, and whichever side refreshes second breaks.
doas systemctl start plebleader-sync (sync, then import), then open https://incineroar.brenise.dev:4442/u/pleb and a replay. Check journalctl -u plebleader-sync and systemctl list-timers plebleader-sync.
Afterwards, remove the worktree with git worktree remove ../nix-pleb-plebleader.

Packages: buildDotnetModule for the fork (generate deps.json with nix build .#…passthru.fetch-deps), buildNpmPackage for the website (start with npmDepsHash = lib.fakeHash, then pin the got: value), and fetchFromGitHub for ArcViewer deploy. The forks expose their packages through their own flakes and become flake inputs in nix-pleb, next to the existing plebsaber-stream input. Use git+https://gitea.satstack.dev/pleb/<repo>.git.
Services: put plebleader-api, plebleader-sync (service and timer) and the Caddy vhost in incineroar/workstation.nix. Don't create a new module file. Use a static user plebleader, StateDirectory=plebleader, and an EnvironmentFile from the sops secret.
Validate with nix build .#nixosConfigurations.incineroar.config.system.build.toplevel --no-link. The user does the switch.

Phase 8: Self-hosted BeatLeader web replay viewer

Upstream, the score row's replay icon opens BeatLeader's web viewer (BL_REPLAYS_URL, replay.beatleader.com), and turning on ArcViewer in the score settings adds a separate altReplay icon (Song/Icons.svelte, webPlayerLink(…, 'arcviewer', …)). Phase 4 pointed BL_REPLAYS_URL at /watch/, so both icons open ArcViewer now. The user wants the upstream behaviour back: the replay icon opens the BeatLeader viewer, and ArcViewer stays a separate icon.

Source: ~/src/BeatLeader/BeatSaber-Web-Replays (webpack 5, npm). Steps:

Fork it to gitea pleb/BeatSaber-Web-Replays with a pleb/main branch, and change as little as possible.
Resolve getApiUrl() (src/utils.js, src/open-replay-decoder.js) to /cors/blapi on non-BeatLeader hosts, the same rule as the website's IS_MIRROR. Check how it fetches the replay file and the score (link=, scoreId=) against the mirror's /cdn/replays/ and /cors/blapi/score/{id}.
Maps: zip-loader.js tries cdn.songs.beatleader.xyz, then /cors/beat-saver2/api/maps/… (a Netlify proxy), then r2cdn/cdn.beatsaver.com. Try /cdn/maps/<hash>.zip first, which already falls back to r2cdn in Caddy. Either proxy /cors/beat-saver2/ to api.beatsaver.com in Caddy or skip it.
Drop or stub analytics (googletagmanager), and check its other runtime fetches (fonts, cdn.jsdelivr.net). Discord and Patreon are plain links.
Add a flake with buildNpmPackage that outputs the built static files (index.html, assets, build, vendor, as in its predeploy script), and add it as a nix-pleb input.
Caddy: serve it at /replay/ on the plebleader vhost.
Website: in mirror mode set BL_REPLAYS_URL to /replay/ instead of /watch/. Leave the explicit ArcViewer links (altReplay, ReplayButton with webPlayer == 'arcviewer') on /watch/.

Check: on /u/pleb the replay icon plays a score in the BeatLeader viewer from local files, and with ArcViewer turned on in settings, the second icon plays it in ArcViewer.

Risks and fallbacks

The server doesn't build or run on Linux, or EF/SQLite can't translate queries. SQL Server is case-insensitive and SQLite isn't, and SQLite has no DateTimeOffset ordering. Fix it endpoint by endpoint, but only for the endpoints the profile, scores and leaderboard pages call (find them in the network tab). Fallback: skip the fork and write a small read-only API in this repo that serves the archive in the response shapes the website expects. That would also be the first step toward the full replacement.
The OAuth refresh chain breaks (more than 14 days without a run, a lost write, or upstream auth changes). The archive keeps everything collected so far. Sync stops, a failing unit makes that visible, and auth login re-seeds the chain.
Maps disappear from BeatSaver. That's why zips are archived as soon as a hash first appears. Old versions that are already gone stay listed in missing.json, which is acceptable (decided 2026-11-08). Cold storage might hold some; check someday, without much hope.
Upstream API changes. The archive stores raw responses, so a change breaks the importer, not the data. Version the importer's mapping.
Upstream code moves fast (the website is updated daily). Pin fork revisions. Rebase only when there's a reason to.
Ranks and pp go stale locally. That's by design: they're upstream's values from the last sync. The UI could show "as of ".

Open questions for the user

Settled on 2026-11-08: port 4442; access limited to incineroar and snorlax; map data limited to played maps; OAuth for auth; sync daily at 17:00 America/Los_Angeles.

Also settled: skip attempts and their replays entirely. Where BL and SS scores overlap, keep one per leaderboard by converting ScoreSaber scores to BeatLeader scores (phase 5).

Nothing is open right now.

Where the earlier work lives

~/ops/beatsaber/ss-scraper/: ScoreSaber replay downloader. The legacy account is done; it has no git commits yet.
~/ops/beatsaber/playlist-tool/src/clients/{beatleader,scoresaber,beatsaver}/: generated Python OpenAPI clients. Use them as a reference for the response models.
~/ops/beatsaber/backups/: UserData backups. They leave the replay folders out.
~/ops/beatsaber/beatleader/beatleader-website.code-workspace: points at ~/src/beatleader/... (lowercase). The checkouts are actually in ~/src/BeatLeader/, so fix that path.
~/src/BeatLeader/bl-dev-sandbox is a BSOR v2 format experiment, not a dev environment. Ignore it.

68 KiB Raw Blame History Unescape Escape