plebsaber.stream/docs/plan.md

Plan: a personal BeatLeader on incineroar

Goal: run my own BeatLeader instance on incineroar that holds only my profile (pleb) and play history, with ScoreSaber history merged in, and keep it in sync with upstream BeatLeader. The minimum viable version stays a fork of upstream rather than a rewrite. Replacing BeatLeader with my own system is a later project. This plan makes sure the data I collect is ready for that.

Written 2026-11-08. The numbers under "Facts" were checked live on that date.

Decisions

Topic	Decision
Server	Fork BeatLeader/beatleader-server (ASP.NET Core, net10.0). Swap SQL Server for SQLite and Cloudflare R2 for the local filesystem. Turn off every background job that recalculates ranks or pp.
Frontend	Fork BeatLeader/beatleader-website (Svelte 5 + rollup). Change as little as possible.
Replay viewer	Self-host ArcViewer's prebuilt WebGL build (the deploy branch) for both BeatLeader (.bsor) and ScoreSaber (.dat) replays. No Unity build.
Source of truth	A raw, append-only archive of unmodified API responses and replay files. The SQLite database can always be rebuilt from it, and so can any future replacement system.
Exporter and sync	TypeScript in this repo, building on src/lib/server/beatleader.ts, beatleaderAuth.ts, scoresaber.ts and beatsaver.ts. It writes only to the archive.
BeatLeader auth	OAuth 2.0 (profile offline_access). The sync owns its own chain of refresh tokens (scripts/archive/cli.ts auth login, seeded 2026-11-08). Unranked ratings do not need it: they come from the anonymous /leaderboards/hash/{hash} (see Facts).
Map files	Archive the BeatSaver zip of every map I've played, using beatsaver.ts. Don't unpack them ahead of time: ArcViewer loads the zip directly, and the server unpacks on demand only if a mirror code path ever needs map files (phase 2, step 9). Neither ever needs BeatSaver.
Sync schedule	Daily at 17:00 America/Los_Angeles, incineroar's timezone.
Access	Only incineroar and snorlax, through Caddy's remote_ip allowlist.
Map data scope	Only maps I've played: BeatLeader leaderboards and ratings, plus the BeatSaver metadata and zip.
Importer	C# inside the server fork, so it shares the EF models. It reads the archive and upserts into SQLite.
Upstream values	Keep them as they are: pp, ranks, star ratings, and the skill triangle. The local instance has one player, so recalculating would turn every rank into #1.
Attempts	Skipped. No failed or quit attempt replays, and no attempt records (/scoresstats). Leaving them out is less work: it's one endpoint the exporter simply doesn't call. Best scores, with their scoreImprovement, are the play history.
ScoreSaber merge	Convert ScoreSaber scores into BeatLeader Score rows. Each leaderboard keeps one score, chosen across BL, SS main and SS legacy by BeatLeader's own rule. BeatLeader's "improvement over previous best" display then covers ScoreSaber plays too. No separate table.
Identity	One local player: id 76561199407393962, name pleb. The legacy Steam id 76561197962107242 (i5u1d0r) becomes a linked alt of that player.
Port	https://incineroar.brenise.dev:4442. Caddy uses 4430–4441 on incineroar, so 4442 is the next free one (4444 is free too). The API listens on 127.0.0.1:8042, which is unused.

Facts (checked 2026-11-08)

My accounts

Source	Id	Scores	Replays
BeatLeader pleb	76561199407393962	3,565 best scores (2,492 unranked); 3,559 play count	one .bsor per score on cdn.replays.beatleader.com, about 1.3 MB each
BeatLeader attempts (not exported)	same	16,518 attempt records (/player/{id}/scoresstats), including fails and quits	non-finished attempts are at /otherreplays/<name>.bsor, auth-gated
BeatLeader legacy	76561197962107242 (i5u1d0r)	3
ScoreSaber main	76561199407393962 (pleb)	2,107 (814 ranked), 1,474 unique maps, first on 2023-11-17	1,034 have hasReplay
ScoreSaber legacy	76561197962107242 (i5u1d0r), inactive	199 (161 ranked), first seen 2022-05-07	58 have replays, already downloaded to ~/ops/beatsaber/ss-scraper/replays/76561197962107242/ (.dat plus a .json with metadata each)

The local mods keep replays too, as a fallback source. The Windows BSManager instance (~/Windows/Users/pleb/BSManager/BSInstances/<ver>/, mounted with mountwin on incineroar) has UserData/BeatLeader/Replays/ and UserData/ScoreSaber/Replays/. ScoreSaber keeps one file per player/hash/difficulty/characteristic, so it can recover some scores whose server copy has hasReplay: false. ~/ops/beatsaber/backups deliberately leaves these folders out.

Unranked ratings and auth (corrected 2026-11-08)

• Every difficulty object has stars, passRating, accRating, techRating (the skill triangle), predictedAcc, plus modifiersRating with ss*/fs*/sf*/bfs*/bsf* variants of each rating.
• On score lists (/player/{id}/scores) and /leaderboard/{id}, unranked ratings are filled in only when the requesting user has profileSettings.showAllRatings == true. That check uses HttpContext.CurrentUserID, which reads only the beatleader.com cookie session (with an Issued claim). An OAuth bearer token does not count: tested, bearer and anonymous both return stars/pass/acc/techRating: null.
• GET /leaderboards/hash/{hash} returns full ratings for unranked difficulties, anonymously, including all modifiersRating variants. Tested on an unranked ExpertPlus (9db01c9b…): stars 8.33, pass 3.92, acc 9.75, tech 7.70. Phase 1, step 6 already fetches this for every played hash, so the importer takes difficulty ratings from these snapshots, not from the nulls embedded in the scores.
• Bearer tokens only work on /oauth2/identity, clan management and ReeSabers presets (CurrentOauthUserID). GET /user with a bearer token is 401. The exporter doesn't actually need auth; the token chain is kept for identity checks and in case a cookie-gated endpoint turns out to matter.
• The sync's chain lives in scripts/archive/auth.ts: login via the loopback redirect http://localhost:5173/auth/beatleader/callback (registered on the OAuth app, as is https://plebsaber.stream/auth/beatleader/callback), token store <PLEBLEADER_DIR>/auth/beatleader_tokens.json (default .data/plebleader/, mode 0600). auth status checks identity and ratings.
• Token lifetimes: access tokens last 3,600 s and refresh tokens 14 days. Each refresh rotates the refresh token and invalidates the old one. So the sync job needs its own refresh token, separate from the plebsaber.stream site's, or the two invalidate each other. It persists every new refresh token atomically before using the new access token. The daily sync stays well within the 14 days. If the chain breaks (no run for 14 days, or a failed write), log in once more through plebsaber.stream's /auth/beatleader/login and seed the new token.

BeatLeader endpoints the exporter uses

Base https://api.beatleader.com. Rate limit: 50 GETs per 10 s overall, 10 per 10 s on /players, 8 per second on /maps. Swagger is at /swagger/index.html.

Data	Endpoint	Notes
identity and settings	GET /user	confirms the token works (player.id)
profile	GET /player/{id}?stats=true	scoreStats, history, pinned scores, badges
best scores	GET /player/{id}/scores?count=100&page=N&sortBy=date&order=desc&includeIO=true	count ≤ 100. Each score includes leaderboard.song and leaderboard.difficulty (with ratings), replay, offsets, scoreImprovement. Incremental runs use time_from=<unix>.
replay stats	GET /score/statistic/{scoreId}	the accuracy graph, hit tracker and so on, from the server's scorestats storage
map data	GET /leaderboards/hash/{hash}	every difficulty of a song, with ratings when the request is authenticated
rank history	GET /player/{id}/history, /history/triangle, /accgraph
live feed (optional)	wss://sockets.api.beatleader.com/scores	all scores globally. Filter on my id for near-real-time sync.

BeatSaver (public, no auth): BeatSaverAPI.getMapByHash(hash) in src/lib/server/beatsaver.ts calls GET https://api.beatsaver.com/maps/hash/{hash}. Each entry in versions[] carries its own hash and downloadURL (the zip). Each map version has its own hash, and scores refer to a version hash, so download the version whose hash matches. Old versions may only be on the CDN or nowhere at all. When a version or map is gone from BeatSaver, take it from the game's own folders on the Windows side instead: Beat Saber_Data/CustomLevels and CustomWIPLevels in the BSManager instance. The folders there are unpacked maps, so compute their hash the way the game does: SHA-1 over Info.dat plus each difficulty file, in order.

The server's MapsPath (MapPostprocessor/MapDownloader.cs) wants <MapsPath>/<hash>/ folders containing an unpacked map with Info.dat. It checks the lowercase and uppercase hash. Only when a folder is missing does it fetch beatsaver.com/api/maps/hash/{hash} and unpack the zip.

ScoreSaber (public, no auth): GET https://scoresaber.com/api/player/{id}/full, GET /api/player/{id}/scores?limit=100&page=N&sort=recent (each item has score.hasReplay), and GET /api/v2/scores/{scoreId}/replay (.dat). The ScoreSaber API doesn't send CORS headers, so it only works server-side. ~/ops/beatsaber/ss-scraper/download_replays.py already handles pacing (30 s minimum between replays), 429 backoff, and resuming. That pacing is outdated: as of 2026-11-08 ScoreSaber sends x-ratelimit-{limit,remaining,reset}-{short,medium,long} headers. The replay endpoint allows 20 (short), 60/min (medium) and 360/hour (long, the one that binds); player endpoints allow 25/100/400. scripts/archive paces replays at 10.5 s and pauses whenever a window's remaining count hits 0, so 1,034 replays take about 3 h instead of 9.

Upstream server: what the fork has to change

These checkouts are in ~/src. Run git pull before relying on them.

• ~/src/BeatLeader/beatleader-server (master was at 2026-07-19 1474312; the live branch rebalanced-pp was updated 2026-10-25). Its public submodules are initialized: beatleader-server-models, beatleader-parser, ReplayDecoder, MapPostprocessor, RatingAPI. Two submodules are private and unreachable: beatleader-server-private and bs-simulator. Upstream already copes with that. Startup.cs falls back to ReplayRecalculatorStub, and nothing in the public code imports a namespace that only the private repos define. Verified 2026-11-08: it builds on Linux with nixpkgs dotnet-sdk_10 (10.0.403) with both private folders empty: 0 errors, about 2,300 warnings, about 1 minute.
• Database: UseSqlServer for two contexts (Startup.cs:380-386): AppContext (DefaultConnection) and StorageContext (StorageConnection, which holds PlayerLeaderboardStats). There are 570 SQL Server migrations.
• Z.EntityFramework.Extensions.EFCore is commercial, with a monthly trial. Call sites: about 92 BulkSaveChanges, 56 BulkUpdate, 19 BulkDelete, 3 BulkInsert.
• Raw SQL in 5 places: DBCC CHECKIDENT (Services/DailyRefresh.cs), TABLESAMPLE (ScoreController.GetRandomScore), TRUNCATE and admin statements (Controllers/Admin/AdminController.cs).
• Storage: all 31 helpers in Utils/S3Helper.cs go to Cloudflare R2 (S3AccountID.r2.cloudflarestorage.com) and return https://cdn.{container}.beatleader.com/{file}. Containers: replays, otherreplays, assets, previews, scorestats, playlists, unicode, configs, replayedvalues, reepresets, songs, attempts. Nothing calls S3 outside this file.
• Background jobs (Startup.cs:455-478): EveryMinuteRefresh, MinuteRefresh, HourlyRefresh, DailyRefresh, HistoryService, BotService, RankingService, SongService, RichPresenceService and MapRefresh run only when ServicesHost=YES. These always run: ConstantsService, RefreshTaskService, ClanTaskService, ClanMessageService, ClanCallbackService, SearchService (Lucene), OauthService, LeaderboardPlayerStatsService.
• Config keys, all from IConfiguration: DefaultConnection, StorageConnection, S3*, MapsPath (default /home/maps), SteamKey, SteamApi, BeatSaverId/Secret, Openiddict* (production expects ../keys/openiddict_certificate*.pfx; development uses dev certs), plus optional OAuth secrets for Patreon, Twitch, Discord and others. Those providers register only when their keys are set.
• The star-rating models are public: RatingAPI/ (C# with ONNX models) and beatleader-analyzer. We don't need them for the minimum version, because we keep upstream's ratings. They are what would let a later replacement rate maps itself.

Upstream website

~/src/BeatLeader/beatleader-website (2026-11-07). In src/network/queues/beatleader/api-queue.js, on any host it doesn't recognize the API base is /cors/blapi/, which is relative to the page. So our host only needs Caddy to proxy /cors/blapi/* to the API, with no code change. It rewrites assets to /cors/cdn-assets-bl/ and /cors/cdnbeatsaver/ in the same way. BL_REPLAYS_URL and BL_ANALYZER_URL fall back to the public replay.beatleader.com and need a patch. Build with npm run build (rollup) into public/.

ArcViewer

~/src/AllPoland/ArcViewer (Unity 6000.0.67f1). The deploy branch on GitHub holds a ready WebGL site (index.html, Build/, TemplateData/). It loads replays from ?replayURL=<url> and recognizes ScoreSaber replays by their magic header (ScoreSaberDecoder.IsScoreSaberReplay, including the legacy format), so it works for both .bsor and .dat. The map comes from id=<bsr> (BeatSaver) or url=<zip>. Serving it from the same origin as the replays avoids CORS and mixed-content problems.

incineroar

• Caddy vhosts are in ~/ops/nix-pleb/systems/brenise.dev/incineroar/workstation.nix. They follow the pattern "${host_fqdn}:44xx" with ${tlsConfig} (Namecheap DNS-01) and a remote_ip allowlist (127.0.0.1/32 ::1/128 192.168.1.34/32 192.168.1.35/32 192.168.1.36/32 10.19.21.34/32). The host firewall is off; Caddy does the gating.
• Disk: / has 479 GB free and /home 684 GB. The archive should be 10–25 GB: about 4.6 GB of best-score replays, about 1.4 GB of ScoreSaber replays, and BeatSaver zips for roughly 2,000–3,000 played maps (ScoreSaber alone has 1,474 unique hashes), at a few MB each. Actual (2026-11-08): 21 GB with 1,897 zips at about 8 MB each. Fetching the remaining ~630 older maps later adds about 5 GB.
• Allowed clients: incineroar (127.0.0.1/32, ::1/128, 192.168.1.36/32) and snorlax (192.168.254.21/32, the address the amelia fallback vhost already allows).
• Timezone: America/Los_Angeles.
• nix-pleb rule: put host app config in an existing <system>/<purpose>.nix (here workstation.nix), not a new module file. Never use nixos-rebuild switch without asking; use build. Never decrypt sops secrets: ask the user to add the values.

Layout

plebsaber.stream/                      (this repo)
  docs/plan.md                         this file
  scripts/archive/                     exporter and sync CLI (TypeScript, node ≥ 22)
  src/lib/server/beatleader.ts         extend: bearer auth from the sync's token store, paging, time_from, raw passthrough
  src/lib/server/scoresaber.ts         extend: player scores, replay download

gitea pleb/beatleader-server           fork, branch `pleb/main` (based on upstream `rebalanced-pp`)
  MIRROR.md                            what MirrorMode changes and how to run it
  appsettings.Mirror.json              incineroar values (ASPNETCORE_ENVIRONMENT=Mirror)
  Tools/ArchiveImporter/               C# console app sharing AppContext/StorageContext
gitea pleb/beatleader-server-models    fork of the models submodule, branch `pleb/main`
gitea pleb/beatleader-website          fork, branch `pleb/main`

/var/lib/plebleader/                   on incineroar, user `plebleader`
  archive/                             source of truth, append-only
    beatleader/
      user.json                        latest /user (no secrets)
      players/<id>/profile/<YYYY-MM-DD>.json
      players/<id>/history.json, triangle.json, accgraph.json
      scores/<scoreId>.json            one ScoreResponse each, unmodified
      scorestats/<scoreId>.json
      leaderboards/<hash>/<YYYY-MM-DD>.json   new snapshot only when the content changed
      replays/<original filename>.bsor
    scoresaber/<playerId>/
      player/<YYYY-MM-DD>.json
      scores/<scoreId>.json            the full item (score plus leaderboard)
      replays/<scoreId>.dat
    beatsaver/
      maps/<hash>.json                 getMapByHash response, unmodified
      zips/<hash>.zip                  the map version matching that hash
      missing.json                     hashes BeatSaver no longer serves, with their source if recovered
    game-levels/<hash>/                maps recovered from the Windows CustomLevels when BeatSaver lacks them
    local-mods/{beatleader,scoresaber}/...   copied from the Windows UserData
    state.json                         sync cursors (last timepost per source/feed)
  db/app.db, db/storage.db             SQLite, rebuilt from the archive
  storage/                             S3 replacement (containers as dirs); replays/ and ssreplays/ symlink into archive/
  maps/<hash>/                         MapsPath: unpack cache, filled on demand from archive zips (phase 2, step 9)
  auth/beatleader_tokens.json          the sync's own OAuth token chain (rotates on refresh)
  secrets → EnvironmentFile from sops  BL_CLIENT_ID, BL_CLIENT_SECRET

Phases

Each phase ends with a check. Phase 1 protects the data and doesn't depend on the fork, so do it first.

Phase 0: Prep

1. Done 2026-11-08 (all were already current; upstream server master 1474312, rebalanced-pp 52a50ed, website 2b3ef891, ArcViewer deploy c776256, ScoreSaber website 836d789). Update the checkouts: git -C ~/src/BeatLeader/beatleader-server pull --recurse-submodules, and the same for beatleader-website, ~/src/AllPoland/ArcViewer (also git fetch origin deploy), and ~/src/ScoreSaber/website. The ScoreSaber website is public now; use it to check the score and replay API shapes.
2. Done 2026-11-08. Create the Gitea forks pleb/beatleader-server and pleb/beatleader-website with the gitea-api skill (mirror or push). Keep upstream as a remote so you can rebase later. Created as private, non-mirror imports through POST /repos/migrate (the Gitea token lacks write:user, which POST /user/repos needs), with default branch pleb/main. The ~/src checkouts keep origin = GitHub and gained a gitea remote; pleb/main is checked out there. The server's pleb/main starts from rebalanced-pp, which is master plus 4 commits and what the live API most likely runs, so the entities match the archived responses. pleb/beatleader-server-models was forked too (see phase 2, step 7); the server's .gitmodules points at it over SSH.
3. Bring this repo's flake up to date (it's still on nixos-25.05) only if the exporter gets packaged here.
4. Ask the user to add a sops secret plebleader.env for incineroar, holding BL_CLIENT_ID and BL_CLIENT_SECRET. These are the same OAuth app as plebsaber.stream; check that its registered callback URLs include the one used for seeding. Agents never decrypt it. For local runs before deployment, use the gitignored .env in this repo.
5. Done 2026-11-08 (scripts/archive/cli.ts auth login; refresh rotation tested; chain expires about 2026-11-22 unless refreshed). Seed the sync's token chain. Add a small scripts/archive command (auth login) that runs the same authorization-code flow as /auth/beatleader/login with scope=profile offline_access. It catches the callback on a loopback port, or reuses a plebsaber.stream callback URL the OAuth app already allows. It writes auth/beatleader_tokens.json. Don't copy the plebsaber.stream site's own refresh token, because rotation would break whichever side refreshes second.

Phase 1: Full export (one-time backup)

Scope (decided 2026-11-08): all player data for both accounts, all history: profiles, BL scores, replays, score stats, SS scores and replays, plus BL leaderboard JSON (ratings) for every played hash. Only BeatSaver map files are limited to maps played in the last two years (--maps-since 2024-11-08, DEFAULT_MAPS_SINCE in scripts/archive/config.ts). Exception: legacy ScoreSaber scores that have an archived replay get their map fetched whatever its age, so those replays render (58 replays, 53 maps; mapHashes() in cli.ts). Legacy scores without a replay don't get a map. Older map zips come later with export maps --maps-since all; existing files are skipped, so that only fetches what's missing. Replays of older scores won't render a map in ArcViewer until then. SS main has 1,034 replays (about 3 h at 10.5 s each). The 58 SS legacy replays are copied from ss-scraper's download instead of being fetched again.

Commands: node scripts/archive/cli.ts export bl|ss|ss-replays|maps|all [--since …] and report. The archive lives in .data/plebleader/archive/ (set PLEBLEADER_DIR to change the root) until /var/lib/plebleader exists on incineroar; then move it. Logs go in .data/plebleader/logs/. Profile, history, triangle and accgraph are stored as dated snapshots in players/<id>/<kind>/<YYYY-MM-DD>.json, written only when changed.

Build scripts/archive/ as a small CLI (export bl, export ss, export maps, import-local-mods, sync). Rules:

• Store responses byte-for-byte. Never reshape them on the way in.
• Make every step resumable: skip files that already exist, keep cursors in state.json, and write to temp files before renaming.
• Respect rate limits. Use about 4 requests per second to BeatLeader and honor Retry-After (the existing requestWith429Retry handles 429s). For ScoreSaber replays, stay under the 360/hour limit (10.5 s pacing) and honor the x-ratelimit-* headers.
• Send User-Agent: plebsaber-archive/0.1. Don't put an email address in it.

Steps:

1. Auth check. node scripts/archive/cli.ts auth status: /oauth2/identity must return 76561199407393962, and /leaderboards/hash/{hash} must return non-null stars for an unranked difficulty. If either fails, stop.
2. BeatLeader scores. Page through /player/{id}/scores with sortBy=date&order=desc&count=100&includeIO=true until it's exhausted (about 36 pages). Do the same for the legacy id (3 scores).
3. Attempts: skipped. Don't call /scoresstats or /otherreplays.
4. Replays. Fetch every scores/*.replay from the CDN (no auth, CORS *). Keep the original filenames.
5. Score stats. GET /score/statistic/{id} for each best score.
6. BeatLeader map data. Collect the set of played hashes: every unique song.hash in BL scores, plus leaderboard.songHash in both ScoreSaber accounts. Normalize them to lowercase. For each, call GET /leaderboards/hash/{hash} (no auth needed). That gives ratings and the triangle for every difficulty, ranked or not. This is the only source of unranked ratings; the scores from step 2 have them nulled.
7. Map files from BeatSaver. For each played hash, call BeatSaverAPI.getMapByHash(hash) and save the response to beatsaver/maps/<hash>.json. Then download the zip for the versions[] entry whose hash matches, to beatsaver/zips/<hash>.zip. Extend beatsaver.ts with a downloadMapZip(version) that streams to a temp file, checks it's a valid zip containing Info.dat, then renames it. Go about 2 requests per second and honor 429s. Write each hash that can't be resolved (deleted map, or an old version gone from the CDN) to missing.json.
8. Recover missing maps from the game: optional, low priority. Decided 2026-11-08: maps gone from BeatSaver are accepted as missing. They were most likely updated or removed for a reason. The user's online backups and local map archives from before 2026 don't have them. Older cold storage might, so that's a "someday" check, not a task. The current BSManager instance's CustomLevels may hold a few recent ones and is cheap to try: with mountwin, hash every folder in Beat Saber_Data/CustomLevels and CustomWIPLevels, and copy those whose hash is in missing.json to game-levels/<hash>/. Whatever is still missing stays listed; those replays just can't render the map.
9. Unpack for the server: dropped 2026-11-08, replaced by on-demand unpacking (phase 2, step 9). ArcViewer loads url=/cdn/maps/<hash>.zip (Caddy serves the archive's zips) instead of id=<bsr> from BeatSaver, so it never needs unpacked folders.
10. Profile and history. Save /player/{id}?stats=true, /history, /history/triangle, /accgraph, and /user (strip anything secret).
11. ScoreSaber main. Page through all scores (22 pages), then download the 1,034 replays. Either port the download_replays.py logic to TypeScript or run it as-is with -o archive/scoresaber, and normalize the layout.
12. ScoreSaber legacy. Copy the 58 replays and the metadata from ~/ops/beatsaber/ss-scraper/replays/76561197962107242/ into archive/scoresaber/76561197962107242/ (the filenames start with the scoreId). Fetch the 199 score items again so the archive has complete, unmodified responses.
13. Local mod replays. With the Windows partition mounted (mountwin, then umount ~/Windows afterwards), copy UserData/BeatLeader/Replays and UserData/ScoreSaber/Replays into archive/local-mods/. Afterwards, match the ScoreSaber files against scores that have hasReplay: false by player/hash/difficulty/characteristic.

Check: the counts match the live API: 3,565 BL scores, 3 legacy, 2,107 plus 199 SS scores, and replay files for every score with a replay URL or hasReplay. Every played hash should have a BeatLeader leaderboard snapshot, plus a BeatSaver zip or recovered game folder, unless it's listed in missing.json. Write a report subcommand that prints counts and what's missing, and paste its output into this file.

Report, 2026-11-08 (phase 1 complete). The ScoreSaber replays finished with zero 429s.

Window:  scores since all time, BeatSaver maps since 2024-11-08
BeatLeader scores        3568 in window (3568 archived total, both accounts)
BeatLeader replays       3568/3568
BeatLeader score stats   3568/3568
ScoreSaber 76561199407393962: 2107 scores, replays 1034/1034
ScoreSaber 76561197962107242: 199 scores, replays 58/58
Played hashes            2560
  BL leaderboard snapshot 2552/2560
  BeatSaver zip           1897/1926 in maps window (missing.json lists 29)
Archive size             21.78 GB

Steps 8 (game-folder recovery) and 13 (local mod replays) are still open. Step 9 was dropped in favour of on-demand unpacking.

The 29 missing hashes are all "hash unknown to BeatSaver", and none belongs to a legacy score with a replay (all 53 of those maps are archived). The 8 hashes without a BL leaderboard are ScoreSaber-only maps BeatLeader never saw. Zips average about 8 MB (15 GB for 1,897), more than the estimate below.

Phase 2: Server fork on SQLite

Done 2026-11-08 (pleb/main at 1741ce9, models fork at 8f233e8). What differs from the steps below:

• Step 2: Microsoft.Data.SqlClient stays as a direct package, because one controller catches SqlException. One controller had an unused using BeatLeader_Server.Migrations; that broke the build once the old migrations were gone. The fresh migrations create 155 tables and 217 indexes. WAL and migrations are applied at startup by Utils/MirrorSetup.cs; busy_timeout comes from Default Timeout=30 in the connection string.
• Step 3: Utils/BulkCompat.cs sits in the global namespace, like Z's own extensions, because some callers have no EF using. 49 of the BulkUpdate calls pass ColumnInputExpression, often on detached stubs that carry only the key and those columns, so the shim marks only the listed columns modified and detaches the stubs after saving. A plain UpdateRange would have wiped every other column. Not exercised at runtime yet; the importer will be the first real user.
• Step 4: the two index-defragmentation scripts became PRAGMA optimize and REINDEX; ANALYZE;.
• Step 5: GetS3Client returns a LocalStorageClient (an AmazonS3Client subclass) when StoragePath is set; the six S3 primitives branch on it. Upstream behaviour is unchanged without StoragePath.
• Step 6: MirrorMode also skips RefreshTaskService and MapRefresh, and the external OAuth providers. Two surprises showed up in testing: GetLazy fetched unknown players from Steam (it now returns null, so 404), and the BeatSaver login is registered unconditionally and validated on every request (it gets placeholder credentials, otherwise every request was a 500).
• Step 7: instead of patching about 35 showRatings decisions, the four HideRatings() leaves return early when MirrorSettings.ShowAllRatings (in the models fork) is set; MirrorSetup sets it. Sorting and filtering by stars on unranked maps still follow the requesting user's settings; patch Utils/{Map,Leaderboard}ListUtils/Sort.cs if that matters.
• Step 8: appsettings.Mirror.json loads with ASPNETCORE_ENVIRONMENT=Mirror; MIRROR.md in the fork documents it. OpenIddict uses ephemeral keys. Run the DLL with --contentRoot <bin> from a working directory in the state dir, because data protection keys go to ../keys/. dotnet run ignores ASPNETCORE_URLS in favour of launchSettings.json.

Check result: on empty files, /swagger/index.html 200, /player/76561199407393962 404, /players and /leaderboards 200 with empty pages, no fail: lines in the log.

Work on the pleb/main branch of pleb/beatleader-server. Keep each change small and in its own commit, so upstream rebases stay manageable.

1. Build on Linux before changing anything: nix shell nixpkgs#dotnet-sdk_10 -c dotnet build BeatLeader-Server.csproj (nixpkgs has SDK 10.0.403). Record which errors come from the missing private submodules versus the platform. Delete or exclude the empty beatleader-server-private and bs-simulator folders, and stub whatever they turn out to supply.

2. SQLite. Replace Microsoft.EntityFrameworkCore.SqlServer with ...Sqlite, and point both AddDbContextFactory calls at UseSqlite (Data Source=/var/lib/plebleader/db/app.db and storage.db). On connection open, set PRAGMA journal_mode=WAL; busy_timeout=5000;. Move the 570 SQL Server migrations aside and generate one fresh migration per context (dotnet ef migrations add InitialSqlite --context AppContext, and the same for StorageContext), so later schema changes still work.

3. Drop Z.EntityFramework.Extensions. Add Utils/BulkCompat.cs with extension methods of the same names and shapes: BulkSaveChanges → SaveChanges, BulkUpdate → UpdateRange + SaveChanges, BulkDelete → RemoveRange + SaveChanges, BulkInsert → AddRange + SaveChanges. With one player, performance doesn't matter, and about 170 call sites keep compiling unchanged.

4. Raw SQL. DBCC CHECKIDENT → DELETE FROM sqlite_sequence WHERE name IN (...). TABLESAMPLE → ORDER BY RANDOM() LIMIT 1. TRUNCATE → DELETE FROM.

5. Storage. Rewrite the bodies of the 31 S3Helper methods to read and write ${StoragePath}/{container}/{filename}, and return ${PublicBaseUrl}/cdn/{container}/{filename}. GetS3Client can return a client that's never used, so the method signatures don't change.

6. Background jobs. Leave ServicesHost unset. Add a config flag (MirrorMode=true) that also skips registering RankingService, the clan services, OauthService and LeaderboardPlayerStatsService. Keep SearchService and ConstantsService if they start cleanly, and point Lucene's index directory at the state dir.

7. Auth. For the minimum version the site is read-only behind Caddy's IP allowlist. No login is needed: the website shows a public profile. If the ratings in API responses still depend on ShowAllRatings for the requesting user, then in MirrorMode treat requests as coming from 76561199407393962, or just store ratings so they always show. That means patching the showRatings checks, which all read ProfileSettings.ShowAllRatings, to true.

8. Configuration. Put an appsettings.Production.json template in the fork and the real values in the NixOS unit's environment (ASPNETCORE_URLS=http://127.0.0.1:8042, ConnectionStrings/DefaultConnection, StoragePath, PublicBaseUrl, MapsPath, MirrorMode). Use development OpenIddict certs, or generate a self-signed pfx into the state dir.

9. On-demand map unpacking (deferred; do it only when needed). MapDownloader.Map(hash) (in the MapPostprocessor submodule) returns MapsPath/<hash>/ when it exists and otherwise downloads from BeatSaver. Add a step before the download: if <MapZipsPath>/<hash>.zip exists (the archive's beatsaver/zips/), extract it to a temp folder under MapsPath, check it contains Info.dat, rename it to <hash>/ and return that. Accept upper- and lowercase hashes, like Map does. In MirrorMode, never fall through to BeatSaver: return null instead. That makes MapsPath a cache that can be deleted at any time.

   • Technical debt: low. It's about 20 lines and changes no interfaces. The catch is where it lives: MapDownloader is in MapPostprocessor, so editing it means a fourth fork. Its methods aren't virtual, so they can't be overridden. Without a fork, add a small Utils/LocalMapZips.EnsureUnpacked(hash) in the server and call it before _downloader.Map(...) at the three call sites (ReplayController, EarthDayController, MapRefresh). That's a few extra lines per rebase and no submodule fork, so prefer it.
   • Why deferred: no read path of the mirror calls MapDownloader today. Its callers are MapRefresh (off in MirrorMode), the Earth Day event controller, and replay upload and recalculation in ReplayController. It becomes necessary if the mirror ever recalculates score stats (for example the optional .dat → .bsor conversion in phase 5) or accepts replay uploads.

Check: the server starts on an empty SQLite file, /swagger/index.html loads, and GET /player/76561199407393962 returns 404 rather than 500.

Phase 3: Importer

Tools/ArchiveImporter lives in the fork, references the main project, and uses its AppContext/StorageContext. Run it as a one-shot that is idempotent and upserts by upstream id.

• beatleader/scores/*.json (ScoreResponse) → Song, DifficultyDescription (keep the upstream ratings and ModifiersRating as they are), Leaderboard, Score (with upstream Id, Rank, Pp, Weight, and Timepost), ScoreImprovement, ReplayOffsets. Rewrite Replay to the local /cdn/replays/<file>.
• leaderboards/<hash>/latest updates the difficulty ratings, so maps you played but have no best score for still get them.
• beatsaver/maps/<hash>.json fills in Song fields that BeatLeader leaves out (BeatSaver key, description, tags, upload date). The server doesn't read map files on any mirror path; if one appears, phase 2, step 9 unpacks them on demand.
• players/<id>/profile/latest → Player, PlayerScoreStats, ProfileSettings, plus history rows from /history. Insert the legacy id as an AccountLink or alt pointing to the main player, so PlayerIdToMain resolves /u/76561197962107242 to pleb. Its 3 BL scores go through the phase 5 merge like any other candidate.
• StorageContext (attempts) stays empty, because attempts aren't exported.
• scorestats/*.json → the scorestats storage container, using the filename the server expects (check ScoreController.GetStatistic).

The response DTOs aren't the entities, and this mapping is the largest single piece of work. Use beatleader-server-models/Models/* (entities) and ResponseUtils.cs (entity → DTO) as the reference, working backwards.

Check: the stock website (phase 4) shows my profile with the right score count, the pp and ranks from upstream, unranked star ratings, and the triangle on the profile.

Phase 4: Website, viewer, Caddy

1. In the website fork, point BL_REPLAYS_URL (and the "watch replay" links) at /arcviewer/?replayURL=<encoded local replay url>&url=<encoded /cdn/maps/<hash>.zip>. Fall back to id=<bsr> only for hashes that have no local zip. Hide or stub the features that can't work offline: clans, events, friends, the Socket page, and the stage analyzer and ppai2 calls in StarRatingGraph.svelte / star-generator.js. Leave the /cors/blapi/ resolution as it is.

2. Package the ArcViewer deploy branch with fetchFromGitHub at a pinned revision, as static files.

3. Add a Caddy vhost in incineroar/workstation.nix:

   "${host_fqdn}:4442" = {
     # Personal BeatLeader mirror (plebleader). Website + ArcViewer static,
     # API on 8042, replays/assets/maps from the state dir.
     # Only incineroar itself and snorlax (192.168.254.21).
     extraConfig = ''
       ${tlsConfig}
       @plebleader_allowed remote_ip 127.0.0.1/32 ::1/128 192.168.1.36/32 192.168.254.21/32
       handle @plebleader_allowed {
         handle_path /cors/blapi/* { reverse_proxy http://127.0.0.1:8042 }
         handle_path /cdn/maps/* { root * /var/lib/plebleader/archive/beatsaver/zips; file_server }
         handle_path /cdn/* { root * /var/lib/plebleader/storage; file_server }
         handle_path /arcviewer/* { root * ${arcviewer}; file_server }
         handle_path /cors/cdn-assets-bl/* { reverse_proxy https://cdn.assets.beatleader.com { header_up Host {upstream_hostport} } }
         handle_path /cors/cdnbeatsaver/* { reverse_proxy https://cdn.beatsaver.com { header_up Host {upstream_hostport} } }
         handle { root * ${plebleader-website}; try_files {path} /index.html; file_server }
       }
       handle { respond "Forbidden" 403 }
     '';
   };
   

   ArcViewer's WebGL build may need Content-Encoding headers for .br/.gz files in Build/. Check which ones the deploy branch ships.

Check: in Chromium, https://incineroar.brenise.dev:4442/u/pleb shows the profile and scores, and a replay opens in ArcViewer.

Phase 5: Convert ScoreSaber scores and merge

ScoreSaber scores become BeatLeader Score rows, and each leaderboard keeps exactly one score. BeatLeader already shows how a score compares with the previous best (ScoreImprovement), so after the merge that comparison covers ScoreSaber plays too. This is importer logic in the fork. The archive keeps the raw ScoreSaber responses as they are.

1. Provenance columns. Add nullable Source (null = BeatLeader, "scoresaber") and SourceScoreId / SourcePlayerId to Score. This goes into the fresh SQLite migration from phase 2. The website uses them for a small "SS" badge, and the importer uses them to stay idempotent.
2. Convert each ScoreSaber item (scoresaber/*/scores/*.json) into a candidate Score:
   • Leaderboard: leaderboard.songHash (lowercased), plus leaderboard.difficulty.difficulty. ScoreSaber uses 1/3/5/7/9 for Easy…ExpertPlus, the same numbers as BeatLeader's difficulty.value. gameMode SoloStandard → mode Standard (strip Solo). Resolve it to the BL leaderboard from leaderboards/<hash> (phase 1, step 6). If BL has none, create Song/DifficultyDescription/Leaderboard from the BeatSaver JSON, with no ratings.
   • BaseScore, ModifiedScore, Modifiers (the same comma-separated codes), MissedNotes, BadCuts, FullCombo, MaxCombo, Timeset/Timepost come from timeSet. Set Platform = "scoresaber" and Hmd = unknown, unless the codes are mapped.
   • Accuracy = BaseScore / maxScore, taking maxScore from the BL difficulty. ScoreSaber reports maxScore: 0 on many unranked leaderboards. Check this against how ReplayUtils computes BL accuracy.
   • Pp: ReplayUtils.PpFromScore(score, difficulty), using the archived BL ratings, when the BL difficulty is ranked; otherwise 0, as BeatLeader does. ScoreSaber's own pp and rank aren't comparable, so keep them only in the archive. Set Rank to 0, and have the website show "—" for ScoreSaber-sourced rows.
   • Id = 1_000_000_000 + ssScoreId. ScoreSaber ids are about 95M and upstream BL ids about 151M, so this can't collide and still fits in an int32.
   • Replay = /cdn/ssreplays/<ssScoreId>.dat when the archive has one (from the server, or recovered from local mod replays); otherwise null. No ReplayOffsets and no score stats, because the server can't decode .dat.
3. Merge per leaderboard. The candidates are the upstream BL best, the ScoreSaber main best, the ScoreSaber legacy best, and the legacy BL score. Sort them by time and fold them with BeatLeader's own ReplayUtils.IsNewScoreBetter(old, new): compare by pp when either score has pp, otherwise by modified score, where a no-fail score loses to a clean one. The winner becomes the leaderboard's Score.
   • ScoreImprovement on the winner is the difference from the best before it in the fold: score, accuracy, pp, misses, bad cuts, modifiers, timeset. If the winner is the upstream BL score and nothing from ScoreSaber came between it and its previous BL best, keep upstream's scoreImprovement as it is.
   • Candidates that lose stay in the archive only.
4. Player totals. Keep upstream's PlayerScoreStats, pp and rank. They won't count ScoreSaber scores that won. Recomputing the weighted pp total locally is an optional later step; label the profile numbers "BeatLeader, as of ".
5. Website. Show the "SS" badge on ScoreSaber-sourced scores, hide the accuracy graph and stats for them (there are no score stats), and send their replay button to /arcviewer/?replayURL=/cdn/ssreplays/<id>.dat&url=/cdn/maps/<hash>.zip. Every score shows under the name pleb; i5u1d0r survives only as SourcePlayerId.

Optional later: convert .dat into .bsor, using ArcViewer's ScoreSaberDecoder as the reference for the format. The server's statistics code could then produce score stats for ScoreSaber-sourced scores too.

Check: the leaderboard count is the union of the BL and SS leaderboards I played, with exactly one score each. The importer prints how many winners came from each source. Spot-check a map where an old legacy ScoreSaber score beats the BL score, and confirm the improvement display and the ArcViewer replay.

Phase 6: Ongoing sync

The sync subcommand runs once a day from a systemd timer on incineroar: OnCalendar=*-*-* 17:00:00 America/Los_Angeles with Persistent=true, so a run missed while the machine was off happens at the next boot. By 5 pm the day's play is usually over.

1. BeatLeader scores: /player/{id}/scores?sortBy=date&order=desc&time_from=<cursor-1d>. Overlap a day and skip ids you already have. New score ids → save the score JSON, the replay, the score stats, and a leaderboard refresh for that hash.
2. Attempts: skipped (see Decisions).
3. New maps: for every hash not in the archive yet, do phase 1, steps 6–9: BeatLeader leaderboard, then the BeatSaver JSON and zip (nothing to unpack). The game's own folders are only a manual fallback, since sync can't mount Windows. List unresolved hashes in the run summary.
4. Profile: one dated snapshot per run, plus /history.
5. Ratings drift: on Sundays' run, re-fetch /leaderboards/hash/{hash} for every map I've played, and store a new snapshot only when it changed. Upstream rebalances ratings (see the rebalanced-pp branch), and keeping the old snapshots preserves the history.
6. ScoreSaber main: sort=recent until reaching a known score id, then download the new replays at the slow pace. A day's worth fits easily in one run. Legacy is frozen; skip it.
7. Token upkeep: refresh the OAuth token at the start of each run and write the rotated refresh token to auth/beatleader_tokens.json (temp file plus rename) before doing anything else. A daily run stays far inside the 14-day window. If the refresh fails, the unit fails loudly and the message says to run auth login again. Alloy ships journal lines to Loki, so set up an alert there.
8. Once the archive has changed, run the importer incrementally, through a --since flag or by watching state.json. That includes the phase 5 merge for every leaderboard that got a new BL or SS score.

Optional later: a manual sync run (systemctl start plebleader-sync) for when I want results before 5 pm.

Check: play a map, and after the next 17:00 run the score, its replay and the map file show up on the local site, and the replay opens in ArcViewer with the local zip.

Phase 7: NixOS deployment on incineroar

• Packages: buildDotnetModule for the fork (generate deps.json with nix build .#…passthru.fetch-deps), buildNpmPackage for the website (start with npmDepsHash = lib.fakeHash, then pin the got: value), and fetchFromGitHub for ArcViewer deploy. The forks expose their packages through their own flakes and become flake inputs in nix-pleb, next to the existing plebsaber-stream input. Use git+https://gitea.satstack.dev/pleb/<repo>.git.
• Services: put plebleader-api, plebleader-sync (service and timer) and the Caddy vhost in incineroar/workstation.nix. Don't create a new module file. Use a static user plebleader, StateDirectory=plebleader, and an EnvironmentFile from the sops secret.
• Validate with nix build .#nixosConfigurations.incineroar.config.system.build.toplevel --no-link. The user does the switch.

Risks and fallbacks

• The server doesn't build or run on Linux, or EF/SQLite can't translate queries. SQL Server is case-insensitive and SQLite isn't, and SQLite has no DateTimeOffset ordering. Fix it endpoint by endpoint, but only for the endpoints the profile, scores and leaderboard pages call (find them in the network tab). Fallback: skip the fork and write a small read-only API in this repo that serves the archive in the response shapes the website expects. That would also be the first step toward the full replacement.
• The OAuth refresh chain breaks (more than 14 days without a run, a lost write, or upstream auth changes). The archive keeps everything collected so far. Sync stops, a failing unit makes that visible, and auth login re-seeds the chain.
• Maps disappear from BeatSaver. That's why zips are archived as soon as a hash first appears. Old versions that are already gone stay listed in missing.json, which is acceptable (decided 2026-11-08). Cold storage might hold some; check someday, without much hope.
• Upstream API changes. The archive stores raw responses, so a change breaks the importer, not the data. Version the importer's mapping.
• Upstream code moves fast (the website is updated daily). Pin fork revisions. Rebase only when there's a reason to.
• Ranks and pp go stale locally. That's by design: they're upstream's values from the last sync. The UI could show "as of ".

Open questions for the user

Settled on 2026-11-08: port 4442; access limited to incineroar and snorlax; map data limited to played maps; OAuth for auth; sync daily at 17:00 America/Los_Angeles.

Also settled: skip attempts and their replays entirely. Where BL and SS scores overlap, keep one per leaderboard by converting ScoreSaber scores to BeatLeader scores (phase 5).

Nothing is open right now.

Where the earlier work lives

• ~/ops/beatsaber/ss-scraper/: ScoreSaber replay downloader. The legacy account is done; it has no git commits yet.
• ~/ops/beatsaber/playlist-tool/src/clients/{beatleader,scoresaber,beatsaver}/: generated Python OpenAPI clients. Use them as a reference for the response models.
• ~/ops/beatsaber/backups/: UserData backups. They leave the replay folders out.
• ~/ops/beatsaber/beatleader/beatleader-website.code-workspace: points at ~/src/beatleader/... (lowercase). The checkouts are actually in ~/src/BeatLeader/, so fix that path.
• ~/src/BeatLeader/bl-dev-sandbox is a BSOR v2 format experiment, not a dev environment. Ignore it.