Record phase 0 forks, phase 1 completion and phase 2 server fork in the plan

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-08 18:43:31 -08:00
co-authored by Claude Opus 5.5
parent a759308711
commit 48dc0db04b
+69 -8
View File
@@ -138,8 +138,9 @@ These checkouts are in `~/src`. Run `git pull` before relying on them.
unreachable**: `beatleader-server-private` and `bs-simulator`. Upstream
already copes with that. `Startup.cs` falls back to `ReplayRecalculatorStub`,
and nothing in the public code imports a namespace that only the private
repos define. **I never compiled it**: an agent build of third-party code was
blocked in that session. Building on Linux is the first task in phase 2.
repos define. **Verified 2026-11-08:** it builds on Linux with nixpkgs
`dotnet-sdk_10` (10.0.403) with both private folders empty: 0 errors,
about 2,300 warnings, about 1 minute.
- Database: `UseSqlServer` for two contexts (`Startup.cs:380-386`):
`AppContext` (`DefaultConnection`) and `StorageContext`
(`StorageConnection`, which holds `PlayerLeaderboardStats`). There are 570
@@ -225,8 +226,11 @@ plebsaber.stream/ (this repo)
src/lib/server/beatleader.ts extend: bearer auth from the sync's token store, paging, time_from, raw passthrough
src/lib/server/scoresaber.ts extend: player scores, replay download
gitea pleb/beatleader-server fork, branch `pleb/main`
gitea pleb/beatleader-server fork, branch `pleb/main` (based on upstream `rebalanced-pp`)
MIRROR.md what MirrorMode changes and how to run it
appsettings.Mirror.json incineroar values (ASPNETCORE_ENVIRONMENT=Mirror)
Tools/ArchiveImporter/ C# console app sharing AppContext/StorageContext
gitea pleb/beatleader-server-models fork of the models submodule, branch `pleb/main`
gitea pleb/beatleader-website fork, branch `pleb/main`
/var/lib/plebleader/ on incineroar, user `plebleader`
@@ -264,14 +268,24 @@ the fork, so do it first.
### Phase 0: Prep
1. Update the checkouts:
1. **Done 2026-11-08** (all were already current; upstream server `master`
`1474312`, `rebalanced-pp` `52a50ed`, website `2b3ef891`, ArcViewer `deploy`
`c776256`, ScoreSaber website `836d789`). Update the checkouts:
`git -C ~/src/BeatLeader/beatleader-server pull --recurse-submodules`, and
the same for `beatleader-website`, `~/src/AllPoland/ArcViewer` (also
`git fetch origin deploy`), and `~/src/ScoreSaber/website`. The ScoreSaber
website is public now; use it to check the score and replay API shapes.
2. Create the Gitea forks `pleb/beatleader-server` and `pleb/beatleader-website`
2. **Done 2026-11-08.** Create the Gitea forks `pleb/beatleader-server` and `pleb/beatleader-website`
with the `gitea-api` skill (mirror or push). Keep `upstream` as a remote so
you can rebase later.
Created as private, non-mirror imports through `POST /repos/migrate` (the
Gitea token lacks `write:user`, which `POST /user/repos` needs), with
default branch `pleb/main`. The `~/src` checkouts keep `origin` = GitHub and
gained a `gitea` remote; `pleb/main` is checked out there. The server's
`pleb/main` starts from `rebalanced-pp`, which is `master` plus 4 commits
and what the live API most likely runs, so the entities match the archived
responses. `pleb/beatleader-server-models` was forked too (see phase 2,
step 7); the server's `.gitmodules` points at it over SSH.
3. Bring this repo's flake up to date (it's still on `nixos-25.05`) only if the
exporter gets packaged here.
4. Ask the user to add a sops secret `plebleader.env` for incineroar, holding
@@ -386,21 +400,26 @@ leaderboard snapshot, plus a BeatSaver zip or recovered game folder, unless
it's listed in `missing.json`. Write a `report` subcommand that prints counts
and what's missing, and paste its output into this file.
**Report, 2026-11-08** (ScoreSaber main replays still downloading):
**Report, 2026-11-08 (phase 1 complete).** The ScoreSaber replays finished
with zero 429s.
```
Window: scores since all time, BeatSaver maps since 2024-11-08
BeatLeader scores 3568 in window (3568 archived total, both accounts)
BeatLeader replays 3568/3568
BeatLeader score stats 3568/3568
ScoreSaber 76561199407393962: 2107 scores, replays 435/1034
ScoreSaber 76561199407393962: 2107 scores, replays 1034/1034
ScoreSaber 76561197962107242: 199 scores, replays 58/58
Played hashes 2560
BL leaderboard snapshot 2552/2560
BeatSaver zip 1897/1926 in maps window (missing.json lists 29)
Archive size 21.15 GB
Archive size 21.78 GB
```
Steps 8 (game-folder recovery), 9 (unpack into `maps/`) and 13 (local mod
replays) are still open. Step 9 is needed before phase 4; it can happen when
the archive moves to `/var/lib/plebleader`.
The 29 missing hashes are all "hash unknown to BeatSaver", and none belongs to
a legacy score with a replay (all 53 of those maps are archived). The 8 hashes
without a BL leaderboard are ScoreSaber-only maps BeatLeader never saw. Zips
@@ -408,6 +427,48 @@ average about 8 MB (15 GB for 1,897), more than the estimate below.
### Phase 2: Server fork on SQLite
**Done 2026-11-08** (`pleb/main` at `1741ce9`, models fork at `8f233e8`).
What differs from the steps below:
- Step 2: `Microsoft.Data.SqlClient` stays as a direct package, because one
controller catches `SqlException`. One controller had an unused
`using BeatLeader_Server.Migrations;` that broke the build once the old
migrations were gone. The fresh migrations create 155 tables and 217
indexes. WAL and migrations are applied at startup by `Utils/MirrorSetup.cs`;
`busy_timeout` comes from `Default Timeout=30` in the connection string.
- Step 3: `Utils/BulkCompat.cs` sits in the global namespace, like Z's own
extensions, because some callers have no EF `using`. 49 of the `BulkUpdate`
calls pass `ColumnInputExpression`, often on detached stubs that carry only
the key and those columns, so the shim marks only the listed columns
modified and detaches the stubs after saving. A plain `UpdateRange` would
have wiped every other column. **Not exercised at runtime yet**; the
importer will be the first real user.
- Step 4: the two index-defragmentation scripts became `PRAGMA optimize` and
`REINDEX; ANALYZE;`.
- Step 5: `GetS3Client` returns a `LocalStorageClient` (an `AmazonS3Client`
subclass) when `StoragePath` is set; the six S3 primitives branch on it.
Upstream behaviour is unchanged without `StoragePath`.
- Step 6: `MirrorMode` also skips `RefreshTaskService` and `MapRefresh`, and
the external OAuth providers. Two surprises showed up in testing:
`GetLazy` fetched unknown players from Steam (it now returns null, so 404),
and the BeatSaver login is registered unconditionally and validated on
every request (it gets placeholder credentials, otherwise every request
was a 500).
- Step 7: instead of patching about 35 `showRatings` decisions, the four
`HideRatings()` leaves return early when `MirrorSettings.ShowAllRatings`
(in the models fork) is set; `MirrorSetup` sets it. Sorting and filtering by
stars on unranked maps still follow the requesting user's settings; patch
`Utils/{Map,Leaderboard}ListUtils/Sort.cs` if that matters.
- Step 8: `appsettings.Mirror.json` loads with `ASPNETCORE_ENVIRONMENT=Mirror`;
`MIRROR.md` in the fork documents it. OpenIddict uses ephemeral keys.
Run the DLL with `--contentRoot <bin>` from a working directory in the state
dir, because data protection keys go to `../keys/`. `dotnet run` ignores
`ASPNETCORE_URLS` in favour of `launchSettings.json`.
Check result: on empty files, `/swagger/index.html` 200,
`/player/76561199407393962` 404, `/players` and `/leaderboards` 200 with empty
pages, no `fail:` lines in the log.
Work on the `pleb/main` branch of `pleb/beatleader-server`. Keep each change
small and in its own commit, so upstream rebases stay manageable.