Record phase 0 forks, phase 1 completion and phase 2 server fork in the plan

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-08 18:43:31 -08:00
co-authored by Claude Opus 5.5
parent a759308711
commit 48dc0db04b
+69 -8
View File
@@ -138,8 +138,9 @@ These checkouts are in `~/src`. Run `git pull` before relying on them.
unreachable**: `beatleader-server-private` and `bs-simulator`. Upstream unreachable**: `beatleader-server-private` and `bs-simulator`. Upstream
already copes with that. `Startup.cs` falls back to `ReplayRecalculatorStub`, already copes with that. `Startup.cs` falls back to `ReplayRecalculatorStub`,
and nothing in the public code imports a namespace that only the private and nothing in the public code imports a namespace that only the private
repos define. **I never compiled it**: an agent build of third-party code was repos define. **Verified 2026-11-08:** it builds on Linux with nixpkgs
blocked in that session. Building on Linux is the first task in phase 2. `dotnet-sdk_10` (10.0.403) with both private folders empty: 0 errors,
about 2,300 warnings, about 1 minute.
- Database: `UseSqlServer` for two contexts (`Startup.cs:380-386`): - Database: `UseSqlServer` for two contexts (`Startup.cs:380-386`):
`AppContext` (`DefaultConnection`) and `StorageContext` `AppContext` (`DefaultConnection`) and `StorageContext`
(`StorageConnection`, which holds `PlayerLeaderboardStats`). There are 570 (`StorageConnection`, which holds `PlayerLeaderboardStats`). There are 570
@@ -225,8 +226,11 @@ plebsaber.stream/ (this repo)
src/lib/server/beatleader.ts extend: bearer auth from the sync's token store, paging, time_from, raw passthrough src/lib/server/beatleader.ts extend: bearer auth from the sync's token store, paging, time_from, raw passthrough
src/lib/server/scoresaber.ts extend: player scores, replay download src/lib/server/scoresaber.ts extend: player scores, replay download
gitea pleb/beatleader-server fork, branch `pleb/main` gitea pleb/beatleader-server fork, branch `pleb/main` (based on upstream `rebalanced-pp`)
MIRROR.md what MirrorMode changes and how to run it
appsettings.Mirror.json incineroar values (ASPNETCORE_ENVIRONMENT=Mirror)
Tools/ArchiveImporter/ C# console app sharing AppContext/StorageContext Tools/ArchiveImporter/ C# console app sharing AppContext/StorageContext
gitea pleb/beatleader-server-models fork of the models submodule, branch `pleb/main`
gitea pleb/beatleader-website fork, branch `pleb/main` gitea pleb/beatleader-website fork, branch `pleb/main`
/var/lib/plebleader/ on incineroar, user `plebleader` /var/lib/plebleader/ on incineroar, user `plebleader`
@@ -264,14 +268,24 @@ the fork, so do it first.
### Phase 0: Prep ### Phase 0: Prep
1. Update the checkouts: 1. **Done 2026-11-08** (all were already current; upstream server `master`
`1474312`, `rebalanced-pp` `52a50ed`, website `2b3ef891`, ArcViewer `deploy`
`c776256`, ScoreSaber website `836d789`). Update the checkouts:
`git -C ~/src/BeatLeader/beatleader-server pull --recurse-submodules`, and `git -C ~/src/BeatLeader/beatleader-server pull --recurse-submodules`, and
the same for `beatleader-website`, `~/src/AllPoland/ArcViewer` (also the same for `beatleader-website`, `~/src/AllPoland/ArcViewer` (also
`git fetch origin deploy`), and `~/src/ScoreSaber/website`. The ScoreSaber `git fetch origin deploy`), and `~/src/ScoreSaber/website`. The ScoreSaber
website is public now; use it to check the score and replay API shapes. website is public now; use it to check the score and replay API shapes.
2. Create the Gitea forks `pleb/beatleader-server` and `pleb/beatleader-website` 2. **Done 2026-11-08.** Create the Gitea forks `pleb/beatleader-server` and `pleb/beatleader-website`
with the `gitea-api` skill (mirror or push). Keep `upstream` as a remote so with the `gitea-api` skill (mirror or push). Keep `upstream` as a remote so
you can rebase later. you can rebase later.
Created as private, non-mirror imports through `POST /repos/migrate` (the
Gitea token lacks `write:user`, which `POST /user/repos` needs), with
default branch `pleb/main`. The `~/src` checkouts keep `origin` = GitHub and
gained a `gitea` remote; `pleb/main` is checked out there. The server's
`pleb/main` starts from `rebalanced-pp`, which is `master` plus 4 commits
and what the live API most likely runs, so the entities match the archived
responses. `pleb/beatleader-server-models` was forked too (see phase 2,
step 7); the server's `.gitmodules` points at it over SSH.
3. Bring this repo's flake up to date (it's still on `nixos-25.05`) only if the 3. Bring this repo's flake up to date (it's still on `nixos-25.05`) only if the
exporter gets packaged here. exporter gets packaged here.
4. Ask the user to add a sops secret `plebleader.env` for incineroar, holding 4. Ask the user to add a sops secret `plebleader.env` for incineroar, holding
@@ -386,21 +400,26 @@ leaderboard snapshot, plus a BeatSaver zip or recovered game folder, unless
it's listed in `missing.json`. Write a `report` subcommand that prints counts it's listed in `missing.json`. Write a `report` subcommand that prints counts
and what's missing, and paste its output into this file. and what's missing, and paste its output into this file.
**Report, 2026-11-08** (ScoreSaber main replays still downloading): **Report, 2026-11-08 (phase 1 complete).** The ScoreSaber replays finished
with zero 429s.
``` ```
Window: scores since all time, BeatSaver maps since 2024-11-08 Window: scores since all time, BeatSaver maps since 2024-11-08
BeatLeader scores 3568 in window (3568 archived total, both accounts) BeatLeader scores 3568 in window (3568 archived total, both accounts)
BeatLeader replays 3568/3568 BeatLeader replays 3568/3568
BeatLeader score stats 3568/3568 BeatLeader score stats 3568/3568
ScoreSaber 76561199407393962: 2107 scores, replays 435/1034 ScoreSaber 76561199407393962: 2107 scores, replays 1034/1034
ScoreSaber 76561197962107242: 199 scores, replays 58/58 ScoreSaber 76561197962107242: 199 scores, replays 58/58
Played hashes 2560 Played hashes 2560
BL leaderboard snapshot 2552/2560 BL leaderboard snapshot 2552/2560
BeatSaver zip 1897/1926 in maps window (missing.json lists 29) BeatSaver zip 1897/1926 in maps window (missing.json lists 29)
Archive size 21.15 GB Archive size 21.78 GB
``` ```
Steps 8 (game-folder recovery), 9 (unpack into `maps/`) and 13 (local mod
replays) are still open. Step 9 is needed before phase 4; it can happen when
the archive moves to `/var/lib/plebleader`.
The 29 missing hashes are all "hash unknown to BeatSaver", and none belongs to The 29 missing hashes are all "hash unknown to BeatSaver", and none belongs to
a legacy score with a replay (all 53 of those maps are archived). The 8 hashes a legacy score with a replay (all 53 of those maps are archived). The 8 hashes
without a BL leaderboard are ScoreSaber-only maps BeatLeader never saw. Zips without a BL leaderboard are ScoreSaber-only maps BeatLeader never saw. Zips
@@ -408,6 +427,48 @@ average about 8 MB (15 GB for 1,897), more than the estimate below.
### Phase 2: Server fork on SQLite ### Phase 2: Server fork on SQLite
**Done 2026-11-08** (`pleb/main` at `1741ce9`, models fork at `8f233e8`).
What differs from the steps below:
- Step 2: `Microsoft.Data.SqlClient` stays as a direct package, because one
controller catches `SqlException`. One controller had an unused
`using BeatLeader_Server.Migrations;` that broke the build once the old
migrations were gone. The fresh migrations create 155 tables and 217
indexes. WAL and migrations are applied at startup by `Utils/MirrorSetup.cs`;
`busy_timeout` comes from `Default Timeout=30` in the connection string.
- Step 3: `Utils/BulkCompat.cs` sits in the global namespace, like Z's own
extensions, because some callers have no EF `using`. 49 of the `BulkUpdate`
calls pass `ColumnInputExpression`, often on detached stubs that carry only
the key and those columns, so the shim marks only the listed columns
modified and detaches the stubs after saving. A plain `UpdateRange` would
have wiped every other column. **Not exercised at runtime yet**; the
importer will be the first real user.
- Step 4: the two index-defragmentation scripts became `PRAGMA optimize` and
`REINDEX; ANALYZE;`.
- Step 5: `GetS3Client` returns a `LocalStorageClient` (an `AmazonS3Client`
subclass) when `StoragePath` is set; the six S3 primitives branch on it.
Upstream behaviour is unchanged without `StoragePath`.
- Step 6: `MirrorMode` also skips `RefreshTaskService` and `MapRefresh`, and
the external OAuth providers. Two surprises showed up in testing:
`GetLazy` fetched unknown players from Steam (it now returns null, so 404),
and the BeatSaver login is registered unconditionally and validated on
every request (it gets placeholder credentials, otherwise every request
was a 500).
- Step 7: instead of patching about 35 `showRatings` decisions, the four
`HideRatings()` leaves return early when `MirrorSettings.ShowAllRatings`
(in the models fork) is set; `MirrorSetup` sets it. Sorting and filtering by
stars on unranked maps still follow the requesting user's settings; patch
`Utils/{Map,Leaderboard}ListUtils/Sort.cs` if that matters.
- Step 8: `appsettings.Mirror.json` loads with `ASPNETCORE_ENVIRONMENT=Mirror`;
`MIRROR.md` in the fork documents it. OpenIddict uses ephemeral keys.
Run the DLL with `--contentRoot <bin>` from a working directory in the state
dir, because data protection keys go to `../keys/`. `dotnet run` ignores
`ASPNETCORE_URLS` in favour of `launchSettings.json`.
Check result: on empty files, `/swagger/index.html` 200,
`/player/76561199407393962` 404, `/players` and `/leaderboards` 200 with empty
pages, no `fail:` lines in the log.
Work on the `pleb/main` branch of `pleb/beatleader-server`. Keep each change Work on the `pleb/main` branch of `pleb/beatleader-server`. Keep each change
small and in its own commit, so upstream rebases stay manageable. small and in its own commit, so upstream rebases stay manageable.