diff --git a/docs/plan.md b/docs/plan.md index 8e11eef..2dea312 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -138,8 +138,9 @@ These checkouts are in `~/src`. Run `git pull` before relying on them. unreachable**: `beatleader-server-private` and `bs-simulator`. Upstream already copes with that. `Startup.cs` falls back to `ReplayRecalculatorStub`, and nothing in the public code imports a namespace that only the private - repos define. **I never compiled it**: an agent build of third-party code was - blocked in that session. Building on Linux is the first task in phase 2. + repos define. **Verified 2026-11-08:** it builds on Linux with nixpkgs + `dotnet-sdk_10` (10.0.403) with both private folders empty: 0 errors, + about 2,300 warnings, about 1 minute. - Database: `UseSqlServer` for two contexts (`Startup.cs:380-386`): `AppContext` (`DefaultConnection`) and `StorageContext` (`StorageConnection`, which holds `PlayerLeaderboardStats`). There are 570 @@ -225,8 +226,11 @@ plebsaber.stream/ (this repo) src/lib/server/beatleader.ts extend: bearer auth from the sync's token store, paging, time_from, raw passthrough src/lib/server/scoresaber.ts extend: player scores, replay download -gitea pleb/beatleader-server fork, branch `pleb/main` +gitea pleb/beatleader-server fork, branch `pleb/main` (based on upstream `rebalanced-pp`) + MIRROR.md what MirrorMode changes and how to run it + appsettings.Mirror.json incineroar values (ASPNETCORE_ENVIRONMENT=Mirror) Tools/ArchiveImporter/ C# console app sharing AppContext/StorageContext +gitea pleb/beatleader-server-models fork of the models submodule, branch `pleb/main` gitea pleb/beatleader-website fork, branch `pleb/main` /var/lib/plebleader/ on incineroar, user `plebleader` @@ -264,14 +268,24 @@ the fork, so do it first. ### Phase 0: Prep -1. Update the checkouts: +1. **Done 2026-11-08** (all were already current; upstream server `master` + `1474312`, `rebalanced-pp` `52a50ed`, website `2b3ef891`, ArcViewer `deploy` + `c776256`, ScoreSaber website `836d789`). Update the checkouts: `git -C ~/src/BeatLeader/beatleader-server pull --recurse-submodules`, and the same for `beatleader-website`, `~/src/AllPoland/ArcViewer` (also `git fetch origin deploy`), and `~/src/ScoreSaber/website`. The ScoreSaber website is public now; use it to check the score and replay API shapes. -2. Create the Gitea forks `pleb/beatleader-server` and `pleb/beatleader-website` +2. **Done 2026-11-08.** Create the Gitea forks `pleb/beatleader-server` and `pleb/beatleader-website` with the `gitea-api` skill (mirror or push). Keep `upstream` as a remote so you can rebase later. + Created as private, non-mirror imports through `POST /repos/migrate` (the + Gitea token lacks `write:user`, which `POST /user/repos` needs), with + default branch `pleb/main`. The `~/src` checkouts keep `origin` = GitHub and + gained a `gitea` remote; `pleb/main` is checked out there. The server's + `pleb/main` starts from `rebalanced-pp`, which is `master` plus 4 commits + and what the live API most likely runs, so the entities match the archived + responses. `pleb/beatleader-server-models` was forked too (see phase 2, + step 7); the server's `.gitmodules` points at it over SSH. 3. Bring this repo's flake up to date (it's still on `nixos-25.05`) only if the exporter gets packaged here. 4. Ask the user to add a sops secret `plebleader.env` for incineroar, holding @@ -386,21 +400,26 @@ leaderboard snapshot, plus a BeatSaver zip or recovered game folder, unless it's listed in `missing.json`. Write a `report` subcommand that prints counts and what's missing, and paste its output into this file. -**Report, 2026-11-08** (ScoreSaber main replays still downloading): +**Report, 2026-11-08 (phase 1 complete).** The ScoreSaber replays finished +with zero 429s. ``` Window: scores since all time, BeatSaver maps since 2024-11-08 BeatLeader scores 3568 in window (3568 archived total, both accounts) BeatLeader replays 3568/3568 BeatLeader score stats 3568/3568 -ScoreSaber 76561199407393962: 2107 scores, replays 435/1034 +ScoreSaber 76561199407393962: 2107 scores, replays 1034/1034 ScoreSaber 76561197962107242: 199 scores, replays 58/58 Played hashes 2560 BL leaderboard snapshot 2552/2560 BeatSaver zip 1897/1926 in maps window (missing.json lists 29) -Archive size 21.15 GB +Archive size 21.78 GB ``` +Steps 8 (game-folder recovery), 9 (unpack into `maps/`) and 13 (local mod +replays) are still open. Step 9 is needed before phase 4; it can happen when +the archive moves to `/var/lib/plebleader`. + The 29 missing hashes are all "hash unknown to BeatSaver", and none belongs to a legacy score with a replay (all 53 of those maps are archived). The 8 hashes without a BL leaderboard are ScoreSaber-only maps BeatLeader never saw. Zips @@ -408,6 +427,48 @@ average about 8 MB (15 GB for 1,897), more than the estimate below. ### Phase 2: Server fork on SQLite +**Done 2026-11-08** (`pleb/main` at `1741ce9`, models fork at `8f233e8`). +What differs from the steps below: + +- Step 2: `Microsoft.Data.SqlClient` stays as a direct package, because one + controller catches `SqlException`. One controller had an unused + `using BeatLeader_Server.Migrations;` that broke the build once the old + migrations were gone. The fresh migrations create 155 tables and 217 + indexes. WAL and migrations are applied at startup by `Utils/MirrorSetup.cs`; + `busy_timeout` comes from `Default Timeout=30` in the connection string. +- Step 3: `Utils/BulkCompat.cs` sits in the global namespace, like Z's own + extensions, because some callers have no EF `using`. 49 of the `BulkUpdate` + calls pass `ColumnInputExpression`, often on detached stubs that carry only + the key and those columns, so the shim marks only the listed columns + modified and detaches the stubs after saving. A plain `UpdateRange` would + have wiped every other column. **Not exercised at runtime yet**; the + importer will be the first real user. +- Step 4: the two index-defragmentation scripts became `PRAGMA optimize` and + `REINDEX; ANALYZE;`. +- Step 5: `GetS3Client` returns a `LocalStorageClient` (an `AmazonS3Client` + subclass) when `StoragePath` is set; the six S3 primitives branch on it. + Upstream behaviour is unchanged without `StoragePath`. +- Step 6: `MirrorMode` also skips `RefreshTaskService` and `MapRefresh`, and + the external OAuth providers. Two surprises showed up in testing: + `GetLazy` fetched unknown players from Steam (it now returns null, so 404), + and the BeatSaver login is registered unconditionally and validated on + every request (it gets placeholder credentials, otherwise every request + was a 500). +- Step 7: instead of patching about 35 `showRatings` decisions, the four + `HideRatings()` leaves return early when `MirrorSettings.ShowAllRatings` + (in the models fork) is set; `MirrorSetup` sets it. Sorting and filtering by + stars on unranked maps still follow the requesting user's settings; patch + `Utils/{Map,Leaderboard}ListUtils/Sort.cs` if that matters. +- Step 8: `appsettings.Mirror.json` loads with `ASPNETCORE_ENVIRONMENT=Mirror`; + `MIRROR.md` in the fork documents it. OpenIddict uses ephemeral keys. + Run the DLL with `--contentRoot ` from a working directory in the state + dir, because data protection keys go to `../keys/`. `dotnet run` ignores + `ASPNETCORE_URLS` in favour of `launchSettings.json`. + +Check result: on empty files, `/swagger/index.html` 200, +`/player/76561199407393962` 404, `/players` and `/leaderboards` 200 with empty +pages, no `fail:` lines in the log. + Work on the `pleb/main` branch of `pleb/beatleader-server`. Keep each change small and in its own commit, so upstream rebases stay manageable.