Record phase 3 importer and fetch full player history

The importer needs every day of /history for the triangle's monthly points;
the API returns only 50 by default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-11-08 19:21:46 -08:00
co-authored by Claude Opus 5.5
parent d75448af66
commit f7567c4c4f
2 changed files with 52 additions and 2 deletions
+49 -1
View File
@@ -543,6 +543,54 @@ loads, and `GET /player/76561199407393962` returns 404 rather than 500.
### Phase 3: Importer
**Importer done 2026-11-09** (`pleb/main` at `737575b`, not pushed yet). Usage
and the full file → table mapping are in the fork's `MIRROR.md`. A full import
takes about 11 s, and an unchanged rerun writes 0 rows. What differs from the
steps below:
- The archived responses are deserialized straight into the entity types,
because their property names match. Only scalar columns are copied onto the
tracked rows; keys, foreign keys and navigations are set explicitly. Leaderboard
snapshots are the main source for `Song`/`DifficultyDescription`. A score's
embedded leaderboard is a fallback only, and no score needed it.
- One-to-one children (`ScoreImprovement`, `ReplayOffsets`, `ModifiersRating`,
`DifficultyStatistics`, `ModifiersMap`, stats, settings) get local ids, so
they can't collide with upstream ids. Scores, difficulties, songs,
leaderboards, mappers and history rows keep upstream ids.
- `PlayerScoreStatsHistory` lives in **`StorageContext`**, so that database
isn't empty. The exporter had saved only 50 days of `/history` (the API
default); it now asks for `count=100000` (1,138 days for pleb). The triangle
needs those monthly points.
- No legacy `Player` row. An `AccountLink` (`PCOculusID` = legacy id) plus a
`// pleb:` patch in `PlayerIdToMain` resolves `/player/76561197962107242`
to pleb. Upstream only checks Steam-range ids against Oculus links. The
legacy account's 3 BL scores are skipped until phase 5.
- `StoragePath/replays` is a symlink to the archive's `beatleader/replays`.
Score stats are copied to `StoragePath/scorestats/<scoreId>.json`.
- Clans aren't mirrored (`ClanOrder` is blank).
Imported: 2,552 songs, 10,104 difficulties, 10,098 leaderboards, 628 mappers,
3,565 scores, 3,568 score stats, 1,138 history rows. API check against the
archived upstream responses, on a scratch DB with `ASPNETCORE_ENVIRONMENT=Mirror`:
- `/player/{id}?stats=true` matches. The only differences are row ids,
clans, and `linkedIds`, which now lists the alt.
- `/player/{id}/scores?includeIO=true`: 3,565 total. Scores, improvements and
offsets match, and unranked difficulties now carry stars and the skill
triangle ratings.
- `/accgraph` and `/history/triangle` match. `/score/statistic/{id}` redirects
to the local file.
- **Open: `/leaderboard/{id}` and `/leaderboard/scores/{id}` return 500.**
`LeaderboardController.GeneralScores` projects
`Player.Clans.OrderBy(…ClanOrder…).Take(1)`, which needs SQL `APPLY`, and
SQLite has none. The same pattern appears about 8 times, in
`ScoreController`, `ReplayController`, `LeaderboardController` and
`ClanUtils`. Fix it in a separate server commit before phase 4. For example,
skip the clan projection in MirrorMode (there are no clans), or move the
ordering client-side.
The check below (stock website) is still to do in phase 4.
`Tools/ArchiveImporter` lives in the fork, references the main project, and
uses its `AppContext`/`StorageContext`. Run it as a one-shot that is idempotent
and upserts by upstream id.
@@ -563,7 +611,7 @@ and upserts by upstream id.
an `AccountLink` or alt pointing to the main player, so `PlayerIdToMain`
resolves `/u/76561197962107242` to `pleb`. Its 3 BL scores go through the
phase 5 merge like any other candidate.
- `StorageContext` (attempts) stays empty, because attempts aren't exported.
- `StorageContext` holds `PlayerScoreStatsHistory`; its attempts tables stay empty, because attempts aren't exported.
- `scorestats/*.json` → the `scorestats` storage container, using the filename
the server expects (check `ScoreController.GetStatistic`).
+3 -1
View File
@@ -26,7 +26,9 @@ export async function exportProfile(playerId: string): Promise<void> {
const dir = (kind: string) => bl('players', playerId, kind);
const endpoints: [string, string][] = [
['profile', `/player/${playerId}?stats=true`],
['history', `/player/${playerId}/history`],
// count: days of history; the default is 50, which loses the triangle's
// monthly points. Upstream keeps one row per day since the first score.
['history', `/player/${playerId}/history?count=100000`],
['triangle', `/player/${playerId}/history/triangle`],
['accgraph', `/player/${playerId}/accgraph`]
];