Add the plebleader daily sync and package the archive CLI

`sync` refreshes the token chain first, then fetches new BeatLeader and
ScoreSaber scores (with a day of overlap), their replays, score stats,
leaderboards and map zips, and re-snapshots every played hash on Sundays.
The flake's plebleader-archive package runs the CLI with Node's type
stripping.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/plan.md

@@ -543,6 +543,54 @@ loads, and `GET /player/76561199407393962` returns 404 rather than 500.
 
 ### Phase 3: Importer
 
+**Importer done 2026-11-09** (`pleb/main` at `e3f991c`, pushed to Gitea). Usage
+and the full file → table mapping are in the fork's `MIRROR.md`. A full import
+takes about 11 s, and an unchanged rerun writes 0 rows. What differs from the
+steps below:
+
+- The archived responses are deserialized straight into the entity types,
+  because their property names match. Only scalar columns are copied onto the
+  tracked rows; keys, foreign keys and navigations are set explicitly. Leaderboard
+  snapshots are the main source for `Song`/`DifficultyDescription`. A score's
+  embedded leaderboard is a fallback only, and no score needed it.
+- One-to-one children (`ScoreImprovement`, `ReplayOffsets`, `ModifiersRating`,
+  `DifficultyStatistics`, `ModifiersMap`, stats, settings) get local ids, so
+  they can't collide with upstream ids. Scores, difficulties, songs,
+  leaderboards, mappers and history rows keep upstream ids.
+- `PlayerScoreStatsHistory` lives in **`StorageContext`**, so that database
+  isn't empty. The exporter had saved only 50 days of `/history` (the API
+  default); it now asks for `count=100000` (1,138 days for pleb). The triangle
+  needs those monthly points.
+- No legacy `Player` row. An `AccountLink` (`PCOculusID` = legacy id) plus a
+  `// pleb:` patch in `PlayerIdToMain` resolves `/player/76561197962107242`
+  to pleb. Upstream only checks Steam-range ids against Oculus links. The
+  legacy account's 3 BL scores are skipped until phase 5.
+- `StoragePath/replays` is a symlink to the archive's `beatleader/replays`.
+  Score stats are copied to `StoragePath/scorestats/<scoreId>.json`.
+- Clans aren't mirrored (`ClanOrder` is blank).
+
+Imported: 2,552 songs, 10,104 difficulties, 10,098 leaderboards, 628 mappers,
+3,565 scores, 3,568 score stats, 1,138 history rows. API check against the
+archived upstream responses, on a scratch DB with `ASPNETCORE_ENVIRONMENT=Mirror`:
+
+- `/player/{id}?stats=true` matches. The only differences are row ids,
+  clans, and `linkedIds`, which now lists the alt.
+- `/player/{id}/scores?includeIO=true`: 3,565 total. Scores, improvements and
+  offsets match, and unranked difficulties now carry stars and the skill
+  triangle ratings.
+- `/accgraph` and `/history/triangle` match. `/score/statistic/{id}` redirects
+  to the local file.
+- `/leaderboard/{id}` and `/leaderboard/scores/{id}` first returned 500.
+  About 10 projections order `Player.Clans` by an expression over the outer
+  row's `ClanOrder`, which needs SQL `APPLY`, and SQLite has none. **Fixed**
+  in `e3f991c`: `Utils/MirrorQueryInterceptor.cs`, an EF query interceptor
+  registered only in MirrorMode, rewrites any ordering of `Player.Clans` to
+  ordering by clan id. One file instead of patching every call site; the
+  mirror has no clans, so the order doesn't matter. The leaderboard,
+  clan-ranking and context variants now return 200.
+
+The check below (stock website) is still to do in phase 4.
+
 `Tools/ArchiveImporter` lives in the fork, references the main project, and
 uses its `AppContext`/`StorageContext`. Run it as a one-shot that is idempotent
 and upserts by upstream id.
@@ -563,7 +611,7 @@ and upserts by upstream id.
   an `AccountLink` or alt pointing to the main player, so `PlayerIdToMain`
   resolves `/u/76561197962107242` to `pleb`. Its 3 BL scores go through the
   phase 5 merge like any other candidate.
-- `StorageContext` (attempts) stays empty, because attempts aren't exported.
+- `StorageContext` holds `PlayerScoreStatsHistory`; its attempts tables stay empty, because attempts aren't exported.
 - `scorestats/*.json` → the `scorestats` storage container, using the filename
   the server expects (check `ScoreController.GetStatistic`).
 
@@ -577,6 +625,65 @@ on the profile.
 
 ### Phase 4: Website, viewer, Caddy
 
+**Steps 1–2 done and both checks passed 2026-11-08**, on a dev stand-in for
+the vhost: the importer into `.data/plebleader/dev/db/`, the API on 8042
+(`PublicBaseUrl=http://127.0.0.1:8890`), and an ad-hoc `nix shell
+nixpkgs#caddy` on `http://127.0.0.1:8890` with the routes below
+(`.data/plebleader/dev/{run-api.sh,Caddyfile}`). Use `127.0.0.1`, not
+`localhost`: the website sends `localhost` to the real upstream API. Commits:
+server `7cc964a`, website `f5194ee3`, `9ebe84ea`, `8d540642`, all local
+(not pushed yet). Step 3 (nix-pleb) is still open.
+
+- **Phase 3 check passed.** `/u/pleb`: 11,608.35pp, #3,861 global, US #1,643,
+  672 ranked and 3,565 total scores, unranked difficulties with stars (8.33★
+  on the Phase 1 test map), the skill triangle with monthly history. The
+  leaderboard page and `/score/{id}` (accuracy graphs from local score stats)
+  work too.
+- **Server fix:** the profile page calls `GET /v2/player/{id}/hydrate`, which
+  the live API has and the public source doesn't (404, "Player not found").
+  `Controllers/MirrorHydrateController.cs` rebuilds it from
+  `PlayerController.Get` plus the `profile` object that the website's
+  `process-hydrate.js` reads.
+- **Server run note:** `dotnet build` doesn't copy `wwwroot/` into `bin/`.
+  Without it `PreviewController`'s constructor throws on `/images/Star.png`,
+  and any controller that depends on it (`/user`, `/user/config`) returns 500.
+  The dev run passes `--webroot <src>/wwwroot`; the Nix package uses `dotnet
+  publish`, which copies it.
+- **Website, as built:** `IS_MIRROR` in `api-queue.js` is true when the API
+  resolves to `/cors/blapi/`. In that case `BL_REPLAYS_URL` is `/watch/` and
+  score stats come from `/cdn/scorestats/`. Every hardcoded
+  `allpoland.github.io/ArcViewer/` link points at `/watch/` (except the clan
+  page's embed). `public/watch/index.html` takes BL replay-viewer or ArcViewer
+  parameters (`link`, `replayURL`, `scoreId`, `id`, optional `hash`) and
+  redirects to `/arcviewer/?replayURL=…&url=/cdn/maps/<hash>.zip&noProxy=true`.
+  It reads the hash from the replay filename
+  (`<score>-<player>-<diff>-<mode>-<HASH>.bsor`); for `scoreId` it first asks
+  `/cors/blapi/score/{id}` for the replay URL. Phase 5 can pass `hash=` for
+  `.dat` replays. The analyzer icons are off by default (an upstream Patreon
+  service). Clans, events, the socket page and the star-rating graph are still
+  visible; they just show nothing.
+- **ArcViewer facts:** without `noProxy=true` it routes every non-whitelisted
+  URL through `cors.bsmg.dev`, which can't reach a LAN host. When both `id`
+  and `url` are given, `id` wins, so never send both. With no map parameter
+  it looks up the replay's own hash on BeatSaver. The `deploy` build ships
+  uncompressed `Build/` files, so no `Content-Encoding` headers are needed.
+  Its own calls to `api.beatleader.com` (player card, leaderboard info) fail
+  from this origin, so the player card stays blank; fixing that needs a Unity
+  build, so it's left as is.
+- **Map zips:** BeatSaver's CDN serves `https://r2cdn.beatsaver.com/<hash>.zip`
+  with `Access-Control-Allow-Origin: *` (it rejects HEAD, use GET). Caddy
+  redirects to it when the archive has no zip, so the website always links
+  the local path and the ~630 older maps outside `--maps-since` still play.
+- **Nix:** the website fork's `flake.nix` builds `packages.default` (yarn
+  v1 offline, ~16 s; deps hash `sha256-ThGvMJfU…`). Its `postPatch` turns the
+  themes' Google Fonts `@import url(...)` into `@import (css) url(...)`,
+  because less would fetch them at build time. ArcViewer `deploy` at
+  `c776256497b66f7c91a74162cfcd943b0f45ee2e` is
+  `sha256-4A+aVnFrKyWQjFnH2N/CwoVIlwrcgV2fYYrtaOiodo0=` for `fetchFromGitHub`.
+
+Step 3 was done together with phase 7 (see there): the vhost below is in
+nix-pleb, waiting for the switch.
+
 1. In the website fork, point `BL_REPLAYS_URL` (and the "watch replay" links)
    at `/arcviewer/?replayURL=<encoded local replay url>&url=<encoded /cdn/maps/<hash>.zip>`.
    Fall back to `id=<bsr>` only for hashes that have no local zip. Hide or stub
@@ -597,20 +704,26 @@ on the profile.
        @plebleader_allowed remote_ip 127.0.0.1/32 ::1/128 192.168.1.36/32 192.168.254.21/32
        handle @plebleader_allowed {
          handle_path /cors/blapi/* { reverse_proxy http://127.0.0.1:8042 }
-         handle_path /cdn/maps/* { root * /var/lib/plebleader/archive/beatsaver/zips; file_server }
+         handle_path /cdn/maps/* {
+           root * /var/lib/plebleader/archive/beatsaver/zips
+           @nozip not file
+           redir @nozip https://r2cdn.beatsaver.com{path} 302
+           file_server
+         }
          handle_path /cdn/* { root * /var/lib/plebleader/storage; file_server }
          handle_path /arcviewer/* { root * ${arcviewer}; file_server }
          handle_path /cors/cdn-assets-bl/* { reverse_proxy https://cdn.assets.beatleader.com { header_up Host {upstream_hostport} } }
          handle_path /cors/cdnbeatsaver/* { reverse_proxy https://cdn.beatsaver.com { header_up Host {upstream_hostport} } }
-         handle { root * ${plebleader-website}; try_files {path} /index.html; file_server }
+         # {path}/index.html serves /watch/; without it the SPA's 404 answers
+         handle { root * ${plebleader-website}; try_files {path} {path}/index.html /index.html; file_server }
        }
        handle { respond "Forbidden" 403 }
      '';
    };
    ```
 
-   ArcViewer's WebGL build may need `Content-Encoding` headers for `.br`/`.gz`
-   files in `Build/`. Check which ones the `deploy` branch ships.
+   Checked: the `deploy` branch ships uncompressed `Build/` files, so no
+   `Content-Encoding` headers are needed.
 
 **Check:** in Chromium, `https://incineroar.brenise.dev:4442/u/pleb` shows the
 profile and scores, and a replay opens in ArcViewer.
@@ -727,6 +840,51 @@ the local zip.
 
 ### Phase 7: NixOS deployment on incineroar
 
+**Built 2026-11-09, not switched yet.** nix-pleb branch `plebleader`
+(`2ada4b7`, in the worktree `~/ops/nix-pleb-plebleader`, based on `main`
+`981f9e7`) adds the inputs `plebleader-server` (server `0481759`) and
+`plebleader-website` (`8d540642`), both following `nixpkgs-unstable`, plus to
+`incineroar/workstation.nix`: user and group `plebleader` (with `caddy` in
+the group), `plebleader-api`, a one-shot `plebleader-import`, ArcViewer
+`fetchFromGitHub` and the 4442 vhost. The toplevel builds, and the system's
+Caddy adapts the generated Caddyfile. The sync unit and timer and the sops
+secret wait for phase 6's `sync` command. What differs from the bullets
+below:
+
+- Server fork: the private upstream submodules (`beatleader-server-private`,
+  `bs-simulator`) are removed from `.gitmodules`, because a
+  submodule-aware fetch fails on them. The flake sets `self.submodules`; the
+  nix-pleb input uses `?ref=pleb/main&submodules=1` over SSH (the models fork's
+  scp-style URL fetches fine). The package's `version` is a fixed `0.1.0`:
+  `buildDotnetModule` hands it to MSBuild, and a git short rev made restore
+  fail with no message (`NU1105` only in `project.assets.json`). 336 NuGet
+  packages in `deps.json`. The package was tested by hand (importer and API
+  on a scratch DB, all endpoints 200, `wwwroot` included).
+- Website: the yarn v1 flake from phase 4, not `buildNpmPackage`.
+- Units: `WorkingDirectory=/var/lib/plebleader/app` (data protection keys go
+  to `../keys`), `StateDirectory` for `app`, `db`, `storage`, `maps`,
+  `StateDirectoryMode=0750` and `UMask=0027`, so Caddy reads through the group
+  and nothing is world-readable. Paths and `PublicBaseUrl` come from the
+  fork's `appsettings.Mirror.json`, so the units only set
+  `ASPNETCORE_ENVIRONMENT=Mirror`, `ASPNETCORE_URLS` and `HOME`.
+
+To deploy (the user does this):
+
+1. Merge `plebleader` into nix-pleb `main`. The main checkout has uncommitted
+   `flake.lock` changes, so commit or stash those first, or cherry-pick the
+   commit and rerun `nix flake lock` if `flake.lock` conflicts. Then
+   `sudo --preserve-env=SSH_AUTH_SOCK nixos-rebuild switch --flake .#incineroar`.
+   `plebleader-api` starts on empty databases.
+2. Move the archive in (21 GB; `/home` and `/` are separate filesystems, so
+   this is a copy):
+   `doas cp -a ~/ops/beatsaber/plebsaber.stream/.data/plebleader/archive /var/lib/plebleader/`,
+   then `doas chown -R plebleader:plebleader /var/lib/plebleader/archive` and
+   `doas chmod -R g+rX,o-rwx /var/lib/plebleader/archive`. Keep the token chain
+   (`.data/plebleader/auth`) where it is until phase 6 moves the sync.
+3. `doas systemctl start plebleader-import`, then open
+   `https://incineroar.brenise.dev:4442/u/pleb` and a replay.
+4. Afterwards, remove the worktree with `git worktree remove ../nix-pleb-plebleader`.
+
 - Packages: `buildDotnetModule` for the fork (generate `deps.json` with
   `nix build .#…passthru.fetch-deps`), `buildNpmPackage` for the website (start
   with `npmDepsHash = lib.fakeHash`, then pin the `got:` value), and

flake.nix

@@ -44,12 +44,35 @@
           mainProgram = "plebsaber-stream";
         };
       };
+    # The plebleader archive CLI (scripts/archive, docs/plan.md): plain
+    # TypeScript with only Node built-ins, run through Node's type stripping.
+    mkArchiveCli = pkgs:
+      pkgs.stdenvNoCC.mkDerivation {
+        pname = "plebleader-archive";
+        version = "0.1.0";
+        src = ./scripts/archive;
+        nativeBuildInputs = [ pkgs.makeWrapper ];
+        installPhase = ''
+          runHook preInstall
+          mkdir -p $out/lib/plebleader-archive
+          cp *.ts $out/lib/plebleader-archive/
+          makeWrapper ${pkgs.nodejs_22}/bin/node $out/bin/plebleader-archive \
+            --add-flags "--experimental-strip-types --disable-warning=ExperimentalWarning" \
+            --add-flags "$out/lib/plebleader-archive/cli.ts"
+          runHook postInstall
+        '';
+        meta = {
+          description = "Archive exporter and daily sync for the personal BeatLeader mirror";
+          mainProgram = "plebleader-archive";
+        };
+      };
   in rec {
     packages = forAllSystems (system: let
       pkgs = import nixpkgs { inherit system; };
       pkg = mkPackage pkgs;
     in {
       plebsaber-stream = pkg;
+      plebleader-archive = mkArchiveCli pkgs;
       default = pkg;
     });
 

scripts/archive/beatleader.ts

@@ -3,11 +3,11 @@
 import * as path from 'node:path';
 import { BL_API } from './config.ts';
 import { get, limiters } from './http.ts';
-import { archivePath, eachWithProgress, exists, listFiles, readJson, save, saveSnapshotIfChanged, today, updateState } from './store.ts';
+import { archivePath, eachWithProgress, exists, listFiles, readJson, readState, save, saveSnapshotIfChanged, today, updateState } from './store.ts';
 
 const bl = (...p: string[]) => archivePath('beatleader', ...p);
 
-type BLScore = { id: number; timepost: number; replay?: string | null; leaderboard: { song: { hash: string } } };
+export type BLScore = { id: number; playerId: string; timepost: number; replay?: string | null; leaderboard: { song: { hash: string } } };
 
 async function blGet(pathAndQuery: string, accept?: number[]) {
 	return get(`${BL_API}${pathAndQuery}`, { limiter: limiters.beatleader, accept });
@@ -26,7 +26,9 @@ export async function exportProfile(playerId: string): Promise<void> {
 	const dir = (kind: string) => bl('players', playerId, kind);
 	const endpoints: [string, string][] = [
 		['profile', `/player/${playerId}?stats=true`],
-		['history', `/player/${playerId}/history`],
+		// count: days of history; the default is 50, which loses the triangle's
+		// monthly points. Upstream keeps one row per day since the first score.
+		['history', `/player/${playerId}/history?count=100000`],
 		['triangle', `/player/${playerId}/history/triangle`],
 		['accgraph', `/player/${playerId}/accgraph`]
 	];
@@ -37,12 +39,12 @@ export async function exportProfile(playerId: string): Promise<void> {
 	}
 }
 
-/** Page through best scores newest-first from `sinceUnix`, saving each score object. */
-export async function exportScores(playerId: string, sinceUnix: number): Promise<void> {
+/** Page through best scores newest-first from `sinceUnix`, saving each score object. Returns the new ones. */
+export async function exportScores(playerId: string, sinceUnix: number): Promise<BLScore[]> {
 	console.log(`BeatLeader scores for ${playerId} since ${new Date(sinceUnix * 1000).toISOString()}`);
-	let added = 0;
+	const added: BLScore[] = [];
 	let seen = 0;
-	let newest = 0;
+	let newest = Number(readState()[`beatleader:${playerId}`]?.newestTimepost ?? 0);
 	for (let page = 1; ; page++) {
 		const qs = `count=100&page=${page}&sortBy=date&order=desc&includeIO=true&time_from=${sinceUnix}`;
 		const res = await blGet(`/player/${playerId}/scores?${qs}`);
@@ -54,12 +56,18 @@ export async function exportScores(playerId: string, sinceUnix: number): Promise
 			if (exists(file)) continue;
 			// The page is parsed and each element re-serialized: same content, no reshaping.
 			save(file, JSON.stringify(score));
-			added++;
+			added.push(score);
 		}
-		console.log(`  page ${page}: ${seen}/${json.metadata.total} seen, ${added} new`);
+		console.log(`  page ${page}: ${seen}/${json.metadata.total} seen, ${added.length} new`);
 		if (json.data.length < 100 || seen >= json.metadata.total) break;
 	}
 	updateState(`beatleader:${playerId}`, { scoresSince: sinceUnix, newestTimepost: newest, lastScoresRun: new Date().toISOString() });
+	return added;
+}
+
+/** Newest archived score time for a player (unix seconds), 0 when there is none. */
+export function newestScoreTime(playerId: string): number {
+	return archivedScores().reduce((max, s) => (s.playerId === playerId ? Math.max(max, s.timepost) : max), 0);
 }
 
 export async function exportReplays(scores: BLScore[]): Promise<void> {

scripts/archive/cli.ts

@@ -9,6 +9,10 @@
 //                                   plus all legacy ScoreSaber maps)
 //   export all         [--since D]  bl, ss, maps (not ss-replays)
 //   report             [--since D]  counts and gaps in the archive
+//   sync    [--ratings] [--no-ss-replays]
+//                                   daily incremental run (phase 6): token refresh, new BL and SS
+//                                   scores with their replays, stats, leaderboards and maps.
+//                                   --ratings re-snapshots every played hash (default on Sundays)
 //
 // --since: YYYY-MM-DD, unix seconds, or "all" (default: all). --maps-since defaults to the two-year
 // window (DEFAULT_MAPS_SINCE). Every command skips files that already exist, so reruns only fill gaps.
@@ -16,10 +20,10 @@ import * as fs from 'node:fs';
 import * as path from 'node:path';
 import { ARCHIVE_DIR, BL_API, DEFAULT_MAPS_SINCE, LEGACY_PLAYER_ID, MAIN_PLAYER_ID, PLAYER_IDS, TOKENS_FILE, USER_AGENT, loadDotEnv, parseSince } from './config.ts';
 import { getAccessToken, login, readTokens, refresh } from './auth.ts';
-import { archivedScores, blHashes, exportLeaderboards, exportProfile, exportReplays, exportScoreStats, exportScores } from './beatleader.ts';
-import { archivedSSScores, exportSSPlayer, exportSSReplays, exportSSScores, ssHashes } from './scoresaber.ts';
+import { archivedScores, blHashes, exportLeaderboards, exportProfile, exportReplays, exportScoreStats, exportScores, newestScoreTime } from './beatleader.ts';
+import { archivedSSScores, exportSSPlayer, exportSSReplays, exportSSScores, newestSSScoreTime, ssHashes } from './scoresaber.ts';
 import { exportMaps } from './beatsaver.ts';
-import { archivePath, exists, listFiles, readState } from './store.ts';
+import { archivePath, exists, listFiles, readState, updateState } from './store.ts';
 
 function fmt(ms: number): string {
 	const d = new Date(ms);
@@ -106,6 +110,64 @@ async function exportSS(since: number): Promise<void> {
 	await exportLeaderboards([...new Set(PLAYER_IDS.flatMap((id) => ssHashes(id, since)))].sort());
 }
 
+/** Overlap of each incremental window, so scores posted late or with a skewed clock aren't missed. */
+const SYNC_OVERLAP_S = 24 * 3600;
+
+/**
+ * Phase 6: the daily run. Main account only; the legacy accounts are frozen. Every step skips
+ * what's already archived, so a rerun (or a run after a missed day) just fills the gap.
+ */
+async function sync(opts: { ratings: boolean; ssReplays: boolean }): Promise<number> {
+	const started = new Date();
+	// Token upkeep first: persist the rotated refresh token before anything else, and fail loudly.
+	const tokens = await refresh();
+	console.log(`Token refreshed; refresh chain valid until about ${fmt(tokens.refresh_expires_at)}\n`);
+
+	await exportProfile(MAIN_PLAYER_ID);
+	const blSince = Math.max(0, newestScoreTime(MAIN_PLAYER_ID) - SYNC_OVERLAP_S);
+	const newBL = await exportScores(MAIN_PLAYER_ID, blSince);
+	// The whole window, not just the new scores, so a replay or stats file that failed yesterday is retried.
+	const blWindow = archivedScores(blSince).filter((s) => s.playerId === MAIN_PLAYER_ID);
+	await exportReplays(blWindow);
+	await exportScoreStats(blWindow);
+	// A new score can change a leaderboard (status, ratings), so re-snapshot its hash.
+	const newBLHashes = [...new Set(newBL.map((s) => s.leaderboard.song.hash.toLowerCase()))].sort();
+	await exportLeaderboards(newBLHashes, true);
+
+	await exportSSPlayer(MAIN_PLAYER_ID);
+	const ssSince = Math.max(0, newestSSScoreTime(MAIN_PLAYER_ID) - SYNC_OVERLAP_S);
+	const newSS = await exportSSScores(MAIN_PLAYER_ID, ssSince);
+	const newSSHashes = [...new Set(newSS.map((it) => it.leaderboard.songHash.toLowerCase()))].sort();
+	await exportLeaderboards(newSSHashes.filter((h) => !newBLHashes.includes(h)));
+	if (opts.ssReplays) await exportSSReplays(MAIN_PLAYER_ID, ssSince);
+
+	// Maps for everything played in either window; existing zips are skipped.
+	const windowHashes = [...new Set([...blWindow.map((s) => s.leaderboard.song.hash.toLowerCase()), ...ssHashes(MAIN_PLAYER_ID, ssSince)])].sort();
+	await exportMaps(windowHashes);
+	const noZip = windowHashes.filter((h) => !exists(archivePath('beatsaver', 'zips', `${h}.zip`)));
+
+	if (opts.ratings) {
+		// Upstream rebalances ratings; a new dated snapshot is written only when a hash changed.
+		await exportLeaderboards(playedHashes(0), true);
+	}
+
+	const summary = {
+		lastRun: started.toISOString(),
+		finished: new Date().toISOString(),
+		newBeatLeaderScores: newBL.map((s) => s.id),
+		newScoreSaberScores: newSS.map((it) => it.score.id),
+		ratingsRefreshed: opts.ratings,
+		mapsWithoutZip: noZip
+	};
+	updateState('sync', summary);
+	console.log('');
+	console.log(`Sync done: ${newBL.length} new BeatLeader scores, ${newSS.length} new ScoreSaber scores${opts.ratings ? ', ratings refreshed' : ''}.`);
+	if (noZip.length) console.log(`Maps without a zip (see beatsaver/missing.json; the game's CustomLevels are the manual fallback): ${noZip.join(' ')}`);
+	const failures = readState().failures;
+	if (failures && Object.keys(failures).length) console.warn(`Recorded failures (retried on the next run): ${JSON.stringify(failures)}`);
+	return 0;
+}
+
 function report(since: number, mapsSince: number): void {
 	const count = (...p: string[]) => listFiles(archivePath(...p)).length;
 	const day = (t: number) => (t ? new Date(t * 1000).toISOString().slice(0, 10) : 'all time');
@@ -218,12 +280,17 @@ async function main(): Promise<number> {
 				return 0;
 		}
 	}
+	if (cmd === 'sync') {
+		// Sundays in the machine's timezone (America/Los_Angeles on incineroar).
+		const ratings = args.includes('--ratings') || new Date().getDay() === 0;
+		return sync({ ratings, ssReplays: !args.includes('--no-ss-replays') });
+	}
 	if (cmd === 'report') {
 		report(since, mapsSince);
 		return 0;
 	}
 
-	console.error('Usage: node scripts/archive/cli.ts <auth login|refresh|status | export bl|ss|ss-replays|maps|all | report> [--since YYYY-MM-DD|all] [--maps-since YYYY-MM-DD|all]');
+	console.error('Usage: node scripts/archive/cli.ts <auth login|refresh|status | export bl|ss|ss-replays|maps|all | report | sync [--ratings] [--no-ss-replays]> [--since YYYY-MM-DD|all] [--maps-since YYYY-MM-DD|all]');
 	return 2;
 }
 

scripts/archive/scoresaber.ts

@@ -26,10 +26,10 @@ export async function exportSSPlayer(playerId: string): Promise<void> {
 	console.log(`ScoreSaber player ${playerId}: ${saveSnapshotIfChanged(ss(playerId, 'player'), r.body) ? 'saved' : 'unchanged'}`);
 }
 
-/** Page `sort=recent` until scores are older than `sinceUnix`. */
-export async function exportSSScores(playerId: string, sinceUnix: number): Promise<void> {
+/** Page `sort=recent` until scores are older than `sinceUnix`. Returns the new items. */
+export async function exportSSScores(playerId: string, sinceUnix: number): Promise<SSItem[]> {
 	console.log(`ScoreSaber scores for ${playerId} since ${new Date(sinceUnix * 1000).toISOString()}`);
-	let added = 0;
+	const added: SSItem[] = [];
 	let seen = 0;
 	for (let page = 1; ; page++) {
 		const r = await get(`${SS_API}/player/${playerId}/scores?limit=100&page=${page}&sort=recent`, { limiter: limiters.scoresaber });
@@ -44,12 +44,18 @@ export async function exportSSScores(playerId: string, sinceUnix: number): Promi
 			const file = ss(playerId, 'scores', `${item.score.id}.json`);
 			if (exists(file)) continue;
 			save(file, JSON.stringify(item));
-			added++;
+			added.push(item);
 		}
-		console.log(`  page ${page}: ${seen} in window, ${added} new (total on server ${json.metadata.total})`);
+		console.log(`  page ${page}: ${seen} in window, ${added.length} new (total on server ${json.metadata.total})`);
 		if (reachedCutoff || json.playerScores.length < 100) break;
 	}
 	updateState(`scoresaber:${playerId}`, { scoresSince: sinceUnix, lastScoresRun: new Date().toISOString() });
+	return added;
+}
+
+/** Newest archived ScoreSaber score time for a player (unix seconds), 0 when there is none. */
+export function newestSSScoreTime(playerId: string): number {
+	return archivedSSScores(playerId).reduce((max, it) => Math.max(max, Math.floor(Date.parse(it.score.timeSet) / 1000)), 0);
 }
 
 /** Download replays for archived scores with hasReplay, oldest first, ~10.5 s apart. */