v1.7: Postgres migration (WIP — tasks 1-4 done, 5 has open defects)

.github/workflows/test.yml

@@ -0,0 +1,30 @@
+name: Tests
+on: [push, pull_request]
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        backend: [pg, sqlite]
+    services:
+      postgres:
+        image: postgres:16
+        env:
+          POSTGRES_PASSWORD: postgres
+          POSTGRES_USER: postgres
+          POSTGRES_DB: postgres
+        ports: ['5432:5432']
+        options: >-
+          --health-cmd pg_isready --health-interval 10s
+          --health-timeout 5s --health-retries 5
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 20
+      - run: npm ci
+      - run: npm test
+        env:
+          TEST_BACKEND: ${{ matrix.backend }}
+          TEST_DATABASE_URL: postgresql://postgres:postgres@localhost:5432/postgres

README.md

@@ -175,6 +175,46 @@ npm run dev
 
 Visit the client dev server's printed URL (usually `http://localhost:5173`).
 
+## Running the tests
+
+```bash
+npm test          # Postgres backend (default) - boots a throwaway container
+npm run test:sqlite   # SQLite backend, no container needed
+npm run test:all      # both, sqlite then pg
+```
+
+The Postgres backend needs a container runtime that speaks the Docker API.
+`tests/setup/pg-global.js` boots one shared Postgres 16 container via
+[Testcontainers](https://node.testcontainers.org) and each test file carves
+out its own database from it (`tests/helpers/backend.js`), so files can't
+see each other's rows.
+
+- **Docker**: works out of the box, nothing to configure.
+- **Podman** (what this repo's containers were validated against; no
+  `docker` binary required): start the user socket once per login session
+  and Testcontainers will find it automatically -
+  `tests/setup/pg-global.js` points `DOCKER_HOST` at the Podman socket
+  itself if `DOCKER_HOST` isn't already set, so no per-developer config is
+  needed:
+
+  ```bash
+  systemctl --user start podman.socket
+  ```
+
+  Rootless Podman can't grant Testcontainers' Ryuk reaper the privileges it
+  wants, so the harness also sets `TESTCONTAINERS_RYUK_DISABLED=true` by
+  default when using Podman. With Ryuk off, the container is stopped and
+  removed by `teardown()` in `pg-global.js` at the end of the run instead -
+  if a run is killed hard enough to skip that (e.g. `SIGKILL`), clean up any
+  leftovers with `podman ps -a` / `podman rm -f`.
+- **CI** sets `TEST_DATABASE_URL` directly against a Postgres service
+  container (see `.github/workflows/test.yml`) and never touches
+  Testcontainers at all.
+
+To point manually at a different runtime or disable Ryuk yourself, set
+`DOCKER_HOST` and/or `TESTCONTAINERS_RYUK_DISABLED` before running the
+tests - the harness only fills these in when they're unset.
+
 ## Live Events
 
 At most one event is active globally at a time. Each is a set of tunable

docs/postgres-migration-runbook.md

@@ -0,0 +1,248 @@
+# Postgres Migration — Backup, Cutover and Rollback Runbook
+
+**Status: NOT READY TO RUN.** Parts B and C describe machinery that is still
+being built (see §0). **Part A is valid today and you should do it now.**
+
+Applies to the Unraid install at `/mnt/user/appdata/rackstack-server/data`.
+
+---
+
+## 0. What actually exists right now
+
+| Piece | Task | State |
+|---|---|---|
+| Async db interface | 1 | ✅ merged to branch |
+| Facade + SQLite driver | 2 | ✅ merged to branch |
+| Postgres test harness | 3 | ✅ merged to branch |
+| Postgres schema + driver | 4 | ✅ merged to branch |
+| `identities` auth split | 5 | ⚠️ **built but has 4 unfixed defects** |
+| **The migrator itself** (`npm run migrate:pg`) | 6 | ❌ **not written** |
+| Auto-migrate on boot | 7 | ❌ not written |
+| Compose/Unraid/docs | 8 | ❌ not written |
+
+**There is no migration tool yet.** Task 5's open defects include one that
+causes a permanent account lockout on partial write, and one where SQLite's
+foreign-key check runs after `COMMIT` so it cannot roll back the rebuild it is
+supposed to guard. Do not point this branch at production data.
+
+---
+
+## Part A — Back up now (do this today)
+
+Valid regardless of migration timing, and it is what unblocks verification.
+
+### A1. Stop the container first
+
+```
+Unraid → Docker → rackstack-server → Stop
+```
+
+**Do not skip this.** SQLite keeps recently-committed writes in a separate
+write-ahead log. Copying a running database captures the main file without the
+log, so the newest progress — potentially hours of it — is silently absent from
+your backup. The copy will look valid and open fine. It will just be stale.
+
+### A2. Copy all three files
+
+```bash
+cd /mnt/user/appdata/rackstack-server
+mkdir -p /mnt/user/backups/rackstack/$(date +%F)
+cp -av data/rackstack.db     /mnt/user/backups/rackstack/$(date +%F)/
+cp -av data/rackstack.db-wal /mnt/user/backups/rackstack/$(date +%F)/ 2>/dev/null || true
+cp -av data/rackstack.db-shm /mnt/user/backups/rackstack/$(date +%F)/ 2>/dev/null || true
+ls -la /mnt/user/backups/rackstack/$(date +%F)/
+```
+
+`-wal` and `-shm` may legitimately be absent if the database was closed cleanly
+— that is why those two lines tolerate failure. `rackstack.db` must be there.
+
+### A3. Verify the backup opens and holds what you expect
+
+```bash
+cd /mnt/user/backups/rackstack/$(date +%F)
+sqlite3 rackstack.db "PRAGMA integrity_check;"
+sqlite3 rackstack.db "SELECT name FROM sqlite_master WHERE type='table' ORDER BY name;"
+sqlite3 rackstack.db "SELECT count(*) AS users FROM users;"
+sqlite3 rackstack.db "SELECT count(*) AS saves FROM saves;"
+sqlite3 rackstack.db "SELECT user_id, length(data) AS bytes, datetime(last_save/1000,'unixepoch') FROM saves;"
+```
+
+`integrity_check` must print `ok`. Record the user and save counts — they are
+what the migration verifies against later.
+
+### A4. Restart the container
+
+```
+Unraid → Docker → rackstack-server → Start
+```
+
+Confirm you can log in and your save is intact before walking away.
+
+### A5. Send the export for verification
+
+Copy that dated backup folder somewhere I can read it. It is used to prove the
+migration reproduces every save byte-for-byte, and to confirm each stored
+`provider_id` matches what SuperTokens will compute in v1.8 — the check that
+prevents a player silently landing on a brand-new empty save after the auth
+change.
+
+If you would rather not share save contents, the counts and shapes from A3 plus
+the output of
+
+```bash
+sqlite3 rackstack.db "SELECT id, provider, provider_id, username IS NOT NULL FROM users;"
+```
+
+are enough for the identity-mapping half, though not for the byte-for-byte half.
+
+---
+
+## Part B — Cutover (once Tasks 6–8 land)
+
+### B1. Stand up Postgres
+
+Add an official `postgres:16` container in Unraid with its **own** appdata path
+(not rackstack's). Create a database and a user for it:
+
+```sql
+CREATE USER rackstack WITH PASSWORD 'choose-something-long';
+CREATE DATABASE rackstack OWNER rackstack;
+```
+
+### B2. Take a fresh backup
+
+Repeat Part A. The backup from a week ago is not the one you want to restore
+from if something goes wrong tonight.
+
+### B3. Configure rackstack
+
+Set on the rackstack container:
+
+```
+DATABASE_URL=postgresql://rackstack:PASSWORD@192.168.x.x:5432/rackstack
+```
+
+Three ways this line commonly goes wrong:
+
+- It must be `postgresql://`. `postgres://` is rejected outright.
+- Use the host's LAN IP, not `localhost` or `127.0.0.1` — inside the container
+  those point at the container itself.
+- **Leave the `/app/data` volume mapping in place.** It is the migration source
+  and your rollback path. Removing it is the one irreversible mistake here.
+
+### B4. Start and watch
+
+```
+Unraid → Docker → rackstack-server → Start, then the log
+```
+
+Expected: a `[migrate]` line per table with a verified row count, then
+`committed`, then `listening on :3000`.
+
+Compare those counts against what you recorded in A3. They must match exactly.
+
+If migration fails, **the container refuses to start by design.** That is not a
+malfunction. Serving an empty game over live save data is worse than being
+down: a stopped container gets investigated, an empty leaderboard might not be
+noticed until saves have been overwritten on top of it. The log names the table
+that failed.
+
+### B5. Verify before you walk away
+
+- Log in with **both** Discord and GitHub if you use both.
+- Confirm your wafers/racks match what you had.
+- Check the admin dashboard loads — that proves `SUPER_ADMIN_IDS` still
+  resolves, which it should, because `users.id` is deliberately unchanged by
+  this migration.
+- Check the leaderboard renders with real numbers, not blanks. Blank
+  `rungsClaimed` values would indicate the camelCase-alias problem.
+
+---
+
+## Part C — Rollback
+
+Four layers, cheapest first. **Read C0 before you need it.**
+
+### C0. The one-way door
+
+Rollback is free **until players start making progress on Postgres.**
+
+Your SQLite file is never modified or deleted by the migration, so reverting to
+it always works mechanically. But it is frozen at the moment of cutover. Once
+players have been on Postgres for a while, going back to SQLite silently
+discards everything earned since — and nobody gets an error, they just find
+their progress reverted.
+
+So: verify immediately after cutover (B5), and decide fast. A rollback ten
+minutes after cutover costs nothing. A rollback three days later costs three
+days of everyone's progress.
+
+If you must roll back late, take a Postgres dump first so the interim progress
+is at least recoverable:
+
+```bash
+pg_dump -U rackstack -h 192.168.x.x rackstack > rackstack-postgres-$(date +%F).sql
+```
+
+### C1. Migration failed, container won't start
+
+Nothing was committed — the migrator runs in a single transaction and rolls the
+whole thing back on any verification mismatch. Your SQLite data is untouched.
+
+```
+Remove DATABASE_URL from the container config → Start
+```
+
+You are back on SQLite exactly as before. Send me the `[migrate]` log lines.
+
+### C2. Migration succeeded but the app misbehaves
+
+```
+Remove DATABASE_URL → Restart
+```
+
+Back on SQLite. Subject to C0 — fine immediately after cutover, lossy later.
+
+Leave the Postgres database in place rather than dropping it; it is evidence.
+
+### C3. SQLite data itself looks wrong
+
+```
+Stop the container
+cd /mnt/user/appdata/rackstack-server/data
+mv rackstack.db rackstack.db.suspect
+cp -av /mnt/user/backups/rackstack/<DATE>/rackstack.db* .
+Start the container
+```
+
+Restore all three files if all three were backed up. Keep the suspect copy.
+
+### C4. Full revert to the previous release
+
+Point the container back at the last known-good image tag:
+
+```
+Repository: ghcr.io/neverendingcode/rackstack-server:1.6.0
+```
+
+then apply C3. v1.6.0 has no knowledge of Postgres or of the `identities`
+table, so it needs a pre-migration SQLite file — which is exactly what your
+Part A backup is.
+
+**Do not change `JWT_SECRET` at any point during any of this.** Changing it
+logs out every player, which looks alarmingly like data loss and will send you
+chasing the wrong problem.
+
+---
+
+## Quick reference
+
+| Situation | Action |
+|---|---|
+| Before you touch anything | Part A, container stopped, all three files |
+| Migration failed | Remove `DATABASE_URL`, restart — nothing was committed |
+| App misbehaving right after cutover | Remove `DATABASE_URL`, restart |
+| App misbehaving days after cutover | `pg_dump` first, then remove `DATABASE_URL` — you will lose interim progress |
+| Save data looks wrong | Restore backup files (C3) |
+| Need the old version back | Pin image to `1.6.0`, then C3 |
+| Logins broke | Check `JWT_SECRET` is unchanged before anything else |