AssessNow — Hosting a Proctored Exam Platform on a Mac mini

A mentoring log, not a runbook. Each section is one small, practicable part: what the piece is, why it's there, the actual config that's running, one thing you can do right now to see it work, and the check that proves you understood it.

AssessNow runs proctored technical assessments — candidates open a link, grant camera/mic/screen, answer questions, and run code that the platform executes. That code is untrusted code from strangers on the internet. And the whole thing is hosted on a Mac mini sitting in a room, on a residential-ish uplink, with no open ports on the router and no cloud hosting bill.

That combination — public HTTPS, untrusted code execution, zero inbound firewall holes, one box — is what makes the hosting interesting. Here's how the pieces fit.

0. The mental model: the request path, end to end

Before any config, hold this in your head. Everything below is a detail of this diagram.

Candidate browser
   │  HTTPS  candidate.assessnow.online
   ▼
Cloudflare edge  ── TLS terminated here ──┐
   │                                       │  (apex assessnow.online → Cloudflare Pages,
   │  tunnel (outbound-only, from the mini)│   the marketing site — never touches the mini)
   ▼
cloudflared  (container on the mini)
   │  http://caddy:4820
   ▼
Caddy  (edge proxy, host-based routing)
   ├── recruiter.assessnow.online → dashboard:80  ─┐
   └── candidate.assessnow.online → frontend:80   ─┤ each of these is *itself* a Caddy
                                                    │ serving a built SPA and proxying
                                                    │ /api + /auth → api:4820
                                                    ▼
                                              api (Node/Express) ──▶ postgres
                                                    │              └▶ redis
                                                    ▼
                                              redis queue ──▶ worker  (untrusted code runs HERE,
                                                                       on a network with no internet)

Two claims in that diagram are the whole design:

1. Nothing on the mini listens to the internet. The mini dials out to Cloudflare.
2. The container that runs candidate code cannot reach the internet or the database. It can reach exactly one thing: Redis.

Everything else is plumbing in service of those two.

1. Cloudflare Tunnel — public HTTPS with zero open ports

The idea. Normally, "put this on the internet" means: get a static IP or dynamic DNS, forward ports 80/443 on the router to the machine, get a TLS cert, and now your box is being port-scanned by everyone forever. A tunnel inverts it. A daemon on the mini opens a persistent outbound connection to Cloudflare's edge. Traffic arrives at Cloudflare, and Cloudflare pushes it down the connection the mini already opened. The router never accepts an inbound connection. There is no port to scan.

What's running (cloudflared-config.yml):

tunnel: assessnow
credentials-file: /etc/cloudflared/1922b192-....json

ingress:
  # The apex (assessnow.online) is served by Cloudflare Pages (marketing
  # site) and is intentionally not routed through this tunnel.
  - hostname: recruiter.assessnow.online
    service: http://caddy:4820
  - hostname: candidate.assessnow.online
    service: http://caddy:4820
  - service: http_status:404

ingress is an ordered match list — first hostname that matches wins, and the last entry (no hostname) is the catch-all. The http_status:404 at the bottom is deliberate: anything reaching the tunnel with a Host header you didn't authorise gets a 404, not a peek at your app.

The credentials file is the tunnel's identity — it's how Cloudflare knows this daemon is allowed to serve *.assessnow.online. It lives on the host, not in the image:

volumes:
  - ./cloudflared-config.yml:/etc/cloudflared/config.yml:ro
  - /Users/fadhilhanri/.cloudflared/1922b192-....json:/etc/cloudflared/...json:ro
  - /Users/fadhilhanri/.cloudflared/cert.pem:/etc/cloudflared/cert.pem:ro

Read-only bind mounts from the host. Rebuild the image a hundred times; the secret never enters a layer.

The consequence people miss: TLS is terminated at Cloudflare, not on the mini. Everything from cloudflared inward is plain HTTP. That's why the Caddyfile says :4820 and not assessnow.online { tls ... } — Caddy's famous auto-HTTPS is deliberately not used here. It would be pointless (Cloudflare already did it) and impossible (no port 80 reachable for the ACME challenge).

docker compose logs cloudflared | grep -i "registered\|connection"

2. Caddy — two layers of routing, and why there are three Caddyfiles

This is the part that confuses people reading the repo for the first time. There are three Caddy configs, and they do different jobs.

Layer 1: the edge Caddy (/Caddyfile) — routes by hostname

:4820 {
	encode gzip

	@recruiter host recruiter.assessnow.online
	handle @recruiter {
		reverse_proxy dashboard:80
	}

	@candidate host candidate.assessnow.online
	handle @candidate {
		header {
			-Server
			X-Content-Type-Options "nosniff"
			Referrer-Policy "strict-origin-when-cross-origin"
			Strict-Transport-Security "max-age=31536000; includeSubDomains"
		}
		reverse_proxy frontend:80
	}

	handle { ...same headers... reverse_proxy frontend:80 }   # fallback
}

One port, two hostnames, two different apps. dashboard:80 and frontend:80 are Docker DNS names — inside a Compose network, the service name resolves to the container. No IPs, no ports on the host.

Note -Server: strip the header that tells an attacker what you're running. Free, do it.

Layer 2: the per-app Caddy (frontend/Caddyfile, dashboard/Caddyfile) — routes by path

Each frontend image is a two-stage build: Node builds the SPA, then the artifacts get copied into a caddy:2 image.

FROM node:20-bookworm-slim AS build
...
RUN npx vite build --base=/

FROM caddy:2
COPY --from=build /app/dist /srv
COPY frontend/Caddyfile /etc/caddy/Caddyfile

And that inner Caddyfile:

:80 {
	@api path /auth/* /api/*
	handle @api {
		reverse_proxy api:4820
	}

	handle {
		root * /srv
		try_files {path} /index.html   # SPA fallback — /t/<token> is a client route
		file_server
	}
}

Why not just point the edge Caddy at the API directly? Because of one word: same-origin. The candidate app is served from candidate.assessnow.online, and its API calls go to candidate.assessnow.online/api/*. Same origin → the session cookie is sent without SameSite gymnastics, no CORS preflights, no credentials: 'include' bugs at 2 a.m. before an exam window. The API appears to live at the same address as the app; it just doesn't.

The try_files {path} /index.html line is what makes /t/<abc123> work. That path doesn't exist on disk — it's a client-side route. Without the fallback, every candidate link is a 404.

3. Compose as the unit of deployment

The mini doesn't "have Node installed and run the app." It runs one command:

docker compose up -d --build

Every service — Postgres, Redis, API, two worker pools, two frontends, Caddy, the tunnel — is declared in docker-compose.yml. That file is the deployment. Three things in it are worth internalising.

restart: unless-stopped

The power flickers. The mini reboots. Docker's daemon comes back, and every container marked unless-stopped comes back with it. This is what replaced the old Electron deploy's embarrassing manual step: "after every reboot, unlock the app and press Start runner." Now the only manual step is that the host must reach a desktop session (FileVault, auto-login) with the container runtime set to launch at login.

depends_on with conditions (not just names)

api:
  depends_on:
    postgres:
      condition: service_healthy
    migrate:
      condition: service_completed_successfully

Plain depends_on: [postgres] only waits for the container to start — which for Postgres means "the process exists," not "the database accepts connections." That's a classic startup race: the API boots, connects, gets ECONNREFUSED, crashes, restarts, and you learn to live with a noisy log. The fix is Postgres's healthcheck:

healthcheck:
  test: ["CMD-SHELL", "pg_isready -U assessnow"]
  interval: 5s
  timeout: 3s
  retries: 20

condition: service_healthy waits for that to pass. Startup order becomes a fact, not a hope.

The one-off migrate job

migrate:
  command: ["node", "scripts/migrate.js"]
  depends_on:
    postgres: { condition: service_healthy }
  restart: "no"

Schema migrations don't run inside the API's boot sequence. They run in a separate container that runs once and exits, and both api and analysis-worker wait on service_completed_successfully.

Why bother? Because the day you run docker compose up -d --scale api=3, three API containers boot simultaneously and three of them try to ALTER TABLE. Migration tools usually take an advisory lock, so you probably survive — but "probably" is not a deployment strategy. Hoisting migrations into a one-off job means there is exactly one migrator, always, by construction. (The comment in the file labels this OPS-1.)

docker compose ps            # who's up, who's healthy
docker compose logs migrate  # ran once, exited 0

4. The two networks — the most important 8 lines in the repo

AssessNow executes code written by candidates. Assume, always, that one of them will try to escape the sandbox. server/src/sandbox.js runs the code in a locked-down node:vm inside a child process — that's fence #1. Fence #2 is the network topology, and it's the one that saves you when fence #1 has a bug:

networks:
  default:                 # has internet — everything else
  sandbox_internal:
    internal: true         # NO gateway → no internet, no host, nothing

worker:
  command: ["node", "src/worker.js"]
  networks:
    - sandbox_internal     # ← ONLY this one
  read_only: true
  tmpfs: [/tmp]
  cap_drop: [ALL]
  security_opt: [no-new-privileges:true]
  pids_limit: 256
  mem_limit: 512m

Read that list slowly, because each line is a specific attack it kills:

Notice that redis is on both networks. That's the deliberate seam: the API (on default) pushes a job into Redis; the worker (on sandbox_internal) pulls it out. Redis is the only thing that spans the airlock, and it carries data, not connectivity.

This is defence in depth done properly: not "we have a sandbox," but "assume the sandbox is broken — what can the attacker reach?" Answer: one Redis instance and 512 MB of RAM.

docker compose exec worker sh -c "getent hosts postgres || echo 'no route to postgres'"
docker compose exec worker sh -c "wget -qO- -T3 https://example.com || echo 'no internet'"
docker compose exec worker sh -c "touch /app/x || echo 'read-only rootfs'"

5. What is durable and what is disposable

Self-hosting means you own the durability question. Sort every piece of state into one of two boxes:

Disposable (rebuilt by docker compose up --build)
Container filesystems, the built SPAs, the API process, Redis (--save "" --appendonly no — deliberately not persisted; it's a queue and a rate-limit counter, not a database).

Durable (lose this and you're finished)

• pgdata — the named volume holding Postgres. Every assessment, attempt, invitation, integrity event.
• server/.env — SESSION_SECRET, OAuth secrets, Xendit keys. Not in git, not in the image.
• ~/.cloudflared/*.json + cert.pem — the tunnel's identity.
• Proctoring media in Cloudflare R2 (off-machine already — good).

The honest gap: docker compose down -v deletes pgdata. One flag. SHARED-BACKEND-PLAN.md §10 says a nightly pg_dump on 7-day rotation should mirror the old com.assesslocal.backup LaunchAgent — but that plan is a proposal, and the old LaunchAgent backs up the retired Electron JSON store, not Postgres. Until a pg_dump cron exists, the system of record for a live product has no automated off-machine backup. See §7.

6. The macOS-specific bits (the part no tutorial covers)

Docker on macOS is not Docker on Linux — containers run inside a Linux VM (Docker Desktop / OrbStack / Colima). Consequences that actually bit this deploy:

• user: "501:20" on cloudflared. That's a macOS UID/GID — 501 is the first human user, 20 is staff. It's there so the container's process can read the bind-mounted credentials with the host's file ownership intact.
• The runtime must start at login. Containers can't restart if the VM isn't running. With FileVault on (and it must be — candidate PII and ID photos are on that disk), the mini can't fully boot unattended: someone types the disk password. That's an accepted trade: physical security and encryption over unattended reboot.
• The mini must never sleep. sleep 0, womp 1 (wake for network), autorestart 1 (start after power failure). Display can sleep; the machine cannot, or the tunnel drops mid-exam.
• It is a single point of failure, on purpose. No hosting bill, and the architecture is container-portable — docker compose up on a VPS is the escape hatch, with no rewrite. That was the explicit trade in the plan's decision table.

7. What actually breaks first

Ranked by "how likely is this to ruin an exam window":

1. Power / internet at the mini. No redundancy. A UPS is the cheapest resilience you can buy; wired Ethernet beats Wi-Fi (the deploy log admits it's still on Wi-Fi, en1).
2. No Postgres backup. §5. This is the highest-severity gap in the whole setup and it costs an hour to close.
3. Compose's depends_on: [caddy] on the tunnel is start-order, not readiness. If cloudflared registers before Caddy is accepting connections, you get 502s at the edge for a few seconds after boot. Harmless — unless it happens during an exam.
4. CPU contention. analysis-worker (face-count, diarization) is CPU-heavy and shares the mini with everything else. The compose file already flags it: scale it or move it to its own host.
5. Stale docs. docs/DEPLOY-MACMINI.md and PROMPT-DEPLOY-MACMINI.md describe the retired Electron deploy (npm run desktop, port 4820 on the host, cloudflared as a LaunchDaemon). Anyone — including a future you, or an agent reading the repo — who follows them will deploy the wrong architecture.

Your reps

Reading questions (answer before checking the files)

1. A candidate's browser requests https://candidate.assessnow.online/api/run with a session cookie. Name every process it passes through, in order, and say exactly where TLS ends. Then: which of those hops would have to change if you moved the whole stack to a VPS tomorrow?
2. redis is on both default and sandbox_internal. Suppose you "simplified" it onto default only. Compose still starts. What breaks, when, and what would the error look like? (Now the real one: suppose instead you put worker on default as well as sandbox_internal, so it could report metrics somewhere. What security property did you just delete, and would any test catch it?)
3. server/.env sets USE_QUEUE=1, and docker-compose.yml has a commented-out SANDBOX_INPROC_FALLBACK: "0" with the note "Fail closed if the code-run queue is down so execution never silently moves off the hardened worker onto the api." Trace it: if Redis dies mid-exam as currently configured, where does candidate code execute? Which of the six protections in the §4 table does it still have?