Courses › System Design › Question Breakdowns

Design LeetCode

Async judging, layered sandboxes for untrusted code, contest spikes and penalty-based leaderboards.

Design an online judge like LeetCode: users browse problems, write code in the browser, submit it, and get a verdict (Accepted, Wrong Answer, Time Limit Exceeded…) after it runs against hidden test cases. Contests add a live leaderboard and a huge submission spike. It's a question about running untrusted code safely, async job processing, scaling for contest spikes and ranking.

Requirements

Functional

  • Users can browse problems and view a problem's statement and examples
  • Users can submit code in several languages and get a verdict with runtime/memory
  • Users can run code against their own custom input
  • During a contest, users see a live leaderboard

Out of scope: discussions, editorial content, user profiles beyond submission history, plagiarism detection.

Non-functional

  • Security: user code is untrusted and must be fully isolated — no network, no access to other submissions or the host
  • Fast feedback: verdict within ~5 seconds typically
  • Scale: ~10k submissions/minute normally; a contest with 100k participants can hit thousands of submissions/second in the first minutes
  • Fair and consistent: the same code gets the same verdict and comparable runtimes
  • Availability for browsing; submissions may queue briefly under load

[!TIP] The core is an async job queue Executing code takes seconds and must never block a web server. Submit → enqueue → workers run it in a sandbox → store the verdict → the client polls or is pushed the result. Once that's on the board, spend your time on the sandbox and the contest spike.

Capacity estimates

Core Entities

  • Problem — statement, examples, limits (time, memory), per-language templates, hidden test cases (in object storage)
  • Submission — user, problem, language, code, status, verdict, runtime, memory, contest ID
  • Contest — start, end, problems, scoring rules
  • Leaderboard entry — contest, user, score, penalty

API

GET  /v1/problems?tag=dp&cursor=...
GET  /v1/problems/{slug}

POST /v1/problems/{slug}/submissions   { "language": "python3", "code": "..." }
→ 202 { "submissionId": "sub_918", "status": "queued" }

GET  /v1/submissions/{id}   → { "status": "done", "verdict": "Accepted", "runtimeMs": 44, "memoryMb": 17.1 }
GET  /v1/contests/{id}/leaderboard?page=1

High-Level Design

user = client "Browser" @0,1
api = service "API Servers" @1,1
pdb = store "Problems DB" @1,0
sdb = store "Submissions DB" @2,0
q = queue "Submission Queue\n(per language)" @2,1
w = service "Judge Workers\n(sandboxed)" @3,1
tests = blob "Test Cases (S3)" @4,1
lb = cache "Contest Leaderboard\n(Redis)" @2,2
user -> api
api -> pdb
api -> sdb : status queued
api -> q
w -> q : pull
tests --> w : cached locally
w -> sdb : verdict
w -> lb : on accept
user -> api : poll / SSE
  • Submit: the API stores the submission as queued and pushes it onto a queue.
  • Judge: a worker pulls it, compiles and runs the code in a sandbox against the problem's test cases (cached locally), and stops at the first failure.
  • Report: the worker writes the verdict; the client polls (or gets an SSE event). For contests, an accepted verdict updates the leaderboard.

Deep Dives

1. Sandboxing untrusted code

User code will try to fork-bomb, read files, open sockets, allocate 64 GB and loop forever. Layers of defense:

Containers alone share the host kernel; for stronger isolation use gVisor or Firecracker microVMs, at a small startup cost. Keep a warm pool of ready sandboxes per language so a submission doesn't pay container startup.

2. Judging a submission

  • Compile once, then run each test case with the time and memory limits. Compare output exactly or with a per-problem checker (floating-point tolerance, any valid answer).
  • Stop on first failure and report which test failed (without revealing hidden input).
  • Test data is large and read-heavy: store it in S3, cache on worker disks by problem version.
  • Measure runtime with CPU time, not wall-clock, and run on homogeneous worker instances so runtimes are comparable.

3. Run vs. submit

"Run" (custom input, a few tests) and "Submit" (all hidden tests) have different costs and urgency. Use separate queues so a flood of runs doesn't delay submissions, and rate-limit runs per user.

4. Surviving the contest spike

  • Pre-scale workers before a scheduled contest start — autoscaling reacts too slowly for a spike that hits in seconds.
  • The queue absorbs the burst: users see "Judging…" a little longer rather than errors.
  • Per-user limits (e.g. one in-flight submission per problem, a short cooldown) stop retry storms.
  • Prioritize contest submissions over practice ones during the contest.

[!WARNING] Don't let a worker crash lose a submission Workers ack the queue message only after writing the verdict. If a worker dies, the message becomes visible again and another worker re-judges it — judging is deterministic, so running twice is harmless.

5. Contest leaderboard

ACM-style ranking: more problems solved ranks higher; ties broken by total penalty (time to each accept + 5–20 minutes per wrong attempt). Encode both into one sortable score — e.g. solved × 10^9 − penaltySeconds — and keep it in a Redis sorted set per contest (see Design a Leaderboard). Update it when a submission is accepted; recompute from submissions if needed. Many contests freeze the public leaderboard for the final hour.

6. Pushing results

Polling every second is fine at normal load. At contest scale (100k users each polling), switch to SSE/WebSocket notifications: the worker publishes submission_done to pub/sub, and the connection server holding that user's socket pushes it.

What's expected at each level

  • Mid-level: submissions through a queue to workers, verdicts stored and polled, containers for isolation.
  • Senior: a layered sandbox (namespaces, cgroups, seccomp, no network), warm pools, idempotent judging with late acks, separate run/submit queues.
  • Staff: contest pre-scaling and prioritization, runtime fairness, gVisor/Firecracker trade-offs, penalty-based ranking and leaderboard freezes, test data distribution to workers.