Courses › System Design › Question Breakdowns

Design a Data Labeling Platform

Leased tasks for human workers, gold tasks, review and consensus, model-assisted labeling and SLAs.

Design a data labeling platform like Scale AI's (or an in-house one like Tesla's autopilot labeling): customers upload images, video, text or LLM conversations; a large workforce of human labelers annotates them (boxes, classes, transcriptions, preference rankings); and labels are quality-checked before being delivered as training data. It's a question about task distribution to humans, quality control (consensus, gold tasks, review), throughput and SLAs, large media handling, and model-assisted labeling.

Requirements

Functional

  • Customers create projects with instructions and a label schema, and upload batches of items
  • Items are split into tasks and assigned to labelers with the right skills; labelers annotate in a web tool
  • Labels go through quality control (review, consensus, gold tasks) before acceptance
  • Customers track progress and export accepted labels
  • (Variant) RLHF: labelers rank or write LLM responses

Out of scope: labeler payments and hiring, the annotation UI's drawing tools, model training.

Non-functional

  • Throughput: ~10M tasks/day across ~50k concurrent labelers
  • Quality: label accuracy targets (e.g. 98%) per project, measured continuously
  • SLA: batches done within a deadline (hours to days); urgent batches prioritized
  • Exactly one accepted label per task (no lost or duplicated work), no task stuck forever
  • Security: customer data visible only to approved labelers; some data must stay in specific regions

[!TIP] It's a work queue where the workers are people Humans are slow (seconds to minutes per task), inconsistent, and sometimes adversarial. The design is a job scheduler with leases — tasks claimed, timed out and reassigned — plus a quality system that measures each labeler and routes work accordingly.

Capacity estimates

Core Entities

  • Project — customer, instructions, label schema, quality targets, required skills, region
  • Item — project, media URI, metadata; Task — item, type (label / review / consensus), status, priority, lease
  • Labeler — skills, qualifications per project, accuracy score, region, current task
  • Annotation — task, labeler, label data (JSON), time spent, status (submitted / accepted / rejected)
  • Gold task — item with a known correct answer, for scoring labelers

API

POST /v1/projects/{id}/batches   { "items": [{ "uri": "s3://cust/img_1.jpg" }, ...], "priority": "standard", "deadline": "..." }

# labeler tool
POST /v1/tasks/next              → { "taskId": "t_3", "item": { "url": "https://...signed" }, "instructions": "...", "leaseUntil": "..." }
POST /v1/tasks/{id}/heartbeat
POST /v1/tasks/{id}/submit       { "annotations": [{ "type": "box", "label": "car", "xywh": [10, 20, 80, 40] }] }

GET  /v1/batches/{id}            → { "done": 81233, "total": 100000, "accuracy": 0.982 }
GET  /v1/batches/{id}/export

High-Level Design

cust = client "Customer" @0,0
api = service "Project / Batch API" @1,0
media = blob "Media Storage" @2,0
pre = service "Pre-labeling Models" @3,0
tasks = store "Task Store" @1,1
router = service "Task Router\n(skills, priority)" @2,1
lab = client "Labeler Tool" @3,1
qc = service "Quality Control\n(review, consensus, gold)" @2,2
scores = store "Labeler Scores" @1,2
cust -> api : upload batch
api -> media
api -> tasks
pre -> tasks : suggested labels
router -> tasks : claim with lease
lab -> router : next task
lab -> media : signed URL
lab -> qc : submit
qc -> tasks : accept / rework
qc -> scores
router -> scores
  • A batch upload stores media (or references customer buckets) and creates tasks; pre-labeling models attach suggested labels.
  • A labeler asks for work; the task router picks a task the labeler is qualified for (skills, accuracy, region), by priority and deadline, and leases it.
  • The labeler annotates, using signed media URLs, and submits.
  • Quality control decides: accept, send to review, or create consensus tasks; outcomes update labeler scores, which feed routing.

Deep Dives

1. Leasing tasks to people

A task is claimed with a conditional update (status = 'available' → 'in_progress', labeler, lease_until = now() + 10 min), extended by heartbeats from the tool. Abandoned tasks (closed tab, logged off) return to the pool when the lease expires. Submissions are accepted only from the current lease holder (fencing), so a late submission after reassignment doesn't create a duplicate. See Design a Job Scheduler for the same pattern with machines.

2. Quality control

Labeler accuracy (from gold and review) drives routing: high-accuracy labelers get harder projects and fewer reviews; low accuracy triggers retraining or removal. Gold tasks must be indistinguishable from real ones, or labelers learn to spot them.

[!WARNING] Speed incentives hurt quality If labelers are paid per task, some will rush or game the tool. Track time per task and accuracy together, flag outliers (implausibly fast, copy-paste answers in RLHF), and never let a labeler review their own work.

3. Model-assisted labeling

Run a model to pre-label (boxes, transcripts, draft answers); labelers correct rather than start from scratch — often several times faster. Track how often humans change the pre-labels: low edit rates on easy items mean those can go to cheaper review-only flows; active learning sends the model's least-confident items to humans first.

4. Large media

Video and lidar items are huge. Labelers stream media through signed URLs from object storage or a CDN near them, not through the API. Split long videos into clip tasks with overlap, and stitch track IDs across clips in a post-processing step.

5. Priorities, deadlines and fairness

Queue tasks by deadline slack and customer priority; reserve qualified labeler capacity for enterprise SLAs. Monitor projected completion per batch (remaining tasks ÷ current throughput of qualified labelers) and alert or rebalance before an SLA is missed.

6. Data security

Per-project access lists and region constraints are enforced by the router (only qualified, approved labelers in allowed regions see a project's tasks). Media URLs are short-lived; the tool blocks downloads and screenshots where possible; all access is audit-logged. Sensitive customers may require labelers in secured facilities.

What's expected at each level

  • Mid-level: batches split into tasks, a queue labelers pull from, submissions stored and exported.
  • Senior: leases with heartbeats and fencing, a QC pipeline (gold, review, consensus), labeler scores driving routing, signed URLs for media.
  • Staff: cost/quality trade-offs per project, model-assisted labeling with active learning, SLA forecasting and priority scheduling, adversarial labelers, and data security across a global workforce.