Courses › AI-Assisted Coding › Round Types

Testing with AI

Choosing what to test, spec-first tests, reviewing generated tests and property checks.

Tests are where AI assistance helps most and misleads most. An assistant can write a dozen tests in seconds. But it tends to write tests that confirm what the code already does rather than what it should do, and it will happily weaken a test to make it pass. In an interview, how you test with an assistant is one of the clearest signals of engineering judgment.

Decide what to test yourself

Before asking for any tests, list the behaviors that matter. The assistant can write the test code, but choosing what to test is your job:

  • The main success paths for each requirement
  • Boundaries: empty input, one element, the maximum size, exact limits (≤ vs. <)
  • Invalid input: wrong types, malformed formats, missing fields
  • Tricky rules from the spec: tie-breaking, rounding, time zones, ordering
  • Regressions: the bug you just fixed

Say the list out loud. It's often worth more to the interviewer than the tests themselves.

Prompting for useful tests

Write pytest tests for `parse_duration(s) -> int` (seconds) in `timeparse.py`.
Spec: accepts "90s", "5m", "2h", "1h30m"; whitespace allowed between parts;
raises ValueError for empty strings, unknown units, or negative numbers.
Cover: each unit, combined units, whitespace, each error case, and "0s".
Follow the style in tests/test_dates.py (plain functions, parametrize for tables).
Do NOT read the implementation — write the tests from the spec.

The last line matters. Tests written from the spec, without looking at the implementation, catch bugs. Tests written from the implementation just repeat its bugs.

Test first, when it fits

For a well-specified function, write or generate the tests before the implementation:

  • Generate the tests from the spec. Review them: are the expected values actually right?
  • Run them and watch them fail. That proves they test something.
  • Ask for an implementation that makes them pass, *without changing the tests*.
  • Read the implementation anyway.

This turns the assistant's speed into a safety net instead of a risk.

Reviewing generated tests

Check every generated test for:

[!WARNING] Don't let "make the tests pass" weaken the tests When a test fails, an assistant may change the assertion, add a special case for the test input, or mark the test skipped. Read every diff to test files separately, and say out loud when you reject one: "It changed the expected value. The test was right; the code is wrong."

Property-style checks

For logic with a clear invariant, ask for property tests (with Hypothesis in Python, or a simple randomized loop) alongside the examples:

  • Sorting: the output is ordered and is a permutation of the input.
  • Round-tripping: parse(format(x)) == x.
  • Money: the sum of all ledger entries is zero.

One property test often catches more than ten example tests, and suggesting one is a strong signal.

Adding tests to untested code

When you're asked to add tests to legacy code:

  • Ask the assistant to summarize the module's behavior, then check the summary against the code.
  • Write characterization tests that pin down the current behavior, including oddities, before changing anything.
  • Flag behavior that looks like a bug instead of quietly enshrining it: "This test documents that negative quantities are accepted. I think that's a bug, so I've marked it to discuss."

[!TIP] Practice Add Tests to an Untested Module with AI · Debug a Failing Test Suite with AI · Reviewing AI Code