API-DD

Practical guide

The contrast test for verifying AI-generated code

A Go technique for proving that a test rejects a plausible wrong alternative instead of merely repeating the generated implementation.

By API-DD

An agent receives this task:

Add a capacity-limited buffer.

It implements Push, adds tests, and leaves everything green. In review, the change looks complete.

But the task allowed at least two incompatible behaviors when the buffer becomes full:

Both are reasonable. Only one can be the module's contract.

If the tests pass with both, they do not verify that decision. They verify something less important, such as whether the first item can be inserted.

This article proposes one concrete check:

A test specifies a decision only when it rejects a wrong but plausible alternative.

We will call a case designed to separate those two alternatives a contrast test.

The green test that decides nothing#

Suppose we have this Go API:

subject, _ := buffer.New(1)
err := subject.Push("first")

The agent generates this test:

func TestPushStoresAnItem(t *testing.T) {
    subject, err := buffer.New(1)
    if err != nil {
        t.Fatalf("new buffer: %v", err)
    }

    if err := subject.Push("first"); err != nil {
        t.Fatalf("push: %v", err)
    }

    got := subject.Items()
    if len(got) != 1 || got[0] != "first" {
        t.Fatalf("expected first item, got %v", got)
    }
}

The test is correct, but it does not cover the risky decision. It passes with an implementation that rejects items when full and with one that overwrites the previous item.

The problem is not abstractly “missing coverage.” It is the absence of a case that distinguishes two semantics.

Write down both alternatives first#

Before adding another test, describe the contrast without code:

Capacity: 1
Initial state: [first]
Action: Push(second)

Chosen contract:
  returns Full
  final state: [first]

Alternative to reject:
  returns nil
  final state: [second]

This small comparison does two things.

First, it forces us to confirm a decision that the original task did not contain. If it does not exist in the ticket, documentation, or a current consumer, we should not let the agent choose it silently.

Second, it shows exactly what the test must observe: the result of Push and the later contents. We do not need to inspect indexes, internal calls, or the buffer's data structure.

The contrast test#

The case can be written like this:

func TestFullBufferRejectsNewItemAndKeepsExistingOne(t *testing.T) {
    subject, err := buffer.New(1)
    if err != nil {
        t.Fatalf("new buffer: %v", err)
    }

    if err := subject.Push("first"); err != nil {
        t.Fatalf("push first item: %v", err)
    }

    err = subject.Push("second")
    if !errors.Is(err, buffer.ErrFull) {
        t.Fatalf("expected full error, got %v", err)
    }

    got := subject.Items()
    if len(got) != 1 || got[0] != "first" {
        t.Fatalf("expected first item to remain, got %v", got)
    }
}

The name states the difference. Both assertions are necessary:

Together they protect the guarantee: rejection is visible and does not alter the existing contents.

Check the test with a mutation#

After obtaining green, introduce the alternative you want to reject for a single test run.

The correct implementation might contain:

func (b *Buffer) Push(item string) error {
    if len(b.items) == b.capacity {
        return ErrFull
    }
    b.items = append(b.items, item)
    return nil
}

Temporarily replace it with the opposite behavior:

func (b *Buffer) Push(item string) error {
    if len(b.items) == b.capacity {
        b.items = append(b.items[1:], item)
        return nil
    }
    b.items = append(b.items, item)
    return nil
}

Run only the case:

go test ./buffer -run TestFullBufferRejectsNewItemAndKeepsExistingOne

The expected result is a semantic failure:

expected full error, got <nil>

Restore the implementation and run the test again.

This manual mutation answers a question that a coverage percentage cannot: does the test detect the wrong decision we care about?

There is no need to mutate every line. Test the alternatives that were plausible before the implementation was known.

Do not derive the alternative from the diff#

The contrast must come from the task, its consumers, or an explicit decision. If we invent it after reading the generated code, we risk validating only the form chosen by the agent.

A useful sequence is:

Other possible contrasts include:

Each difference deserves a case when it matters to a consumer. It does not need three cases with different data when all three reject exactly the same alternative.

Three questions for reviewing a generated test#

Ask these questions in the pull request:

If we cannot answer the first, the test probably demonstrates a triviality. If the second fails, code and test may repeat the same invention. If the third fails, the test fixes the interior instead of the contract.

A short prompt for obtaining the contrast#

Before requesting tests or implementation, add:

For every observable decision in this task:

1. Propose two incompatible but plausible behaviors.
2. State which one is confirmed by the repository or documentation.
3. If neither is confirmed, ask a question and do not decide by probability.
4. For the confirmed behavior, write the smallest case that rejects the alternative.
5. Explain which concrete mutation must make that test fail.

Do not use details from the proposed implementation to define the contrast.

The goal is not to produce more tests. It is to make every new test name the decision it protects and the alternative it rejects.

Green does not mean verified. First prove that the case can recognize the wrong decision.