# Eval-Driven Development (EDD)

Connecting an LLM to MCP tools, APIs, or terminals turns a chatbot into a decision-maker. Failures rarely look like stack traces. They look like a wrong tool, a hallucinated parameter, or an infinite retry loop.

EDD treats prompts and tool schemas as version-controlled, evaluated contracts. Agent Lifecycle Kit ships the harness, CI gates, and production closed loop.

## The loop (same shape as TDD)

1. **Red - Define intent.** JSONL cases and YAML metrics assert the tool (and arguments) you expect, and that chatty questions do not invent tool calls.
2. **Green - Implement the interface.** Register the MCP/tool contract and minimal system instructions. Run until asserts pass.
3. **Refactor - Refine context.** Iterate descriptions and constraints without breaking existing cases. Gate merges with `kit eval ci --threshold-routing 95`.

```mermaid
flowchart LR
  intent[Agent intent] --> edd[Red / green / refactor]
  edd --> ci[CI threshold gate]
  ci --> ship[Ship]
  ship --> prod[Prod traces + shadow judge]
  prod -->|miss| edd
```

## Why run it this way

| Capability | Outcome |
|------------|---------|
| **Context isolation** | Fresh context per case. No cross-test contamination. |
| **Deterministic mocks** | Measure routing and extraction, not third-party latency. |
| **Dual-layer asserts** | Schema/shape plus argument meaning, task completion, criteria judge, optional LLM-as-a-judge. |
| **CI quality gates** | `kit eval ci --threshold-routing 95` blocks routing drift; safety suite runs in `kit check`. |
| **Closed-loop telemetry** | Production misses become `.jsonl` cases (`kit eval dataset from-trace`); `kit eval shadow` samples live turns into the suite. |
| **Dataset hygiene** | `kit eval dataset lint\|dedupe\|synthesize\|from-trace` keeps suites valid and scalable. |

Bare `kit eval` still validates which Kit skill activates. `kit eval run|watch|report|ci` validates how an agent calls tools. Use EDD whenever you change prompts, tool schemas, or routing.

## Quick start

```bash
curl -fsSL https://raw.githubusercontent.com/mzworthington/agent-lifecycle-kit/main/install.sh | sh
kit init . --mcp default --hook

kit eval run --suite evals/edd/architecture_routing.yaml --model scripted
kit eval ci --threshold-routing 95 --out out/reports
kit eval report --format md --out out/reports
kit eval watch --suite evals/edd/architecture_routing.yaml --target evals/edd
kit eval dataset lint --dataset evals/edd/architecture_routing.jsonl
```

`agent-kit` is an alias for `kit`.

## Cursor, Copilot, and API keys

Cursor and GitHub Copilot are **IDE hosts**. They already get Kit skills and the same bootstrap (`AGENTS.md` → `.cursorrules` and `.github/copilot-instructions.md` via `kit export-rules` / `kit init`). Daily work and the merge gate use `--model scripted`. You do **not** need an OpenAI (or any provider) API key for that.

The live-eval key is a **different job**: call a real model over HTTP and ask whether *that* model picks the right tool. Cursor Chat and Copilot Chat are not HTTP eval drivers. The harness cannot send cases into the model sitting in your editor.

```mermaid
flowchart TD
  start["kit eval run / ci"] --> model{"--model?"}
  model -->|"scripted or mock<br/>CI default"| scripted["Local keyword driver<br/>no HTTP, no key"]
  model -->|"provider id<br/>e.g. gpt-4o-mini"| key{"KIT_EVAL_API_KEY or<br/>OPENAI_API_KEY set?"}
  key -->|no| scripted
  key -->|yes| live["POST /chat/completions<br/>Bearer token"]
  live --> agent["Agent under test"]
  live --> judge["LLM-as-judge when that metric is on"]
```

`--model scripted` never spends, even if a key is in the environment. Cases tagged `requires-live` are skipped on the scripted driver.

### When a key is used

The runner takes the first non-empty value:

1. `KIT_EVAL_API_KEY` (preferred; this is the secret nightly CI looks for)
2. `OPENAI_API_KEY`
3. `ANTHROPIC_API_KEY`

That value is sent as `Authorization: Bearer …` to an **OpenAI-compatible** `{baseUrl}/chat/completions`. Base URL resolution: `KIT_EVAL_BASE_URL`, then `OPENAI_BASE_URL`, then `https://api.openai.com/v1`.

`ANTHROPIC_API_KEY` is only useful if `KIT_EVAL_BASE_URL` points at a gateway that accepts Anthropic keys on the OpenAI request shape. Anthropic’s native Messages API is not this client.

The same key is reused for:

- **Agent:** given this prompt and these tools, which call do you make?
- **Judge:** optional second completion for `llm_as_judge` / `criteria_judge` / live `task_completion` (falls back to a local heuristic when the model is scripted or no key is set)

Optional: `KIT_EVAL_MODEL`, `KIT_EVAL_TOKEN_USD_PER_1K`. Local OpenAI-compatible servers (Ollama, OpenRouter, and similar) work by setting `KIT_EVAL_BASE_URL`.

## Metrics and suites

Beyond tool selection and schema shape, suites can assert:

| Area | Metrics / tooling |
|------|-------------------|
| Outcome quality | `argument_correctness`, `task_completion`, `criteria_judge` |
| MCP / multi-step | `mcp_use`, `plan_adherence`, `step_efficiency`, trajectory traces in reports |
| Safety | `evals/edd/safety.yaml` (injection + no-tool; in `kit check` and nightly live) |
| Extensibility | `type: plugin` modules; `kit eval dataset lint\|dedupe\|synthesize\|from-trace` |

Full metric table and harness layout: [evals/edd/README.md](../evals/edd/README.md).

### CI

| Job | Key | Model | Purpose |
|-----|-----|--------|---------|
| **Verify** in [`.github/workflows/ci.yml`](../.github/workflows/ci.yml) | unused | hardcoded `scripted` via `kit check` | Merge gate: harness + keyword routing + safety + recovery |
| Nightly [`.github/workflows/edd-live.yml`](../.github/workflows/edd-live.yml) | **requires** `KIT_EVAL_API_KEY` | repo variable `KIT_EVAL_MODEL` | Live paraphrases, prompt-injection, multi-tool, safety |
| `pnpm check` / `kit check` | unused | hardcoded `scripted` | Same as Verify, locally |

Nightly **skips the whole job** if `KIT_EVAL_API_KEY` is empty. It does not fall through to `OPENAI_API_KEY`.

Verify and the nightly live job (plus Pages deploy) publish a **job summary**: what the gate means, then an EDD overview table and collapsible full report via `kit eval report --github-summary`.

### Local live run

```bash
export KIT_EVAL_API_KEY='…'   # or OPENAI_API_KEY
# optional: export KIT_EVAL_BASE_URL='https://api.openai.com/v1'
kit eval run --suite evals/edd/architecture_routing.yaml --model gpt-4o-mini
```

You should then see `requires-live` cases execute instead of “Skipping N requires-live case(s)”.

## Next

| Resource | Purpose |
|----------|---------|
| [SOPs/eval-driven-development.md](../SOPs/eval-driven-development.md) | Day-to-day procedure |
| [evals/edd/README.md](../evals/edd/README.md) | Suites, metrics, layout |
| [SOPs/edd-production-telemetry.md](../SOPs/edd-production-telemetry.md) | `kit.*` spans, `kit eval shadow`, from-trace, drift |
| [skills/agent-orchestrator/SKILL.md](../skills/agent-orchestrator/SKILL.md) | Feature lifecycle around EDD |