Agent Lifecycle Kit

Eval suites

Offline and live Eval-Driven Development suites for Agent Lifecycle Kit: routing, schemas, Cloudflare ops, and the teaching demo you can paste into a PR.

Suite reference for Kit’s agent eval harness. Day-to-day steps in SOPs/eval-driven-development.md. Companion: SOPs/edd-production-telemetry.md.

Layout

evals/edd/
├── README.md
├── system_prompt.md
├── kit_knowledge_prompt.md
├── demo.yaml|.jsonl          ← first-hour teaching suite
├── examples/before-after.md
├── examples/eval-report.md
├── examples/prod-trace.json
├── examples/otel-agent-loop.json   ← kit.* OTel span fixture
├── examples/prod-turns.jsonl      ← shadow-eval NDJSON corpus
├── architecture_routing.yaml|.jsonl
├── architecture_self_correction.yaml|.jsonl
├── architecture_terminal.yaml|.jsonl
├── kit_knowledge.yaml|.jsonl
├── cloudflare_ops.yaml|.jsonl
├── safety.yaml|.jsonl
└── tools/*.json

Drivers

| Driver | When | What it proves | |--------|------|----------------| | scripted (CI default) | kit eval ci, PR CI (kit check), Cursor / Copilot daily work | Harness, schema, keyword routing. **Not** a product LLM test. No API key. | | Live model | Nightly `.github/workflows/edd-live.yml` when KIT_EVAL_API_KEY is set | Paraphrases, prompt-injection, multi-tool, and safety cases tagged requires-live |

Cursor and GitHub Copilot are IDE hosts (AGENTS.md → .cursorrules / .github/copilot-instructions.md). They are **not** the live eval driver: kit eval never calls Cursor Chat or Copilot Chat. Env resolution, CI jobs, and a local live example: docs/edd.md (section *Cursor, Copilot, and API keys*).

Do not extend the scripted driver to pass requires-live cases. Add JSONL rows instead.

Quick start

First-hour teaching suite (six cases, before/after story in examples/before-after.md):

kit eval run --suite evals/edd/demo.yaml --model scripted
kit eval ci --suite evals/edd/demo.yaml --threshold-routing 95 --model scripted --out out/reports
kit eval report --format md --out out/reports

Full regression / CI suites:

kit eval run --suite evals/edd/architecture_routing.yaml --model scripted
kit eval ci --suite evals/edd/kit_knowledge.yaml --threshold-routing 95 --model scripted --out out/reports
kit eval ci --suite evals/edd/cloudflare_ops.yaml --threshold-routing 95 --model scripted --out out/reports
kit eval ci --suite evals/edd/architecture_routing.yaml --threshold-routing 95 --out out/reports
kit eval ci --suite evals/edd/safety.yaml --threshold-routing 95 --model scripted --out out/reports
kit eval watch --suite evals/edd/architecture_routing.yaml --target evals/edd

agent-kit aliases kit.

Metrics

| Type | Asserts | |------|---------| | tool_selection | Correct tool, expect.no_tool, or ordered expect.tools[] | | schema_match | Valid JSON object args (type/shape; not value meaning) | | argument_correctness | expect.arguments_contains / per-call args match intent meaning | | task_completion | User goal achieved (expect.goal or expected tool plan); scripted heuristic or live judge | | criteria_judge | Written suite criteria + threshold (0-1); per-criterion reasons | | mcp_use | Only catalog MCP tools; expected MCP capability when intent requires it | | plan_adherence | Ordered expect.tools[] / expect.tool matches trajectory steps | | step_efficiency | Tool step count <= max_steps (defaults to plan length or 1) | | plugin | Consumer module (module: ./plugin.mjs) receives case + trajectory | | llm_as_judge | Semantic accuracy / hallucination / tone (skipped when expect.no_tool) | | self_correction | Param updates after injected errors | | terminal_fallback | Circuit breaker stops endless retries |

Harness layout (hexagonal)

flowchart TB
  runner[EvalRunner]
  assert[run-assertions]
  arg[argument-correctness]
  mcp[mcp-use / plan-metrics]
  synth[synthesize / dataset-hygiene]
  orch[run-judges]
  pure[judge local heuristics]
  http[judge-provider HTTP adapter]
  plug[metric-plugin loader]
  trace[failure-trace + redact]
  runner --> assert
  assert --> arg
  assert --> mcp
  assert --> orch
  assert --> plug
  orch --> pure
  orch --> http
  runner --> trace
  cli[kit eval dataset] --> synth

Pure metric and judge logic stays inward. OpenAI-compatible HTTP and dynamic plugin imports live only in adapters.

Safety suite

Gateable injection / no-tool suite: evals/edd/safety.yaml. kit check runs it plus architecture routing, kit-knowledge, Cloudflare ops, self-correction, and terminal-fallback via EDD_CI_SUITES.

Dataset hygiene

kit eval dataset lint --dataset evals/edd/architecture_routing.jsonl
kit eval dataset dedupe --dataset path.jsonl --out path.deduped.jsonl
kit eval dataset synthesize --dataset path.jsonl --count 2 --out path.syn.jsonl
kit eval dataset from-trace --trace evals/edd/examples/prod-trace.json --out out/prod.jsonl
kit eval shadow --infile evals/edd/examples/prod-turns.jsonl --sample 1 --seed 1 --out out/shadow-fails.jsonl

Synthetic paraphrases keep expectations, add tags synthetic + requires-live.

Production telemetry (closed loop)

Promote production misses into the suite with the same kit.* fields as eval cases:

kit eval shadow --infile evals/edd/examples/prod-turns.jsonl --sample 1 --seed 1 --out out/shadow-fails.jsonl
kit eval dataset from-trace --trace evals/edd/examples/prod-trace.json --out out/prod.jsonl

Fixtures: examples/otel-agent-loop.json, examples/prod-turns.jsonl, examples/prod-trace.json. Procedure: SOPs/edd-production-telemetry.md.

Tags

| Tag | Meaning | |-----|---------| | routing | Counts toward routing accuracy | | requires-live | Skipped when the scripted driver is in use | | prod-derived | Converted from a production miss via productionTraceToJsonl | | prompt-injection | Instruction-override attempts |

Reports

| Artifact | Role | |----------|------| | out/reports/eval-report.md | Stable alias for PR review | | out/reports/edd-report.md | Same Markdown body | | out/reports/edd-report.json | Machine-readable results | | GitHub Actions job summary | Overview table + collapsible full report (kit eval report --github-summary) |

Includes pass rate, tokens/latency, routing + schema adherence, and failure traces. Example: examples/eval-report.md.

CI workflows (.github/workflows/ci.yml Verify, edd-live.yml) write a short “what this gate means” preamble plus the EDD overview into the run **Summary** tab.

Live models (optional): KIT_EVAL_API_KEY first, then OPENAI_API_KEY, then ANTHROPIC_API_KEY. Optional KIT_EVAL_BASE_URL / OPENAI_BASE_URL (OpenAI-compatible /chat/completions; default https://api.openai.com/v1), KIT_EVAL_TOKEN_USD_PER_1K, KIT_EVAL_MODEL. Nightly CI only reads KIT_EVAL_API_KEY.