Skip to content

Evaluation and Optimization

This example exposes one extraction prompt to supervised optimization with a metric that returns both a score and actionable feedback. Whether it improves requires measurement; the listing is not evidence of a quality gain.

Complete Program

> adapter: pydantic
> model: openai:gpt-5.6-luna

@extract_owner(ticket: str) -> str:
  > optimize: owner_prompt:
    >> Read the support ticket below.
    Ticket: <ticket>
    The team that owns the next action is [owner]
  = `owner`

@eval: extract_owner:
  > data: tickets:
    = ```
    return [
      ("Payments: retries fail after authorization", {"owner": "Payments"}),
      ("Search indexing is twelve hours behind", {"owner": "Search"}),
      ("Identity reports invalid refresh tokens", {"owner": "Identity"}),
    ]
    ```

  > test_data: tickets:
    = ```
    return [
      ("Checkout cannot create payment intents", {"owner": "Payments"}),
      ("New documents do not appear in results", {"owner": "Search"}),
    ]
    ```

  > metric: exact_owner(tickets):
    = ```
    actual = extract_owner(tickets).strip().casefold()
    wanted = expected["owner"].casefold()
    if actual == wanted:
        return 1.0, None
    return 0.0, f"returned {actual!r}; expected {wanted!r}"
    ```

= `extract_owner("Identity: sessions expire immediately after login")`

The leading >> opens the template. Its continuation lines belong to the same template, so this span makes one logical model call. Do not add >> to each continuation line.

Run Tests and Evals

kedi program.kedi --eval

Normal evaluation uses matching test_data when present; otherwise it falls back to data. It reports one selected score, not separate training and test scores.

Run GEPA

Install the optional optimization dependencies and configure the selected provider. See GEPA prerequisites. The default optimizer is mock, so select gepa explicitly for real search.

kedi program.kedi \
  --optimize \
  --optimizer gepa \
  --optimizer-max-metric-calls 30 \
  --eval

GEPA trains on data, validates against matching test_data, and stores only an optimized prefix. The source prompt and its output schema remain authoritative.

These two validation cases influence candidate selection. They are not an untouched final test set. Freeze separate held-out cases before comparing the baseline and optimized artifact, with the same model and repeated sampling. An eval score below one does not by itself produce a nonzero CLI exit status.

Generated files are:

  • program.kedi.optimized.json for selected prompt prefixes;
  • program.kedi.optimized_scores.json for score metadata;
  • program.kedi.gepa/ for per-span checkpoints.

Use --optimizer-fresh to remove these optimization artifacts before a new GEPA run. --no-cache instead controls generated > auto: code and does not reset optimizer state.

Dataset Shape

Explicit (input, expected_dict) rows are the least ambiguous format across evaluation and optimization. For a procedure with multiple parameters, use ((arg1, arg2), expected_dict). Metrics should be deterministic, stable, and cheap compared with the model call; vague feedback gives the optimizer little usable evidence.

For a no-network starting point, run the Validation Workflow, including its intentionally failing variant. Use the Reproducibility Protocol before interpreting an optimization result as an improvement.