Skip to main content

​
Overview

AI Metric Generation lets you automatically create evaluation metrics from your product’s specifications. Instead of manually crafting judge prompts and configuring evaluation parameters, the AI analyzes your specifications and generates ready-to-use metrics.
Evaluation parameters are automatically selected based on each specification’s description and test type. The generated judge prompt follows a format optimized for reliable LLM-based evaluation across different evaluator models.

​
Requirements

  • A product with a description
  • At least one specification of type POLICY with a test type assigned (Accuracy, Security, or Behavior)
CAPABILITY and INABILITY specifications cannot be used for AI metric generation because they do not have a test type.

​
How to Generate Metrics

There are two ways to trigger AI metric generation from the dashboard:

​
From the Specifications Page

  1. Navigate to your product’s Specifications tab
  2. Open the dropdown menu on the specification you want to generate metrics for
  3. Click Generate Metrics — this takes you to the generation page with that specification pre-selected
  4. Click Generate and wait for the AI to process
  5. Review the generated candidates — edit, save, or discard each one

​
From the Metrics Page

  1. Navigate to your product’s Metrics tab
  2. Click Generate Metrics with AI
  3. Select the specifications you want to generate metrics for
  4. Click Generate and wait for the AI to process
  5. Review the generated candidates — edit, save, or discard each one
In both cases, the AI analyzes your product name, description, and the selected specifications to generate tailored metrics.

​
Complete a Metric with AI

When you already know what you want to measure but don’t want to write the judge prompt by hand, use Complete with AI. You provide the metric’s name, description, and evaluation type as the source of truth, and the AI fills in the rest: the judge prompt and the evaluation parameters (plus a recommended evaluator model for AI Evaluation). This is available in two places:
  • The manual metric form — when creating a metric from scratch (Metrics tab → New Metric → blank), fill in the name, description, and evaluation type, then click Complete with AI.
  • The AI generation review step — when editing a generated candidate, click Complete with AI to regenerate its judge prompt and evaluation parameters from the candidate’s current name, description, and tags.
Your name, description, and tags are never modified — they are the inputs the AI completes from. If you have already written a judge prompt, the dashboard asks for confirmation before replacing it.
Complete with AI is available for AI Evaluation and Human Evaluation metrics:
  • For AI Evaluation, it produces a binary LLM-judge prompt and recommends an evaluator model.
  • For Human Evaluation, it produces annotator guidelines on a 0–100 scoring scale (no evaluator model, since a human assigns the score).
Self-Hosted metrics are not supported because they have no judge prompt. See Evaluation Types for the differences between these scoring methods.

​
Evaluation Parameter Selection

In both spec-driven generation and completion mode, the AI automatically selects the evaluation parameters each metric needs. In spec mode these are inferred from the specification description; in completion mode they are inferred from the metric description you provided. For example:
  • A specification about citation accuracy or knowledge-grounded answers will include retrieval_context so the judge can verify answers against retrieved source material.
  • A specification about internal processes, workflows, or tool orchestration will include traces (and often tools_used) so the judge can inspect the execution path — not just the final output.
  • A specification about refusal or safety boundaries typically only needs input and actual_output, since compliance is fully observable from what was asked and answered.
If a metric includes traces or retrieval_context as evaluation parameters, the product under test must capture that data during test execution (e.g., via tracing integrations). If the data is not available at evaluation time, the evaluation will fail.

​
Generated Metric Properties

Each AI-generated metric candidate includes:

​
Specification Linking

When you save a metric generated from a specification, it is automatically linked to that specification. This creates a traceable connection between your requirements and your evaluation criteria. Metrics created via Complete with AI are not automatically linked to a specification — you can link them manually from the Specification Hub if needed. You can view linked specifications directly from a metric’s detail page, and manage metric-specification links from the Specification Hub.