Skip to main content

​
What is an Evaluation?

An evaluation in Galtea represents the assessment of traces from a session using the evaluation criteria of a metric. Evaluations are now directly linked to sessions, allowing for comprehensive evaluation of full sessions containing multiple traces for multi-turn dialogues.
Evaluations do not call your AI product — they score outputs that have already been generated. Run your product first to produce outputs, then trigger the evaluation to score them.
You can create evaluations from the Galtea dashboard (using Endpoint Connections) or programmatically using the Galtea SDK.

​
Evaluation Lifecycle

Evaluations follow a specific lifecycle:
1

Creation

Trigger an evaluation. It will appear in the session’s details page with the status “pending”
2

Processing

Galtea’s evaluation system processes the evaluation using the evaluation criteria of the selected metric
3

Completion

Once processed, the status changes to “completed” and the results are available

​
SDK Integration

The Galtea SDK allows you to create, view, and manage evaluations programmatically. This is particularly useful for organizations that want to automate their evaluation process or integrate it into their CI/CD pipeline.

Evaluation Service SDK

Manage evaluations using the Python SDK

GitHub Actions

Learn how to set up GitHub Actions to automatically evaluate new versions

​
Evaluation Properties

​
Result Properties

Once an evaluation completes, the following fields are available:
​
Enum
The current status of the evaluation. Possible values:
  • Pending: The evaluation has been created but not yet processed.
  • Pending Human: The evaluation is waiting for a human annotator to review and score it. This status is used for metrics with source: "human_evaluation".
  • Success: The evaluation was processed successfully.
  • Failed: The evaluation encountered an error during processing. Check the canRetry field to determine if this evaluation can be retried.
  • Skipped: The evaluation could not produce a score. The error message names the cause. Common causes are:
    • Metric validation failed, for example a required parameter is missing.
    • The organization ran out of credits.
    • The metric could not read the actual output, for example plain text where JSON Field Match expects a JSON object.
    • The judge model was unavailable.
    Check the canRetry field to see whether the evaluation is eligible for a retry. A retry re-scores the output already stored on the session. When the cause is that output, send a new inference result instead.
  • Outdated: The evaluation was produced by a Monitor, and the session then reopened and gained a new turn. The score describes an older version of the conversation, so it no longer counts as current: charts and averages leave it out, and the monitor scores the session again once it closes. The row keeps its score and its reason, so you can still read what it found.
​
Number
The score assigned to the output by the metric’s evaluation criteria. Example: 0.85
​
Text
The explanation of the score assigned to the output by the metric’s evaluation criteria.
​
Text
The error message if the evaluation failed during processing.
​
Number
The number of retry attempts made for this evaluation. Starts at 0 and increments each time the evaluation is retried. Used to track retry history and enforce retry limits.
​
Boolean
default:"false"
Indicates whether the evaluation is eligible for the retry operation. When false, it cannot be retried at all. When true, it can, but a retry does not always help: canRetry is a permission, not a promise. A retry re-scores the output already stored on the session, so it succeeds for a temporary cause (an evaluation processing error, insufficient credits, a judge model that was unavailable) and skips again when the cause is the stored output itself. Defaults to false.
​
Text
The ID of the human evaluator assigned to this evaluation, if applicable.
​
Number
The human-provided annotation score, separate from the AI-generated score. When a human annotates an AI-evaluated metric, this field stores their score without overwriting the original AI score.
​
Text
The human-provided annotation reason, separate from the AI-generated reason. When a human annotates an AI-evaluated metric, this field stores their reasoning without overwriting the original AI reason.
​
Text
The timestamp when the human evaluator started reviewing this evaluation, if applicable.
​
Text
The timestamp when the human evaluator submitted their evaluation or annotation.

Concepts overview

How Galtea’s concepts connect — diagram + per-entity quick reference.

Dataset

A dataset of test cases for evaluating product performance

Test Case

Each challenge in a dataset for evaluating product performance

Session

A full conversation between a user and an AI system.

Metric

Ways to evaluate and score product performance