Skip to main content

​
What is a Run?

A Run is the record of one launch. Every launch creates a Run. Every evaluation of that launch points back at it, and so does every session the launch created. The Run answers three questions about a launch: who started it, what it produced, and how it ended. A Run belongs to one product. It usually also names the version you launched against.
A Run is an entity you can list, open and delete. It is not the SDK method evaluations.run(). That method starts a launch. The Run is the record that the launch leaves behind.
A Run is the development counterpart of a Monitor. A Monitor scores production sessions on its own, with no launch. A Run records a launch you triggered. The record survives the queue. Before Runs, a launch was visible only as a queue job, and the queue drops old jobs. A Run stays, so you can still read last week’s launch.

​
Every launch creates a Run

You do not have to create a Run. Galtea creates one for you whenever you launch evaluations, from the dashboard, the CLI, the SDK or the REST API. Every launch response carries the id. A launch that returns evaluations gives it on each one, as runId. A launch that queues work or covers a batch returns it beside the job: runId for a single run, runIds for a call that opens one per product or per version. These launches create a Run:
  • An evaluation of a version against its test cases.
  • An evaluation of one session, or of several sessions, production sessions included. One call creates one Run per product, because a Run belongs to a single product.
  • An evaluation of a single inference result.
  • A single-turn evaluation.
  • An inference generation, with or without metrics.
  • A metric replay. One call creates one Run per product, for the same reason.
  • A test case replay. One call creates one Run per version, because each version runs as its own job.
A launch rejected on its own contents, such as an unknown metric, creates no Run. You get the error in the response, so there is nothing to record. Two cases create no Run:
  • Monitor evaluations. A Monitor evaluation belongs to its Monitor, not to a launch. An evaluation carries a Run or a Monitor, never both.
  • Retries. A retried evaluation keeps the Run it was created with. It does not start a new one.

​
Opening a Run yourself

You can also open a Run yourself, and attribute your own sessions and evaluations to it. Such a Run records work you performed, so it names no launch of the platform. The steps are:
  1. Create the Run with POST /runs, with galtea runs create, or with galtea.runs.start().
  2. Pass the returned id as runId, or as run_id in the SDK, on each call that creates a session or an evaluation. Four calls accept it: POST /sessions, POST /evaluations/fromSession, POST /evaluations/fromInferenceResult and POST /evaluations/singleTurn.
  3. Close the Run with POST /runs/{id}/close, with galtea runs close, or by leaving the SDK with block.
Pass a runId only for a Run you opened with POST /runs. A Run the platform opened already records its own launch, so it refuses work you attribute, and the answer says so. POST /evaluations/fromVersion, POST /evaluations/fromSessions and POST /inferenceResults/generate refuse a runId outright, because each one opens its own Run.
You can also set your own customId on a Run, for example a build number from your pipeline. A customId must be unique within its product among runs that are not deleted. Deleting a run frees its customId.

​
The Run number

Each Run has a number, shown as #42. Numbers start at 1 in each product and count up. A number is never reused, so a deleted Run never gives its number to a later one. The number is a label. Address a Run by its id, not by its number, and do not compare numbers across products.

​
Status

A Run has one of four statuses:
  • RUNNING — the work is in progress. Every new Run starts here.
  • COMPLETED — the work finished and created its evaluations.
  • FAILED — no session of the launch completed. A launch that fails its endpoint check ends here with zero evaluations. A launch whose every session failed also ends here. Its sessions still hold evaluations: a session with a failed turn has each metric SKIPPED, and a session whose turns all completed is scored. The error field says why.
  • CANCELLED — you cancelled the Run, or the queue job that carried it out.
A Run the platform opened settles with its own launch. The launch has one outcome, and that outcome becomes the status of the Run. A Run you opened settles only when you close it, and it always ends as COMPLETED. You performed the work yourself, so the platform has no outcome of its own to report. COMPLETED, FAILED and CANCELLED are final. Each one also sets finishedAt. A Run that already holds a final status cannot change again, so closing it a second time returns an error.

​
The sessions of a Run

The sessions of a Run are a union: the sessions the Run created, plus the distinct sessions of its evaluations. Galtea counts that union when you read the Run. A launch that generates inferences creates sessions, so the Run holds them even when it evaluates no metric. A launch over sessions that already exist creates none, and reaches its sessions through its evaluations. A session names at most one Run: the Run that created it. The session carries it as runId. A session you log yourself carries no runId. That covers a session you open with POST /sessions and no run id, a trace file import, and OpenTelemetry traces. The session count of a Run is wider than that field. It also counts a session the Run only scored. The two GET /sessions filters keep the difference. runIds returns the sessions a Run created. involvedInRunIds returns the sessions it created or scored.

​
Running a Run again

POST /runs/{id}/relaunch repeats a Run. It answers with a new Run, which names the one it repeats as relaunchedFromRunId. The source Run is never touched: its evaluations stay exactly as they were, so you can compare the two. The kind of the source decides what “again” means. You do not choose:
  • A Run of kind GENERATE_INFERENCES is re-launched. Galtea drives the version again, creates new sessions, and scores them.
  • A Run of any other kind is re-evaluated. Galtea scores sessions that already exist again, with the same metrics, and creates no session.
  • A Run you performed yourself carries no kind. It is re-evaluated too. Galtea reads the Run’s own evaluations for the sessions and the metrics to score again. No session is created, because only you can produce one.
Two bodies are accepted, both optional:
  • versionId runs the same tests against a different version. It is accepted only on a re-launch, because a re-evaluation scores sessions that already belong to a version.
  • useCurrentTestCases resolves the inputs again instead of repeating the recorded ones. On a re-launch it uses every test case the same datasets hold today. On a re-evaluation of a version it uses every session that version holds today. It changes nothing where the original request listed its test cases or sessions one by one, because there is no container to resolve again.
A Run whose evaluations name no session is refused, because there is nothing to score again. So is one that scored more sessions than a single repetition can cover; score those from the sessions list instead, in batches. The answer names the reason.

​
Cancelling a run

POST /runs/{id}/cancel stops a Run that is still running, and the work behind it. The sessions and evaluations it already created are kept. Only a Run the platform queued a job for can be cancelled. It is the one carrying a jobId. Every other Run finishes inside the request that started it, so by the time you read it there is nothing left to stop, and a Run you opened yourself carries no job at all.

​
Reading the scores of a Run

GET /runs/{id}/metric-summary reports how each metric scored across the Run: the mean and the observed range, over the evaluations that carry a score. A reviewer’s score wins over the judge’s, the same rule every reported number uses. A metric whose evaluations are all still pending has no row.

​
Deleting a Run

DELETE /runs/{id} hides the Run and every evaluation it holds. The sessions stay, and so does everything else that scored them. A Monitor’s scores of the same sessions survive, and a session scored by two Runs keeps the scores of the other Run.

​
Finding runs in the dashboard

Open a product and pick Runs in the sidebar. It lists every run of that product. The sidebar keeps Runs behind More unless you turn on Show all tabs in your settings. To see the runs of one version only, open that version and pick its Runs tab. Open a row to reach the run’s own page. It has three tabs:
  • Overview — a Run card with the status, the version, the finish time and the counts, a What it ran card with the kind, the inputs, the error and the request, and how each metric scored. For a run you opened yourself, the second card says instead that you performed the work.
  • Sessions — every session the run created or scored.
  • Evaluations — the evaluations the run launched.
The What it ran card carries the actions. Cancel is shown while the run is still running and the platform queued a job for it. Otherwise Run again is shown, for a run Galtea can repeat. The two are never shown together. Evaluation tables and session tables both show a Run column and a Runs filter in the Filters menu, so you can narrow either list down to one run. In both, the column links to the run’s page. Session tables hide both in the Production view, because production traffic never belongs to a Run.

​
API and CLI

The Galtea Python SDK exposes runs as galtea.runs: start(), list(), get(), get_by_custom_id(), close() and delete(). start() returns a context manager, so the sessions and evaluations you create inside a with block join the run, and leaving the block closes it.

Run Service

Open, group and close runs programmatically
The REST API offers POST /runs, POST /runs/{id}/close, POST /runs/{id}/relaunch, POST /runs/{id}/cancel, GET /runs, GET /runs/{id}, GET /runs/{id}/metric-summary and DELETE /runs/{id}. GET /evaluations accepts a runIds filter, so you can read every evaluation of a run. Three session operations name a Run. POST /sessions accepts an optional runId, so you can add a session to a Run you opened. It is refused on a production session. GET /sessions?runIds= returns the sessions a Run created. GET /sessions?involvedInRunIds= returns the sessions a Run created or scored. The SDK write methods take the same id as run_id: sessions.create, sessions.get_or_create, evaluations.create, evaluations.create_single_turn, evaluations.run (only when you pass an agent, because the queued mode performs its own launch) and traces.create_and_evaluate. evaluations.list reads them back with the same run_id filter, and each Session and Evaluation carries its own run_id. The CLI mirrors those operations: galtea runs list, galtea runs get, galtea runs create, galtea runs close, galtea runs relaunch, galtea runs cancel, galtea runs get-metric-summary and galtea runs delete.

​
Run Properties

​
string
Unique identifier of the run. Use it to address the run, and to link evaluations to it.
​
string
required
The ID of the product the run belongs to. A run always names exactly one product.
​
string
The ID of the version the run evaluated. It is empty for a metric replay, because a replay can span several versions of one product, so no single version would be correct.
​
string
The ID of the user who started the run. It is empty when the run came from an automation with no signed-in user.
​
int
The run number within its product, shown as #42. It starts at 1, counts up, and is never reused.
​
string
Your own identifier for the run, for example a build number. It must be unique within the product among runs that are not deleted.
​
string
The ID of the run this one repeats. POST /runs/{id}/relaunch sets it. It is empty on every other run.
​
Enum
Who opened the run. LAUNCH when a launch endpoint opened it; the run then ends with the launch it holds. CALLER when POST /runs opened it; only POST /runs/{id}/close ends it.
​
Enum
The lifecycle status of the run. Possible values: RUNNING, COMPLETED, FAILED, CANCELLED. The last three are final and also set finishedAt.
​
Enum
What the platform did in this run. It is empty on a run you opened with POST /runs. Possible values:
  • GENERATE_INFERENCES: the platform generated sessions from test cases, and scored them when the request named a metric. POST /evaluations/fromVersion, POST /evaluations/fromTestCases and POST /inferenceResults/generate reach it.
  • EVALUATE_SESSION: the platform scored one existing session, inference result or single turn. POST /evaluations/fromSession, POST /evaluations/fromInferenceResult and POST /evaluations/singleTurn reach it.
  • EVALUATE_SESSION_BATCH: the platform scored several existing sessions in one call. POST /evaluations/fromSessions reaches it.
  • REPLAY_METRIC: the platform scored existing sessions again with a metric. POST /evaluations/fromMetrics reaches it.
​
string
The ID of the queue job that carried out the run. It is empty when the work finished inside the request, with no job. A run carrying one can be cancelled.
​
string
Why the run failed. It is empty unless the status is FAILED.
​
object
The launch request as the platform received it. It is empty on a run you opened with POST /runs.
​
object
What the request resolved to at dispatch: the test cases, sessions or metrics the platform worked on. Its keys depend on kind. It is empty on a run you opened with POST /runs.
​
string
Timestamp of when the run was created (ISO 8601 format).
​
string
Timestamp of when the run reached a final status (ISO 8601 format). It is empty while the run is RUNNING.
​
int
How many evaluations the run holds. Galtea counts it when you read the run; it is not stored.
​
int
How many distinct sessions the run created or scored. Galtea counts it when you read the run; it is not stored. A deleted session is not counted, so this total always matches what GET /sessions?involvedInRunIds= returns.

Concepts overview

How Galtea’s concepts connect — diagram + per-entity quick reference.

Product

A functionality or service being evaluated

Version

A specific iteration of a product

Evaluation

The assessment of an evaluation using a specific metric’s criteria

Monitor

An always-on evaluation rule for production sessions