Direct Inferences and Evaluations from the Platform

Learn how to configure Endpoint Connections to run inferences and evaluations directly from the Galtea Dashboard, without writing any SDK code.

Galtea allows you to run inferences against your AI system and evaluate its responses directly from the Dashboard, without writing any SDK code. This is made possible by Endpoint Connections, which tell Galtea how to call your API, extract the response, and manage session state across turns.

This guide covers the platform-based workflow. If you prefer to generate inferences programmatically (e.g., in a CI/CD pipeline or custom script), see the SDK tutorials instead.

Prerequisites

Before you begin, make sure you have the following set up in the Galtea Dashboard:

• A Product representing your AI system
• A Dataset with at least one Test Case to run against your endpoint

Workflow Overview

1. Create an Endpoint Connection

   Define how Galtea should call your AI endpoint — URL, authentication, request format, and response extraction.

2. Create a Version with the Endpoint Connection

   Create a new version of your product and attach the endpoint connection to it.

3. Run an Evaluation from the Dashboard

   Open the version and choose the datasets and metrics to run. Galtea calls your endpoint for each test case, records the traces, and evaluates them.

4. Evaluate Again with Other Metrics

   Optional. Score an existing session again with different metrics, without calling your endpoint a second time.

Step 1: Create an Endpoint Connection

Navigate to your product in the Dashboard, switch to Development view mode, and open the Connections section in the sidebar (under Product), then the Endpoints tab. Click New Endpoint Connection and configure the following:

1. Name — A descriptive name (e.g., “Production Chat API”).
2. Type — Select CONVERSATION for the primary request/response endpoint.
3. URL — The full URL of your AI endpoint (e.g., https://api.company.com/v1/chat).
4. HTTP Method — Typically POST.
5. Authentication — Choose the auth type (Bearer, API_KEY, Basic, or None) and provide the token.
6. Input Template — A Jinja2 template that defines the request body Galtea will send.
7. Output Mapping — JSONPath expressions that tell Galtea how to extract values from the response.

Input Template

The input template uses Jinja2 syntax with placeholders that Galtea fills automatically. At minimum, use {{ input.user_message }} to inject the test case input:

{
  "model": "gpt-4",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "{{ input.user_message }}"}
  ]
}

For multi-turn conversations, use past_turns to include conversation history:

{
  "model": "gpt-4",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {% for turn in past_turns %}
    {"role": "user", "content": "{{ turn.input }}"},
    {"role": "assistant", "content": "{{ turn.output }}"},
    {% endfor %}
    {"role": "user", "content": "{{ input.user_message }}"}
  ]
}

See Endpoint Connection — Input Template for the full list of available placeholders and advanced template examples.

Output Mapping

The output mapping tells Galtea how to extract values from the API response using JSONPath expressions. The output key is required:

{
  "output": "$.choices[0].message.content"
}

You can also extract additional values to store as session metadata:

{
  "output": "$.choices[0].message.content",
  "retrieval_context": "$.choices[0].retrieval_context",
  "session_id": "$.metadata.session_id"
}

Any extra key beyond the special keys (output, retrieval_context, session_id, traces) is saved to the session metadata and becomes available as a {{ key }} placeholder in subsequent turns.

Sending an extracted value back (the round-trip)

Extracting a value is only half of stateful multi-turn handling. If your agent returns an identifier on the first turn — a thread_id, conversation_id, or any token that groups subsequent messages — you must also reference it in the Input Template so Galtea sends it back on the next turn. Extraction alone stores the value but never resends it.

The round-trip has two coordinated sides:

Output Mapping (extract)

{
  "output": "$.content",
  "thread_id": "$.thread_id"
}

Turn by turn:

1. Turn 1 — thread_id has no value yet, so {{ thread_id }} renders as an empty string. Your agent creates a new thread and returns its id in the response.
2. Extraction — Output Mapping captures $.thread_id into session metadata.
3. Turn 2+ — {{ thread_id }} now renders the captured id, so every following message is routed to the same thread.

Configuring the Output Mapping but leaving a hardcoded id (or omitting {{ thread_id }} entirely) in the Input Template is the most common mistake: the value is captured but never resent, so the agent either starts a fresh thread every turn or — worse, with a fixed id — collides every conversation onto one shared thread and leaks context across unrelated test cases.

On the first turn, undefined placeholders render as empty strings. If your agent rejects an empty thread_id (rather than treating it as “create a new thread”), omit the field entirely on the first turn with a conditional, minding the trailing comma so the body stays valid JSON:

{
  "message": { "type": "human", "content": "{{ input.user_message }}" }{% if thread_id %},
  "thread_id": "{{ thread_id }}"{% endif %}
}

See Templates & Mapping — State Management for the full reference on how extracted values are stored and reused across turns.

Step 2: Create a Version with the Endpoint Connection

Navigate to your product and create a new Version. When configuring the version:

1. Fill in the version name, model, and any other relevant properties.
2. Turn on the Link Connection toggle. It is off by default; turning it on reveals the connection fields.
3. Choose HTTP Endpoint as the Conversation Target.
4. In the Endpoint Connection field, select the endpoint connection you created in Step 1.

Once Link Connection is on, the Endpoint Connection is the only required connection. For most integrations, this single endpoint handles the entire interaction lifecycle.

If your AI system requires separate endpoints for session initialization or cleanup, you can optionally configure Initialization and Finalization endpoint connections. These also appear only under Link Connection with the HTTP Endpoint target. See Endpoint Connection — Multi-Step Session Lifecycle for details.

Step 3: Run an Evaluation

Once your version is set up with an endpoint connection, you can run it from the Dashboard. Evaluations start from the version, not from the dataset:

1. Navigate to your product’s Versions section.
2. Open the version with the configured endpoint connection, or use the Run Evaluation button on its row.
3. Pick what to run in the dialog:
   • Specifications (recommended): every runnable Specification is preselected, and its linked datasets and metrics are taken from it.
   • Datasets & Metrics: choose the datasets and Metrics yourself.
4. Start the run.

The Dashboard then opens the version’s Sessions tab, where the run’s new sessions are already listed. Their evaluations appear on the Evaluations tab as each inference finishes.

Galtea will iterate through each test case, call your endpoint using the configured endpoint connection, record the resulting Traces, and evaluate them with the metrics you selected. Each test case produces a session with one or more traces depending on whether it’s a single-turn or multi-turn test.

Step 4: Evaluate Again with Other Metrics

The run above already produces scores. Use this step only when you want to score inferences you already have, for example against a metric you added later. It reuses the recorded inferences and does not call your endpoint again:

1. Navigate to the session in the Dashboard.
2. Click Run Evaluation.
3. Select the Metrics you want to use.
4. Run the evaluation.

The Dashboard then opens that session’s Evaluations tab, where the new evaluations are already listed.

Galtea will assess each trace using the selected metrics and provide scores and explanations.

For single-turn tests, metrics like Factual Accuracy and Answer Relevancy work well. For multi-turn conversations, consider Knowledge Retention, Role Adherence, and Conversation Completeness.

Collecting Spans During Direct Inference

There are three ways to collect spans during Direct Inference:

1. Output Mapping (no code) — Extract spans from the API response using a traces key in your output mapping.
2. SDK set_context (in your handler) — Pass {{ trace_id }} to your endpoint and use the SDK to create spans from within the handler.
3. W3C Trace Context Propagation (zero code) — Enable the traceparent header to automatically correlate your OTEL spans with Galtea traces.

Option 1: Extract Spans via Output Mapping

If your endpoint returns span data in its response, you can extract it using the traces key in the output mapping. Galtea will store each span object linked to the trace automatically.

Example API response:

{
  "response": "The answer is 42.",
  "traces": [
    {
      "name": "retrieve_context",
      "type": "RETRIEVER",
      "latencyMs": 120,
      "inputData": { "query": "meaning of life" },
      "outputData": { "documents": ["..."] }
    },
    {
      "name": "generate_response",
      "type": "GENERATION",
      "latencyMs": 350,
      "inputData": { "context": "..." },
      "outputData": { "text": "The answer is 42." }
    }
  ]
}

Output Mapping:

{
  "output": "$.response",
  "traces": "$.traces"
}

Galtea extracts the traces array and creates Span entities linked to the trace. Each object in the array must contain at least a name field and can include any Span properties:

Property	Type	Required	Description
name	string	Yes	Name of the traced operation
type	string	No	One of: SPAN, GENERATION, EVENT, AGENT, TOOL, CHAIN, RETRIEVER, EVALUATOR, EMBEDDING, GUARDRAIL
description	string	No	Human-readable description of the operation
inputData	object	No	Input parameters passed to the operation
outputData	object	No	Result returned by the operation
error	string	No	Error message if the operation failed
latencyMs	number	No	Execution time in milliseconds
metadata	object	No	Additional custom metadata
startTime	string	No	ISO 8601 timestamp when the operation started
endTime	string	No	ISO 8601 timestamp when the operation completed
parentTraceId	string	No	ID of the parent span for hierarchical relationships

This approach requires no SDK code in your endpoint handler — it works purely through configuration.

Option 2: Use set_context in Your Endpoint Handler

When running evaluations via Direct Inference, you can collect spans from your endpoint handler by linking the trace_id to your span context. Galtea automatically sends the X-Galtea-Inference-Id HTTP header with every Direct Inference request, so your endpoint can read it directly — no template changes needed.

Alternatively, you can pass {{ trace_id }} in the input template if you prefer to receive it in the request body.

1. Get the trace_id

From the HTTP header (recommended) — available automatically on every Direct Inference call:

trace_id = request.headers.get("X-Galtea-Inference-Id")

From the input template — include the placeholder in your endpoint connection’s input template:

{
  "model": "gpt-4",
  "messages": [
    {"role": "user", "content": "{{ input.user_message }}"}
  ],
  "metadata": {
    "trace_id": "{{ trace_id }}"
  }
}

2. Use set_context in Your Endpoint Handler

In your API endpoint, extract the trace_id from the request and use the SDK’s set_context / clear_context to associate spans with it:

@traced(type=SpanType.AGENT)
def run_agent(query: str) -> str:
    # Your agent logic here — all nested @traced calls
    # will be linked to the trace automatically
    return "Agent response to: " + query


def my_endpoint_handler(request):
    """Your API endpoint that Galtea calls during Direct Inference."""
    body = request.json()
    user_input = body["messages"][-1]["content"]
    trace_id = body["metadata"]["trace_id"]

    # Set Galtea context so all @traced calls are linked to this trace
    token = set_context(trace_id=trace_id)
    try:
        response = run_agent(user_input)
    finally:
        # Flush spans to Galtea and clear context
        clear_context(token)

    return {"choices": [{"message": {"content": response}}]}

All @traced-decorated functions called while the context is active will be automatically linked to the trace in Galtea.

For a complete guide on tracing setup, decorators, and context managers, see the Tracing Agent Operations tutorial.

Option 3: W3C Trace Context Propagation

If your service is instrumented with OpenTelemetry, you can automatically correlate your internal spans with Galtea traces using the W3C Trace Context standard — no code changes required on your side.

Prerequisites: Your service must be instrumented with OpenTelemetry and must receive the traceparent header Galtea sends with each Direct Inference request. This option correlates your spans with the traces Direct Inference creates. To create production sessions from OTel traces without Direct Inference, see Monitor Real User Traffic via OpenTelemetry.

How it works

When enabled, Galtea creates a unique W3C trace ID for each direct inference call and injects a traceparent header into the outbound request to your endpoint. Your OTEL-instrumented service automatically creates child spans under that OpenTelemetry trace. Galtea stores that W3C trace ID on the Trace, in its OpenTelemetry Trace ID field, so a collector can correlate the two.

1. Enable trace context propagation on your endpoint connection

In your Conversation endpoint connection, expand the Advanced Options section and scroll to Headers. Check the Enable W3C trace context propagation checkbox. This adds a traceparent header that Galtea will populate with the correct trace and span IDs on each inference call:

traceparent: 00-{trace-id}-{span-id}-01

When the checkbox is unchecked, no traceparent header is sent to your endpoint, and trace context is not propagated.

This option is only available for Conversation endpoint connections — the ones that handle inference calls. Initialization and Finalization endpoints are session lifecycle calls where trace correlation does not apply.

The trace ID is stored on each trace and visible in the dashboard, so you can correlate traces even without checking your observability platform.

2. Configure your service for OTEL

Ensure your service has OpenTelemetry instrumentation enabled. Most frameworks support auto-instrumentation which requires no code changes:

# Python example
pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install

3. Send your spans to Galtea

Point your OTel exporter or Collector at Galtea’s public OTel endpoint. Send OpenTelemetry Traces to Galtea covers the endpoint, the Authorization: Bearer <API key> requirement, the exporter and Collector configuration, and a curl check to verify your key.

Once configured, spans from your service will be automatically linked to the corresponding Galtea traces via the shared trace ID.

This approach only works for Direct Inference (where Galtea initiates the call to your endpoint). For SDK-based connections where your code calls the Galtea API, use the SDK’s span context mechanism instead.

To control how span content maps to Span records (the galtea.span.* attributes and the automatically mapped OTel fields), see How span content maps to Span records. To create production sessions from OTel traffic without Direct Inference, see Monitor Real User Traffic via OpenTelemetry.

Learn More

Endpoint Connection

Full reference for configuring endpoint connections

Version

Learn about versions and how endpoint connections integrate with them

Evaluations

Understand how evaluations work

Metrics

Browse available metrics for evaluating your AI

Tracing Agent Operations

Capture and analyze your agent’s internal operations