> ## Documentation Index
> Fetch the complete documentation index at: https://docs.galtea.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluating Conversations

> Learn how to evaluate multi-turn conversations using Galtea's session-based workflow across test-based, past conversations, and production monitoring.

To accurately evaluate interactions within a dialogue, you can use Galtea's session-based workflow. This approach allows you to log an entire conversation and then run evaluations on all of its turns at once.

Certain metrics are specifically designed for conversational analysis and require the full context:

* **Role Adherence**: Measures how well the AI stays within its defined role.
* **Knowledge Retention**: Assesses the model's ability to remember and use information from previous turns.
* **Conversation Completeness**: Evaluates whether the conversation has reached a natural and informative conclusion.
* **Conversation Relevancy**: Assesses whether each turn in the conversation is relevant to the ongoing topic.

## The Session-Based Workflow

Every evaluation follows the same two steps: first get the conversation into a [Session](/concepts/product/version/session), then evaluate that session.

<Steps>
  <Step title="Capture the conversation in a session">
    A [Session](/concepts/product/version/session) holds all the turns of one conversation. How the turns get there depends on your use case (simulated, ingested, or live). See [step 1](#1-capture-the-conversation-in-a-session).
  </Step>

  <Step title="Evaluate the session">
    Once the turns are logged, evaluate the whole conversation at once with `evaluations.create()`. See [step 2](#2-evaluate-the-session).
  </Step>
</Steps>

## 1. Capture the conversation in a session

How turns get into a session depends on where the conversation comes from. Pick the use case that matches yours. Each one ends with a logged `session` you evaluate in [step 2](#2-evaluate-the-session).

<Tabs>
  <Tab title="Test-based evaluation">
    Use this when you have test cases. Galtea's [Conversation Simulator](/concepts/product/test/case/conversation-simulator) drives a synthetic user against your own agent, so you provide an agent function. It requires `test_case_id`.

    First, define an agent function that connects Galtea to your product:

    <Tabs>
      <Tab title="Simple">
        The quickest way to get started. Your function receives just the latest user message as a string.

        ```python theme={"system"} theme={"system"}
        def my_agent(user_message: str) -> str:
            # In a real scenario, call your model here
            return f"Your model output to: {user_message}"
        ```
      </Tab>

      <Tab title="Chat History">
        Use this when your agent needs the full conversation context. Your function receives the message list in the OpenAI format (`{"role": "...", "content": "..."}`).

        ```python theme={"system"} theme={"system"}
        def my_agent(messages: list[dict]) -> str:
            # messages follows the standard chat format:
            # [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}, ...]
            user_message = messages[-1]["content"]
            return f"Your model output to: {user_message}"
        ```
      </Tab>

      <Tab title="Structured">
        For full control over input and output — including optional usage tracking, cost tracking, and retrieval context for RAG evaluations.

        ```python theme={"system"} theme={"system"}
        def my_agent(input_data: AgentInput) -> AgentResponse:
            user_message = input_data.last_user_message_str()

            # Access structured input fields from the first user message's metadata
            # (e.g. when test case input is {"user_message": "hello", "chat_type": "support"})
            first_msg = input_data.messages[0] if input_data.messages else None
            chat_type = first_msg.metadata.get("chat_type") if first_msg and first_msg.metadata else None

            # In a real scenario, call your model here
            model_output = f"Your model output to: {user_message}"
            # Return AgentResponse with optional usage/cost tracking
            return AgentResponse(
                content=model_output,
                usage_info={"input_tokens": 100, "output_tokens": 50},
            )
        ```
      </Tab>
    </Tabs>

    <Info>
      All three signatures work with `evaluations.run()`, `inference_results.generate()`, and `simulator.simulate()`. Both sync and async functions are supported. The SDK auto-detects which signature you're using from the type hint on the first parameter.

      For the full list of fields available on `AgentInput` (including structured input access via message metadata), see the [AgentInput reference](/sdk/api/agent/input).
    </Info>

    Then fetch a test case, create a session, and run the simulator with your agent:

    ```python theme={"system"}
    # Fetch your test cases (created from a CSV of behavior tests)
    test_cases = galtea_client.test_cases.list(test_id=behavior_test.id)
    if not test_cases:
        raise ValueError("No test cases found")

    # Take one test case (loop over `test_cases` to evaluate them all)
    test_case = test_cases[0]

    # Create a session for the test case, then let the simulator drive the
    # conversation between a synthetic user and your agent function (`my_agent`)
    session = galtea_client.sessions.create(version_id=version_id, test_case_id=test_case.id)
    galtea_client.simulator.simulate(
        session_id=session.id,
        agent=my_agent,
        max_turns=test_case.max_iterations or 10,
    )
    ```

    <Info>
      See the full simulation workflow in [Simulating Conversations](/sdk/tutorials/simulating-conversations).
    </Info>
  </Tab>

  <Tab title="Past conversations (offline ingestion)">
    Use this when the conversation already happened outside Galtea. Create a session (no `test_case_id`) and log all of its turns in a single batch.

    ```python theme={"system"}
    # The conversation already happened: create a session (no test_case_id) and log every turn
    session = galtea_client.sessions.create(
        version_id=version_id,
        custom_id="EXTERNAL_CONVERSATION_ID",  # optional: map to your own conversation ID
        is_production=True,  # set to True when these are real users
    )

    conversation_turns = [
        {"role": "user", "content": "What are some lower-risk investment strategies?"},
        {
            "role": "assistant",
            "content": "For lower-risk investments, consider diversified index funds, bonds, or Treasury securities.",
            "retrieval_context": "Low-risk investment options include index funds, government bonds, and Treasury securities.",
        },
        {"role": "user", "content": "With age, should the investment strategy change?"},
        {
            "role": "assistant",
            "content": "Yes, many advisors recommend shifting to more conservative investments as you approach retirement.",
            "retrieval_context": "Financial advisors typically recommend a more conservative asset allocation as investors near retirement age.",
        },
    ]

    # Log all turns at once
    galtea_client.inference_results.create_batch(session_id=session.id, conversation_turns=conversation_turns)
    ```
  </Tab>

  <Tab title="Monitoring (production)">
    Use this for real-time logging of user interactions from your live product. Create the session with `is_production=True` and log turns as they happen, or batch them at the end.

    <Tabs>
      <Tab title="Log turns individually">
        ```python theme={"system"}
        # Create a production session and log each turn as it happens in your live app
        session = galtea_client.sessions.create(version_id=version_id, is_production=True)


        def your_product(user_input: str) -> str:
            return f"This is a simulated response to '{user_input}'"


        def handle_turn(user_input: str) -> str:
            model_output = your_product(user_input)
            galtea_client.inference_results.create(session_id=session.id, input=user_input, output=model_output)
            return model_output


        # Simulate production interactions
        handle_turn("Hello!")
        handle_turn("What services do you offer?")
        ```
      </Tab>

      <Tab title="Log turns in a batch">
        ```python theme={"system"}
        # Or, if you already have the full transcript, create the session and log all turns at once
        session = galtea_client.sessions.create(version_id=version_id, is_production=True)

        conversation_turns = [
            {"role": "user", "content": "What are some lower-risk investment strategies?"},
            {
                "role": "assistant",
                "content": "For lower-risk investments, consider diversified index funds, bonds, or Treasury securities.",
            },
        ]

        galtea_client.inference_results.create_batch(session_id=session.id, conversation_turns=conversation_turns)
        ```
      </Tab>
    </Tabs>

    <Info>
      See the dedicated guide: [Monitor Production Responses](/sdk/tutorials/monitor-production-responses-to-user-queries).
    </Info>
  </Tab>
</Tabs>

## 2. Evaluate the session

Whichever use case you picked, you now have a `session` with all its turns logged. Evaluate the whole conversation at once, either by passing [Specification](/concepts/product/specification) IDs so the API resolves their linked metrics automatically (recommended), or by listing metrics explicitly.

<Tabs>
  <Tab title="Using specification IDs">
    ```python theme={"system"}
    # Evaluate the whole conversation; metrics are resolved from the specifications automatically
    galtea_client.evaluations.create(
        session_id=session.id,
        specification_ids=[conversation_spec.id],
    )
    ```
  </Tab>

  <Tab title="Using metrics">
    ```python theme={"system"}
    # Evaluate the whole conversation by listing metrics explicitly
    galtea_client.evaluations.create(
        session_id=session.id,
        metrics=[
            {"name": "Conversation Relevancy"},
            {"name": "Role Adherence"},
            {"name": "Knowledge Retention"},
        ],
    )
    ```
  </Tab>
</Tabs>

<Info>
  For test-based evaluation, run this evaluation inside your loop so every simulated session is evaluated.
</Info>

## Custom Metrics with Full Conversation Access

When you use `CustomScoreEvaluationMetric`, your `measure()` method always receives an `inference_results` parameter containing `InferenceResult` objects. For session evaluations this includes all turns; for single inference result evaluations it contains one item. This enables conversation-level scoring (e.g., consistency checks, cross-turn analysis).

```python theme={"system"}
from galtea import CustomScoreEvaluationMetric, InferenceResult


class ConversationConsistency(CustomScoreEvaluationMetric):
    """Scores how consistently the assistant responds across all turns."""

    def __init__(self):
        super().__init__(name=metric_name)

    def measure(self, *args, inference_results: list[InferenceResult] | None = None, **kwargs) -> float:
        if not inference_results:
            return 0.0
        # Access the full conversation for cross-turn analysis
        assistant_outputs = [ir.actual_output for ir in inference_results if ir.actual_output]
        if len(assistant_outputs) < 2:
            return 1.0
        # Your custom logic here (e.g., check for contradictions across turns)
        return 0.9


galtea_client.evaluations.create(
    session_id=session.id,
    metrics=[
        {"name": "Role Adherence"},
        {"score": ConversationConsistency()},  # Custom multi-turn metric
    ],
)
```

<Info>
  Each `InferenceResult` object provides `actual_input`, `actual_output`, `latency`, `index`, and other fields. Retrieval context is not a field on the object; it is derived from the turn's `RETRIEVER` trace spans. See the [custom metrics tutorial](/sdk/tutorials/evaluate-with-custom-metrics) for more on custom metrics.
</Info>

## Learn More

<CardGroup cols={2}>
  <Card title="Session" icon="clock-rotate-left" iconType="solid" href="/concepts/product/version/session">
    A full conversation between a user and an AI system.
  </Card>

  <Card title="Inference Result" icon="arrow-right-from-bracket" iconType="solid" href="/concepts/product/version/session/inference-result">
    A single turn in a conversation between a user and the AI.
  </Card>

  <Card title="Evaluation" icon="clipboard-check" iconType="solid" href="/concepts/product/version/session/evaluation">
    The assessment of an evaluation using a specific metric's criteria
  </Card>

  <Card title="Conversation Simulator" icon="user-group" iconType="solid" href="/concepts/product/test/case/conversation-simulator">
    Test your conversational AI with simulated multi-turn conversations
  </Card>
</CardGroup>