Simulating User Conversations

Learn how to use Galtea’s Conversation Simulator to test your AI with simulated multi-turn conversations.

Galtea’s Conversation Simulator allows you to test your AI products — chatbots, assistants, and agents — by simulating realistic multi-turn user interactions. This guide walks you through integrating your agent and running simulations using Behavior Tests.

Using the specification-driven workflow? If you’ve defined Specifications with linked metrics and tests, the evaluation pipeline handles simulation automatically — no manual session creation needed. See Evaluations from Specs.

Agent Integration Options

Simple

The quickest way to get started. Your function receives just the latest user message as a string.

def my_agent(user_message: str) -> str:
    # In a real scenario, call your model here
    return f"Your model output to: {user_message}"

Chat History

Use this when your agent needs the full conversation context. Your function receives the message list in the OpenAI format ({"role": "...", "content": "..."}).

def my_agent(messages: list[dict]) -> str:
    # messages follows the standard chat format:
    # [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}, ...]
    user_message = messages[-1]["content"]
    return f"Your model output to: {user_message}"

Structured

For full control over input and output — including optional usage tracking, cost tracking, and retrieval context for RAG evaluations.

def my_agent(input_data: AgentInput) -> AgentResponse:
    user_message = input_data.last_user_message_str()

    # Access structured input fields from the first user message's metadata
    # (e.g. when test case input is {"user_message": "hello", "chat_type": "support"})
    first_msg = input_data.messages[0] if input_data.messages else None
    chat_type = first_msg.metadata.get("chat_type") if first_msg and first_msg.metadata else None

    # In a real scenario, call your model here
    model_output = f"Your model output to: {user_message}"
    # Return AgentResponse with optional usage/cost tracking
    return AgentResponse(
        content=model_output,
        usage_info={"input_tokens": 100, "output_tokens": 50},
    )

All three signatures work with evaluations.run(), inference_results.generate(), and simulator.simulate(). Both sync and async functions are supported. The SDK auto-detects which signature you’re using from the type hint on the first parameter.

For the full list of fields available on AgentInput (including structured input access via message metadata), see the AgentInput reference.

Conversation Simulation Workflow

1. Implement Your Agent

   Define an agent function with one of the supported signatures above.

2. Prepare Scenario Data

   Create a CSV file with scenario data, or generate test cases from your specifications. Each row describes a user goal, persona, scenario, and the first user message.

3. Create a Test and Sessions

   Upload your scenario CSV to create a test, or use AI-generated tests from the dashboard. The platform generates a session for each test case.

4. Run the Simulator with Your Agent

   Use simulator.simulate() to execute the conversation between your agent and the simulated user for each session.

5. Evaluate the Results

   After simulation, trigger evaluations via evaluations.create() or review results in the dashboard’s Analytics tab.

Step-by-Step Guide

1. Create a Test and Sessions

First, create behavior test cases with user personas and goals. You can generate these from your product description or upload a CSV:

# Create a test suite using the behavior test options
# This can be done via the Dashboard or programmatically as shown here
test = galtea_client.tests.create(
    product_id=product.id,
    name=test_name,
    type="BEHAVIOR",
    # This time we provide a path to a CSV file with behavior tests, but you can also have Galtea generate them if you do not provide a CSV file
    test_file_path="path/to/behavior_test.csv",
)

# Get your test cases
# If Galtea is generating the test for you, it might take a few moments to be ready
test_cases = galtea_client.test_cases.list(test_id=test.id)

Once generation completes, you’ll see the resulting test cases in your dashboard. For the CSV upload format, see Behavior Tests.

2. Run the Conversation Simulator

For each test case/session, use the simulator to run the full simulation with your agent function:

# Define your agent function (see Agent Integration Options for all signatures)
def my_agent(user_message: str) -> str:
    return f"Response to: {user_message}"


# Run simulations with your agent function
for test_case in test_cases:
    session = galtea_client.sessions.create(version_id=version.id, test_case_id=test_case.id)

    result = galtea_client.simulator.simulate(session_id=session.id, agent=my_agent, max_turns=10)

    # Analyze results
    print(f"Scenario: {test_case.scenario}")
    print(f"Completed {result.total_turns} turns")
    print(f"Success: {result.finished}")
    if result.stopping_reason:
        print(f"Ended because: {result.stopping_reason}")

You can optionally use the @trace decorator to capture internal operations during simulation. Traces are automatically collected and saved per turn. See the Tracing Agent Operations guide for more details.

3. Evaluate the Session

    # After each simulation, you can create an evaluation
    evaluations = galtea_client.evaluations.create(
        session_id=session.id,
        metrics=[{"name": "Role Adherence"}],  # Replace with your metrics
    )
    for evaluation in evaluations:
        print(f"Evaluation created: {evaluation.id}")

When using structured JSON inputs in test cases (e.g. {"user_message": "hello", "chat_type": "support"}), the extra fields are available in messages[0].metadata on the AgentInput. See AgentInput reference for details.

Advanced Usage: RAG Agents with Retrieval Context

For Retrieval-Augmented Generation (RAG) agents, you can return the context that was retrieved and used to generate the response. This context will be logged with the inference result, enabling evaluations with metrics like Faithfulness and Contextual Relevancy.

def my_rag_agent(input_data: galtea.AgentInput) -> galtea.AgentResponse:
    user_message = input_data.last_user_message_str()

    # Your RAG logic to retrieve context and generate a response
    retrieved_docs = vector_store.search(user_message)
    response_content = llm.generate(prompt=user_message, context=retrieved_docs)

    return galtea.AgentResponse(
        content=response_content,
        retrieval_context=retrieved_docs,
        metadata={"docs_retrieved": len(retrieved_docs)},
    )

The retrieval_context field is optional and can contain:

• Retrieved document snippets or full documents
• Formatted context strings
• JSON-serializable data structures

By providing retrieval context, you enable Galtea to evaluate the faithfulness of your model’s responses relative to the retrieved information.

Next Steps

Specification-Driven Evaluations

Automate simulation + evaluation from specifications with evaluations.run().

Tracing Agent Operations

Capture internal operations of your AI agent during simulations.