Quickstart

Validate your AI product with Galtea

This quickstart walks you through running your first evaluation in Galtea for a conversational product using the Python SDK. It is intended for a technical audience and covers RAG pipelines, conversational scenarios, and security or safety checks.

Evaluation Workflow Overview

1. Tell Us About Your Product

   Provide details about your product so our models can create tailored testing content.
2. Install SDK & Connect

   Set up the Galtea Python SDK to interact with the platform.
3. Version Your Product

   Track different implementations to compare improvements over time.
4. Create Your Tests

   Create a test dataset to generate the test cases you’ll evaluate.
5. Choose Your Metrics

   Select from our metrics library or bring your own custom metrics.
6. Run Evaluations

   Test your product version with the selected test and metrics.
7. See Your Results

   Explore detailed insights and compare versions on the dashboard.

1. Tell Us About Your Product

If you’re new to Galtea, start by creating an account and registering your first product in the dashboard: Go to the dashboard.

Follow the Onboarding checklist on the product page (this is the guided setup shown inside your product). It will prompt you for product details and help you generate an API key. The product details you provide there are used to tailor generated tests and evaluations.

Keep your Product ID and API key handy for the SDK steps below.

The product information is crucial – it powers our ability to generate synthetic test data that’s specific to your use case.

2. Get Started with the SDK

You can use Galtea from the dashboard or via the Python SDK. This quickstart uses the Python SDK.

1. Get your API key

   In the Galtea dashboard, copy the API key you created during your product page onboarding (or navigate to Settings > Generate API Key).

2. Install the SDK

   pip install galtea
   
3. Connect to the platform

   from galtea import Galtea
   
   # Verify the SDK is properly installed by initializing it
   # Note: Replace "YOUR_API_KEY" with your actual API key from the Galtea dashboard
   galtea = Galtea(api_key="YOUR_API_KEY")
   print("Galtea SDK installed successfully!")
   

3. Version Your Product

Use versions to track iterations of your product and compare evaluation results over time. Each evaluation is tied to a version, so you can see how changes impact metrics.

If you completed the Onboarding checklist on your product page, you already have an initial version (created automatically).

To find IDs in the dashboard, open your product and click the 3-dot menu on any item, then select Copy ID.

Or resolve your product and version by name in the SDK — no IDs to copy:

# Look your product and version up by name — no need to copy IDs from the dashboard.
product = galtea.products.get_by_name(name="Financial Assistant")
version = galtea.versions.get_by_name(product_id=product.id, version_name="v1")

product_id = product.id
version_id = version.id

4. Create Your Tests

Prefer to generate tests automatically? If you’ve defined Specifications for your product, Galtea can generate tests and metrics for you — see the Specification-Driven Evaluations tutorial.

A single conversational product typically includes multiple components. In Galtea, you evaluate each component by creating a different test type, think of each test as a lens on the same system:

• RAG pipelines → create an Accuracy test (type="ACCURACY")
• Security or safety aspects → create a Security & Safety test (type="SECURITY")
• Conversational scenarios → create a Behavior test (type="BEHAVIOR")

You can create one or multiple test types for the same product, depending on which components you want to evaluate.

After creating a test, give it a moment to finish generating test cases. You can track progress in the dashboard.

If you already created a test during onboarding, you can reuse it here—skip the tests.create(...) step and just list its test cases using the test ID from the dashboard.

RAG

Accuracy tests are used to evaluate RAG pipelines and QA-style components within a conversational product. Galtea generates single-turn test cases (input and expected output) from a knowledge base to assess response quality and factual correctness.

test = galtea.tests.create(
    name="rag-accuracy-test",
    type="ACCURACY",
    product_id=product_id,
    ground_truth_file_path="knowledge.md",
    language="english",
    max_test_cases=20,
)

After a brief wait for test case generation, list the test cases:

test_cases = galtea.test_cases.list(test_id=test.id)
print(f"Using test '{test.name}' with {len(test_cases)} test cases.")

Learn more about Accuracy Tests and their configuration options.

Security

Security & Safety tests probe safety and security boundaries (e.g., misuse, toxicity, data leakage). For this quickstart, we’ll create a small Misuse test.

test = galtea.tests.create(
    name="misuse-security-test",
    type="SECURITY",
    product_id=product_id,
    variants=["misuse"],
    strategies=["original"],  # original must always be included
    max_test_cases=20,
)

After a brief wait for test case generation, list the test cases:

test_cases = galtea.test_cases.list(test_id=test.id)
print(f"Using test '{test.name}' with {len(test_cases)} test cases.")

Learn more about Security & Safety Tests, threats, and strategies.

Conversational

Behavior tests evaluate multi-turn conversations. Each test case defines a goal/persona/scenario and is meant to be run with the Conversation Simulator.

test = galtea.tests.create(
    name="conversation-behavior-test",
    type="BEHAVIOR",
    product_id=product_id,
    language="english",
    max_test_cases=20,
    strategies=["written"],
)

After a brief wait for test case generation, list the test cases:

test_cases = galtea.test_cases.list(test_id=test.id)
print(f"Using test '{test.name}' with {len(test_cases)} test cases.")

Learn more about Behavior Tests and Conversation Simulation.

5. Choose Your Metrics

Metrics define how evaluation outputs are scored. Galtea provides built-in metrics for common use cases (factual accuracy, faithfulness, toxicity, role adherence, and many more), and you can also create custom metrics with your own judge prompts. You can also generate metrics from specifications — the AI creates tailored judge prompts for each spec.

RAG

For a RAG-style Accuracy test, a good default is Factual Accuracy.

metric = galtea.metrics.get_by_name(name="Factual Accuracy")

Security

For Misuse Security & Safety tests, we’ll use Misuse Resilience.

metric = galtea.metrics.get_by_name(name="Misuse Resilience")

Conversational

For conversational evaluations, start with a metric like Role Adherence.

metric = galtea.metrics.get_by_name(name="Role Adherence")

6. Run the Evaluation

Now connect Galtea to your product. You define an agent function — a wrapper that takes a test case input, calls your AI system, and returns its output.

The SDK supports three agent function signatures — pick the one that matches how much context your agent needs:

Simple

The quickest way to get started. Your function receives just the latest user message as a string.

def my_agent(user_message: str) -> str:
    # In a real scenario, call your model here
    return f"Your model output to: {user_message}"

Chat History

Use this when your agent needs the full conversation context. Your function receives the message list in the OpenAI format ({"role": "...", "content": "..."}).

def my_agent(messages: list[dict]) -> str:
    # messages follows the standard chat format:
    # [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}, ...]
    user_message = messages[-1]["content"]
    return f"Your model output to: {user_message}"

Structured

For full control over input and output — including optional usage tracking, cost tracking, and retrieval context for RAG evaluations.

def my_agent(input_data: AgentInput) -> AgentResponse:
    user_message = input_data.last_user_message_str()

    # Access structured input fields from the first user message's metadata
    # (e.g. when test case input is {"user_message": "hello", "chat_type": "support"})
    first_msg = input_data.messages[0] if input_data.messages else None
    chat_type = first_msg.metadata.get("chat_type") if first_msg and first_msg.metadata else None

    # In a real scenario, call your model here
    model_output = f"Your model output to: {user_message}"
    # Return AgentResponse with optional usage/cost tracking
    return AgentResponse(
        content=model_output,
        usage_info={"input_tokens": 100, "output_tokens": 50},
    )

All three signatures work with evaluations.run(), inference_results.generate(), and simulator.simulate(). Both sync and async functions are supported. The SDK auto-detects which signature you’re using from the type hint on the first parameter.

For the full list of fields available on AgentInput (including structured input access via message metadata), see the AgentInput reference.

galtea.evaluations.run() does the rest in one call: it finds the specifications linked to your version, resolves their tests and metrics, runs your agent on every test case, and submits the results for scoring.

# One call resolves your specifications, their tests and metrics, runs your agent
# on every test case, and submits the results for scoring.
result = galtea.evaluations.run(
    version_id=version_id,
    agent=my_agent,
)
print(f"Launched {result['testCaseCount']} test cases across {len(result['specifications'])} specifications")

Prefer to drive each test case yourself? The Run Test-Based Evaluations tutorial shows the manual per-test-case loop for full control.

7. See Your Results

After launching, block until scoring finishes and read a summary straight from the SDK — a pass/fail count, plus any sessions that failed to run so you can triage them:

# Block until every evaluation leaves PENDING (defaults: up to 300s, polling every 5s).
completed = galtea.evaluations.wait_for(
    evaluation_ids=[e.id for e in result["evaluations"]],
)

# A readable pass/fail summary. Each metric defines its own passing threshold;
# 0.5 is used here as a simple example — adjust it to your metrics.
pass_threshold = 0.5
scored = [e for e in completed if e.score is not None]
passed = [e for e in scored if e.score >= pass_threshold]
print(f"{len(passed)}/{len(scored)} evaluations scored >= {pass_threshold}")

# Triage: list sessions whose agent run failed and produced no output to score.
failed_sessions = galtea.sessions.list(version_id=version_id, status="FAILED")
if failed_sessions:
    print(f"{len(failed_sessions)} sessions failed to run — inspect them:")
    for session in failed_sessions:
        print(f"  session {session.id}")

print(f"View full results at: https://platform.galtea.ai/product/{product_id}")

For the full picture, open your product in the Galtea platform and head to Analytics. Start with the aggregate scores to get a quick baseline, then slice by version, test, and metric to see where performance shifts.

The most useful part is drilling into individual test cases: you can inspect the exact prompt/output pair (or the full conversation for scenarios) and understand which specific inputs move the score up or down. This makes it easier to identify failure patterns, validate fixes in new versions, and iterate with confidence as you move toward production.

Next Steps

Congratulations! You’ve completed your first evaluation with Galtea.

You can now explore more advanced features like:

Tracing Agent Operations

Learn how to capture and analyze the internal operations of your AI agent.

Evaluating Production

Learn how to log and evaluate user queries from your production environment.

Specification-Driven Evaluations

Use specifications to automate test type selection and metric generation.

Dive Deeper

Explore these concepts to tailor tests, metrics, and workflows to your product. The Concepts overview shows how they all connect at a glance.

Concepts overview

How Galtea’s concepts connect — diagram + per-entity quick reference.

Product

A functionality or service being evaluated

Version

A specific iteration of a product

Test

A set of test cases for evaluating product performance

Session

A full conversation between a user and an AI system.

Inference Result

A single turn in a conversation between a user and the AI.

Evaluation

The assessment of an evaluation using a specific metric’s criteria

Metric

Ways to evaluate and score product performance

If you have any questions or need assistance, contact us at [e-mail].