Create a Custom Test

Learn how to create and upload custom tests using the SDK

Looking for the fastest way to create tests? If you’ve defined Specifications, Galtea can generate tests automatically — including type, variants, and test cases. Use this tutorial when you need full control over test creation or want to upload your own test data.

You can create Tests by uploading your own test data or by having Galtea generate test cases from a knowledge file.

Creating Tests with the SDK

The Galtea SDK provides methods to create Accuracy Tests, Security & Safety Tests, and Behavior Tests.

Upload Existing Test File

If you have a prepared test file in CSV format, you can upload it directly.

# Upload a pre-existing test file to the Galtea Platform
test = galtea.tests.create(
    name="financial-qa-test-" + run_identifier,
    type="ACCURACY",
    product_id=product_id,
    test_file_path="path/to/accuracy_test.csv",
)

The test file must follow the structure specified for:

• Accuracy Tests: Standard format with input, expected_output, etc.
• Security & Safety Tests: Standard format with adversarial inputs
• Behavior Tests: Conversation simulator format with goal, user_persona, etc.

If the file is not correctly formatted, test cases will not be created automatically, but you can still add them manually.

Behavior Tests: Use type="BEHAVIOR" to create conversation simulation tests that enable multi-turn dialogue evaluation with simulated users. See the Conversation Simulator Tutorial for complete examples.

Generate Test from Knowledge Base

You can provide a knowledge base file (e.g., PDF, TXT) and have Galtea generate the test cases for you. This is a powerful feature for creating comprehensive tests with minimal effort.

# Generate a test from a knowledge base file (PDF, TXT, etc.)
generated_test = galtea.tests.create(
    name="generated-financial-qa-test-" + run_identifier,
    type="ACCURACY",
    product_id=product_id,
    ground_truth_file_path="path/to/knowledge.md",
    language="english",
    max_test_cases=50,  # Limit the number of generated test cases
)

Next Steps

Run Test-Based Evaluations

Run your test against a specific product version.

API Reference

View the complete Test Service API reference.