Skip to main content

​
Returns

Returns a Test object for the given parameters.

​
Examples

  • Quality Test
  • Red Teaming
  • Scenario Test
  • Quality Custom Test from CSV
Create a quality test to evaluate the quality and correctness of outputs.
test = galtea.tests.create(
    name="example-test-tutorial",
    type="QUALITY",
    product_id="YOUR_PRODUCT_ID",
    ground_truth_file_path="path/to/knowledge_file.pdf",
    language='english',
    variants=["paraphrased", "typos", "incorrect"],
    max_test_cases=100
)

​
Parameters

​
name
string
required
The name of the test.
​
type
string
required
The type of test. Possible values:
  • QUALITY: Tests that evaluate the quality and correctness of outputs
  • RED_TEAMING: Tests that evaluate security, safety, and bias aspects
  • SCENARIOS: Tests that use conversation simulation to evaluate multi-turn dialogue interactions
​
product_id
string
required
The ID of the product you want to evaluate.
​
ground_truth_file_path
string
Path to a local file containing the knowledge base. This file is uploaded to Galtea, which then generates test cases based on its content.
Supported formats include .pdf, .txt, .json, .html, .md, and .zip. Upload a .zip archive to generate test cases from multiple documents at once.
​
few_shot_examples
string
Optional few-shot examples to provide more context to our system about how the test cases should be generated. This can help our system better understand the expected format and style wanted for the test cases. Example:
Q: What is the capital of France?
A: The capital of France is Paris.
Q: What is the capital of Germany?
A: The capital of Germany is Berlin.
This field only applies if tests are generated by Galtea and are of type QUALITY.
​
custom_user_focus
string
Narrow down the scope of generated scenarios by describing a specific type of user, context, or situation. This helps ensure test cases align with your most relevant goals and flows. Example: “A medical professional specialized in dementia with more than 15 years on the field.”
This field only applies if tests are generated by Galtea and are of type SCENARIOS.
​
language
string
The language for generating synthetic test cases if ground_truth_file_path is provided. Defaults to the language detected in the ground truth file. If provided, it should be written in English and be a valid language name from the ISO 639 standard (e.g., “english”, “spanish”, “french”). More information can be found at https://en.wikipedia.org/wiki/List_of_ISO_639_language_codes.
​
variants
list[string]
A list of strings that specifies how to generate test cases. The meaning of this parameter depends on the test type:
  • For RED_TEAMING tests: Specifies which threat categories to generate test cases for (e.g., ["data_leakage", "Misuse"]).
  • For QUALITY tests: Specifies which evolutions to apply to the base test cases, creating variations like paraphrased questions or adding typos (e.g., ["paraphrased", "typos"]).
Note on Terminology: In the context of Red Teaming, this parameter is used to select “Threats” (as described in Threat Categories). In other test types, it refers to specific variations or evolutions.
​
strategies
list[string]
A list of strings that specifies how to generate test cases related to its style.
  • For RED_TEAMING tests: Strategies are techniques for modifying or obfuscating prompts generated for each threat (as specified in variants for Red Teaming tests). The original strategy must always be included. See the full list of red teaming strategies.
  • For SCENARIOS tests: Strategies define the conversation style. Currently written and spoken are supported, which influence the tone and formality of the generated dialogues. If not specified, the default strategy is written.
​
custom_variant_description
str
Description for guiding synthetic data generation for security tests. This parameter is used to define the behavior of the Custom threat when it is included in the variants list for RED_TEAMING tests.
​
max_test_cases
int
An optional integer to limit the maximum number of test cases generated by Galtea when ground_truth_file_path is used.
​
test_file_path
string
Path to a local CSV file containing predefined test cases. This file is uploaded to Galtea.File Format by Test Type:
  • QUALITY/RED_TEAMING: Standard format with input, expected_output, tag, source columns
  • SCENARIOS: Conversation simulator format with goal, user_persona, initial_prompt, stopping_criterias, max_iterations, scenario columns
See the Conversation Simulator Tutorial for detailed SCENARIOS CSV format examples.
​
source_test_id
string
The ID of an existing QUALITY test. If provided when creating a SCENARIOS test, Galtea will use the approved test cases from the source quality test to generate conversational scenarios. This allows you to transform single-turn Q&A pairs into multi-turn dialogues.
This parameter is only applicable when type is SCENARIOS and cannot be used with ground_truth_file_path or test_file_path.