Generate dataset configuration from specifications

Derives dataset type, variant, and name from specifications (rule-based).

With automaticCreation: true this does more than return a suggestion: it creates the datasets and their test-case generation continues on the background worker, so each one comes back with status PENDING.

Accuracy (QUALITY) specifications need groundTruthUri, a knowledge base file uploaded beforehand, because an accuracy dataset grades answers against known-correct ones and a specification never states what the correct answer is.

See Datasets.

POST /tests/generate-config

Generate dataset configuration from specifications

cURL

curl --request POST \
  --url https://api.galtea.ai/tests/generate-config \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '
{
  "specificationIds": [
    "spec_123",
    "spec_456"
  ],
  "automaticCreation": false,
  "maxTestCases": 30,
  "groundTruthUri": "<string>"
}
'

200

{
  "config": [
    {
      "name": "Scenarios for polite refusals",
      "type": "SCENARIOS",
      "specificationId": "spec_123",
      "productId": "prod_123",
      "maxTestCases": 30,
      "variants": [
        "custom"
      ],
      "customVariantDescription": "<string>",
      "strategies": [
        "written"
      ]
    }
  ],
  "tests": [
    {
      "id": "test_123",
      "productId": "prod_123",
      "userId": "user_123",
      "name": "Quality Test",
      "type": "QUALITY",
      "models": [
        "gpt-4",
        "claude-3"
      ],
      "groundTruthUri": "s3://my-bucket/tests/test_123/ground-truth.csv",
      "uri": "s3://my-bucket/tests/test_123/test.csv",
      "error": "<string>",
      "status": "SUCCESS",
      "fewShot": "Example few-shot learning data",
      "languageCode": "es-MX",
      "backgroundNoiseProfile": "street",
      "backgroundNoiseLevel": "medium",
      "variants": [
        "rag",
        "summarization"
      ],
      "customVariantDescription": "Custom variant description",
      "strategies": [
        "original"
      ],
      "customUserPersona": "Business analyst",
      "maxTestCases": 100,
      "sourceTestId": "<string>",
      "dataCatalogUri": "s3://my-bucket/tests/test_123/data-catalog.json",
      "metadata": {
        "key": "value"
      },
      "specificationId": "spec_123",
      "isExtendable": true,
      "createdAt": "2023-11-07T05:31:56Z",
      "deletedAt": "2023-11-07T05:31:56Z"
    }
  ]
}

Authorizations

Authorization · string · header · required

API key authorization. Pass your API key in the Authorization header as a Bearer token. Both new (gsk_*) and legacy (gsk-) API keys are accepted, e.g. Authorization: Bearer gsk_... or Authorization: Bearer gsk-....

Body

application/json

specificationIds · string[] · required

IDs of the specifications to generate dataset configs for

Minimum array length: 1

Example:

["spec_123", "spec_456"]

automaticCreation · boolean

When true, creates the datasets from the generated config and starts their test-case generation on the background worker.

Example:

false

maxTestCases · integer

How many test cases to generate per dataset. Defaults to 3. The upper limit is per test type and deployment-configured, and is enforced when the dataset is created (automaticCreation: true).

Required range: x > 0

Example:

30

groundTruthUri · string<uri>

Presigned upload URL of the knowledge base file, uploaded beforehand. Required for accuracy (QUALITY) specifications, ignored for the other test types.

Response

200

application/json

Dataset configuration generated, and the datasets created when asked for

config · object[] · required

One generated dataset config per specification

Show child attributes

tests · object[]

The created datasets, present only when automaticCreation was true. Each comes back PENDING, with its test-case generation running on the background worker.

Show child attributes