GitHub Actions

Learn how to integrate Galtea’s evaluation capabilities into your GitHub Actions workflow

Initial Setup

To initialize the Galtea class, you need to provide your API key obtained in the settings page of the Galtea platform.

Before you can use GitHub Actions with Galtea, you need to perform some credentials and variables configuration:

• Configure Repository Secrets and Variables
  • Go to your repository’s “Settings” tab
  • Navigate to “Secrets and variables” > “Actions”
  • Add the following:
    • Secret: GALTEA_API_KEY - Your Galtea API key
    • Variable: GALTEA_PRODUCT_ID - Your Galtea Product ID
  • Click “Add” after each entry

Dependencies

Create a requirements.txt file in your repository and add the dependencies required for your project. At minimum, you’ll need galtea library:

requirements.txt

galtea

Create your GitHub Action

Create a .github/workflows/evaluate.yml file in your repository with the following content:

.github/workflows/evaluate.yml

name: Galtea Evaluation

on:
  push:
  pull_request:
  workflow_dispatch:

jobs:
  evaluate:
    env:
      GALTEA_API_KEY: ${{ secrets.GALTEA_API_KEY }}
      GALTEA_PRODUCT_ID: ${{ vars.GALTEA_PRODUCT_ID }}
    runs-on: ubuntu-latest
    steps:
     - name: Checkout
       uses: actions/checkout@v4

     - name: Setup Python
       uses: actions/setup-python@v5
       with:
         python-version: '3.10'

     - name: Install dependencies
       run: |
        pip install -r requirements.txt

     - name: Run Evaluation
       run: |
        python evaluate.py

Create your Test Script

Create an evaluate.py file in your repository with the following content.

You give galtea.evaluations.run(version_id=..., agent=my_agent) one thing: a Python function that calls your agent. From there it does the whole evaluation in one call. It finds the Specifications of your product, resolves their linked Datasets and Metrics, calls my_agent once per Test Case inside this CI job, and submits the answers for scoring. Your agent does not need to be deployed anywhere, so this also works as a pre-deploy gate.

import os

from galtea import Galtea

# GALTEA_API_KEY and GALTEA_PRODUCT_ID come from the workflow's `env:` block above.
# The GITHUB_* ones are provided by GitHub Actions itself, so you never declare them.
galtea = Galtea(api_key=os.environ["GALTEA_API_KEY"])

PRODUCT_ID = os.environ["GALTEA_PRODUCT_ID"]
COMMIT_SHA = os.environ["GITHUB_SHA"]

# One version per workflow run, named after the commit so every result is traceable to the
# code that produced it. Version names are unique per product, and the same commit is built
# again on `pull_request` and on every job re-run, so the run id and attempt keep it unique.
version = galtea.versions.create(
    name=f"ci-{COMMIT_SHA[:7]}-{os.environ['GITHUB_RUN_ID']}.{os.environ['GITHUB_RUN_ATTEMPT']}",
    product_id=PRODUCT_ID,
)
if version is None:
    raise RuntimeError("Could not create the version — check GALTEA_API_KEY and GALTEA_PRODUCT_ID")


# Your product under test. Galtea calls it once per test case.
def my_agent(user_message: str) -> str:
    # In a real scenario, this would call your actual AI model or API
    return "This is a placeholder model answer."


# One call runs the whole evaluation: it finds the product's specifications, resolves their
# linked datasets and metrics, runs the agent on every test case, and submits the results.
result = galtea.evaluations.run(version_id=version.id, agent=my_agent)

print(f"Evaluated {result['testCaseCount']} test cases against version {version.name}")

run() discovers work through specifications, so each one needs both its metrics and a dataset linked. A specification missing either is skipped silently, and the run evaluates nothing. That setup is done once (see Writing Specifications), not on every CI run. If you need per-test-case control instead, see Run Dataset-Based Evaluations.

Success! 🎉 Your GitHub Actions workflow is now configured to run evaluations with Galtea. Each time you push changes, it will automatically evaluate your product using the latest version of your code.