Monitor Production Responses

Learn how to log and evaluate user queries from your production environment.

You can use Galtea to log and evaluate real user interactions from your production environment. This helps you monitor your product’s performance over time.

This tutorial evaluates production sessions on demand, one call at a time. To score production sessions automatically and continuously, create a Monitor instead: an always-on rule that samples and scores a product’s production sessions with a set of metric families, with no run to trigger by hand.

Recommended: Use Your Own Conversation IDs

If your application already has its own conversation or chat ids, you can keep using them as the handle for a Galtea session, so you never store Galtea’s own session id. First, create the session once with galtea.sessions.create(custom_id=<your id>, ...). Then log every turn with session_custom_id=<your id> (plus a version_id or product_id anchor): each call finds that session by your custom id and appends to it.

# At the start of the conversation, create the session ONCE with YOUR own id.
galtea.sessions.create(
    custom_id="conversation-42",  # your own conversation id
    version_id=VERSION_ID,
    is_production=True,
)


# In your application's request handler...
def handle_user_query_with_custom_id(conversation_id: str, user_query: str) -> str:
    # Your logic to get a response from your model
    model_response = your_product_function(user_query)

    # Log and evaluate in one call, using YOUR conversation id as the handle.
    # Galtea finds the existing session by that id and appends the turn to it.
    galtea.inference_results.create_and_evaluate(
        session_custom_id=conversation_id,  # the id you created the session with
        version_id=VERSION_ID,  # or product_id=product_id to find it under the product
        input=user_query,
        output=model_response,
        metrics=[
            {"name": "Role Adherence"},
            {"name": "Answer Relevancy"},
        ],
    )

    return model_response


# Test the handler: both calls land in the same session
handle_user_query_with_custom_id("conversation-42", "What are your business hours?")
handle_user_query_with_custom_id("conversation-42", "Are you open on weekends?")

The same works for logging turns without evaluating them:

# Create the session once, up front, with your own conversation id.
galtea.sessions.create(
    custom_id="conversation-1234",
    version_id=VERSION_ID,
    is_production=True,
)


def handle_conversation_turn(conversation_id: str, user_input: str) -> str:
    # Replace this with your actual model call
    model_output = f"This is a simulated response to '{user_input}'"

    # Log each turn under your own conversation id. Galtea finds the existing
    # session by that id and appends the turn to it.
    galtea.inference_results.create(
        session_custom_id=conversation_id,
        version_id=VERSION_ID,  # or product_id=product_id
        input=user_input,
        output=model_output,
    )
    return model_output


# This would happen dynamically in your application.
for question in [
    "What are some lower-risk investment strategies?",
    "With age, should the investment strategy change?",
    "Great, thanks!",
]:
    handle_conversation_turn("conversation-1234", question)

Sessions are created only through the sessions API. If no session matches your session_custom_id and anchor, the call returns a 400 error. Create the session first with galtea.sessions.create(custom_id=...).

If your system already emits OpenTelemetry traces, you can create production sessions without any SDK calls. See Monitor Real User Traffic via OpenTelemetry.

Single-Turn Production Monitoring

For simple, single-turn interactions, create a production session and use galtea.inference_results.create_and_evaluate() to log and evaluate the interaction in a single call.

You can pass Specification IDs so the API resolves their linked metrics automatically (recommended), or list metrics explicitly.

Using specification IDs

# In your application's request handler...
def handle_user_query_with_specifications(user_query: str, retrieval_context: str | None = None) -> str:
    # Your logic to get a response from your model
    model_response = your_product_function(user_query, retrieval_context)

    # Log and evaluate the interaction, resolving metrics from your specifications
    session = galtea.sessions.create(version_id=VERSION_ID, is_production=True)
    galtea.inference_results.create_and_evaluate(
        session_id=session.id,
        input=user_query,
        output=model_response,
        retrieval_context=retrieval_context,
        specification_ids=specification_ids,
    )

    return model_response


# Test the handler
handle_user_query_with_specifications("What are your business hours?", "Business hours: 9am-5pm Monday-Friday")

Using metrics

# In your application's request handler...
def handle_user_query(user_query: str, retrieval_context: str | None = None) -> str:
    # Your logic to get a response from your model
    model_response = your_product_function(user_query, retrieval_context)

    # Log and evaluate the interaction in Galtea
    session = galtea.sessions.create(version_id=VERSION_ID, is_production=True)
    galtea.inference_results.create_and_evaluate(
        session_id=session.id,
        input=user_query,
        output=model_response,
        retrieval_context=retrieval_context,
        metrics=[
            {"name": "Role Adherence"},
            {"name": "Answer Relevancy"},
            {"name": "Faithfulness"},
        ],
    )

    return model_response


# Test the handler
handle_user_query("What are your business hours?", "Business hours: 9am-5pm Monday-Friday")

Multi-Turn Production Monitoring (Conversations)

For multi-turn conversations, use the session-based workflow to log the entire interaction first and then evaluate it. If your application has its own conversation ids, prefer the session_custom_id flow above: create the session once with a custom_id in step 1, then pass session_custom_id (plus version_id or product_id) instead of session_id in step 2.

1. 1. Create a Session

   First, create a session at the start of the conversation. For production monitoring, make sure to set is_production=True.

   # Use is_production=True for real user interactions
   session = galtea.sessions.create(
       custom_id="CLIENT_PROVIDED_SESSION_ID",  # Optional: a custom ID to associate this session in Galtea Platform to the one in your real application.
       version_id=VERSION_ID,
       is_production=True,
   )
   
2. 2. Log Conversation Turns

   Next, log the user-assistant interactions. You can do this individually as each turn happens or in a single batch after the conversation ends.

   Log Turns Individually

   This approach is useful for logging interactions in real-time in a live application.

   def get_model_response(user_input: str) -> str:
       # Replace this with your actual model call
       model_output = f"This is a simulated response to '{user_input}'"
       return model_output
   
   
   # This would happen dynamically in your application.
   user_questions = [
       "What are some lower-risk investment strategies?",
       "With age, should the investment strategy change?",
       "Great, thanks!",
   ]
   
   for question in user_questions:
       model_response = get_model_response(question)
       # Log the turn to Galtea right after it happens
       inference_result = galtea.inference_results.create(session_id=session.id, input=question, output=model_response)
   

   Log Turns in a Batch

   If you have the entire conversation history, you can log all turns at once for efficiency.

   # The conversation must be in the standard format: a list of role/content dictionaries
   conversation_turns = [
       {"role": "user", "content": "What are some lower-risk investment strategies?"},
       {
           "role": "assistant",
           "content": "For lower-risk investments, consider diversified index funds, bonds, or Treasury securities.",
       },
       {"role": "user", "content": "With age, should the investment strategy change?"},
       {
           "role": "assistant",
           "content": "Yes, many advisors recommend shifting to more conservative investments as you approach retirement.",
       },
       {"role": "user", "content": "Great, thanks!"},
       {"role": "assistant", "content": "You're welcome!"},
   ]
   
   galtea.inference_results.create_batch(session_id=session_batch.id, conversation_turns=conversation_turns)
   
3. 3. Evaluate the Session

   Finally, once the conversation is complete and all turns are logged, you can run an evaluation on the entire session. Pass Specification IDs so the API resolves their linked metrics automatically (recommended), or list metrics explicitly.

   Using specification IDs

   # Or resolve metrics from your product's specifications instead of listing them
   galtea.evaluations.create(session_id=session.id, specification_ids=specification_ids)
   

   Using metrics

   galtea.evaluations.create(session_id=session.id, metrics=METRICS_TO_EVALUATE)
   

For more details on evaluating multi-turn conversations, see the Evaluating Conversations guide.

Reading Monitor Results

If you have set up a Monitor on the Galtea dashboard to continuously evaluate a sample of your production sessions, you can pull its results with a single call:

curl -sS -H "Authorization: Bearer ${GALTEA_API_KEY}" \
  "https://api.galtea.ai/monitors/${MONITOR_ID}/results?days=7"

The days query parameter sets the trailing window to aggregate, bucketed by day. It defaults to 7 and accepts up to 90.

The response is built to give you an honest read of your product’s quality, not just a bare average. Keep these ideas in mind when you interpret it:

• Each metric family gets its own score series. A metric family is a metric and every later revision of it (for example, when you edit its judge prompt), grouped as one line of history. The response returns one series per family bound to the monitor, each with a whole-range headline score and one point per day bucket.
• Scores are sampling-weighted. A monitor does not evaluate every session; it evaluates a sample to control cost. Sessions from a period when sampling was lower count for more in the average, so a period with a smaller sample does not silently pull the score toward its own noise.
• A confidence interval tells you how much to trust a score. The confidence interval is the range that most likely contains the true score. Galtea publishes a 95% interval once a bucket has scored at least 30 sessions. Below that floor, confidenceInterval is null and collectingData is true, so you know the score is a preview, not a settled number, until more sessions land.
• The worst metric family drives the headline, never an average. worstMetric is the lowest-scoring family over the whole range, not a blended score across families. Use it to see which quality dimension needs attention first.
• Markers flag context changes. markers.metricRevisions marks when a bound metric family was edited, and markers.samplingChanges marks when the sampling rate changed. Both can explain a shift in the score series that isn’t a real quality change.
• Spend shows your current billing cycle. spend.spentCredits and spend.pendingCredits are the credits already used and reserved for evaluations still running this cycle; spend.capCredits is the monthly cap you configured for the monitor (null if you did not set one). If spend nears the cap, the monitor lowers its sampling rate to stay within budget, which is exactly what the sampling-change markers and sampling-weighted scores are designed to absorb.
• Sessions are listed, not just counted. sessions.scored lists the most recently evaluated sessions (capped at 200), each with the turn count it was scored at. sessions.waiting lists production sessions that have not closed yet (capped at 100); a session is scored only once it closes. A session closes when it is explicitly finished, or when the product’s auto-close setting closes it after inactivity.

Next Steps

Evaluating Conversations

Evaluate full multi-turn production conversations.

Tracing Agent Operations

Capture and analyze internal agent operations in production.