> ## Documentation Index > Fetch the complete documentation index at: https://docs.galtea.ai/llms.txt > Use this file to discover all available pages before exploring further. # Monitor Production Responses > Learn how to log and evaluate user queries from your production environment. You can use Galtea to log and evaluate real user interactions from your production environment. This helps you monitor your product's performance over time. This tutorial evaluates production sessions on demand, one call at a time. To score production sessions automatically and continuously, create a [Monitor](/concepts/product/monitor) instead: an always-on rule that samples and scores a product's production sessions with a set of metric families, with no run to trigger by hand. ### Recommended: Use Your Own Conversation IDs If your application already has its own conversation or chat ids, you can keep using them as the handle for a Galtea session, so you never store Galtea's own session id. First, create the session once with `galtea.sessions.create(custom_id=, ...)`. Then log every turn with `session_custom_id=` (plus a `version_id` or `product_id` anchor): each call finds that session by your custom id and appends to it. ```python theme={"system"} # At the start of the conversation, create the session ONCE with YOUR own id. galtea.sessions.create( custom_id="conversation-42", # your own conversation id version_id=VERSION_ID, is_production=True, ) # In your application's request handler... def handle_user_query_with_custom_id(conversation_id: str, user_query: str) -> str: # Your logic to get a response from your model model_response = your_product_function(user_query) # Log and evaluate in one call, using YOUR conversation id as the handle. # Galtea finds the existing session by that id and appends the turn to it. galtea.inference_results.create_and_evaluate( session_custom_id=conversation_id, # the id you created the session with version_id=VERSION_ID, # or product_id=product_id to find it under the product input=user_query, output=model_response, metrics=[ {"name": "Role Adherence"}, {"name": "Answer Relevancy"}, ], ) return model_response # Test the handler: both calls land in the same session handle_user_query_with_custom_id("conversation-42", "What are your business hours?") handle_user_query_with_custom_id("conversation-42", "Are you open on weekends?") ``` The same works for logging turns without evaluating them: ```python theme={"system"} # Create the session once, up front, with your own conversation id. galtea.sessions.create( custom_id="conversation-1234", version_id=VERSION_ID, is_production=True, ) def handle_conversation_turn(conversation_id: str, user_input: str) -> str: # Replace this with your actual model call model_output = f"This is a simulated response to '{user_input}'" # Log each turn under your own conversation id. Galtea finds the existing # session by that id and appends the turn to it. galtea.inference_results.create( session_custom_id=conversation_id, version_id=VERSION_ID, # or product_id=product_id input=user_input, output=model_output, ) return model_output # This would happen dynamically in your application. for question in [ "What are some lower-risk investment strategies?", "With age, should the investment strategy change?", "Great, thanks!", ]: handle_conversation_turn("conversation-1234", question) ``` Sessions are created only through the sessions API. If no session matches your `session_custom_id` and anchor, the call returns a `400` error. Create the session first with `galtea.sessions.create(custom_id=...)`. If your system already emits OpenTelemetry traces, you can create production sessions without any SDK calls. See [Monitor Real User Traffic via OpenTelemetry](/sdk/tutorials/monitor-real-user-traffic-with-opentelemetry). ### Single-Turn Production Monitoring For simple, single-turn interactions, create a production session and use `galtea.inference_results.create_and_evaluate()` to log and evaluate the interaction in a single call. You can pass [Specification](/concepts/product/specification) IDs so the API resolves their linked metrics automatically (recommended), or list metrics explicitly. ```python theme={"system"} # In your application's request handler... def handle_user_query_with_specifications(user_query: str, retrieval_context: str | None = None) -> str: # Your logic to get a response from your model model_response = your_product_function(user_query, retrieval_context) # Log and evaluate the interaction, resolving metrics from your specifications session = galtea.sessions.create(version_id=VERSION_ID, is_production=True) galtea.inference_results.create_and_evaluate( session_id=session.id, input=user_query, output=model_response, retrieval_context=retrieval_context, specification_ids=specification_ids, ) return model_response # Test the handler handle_user_query_with_specifications("What are your business hours?", "Business hours: 9am-5pm Monday-Friday") ``` ```python theme={"system"} # In your application's request handler... def handle_user_query(user_query: str, retrieval_context: str | None = None) -> str: # Your logic to get a response from your model model_response = your_product_function(user_query, retrieval_context) # Log and evaluate the interaction in Galtea session = galtea.sessions.create(version_id=VERSION_ID, is_production=True) galtea.inference_results.create_and_evaluate( session_id=session.id, input=user_query, output=model_response, retrieval_context=retrieval_context, metrics=[ {"name": "Role Adherence"}, {"name": "Answer Relevancy"}, {"name": "Faithfulness"}, ], ) return model_response # Test the handler handle_user_query("What are your business hours?", "Business hours: 9am-5pm Monday-Friday") ``` ### Multi-Turn Production Monitoring (Conversations) For multi-turn conversations, use the session-based workflow to log the entire interaction first and then evaluate it. If your application has its own conversation ids, prefer the `session_custom_id` flow above: create the session once with a `custom_id` in step 1, then pass `session_custom_id` (plus `version_id` or `product_id`) instead of `session_id` in step 2. First, create a session at the start of the conversation. For production monitoring, make sure to set `is_production=True`. ```python theme={"system"} # Use is_production=True for real user interactions session = galtea.sessions.create( custom_id="CLIENT_PROVIDED_SESSION_ID", # Optional: a custom ID to associate this session in Galtea Platform to the one in your real application. version_id=VERSION_ID, is_production=True, ) ``` Next, log the user-assistant interactions. You can do this individually as each turn happens or in a single batch after the conversation ends. This approach is useful for logging interactions in real-time in a live application. ```python theme={"system"} def get_model_response(user_input: str) -> str: # Replace this with your actual model call model_output = f"This is a simulated response to '{user_input}'" return model_output # This would happen dynamically in your application. user_questions = [ "What are some lower-risk investment strategies?", "With age, should the investment strategy change?", "Great, thanks!", ] for question in user_questions: model_response = get_model_response(question) # Log the turn to Galtea right after it happens inference_result = galtea.inference_results.create(session_id=session.id, input=question, output=model_response) ``` If you have the entire conversation history, you can log all turns at once for efficiency. ```python theme={"system"} # The conversation must be in the standard format: a list of role/content dictionaries conversation_turns = [ {"role": "user", "content": "What are some lower-risk investment strategies?"}, { "role": "assistant", "content": "For lower-risk investments, consider diversified index funds, bonds, or Treasury securities.", }, {"role": "user", "content": "With age, should the investment strategy change?"}, { "role": "assistant", "content": "Yes, many advisors recommend shifting to more conservative investments as you approach retirement.", }, {"role": "user", "content": "Great, thanks!"}, {"role": "assistant", "content": "You're welcome!"}, ] galtea.inference_results.create_batch(session_id=session_batch.id, conversation_turns=conversation_turns) ``` Finally, once the conversation is complete and all turns are logged, you can run an evaluation on the entire session. Pass [Specification](/concepts/product/specification) IDs so the API resolves their linked metrics automatically (recommended), or list metrics explicitly. ```python theme={"system"} # Or resolve metrics from your product's specifications instead of listing them galtea.evaluations.create(session_id=session.id, specification_ids=specification_ids) ``` ```python theme={"system"} galtea.evaluations.create(session_id=session.id, metrics=METRICS_TO_EVALUATE) ``` For more details on evaluating multi-turn conversations, see the [Evaluating Conversations](/sdk/tutorials/evaluating-conversations) guide. ## Reading Monitor Results If you have set up a [Monitor](https://platform.galtea.ai/) on the Galtea dashboard to continuously evaluate a sample of your production sessions, you can pull its results with a single call: ```bash theme={"system"} curl -sS -H "Authorization: Bearer ${GALTEA_API_KEY}" \ "https://api.galtea.ai/monitors/${MONITOR_ID}/results?days=7" ``` The `days` query parameter sets the trailing window to aggregate, bucketed by day. It defaults to 7 and accepts up to 90. The response is built to give you an honest read of your product's quality, not just a bare average. Keep these ideas in mind when you interpret it: * **Each metric family gets its own score series.** A metric family is a metric and every later revision of it (for example, when you edit its judge prompt), grouped as one line of history. The response returns one series per family bound to the monitor, each with a whole-range `headline` score and one point per day bucket. * **Scores are sampling-weighted.** A monitor does not evaluate every session; it evaluates a sample to control cost. Sessions from a period when sampling was lower count for more in the average, so a period with a smaller sample does not silently pull the score toward its own noise. * **A confidence interval tells you how much to trust a score.** The confidence interval is the range that most likely contains the true score. Galtea publishes a 95% interval once a bucket has scored at least 30 sessions. Below that floor, `confidenceInterval` is `null` and `collectingData` is `true`, so you know the score is a preview, not a settled number, until more sessions land. * **The worst metric family drives the headline, never an average.** `worstMetric` is the lowest-scoring family over the whole range, not a blended score across families. Use it to see which quality dimension needs attention first. * **Markers flag context changes.** `markers.metricRevisions` marks when a bound metric family was edited, and `markers.samplingChanges` marks when the sampling rate changed. Both can explain a shift in the score series that isn't a real quality change. * **Spend shows your current billing cycle.** `spend.spentCredits` and `spend.pendingCredits` are the credits already used and reserved for evaluations still running this cycle; `spend.capCredits` is the monthly cap you configured for the monitor (`null` if you did not set one). If spend nears the cap, the monitor lowers its sampling rate to stay within budget, which is exactly what the sampling-change markers and sampling-weighted scores are designed to absorb. * **Sessions are listed, not just counted.** `sessions.scored` lists the most recently evaluated sessions (capped at 200), each with the turn count it was scored at. `sessions.waiting` lists production sessions that have not closed yet (capped at 100); a session is scored only once it closes. A session closes when it is explicitly finished, or when the product's auto-close setting closes it after inactivity. ## Next Steps Evaluate full multi-turn production conversations. Capture and analyze internal agent operations in production.