Skip to main content
The JSON Field Match metric is one of the Deterministic Metric options in Galtea. It performs a field-level comparison between two JSON objects, giving a more granular and interpretable score than text-based matching when the expected output is structured JSON.
If your use case involves entity extraction where accents or capitalization may vary (e.g., “Sí” vs “SI”), consider JSON Field Match (Normalized) instead.

​
Evaluation Parameters

To compute the JSON Field Match metric, the following parameters are required:
  • actual_output: The JSON output generated by the model (string or object). Supports lenient parsing — JSON wrapped in markdown code fences or embedded in surrounding text is automatically extracted.
  • expected_output: The reference JSON object to compare against. Must be a valid JSON object.

​
How Is It Calculated?

The metric compares top-level fields between the expected and actual JSON outputs:
  1. Parse Inputs expected_output is parsed as a strict JSON object. actual_output is parsed leniently — the metric will attempt to extract a JSON object from markdown code fences (e.g., ```json ... ```) or surrounding text before parsing. The two sides behave differently when parsing fails:
    • actual_output is not a JSON object (for example, your pipeline answered in plain text, or returned nothing): the evaluation is skipped, and its error message says the output could not be read. Nothing broke, so there is no score to report. Retrying the evaluation re-scores the same stored output and skips again, so fix the pipeline and send a new inference result.
    • expected_output is not a JSON object: the evaluation fails. This is an error in your test case, and you have to fix the dataset.
  2. Compare Fields For each top-level key in the expected output, check whether the same key exists in the actual output with an equal value. Comparison uses strict equality — type mismatches (e.g., string "30" vs number 30) count as non-matching. Nested objects are compared by deep equality.
  3. Compute Score The score is calculated as:
    Where total_fields is the number of top-level keys in the expected output, and non_matching_fields is the count of keys that are missing or have different values in the actual output.

​
Interpretation of Scores

  • 1.0 — All expected fields are present in the actual output with matching values. Also returned when the expected output is an empty JSON object {}, provided the actual output is readable JSON.
  • 0.5 — Half of the expected fields match.
  • 0.0 — No fields match.
Extra keys in the actual output that are not in the expected output are ignored. Missing keys in the actual output count as non-matching.

​
Suggested Test Case Types

Use JSON Field Match when evaluating:
  • Structured extraction tasks where the model outputs a JSON document with specific fields.
  • Form-filling or slot-filling agents where each field must match independently.
  • API response validation where the expected output is a known JSON structure.
  • Golden dataset evaluations where partial credit is useful (e.g., 3 out of 4 fields correct).