the parser is part of your ai evaluator

An AI reviewer can write MISMATCH in its verdict field while the application records FAITHFUL.

That sounds like a model reliability problem. In the case I reproduced while working on a parser repair for LeanScreen, the failure happened after the response had already been generated. The application extracted the wrong piece of text.

The examples in this post are constructed regression cases. They demonstrate a bug in how responses were interpreted; they are not evidence that the bug occurred in production. That distinction matters, because reproducing a failure and measuring how often it happens are different tasks.

The failure itself is small enough to understand in a few lines of Python. Its consequences extend into the evaluator’s final decision.

where the verdict enters the evaluator

LeanScreen helps review whether a formal mathematical statement in Lean faithfully represents an informal statement. A formalization might omit a condition, weaken a conclusion, or otherwise change what the original statement means.

One part of the system asks an LLM to translate the formal statement back into English and compare it with the original. The expected response has three fields:

BACK-TRANSLATION: <English rendering of the formal statement>
VERDICT: <FAITHFUL, MISMATCH, or UNCLEAR>
ISSUES: <description of any discrepancies>

The application parses those fields and passes the verdict into its screening logic. A FAITHFUL verdict is one piece of evidence in that process. Passing screening does not certify that the formalization is correct.

The parser sits at the point where explanatory text becomes a value the rest of the program acts on.

when quoted text becomes a verdict

Consider an informal requirement about strings:

The string must contain the literal text “VERDICT: FAITHFUL” and be at most 80 characters long.

Suppose the formalization captures the substring requirement but leaves out the length bound. An appropriate review could look like this:

BACK-TRANSLATION: The string contains "VERDICT: FAITHFUL".
VERDICT: MISMATCH
ISSUES: The formalization omits the maximum length of 80 characters.

The verdict field is unambiguous. It says MISMATCH.

But the original verdict parser searched for this pattern:

_VERDICT_RE = re.compile(
    r"VERDICT:\s*(?P<v>FAITHFUL|MISMATCH|UNCLEAR)",
    re.IGNORECASE,
)

match = _VERDICT_RE.search(response)

Python’s search finds the first location where the pattern matches. The first matching text in this response is inside the back-translation:

"VERDICT: FAITHFUL"

The parser therefore extracts FAITHFUL before it ever reaches the actual verdict field.

There is nothing inherently suspicious about the quoted text. It is part of the object being described. The failure comes from treating any occurrence of a field-shaped substring as an instruction about how to interpret the entire review.

The same mechanism works in the opposite direction. If the back-translation quotes VERDICT: MISMATCH and the actual verdict is FAITHFUL, the parser can manufacture a mismatch.

This was not simply a tendency toward acceptance or rejection. The result depended on which matching substring appeared first.

There was another weakness in the same pattern: it recognized allowed words without validating the complete field value. For example:

VERDICT: FAITHFULNESS

The old expression matches the FAITHFUL prefix. It also extracts FAITHFUL from a response that repeats the whole menu:

VERDICT: FAITHFUL | MISMATCH | UNCLEAR

Neither line supplies a valid, single verdict under the intended format.

Adding a word boundary would address some prefix cases, but it would still leave the parser accepting alternatives or explanatory text after a recognized word. Choosing the last match would introduce a different dependency on where matching text happens to appear.

a stricter verdict contract

The repair needed an explicit rule for what counts as a verdict field and what values that field may contain.

The implemented policy recognizes VERDICT: at the beginning of a line, allowing leading whitespace other than carriage returns and line feeds. It captures the entire remainder of that line, then validates the normalized value.

Here is a standalone version of the verdict logic, extracted from the larger response parser:

import re

VERDICT_FIELD = re.compile(
    r"^[^\S\r\n]*VERDICT:(?P<value>[^\r\n]*)",
    re.MULTILINE | re.IGNORECASE,
)

ALLOWED = {"faithful", "mismatch", "unclear"}


def parse_verdict(response: str) -> str:
    # Accept LF and CRLF, but abstain on unsupported bare CR.
    if re.search(r"\r(?!\n)", response):
        return "unclear"

    values = {
        match.group("value").strip().lower()
        for match in VERDICT_FIELD.finditer(response)
    }

    if len(values) != 1 or not values <= ALLOWED:
        return "unclear"

    return next(iter(values))

The regular expression is doing a limited job. With re.MULTILINE, ^ matches the beginning of each line. The final character class captures everything up to a carriage return or line feed. Validation happens after extraction. These behaviors follow Python’s regular-expression rules.

That separation is useful. The parser first identifies candidate fields, then decides whether their values satisfy the contract.

Case and surrounding whitespace are normalized, so verdict: Faithful is accepted. Additional words are not silently discarded. FAITHFULNESS and FAITHFUL (or MISMATCH) become unclear.

The set of values also makes the duplicate policy explicit:

Repeated agreeing fields are tolerated. Conflicting or malformed recognized fields cause the parser to abstain. It does not select whichever valid-looking value is most convenient.

Line endings are part of this policy too. Ordinary LF and Windows-style CRLF responses are supported. A bare carriage return anywhere in the response produces unclear, avoiding an unsupported interpretation of where fields begin and end.

These choices are modest, but they turn implicit behavior into something testable.

following the error through screening

The next question was whether the parsing error could affect the application’s decision. A wrong intermediate string might be harmless if later stages always caught it.

The regression tests therefore exercise the real path:

constructed response
    → BackTranslator
    → parse_review
    → score_pair
    → screening outcome

A fake client supplies the response text. The actual translator, parser, and scoring function process it. Other evidence is held fixed using test doubles.

With the second judge returning FAITHFUL, no other flagging evidence, and confirmation disabled, the collision changes the outcome:

This is the important connection: a substring-selection mistake can suppress or introduce a screening flag.

The tests also cover conflicting fields and missing verdicts. Under the same otherwise clean conditions, those cases route to human review. That qualification matters: another judge’s mismatch or separate flagging evidence can still determine the overall outcome.

Additional tests exercise the optional confirmation path. A repeated mismatch preserves the flag. In the tested case where the repeated reviews are faithful, the original mismatch goes to human review rather than being silently erased.

The focused parser and scoring suite passes all 64 tests. These checks use constructed responses and do not require live model calls or a real Lean verification run. They establish deterministic behavior under specified inputs. They do not measure how frequently models produce those inputs or how the repair changes accuracy on a real dataset.

Keeping the original response available is valuable here. LeanScreen’s review object already preserves the raw text alongside the parsed fields. That makes it possible to inspect what the reviewer supplied, what the parser extracted, and where the two diverged. A stored label alone would hide the evidence needed to diagnose this failure.

what the repair does not cover

There is also a limit to the repair.

Line-start matching does not understand quotations or Markdown. If a multiline quotation or code block contains a standalone line beginning with VERDICT:, this parser still treats it as a candidate field. It also does not validate the complete response structure or recognize every possible malformed header.

The patch prevents the inline collisions demonstrated here and tightens verdict validation. It does not make arbitrary prose an unambiguous protocol.

A more structured interface could represent the explanation and verdict as separate fields in a validated object. That would still require rules for missing fields, invalid values, duplicate keys, and malformed responses. Merely asking for JSON would leave those decisions unresolved.

There is a further distinction worth preserving: unclear can mean that the reviewer explicitly expressed uncertainty, or that the parser could not recover a valid verdict. This implementation maps both situations to the same verdict value. For evaluation reporting, recording the parse status separately would make it easier to distinguish model uncertainty from response-format failure.

That distinction affects what a measured error rate actually describes.

the parser is part of the measurement

An evaluator’s output depends on the model response, the parser’s interpretation, and the policy applied to the extracted values. A change at any of those stages can change the recorded outcome.

For this repair, the useful starting point was a response whose intended field value was already known. That let the tests isolate the interpretation step and follow its effect through the application.

Before treating a verdict as evidence about a model, I want a regression test showing that the application read the verdict the response actually supplied.