How We Validate Our Own Evaluators

Version 1.0 — Published 2026-07-10

Infrarails AI Labs Private Limited

This document explains the methodology behind Infrarails's evaluation engine — not a headline accuracy number. We publish per-evaluator reliability, hold pre-registered pass bars even on a near-miss, and disclose where our own detectors fall short. Aggregate accuracy claims stay out of scope here by design (see "What this document is not," below) — this is about how we know what we know, and what we do when the answer is uncomfortable.

Why methodology, not a headline number

Most vendors in this category publish a single accuracy figure and stop there. We think that number is close to meaningless without knowing how it was produced — on what data, held against what bar, decided before or after the result was in. So this document describes the discipline itself: the gate process, the per-evaluator reliability scheme, and two worked examples of us publishing a negative finding about our own engine.

1. The gate discipline: we pre-register the bar before we see the result

Before running a benchmark against a new engine version, we commit to a numeric pass bar — AUC, false-pass count, direction of travel — in writing, before the run. If the result misses the bar, we hold it. We do not revise the bar after seeing the score.

A concrete example. In one benchmark cycle, our third gate run scored AUC 0.812 against a pre-registered bar of 0.79 — the AUC criterion passed — but the same run surfaced exactly one false-pass out of 90 held-out cases (a benign, empty-response message mislabeled as a hard case in the source benchmark). Our own pre-registered bar required zero false-passes. We held the gate as not passed, on a single borderline case, rather than relitigate the bar after the fact. The engine only cleared the bar two runs later, once an independent review of the flagged case's label — not a threshold tweak — justified a re-run. The full trajectory across four gate attempts: AUC 0.784 → 0.786 → 0.812 → 0.822 (re-score) / 0.797 (fresh independent replay). Every iteration fixed what it targeted; nothing was patched by loosening the bar.

This is the pattern we want documented: the goalposts don't move after the ball is in the air.

2. Reliability is per-evaluator, not per-platform

We don't publish one number for "how accurate is Infrarails." Accuracy is per-evaluator. Each evaluator is validated independently and, where we have a real benchmark row for it, published with precision, recall, false-positive rate, and false-negative rate. Where an evaluator does not yet have its own measured benchmark, our internal dashboards show a "validation in progress" placeholder — never a fabricated percentage standing in for missing data.

Every evaluator is also assigned a reliability tier, reflecting the honest maturity of its detection approach — not a marketing grade:

These class-level figures describe the typical evidentiary basis for each category, not a live per-evaluator score — we do not publish an invented per-evaluator statistic on any customer-facing surface. As real benchmark rows accumulate for individual evaluators, they replace the class-level tier with a measured value.

3. Honesty commitments: what "verified" actually means

An evaluator only carries a "verified" quality status in our internal registry once it has passed a real, evidence-based check — not once it has shipped. Evaluators without that evidence are labeled "stub" and excluded from live scoring until they clear the bar. This label is deliberately conservative: our own internal review process has, more than once, found evaluators that were marked verified without adequate supporting evidence and downgraded them back to stub rather than let an unproven signal count toward a customer-facing score.

We apply the same discipline to negative findings about our own engine. Two examples:

• The false-pass case above (§1) — held publicly, not quietly patched over, as the gate's own governing record.
• Guard panel disclosure. Our engine-level verdict false-positive rate runs in the 12–30% range on our internal benchmark. Layering an independent, Apache-licensed guard panel on top of the engine projects to roughly 0.4% panel-level false-positive rate on the public XSTest benchmark when both guards in the panel agree — but on CPU-only deployments that panel can time out under load and fall back to the engine's own baseline rate. We disclose this trade-off rather than quote only the more favorable number.

4. Where this stands today, honestly

We describe our capability across three tiers and are explicit about which one this document is describing:

• Unified measurement + governance + compliance platform — this is where the methodology above is most mature and load-bearing today.
• Runtime behavioral control (intercepting and modifying AI interactions inline, not just flagging them after the fact) — the enforcement mechanics are real and code-shipped, but our own detectors currently fail on an estimated 74–89% of safe, novel content when used as an inline blocking gate. That false-block rate is the reason we do not yet recommend blocking-grade enforcement at that detector reliability, and it's the same honest number we hold ourselves to internally before considering this tier production-ready.
• Universal authority across every AI interaction, every industry — not something we claim today. No customer base, no independent audits, and no standards-body participation exist yet to support that claim, and we say so plainly rather than imply otherwise.

What this document is not

This document does not publish a headline accuracy percentage for the platform, and it is not a substitute for one. We consider a bare accuracy claim without a stated dataset, layer, and baseline to be close to meaningless, and we hold ourselves to that same bar internally before it ever reaches an external audience. The only sanctioned external accuracy statement we make today is a hedged, cited range on our own benchmark (§3 above) — this document does not change or extend that.

Questions about this methodology, or requests for evaluator-specific reliability data as part of a due-diligence process, can be directed to the Infrarails team.