Proof, Not Promises: Evaluation, Guardrails and Observability for Forward Deployed Engineers

Part 5 of the FDE skills series. Aater Suleman’s third pattern, earn the right to write, has a people side, covered in Part 3, and a technical side, covered here. The people side gets you a fair hearing. The technical side gives you the evidence. This is the area I work in every day through EvalTrust, so this article is the most personal in the series.

1. Why evaluation is an FDE skill, not a specialist’s job

In his talk, Aater listed poor risk controls among the main reasons AI pilots fail, and pointed out that an agent going wrong in production can cost millions. His answer to the access problem was not to argue harder. It was to show proof: run the system read-only, show what it would have done next to what people actually did, and let the evidence make the case.

That proof doesn’t produce itself. Someone has to design the tests, define what “good” means with the customer, run the comparisons, watch the system in production and explain the results in business terms. On an FDE team, that someone is often you.

There’s also a simple reason it matters more for agents than for older software: agents fail quietly. A traditional application that breaks throws an error. An agent that breaks usually returns a fluent, confident, wrong answer, and no alert fires. I wrote about this in detail in my leader’s guide to agent readiness.

---

2. The four controls: observe, evaluate, guard, gate

I organize this work into four controls. Each answers a different question a customer will ask.

---

3. Evaluate: building a golden set with the customer

Every evaluation starts with ground truth: a set of real cases where the right answer is known and agreed. This is the golden set, and building it is as much a people task as a technical one.

How to build one in an engagement:

1. Sample real cases. A few hundred, drawn from actual historical work, deliberately including the hard and unusual ones, not just the easy majority.
2. Have the customer’s experts label them. What should the system have flagged, answered or done? Use their definitions of “correct,” not yours.
3. Measure expert agreement. Have two experts label the same subset. If they only agree 85% of the time, a system that matches them 85% of the time is performing at human level, and a 99% target is unrealistic.
4. Write down the edge cases. Every disagreement is a policy question the business needs to answer. Bring them to the owner.
5. Version it. The golden set becomes the reference for every model change, prompt change and vendor switch. Keep it under version control like code.

What to measure depends on the system. A quick map:

System type	Core metrics
Classification or flagging	Precision, recall, cost-weighted value at the chosen threshold
RAG and question answering	Retrieval recall, answer correctness, groundedness (faithfulness)
Drafting and summarization	Factual accuracy, completeness, human acceptance and edit rate
Agents taking actions	Task success, tool-call accuracy, escalation correctness, safety violations, cost per task

Part 2 explains how to connect each of these to the business KPI, and why the threshold with the best F1 score is often not the best business decision. For a deeper technical treatment, see my practitioner’s guide to evaluating agentic AI.

A note on LLM judges. Using a language model to grade outputs scales evaluation enormously, but a judge is itself a model that can be wrong. Calibrate it against expert labels on part of the golden set before you trust it, and keep humans reviewing a sample over time.

---

4. Shadow mode: the safest experiment you’ll ever run

Shadow mode is Aater’s “show them A versus B” made concrete. The AI runs on real, live work, but its outputs go only to a log, never to the real system. You then compare what it would have done with what people actually did.

What a good shadow-mode report shows the customer:

• Agreement rate with human decisions, broken down by case type
• Where the AI was right and the human wasn’t, and the reverse; both matter
• Severity of disagreements: a wrong low-risk suggestion isn’t the same as a wrong high-risk one
• Safety and policy violations, which should be zero
• Cost and speed compared with the current process

Run it long enough to cover normal variation: busy weeks, quiet weeks, month-end. Then let the business owner decide whether the evidence justifies the next step.

---

5. Guard: designing guardrails that fit the risk

Guardrails are the controls that stop the system from doing harm even when it’s wrong. Match them to the risk of each action.

Layer	Examples
Input checks	Detect prompt injection, out-of-scope requests and sensitive data entering the system
Output checks	Groundedness checks, format and policy validation, blocking sensitive data from leaving
Action limits	Allowed tools only, value caps, rate limits, read-only by default
Human approval	Required for high-impact or irreversible actions, with a clear review screen
Escalation	Hand off to a person when confidence is low or the case is unusual
Reversibility	Every write can be traced and undone

A practical rule: the more irreversible the action, the more layers it needs. Drafting an email needs light checks. Changing a patient record, issuing a payment or modifying production infrastructure needs approval, limits and a full audit trail.

---

6. Observe: monitoring in production

Passing an evaluation once proves nothing about next month. Data changes, users change, vendors update models, and quality drifts without anyone noticing. Production monitoring for AI should track three kinds of signal:

1. System health: errors, latency, cost per task, tool failures. The traditional monitoring most teams already have.
2. Quality: ongoing evaluation on a sample of live traffic, user acceptance and edit rates, escalation rates, groundedness scores.
3. Business outcome: the KPI from the metric card, compared with the baseline.

Instrument agents with traces that capture every step (model calls, retrieved documents, tool calls and their results) so that when something goes wrong you can see exactly why. Open standards like OpenTelemetry make this portable across tools; I walked through a working setup in my article on agent observability with OpenTelemetry.

Set alerts on quality, not just uptime. “Evaluation pass rate dropped below the agreed threshold” should page someone just like “service is down” does.

---

7. Gate: thresholds agreed before each step

A gate is a decision point with criteria agreed in advance. Gates connect all of the above to the right-to-write ladder from Aater’s talk.

The most important word in this section is before. If you set the threshold after seeing the results, you’ll be tempted to move it. Agree on it with the owner, write it down, and treat it like a contract. That discipline is what makes a customer believe your next report.

Gates also run in the delivery pipeline. Every change to a prompt, model or tool should rerun the golden set automatically and block the release if quality drops, just as failing unit tests block a code deploy.

---

8. How to build these skills

• Build a golden set for something you already made. Even 100 hand-labeled cases will change how you think about quality.
• Run a shadow comparison. Take any process with historical decisions, run a model against it and write a one-page evidence report as if for a customer.
• Add traces to an agent and practice debugging a wrong answer from the trace alone.
• Write a gate document: the criteria, who signs off, and what happens if the system fails it.
• Read the frameworks customers will ask about, such as the NIST AI Risk Management Framework, so you can map your evidence to their language.

---

Let’s talk

How do you prove an AI system is ready to act on real data? What evidence has worked for you with skeptical stakeholders? Tell me on LinkedIn or at mohcine@evaltrust.cloud.

---

The FDE skills series

1. Think like a product manager
2. From business KPI to F1 score
3. The people stack
4. Integration and rapid prototyping
5. Evaluation, guardrails and observability (this article)
6. Security, privacy and compliance
7. Domain expertise and scoping
8. Communication and ownership

Sources and notes

• Aater Suleman, talk on forward deployed engineering, Dallas, November 5, 2026: poor risk controls as a cause of failed pilots, “earn the right to write,” and showing A-versus-B evidence instead of arguing for access.
• Example gate criteria are illustrative; real thresholds must be agreed with each customer.