Eval Systems & Quality Gates

How to know the AI got better or worse before users find out

Master execution-based evaluation, visual regression, regression detection, LLM-as-judge with human calibration, eval infrastructure as a product, drift detection, and pre-deployment gates.

Modules in This Level

5.1 Why Code-Gen Evals Are Uniquely Hard

Understand why AI code generation cannot use traditional assertions — non-determinism requires semantic evaluation — and map the eight-level correctness spectrum from syntax parsing to aesthetic judgment that defines the full eval infrastructure Make must build.

5.2 Execution-Based Evaluation

Walk the five-step execution evaluation ladder — AST parsing, TypeScript build, runtime startup, headless render, functional test — and understand why each catches a failure class the previous step cannot, so skipping any rung leaves a category of broken apps shipping to users.

5.3 Visual Regression Testing

Evaluate AI-generated UI with pixel-diff screenshot comparison, design fidelity metrics, and LLM-as-visual-judge — leveraging the Figma frame as ground truth for a design fidelity score that competitors without design context cannot replicate.

5.4 Regression Detection Across Releases

Learn how stratified eval sets — by framework, task type, complexity, and integration — surface category-level regressions that overall pass rates hide, and why explicit go/no-go gates with absolute minimums are the only defense against silent quality degradation across releases.

5.5 LLM-as-Judge and Human-in-the-Loop

Learn to use LLM-as-judge for code quality, accessibility, and design fidelity — what rubric design with scored anchors looks like versus vague prompts — and where judges fail and human review must prevent the generator's blind spots from becoming the judge's blind spots.

5.6 Eval Infrastructure as a Product

Design the eval system as a product with distinct users — agent, platform, product team, and leadership — each needing different views, and trace the pipeline from trigger through execution ladder, LLM judge, screenshot comparison, regression detection, and go/no-go gate.

5.7 The Eval-Driven Development Loop

Use eval failure analysis as product strategy input — tracing a 30% modification failure rate to a context retrieval gap, or a 20% deployment failure to a build/runtime split — so each failure category maps to a specific investment hypothesis, not a vague quality complaint.

5.8 Level 5 Checkpoint

Apply Level 5 eval knowledge to a leadership presentation scenario: outline an end-to-end quality assurance strategy for Make that ensures code quality improves with every release and never silently regresses, covering evaluation tiers, regression gates, and drift detection.

5.9 Model Drift Detection

Understand why coding agents are vulnerable to silent drift — model version changes and prompt distribution shift degrade design token consistency and component decomposition without triggering errors — and learn the monitoring signals that catch drift before users notice.

5.10 Pre-Deployment Gates

Learn why eval passage is not sufficient for production readiness, and master the pre-deployment gates — canary benchmarks, contamination detection, shadow traffic validation, and staged rollout criteria — that close the gap between benchmark scores and real user experience.