Evaluating Gen AI Applications - A Practitioner’s Playbook for Testing What AI Actually Gets Right

di Srinivas Bommena (Autore)

Srini, 2026

Most Gen AI applications do not fail with an error message. They fail while sounding convincing. This practitioner's playbook shows engineering, product, testing, risk and governance teams how to build evaluation systems that expose those failures before they reach users, auditors or regulators. Using the running Meridian case, the book takes you from the first evaluation test to a governed production operating model. It shows how to create versioned datasets, evaluation rubrics, traces, regression suites, human-review workflows, release gates, model-change decisions, red-team evidence and audit-ready release records. Inside the Second Edition Build an evaluation suite from business requirements, failure modes and protected behaviours. Evaluate RAG pipelines, agentic workflows, multimodal applications, prompts, code and reasoning. Govern model migrations and fine-tuning changes using non-compensatory release gates. Apply uncertainty analysis, retrieval metrics, pass@k, human calibration and defensible thresholds. Test MCP and tool-use boundaries, memory failures, prompt injection, provenance and adversarial behaviour. Connect evaluation evidence to OWASP, MITRE ATLAS, C2PA, the EU AI Act and production governance. Practise the methods through seven interactive companion applications built around realistic Meridian scenarios. This is not a prompt book or a catalogue of benchmark scores. It is an engineering guide to deciding what to test, what evidence to retain, when to stop a release and how to keep evaluation effective after deployment. Written for AI engineers, software testers, architects, platform teams, product leaders, risk professionals and AI-governance practitioners responsible for Gen AI systems in production. Create evaluation datasets from incidents, production traces, adversarial examples, edge cases, expert review, and holdout sets. Use observability to monitor drift, regressions, latency, cost, model changes, and production behaviour. Build structured human-in-the-loop evaluation using rubrics, calibration, adjudication, and reviewer agreement. Apply red teaming to prompt injection, jailbreaks, prompt leakage, data exfiltration, and tool misuse. Turn evaluation results into governance evidence for audits, executive oversight, compliance, and release decisions. This book is written for AI platform teams, software engineers, ML engineers, architects, product owners, risk teams, governance leaders, and technology executives responsible for shipping Gen AI systems that must be trusted in production. The future of Gen AI will not be won by teams that merely generate impressive outputs. It will be won by teams that can prove their systems are accurate, grounded, observable, secure, cost-aware, and operating within clearly defined boundaries. Evaluating Gen AI Applications gives you the practical Safety and Validation Engineering framework to build that proof.

• Autore: Srinivas Bommena
• Anno edizione: 2026
• Editore: Srini
• Formato: EPUB3 con DRMFREE
• Lingua: Inglese
• Compatibilità: Tutti i dispositivi (eccetto Kindle)
• Layout: Reflowable
• EAN: 9798235886612