Maya Malamud

For a while, the instinct was to put a "cool" agent responsible for every task. Then the bills and latency came in. Most of that work could've been handled far more cheaply by classic ML.

That's why production teams are shifting to a three-tier cascade: deterministic logic, classic ML triage, and confidence-gated LLM escalation. I'll show this architecture, then how to upgrade it.

Most teams stop at escalating to the LLM only when needed. The real improvement is a teacher-student loop where LLM output on hard cases retrains the classic model beneath it. The LLM's job isn't just to answer what the cheap tier can't; it ensures the cheap tier needs it less next time.

I'll also cover the trap teams fall into: treating the confidence threshold like a technical setting rather than an economic cost-and-risk trade-off that can build or break user trust.

You'll leave with a blueprint for this hybrid cascade and a pattern for using LLMs to teach your models, not just do their work.

Background: Women's acute myocardial infarction (heart attack) is frequently misdiagnosed as anxiety at intake, delaying treatment. These errors can become embedded in clinical datasets, teaching downstream models the same incorrect label as ground truth.

Objectives: To identify structural label error, a systematic mislabeling pattern tied to patient subgroups, in intake diagnoses for female cardiac patients, and to develop a method for detecting and correcting it.

Methods: Using a large clinical database, I identified female patients whose confirmed heart attack diagnoses contradicted their intake label. I applied natural language processing (NLP) to clinical notes to trace the discrepancy to its origin, then developed a three-step correction approach: bias-aware annotation, disaggregated validation, and a conflict layer flagging disagreement between label and evidence.

Results: Clinical notes showed cardiac signal was present from intake in the miscoded cases, indicating the error originated in labeling rather than in the clinical presentation itself. This pattern was consistent enough to function as a false ground truth for any model trained on it.

Conclusions: Structural label error in women's cardiac diagnosis is systematic, not random, and can be identified and corrected using the proposed annotation approach. The same method may apply to other clinical domains where training labels carry embedded diagnostic bias.

Fetal growth monitoring today relies mostly on population reference curves, which can miss meaningful changes in an individual fetus's growth trajectory even when measurements stay within the accepted range. I'll present a proof-of-concept project that estimates an individualized growth trajectory for each fetus from early ultrasound measurements and monitors later measurements against it, using ideas adapted from Statistical Process Control (SPC), a method originally developed for industrial process monitoring.

The approach builds on earlier published research in Quality Engineering, and Dr. Diamanta Benson, Head of the Industrial Engineering and Management Program at The Open University of Israel, and I are now extending it as co-investigators with Beilinson Hospital's Women's and Maternity Hospital, using richer longitudinal data and current statistical and machine learning methods. I'll talk about what it actually takes to make an idea like this usable in a real clinical system, and the main obstacles along the way.

Building a custom eval metric is easy now; every platform lets you do it. The harder part, and the one that actually matters, is knowing which metric will serve your real business KPI.

I call this the Failure-to-Metric method. Start from a real production failure, not a benchmark. Decide if it's even worth a dedicated metric. If it is, design it around the moment things went wrong, not just where they ended up: did the system refuse when it should have, and how costly was the answer when it didn't.

I'll walk through how to recognize which failures actually matter and connect them to the KPI they're quietly damaging, using examples from high-stakes clinical workflows, where a wrong answer isn't just a bad user experience, it's a patient safety issue.

Then I'll show two metrics built this way. A Refusal Rate metric, scoring whether the system correctly declines to answer when it should (higher is better). A Hallucination Specificity Score, scoring how dangerous the answer was when it didn't refuse (lower is better).

You'll leave with the method, two working examples, and a way to decide which of your own production failures deserve a metric.