Three numbers sit within six tenths of a point in AWS's new agent-distillation walkthrough: the distilled 4B student at 0.541 Absolute Success Rate, the 40B-active teacher decoding greedily at 0.540, and an unnamed medium-sized proprietary model at 0.546. The post's headline is the first, framed as a 4B model recovering essentially all of its frontier teacher's quality. The framing holds. What makes the result worth reading is not the recovery but the fact that the training set was filtered by the same scorer that later grades the student, which explains both why the student lands where it does and what it cannot do.
The Pipeline, Stripped of Framing
ShoppingBench publishes 750 synthesized shopping problems, 250 each for product, shop and voucher intents, each with a hidden gold answer and an explicit constraint set. AWS splits them once, by fixed index, into 426 training problems and 324 held-out ones. A teacher — GLM-5.3 on Bedrock, reasoning on, temperature 1.0 — plays the agent over a 2.7M-product catalog, sampled twice per problem to yield 850 trajectories; the benchmark's own scorer then keeps only the runs that solve the task. Each surviving episode becomes one row per step: conversation so far in the user turn, the teacher's reasoning plus tool call in the assistant turn, loss on the assistant turn alone. About 2,600 training and 130 validation rows survive, split by problem rather than by row, because two samples per problem and several steps per sample leave rows sharing near-identical context. Two details are worth copying: every row is rendered with the student's serving chat template, and tool observations are trimmed to the fields the next decision needs rather than letting the 16,384-token limit cut rows arbitrarily.
The Comparison Set Does the Work
The table is a bracket: base 0.297, distilled 0.541, teacher sampled at 0.511, teacher greedy at 0.540, medium frontier model 0.546. Reporting the teacher twice is the honest move, and the post explains why — greedy answers what one attempt gets you, sampling answers what the data looked like. It also means "matched its teacher" is a claim against the weaker of the two teacher settings. Against the greedy teacher and an independent frontier model, the student is a tie, not a win.
A tie is the right reading. On 324 problems a proportion near 0.54 carries a standard error of about 2.8 points, so an unpaired 95% interval spans roughly five points either way: 0.541, 0.540 and 0.546 are one measurement. The post says as much — within run-to-run noise, single run, no seeds — but that caveat should also discipline the phrase "recovers essentially all". It describes aggregates, not per-intent reliability, and it cannot separate the student from the models beside it.
Rejection Sampling Changes the Target
The filter is load-bearing. Keeping only solved runs trains on a conditional distribution — trajectories given success — not on the teacher's policy. A student imitating a best-of-two slice of a teacher that solves roughly half the problems should be expected to beat that teacher's single sampled attempt; 0.541 against 0.511 is the filter showing through, not new capability. The post is candid about this and credits the effect. The symmetric consequence it leaves implicit: the student never sees a failed trajectory, so recovery from its own errors was never in its training distribution.
One Function, Two Jobs
ShoppingBench's scorer filtered the data and also grades the evaluation. The held-out problems are disjoint by index, so this is not example leakage — the split discipline is better than most replications manage. But the objective is not independent either: the student is selected toward behaviour the scorer rewards, and a blind spot in the scorer stays invisible in ASR instead of being punished by it. "Meets every stated constraint" is clean and checkable, and it says nothing about relevance beyond the constraints, query efficiency, or how many steps the agent spent getting there.
The Ablation That Travels
The most reusable finding costs nothing to apply. Training the same student on the same trajectories with the teacher's reasoning stripped — bare tool calls only — drops average ASR from 0.541 to 0.415. Both students clear the 0.297 base, so roughly half the gain comes from imitating the harness's action grammar and query policy, and the other half from the reasoning between calls. The gain holds on all three intents, which is what keeps it from looking like a market-basket coincidence.
Where the Method Stops
Sequence-level distillation plateaus on the hardest problems, and the post names the mechanism: in a multi-step loop an early error compounds because the student never sees the states its own mistakes create. It also names the diagnostic that matters in practice — traces lengthen while accuracy stays flat, well-formed tool calls that re-search with reworded queries until the step limit. The sharpest sentence in the post is that the gap which decides success is between the task and the student, not between the student and the teacher: a 4B model absorbs a teacher ten times larger when the task sits inside its reach, and the same recipe over a harder task produces fluent failure. On-policy training is the stated next stage; this walkthrough stops at stage one.
Two Numbers That Do Not Line Up
Training caps conversations at 16,384 tokens; the endpoint serves the same model at max_model_len 131,072. The deployment accepts context the student never saw in training, which is convenient and misleading as a capability claim — past the training budget the policy is extrapolating. The adapter-on-shared-base pattern is genuinely good: base weights load once, the student attaches as a second inference component against a pinned DJL LMI and vLLM build, so base-versus-student comparison becomes a routing decision instead of a second deployment. It also means the hourly cost is paid whether or not traffic arrives.
The Cost Claim Left Open
The introduction argues that an agent loop bills the frontier model on every step of every task, while a hosted student bills by the hour regardless of volume. That is the right shape of argument, and the post then declines to publish per-task cost or latency, telling readers to measure against their own traffic, sequence lengths and concurrency. Refusing a headline number is more restrained than vendor posts usually manage. It also leaves the return unquantified, and the arithmetic is unforgiving at low volume: a fixed endpoint beats a per-token API only above the traffic level where API spend per hour exceeds instance cost, so utilisation rather than model size decides which side of the line a deployment sits on. Add the training run, the evaluation jobs and the stored trajectories, and the honest unit is cost per solved task on your distribution.
What Travels
Split by problem, not by row. Keep the reasoning in the targets. Report the teacher at both decoding settings. Compare against an independent frontier model, not only the untrained base the student is bound to beat. Treat general ability as a regression check — IFEval prompt-strict falls 0.819 to 0.741 and instruction-strict 0.872 to 0.813, reported rather than hidden. And when the aggregate looks like a tie, believe the interval rather than the third decimal.
Sources
AWS Machine Learning Blog, "Build efficient agents with knowledge distillation on Amazon SageMaker AI", 2 November 2026. https://aws.amazon.com/blogs/machine-learning/build-efficient-agents-with-knowledge-distillation-on-amazon-sagemaker-ai/