Amazon published a hands-on guide on 23 October for Multi-Turn Reinforcement Learning on SageMaker AI. The headline is a jump from 61 to 84 percent on held-out Wordle words for gpt-oss-20b, for about $339 in training tokens. The more instructive number is 37, and why it sits beside 84 is the part worth reading first.
What the pipeline optimises
A SageMaker training job owns the policy model and a LoRA adapter. Each rollout is one training row, and each row holds one thing: a secret word. The service invokes an agent on Bedrock AgentCore through a Lambda forwarder; the agent runs a Strands loop, calls the policy model through a SageMaker Job Runtime endpoint speaking the OpenAI chat API, plays up to six guesses through one guess-word tool, and returns a single scalar reward when the episode ends. GRPO uses the spread of rewards across eight rollouts of the same word to update the adapter.
The dataset carries no target trajectory and no step labels: 640 training words in five batches of 128, plus 128 held-out words with no overlap. Correctness is specified only by a scoring function the customer writes. For a task where nobody can write the right transcript, you supply a way to score the outcome and let the optimiser find the path.
Two scores from one adapter
The training job scores its own validation at temperature 0, where greedy decoding makes curves comparable between runs. That curve peaked at 48 percent at step 90 and finished at 37. Evaluated at serving settings, temperature 1.0, the same weights reach 84 percent at medium reasoning effort against 61 for the base model. Same adapter, same 128 words, a 47-point spread depending on where you look.
That is not a reporting bug but a property of the task class. Greedy decoding collapses a search problem into a fixed policy: after seeing underscore-A-S-T-E the model must weigh PASTE against WASTE, and temperature 0 commits to the highest-logit continuation and never revisits it. In single-turn benchmarks that determinism is a feature. In a multi-turn task where the model's own output becomes the next input, it measures the tendency to walk into a dead end and stay there.
The published cells size it precisely. At the low effort the model trained at, the base model solves 15 percent and the trained model 69 percent with sampling on, while the same trained weights score 37 percent at temperature 0. Decoding alone is worth 32 points, more than the 23-point training gain the post leads with.
Where the gain holds up
The strongest result is not the headline pair but its robustness: the adapter adds 14 to 54 points in all four configurations tested, low and medium effort crossed with temperature 0 and 1.0. Whatever training taught transfers to settings it was not trained under.
The magnitudes need care. Wordle is a teaching task frontier models already play; the intended use is a small model on an internal workflow public models never saw. And 128 held-out words is a small sample. The binomial arithmetic gives a standard error near 4.3 points at a 61 percent rate and 3.2 at 84, so the 23-point gap sits about four standard errors out and is not noise. The two medium-effort sampling runs, 80 and 88, differ by 8 points, which is what a 128-word sample should wander.
A reward that inverted twice
The most reusable section documents a reward design that failed. The first version summed per-guess scores across the game, inherited from a single-turn project where each guess really was an independent sample. Summed into one episode reward it rewarded length: a lost game paid about +0.49, and solving on the first guess scored +0.99 against +1.61 on the third. The authors found it by reading rollout traces, not the loss curve.
The fix splits a dominant outcome term from a clipped shaping term. A solve pays 1.05 to 1.55 depending on turn, a loss after six guesses pays minus 0.5, and never producing a valid guess pays minus 1.5. Shaping, clipped to minus one to one, is weighted 0.04 on a solve and 0.15 on a loss, because adjacent solve tiers are 0.1 apart while the loss tiers are 1.0 apart. At 0.04 a full shaping swing consumes 80 percent of a solve-tier gap, leaving the ordering on about a twentieth of a reward unit of headroom. Change the clip, the turn budget or any shaping term and the inversion must be re-derived by hand.
There is a quieter mismatch. The reward spends half its range, 0.5 of 1.55, encoding how fast the solve happened, while the metric is binary pass-at-one, so the headline says nothing about speed. Shaping cannot simply be zeroed either: GRPO learns from variance inside a group, and two lost games differing in clue consistency still carry a gradient.
What the tokens bought
The cost breakdown is unusually complete: 491.6 million prefill tokens, 159.7 million sampled, 643.3 million train, across 105,486 rollouts in 12 hours 40 minutes, $338.50 at US East list prices. Train tokens dominate at $231.59, 68 percent, because the backward pass scales with trajectory length. More portable than the total: about 1,514 sampled and 6,098 train tokens per episode, roughly $0.0032 per game. Prefill accumulates because every turn re-reads the history. The AgentCore runtime adds about $79, so the platform figure is near $418.
Failure modes for the runbook
Four failures generalise beyond this stack. The default 4,096-token sampling budget ran out mid-reasoning before the first tool call often enough to be raised to the 8,192 maximum. Credential handling starved at 32 concurrent rollouts, where per-call sessions exhausted the provider and hour-long token caching produced 403s as credentials rotated. A rollout that never reaches the policy model fails the whole job with a no-sampling-requests error, so the agent must always return a reward, even the floor one. And at too high a learning rate the model wrote tool calls as prose, every rollout collected the same floor reward and the gradient vanished.
Training reward plateaued near step 15 and validation near step 30, so most of the 100-step budget confirmed a plateau. Redeploying the runtime mid-job kills the job.
What it establishes
It establishes that an episode-level reward with no step labels can move a 20B open-weight model by double digits on a held-out multi-turn task, that the transfer survives four decoding configurations, and that the recipe ships as code. It does not establish that it works on the internal workflows the post targets, that more than 100 steps help, or that per-turn reward would not do better, though UpdateReward exists and this run did not use it.
The durable lesson is evaluation discipline. Measure at the settings you will serve at, publish the decoding configuration beside the number, and treat the training curve as a signal rather than a verdict. A run whose validation says 37 and whose serving evaluation says 84 is not a contradiction; it is what agentic RL looks like when the measurement protocol is treated as part of the method.
Sources
AWS Machine Learning Blog, Multi-Turn Reinforcement Learning on Amazon SageMaker AI: A hands-on guide, 23 October 2026. https://aws.amazon.com/blogs/machine-learning/multi-turn-reinforcement-learning-on-amazon-sagemaker-ai-a-hands-on-guide/
Sample code: https://github.com/aws-samples/sample-multi-turn-rl-wordle