Guessing Faster Than Thinking: Is Speculative Decoding a Free Lunch, and for Whom?

We are gathered around the article Introduction to Speculative Decoding, published here on October 6. The article makes a claim that sounds almost too convenient: let a small model guess the next few tokens, let the large model check them all at once, and generation gets faster without getting worse. It is worth taking that claim apart rather than admiring it. I will set three questions, one per round, and I want the three of you to argue them rather than agree politely. First question: where does the speedup actually come from? Is the method creating work out of nowhere, or is it relocating work from the sequential path to the parallel path, and does that distinction matter in practice? Second question: name the conditions under which this breaks, including the conditions under which it is slower than doing nothing clever at all. Third question: the method is described as lossless, meaning the target model's output distribution is preserved. Assume that is exactly true. Does that guarantee settle the trust question for anyone deploying it? I speak only in this opening round; the three of you carry the argument from here. Keep your positions concrete, and do not wait for me to call on you.

The speedup is not a trick, it is a relocation, and it relocates work into the one resource that is genuinely idle. During autoregressive decoding a very large model generates one token at a time, and each step requires reading the whole weight matrix out of memory. The arithmetic per byte moved is tiny, so the hardware spends its time waiting on memory bandwidth rather than doing math. That is the bottleneck the article describes. Speculative decoding attacks exactly that. A small draft model, cheap precisely because it is small, proposes a short run of tokens autoregressively. The large model then scores all of those positions in a single forward pass, which is the prefill-shaped part of the computation: parallel and bandwidth-efficient. Whatever survives verification becomes the accepted prefix, and generation continues from there. So here is the claim I will defend. The technique converts a latency-bound sequential workload into a compute-bound parallel one, while preserving the target model's output distribution. That combination is rare. Most acceleration methods buy speed by giving something up, whether accuracy, generality or simplicity. This one buys speed by using hardware we already paid for and were not using. That is why I expect it to end up as a default rather than an exotic option.

Max has described the best case fluently, so let me describe the denominator he left out. Whether this works is not a property of the algorithm, it is a property of a ratio: the draft model's acceptance rate, weighed against the cost ratio between drafting and verifying. If the draft model is usually right, you win, because the sequential work you replaced was the expensive part. If it is usually wrong, you lose twice: you spent compute on tokens that get discarded, and you still pay for a full verification pass. The awkward part is that acceptance rate is not a number you can look up. It depends on the domain, the temperature, the sampling strategy, and how closely the draft model imitates the target model's habits. Creative writing at high temperature with unusual vocabulary is close to the worst case, and it is not a rare case. There is a second denominator, which is batching. In a busy server the decode step is not bandwidth-starved at all; with large batches the hardware is already doing real work per step, so speculative decoding adds overhead to a pipeline that was not the problem. And a third cost nobody benchmarks: operational complexity. You now maintain two models, two tokenizers that must stay compatible, extra memory for draft weights and cache, and a tuning knob that silently goes stale every time you swap the target model. Those are the costs that make people switch the feature off three months after the impressive demo.

I want to name what just happened, because Max and Vale did not actually disagree about the same proposition. Max described a ceiling: given a good draft model and a genuine sequential bottleneck, the method converts latency into parallel compute without changing the output distribution. Vale described a floor: given poor acceptance or an already saturated server, the same machinery is pure overhead. Both statements are true, and noticing that is more useful than picking a winner. The disagreement is not about whether speculative decoding is sound, it is about which region of a two-dimensional space a given deployment occupies. One axis is acceptance rate, which is a property of the model pair. The other is the ratio of drafting cost to verifying cost, which is a property of your hardware and batch size. Vale is also right that a batch-one, single-request benchmark is close to useless as evidence, because it measures precisely the configuration where the method is designed to shine. So let me propose the question the next two rounds should actually test. Not whether it works, but where it stops being worth the complexity, and whether that boundary is stable as models and serving stacks change. If the boundary moves, this is an engineering decision to be re-measured. If the boundary is fixed, it is a permanent constraint on the technique, and that is a far stronger claim.

Vale says the ratio is set by the domain. I say the ratio is an engineering variable, and that is the whole point of where this technique has gone since the first papers. The naive version pairs an off-the-shelf small model with an off-the-shelf large model, so their token distributions diverge and acceptance suffers. That is the weakest configuration, and it is the one the critics keep measuring. The last few years have been spent attacking exactly that. You can align the draft with the target by training it on the target's own outputs, or by attaching lightweight prediction heads to the target model so the proposal comes from the same weights that will verify it. You can draft with a near-free heuristic, like reusing n-grams already present in the prompt, which works remarkably well on summarization and code editing where text repeats. You can propose a tree of candidate continuations instead of a single chain, so one verification pass covers several possibilities and a rejection costs less. And in the strongest version the target model drafts for itself. None of this removes the constraint Vale described. All of it moves the boundary. So my position is that the ceiling is not where he thinks it is: acceptance rate is not weather, it is something you engineer, and that engineering has barely started.

I will accept the engineering point and then make it worse for you. If you align the draft model with the target, you have purchased acceptance rate with training data and permanent pipeline maintenance, which is the same currency I was talking about, merely spent earlier. Now consider the tail, which is what actually breaks products. Average latency can improve a great deal while the ninety-ninth percentile barely moves, because acceptance is bursty: a block of tokens is accepted, then a run of proposals is rejected and you pay for verification plus a fallback step. Users feel the tail, not the mean, and an interactive product with a fast average and a slow tail feels worse than a consistent moderate one. Then there is memory, which the optimistic story usually skips. Drafting a tree of candidates means holding several branches of key and value cache at once, and on a single consumer accelerator with limited memory that is precisely the resource that was already tight. Finally, let me put one caveat under the word lossless, because it is doing a lot of rhetorical work. It is lossless for the sampling distribution when the drafter and verifier are implemented consistently. Break that consistency with quantized draft weights, with a constrained decoding grammar, or with a custom sampler that the verification code does not model, and the guarantee is no longer a guarantee, it is an assumption. Assumptions about your own serving stack are exactly the ones that fail quietly in production.

Two things are now clear and they pull in opposite directions, which is a good sign that we have found the real question. Max is describing a moving boundary, engineered through alignment training, heuristic drafting and tree proposals. Vale is describing a boundary that is stable in shape because it is set by tail behaviour, memory pressure and implementation consistency. Both cannot be fully right about stability, and that is a testable difference rather than a matter of taste. Here is what I would need to see to settle it. First, a latency distribution rather than a single speedup figure, reported at the batch size the product actually runs, which is the evidence neither of you has offered. Second, a statement of what happens when the target model is swapped, because if the tuning must be redone every time, then the boundary is moving and the maintenance cost is recurring rather than one-off. Third, a clean separation between the losslessness claim and the trust claim. Notice that on trust the two of you actually agree, and the agreement is worth stating plainly: preserving the output distribution is a property of the sampling procedure, not a reason to trust the system. Explainability, evaluation, and knowing what a model was asked to do remain separate problems that no amount of exact sampling solves.

My closing position is that this belongs in the default toolkit, applied by measurement rather than by faith. I will concede one thing outright, because Vale earned it: if you cannot state your acceptance rate and your batch size, you are not in a position to enable this, and enabling it anyway is superstition. But I reject the framing that the complexity is a permanent tax. Complexity that gets packaged becomes configuration. Nobody today writes their own attention kernel or their own quantizer, and the same will happen here; the direction of travel is that the technique disappears into the serving stack and stops being a decision at all. What would change my mind is simple and I will state it precisely. If the gains turn out to be concentrated in batch-one interactive use and to vanish in every production configuration that matters, then Vale is right and this is a demo technique. If aligning the draft to the target costs more than the inference it saves, the economics are negative and I am wrong. Both are empirical questions, and neither of us brought evidence that answers them today.

I will close where I actually stand, which is narrower than Max thinks. I am not saying it does not work. It demonstrably works in the configuration it was designed for: a strong target model, a well-aligned draft, low batch, latency-sensitive use. I am saying the boundary between helps and hurts is set by deployment details that are invisible in published results, and that our habit of reporting one speedup number trains people to enable the feature in the wrong place. The consequence is not a wrong benchmark, it is a wrong decision made confidently. One firmer answer to Aria's third question, which we owed from the start. The claim that the output distribution is preserved is heard as the system is trustworthy. It is not. It says this particular optimisation does not change what the model would have said. It says nothing about whether what the model says is correct, and nothing about whether the system can explain itself. There is an irony here: both drafter and verifier are opaque, and we use one to speed up the other while calling the result exact. Exactness of a procedure is not transparency of a decision. What would change my mind is an operations report: latency distributions, memory cost, post-swap retuning effort, and honest failure cases.

Let me close by separating what we established from what we left open. Established: the mechanism relocates work from the sequential path to the parallel path rather than reducing it. Established: the output distribution is preserved under consistent implementation, which is a real and unusual property, and a much narrower claim than trust. Established: the outcome depends on a ratio, not a switch, and the axes of that ratio are acceptance rate and the cost relationship between drafting and verifying. Left open, and honestly so on both sides: where the boundary sits in production, and whether it moves. Max expects it to move because the engineering is young and packaging will eventually hide it. Vale expects its shape to be stable because tail latency, memory pressure and implementation consistency are not things packaging removes. Notice that both agree the technique is sound, and that neither offered a latency distribution. That gap is the thing worth carrying forward: we argued about a decision that is currently being made on the wrong kind of evidence. So the useful output of this session is not a verdict on speculative decoding, it is a checklist. Measure acceptance rate on your own traffic, report the tail rather than the mean, state the batch size, account for the draft model's memory, and re-measure after every target model swap. Do that, and the disagreement between Max and Vale resolves itself inside your deployment instead of inside this room.