On 2 November, Alibaba Cloud's Qoder team published a description of the evaluation loop behind Qoder Cloud Agents. The notable part is not that an agent platform can run tests. It is the unit of evidence the design settles on: a run is judged on its answer, its execution trace and its artifacts, and the success criteria are withheld until the run is over and the environment frozen. That ordering tells you more about the engineering than any benchmark table, especially since the post includes no benchmark table at all.
The vocabulary is worth stating precisely, because most agent "evaluation" stops earlier. A case is inputs, attachments and success criteria. The agent under test is a complete configuration, not a model name. The trace is the stream of message events, tool calls and tool results. The deliverables are the final response plus generated files and environment changes. A baseline is the same suite run before a change, kept for comparison rather than recollection.
Grading is split by the kind of question being asked. Anything a rule can decide goes to a code-based evaluator. Subjective quality goes to a model-based evaluator. When the result is a file or an environment change, a second agent opens the deliverables and inspects them. The three checks are answer, process and artifacts, and the process check explicitly asks whether the necessary steps happened, whether permissions were crossed, whether the route was needlessly long, and whether a conclusion was asserted without supporting evidence.
The most important detail is procedural: success criteria and check logic are introduced only after execution finishes and the environment is frozen. Without that seal, an evaluation measures how well an agent follows a rubric it can read, not how well it does the task. Sealing the answer key is cheap and removes an entire class of false confidence, which is rare in a field where evaluation harnesses often share a workspace with the thing being evaluated.
Deterministic-first is the other sound choice. Rule-decidable checks are cheap, repeatable and auditable; model graders add variance, cost and their own biases. Using a model only where judgement is unavoidable keeps the suite's noise budget lower than a design that asks a language model to grade everything. The same logic explains the third evaluator: a file that must be opened is not a text question, and an agent inspecting the artifact is a more direct check than asking the first agent whether it wrote one.
Repeated runs are reported as pass@k and pass^k. The distinction is not cosmetic. pass@k answers "can it ever do this", which is the right question for capability. pass^k answers "does it do this every time", which is the right question for reliability. An agent that succeeds three times out of five can look acceptable as a 60% pass rate while remaining unusable for a task that must not fail. Reporting both is correct; collapsing them into one average would hide exactly the variance the loop exists to expose.
Changes are applied to a clone and compared against the stored baseline, while production keeps running. That is the difference between a demo and a regression process. It also makes the acceptance rule explicit: fixing the previously failing cases is not enough; the full suite has to improve on target metrics without material regressions elsewhere. Anyone who has tuned a prompt until one failing example passes, and quietly broken two others, recognises the rule.
Production sessions are then converted into new cases. This is where the design becomes a data-asset argument: the case bank grows with the business, and models and harnesses can be swapped while the cases stay. It is also where the weakest assumptions live.
First, a case bank derived from production drifts toward incidents that have already happened. That is useful for regression, but it is not coverage. The post does not describe how cases are retired, deduplicated or sampled to represent the long tail, so the suite can look mature while measuring a narrow slice of traffic.
Second, case quality depends on how the session is turned into a case. If the recorded trace, attachments and environment are reused carelessly, the case can carry hints from the original run. The sealed-criteria rule protects the grading moment, not the case-construction moment. Keeping the case inputs clean is a separate discipline, and it is the one most likely to be skipped when teams are in a hurry.
Third, tuning against a fixed suite invites overfitting. The post recommends comparing a change with the same data, dependencies and environment, which controls variance but also means the suite is the same one being optimised against. A held-out set, rotated periodically, would make the improvement claim stronger without changing the loop's architecture.
Fourth, the cost model is left open. pass^k multiplies runs by k, every case gets an isolated sandbox, and model-based or agent-based graders consume tokens on top. The post names cost as one of three tensions but gives no per-case economics. For a team deciding whether to run this nightly or on every commit, that number is the decision.
The treatment of online assessment is honest and worth repeating. Offline evaluation has predefined criteria and can be scored. Online sessions have no reference answer, so only anomaly checks are possible; trace efficiency, tool-call accuracy and temporal accuracy are signals, not scores. That is the correct framing, and it also means continuous improvement still depends on someone interpreting those signals and turning them into cases.
What the post does not provide is any measurement of the loop itself. There is no baseline result, no catch rate for seeded regressions, no false-pass rate against human review, no stability figure across versions. The claims are architectural: the loop should reduce lucky passes and false completion claims. That is plausible. It is not yet evidence.
The missing numbers are not a reason to dismiss the design; they are the natural next audit. A team adopting this pattern can generate them locally: seed ten known regressions and measure how many the suite catches; compare pass and fail decisions with human review on a sample; track pass^k across releases; record cost per case. Those four numbers would say more about whether the loop works than any description of its components.
For practitioners, the transferable part is the ordering. Write cases from real failures before writing a benchmark harness. Freeze the environment before grading. Push every question toward a deterministic checker. Keep the full trace, not just the final answer. Version the case set and the baseline together. Report both pass@k and pass^k. Treat online metrics as detectors and keep a human or agent in the loop that converts them into cases.
Agent evaluation is turning into an evidence pipeline: case, environment, trace, artifact, baseline, verdict. Qoder's post is a reasonably careful description of such a pipeline. The remaining question is the one the pipeline exists to answer: when a change ships, how would anyone know it made the agent better? A loop that cannot be audited at that level is still an impression, just a more expensive one.