Alibaba Cloud's Qoder team described on 2 November how its Cloud Agents product runs a continuous evaluation loop. An evaluation case is defined as inputs, attachments and success criteria; the agent under test is a complete configuration; the trace is the stream of message events, tool calls and tool results. Checks are split three ways: code-based evaluators for anything a rule can decide, model-based evaluators for subjective quality, and a second agent that opens the deliverables when the result is a file or a change to the environment. Success criteria are introduced only after execution finishes and the environment is frozen, so the agent cannot read the answer key. Each case runs in its own sandbox session, repeated runs are reported as pass@k and pass^k, proposed changes are applied to a copy and compared with the pre-change baseline, and production sessions are converted into new evaluation cases.
Our takeThe design choice worth noting is that a correct answer counts for little on its own. Qoder splits grading into answer, process and artifacts, and pushes every question it can toward a deterministic evaluator before involving a model, which is a cheap way to avoid rewarding lucky guesses and false task-complete claims. What the post does not give is a number: no baseline results, no measured catch rate for regressions. So the loop is sound engineering with unquantified value. The next question is whether such business-specific case sets stay private: a question bank built from your own traffic