Property Finder, a MENA real-estate app, moved a vision workload to Amazon Bedrock and found its evaluation labels were the obstacle. Of its labelled samples, 96.8% had been inherited from the incumbent vision-language model's own predictions rather than verified by humans, so every alternative was scored against the model it was meant to replace. Errors were hidden inside those labels: character-level mistakes in watermark text were marked correct, and single-label photo classification forced one room type onto images that showed several. Working with the AWS Generative AI Innovation Center, the team hand-annotated 356 UAE images, held 230 Egypt images as a cross-regional check, used the first 256 UAE images as a development set, and switched overlapping attributes to multi-label output. Micro F1 was the primary metric, reported with macro F1, per-label precision and recall, and per-sample IoU. Prompts were evolved from v0 to v27 in a loop modelled on GEPA, with DSPy's MIPROv2 applied for v27. Each model was then tuned on its own prompt: Llama 4 Maverick 17B Instruct moved from 0.75 with the shared v21 prompt to 0.81 on the development set, against 0.55 for the previous proprietary model on its tuned v22 prompt. On the 356-image evaluation set the two scored 0.80 and 0.56, a 47% relative gain on the development set comparison. Estimated cost fell to $2.35 per 1,000 images, 86% lower, with 40-60% lower inference latency reported from production monitoring. The authors note the evaluation set contains the development images, so the scores are optimistic, and that 356 images can only detect moderate differences between models.
Our takeThe transferable finding is the circularity, not the model swap: labels built from production predictions reward whatever resembles the incumbent, and the same pattern appears in LLM-as-judge pipelines and in regression suites seeded from current output. Two caveats matter. The reported scores are tuned on 256 of the 356 evaluation images, so the gap is an upper bound, and 356 images resolve only moderate differences. Annotation came from one expert plus AWS review, with no inter-annotator agreement figure.