Why I test coding agents on tasks, not just responses

A coding model can produce convincing code and still fail to complete a real development task.

When the model operates through an agent, success also depends on whether it can select tools, provide correctly structured arguments, inspect the repository, execute commands and respond to errors.

That makes evaluation more complicated than asking a model a question and judging the resulting text.

I am interested in testing these capabilities through controlled tasks with explicit success criteria. Rather than assuming that a model's reputation translates directly into useful agent behaviour, I want to observe what happens when it interacts with actual development tools.

Local hardware adds another dimension. Model size, quantisation, available memory, runtime behaviour and context limits all affect which configurations are practical.

The useful question is not simply which model is best. It is which model and runtime can reliably complete the tasks that matter within the available constraints.

That is the motivation behind my experiments with local coding agents, tool-call reliability and repeatable benchmarks.

What I want to measure

A useful evaluation should make it possible to inspect more than the final answer. It should record whether the agent selected and called tools correctly, whether its changes passed relevant checks, and where a task failed.

Repeatable scenarios also make comparisons more meaningful: the same task can be run across different models and runtimes, with results recorded against explicit success criteria.

The goal is not to produce a universal ranking. It is to understand which configurations are dependable for particular development tasks on the hardware available.