Evaluating local AI coding agents through practical tasks

Local language models are increasingly capable, but raw model performance does not tell the whole story when using them as coding agents.

An agent must do more than generate plausible text. It needs to use tools correctly, inspect files, execute commands, make appropriate changes and verify its work.

This project explores how to evaluate those capabilities on local hardware through controlled, reproducible tasks.

The problem

A model can appear capable in a simple conversation but fail when asked to perform a multi-step development task. Tool-call formatting, runtime compatibility, context limits and recovery from errors can all affect the outcome.

The approach

The project is designed around repeatable scenarios, runtime and model profiles, and observable pass/fail results. It explores local inference through runtimes including llama.cpp and MLX.

The emphasis is on agentic coding capability rather than treating response speed as the sole measure of quality.

Areas of investigation

• Tool-call correctness and execution
• File inspection and code modification
• Command execution and testing
• Multi-step task completion
• Runtime compatibility
• Failure detection and recovery

What this project explores

The aim is to make local coding-agent evaluation more systematic, helping identify which combinations of model, runtime and configuration are suitable for particular development tasks.

Technical areas: Python, local LLM inference, llama.cpp, MLX, automated testing and benchmarking.