Making local model integrations more reliable

Small and medium-sized language models can produce a tool call as ordinary text instead of the structured response expected by an AI coding agent.

When that happens, a model may appear to have selected a tool without the client actually receiving an executable tool call.

This project investigates a middleware layer between OpenAI-compatible clients and locally served language models, with the aim of repairing or normalising incompatible tool-call responses.

The problem

Different models and inference servers can behave differently when handling tool calls. Some responses contain recognisable tool-call syntax but fail to conform to the structured format expected by the client.

The approach

The middleware is designed to inspect and normalise responses, handle supported malformed tool-call formats, and address compatibility differences between clients and local inference servers.

The work also considers streaming responses, malformed conversation history and safeguards against repeated identical prompts.

Why it matters

Local AI development requires more than running a model. Reliable integration depends on the behaviour of the entire chain: model, inference server, middleware and client.

This project explores whether a focused compatibility layer can make that chain more predictable without replacing the underlying model or coding agent.

Technical areas: Python, FastAPI, OpenAI-compatible APIs, streaming responses, local LLMs and integration testing.