Optimizing Local Language Model Performance: 128K Context with Qwen 3.5 4B and Adaptations in Llama.cpp
At Atomic Agent, our focus centers on creating open-source AI agents optimized for local deployment. Recently, we shipped our first desktop version, prompting us to re-examine the internal architecture of our model implementations. Unlike the typical approach where agents are built around cloud-centric models with limited local support, we adopted a reverse strategy—designing our system specifically around local models, ranging from 4 billion to 9 billion parameters.
In this article, we detail the significant modifications that enabled us to manage a remarkable 128,000-token context window on a modest 800 MB of RAM, instead of the traditional 4 GB requirement. These enhancements are baked into our customized llama.cpp build and are tailored specifically to empower local models with expanded context capabilities.
Memory-Efficient KV Cache Through Compression
A key solution lies in compressing the key-value (KV) cache. Our custom llama.cpp build integrates TurboQuant—a tool that compresses the cache aggressively, set to a turbo3 compression mode by default. On a MacBook Pro M5, this setup allows a 128K chat session with Qwen 3.5 4B to occupy approximately 800 MB of RAM, a substantial reduction from the standard 4 GB footprint. While this approach introduces some lossy compression, leading to a slight decline in output quality, the trade-off significantly minimizes resource consumption, making large-context interactions feasible on consumer-grade hardware.
Maintaining Stable Prompt Prefixes to Maximize Cache Reuse
To capitalize on cache reuse, we ensure that the prompt prefix remains static throughout the session. This includes the system prompt, tool catalog, and skill list—they stay byte-identical between interactions. Dynamic elements such as memory writes or per-step updates are appended at the end of the prompt. This strategy enables llama.cpp to reuse its KV cache efficiently, drastically reducing startup latency. For instance, a response with Qwen 27B that historically needed over four minutes to initiate can now start in approximately one second during ongoing sessions. Although this improves responsiveness rather than inference speed per se, it enhances the user experience, especially during long conversations.
Externalized State Management and Memory
We shift session state—such as memory and previous tool outputs—outside the model itself. These data are stored on disk, and the prompt presents a bounded slice of this information. In local models, excessive tool outputs are trimmed to only include the most relevant content, particularly focusing on recent results like command pass/fail indicators. When conversations approach the buffer limit, the agent summarizes earlier parts to preserve important context without overwhelming the model, allowing for continuous, coherent dialogue over extended sessions.
Constraining Tool Call Grammar for Reliability
Our system constrains tool invocation syntax via llama.cpp’s grammar rules, which enforce responses to conform to a JSON array of tool calls. This ensures the model cannot produce malformed JSON or invent invalid call structures, even when operating with smaller models. Different model families, such as Qwen and Gemma, have tailored profiles to match their reasoning patterns within this grammar framework. To prevent endless reasoning loops, we cap per-step reasoning tokens at approximately 1500, ensuring efficient and predictable interactions.
Efficient Processing of Accessibility Data
In environments where visual data needs interpretation, our approach leverages the accessibility tree rather than raw images. By transmitting compressed ARIA snapshots instead of full screenshots, a 4B model can interpret web content efficiently, dealing with fewer tokens and maintaining performance.
Loop Guards to Prevent Repetitive Behavior
To mitigate a common challenge with small models—repetitive re-reading of the same file—we implement loop guards. If the system detects three identical calls in succession, it issues a warning. After five such repetitions with no new results, the process is halted, and explanations are provided. This feature helps avoid infinite loops and enhances robustness.
Hardware-Agnostic Sizing and Resource Allocation
All models are provisioned based on available machine RAM. For example, our 4B and Gemma 4 E4B models recommend at least 6 GB of RAM, while larger models like Qwen 3.6 35B-A3B demand around 24 GB. Our automatic context size adjustment ensures that on typical Macs, at least 4 GB or 25% of total RAM is allocated, with the KV cache size capped accordingly. This sizing strategy ensures stable operation across diverse hardware configurations.
User-Friendly Desktop Application with Flexible Deployment
Our desktop application bundles these enhancements into an easy-to-use package. It automatically detects and downloads models compatible with your system, manages execution locally, and prompts for user approval before any file modifications or command executions. The application supports multiple platforms, including Mac (Apple Silicon), Windows (x64 and ARM), and Linux (unsigned builds). Note that on Windows ARM devices, models currently run on CPU, given hardware constraints.
Cloud and Local Hybrid Operation
For scenarios requiring cloud assistance, our Fusion mode enables an intelligent division of labor—cloud-based planning combined with local model execution. This approach minimizes API token usage, reserving cloud interactions for strategic planning while leveraging local resources for actual inference.
Privacy, Licensing, and Community Engagement
We prioritize user privacy by ensuring that anonymous analytics are enabled by default. These metrics do not include message contents, file paths, or API keys. Users can disable data collection anytime via the privacy settings.
Our project is licensed under MIT, and source code is available on GitHub: https://github.com/AtomicBot-ai/atomic-agent.
Final Thoughts: Context Length in Daily Practice
While our setup supports a 128K token context on machines with 16 GB RAM, real-world usage varies significantly. We invite community feedback on typical context sizes in daily interactions. Understanding actual usage patterns will help further refine these configurations and optimize local AI deployment for everyone.
This article reflects our ongoing efforts at Atomic Agent to enhance local AI capabilities, making high-context, resource-efficient models accessible to a broader audience.
