Scrolling through Hacker News while my laptop fan sounds like a jet engine, I stumbled upon a wild "Show HN": boiling down Gemini's Tool Calling into a tiny 26M parameter model. The mad lads at Cactus just open-sourced this pocket-sized beast, promising it runs blazingly fast on budget phones and wearables.
TL;DR: Honey, I Shrunk the LLM
Henry from the Cactus team just dropped Needle, and its specs are beautifully unhinged:
- Microscopic Size: Exactly 26M parameters (smaller than an average frontend dev's
node_modules folder).
- Insane Speed: Hits 6000 tok/s prefill and 1200 tok/s decode directly on consumer devices.
- The Core Insight: Tool calling is essentially a retrieval-and-assembly problem (match query -> grab arguments -> spit out JSON). It doesn't require deep, philosophical reasoning. Massive models are total overkill here.
- The "Heretical" Architecture: Simple Attention Networks. Zero MLPs or FFNs anywhere. The author argues FFN parameters are wasted space if the facts are already provided in the input (like in RAG or tool use scenarios).
- Training Montage: Pretrained on 200B tokens (27 hours on 16 TPU v6e), then post-trained for 45 minutes on 2B tokens of synthetic Gemini data.
- The Flex: It straight-up beats larger models like FunctionGemma-270M and Qwen-0.6B in single-shot tool calling.
What the HN armchair experts are saying
(While it's fresh off the press, you can already predict the dev community's split reactions)
- The Pragmatists: Thank god! Using a 70B parameter model just to turn on a smart bulb or format a JSON string is a massive waste of VRAM and electricity. Building these small ai tools is the future of mobile apps.
- The Skeptics: Dropping FFNs sounds super sus. Sure, it dominates single-shot function calling, but how does it hold up in multi-turn conversational agents with complex context?
- The Local LLM Junkies: Downloading instantly to finetune on a Mac. Doing this locally means no more renting expensive cloud vps instances just to parse some data.
The C4F Verdict: Right tool for the job
We devs love our hype trains. We often throw an entire A100 GPU cluster at a problem that could realistically be solved with Regex, or in this case, a 26M model. Needle is a beautiful reminder that architecture and specific use-cases matter more than brute size, especially for edge computing.
Stop forcing an AI to be Shakespeare when you just need a JSON parser. Kudos to Cactus for open-sourcing it. Go download the weights and play around with it, folks!
Source: Show HN: Needle: We Distilled Gemini Tool Calling into a 26M Model
