On-Device Inference with Liquid AI: 5 Easy Frameworks
On-device inference means running a language model on the phone, laptop or small board itself, without a server. Liquid AI's LFM2.5-230M is built for exactly that. This article is based on Liquid AI's official LFM2.5-230M model card and the LFM2.5-230M docs page, read on 2 November 2026, not on my own benchmark. The old version of this post described tests on a MacBook and a Jetson with YAML configs and "war stories"; none of that came from Liquid AI, and it is removed.
Contents
- What LFM2.5-230M is
- What it is for, and what it is not for
- Five frameworks for on-device inference with LFM2.5
- 1. llama.cpp on CPUs and small boards
- 2. MLX on Apple Silicon
- 3. ONNX Runtime for portability
- 4. Transformers for fine-tuning and testing
- 5. vLLM or SGLang when on-device is not enough
- Settings that matter for on-device inference
- Tool calling in four steps
- Fine-tuning for your on-device inference task
- Reading the benchmarks honestly
- Mistakes to avoid with on-device inference
- FAQ
What LFM2.5-230M is
The model card calls LFM2.5 "a family of hybrid models designed for on-device deployment", and LFM2.5-230M the most compact one. The facts it lists:
- 230M parameters, 14 layers: 8 double-gated convolution blocks plus 6 GQA attention blocks.
- Training budget of 19T tokens; distilled from LFM2.5-350M and refined with multi-stage reinforcement learning.
- Context length 32,768 tokens, vocabulary 65,536, knowledge cutoff mid-2024.
- Ten languages, including English, Chinese, Japanese and Korean (Vietnamese is not listed).
- License: the LFM Open License 1.0 (
license_name: lfm1.0), not Apache or MIT.
Liquid AI reports decode speeds of 213 tokens per second on a Galaxy S25 Ultra and 42 tokens per second on a Raspberry Pi 5. These are the vendor's figures.
What it is for, and what it is not for
The card is direct: use it for data extraction and lightweight on-device agentic pipelines such as tool calling, and do not use it for reasoning-heavy workloads such as advanced math, code generation or creative writing. That is the most important fact for on-device inference planning. A 230M model is a fast, cheap worker for narrow jobs, not a general assistant.
Five frameworks for on-device inference with LFM2.5

1. llama.cpp on CPUs and small boards
The docs page gives a two-line start:
brew install llama.cpp
llama-cli -hf LiquidAI/LFM2.5-230M-GGUF \
--temp 0.1 --top-k 50 --repeat-penalty 1.05
The -hf flag downloads the GGUF model from Hugging Face. The card describes the GGUF repository as quantized and optimized for edge inference and local deployment.
2. MLX on Apple Silicon
The LFM2.5-230M-MLX-8bit repository is listed as the format for fast inference on Mac devices.
3. ONNX Runtime for portability
The LFM2.5-230M-ONNX repository targets cross-platform deployment with ONNX Runtime, useful where Python is not an option.
4. Transformers for fine-tuning and testing
pip install "transformers>=5.2.0" torch accelerate
The docs page uses this install line, while the model card says the quick start is compatible with transformers>=5.0.0. Use the higher one to be safe. The model loads with AutoModelForCausalLM.from_pretrained("LiquidAI/LFM2.5-230M", device_map="auto", dtype="bfloat16"), and generation uses the recommended sampling values.
5. vLLM or SGLang when on-device is not enough
The card lists vLLM and SGLang for high-throughput GPU deployments with the native checkpoint. That is server inference, useful when the same model must handle many users.
Settings that matter for on-device inference
The card's generation parameters are temperature: 0.1, top_k: 50 and repetition_penalty: 1.05. Small models drift with higher temperatures, so start with these. The chat format is ChatML-like with <|im_start|> and <|im_end|>, and tokenizer.apply_chat_template() builds it for you. For tools, the model writes Pythonic calls between <|tool_call_start|> and <|tool_call_end|> by default; you can ask for JSON instead.
Tool calling in four steps
Tool use is the main reason to pick this model for on-device inference agents. The card describes four steps:
- Function definition: give the tools as a JSON list in the system prompt, or pass
tools=...totokenizer.apply_chat_template(). - Function call: the model answers with a Python list of calls between the tool-call tokens.
- Function execution: your code runs the call and returns the result with the
toolrole. - Final answer: the model reads the tool output and answers in plain text.
The card's example asks for the status of a candidate ID, the model emits [get_candidate_status(candidate_id="12345")], and after the tool result it answers with the stage and the interview date. Your application, not the model, runs the function, so validate every argument before you execute it.
Fine-tuning for your on-device inference task
Liquid AI recommends fine-tuning LFM2.5 for a specific use case to get the best results. The card links recipes for continued pre-training and supervised fine-tuning with LoRA through Unsloth, SFT and DPO with LoRA through TRL, and GRPO with either library. For a narrow extraction job, a small fine-tuned model is often the whole point of on-device inference: it is cheap to run, and it only has to do one thing well. Start from the instruction model for quick tests, and from LFM2.5-230M-Base when the card's recipes call for a base model.
Reading the benchmarks honestly
On the card's table, LFM2.5-230M scores 71.71 on IFEval and 43.26 on BFCLv3, ahead of several larger small models listed there. On MMLU-Pro it scores 20.25, while Qwen3.5-0.8B scores 37.42, and on τ²-Bench Telecom it scores 5.26. So it follows instructions and calls functions well for its size, but its general knowledge is limited, as the card itself warns. All numbers are from Liquid AI.
Mistakes to avoid with on-device inference
- Using a 230M model for math or code, which the card advises against.
- Ignoring the license: check the LFM Open License before shipping a product.
- Copying vendor speed figures as your own; measure on your hardware.
- Expecting Vietnamese or other unlisted languages to work well.
To try a small local model on an ordinary Linux machine first, the Ollama on Linux post was tested on a CPU and shows what a tiny model can and cannot do.
FAQ
Is LFM2.5-230M a "liquid neural network"? The model card describes a hybrid of convolution and attention blocks; it does not use that phrase.
Do I need a GPU for on-device inference with this model? No. The vendor speed figures are for a phone and a Raspberry Pi 5, and llama.cpp runs on CPUs.
Can I fine-tune it? Yes. The card recommends fine-tuning and links Unsloth and TRL recipes for SFT, DPO and GRPO, and a base model, LFM2.5-230M-Base.
Which runtime is best for on-device inference? llama.cpp with GGUF for most CPUs and boards, MLX on Macs, ONNX where you need portability.
Comments
Post a Comment