On-Device Inference with Liquid AI: 5 Easy Frameworks

On-device inference means running a language model on the phone, laptop or small board itself, without a server. Liquid AI's LFM2.5-230M is built for exactly that. This article is based on Liquid AI's official LFM2.5-230M model card and the LFM2.5-230M docs page, read on 2 November 2026, not on my own benchmark. The old version of this post described tests on a MacBook and a Jetson with YAML configs and "war stories"; none of that came from Liquid AI, and it is removed.

What LFM2.5-230M is

The model card calls LFM2.5 "a family of hybrid models designed for on-device deployment", and LFM2.5-230M the most compact one. The facts it lists:

  • 230M parameters, 14 layers: 8 double-gated convolution blocks plus 6 GQA attention blocks.
  • Training budget of 19T tokens; distilled from LFM2.5-350M and refined with multi-stage reinforcement learning.
  • Context length 32,768 tokens, vocabulary 65,536, knowledge cutoff mid-2024.
  • Ten languages, including English, Chinese, Japanese and Korean (Vietnamese is not listed).
  • License: the LFM Open License 1.0 (license_name: lfm1.0), not Apache or MIT.

Liquid AI reports decode speeds of 213 tokens per second on a Galaxy S25 Ultra and 42 tokens per second on a Raspberry Pi 5. These are the vendor's figures.

What it is for, and what it is not for

The card is direct: use it for data extraction and lightweight on-device agentic pipelines such as tool calling, and do not use it for reasoning-heavy workloads such as advanced math, code generation or creative writing. That is the most important fact for on-device inference planning. A 230M model is a fast, cheap worker for narrow jobs, not a general assistant.

Five frameworks for on-device inference with LFM2.5

On-device inference with Liquid AI LFM2.5-230M: GGUF for llama.cpp, MLX for Apple Silicon, ONNX, Transformers and vLLM
Which model repository to use with which runtime, from the model card.

1. llama.cpp on CPUs and small boards

The docs page gives a two-line start:

brew install llama.cpp
llama-cli -hf LiquidAI/LFM2.5-230M-GGUF \
  --temp 0.1 --top-k 50 --repeat-penalty 1.05

The -hf flag downloads the GGUF model from Hugging Face. The card describes the GGUF repository as quantized and optimized for edge inference and local deployment.

2. MLX on Apple Silicon

The LFM2.5-230M-MLX-8bit repository is listed as the format for fast inference on Mac devices.

3. ONNX Runtime for portability

The LFM2.5-230M-ONNX repository targets cross-platform deployment with ONNX Runtime, useful where Python is not an option.

4. Transformers for fine-tuning and testing

pip install "transformers>=5.2.0" torch accelerate

The docs page uses this install line, while the model card says the quick start is compatible with transformers>=5.0.0. Use the higher one to be safe. The model loads with AutoModelForCausalLM.from_pretrained("LiquidAI/LFM2.5-230M", device_map="auto", dtype="bfloat16"), and generation uses the recommended sampling values.

5. vLLM or SGLang when on-device is not enough

The card lists vLLM and SGLang for high-throughput GPU deployments with the native checkpoint. That is server inference, useful when the same model must handle many users.

Settings that matter for on-device inference

The card's generation parameters are temperature: 0.1, top_k: 50 and repetition_penalty: 1.05. Small models drift with higher temperatures, so start with these. The chat format is ChatML-like with <|im_start|> and <|im_end|>, and tokenizer.apply_chat_template() builds it for you. For tools, the model writes Pythonic calls between <|tool_call_start|> and <|tool_call_end|> by default; you can ask for JSON instead.

Tool calling in four steps

Tool use is the main reason to pick this model for on-device inference agents. The card describes four steps:

  1. Function definition: give the tools as a JSON list in the system prompt, or pass tools=... to tokenizer.apply_chat_template().
  2. Function call: the model answers with a Python list of calls between the tool-call tokens.
  3. Function execution: your code runs the call and returns the result with the tool role.
  4. Final answer: the model reads the tool output and answers in plain text.

The card's example asks for the status of a candidate ID, the model emits [get_candidate_status(candidate_id="12345")], and after the tool result it answers with the stage and the interview date. Your application, not the model, runs the function, so validate every argument before you execute it.

Fine-tuning for your on-device inference task

Liquid AI recommends fine-tuning LFM2.5 for a specific use case to get the best results. The card links recipes for continued pre-training and supervised fine-tuning with LoRA through Unsloth, SFT and DPO with LoRA through TRL, and GRPO with either library. For a narrow extraction job, a small fine-tuned model is often the whole point of on-device inference: it is cheap to run, and it only has to do one thing well. Start from the instruction model for quick tests, and from LFM2.5-230M-Base when the card's recipes call for a base model.

Reading the benchmarks honestly

On the card's table, LFM2.5-230M scores 71.71 on IFEval and 43.26 on BFCLv3, ahead of several larger small models listed there. On MMLU-Pro it scores 20.25, while Qwen3.5-0.8B scores 37.42, and on τ²-Bench Telecom it scores 5.26. So it follows instructions and calls functions well for its size, but its general knowledge is limited, as the card itself warns. All numbers are from Liquid AI.

Mistakes to avoid with on-device inference

  • Using a 230M model for math or code, which the card advises against.
  • Ignoring the license: check the LFM Open License before shipping a product.
  • Copying vendor speed figures as your own; measure on your hardware.
  • Expecting Vietnamese or other unlisted languages to work well.

To try a small local model on an ordinary Linux machine first, the Ollama on Linux post was tested on a CPU and shows what a tiny model can and cannot do.

FAQ

Is LFM2.5-230M a "liquid neural network"? The model card describes a hybrid of convolution and attention blocks; it does not use that phrase.

Do I need a GPU for on-device inference with this model? No. The vendor speed figures are for a phone and a Raspberry Pi 5, and llama.cpp runs on CPUs.

Can I fine-tune it? Yes. The card recommends fine-tuning and links Unsloth and TRL recipes for SFT, DPO and GRPO, and a base model, LFM2.5-230M-Base.

Which runtime is best for on-device inference? llama.cpp with GGUF for most CPUs and boards, MLX on Macs, ONNX where you need portability.

Comments

Popular posts from this blog

How to Play Minecraft Bedrock Edition on Linux: A Comprehensive Guide for Tech Professionals

Linux desktop frozen? My 5 go-to tricks to try - before forcing a hard reboot

The Ultimate Guide: How to Set Up DXVK in Wine on Linux for Enhanced Gaming Performance