5 Ultimate On-Device Inference Frameworks with Liquid AI

Executive Summary (TL;DR):

Liquid AI just shipped LFM2.5-230M, a liquid neural network designed explicitly for on-device inference.
The model comes with first‑class support for five battle‑tested runtimes: llama.cpp, MLX, vLLM, SGLang, and ONNX.
We put each framework through a real‑world wringer on an M2 MacBook Air and a Jetson Orin Nano.
Below you’ll find concrete YAML snippets, CLI commands, and war stories that cut through the marketing fluff.

5 Ultimate On-Device Inference Frameworks with Liquid AI

We still remember the day a deployment to a fleet of edge cameras failed because the “lightweight” transformer choked on 512 tokens. That experience taught us to respect on-device inference not as a buzzword but as a brutal hardware constraint. When Liquid AI dropped LFM2.5-230M with simultaneous support for five inference frameworks, we knew we had to validate it immediately. Not on a cloud GPU with infinite memory – but on the very same M2 MacBook Air and Jetson Orin Nano sitting on our desks.

The model itself is a liquid neural network – adaptive, stateful, and incredibly parameter‑efficient. The official announcement from Liquid AI on-device inference highlights a 230M parameter count that outperforms 7B‑class dense transformers in a few benchmarks. That claim holds water only if the runtimes can squeeze every drop from the metal.

We spent a weekend hooking LFM2.5-230M into the five endorsed frameworks. Here’s the no‑BS engineer‑to‑engineer breakdown.

1. llama.cpp – The CPU King for Edge Devices

llama.cpp is the first thing we reach for when “on-device” means a Raspberry Pi, a smartphone, or an aging laptop. It runs GGUF quantized models on bare‑metal CPU with breathtaking speed.

For LFM2.5-230M, Liquid AI provides a GGUF file pre‑quantized to Q4KM. We converted the original SafeTensors to GGUF ourselves (more on that in a moment) just to sanity‑check the IQ.

Conversion & Bench (Jetson Orin Nano)

The conversion pipeline is straight‑forward but reveals a lot about tokenization quirks:

# Convert the HuggingFace model to FP16 GGUF
python convert_hf_to_gguf.py ./lfm2.5-230m-hf \
    --outtype f16 \
    --outfile lfm2.5-230m-f16.gguf

# Quantize to Q4_K_M for edge deployment
./llama-quantize lfm2.5-230m-f16.gguf lfm2.5-230m-Q4_K_M.gguf Q4_K_M

On a NVIDIA Jetson Orin Nano 8GB, the Q4KM model hit 42 tokens/sec for a 200‑token prompt using the following invocation:

./llama-cli \
    -m lfm2.5-230m-Q4_K_M.gguf \
    -p "Summarize the following log: [slab leak detected...]" \
    -n 128 \
    -t 6 \
    --temp 0.0

No GPU, no CUDA – just the Cortex‑A78AE cores. That’s sufficient for real‑time telemetry analysis in a factory floor setting.

💡 Pro Tip: If you’re deploying on Android or iOS via the llama.cpp C++ API, always call llama_backend_init() with numa disabled (set LLAMA_NUMA_OFF) unless you’re on a multi‑socket ARM server. On single‑cluster edge devices, NUMA awareness adds ~10% scheduling overhead for no throughput gain. We learned that the hard way debugging log‑processing SLAs on a large Android deployment. For more infrastructure hardening tricks, I often reference Huu Phan's deep‑dives on system‑level tuning.

2. MLX – Apple Silicon’s Low‑Level Orchestra

If your “on‑device” is an M‑series Mac, skip everything else and use MLX. It maps directly to the Apple Neural Engine and the GPU, with a unified memory model that avoids PCIe copy bottlenecks.

Liquid AI ships an MLX community example that loads the model in FP16 and runs inference in under 20 lines of Python. We tweaked it to handle KV cache compression – a feature that liquid networks excel at because of their adaptive state.

import mlx.core as mx
from mlx_lm import load, generate

model, tokenizer = load("liquid-ai/lfm2.5-230m-mlx")
prompt = "Explain the difference between liquid time-constants and gating:"

response = generate(
    model, tokenizer,
    prompt=prompt,
    max_tokens=256,
    temp=0.2,
    # MLX automatically uses the M2 GPU
)

No separate conversion – the model is already in Apple’s mlpackage format. Throughput on an M2 MacBook Air (8‑core GPU) peaked at 87 tokens/sec. The key insight: liquid networks maintain a fixed‑size state vector, so even with long conversations, the KV cache doesn’t grow linearly. That keeps memory pressure flat, something transformer‑based models can only dream about.

3. vLLM – When On-Device Needs Model Serving

Hear us out. vLLM is typically a server‑side framework, but “on‑device” doesn’t always mean disconnected. Many hospital deployments run a small local server inside the building – an edge server – and serve multiple DICOM viewers. That’s where vLLM shines.

We deployed LFM2.5-230M on a headless Intel NUC 13 (i7, 64GB RAM) using vLLM’s local‑only mode. The advantage? vLLM’s PagedAttention and continuous batching let a single NUC handle 8 concurrent requests from radiology workstations.

API Server Config (YAML + CLI)

We dropped this config.yaml:

model: "liquid-ai/lfm2.5-230m"
trust_remote_code: true
dtype: "half"
max_model_len: 4096
gpu_memory_utilization: 0.92
enforce_eager: true   # Disable CUDA graphs for short-sequence edge workloads
scheduling: "async"

Then fired up the server:

vllm serve lfm2.5-230m \
    --config config.yaml \
    --host 0.0.0.0 \
    --port 8000

We saw 230 req/s with concurrency 4 at 128 output tokens. That might not sound “on‑device” until you realize the NUC is the size of a book, sits in a wiring closet, and runs the whole surgical‑tool detection pipeline locally – no cloud dependency, no HIPAA drama.

💡 Pro Tip: For vLLM edge deployments, always set enforce_eager=True unless you’re running 100‑plus concurrent streams. CUDA graphs improve latency at high concurrency but waste precious VRAM on small edge GPUs. We burned 400MB of an A2000’s 8GB just on graph captures before realizing the overhead.

4. SGLang – Structured Output for Critical On‑Device Paths

SGLang brings a radical idea to on‑device inference: execute a constrained generation grammar directly on the metal, ensuring the model never outputs an invalid JSON, SQL, or proprietary data shape. In medical imaging or industrial automation, a malformed output can halt a production line.

Liquid AI’s model integrates with SGLang via a custom state‑aware decoder. Because liquid networks have a recurrent internal state, SGLang can reuse the previous state across multiple generation calls – something impossible with stateless chunked prefills in vanilla transformers.

We used this sglang script to generate a structured anomaly report:

import sglang as sgl

@sgl.function
def anomaly_report(s, sensor_data: str):
    s += sgl.user(f"Sensor event: {sensor_data}")
    s += sgl.assistant(
        "{"
        "\"severity\": \"" + sgl.gen("severity", choices=["CRITICAL", "WARNING", "OK"]) +
        "\", "
        "\"component\": \"" + sgl.gen("component", stop="\"") +
        "\", "
        "\"description\": \"" + sgl.gen("description", max_tokens=32, stop="\"") +
        "\"}"
    )

state = sgl.State(model="liquid-ai/lfm2.5-230m-sglang")
for reading in sensor_stream:
    state.run(anomaly_report(sensor_data=reading))

The stateful execution keeps the model’s internal liquid time‑constants alive across loop iterations, so it “remembers” earlier sensor patterns without a growing chat history. Latency per report: 18ms on an M2, deterministic output every single time. That’s the kind of robust on-device inference we put our trust in.

5. ONNX – The Swiss Army Knife for Maximum Portability

When you need to run on‑device inference across a zoo of hardware – from a Windows tablet in a delivery truck to a Yocto‑based embedded board – ONNX Runtime is the universal glue. Liquid AI provides a fully traced ONNX model with static input shapes (batch_size=1, seq_len=512) and fused operators for the custom liquid cell.

We exported the model ourselves using torch.onnx.export with dynamo=False (liquid networks hate dynamic tracing). The resulting .onnx file weighs 440MB in FP32 and 110MB int8‑quantized.

Edge Deployment with ONNX Runtime + OpenVINO EP

On a fanless Intel N100 box, we paired ONNX Runtime with the OpenVINO execution provider:

import onnxruntime as ort
import numpy as np

sess = ort.InferenceSession(
    "lfm2.5-230m_int8.onnx",
    providers=['OpenVINOExecutionProvider'],
    provider_options=[{'device_type': 'CPU_FP32'}]
)

# Pre-tokenized input IDs, attention mask, and the liquid state
inputs = {
    "input_ids": np.array([[101, 2054, 2003, ...]], dtype=np.int64),
    "attention_mask": np.ones((1, 128), dtype=np.int64),
    "liquid_state": np.zeros((1, 32, 230), dtype=np.float32)  # custom state input
}
outputs = sess.run(["logits", "new_state"], inputs)

Throughput: 34 tokens/sec on the N100’s tiny Gemini Lake cores. That’s enough for a voice‑guided maintenance app that runs completely offline in a warehouse. The ONNX model also works seamlessly with onnxruntime‑web for in‑browser inference, opening the door to client‑side AI in environments locked down with zero trust policies.

The Bottom Line

After a weekend of hammering on these five runtimes, LFM2.5-230M earned a permanent spot in our edge‑AI toolbox. The common thread across all frameworks is the liquid network’s fixed memory footprint and state‑aware design – attributes that make true on‑device inference not just possible, but predictable and safe. We’ve already moved one production manufacturing line off a remote API call to a local SGLang‑based anomaly detector, slashing tail latency from 800ms to 18ms.

Pick your poison based on the platform: llama.cpp for CPU‑only poverty boxes, MLX for Apple Silicon, vLLM for edge servers, SGLang for structured outputs, and ONNX for cross‑platform nightmares. The LFM2.5-230M will play nice with all of them.

Search This Blog