按硬件与负载选择执行路径
结合模型结构、设备资源和请求长度,实测运行配置并保存调优结果。
ADAPTIVE INFERENCE · OPEN SOURCE
自研、开源的 Rust 大模型推理引擎。根据硬件、模型与任务负载选择执行路径,减少运行开销与重复计算,为长会话和智能体任务保留可复用的状态。

01 / PRODUCT CAPABILITIES
结合模型结构、设备资源和请求长度,实测运行配置并保存调优结果。
直接调用后端算子。支持的 Metal 路径通过 megakernel 合并 GPU 调度,缩短执行热路径。
相同模型和兼容缓存配置下,复用匹配的提示词前缀与 KV 页面,减少重复预填充和缓存副本。
保存 KV 与循环状态,通过保留的检查点恢复、回退和分支,减少从头处理历史的需要。
用分页、驻留上限和磁盘淘汰管理状态,控制长会话与反复工具调用积累的缓存。
提供聊天补全接口,分别处理思考内容、可见文本与工具调用,便于应用接收和呈现。
02 / HOW IT WORKS
模型层描述架构和状态,后端提供算子。通过实测选择适用配置,结果绑定模型、硬件与版本。
兼容聊天接口 · 工具调用输出
模型工作流 · 配置选择 · 状态管理
Metal / CUDA / CPU
共享状态:匹配前缀 / KV 页面 / 检查点 / 驻留预算 · 支持范围随配置确认
RESEARCH / ADASPARK
一次多猜几个 token,并不一定更快。候选越多,验证也可能越贵。AdaSpark 在运行中学习验证耗时与候选接受概率,动态决定每一轮值得验证多少、验证哪些。
已在 Imparo 实现 · 论文预印本与复现代码公开
论文评测 · 解码速度最高可达
3.11×Imparo + AdaSpark,相比 llama.cpp 默认 DSpark 配置。
Qwen3-8B · Q4_K_M · Apple M3 Pro / Metal。相同模型与草稿文件,单请求测试;包含引擎与调度的综合收益。草稿模型提出后续 token,同时从当前请求文本提取 n-gram 续写候选。
结合接受概率和预计耗时,选择本轮候选的排序与验证宽度。
目标模型验证候选;实际耗时与接受结果反馈给下一轮选择。
数字说明:3.11× 为论文表 4 评估子集中 Qwen3-8B 的解码速度比值,按各回复速度比取几何平均;不是单次峰值,也不是完整任务耗时比值。基线为 llama.cpp build 11201 的默认 DSpark 三 token 链式验证配置,未额外调参。同一 Imparo 引擎内,仅改变调度策略的速度比值为 1.17–1.52×。
测试范围:Apple M3 Pro(36 GB)· Metal · 单请求顺序运行 · 32,768-token 上下文配置。四个目标模型为 LFM2.5-2.6B、LFM2.5-8B-A1B、Qwen3-4B、Qwen3-8B,使用六个公开数据源的单轮与多轮对话。
这里的在线学习调整的是验证调度,不是实时微调大模型。结果来自论文,本站未重新跑测;不代表全平台、所有模型默认开启,也不是多 Agent 并发性能结论。完整量化、草稿文件与统计排除项见复现说明。
03 / MEASURED PERFORMANCE
公开记录中的 Gemma 4 E4B:分别看输入处理和文字生成,不把两种速度混为一谈。
单位 token/s · 越高越好 · 每组使用相同输入长度
单位 token/s · 越高越好 · 每组使用相同输入长度
| 输入 tokens | 读入 Imparo | 读入 llama.cpp | 生成 Imparo | 生成 llama.cpp |
|---|---|---|---|---|
| 449 | 932 | 556 | 46.6 | 40.5 |
| 5,651 | 1,063 | 570 | 44.2 | 38.5 |
| 16,191 | 1,006 | 522 | 40.0 | 34.5 |
Apple M3 Pro · UD-Q4_K_XL · f16 KV · 512-token 预填充分块 · 各引擎关闭 thinking。同一会话内两轮交错测试取中位数;客户端以首 token 等待时间估算读入速度,以输出流计时生成速度。
节选自仓库公开记录,非本网页重新实测,也不代表所有模型或设备。完整表包含 LFM2 及其他引擎;其中 LFM2 短输入生成速度未领先 oMLX。
04 / GET STARTED
按项目文档确认运行条件,先完成一个可复现的任务。
按模型架构、后端与设备版本检查兼容范围。
按仓库文档构建引擎,设置上下文与缓存预算。
通过聊天补全接口接入应用,验证输入、输出和工具调用格式。
固定模型、设备与请求,记录延迟、内存与任务质量。
模型、后端和加速路径的支持范围以仓库记录为准。性能数字仅对应列明的配置,不代表统一提速比例;缓存共享不等同于全平台连续批处理。AdaSpark 论文结果与下方 Gemma 基准为两组独立实验,不能混用。
优化从哪里来:原生执行与实测调优:以 Rust 连接后端算子,按设备和负载选择运行配置;支持的 Metal 路径采用 megakernel 减少 GPU 调度开销。 上下文复用:相同模型与兼容缓存配置下,共享匹配的提示词前缀与 KV 页面,减少重复读入。 持续状态管理:保存和恢复匹配的会话状态,支持检查点回退与分支,并用分页及驻留预算管理缓存。
AdaSpark 自适应推测解码:已在 Imparo 实现的研究路径。根据运行中观察到的验证耗时与候选接受情况,调整验证树的排序和宽度,并加入当前请求文本中的 n-gram 候选。公开论文及复现脚本见下方链接;不代表所有模型和后端默认开启。
产品资料核对:2026-10-21。基于官方仓库与论文;main 分支与发布安装包可能存在差异,使用前请查看对应版本说明。
阅读实现、检查支持范围,按公开脚本复现性能结果。
ADAPTIVE INFERENCE · OPEN SOURCE
Our open-source LLM inference engine, built in Rust. Execution adapts to hardware, models and workloads, reducing overhead and repeated computation while retaining reusable state for long conversations and agent tasks.

01 / PRODUCT CAPABILITIES
Model structure, device limits and request sizes guide measured configuration choices, saved for reuse.
Call backend kernels directly. Supported Metal paths combine GPU dispatches through megakernels.
Reuse matching prefixes and KV pages under compatible model and cache settings, reducing repeated prefill and duplicate storage.
Persist KV and recurrent state. Retained checkpoints let tasks restore or branch without reprocessing all their history.
Paged allocation, residency limits and disk eviction manage accumulated state in long conversations and tool loops.
A chat-completions endpoint separates reasoning, visible text and tool calls for downstream applications.
02 / HOW IT WORKS
The model layer describes architecture and state; backends supply kernels. Measured configuration choices stay bound to the model, device and version.
Chat endpoint · tool-call output
Model workflows · tuning · state
Metal / CUDA / CPU
Shared state: matching prefixes / KV pages / checkpoints / residency budgets · support is configuration-specific
RESEARCH / ADASPARK
Proposing more tokens is not always faster: verification can cost more, too. AdaSpark learns verification time and candidate acceptance during execution, then adapts which candidates to verify and how many.
Implemented in Imparo · Preprint and reproduction code available
PAPER BENCHMARK · DECODE SPEED UP TO
3.11×Imparo with AdaSpark versus llama.cpp’s default DSpark configuration.
Qwen3-8B · Q4_K_M · Apple M3 Pro / Metal. Same target and drafter files, one request at a time; includes both engine and scheduling gains.Combine draft-model tokens with n-gram continuations from the current request.
Use estimated acceptance and verification time to choose candidate order and tree width.
The target verifies candidates. Measured time and acceptance outcomes inform the next round.
Metric: 3.11× is the geometric mean of per-reply decode-speed ratios for Qwen3-8B on the evaluation subset in Table 4, not a single-run peak or a full-task timing ratio. The baseline is llama.cpp build 11201 with its default three-token DSpark chain, without additional tuning. Holding Imparo fixed, scheduling alone gives a 1.17–1.52× ratio.
Scope: Apple M3 Pro (36 GB), Metal, one request at a time, with a 32,768-token context setting. Four targets—LFM2.5-2.6B, LFM2.5-8B-A1B, Qwen3-4B and Qwen3-8B—on single- and multi-turn conversations from six public sources.
Online learning updates verification scheduling, not the target model’s weights. These paper results were not rerun for this page; they do not establish universal support, default activation or concurrent-agent performance. See setup notes for quantization, drafters and exclusions.
03 / MEASURED PERFORMANCE
Gemma 4 E4B in the published record: input processing and generation are measured separately.
Token/s · higher is better · matched input length per group
Token/s · higher is better · matched input length per group
| Input tokens | Prefill Imparo | Prefill llama.cpp | Decode Imparo | Decode llama.cpp |
|---|---|---|---|---|
| 449 | 932 | 556 | 46.6 | 40.5 |
| 5,651 | 1,063 | 570 | 44.2 | 38.5 |
| 16,191 | 1,006 | 522 | 40.0 | 34.5 |
Apple M3 Pro · UD-Q4_K_XL · f16 KV · 512-token prefill chunks · thinking disabled on every engine. Median of two interleaved rounds in the same session. Client-side prefill estimates use time to first token; decode is timed from the output stream.
Selected repository measurements, not rerun for this page or a claim about every model or device. The full table includes LFM2 and other engines; LFM2 short-prompt decode does not lead oMLX.
04 / GET STARTED
Check the documented requirements and begin with a reproducible task.
Check the model architecture, backend and device requirements.
Follow the build instructions and set context and cache budgets.
Integrate the chat-completions endpoint and validate input, output and tool-call formats.
Pin model, device and requests; record latency, memory and task quality.
Model, backend and acceleration support follow the repository records. Performance is configuration-specific, not a universal speedup; shared caches do not imply continuous batching everywhere. AdaSpark results and the Gemma benchmark below are separate experiments.
Where the optimizations come from:Native execution and measured tuning: Rust connects directly to backend kernels; supported Metal paths use megakernels to reduce GPU dispatch overhead. Context reuse: share matching prefixes and KV pages when models and cache configurations are compatible. Persistent state: restore matching saved boundaries, rewind or branch from retained checkpoints, and manage cache through paging and residency budgets.
AdaSpark adaptive speculative decoding:Implemented in Imparo as a research path. Observed verification times and candidate acceptance guide tree ordering and width, with n-gram candidates from the current request. See the paper and reproduction scripts; this is not a default feature across every model and backend.
Product sources reviewed on 2026-10-21 against official repositories and the paper. Main and released installers can differ; check the relevant release notes.
Read the implementation, check supported configurations and reproduce the published measurements.