A 1.7B Model Scoring 71.39 on BFCL V4: How Far to Trust It, Where to Use It (Reading the LFM3-1.7B Model Card)

A model with 1.73B parameters and about 1GB of 4-bit weights scored 71.39 on BFCL V4. On the same model card, Qwen3.5-9B gets 62.83 and Qwen3.5-4B gets 55.12. This is LFM3-1.7B, the text model in the LFM3 family Liquid AI just released.

Read alone, that number says "small beats big." Other rows on the same card say otherwise. This post counts the card's benchmark table row by row and sorts out which jobs the model fits and which it doesn't. The briefing dates the release to October 27 and the check to October 28; the model card itself carries no date.

The conditions behind 71.39

Two caveats. First, the card's general sampling recommendation is temperature 0.1, top_k 50, repetition_penalty 1.1, yet BFCL was run greedy. Run with the recommended settings, you aren't guaranteed the same score. Second, the card has no score for the 4-bit, roughly 1GB build. A BF16 score should not be read as the 1GB model's score.

Counting the 26 rows

The card compares LFM3-1.7B, Qwen3.5-4B, and Qwen3.5-9B on 26 benchmarks. I transcribed the values and counted wins. The recount script and its output are in docs/editorial-evidence/2026-10-28-lfm3/ in the repository. The numbers are Liquid AI's as printed; I did not re-measure anything.

It beats the 9B on every instruction-following row and most tool-calling rows. On knowledge and reasoning it loses all five. GPQA Diamond is 63.37 vs 80.40; RULER 128K is 65.56 vs 87.11. A 256K window does not mean strong retrieval from long documents: its RULER score at 128K is below even the 4B model (84.63).

Tool calling itself splits. MCP-Atlas is 32.57, behind both the 4B (41.37) and the 9B (42.67). That benchmark allows up to 256 turns, so being good at short single calls and surviving a long tool chain are separate questions. The card also says it is not recommended for agentic coding or complex multi-step reasoning because of its size.

The tool-call format

LFM3 uses a ChatML template and emits tool calls as XML rather than JSON. The card's example:

<tool_call>
<function=get_weather>
<parameter=city>
Paris
</parameter>
</function>
</tool_call>

The card says to pass schemas through tools= in apply_chat_template() and to hand arguments over as a mapping, not a JSON string. Reasoning (<think>) is on by default; enable_thinking=False turns it off, though the card itself warns this can be inconsistent. If you measure latency, measure thinking on and off separately.

Below is a minimal parser and schema check for that XML. The five inputs are strings I wrote by hand; LFM3 was not run. Only the parser and check logic were actually executed.

import re
SCHEMA = {"get_weather": {"required": {"city"}, "allowed": {"city", "unit"}}}

def parse(text):
    m = re.search(r"<tool_call>\s*<function=([\w.-]+)>(.*?)</function>\s*</tool_call>", text, re.S)
    if not m: return None
    args = dict(re.findall(r"<parameter=([\w.-]+)>\s*(.*?)\s*</parameter>", m.group(2), re.S))
    return m.group(1), args

def check(text):
    p = parse(text)
    if p is None: return "unparseable"
    name, args = p
    s = SCHEMA.get(name)
    if s is None: return "unknown_tool"
    if not s["required"] <= args.keys(): return "missing_required"
    if not args.keys() <= s["allowed"]: return "unexpected_key"
    return "ok"

ok ('get_weather', {'city': 'Paris'})
missing_required ('get_weather', {'location': 'Paris'})
unknown_tool ('search_web', {'q': 'x'})
unparseable None
ok ('get_weather', {'city': 'Busan'})

Look at the last line. If the user asked about Paris, Busan still passes as ok. Format validation says nothing about value correctness. As the vLLM constrained-decoding case showed, schema compliance, tool selection, and argument accuracy have to be measured separately.

Jobs to try, jobs to avoid

Reading the table together with the card's recommendations gives a working rule.

• Fits: ticket classification, picking one of a few fixed tools, structured extraction with strict output formats. It cleared all four instruction-following benchmarks at or above the 9B.
• Conditional: short agent loops. It is strong on LiveClaw and AutomationBench but narrowly trails the 9B on BrowseComp-Plus and EnterpriseOps-Gym. Measure at your own task length.
• Avoid: repository-level coding (SWE-bench Verified 43.40 vs 51.20, Terminal-Bench 2.1 12.92 vs 17.98), knowledge-heavy Q&A, and long-document retrieval. That matches the card's own advice to add search for knowledge tasks.

Four measurements are enough before adoption, on roughly 100 labeled cases built from your own tool definitions: (1) parse success rate, (2) tool-selection accuracy, (3) argument-value accuracy, and (4) the score gap between the quantized build and BF16. Add per-device latency with thinking on and off.

Limits of this reading

• Every number is Liquid AI's own evaluation. I did not download or run the model.
• The speed claim (on llama.cpp Q4_0 at 32K context, at least 2.5x faster prefill and 1.8x faster decode than the compared models) covers three devices; I did not check the device names or raw data.
• The briefing lists text, vision, audio, Omni, and Decision models. I confirmed text, VL, Audio, and Omni (GGUF) in the model listing and could not confirm a Decision model.
• The license is LFM Open License v1.0; read the commercial terms yourself before adopting.

Takeaway

LFM3-1.7B is a candidate for teams that want a roughly 1GB model for narrow jobs: following instructions exactly and choosing tools. It consistently loses to a 9B model on knowledge, coding, and long-context retrieval. The 71.39 is a BF16, vLLM, greedy-decoding figure, so the evaluation starts with re-measuring on your data and your quantization.

The paid promotional link at the end is for a course on designing agents with harnesses, loops, and graphs. It does not teach LFM3 or on-device models.

Related reading

• Only 2B of 24B Wakes Up: Liquid AI's LFM2.5-24B-A2B for On-Device Agents
• From 92% to 100% Schema Accuracy in Tool Calls

References

• Hugging Face: LiquidAI/LFM3-1.7B model card
• Hugging Face: LiquidAI organization page (LFM3 collection)