Writing / 27 July 2026

Apple's on-device model vs Qwen 2.5-7B, and the fine-tune I couldn't ship

Written 27 July 2026. Since then the pipeline moved to Qwen 3.5-4B, and Apple's model scored 48.4% on a stricter, human-scored test set.

I compared three ways of running the evaluation step on an iPhone: my Qwen 2.5-7B on the GPU, Apple's built-in model, and Apple's model with a LoRA adapter I trained. The adapter matched the 7B on my test set for $0.76, and I still couldn't ship it. This follows my Core ML vs Core AI post, which has the GPU baseline.

Test 1: my own model on the Neural Engine

Apple's Core AI export recipes for iOS stop at Qwen 3-4B. The 8B is macOS only, so even total success meant a smaller model than the one I ship. And the 4B never loaded on an 8 GB iPhone:

• Raw .aimodel on the Neural Engine: killed by the memory watchdog at load.
• Precompiled on a Mac: the on-device specialization pass ran 10 to 30 minutes at high heat and was killed in all five attempts, each time leaving gigabytes of orphaned cache.
• Forced onto the GPU: the palettized weights inflated during graph materialization and blew the memory budget.
• A macOS-style dynamic export: the engine came up in 180 MB, then died building the graph at the first prefill.

The cause is memory accounting. Core AI holds model weights as counted app memory, while llama.cpp memory-maps them, which is why my 4.3 GB GGUF shows a ~495 MB footprint. iOS 27 also moved Neural Engine memory onto the app's own jetsam ledger. So a 2.4 GB Core AI 4B dies where a 4.3 GB llama.cpp model runs fine. Only Qwen 3-0.6B loaded, at about 43 tok/s, and it's far too small to grade anything reliably.

Test 2: Apple's built-in model

Apple ships a roughly 3B model inside the OS, reachable through the Foundation Models framework. Its weights live on the OS side, so the memory problem above doesn't apply, and there's nothing to download. I built a probe app that replays the pipeline's exact 48 grading prompts on the phone and saves the raw output. All scoring happens on my Mac with the same parser the pipeline uses, so the numbers compare directly with every other model I've run.

My GPU path used about 5% battery for one analysis, hit serious thermal pressure and ran at 3.5 tok/s. Here, three analyses' worth of work didn't move the battery or the thermal state. It isn't a clean Neural Engine vs GPU comparison, because this is a 3B against a 7B, and iOS reports battery in 5% steps.

I got one thing wrong going in. Apple's WWDC26 session shows the context size as 8192, and I assumed iOS 27 had doubled it. The device reports 4096. A 3,500-token transcript plus my ~700-token instructions barely fits.

Over-score is the share of behaviors that did not happen but got marked as done, and it's the error I care most about. The iOS 27 model is better than the older one, but it still credits nearly half of what never happened, and all 14 of its errors go that way. It quotes some line from the transcript and accepts it: one closing line got credited against three separate criteria, and one terse instruction against five.

Test 3: fine-tuning Apple's model

That failure looked like bias, not missing ability. The model followed the output format every time and always quoted real transcript text. It just said yes too easily, and an earlier bake-off had shown this task depends on judgment more than domain knowledge.

So I wrote about 600 synthetic training examples, all fiction, loaded with hard negatives: lines that look like they satisfy a criterion without doing it, and lines from the other speaker that must never be credited. Apple's toolkit asks for 32 GB of memory and my Mac mini has 16, where training crawled at about 95 s per step, roughly 15 hours per epoch. On a rented GPU it took 20 minutes and $0.76, so I trained two learning rates in parallel.

Over-scoring fell from 75% to 6.7% while recall rose to 94.4%. I re-ran the exported adapter through the real Foundation Models framework, not just the training harness, and got identical numbers. Apple's default learning rate overcorrected into a harsh grader that missed half of what people did; a third of that rate landed in the middle.

I made two mistakes. First, my benchmark only scored the met or missed verdict, but the product also writes a short tip for each criterion. Stock Apple wrote 39 tips, all different. My adapter wrote 29 with 5 distinct strings, mostly repeating one phrase I'd lazily reused in the training data. The verdicts improved and the feedback got worse, and I would have shipped it if I hadn't checked. The fix is to leave the tip line out of the training loss.

Second, the test set was mildly contaminated. I wrote the training data in the same session I'd read the benchmark transcripts, and some phrasing leaked: zero exact duplicates and a median line similarity of 0.24, but two or three near-paraphrases. Discounting those puts accuracy around 89.6%. So what I can defend is that it matches the 7B within this test's resolution; 45 of 48 against 42 of 48 was never a meaningful gap.

Why it didn't ship

From Apple's adapter toolkit page: "Version 26.0.0 is the last release of this toolkit and is not compatible with macOS, iOS, iPadOS, or visionOS 27 and later."

An adapter is tied to one exact set of base weights, checked by hash (baseModelSignature), and a mismatch is rejected outright. Training for iOS 27 needs a toolkit containing iOS 27's base weights, which doesn't exist publicly. My phone runs iOS 27, so no amount of compute would get the adapter onto it.

A prompting result that did help the 7B

While testing whether prompting could stand in for fine-tuning, I added three worked examples to the shared instructions and ran 240 criterion decisions:

That fixed 14 over-scores for 3 recall losses. The examples cost about 170 tokens, and because they sit in the cached instructions, they're processed once per analysis, not once per criterion. The same examples made Apple's 3B much worse: 54.3% down to 40.4%, with over-scoring up to 93.3%.

Result

I kept the 7B on the GPU. Apple's model costs almost nothing in battery and heat, and a few hundred targeted examples fix its accuracy for under a dollar, but the tool to deploy that fix has been withdrawn. The inference layer sits behind a protocol with llama.cpp underneath, which is what made this safe to try. The training pipeline is automated, so if Apple ships a toolkit for the current OS, retraining takes 20 minutes and about a dollar.

Method: iPhone 17 (iPhone18,3), iOS 27.0, Apple's system model through Foundation Models, greedy sampling, 180 max output tokens, default guardrails. The probe replays byte-identical prompts exported from my Python harness and computes nothing on the device. Labels: 48 hand-authored decisions across three sessions, and 240 decisions from 15 assembled transcripts for the prompting test. Adapter: Apple's toolkit 26.0.0, LoRA rank 32, bf16, 569 training and 70 validation examples split by transcript, 2 epochs, on a rented A10G. Every number measures agreement with my labels, not clinical correctness.

Sources: Apple, Foundation Models adapter training · Loading and using a custom adapter · WWDC26: What's new in Foundation Models