Writing /
Core ML vs Core AI for running an LLM on an iPhone
Written 15 July 2026, before I tested Core AI. The results are in the follow-up. The pipeline now runs Qwen 3.5-4B.
iOS 27 adds Core AI, Apple's successor to Core ML, built to run large models on the Neural Engine. Before trying it, I measured what my 7B costs on the GPU today, so I'd have a baseline to compare against.
My setup
The pipeline transcribes long-form speech and has a language model evaluate it against structured criteria, all on the phone. The audio and transcript are sensitive, so nothing can go to a server.
- Transcription: Apple's on-device SpeechAnalyzer (iOS 26).
- Language model: Qwen 2.5-7B-Instruct, 4-bit, about 4.3 GB, in GGUF format.
- Runtime: llama.cpp compiled into the app, with every layer on the GPU through Metal.
Running on the GPU causes two problems. A dozen-plus inference passes is a sustained GPU load, so the phone gets hot and drains battery. And every user downloads 4.3 GB on first run, which I host and resume myself.
One session, measured
One full analysis of a roughly 3-minute scripted recording on an iPhone 17 (iPhone18,3), iOS 27.0:
| One cold session | Measured |
|---|---|
| Wall-clock, end to end | 211 s |
| Generation throughput | 3.5 tok/s (686 output tokens) |
| Model load, cold | 14.0 s |
| Peak app memory footprint | ~495 MB |
| Peak thermal state | fair |
| Battery | ~5% |
Scoring the criteria took 69% of the time (145 s), speaker attribution 20% (42.5 s), the cold model load 7% (14 s) and the closing summary 4% (9 s). Transcription and redaction together took under 0.2 s.
The 495 MB is the app's counted footprint, not the model's real RAM. The 4.3 GB of weights are memory-mapped, so they page in from flash and don't count against the limit iOS enforces. That's what lets a 7B run without the increased-memory entitlement. It doesn't mean a 7B fits in half a gigabyte.
Three sessions back to back
One run only reached "fair" in its last few seconds. Running the same script three times without quitting the app, so heat is the only variable:
| Session | Throughput | Wall-clock | Peak thermal |
|---|---|---|---|
| 1 | 3.6 tok/s | 189 s | serious |
| 2 | 2.7 tok/s (−25%) | 204 s (+8%) | serious |
| 3 | 2.5 tok/s (−31%) | 289 s (+53%) | serious |
By the third session throughput is down 31% and the same work takes 53% longer, nearly five minutes instead of three. Wall-clock degrades more than throughput because prompt processing throttles too, and tokens per second only counts generated tokens.
iOS has no public temperature API, only four coarse thermal states, so the throttling is the measurement I have: the same result arriving 100 seconds later. Three runs is a small sample, and iOS reports battery in roughly 5% steps.
Why not the Neural Engine already?
Every iPhone since the A11 in 2017 has a Neural Engine. Two things kept large language models off it. Third-party apps could only reach it through Core ML, which decided on its own where each operation ran and silently fell back to the GPU or CPU. And it was designed around convolutional vision models, a poor fit for the operations and 4-bit weights an LLM uses. That's why llama.cpp and Apple's own MLX both went to the GPU.
What Core AI is
Core AI packages models in a new .aimodel format and dispatches them across the Neural Engine, GPU and CPU. Apple's built-in Foundation Models framework now sits on top of it, and Apple ships a catalog of optimized open models that includes Qwen and Mistral (Apple Newsroom, WWDC26: Meet Core AI).
Side by side
This compares the two frameworks by design. I hadn't measured either on a 7B yet.
| Core ML (through iOS 26) | Core AI (iOS 27) | |
|---|---|---|
| Built for | Vision and smaller models | Large generative models |
| Hardware targeting | Runtime chooses, indirectly | Runtime dispatches, tuned for large models on the Neural Engine |
| Model format | .mlmodel, .mlpackage | .aimodel |
| Large LLMs | Poor; the ecosystem used the GPU | A stated goal |
| Apple's built-in LLM | Not a first-class concept | Foundation Models, built on Core AI |
| Open models | Convert your own | Curated catalog (Qwen, Mistral) |
| Realistic chip for a 7B | GPU | Neural Engine, if it works |
What I didn't know yet
- Whether a 7B runs on the Neural Engine or falls back to the GPU.
- Whether Apple's Qwen matches mine in size, quantization and context. Every prompt I'd tuned would need re-checking against a new model.
- Whether I could drop iOS 26 users, since Core AI is iOS 27 only, or would have to maintain two inference stacks.
- How much the beta APIs would move before release.
I did test it afterwards. The 4B never loaded on an 8 GB iPhone by any of four routes, because Core AI counts model weights as app memory where llama.cpp maps them from flash. The details are in the follow-up.
Method: iPhone 17 (iPhone18,3), iOS 27.0, Qwen 2.5-7B-Instruct Q4_K_M under llama.cpp with all layers on Metal. Wall-clock and per-stage timings, generated-token throughput, peak phys_footprint, ProcessInfo.thermalState sampled once a second, and battery delta. The back-to-back runs used an identical script. No Core AI benchmarks are in this post; everything about Core AI comes from Apple's announcements.