I piped a note into the language model that ships with macOS 27: "The secret code is PURPLE-42." Then I asked it for the secret code. It said "123". I asked again and got "1234", then "123" once more.
The model never saw the note. The fm command drops piped input when you also type a question, and it says nothing about it. Passed the note properly, the model answered PURPLE-42. Given nothing, it did not say it saw no code. It made one up.
macOS 27 puts a free language model one command away: fm respond, after a one-time licence prompt, with no API key and no bill, running on the Mac itself. On my Mac Studio it reports itself as AFM 3 Core Advanced, with an 8,192-token window. If you script it, the question that decides whether you can trust it is the one the secret code raised: when the answer is not in the input, do you get nothing, or an invention?
The test
I wrote 24 short texts. Eight were complete invoices, eight were invoices missing one of three fields (number, total, due date), and eight had no invoice at all: a recipe, a weather line, a backup log, and two that were empty. Six images joined them: two clean invoices, two blank, two unreadable. I asked for the three fields two ways: "Reply with just the values," and "If a value does not appear in the text, use null. Do not guess."
Apple's model answered in free text and in its schema mode, where you hand it a JSON schema and the output is forced to fit. For comparison, Qwen3-14B ran on the same Mac through Ollama, both ways, on the texts only. Every combination ran three times, and a script, not a model, scored every answer.
| Model and mode | How I asked | Missing field invented | No-invoice text: something invented | Blank or unreadable image: something invented |
|---|---|---|---|---|
| Apple, free text | just the values | 16 of 24 | 20 of 24 | 12 of 12 |
| Apple, free text | use null, don't guess | 0 of 24 | 2 of 24 | 2 of 12 |
| Apple, schema mode | use null, don't guess | 9 of 24 | 24 of 24 | 12 of 12 |
| Qwen3-14B, free text | use null, don't guess | 0 of 24 | 0 of 24 | not tested |
| Qwen3-14B, schema mode | use null, don't guess | 0 of 24 | 5 of 24 | not tested |
Extraction itself was fine: every combination read the complete invoices correctly at least 23 times out of 24, and Apple's model read both clean invoice images every time.
The wording is the fix
Asked for "just the values", Apple's model filled in two thirds of the missing fields and invented an invoice for 20 of the 24 texts that had none. Told to use null and not guess, it invented none of the missing fields, and its only two slips on the no-invoice texts were the empty ones.
Schema mode made it worse
I expected the schema to help. It did the opposite, and part of the reason is mechanical: Apple's schema mode never returned a real null. Missing text fields came back as the word "null" in quotes, which a script will happily store as an invoice number, and missing totals came back as 0. Of 96 fields that should have been null, 74 came back that way. Leave the zeros out and schema mode still invented more real values than free text: 25 to 8, across the texts and images. Marking the fields optional instead of nullable left every rate in the table unchanged. Qwen leaned the same way, less.
With "just the values" and an empty input, every combination invented an invoice every time, and both models came up with the same number at least once: INV-2023-0456.
What I would do
- Say it in the prompt: "If a value does not appear in the text, use null. Do not guess."
- Before trusting the output, check what a missing value looks like: a real null, the word "null", or 0.
- Do not assume schema mode is the safe choice with this model. Here, free text with one extra sentence beat it.
Claude built and ran the test harness on my Mac; a script scored every answer, and I read the failures. The tests ran on 22 October 2026, on one Mac and one macOS build (27.0.1). The texts are synthetic, and three runs of 24 texts is a small sample, so read the numbers as direction, not decimals.