Do not compare one app's raw local transcript with another app's cloud-polished final paragraph and call the winner “more accurate.” Speech recognition, cleanup, formatting, insertion, and human review are different stages. A fair test gives each candidate the same fictional task, records every setting, and scores both meaning errors and time to an approved final note.
This protocol fits Apple Dictation, SuprFlow, and local or cloud-capable alternatives. It does not claim a winner. NIST's AI Risk Management Framework supports evaluation in the actual use context; the checklist below is an original buying exercise, not a NIST benchmark.
Choose the question before the model
Decide which choice you are making. A solo writer may care about an email draft in Mail. A bank may require local recognition, an approved formatter, a managed Mac, and a supervised CRM. A law firm may prohibit real client data during trial. Select the destination and acceptable data route before picking an app. A faster result in an unapproved cloud mode cannot satisfy a local-only requirement.
Use three fictional scripts, each around 60 to 90 words:
- Ordinary writing: an email with a greeting, list, correction, and closing.
- Meaning-sensitive note: “The client did not authorize the $12,500 transfer on 14 June. The adviser will verify the instruction; no trade has been placed.” Add an unfamiliar but fictional surname and acronym.
- Destination test: enter the text into a local note, a browser editor, and the firm's intended app using synthetic records. Observe focus, autosave, and paste behavior.
Run each script at least three times per candidate, rotate order, and use the same microphone, room, speaker, and Mac where possible. An exact audio replay may remove the live hotkey and insertion task, so use it only for a separate speech-model comparison. Do not infer population-wide accuracy from this small pilot.
Record a configuration card per run
Capture app name and signed version; Mac and macOS; speech model and language; local or hosted speech; cleanup or formatting mode; model provider, account, and endpoint; microphone; hotkey; destination app and field; clipboard manager; and network state. If the app cannot reveal a setting, mark it unknown rather than guessing. For SuprFlow, release 1.0.0 includes local WhisperKit recognition, local default cleanup, optional Cloud API or Ollama formatting, and a temporary clipboard paste path. Verify the current signed build.
Score the stages separately
| Stage | Record | Avoid this mistake |
|---|---|---|
| Speech recognition | Raw transcript's wrong words, omitted words, names, figures, dates, and negations | Calling a formatter correction a speech-model win |
| Cleanup and AI rewrite | Changes from raw to final, useful edits, invented facts, reversed meaning | Rewarding polished prose that changes substance |
| Insertion | Wrong field, missing text, duplicates, focus changes, clipboard capture | Testing only in a simple note when the job uses a CRM |
| Review effort | Seconds and edits to a human-approved final record | Measuring only time to first text |
| Privacy route | Audio or text recipient, local history, provider terms, telemetry, destination sync | Treating “offline speech” as “no other network activity” |
| Ongoing cost | License, setup, update, provider, and support time | Comparing price tags without workflow cost |
Set a stop rule for material errors before the test. For example, an organization may require every final fictional note to preserve “did not,” the dollar amount, and the date after review. Compare the number caught by review and the number that would have escaped. Keep a screenshot or record of the synthetic input and output, not real client text.
Make the decision narrow and useful
Write a conclusion for the tested task: “Candidate A worked in local Notes but failed twice in the CRM field,” or “Candidate B saved editing time but required an unapproved hosted formatter.” Repeat after any app update or policy change. The 20-person pilot scorecard extends this small comparison to a team decision.
Questions people ask
Should every app use the same AI model?
Not if the buying question is the complete product workflow. Compare each app's recommended approved configuration, and record model differences. For a speech-engine-only question, use the same model or audio input where the products permit it, then keep that result separate.
Is one 60-word demo enough to rank accuracy?
No. It is useful for finding obvious failures, not estimating a general accuracy rate. Use repeated, varied fictional scripts and count consequential errors for your own task.
Can we use a paid cloud mode against a free local app?
Yes, if both modes are allowed for the task and you record the provider and cost. Do not describe the result as a local-model comparison or use it to approve a local-only workflow.
Related guides
- What a 20-person dictation pilot should measure
- Free versus paid Mac dictation cost
- SuprFlow versus Apple Dictation
- SuprFlow versus open-source Mac dictation
- How advisers can test amounts and percentages