Last updated {{EVAL_DATE}}

How accurate is Zonlo's feedback?

Feedback you can't trust is worse than no feedback. This page explains how Zonlo grades what you say, how we measure that grading, and what we haven't validated yet. It is an engineering writeup, not a sales page.

How grading works

When you speak, your iPhone transcribes your voice on-device with Apple's speech recognition. The audio never leaves your phone; only your words, as text, are sent to our server over an encrypted connection.

The server grades the transcript with a large language model against the scenario's rubric: did your reply work in that moment, is there a more natural way to say it, and did the register (politeness level) fit the scene.

For Japanese, we don't ask the model to guess what you said. Every Japanese reply is first parsed by Kagome, a morphological analyzer, which identifies the politeness forms and particles you actually used, deterministically. That analysis is handed to the grader as ground truth, so the register judgment rests on parsed grammar rather than the model's reading of raw text.

How we measure it

We maintain an eval corpus of real learner phrasings for our scenarios, including clusters of valid alternative phrasings: the correct-but-different answers that a strict grader would wrongly mark down. Every change to the grader (prompt, model, or validation logic) runs against the full corpus before it ships. If it grades worse than what's live, it doesn't ship.

We track false-fails as their own number because they are the most damaging mistake Zonlo can make. Telling someone who already freezes when speaking that a correct reply was wrong is exactly the harm this app exists to avoid.

Current results

Measured on {{MODEL_NAME}}, last run {{EVAL_DATE}}. Percentages on a corpus this size should be read as a direction, not a guarantee.

What we haven't validated yet

• The corpus is small and growing. Each new case can move the percentages, which is exactly why we publish the corpus size next to them.
• Japanese labels are still being validated against native sources. The register checks are deterministic, but the human-provided correctness labels for Japanese cases are being cross-checked against native materials before we treat them as settled.
• Follow-up exchanges are new to the corpus. Zonlo's conversations continue past the first line, and grading those later exchanges only recently gained corpus coverage. Expect this section to firm up as those cases grow.

This page updates as the corpus grows and as grader changes ship. The date at the top is the date of the numbers, not of the prose.

Curious how this compares to other apps? Read Zonlo vs Duolingo and Zonlo vs Speak, or see how a conversation works.