The Mental Model Gap
When a user opens a familiar application, they already know roughly what to expect: buttons respond to clicks, forms save data, and undo reverses the last action. AI agents break this expectation because their behaviour is probabilistic, context-dependent, and sometimes non-deterministic.
Users build mental models from every interaction. When an agent behaves unexpectedly, the mental model fractures — and trust erodes faster than it was built.
A mental model is the user’s internal understanding of how a system works, what it can do, and how it will behave in different situations. For traditional software, mental models transfer well between applications. For AI agents, every new capability requires updating the model.
Users do not need to understand how an AI works internally. They need an accurate behavioural model — what the system will do, when it will ask for help, and how to correct it when it is wrong.
Dimensions of Trust
Trust in AI systems is multi-dimensional. Users calibrate trust across several axes simultaneously:
| Dimension | Question the user asks | Erosion trigger |
|---|---|---|
| Competence | Can it do the task correctly? | Obvious errors, low confidence on easy tasks |
| Reliability | Does it work consistently? | Non-determinism, intermittent failures |
| Transparency | Can I see what it is doing? | Opaque actions, missing rationale |
| Accountability | Who is responsible when it fails? | No undo, no audit trail, no escalation path |
| Benevolence | Does it have my interests in mind? | Misaligned goals, unexpected side effects |
Calibrated Trust
The goal is not maximum trust — it is calibrated trust: the user’s trust level matches the system’s actual capability.
Over-trust is dangerous because users stop verifying agent outputs. Under-trust leads to constant overriding, negating the productivity benefits of delegation.
Communicate confidence along with every output. A system that says "I am 95% confident this is correct" helps the user calibrate trust. One that always projects certainty invites over-trust or abrupt loss of confidence when it errs.
Building Mental Models Through Onboarding
First interactions shape the mental model disproportionately. Onboarding should establish:
- Scope — what the agent can and cannot do. Explicit “out of scope” examples prevent false expectations.
- Failure modes — show what happens when the agent is uncertain or when it makes a mistake.
- Recovery paths — demonstrate undo, correction, and escalation before the user needs them.
| Onboarding pattern | Description | Example |
|---|---|---|
| Guided walkthrough | Step-by-step introduction with safe defaults | "Let me show you how I handle meeting scheduling" |
| Shadow mode | Agent acts but does not execute; user reviews | "Here is what I would have done — approve or modify?" |
| Progressive disclosure | Capabilities unlock as trust is demonstrated | Start with read-only, add write access later |
| Exception demonstration | Deliberately show failure and recovery | "Watch what happens when I cannot resolve a conflict" |
Trust Recovery
Even well-designed agents make mistakes. Trust recovery follows a predictable pattern:
- Acknowledge — explicitly name the error without deflection.
- Explain — provide the context that led to the mistake.
- Remediate — undo the action or offer compensation.
- Prevent recurrence — describe what will change to avoid the same error.
Silently correcting an error without informing the user destroys trust faster than the original mistake. Users interpret silence as hiding failures.
Key Takeaways
- Users build mental models from behaviour, not architecture — design for predictable, observable actions.
- Trust is multi-dimensional: competence, reliability, transparency, accountability, and benevolence all matter.
- Aim for calibrated trust, not maximum trust. Over-trust causes unchecked failures; under-trust defeats automation.
- First-run onboarding is the highest-leverage trust-building moment.
- Trust recovery requires explicit acknowledgment, explanation, remediation, and prevention.