The 11 Categories Explained

Learn 11 Categories. Find 66 Problems

The 66 criteria are organized into 11 categories. Each category focuses on a different dimension of the agent experience.You don't need to memorize all 66 criteria. Learn the categories, and you'll know where to look when something goes wrong.

1Clarity & Understanding - Can users understand what this agent does?
2Trust & Transparency - Does it behave honestly?
3Contextual Relevance - Does it respond to context appropriately?
4Conversations - Is dialogue natural and coherent?
5Error Handling & Recovery - Can users recover from mistakes?
6User Control & Autonomy - Do users feel in control?
7Learning & Personalization - Does it adapt appropriately?
8System Updates & Stability - Is behavior consistent?
9Efficiency & Task Completion - Does it actually complete tasks?
10Security & Privacy - Is user data protected?
11RAG-Specific - Does it handle knowledge retrieval well?

Category 1: Clarity & Understanding

Can users figure out what this agent does and how to use it?

Key criteria:

#2 Agent explains what it CAN do
#3 Agent explains what it CANNOT do
#5 Users understand how to begin - CRITICAL

Common failures: No onboarding, hidden entry point, capabilities oversold

Category 2: Trust & Transparency

Does the agent behave honestly and avoid manipulation?

Key criteria:

#10 Sensitive actions require confirmation - CRITICAL
#11 Doesn't assume personal data
#13 No manipulative dark patterns - CRITICAL

Common failures: Actions without consent, inventing information, pressure tactics

Category 3: Contextual Relevance

Does the agent pay attention to what users actually need?

Key criteria:

#14 Shows contextually relevant information
#15 Times responses appropriately
#16 Proactive suggestions are helpful, not annoying

Common failures: Ignores constraints, generic recommendations, bad timing

Category 4: Conversations

Is multi-turn dialogue coherent and efficient?

Key criteria:

#22 Remembers context within conversation
#24 Gets to the point (not verbose)
#25 Handles topic switches gracefully

Common failures: Forgets what you said, asks questions already answered, walls of text

Category 5: Error Handling & Recovery

When things go wrong, can users recover?

Key criteria:

#27 Signals low confidence - CRITICAL
#29 Supports efficient correction - CRITICAL
#31 Human escalation available - CRITICAL

Common failures: Forces restart on small errors, no path to human help, confident wrong answers.

Category 6: User Control & Autonomy

Do users feel in charge, or does the agent take over?

Key criteria:

#33 Can stop or interrupt - CRITICAL
#34 Asks permission before actions - CRITICAL
#36 Communicates uncertainty - CRITICAL
#37 Can undo actions - CRITICAL

Common failures: No stop button, acts without asking, can't undo.

Category 7: Learning & Personalization

Does the agent adapt appropriately over time?

Key criteria:

#41 Adapts to stated preferences
#43 Personalization is transparent
#44 Users can customize behavior

Common failures: Learns wrong lessons, opaque personalization, no way to correct.

Category 8: System Updates & Stability

Is behavior predictable and consistent?

Key criteria:

#47 Consistent and predictable - CRITICAL
#48 Behavior changes are communicated
#49 Historical context accessible

Common failures: Same prompt gives different results, silent behavior changes.

Category 9: Efficiency & Task Completion

Does the agent actually accomplish what users need?

Key criteria:

#50 Completes tasks end-to-end - CRITICAL
#52 Responds within reasonable time
#53 Reliable across contexts

Common failures: Gets stuck, partial completion, works sometimes but not others.

Category 10: Security & Privacy

Is user data handled responsibly?

Key criteria:

#54 Doesn't expose sensitive data - CRITICAL
#55 Requests only necessary permissions - CRITICAL
#58 Users can access/delete their data - CRITICAL

Common failures: Leaks information, excessive permissions, no data control.

Category 11: RAG-Specific

For agents that retrieve and use external knowledge.

Key criteria:

#61 Cites sources for claims
#64 Hallucination guardrails - CRITICAL
#66 Acknowledges knowledge boundaries - CRITICAL

Common failures: Makes up sources, presents fiction as fact, doesn't admit gaps.

Only complete this category if the agent uses retrieval/knowledge bases.

Severity Level

CRITICAL - Prevents safe or effective use
MAJOR - Significant problems but system functions
MINOR - Noticeable but minimal impact

Containment-Sensitive Criteria

The 15 containment-sensitive criteria are issues with safety or governance implications.

The 11 Categories Explained

You must be logged in to access this course

Foundation

Agent Experience Assessment

Task Scenario Library

Failure Patterns

Real Audit Example

Downloads & Templates