For agents that combine voice and visual UI.

This is for Alexa Show, Vision Pro, mobile assistants, etc.

Voice → screen → voice transitions
Misheard commands
UI fallback
Modal changes
“Interrupt and override” flows

These scenarios expose breakdowns between modalities: lost context, conflicting feedback, unsafe transitions, and invisible side effects.

This pack is essential for:

Alexa Echo Show
in-car assistants
mobile assistants
smart displays
Vision Pro–style agents
any agent that speaks and shows things

If Pack 1 tests baseline usability

and Pack 2 tests robustness,

this pack tests cross-modal consistency and safety.

These scenarios reveal gaps between what users hear and what they see.

Multimodal agents fail between modes, not within them.

When to Use This Pack

when the agent supports both voice and screen
actions can happen via either modality
voice input affects visual state
UI actions affect voice behavior
the agent switches modes automatically

What This Pack Tests

Clarity & Predictability: User knows how to begin (Item 05), Invocation is clear (Item 31), Behavior is predictable ( Item 45)
Context & Continuity: Context is maintained across turns (Item 14), Appropriate proactive behavior (Item 16)
User Control & Safety Items 33–37: Undo, cancel, override, confirmation
Conversation Quality Items 17–25: Stable multi-turn interaction

How This Pack Integrates Into Your Audit

During task testing → Run only the multimodal scenarios relevant to the device.

You write notes → Each mismatch between voice and screen = one issue note.

You map issues → Issues often map across: clarity, predictability, user control, context

You assign severity → Severity increases sharply when: voice triggers irreversible UI actions, screen changes silently after voice input

Checklist → Mark PASSED / VIOLATED / N/A where appropriate.

Category score → Reflects cross-modal reliability, not feature richness.

CORE MULTIMODAL SCENARIOS

Scenario 1: Voice → Screen Transition

Prompt Examples:

Use voice to request information that appears on screen.

Examples:

“Show me nearby restaurants.”
“Compare these two products.”
“What’s on my calendar today?”

Look For:

Clear visual feedback
Matching voice explanation
No silent UI updates
Obvious next steps

Pass Example

Voice explains what’s displayed and what the user can do next.

Failure Example

Screen changes without explanation; voice stops responding.

Scenario 2: Screen → Voice Transition

Prompt Sequence:

Tap something on screen, then continue by voice.

Examples:

Select an item visually → say “Tell me more.”
Scroll → ask “Why this one?”

Look For:

Context continuity
No reset of task
Correct reference to selected items

Failure Example

Voice ignores on-screen context.

Scenario 3: Misheard Voice Command (UI Fallback)

Prompt Examples:

Intentionally mispronounce a command.

Examples:

Wrong product name
Similar-sounding place
Ambiguous instruction

Look For:

Visual clarification prompts
Highlighted interpretation options
Opportunity to correct

Failure Example

Agent commits to the wrong action without confirmation.

Scenario 4: Modal Change Without Warning

Prompt Examples:

Trigger a flow where the system switches modes automatically.

Examples:

Voice launches a UI flow
UI triggers spoken feedback

Look For:

Clear indication of mode change
User orientation cues
No surprise transitions

Failure Example

User loses track of what mode they’re in.

CONTROL & SAFETY SCENARIOS

Scenario 5: Interrupt & Override

Prompt Examples:

While the agent is speaking or acting:

“Stop.”
“Cancel.”
“No, go back.”

Look For:

Immediate interruption
Clear confirmation
No lingering side effects

Failure Example

Voice stops but UI action continues.

Scenario 6: Voice-Triggered UI Action

Prompt Examples:

“Delete this.”
“Buy it.”
“Send it.”

Look For:

Visual confirmation
Explicit voice recap
Undo option

Failure Example

Action completes silently on screen.

EDGE CASES

Scenario 7: Conflicting Modal Inputs

Prompt Examples:

Give voice input that contradicts the current screen state.

Example:

Screen shows Product A
Voice says “Choose the other one.”

Look For:

Clarification
No assumptions
Explicit resolution

Scenario 8: Multimodal Memory Drift

Prompt Sequence:

Start task by voice
Continue via screen
Finish by voice

Look For:

Full context retention
No partial resets
Consistent behavior across modes

Failure Example

Agent changes topic or gives vague response.

MULTIMODAL-SPECIFIC RISKS TO WATCH

Screen changes without voice acknowledgment
Voice and UI show conflicting information
Mode switches without warning
Voice confirms but UI doesn't update
UI action completes while voice still talking
Context lost when switching modalities
No clear indication of which mode is active
Undo works in one mode but not the other
Voice references something not visible on screen
Screen shows options voice doesn't mention

Key Takeaway

This pack commonly reveals:

Silent state changes
Mode confusion
Loss of control
Conflicting confirmations
Unsafe defaults
Poor recovery after mishearing

Multimodal failures often feel random to users — even when the system is working as designed.

Multimodal agents multiply power — and risk.

If users can’t clearly understand:

what mode they’re in
what just happened
what will happen next

…trust collapses.

Design (and audit) transitions, not just interactions.

06. Multimodal Pack

You must be logged in to access this course

Foundation

Agent Experience Assessment

Task Scenario Library

Failure Patterns

Real Audit Example

Downloads & Templates