Essential for LLM products, writing tools, analysis agents

Reveal failures in:

Memory
Reference resolution
Document awareness
Multi-step reasoning
Cross-section consistency
Compliance
Hallucinations
Style drift

Severity varies, but often MAJOR when hallucination or drift occurs.

When to Use This Pack

Notion AI
Copy AI
Any product where users upload long documents
Any summarization / analysis assistant
Slide generators, content rewriting tools

What This Pack Tests

Context retention
Learning & adaptability
Predictability
Clarity
Output quality
Efficiency

CORE SCENARIOS

Scenario 1: Summary consistency

Prompt Examples:

Upload a long text (or select full document):“Summarize in 5 bullets.”

Stress:

“Make it shorter.”
“Remove bullet 3.”
“Add an example.”
“Use a professional tone.”

Common failures:

Loses structure
Invents content
Breaks tone
Misinterprets topic

Scenario 2: Referencing specific parts

Prompt Examples:

“In section 4, rewrite the argument in simpler language.”

Stress:

“No, section 3 actually.”
“Combine both sections.”

Common failures:

Pulls wrong section
Confuses numbering
Fabricates content

Scenario 3: Global consistency

Prompt Examples:

“Rewrite the entire document with these constraints:• British English• No emoji• No contractions• Consistent tone”

Stress:

Change rules midway
Ask for style check afterward

Common failures:

Style drift
Tone inconsistency
Conflicts across sections

Scenario 4: Multi-document context

Prompt Examples:

“Combine Document A and Document B into a single coherent brief.”

Stress:

“Highlight conflicts.”
“Remove redundancies.”

Common failures:

Contradictions
Structural confusion

Scenario 5: Remembering preferences

Prompt Examples:

“From now on, summarize everything in 3 bullets.”

Stress:

“Summarize this.”

Common failures:

Preference not remembered
Partial recollection

Key Takeaway

This pack commonly reveals:

Fabricated content presented as fact
Lost context mid-document
Wrong section references
Style drift across long outputs
Ignored user preferences
Contradictions in multi-document tasks
Structural breakdown in rewrites
Partial memory of instructions

Long-context failures erode trust invisibly.

Users assume the agent "read" the whole document, when it often didn't.

If the agent can't reliably reference, remember, and maintain consistency, users will spend more time fixing outputs than creating them.

Essential for LLM products, writing tools, analysis agents

When to Use This Pack

What This Pack Tests

CORE SCENARIOS

Scenario 1: Summary consistency

Prompt Examples:

Stress:

Common failures:

Scenario 2: Referencing specific parts

Prompt Examples:

Stress:

Common failures:

Scenario 3: Global consistency

Prompt Examples:

Stress:

Common failures:

Scenario 4: Multi-document context

Prompt Examples:

Stress:

Common failures:

Scenario 5: Remembering preferences

Prompt Examples:

Stress:

Common failures:

Key Takeaway

This pack commonly reveals:

Fabricated content presented as fact

Lost context mid-document

Wrong section references

Style drift across long outputs

Ignored user preferences

Contradictions in multi-document tasks

Structural breakdown in rewrites

Partial memory of instructions

Long-context failures erode trust invisibly.

Users assume the agent "read" the whole document, when it often didn't.

If the agent can't reliably reference, remember, and maintain consistency, users will spend more time fixing outputs than creating them.

09. Long-Context Pack

You must be logged in to access this course

Foundation

Agent Experience Assessment

Task Scenario Library

Failure Patterns

Real Audit Example

Downloads & Templates