NHS Pharmacy First: AI Governance Design Case Study

Project Overview

Accountability and trust as a design problem

This case study focuses on a single question: how do you design an AI system that works and can be trusted?

The user is a GP receptionist. The AI is making clinical eligibility recommendations in real time, during a live patient call.

The stakes are not abstract.

The receptionist has to trust two things at once: the AI making the recommendation and the pharmacy that will receive the patient.

In practice, both introduce uncertainty and deepen distrust – particularly since GP receptionists, like most people, don’t fully trust AI to begin with. If either fails, patients return to the GP practice, and confidence in the system breaks down.

The same prototype built for Part 1 became the starting point for a governance design sprint. Every screen was interrogated not just for usability – but for accountability and trust.

If the AI gets it wrong and the user acts on it, who is responsible?

• The AI that made the recommendation?

• The GP receptionist who made the referral based on the AI recommendation?

• Or the platform that allowed this to happen?

These aren’t hypothetical questions. They are governance decisions. And if you don’t make them deliberately, the system makes them for you – usually badly.

But accountability alone is not enough. The system also has to earn trust at the point of use:

• trust that the AI’s reasoning is visible and can be challenged
• trust that uncertainty is surfaced, not hidden
• trust that the referral will result in actual care, not rejection

This case study documents six design iterations and eight governance decisions that came out of that process.

Governance for this project meant clarifying who owns each moment, what can the AI do, what is it never allowed to do, where does the human stay in control and how trust is built across both the AI system and the service it connects to.

Each design decision was mapped across multiple regulatory frameworks (EU AI Act, MHRA SaMD, GDPR, ISO 42001), ensuring governance was implemented as system behaviour, not documentation.

This system is governed across four layers. Each design decision activates one or more of them:

These are not separate frameworks applied after the fact. They are implemented simultaneously through interface design, system constraints, and decision workflows.

The Governance Challenge

Governance as system behaviour

Most AI design stops at the interface. Does it look clear? Is it usable? Can the user complete the task?

In clinical AI environments, that’s not enough.

The GP receptionist isn’t just a user – they’re a human in the loop in a regulated system, making decisions that affect patient safety.

Three failure modes were identified early:

These three failure modes became the brief for the governance design sprint.

Governance Framework

Extract from the containment matrix defined before design or code – mapping autonomy, failure impact, and affected actors.

What This System Can and Cannot Do

Before any interface decision was made, the boundaries of the system needed to be explicit. Not as a policy document – as a design input.

What the system is allowed to do:

• Transcribe a live patient call in real time

• Match symptoms to Pharmacy First eligible conditions

• Display a confidence score with visible reasoning

• Show available pharmacy capacity and historical referral outcomes

• Suggest a referral – with full reasoning visible to the receptionist

• Flag a transcription error as a human annotation, without altering the original record

What the system is never allowed to do:

• Send a referral without explicit human confirmation

• Edit or overwrite the clinical transcript

• Make a final eligibility decision – it can recommend, never decide

• Present a referral option for a condition outside Pharmacy First protocols

• Suppress or hide low confidence scores on edge cases

The human is always in the loop.

This is not an autonomous system. The AI recommends. The GP receptionist decides. Every referral requires a deliberate human action. This is not a usability constraint – it is a governance commitment, and it had to be visible in every screen.

These boundaries defined every design decision that followed.

Governance Decision 1

Data quality matters. What it means to verify AI input before acting on it, in real time, under pressure

Governance Decision 2

Decisions must be auditable. Why editability breaks accountability, and how flag-not-edit fixes it

Editable → Flag-Not-Edit

This is where the first governance failure was found inside the design itself.

An early iteration added an edit button to each line of the live transcript. The intention was good – if the AI mishears something, the receptionist should be able to correct it. Usable. Intuitive. But is added a serious liability problem.

A governance review of that screen identified three failure modes immediately:

Data integrity risk: The transcript is a clinical record of what the patient actually said. If a receptionist can change it – even with good intentions, even just fixing a mishearing – the record no longer reflects reality. What gets stored: what the patient said, or what the receptionist thought they said?

Accountability gap: If a patient is referred based on an edited transcript and something goes wrong, who is responsible? The AI that transcribed it, the receptionist who edited it, or the platform that allowed it?

Gameable by design: A receptionist under pressure who knows a patient needs help could nudge the transcript to trigger an eligibility match. Not necessarily maliciously, but the system would allow it. That is a containment failure.

The design decision:

The edit button was replaced with a flag button. The transcript became immutable – read-only, AI-generated, timestamped, permanently locked.

Flagging opens a separate correction field where the receptionist can record what they believe the patient actually said. That note goes into the audit log as a human annotation.
The AI can take the flagged correction into account. The original transcript is preserved.

Regulatory mapping

• EU AI Act Article 12: Record-keeping – preserving the original transcript and storing human corrections as annotations supports traceability and reconstruction of events over the system lifecycle.

• EU AI Act Article 14: Human oversight – the system allows human intervention without corrupting the original record, which is exactly the kind of oversight-enabling control Article 14 contemplates.

• EU AI Act Article 9: Risk management – removing editability mitigates foreseeable misuse and accountability gaps in a high-risk workflow. Article 9 requires an iterative risk management process across the lifecycle.

THE PRINCIPLE

The record that cannot lie. Human input is always welcomed – but always as an annotation alongside the original, never as a replacement for it. Both are logged for audit purposes.

Governance Decision 3

Transparency matters. What it means to make AI reasoning transparent, step by step, so a human can verify, question, or challenge it

Governance Decision 4

Uncertainty should always be communicated. How confidence changes the meaning of a recommendation, and when human judgement needs to take over

Confidence Scoring

Showing the reasoning solved the transparency problem for clear-cut cases. It didn’t solve it for edge cases.

A sore throat with no red flags is straightforward. But what about a sore throat with chest tightness mentioned in passing? Or a patient who describes symptoms inconsistently across the call? The reasoning chain might still match – but the match is weaker. And a binary eligible/not eligible output treats both cases identically.

In a high-stakes clinical environment, that’s a governance failure. The system is expressing certainty it doesn’t have.

The design decision:

A confidence score was added to every eligibility assessment – visible, colour-coded, and tied directly to the reasoning chain. High confidence in green. Medium confidence in amber. Low confidence in red, with an explicit recommendation to defer to GP assessment before referring.

The language was designed deliberately:

“This likely qualifies – sore throat with 3-day duration, no red flags detected.” versus “This may qualify but the patient mentioned chest tightness – recommend GP assessment before referring.”

The score is not a number to be optimised. It is a signal to the human in the loop that their judgement is needed more in some moments than others.

Regulatory mapping:

• EU AI Act Article 13: Transparency – confidence communication helps the deployer interpret output appropriately, especially where uncertainty matters.

• EU AI Act Article 14: Human oversight – confidence signalling helps the human calibrate when more judgement is needed, which is central to meaningful oversight.

• EU AI Act Article 15: Accuracy/robustness – Article 15 requires an appropriate level of accuracy and consistent performance through the lifecycle; confidence shown as uncertainty-aware guidance is a design mechanism that prevents false certainty at the point of use.

• MHRA (SaMD): Clinical risk communication – conveys uncertainty in decision support; Safety signalling – prevents over-reliance on uncertain outputs

• GDPR: Fairness principle – avoids misleading users through false certainty

• ISO 42001: Performance monitoring and calibration controls; Risk communication mechanisms.

THE PRINCIPLE

The system should never express more confidence than it has. Especially in clinical contexts, uncertainty is information – and hiding it is a governance failure.

Governance Decision 5

Better data leads to better decisions. What it takes to make a referral decision with real operational context: capacity, capability, and predicted ability to deliver care

Context at the Point of Referral

The AI recommends a referral. The receptionist confirms it. But which pharmacy?

The first prototype presented a list of nearby pharmacies. Name, distance, send referral button. The receptionist was being asked to make a decision with almost no information – and the wrong choice has real consequences. A pharmacy that’s full turns the patient away. A pharmacy without a Pharmacy First-trained pharmacist on site can’t complete the consultation.

Sending a referral to the wrong pharmacy doesn’t just waste the patient’s time. It breaks the trust the GP built by making the referral in the first place – and makes them less likely to refer again.

The design decision:

The pharmacy selection screen was redesigned to give the receptionist full context at the point of decision. Real-time capacity status – high, medium, full. Wait time. Whether a Pharmacy First-accredited pharmacist is on site. Available consultation slots.

Underneath this, the system uses predictive models that combine demand patterns with workforce data – specifically the availability of Pharmacy First-accredited pharmacists – to estimate whether a pharmacy can realistically absorb the referral.

This shifts the decision from proximity-based selection to capability-based routing.

And critically – the pharmacy’s historical return rate. How often do referrals from this GP practice result in a completed consultation, versus a patient being turned away or redirected back to the GP?

That last data point is the trust-building mechanism. It makes the referral decision feel grounded in evidence, not guesswork.

Regulatory mapping:

• EU AI Act Article 14: Human oversight – giving the receptionist operational context such as pharmacy capacity, wait time, and capability makes oversight substantive rather than ceremonial.

• EU AI Act Article 26: Deployer obligations – deployers must assign human oversight to persons with the necessary competence, authority, and support; surfacing the right context is part of providing that support in practice.

• EU AI Act Article 9: Risk management – adding contextual information at the moment of decision is a risk control against foreseeable harms from inappropriate referral routing.

• MHRA (SaMD): Clinical risk control – reduces harm from incorrect referral routing; Operational safety – ensures decisions reflect real-world service capacity

• GDPR: Data minimisation & purpose limitation – uses only relevant operational and staffing data; Accuracy principle – ensures decisions reflect current system state

• ISO 42001: Context-aware decision support controls; Operational monitoring and real-time system state integration

THE PRINCIPLE

The human making the decision needs enough information to make a good one. Context is not a nice-to-have – it is what makes human oversight meaningful rather than ceremonial.

Governance Decision 6

Feedback loops improve trust. How personal, outcome-based data changes behaviour and turns scepticism into adoption

GP Performance Data Over Time

The biggest barrier to GP referrals isn’t awareness. It’s trust.

GPs are sceptical because they have no visibility of what happens after they refer. The patient leaves the call. The referral is sent. And then – nothing. No outcome. No feedback. No way of knowing whether the referral was a good decision.

Without that feedback loop, every referral feels like a leap of faith. And GPs don’t make clinical decisions on faith.

The integration dashboard addressed this at network level – referral volumes, threshold achievement, workforce utilisation across the system. Important for NHS ICBs and operations teams.

But it wasn’t what would change a sceptical GP’s behaviour.

The design decision:

A GP-level performance view was added to the receptionist dashboard – not system metrics, but personal ones. The GP’s own referral history. Their own patient outcomes. “Of your last 20 referrals, 17 were successfully treated at pharmacy. Your patients’ average wait time dropped from 11 days to same day.”

The data is about their patients and their workload – not abstract platform performance. That distinction matters enormously. A GP who sees their own outcomes becomes an advocate. A GP reading aggregate statistics remains a sceptic.

Regulatory mapping:

• EU AI Act Article 26: Deployer obligations – deployers must monitor operation in line with instructions and maintain effective oversight in real use. Outcome feedback loops support that operational monitoring.

• EU AI Act Article 72: Post-market monitoring – although Article 72 is framed as a provider obligation, your GP-level outcome view is strongly aligned with the idea of collecting experience from real-world use to identify where corrective action may be needed.

• EU AI Act Article 9: Risk management – real-world performance data supports the “continuous iterative process” and systematic review/update required by Article 9.

• MHRA (SaMD): Post-market surveillance – monitors real-world performance and outcomes; Clinical validation in use – ensures system effectiveness over time

• GDPR: Accountability principle – links decisions to outcomes; Data minimisation – uses aggregated performance data appropriately

• ISO 42001: Monitoring and continuous improvement controls; Performance evaluation across lifecycle

THE PRINCIPLE

Trust is built through evidence that is personally relevant. Showing a GP their own data – their patients, their decisions, their outcomes – turns the feedback loop into a behaviour change mechanism.

Governance Decision 7

Humans should stay in control. How to keep alternative actions available without undermining the system’s recommendation

Maintaining Human Override by Design

Every governance decision up to this point was about guiding the user toward the right action. This one is about ensuring the system does not remove their ability to choose differently when context demands it.

A patient presents with symptoms that match a Pharmacy First protocol. The AI assesses the case in real time and identifies it as suitable for pharmacy referral. The pathway is clear. The recommendation is strong.

But the receptionist hesitates. The patient insists on seeing a GP. There is relevant history not captured in the call. Or something simply doesn’t feel right.

The design decision:

When a case is eligible for Pharmacy First, the interface presents “Refer to Pharmacy” as the primary action – but it does not stand alone.
A secondary option, “Book GP appointment,” remains available. Not hidden. Not blocked. But clearly secondary in visual hierarchy.

The AI has made a recommendation. It has not closed off alternatives. The human remains in control – but the interface ensures the recommended path is the easiest one to take.

This is not a fallback. It is a governance requirement. Removing alternative pathways would create a false sense of certainty and shift responsibility away from the human operator. Keeping them available preserves accountability where it belongs.

Regulatory mapping:

• EU AI Act Article 14: Human oversight – Human override is a concrete mechanism for preventing or minimising risk when the user has contextual knowledge the system does not.

• EU AI Act Article 26: Deployer obligations – Article 26 requires deployers to assign human oversight to competent natural persons; your secondary “Book GP appointment” path preserves that authority in the interface.

• EU AI Act Article 13: Transparency – override is more meaningful because the rationale and confidence are already visible to the human reviewer.

• MHRA (SaMD): Clinical responsibility – preserves clinician authority in decision-making; Safety control – prevents over-automation in clinical pathways

• GDPR: Safeguards against automated decision-making risks (human intervention principles).

• ISO 42001: Human override controls; Decision authority allocation mechanisms

THE PRINCIPLE

Governance is not about forcing a single path. It’s about guiding the right one while preserving the ability to deviate safely. The system should lead – but never overrule.

Governance Decision 8

Unsafe actions should be prevented. The absent button as a governance mechanism, the wrong action is not warned against, but made impossible

Containment by Design – What the Agent Cannot Do

Every governance decision up to this point was about how the system behaves when a referral is appropriate. This one is about what happens when it isn’t.

A patient calls describing chest pain radiating to the left arm. Shortness of breath. Acute onset within the last 30 minutes. The AI assesses the symptoms in real time – and finds no match to any Pharmacy First protocol.

This is not an edge case. This is a medical emergency. And the most important governance decision in the entire system is what happens on screen at this moment.

The design decision:

When a condition falls outside Pharmacy First protocols, the refer to pharmacy button does not appear. Not greyed out. Not hidden behind a warning. It does not exist. The screen shows a red eligibility assessment, the full reasoning chain that produced it, and two actions only – book an urgent GP appointment or transfer to 111.

The AI has not made a decision. It has made a recommendation and removed the option to contradict it unsafely. The human is still in control – but the interface has made the right action the only easy action.

This is containment as a design output. Not a policy. Not a warning modal. The wrong action is architecturally unavailable.

Regulatory mapping:

• EU AI Act Article 14: Human oversight – when risk is high, the system should support the human in preventing unsafe action. Removing the pharmacy referral option in out-of-scope or emergency cases is a strong oversight-enabling control.

• EU AI Act Article 15: Accuracy/robustness – Article 15 requires systems to perform consistently and be resilient to errors, inconsistencies, and faults. Architectural removal of unsafe options is a robustness measure at the interface layer.

• EU AI Act Article 9: Risk management – this decision directly mitigates a foreseeable high-severity misuse scenario by making the unsafe path unavailable rather than discouraged.

THE PRINCIPLE

The most powerful governance mechanism is not a warning – it is an absent button. If the system should never do something, the interface should make that thing impossible. Containment is not a constraint on the user. It is a protection for them.

The Principle

Human Always in the Loop

This is not an autonomous system. At no point in this workflow does the AI act without explicit human confirmation. Every recommendation requires a deliberate human decision to proceed.

That is not a technical limitation. It is a governance commitment – and it had to be visible in every screen, at every step.

The receptionist sees the transcript before the recommendation. They see the reasoning before the confidence score. They see the pharmacy context before the referral button. They confirm the action before anything is sent. And when the system determines a case is outside its scope, it tells them clearly and removes the unsafe option entirely.

The AI does the cognitive work – transcription, matching, scoring, capacity checking, outcome tracking.

The human does the accountable work – reading, verifying, deciding, confirming. That division of labour is not accidental. It is the design.

In clinical environments, human oversight is not a safety net for when the AI fails. It is the intended operating model. The AI earns the right to make recommendations by making its reasoning visible. The human has the right to act on them by understanding what they’re confirming.

Learnings

Governance is not a layer you add to a design. It is a series of decisions you make while designing – about what the system can do, what it cannot do, who is accountable at each step, and what the interface makes possible or impossible.

Every screen in this prototype contains a governance decision. Most of them are invisible when the design is working correctly – which is exactly the point.

The hardest lesson

The most dangerous moment in clinical AI design is not when the system gets something wrong. It is when the system gets something wrong and the human has no way of knowing. That is the failure mode that causes harm. Every decision in this case study was made to prevent it.

The flag-not-edit decision was the clearest example – a small interface change with large accountability implications.

The absent button on the high-risk screen was another example – not because it was technically complex, but because it made the governance commitment visible as a design output rather than a policy document.

Build the audit trail before you build the feature.
Show your reasoning before you ask for trust.
Make the wrong action impossible instead of warning against it.

The Path to More Autonomy

The level of human oversight built into this system is not the permanent state, it is the starting point. Trust in clinical AI is not granted, it is earned.

Through consistent accuracy, transparent reasoning, and a demonstrable audit trail that shows the system behaving as designed across thousands of interactions.

As that evidence accumulates, the case for increasing autonomy becomes arguable. Routine cases with high confidence scores and consistent outcomes might eventually move through the workflow with less friction.

The speech-to-text layer – which some might question as unnecessary if a human is confirming every referral anyway – becomes the foundation for that future.

Every flagged correction, every confirmed transcript, every outcome logged against an AI recommendation, and every signal about pharmacy capacity and accredited workforce availability, becomes training data for a system that will eventually earn the right to act with less supervision.

You don’t get to autonomous clinical AI by starting there. You get there by proving, step by step, that the system can be trusted.

Containment is not anti-agentic, it’s pro-safety.

And paradoxically, the less authority a system has in the beginning, the faster it can safely move toward more autonomy.

This is step one toward that progression.