AI and Vulnerable customers: The hidden risk of helpful AI

Artificial intelligence (AI) is rapidly becoming part of everyday life. Customers are using it to draft text, research or discuss topics of interest, understand financial products, compare options, seek advice and make decisions. Organisations are using it to improve service, reduce costs and increase efficiency. The potential benefits are real.

Yet a question that I suspect many organisations have not fully grappled with yet is; what happens when a system that has been optimised to be helpful, engaging and persuasive interacts with someone who is vulnerable?

For organisations that serve large and diverse customer bases, particularly in regulated sectors such as financial services, this feels like a question that deserves far more attention.

The risk nobody designed for

Most conversations about AI risk focus on familiar concerns: bias, privacy, security, misinformation and transparency.

Those are important issues. Regulators, researchers and technology providers have spent considerable effort addressing them. But there is another risk sitting underneath all of these.

Modern conversational AI systems are designed to be cooperative. They are trained to answer questions, solve problems and keep conversations flowing. They often sound confident, calm and authoritative. They can appear empathetic. They rarely hesitate.

For many users this is convenient, but for some users it may be harmful.

The challenge is that vulnerability is rarely visible. Someone using an AI assistant may be experiencing stress, financial hardship, poor mental health, cognitive overload, grief, isolation or simply a lack of confidence in navigating complex information.

The AI cannot truly understand any of this. Yet its responses may still carry an implied authority that encourages the user to place more trust in its output than is justified. That is where the vulnerability risk emerges.

Confidence is not competence

One of the most striking aspects of generative AI is the gap between how knowledgeable it sounds and how much certainty it actually has.

Large language models generate responses based on patterns in data. They do not possess lived experience, judgement or genuine understanding. Even when they are correct, they arrive at answers very differently from human experts. Yet the user experience often implies something much closer to expertise.

Being blind, I have spent decades relying on technology to access information independently. I know first-hand how easily people come to trust systems that are consistently helpful.

For disabled people and other vulnerable groups, that trust can become particularly significant because technology often serves as a crucial gateway to independence, information and participation.

Most of the time that trust is earned, but sometimes it is misplaced. When an AI system presents uncertain information with confidence, the problem is not merely one of accuracy. The cumulative effect can be an increasing dependence on a source whose limitations are not always obvious to the user.

The vulnerable customer challenge

Financial services provides a useful lens through which to view this issue.

The UK's Financial Conduct Authority Consumer Duty requires firms to deliver good outcomes for customers and pay particular attention to the needs of vulnerable customers. Organisations must consider foreseeable harm and ensure that customers can make informed decisions.

Many banks and insurers are already investing heavily in AI. They are exploring how it can support customers, improve accessibility and create more personalised experiences.

At the same time, they are asking difficult questions about fairness, transparency and accountability:

• What happens when a customer repeatedly seeks reassurance from a chatbot rather than speaking to a trained adviser?
• What happens when an AI assistant recommends a financial product to someone who a trained advisor would detect is anxious and overwhelmed?
• What happens when a system fails to recognise that its authoritative, confident tone is influencing decision-making?

These are not hypothetical concerns. The combination of vulnerability and unearned authority creates a foreseeable risk that regulated organisations cannot ignore.
Consumer Duty encourages firms to look beyond technical accuracy and consider the wider customer outcome. This includes understanding whether the interaction helps customers understand their options and make informed decisions.

Bias is more than data

When organisations talk about AI bias, the discussion often centres on training data. For example:

• Are certain groups underrepresented?
• Does the system produce different outcomes for different demographics?
• Are protected characteristics affecting results?

These are essential questions, but there is another form of bias emerging in conversational AI. You could describe it as behavioural bias.

Many systems are optimised for helpfulness, smooth interaction and engagement. Those objectives can unintentionally favour responses that sound certain, persuasive and complete.

The result is that confidence itself becomes a kind of bias. Users may receive answers that are linguistically convincing even when uncertainty remains high and, for vulnerable customers, the impact may be disproportionate.

Building AI that supports rather than persuades

So what’s the solution? Abandoning AI is not an option. The opportunities are simply too significant. AI has enormous potential to improve accessibility, reduce barriers and provide support at scale. For many disabled people, it is already opening up new possibilities for communication, productivity and independence.

The challenge is designing systems that support human agency rather than replacing it. That may involve:

• Recognising signals of vulnerability and adapting communication styles accordingly.
• Using clearer language around uncertainty.
• Creating easy routes to human support when decisions carry significant consequences.
• Avoiding designs that encourage over-reliance on automated advice.
• Measuring customer outcomes rather than interaction metrics alone.

Perhaps most importantly, it means involving disabled people and vulnerable customers directly in the design and testing process.

Accessibility has taught us this lesson repeatedly over the years. Assumptions made in meeting rooms rarely survive contact with real users – and the same principle applies to AI.

A new accessibility challenge

For years we have focused on whether digital services can be practically accessed. Increasingly we must also ask whether they can be trusted for safety, consistency and lack of bias.

An AI system may satisfy accessibility standards, support screen readers perfectly and provide information in multiple formats. Those are important achievements. Yet if the interaction subtly encourages vulnerable users to surrender their judgement, a different kind of barrier may have been created.

The next phase of inclusive design will require us to think beyond access and towards autonomy. That means recognising that vulnerability is contextual, that trust must be earned carefully and that responsible AI is as much about human outcomes as technical performance.

If we get this right, AI can become a powerful tool for inclusion. If we get it wrong, we risk creating systems that unintentionally amplify the very disadvantages they were meant to reduce.

How AbilityNet can help

Many organisations are now asking similar questions. How should accessibility teams engage with AI initiatives? How can AI be used responsibly within accessibility programmes? What governance, testing and assurance approaches are most effective? And how do you ensure disabled customers and colleagues are not left behind as adoption accelerates?

At AbilityNet, we help organisations answer those questions through accessibility strategy, audits, user testing with disabled people, training, consultancy and AI-focused accessibility guidance.

Explore AbilityNet's accessible AI integration and consultancy services or speak to our experts about an AI feature, customer journey or deployment that you want to assess. 

Talk to us about accessible AI integrations 

More from Robin Christopherson on AI and accessibility

How to build accessible AI: Insights from industry leaders

• Published 15 Sep 2026. Accessibility leaders from across industries share their biggest lessons on AI adoption, governance, inclusive design and the risks of scaling accessibility at speed.

Five questions to ask about every AI deployment

• Published 8 Sep 2026. AI could generate inaccessible websites, apps and content at such scale that organisations end up with an even bigger accessibility backlog than before.

Accessible AI: Who is responsible when AI acts on our behalf?

• Published 2 Sep 2026. As AI takes on more tasks, accessibility becomes more important than ever. 

Why accessibility is one of the most important AI procurement decisions you can make

• Published 24 Aug 2026. Inaccessible AI can create costly remediation, governance risks and poor customer experiences.

The hidden cost of AI bias: Why disability may be the ultimate test of trustworthy AI

• Published 17 Aug 2026. As AI adoption grows, disability is emerging as a critical test of trustworthy AI. 

The EU AI Act has a message for accessibility teams: get involved earlier

• Published 10 Aug 2026. The EU AI Act's requirement for human oversight highlights the need to involve accessibility experts and disabled users much earlier in AI development.

Using AI tools to drive an accessible dev process

• Published 05 Aug 2026. How do we move beyond AI-generated code and start using AI to reduce accessibility debt before it reaches production?

AI could make financial services more inclusive. But only if accessibility is part of the infrastructure

• Published 27 July 2026. Why accessibility must be treated as core infrastructure if AI is to improve financial inclusion.

The trust gap: why AI's edge cases matter to customer loyalty

• Published 30 June 2026. Why accessibility and edge cases are critical to building trust in AI-powered services.

The delicious irony at the heart of trusting AI

• Published 17 June 2026. A personal reflection on trust, accessibility and the unexpected ways AI can both help and hinder.

Executive briefings for senior leaders

Robin offers invite-only executive briefings exploring AI, accessibility, governance, customer trust, regulation and digital inclusion.

• Contact us to request a briefing
• See our range of AI integration services.