AINorthern IrelandBusiness

Questions To Ask AI Vendors: A Procurement Checklist For Councils

A practical list of questions councils should ask AI vendors during procurement: data handling, accuracy, accessibility, security, transparency, exit and support.

Geoffrey Moffett 4 Min Read

Quick Answer

Before buying AI, councils should ask vendors where data is stored and processed, whether it's used to train models, how accuracy is tested and monitored, how bias is addressed, how the product meets accessibility standards, what security certifications apply, how costs scale and how the council can leave with its data. Answers should be in writing and assessed by information governance, IT and legal colleagues.

Key Takeaways

  • Define the problem first, so you can judge whether the product solves it.
  • Data handling and training use are the most important questions.
  • Ask for evidence of accuracy testing, not just demos.
  • Plan your exit before you sign.
In This Article

AI is in every sales pitch now: chatbots, transcription, document analysis, "smart" customer service. Some products are genuinely useful. Others are an old product with a chat box bolted on. Knowing the right questions to ask AI vendors protects your council's data, money and reputation, and helps you buy things that actually work.

This is general guidance. Follow your council's procurement rules and involve procurement, legal, information governance and IT colleagues.

Before Any Questions: Define The Problem

Write a one-page problem statement:

  • What task or service are we trying to improve?
  • Who uses it (staff, residents, both)?
  • How do we do it now and what does it cost in time?
  • What would success look like in six months?
  • What's the risk if the AI gets it wrong?

If you can't answer these, you're not ready to buy.

Questions To Ask AI Vendors About Data

  • Where is our data processed and stored? In which countries?
  • Is our data, including prompts and uploaded documents, used to train or improve your models or anyone else's? Can we switch that off contractually, not just in settings?
  • Which sub-processors, including AI model providers, do you use?
  • How long is data retained, and can we delete it?
  • Can you support our data protection impact assessment with documentation?
  • How do you handle personal and special category data?

The ICO's guidance on AI and data protection is a useful reference for what good answers look like.

Questions About Accuracy

  • How do you test accuracy? Can you share the method and results?
  • How does the product behave when it doesn't know the answer?
  • Can we test it with our own content and typical questions before committing?
  • How do you monitor accuracy after launch, and what reports will we see?
  • How are errors reported and fixed?

Demos are designed to impress. Ask to run your own test questions, including awkward ones.

Questions About Bias And Fairness

  • How have you tested for bias across different groups?
  • Can the product be used in a way that affects decisions about individuals? If so, what safeguards exist?
  • What information can you provide to support our equality screening?

Questions About Transparency

  • Can residents be told clearly when they're interacting with AI?
  • Can we explain, in plain terms, how the system works?
  • Will you support us if we publish information about the tool, for example using approaches like the UK Government's algorithmic transparency recording standard?

Questions About Accessibility

  • Does the product meet the accessibility standards public sector websites and apps must follow?
  • Can you provide an accessibility statement or audit?
  • How does it work with screen readers and keyboard navigation?

Questions About Security

  • What security certifications and testing do you have?
  • How is access controlled? Does it support our single sign-on?
  • What's your incident notification process?

Questions About Language And Local Context

  • How well does it handle Northern Ireland place names, local terms and accents (for voice tools)?
  • If language support is relevant to your council, what's the quality of any Irish or Ulster-Scots support, and how would outputs be checked by qualified people?

Questions About Cost And Contract

  • How does pricing scale with usage? What happens if usage spikes?
  • Are there extra costs for integration, training or support?
  • What's the minimum term?
  • How will price changes be handled?

Questions About Exit

  • How do we get our data and configuration back, in what format?
  • What happens to our data when the contract ends?
  • Can we switch providers without starting from scratch?

Questions About Support And Training

  • What training do you provide for staff?
  • Who do we contact when something goes wrong, and how quickly will they respond?
  • How do you communicate changes to the AI model behind the product?

That last one matters. When an underlying model changes, the product's behaviour can change too.

Scoring Answers

Create a simple scoring grid with your colleagues. Weight data handling, accuracy and exit highly. Insist on written answers. Vague responses such as "we take security very seriously" should score low.

A Made-Up Example: The Transcription Tool

Imagine a supplier offers an AI tool to transcribe and summarise meetings. The demo is impressive. The council's digital team asks to test it with a recording of a public, published meeting featuring local accents and place names. The transcript struggles with several place names and mixes up two speakers. The supplier's written answers also reveal that recordings are processed outside the UK by a sub-processor not mentioned in the sales material.

None of this means the tool is unusable, but it changes the conversation: about accuracy checks, about data location and contracts, and about which meetings it would be appropriate for. The council would only have discovered these issues after signing if it had relied on the demo.

Pitfalls To Avoid

  • Buying the demo. Test with your own data.
  • Ignoring AI in existing contracts. Review supplier updates.
  • No success measures. Define them before the pilot.
  • No internal capability. Someone in the council needs to understand the tool well enough to manage it.

For the governance that sits around this, see our guide to an AI policy for councils.

How Starien Can Help Train Council Teams

We help organisations understand AI well enough to ask good questions and use tools well. Our AI Coaching can build that confidence in your digital, comms and procurement teams. Get in touch to talk about a session.

Frequently Asked Questions

What are the most important questions to ask AI vendors?

Where data is processed and stored, whether it's used to train models, how accuracy and bias are tested, what happens when the AI is wrong, how it meets accessibility and security requirements, how pricing scales and how you can exit.

Do AI features in existing software need assessment?

Yes. When an existing supplier adds AI features, ask the same questions and consider whether your data protection and equality assessments need updating.

Should councils pilot AI tools before buying?

Where procurement rules allow, a time-limited pilot with clear success measures and test data is a sensible way to check real-world performance before committing.

Written By

Geoffrey Moffett

More Articles
More In AI

Back To Top