# Measuring AI Impact In Mid Ulster Without Vanity Metrics

URL: https://www.starien.com/blog/measuring-ai-impact-in-mid-ulster-council
Published: 2026-10-23
Author: Geoffrey Moffett
Topics: Mid Ulster, Mid Ulster District Council, measurement, performance, local government, AI

> How a council like Mid Ulster could measure the real impact of AI in 2027: time saved, quality, resident outcomes and risks, rather than counting logins and posts.

## Quick answer

Measuring AI impact in Mid Ulster council teams means looking at what changes for staff and residents: time saved on specific tasks, quality of outputs, faster or clearer services and fewer errors, alongside risks and problems. Logins, prompts and post counts are vanity metrics. Use simple before-and-after comparisons and review measures regularly as AI use evolves.

## Key takeaways

- Count outcomes, not activity: time saved, quality and resident benefit.
- Measure specific tasks before and after, rather than vague overall impressions.
- Track risks and errors as well as benefits.
- Keep measurement simple enough that busy teams will actually do it.

"We've trained 120 staff and they've used AI 5,000 times this quarter." It sounds impressive, but what did it achieve? Did residents get faster answers? Did officers spend more time on complex cases? Did anything go wrong? Measuring AI impact in Mid Ulster council teams, or any council, means looking past activity to outcomes.

Here's a practical approach for 2027 that a council like Mid Ulster could adapt. It's general guidance, not a description of current practice, and because AI use will evolve quickly, the measures themselves should be reviewed every few months.

## Why Measuring AI Impact In Mid Ulster Matters

Councils are accountable for how they spend time and money. Elected members, auditors and residents will reasonably ask whether AI is worth it. Good measurement helps:

- decide where to invest more effort;
- stop doing things that don't work;
- spot risks early;
- explain the value honestly.

For a district with practical, no-nonsense expectations, honest numbers matter more than glossy claims.

## The Vanity Metrics To Avoid

These numbers are easy to collect and tempting to report, but they say little on their own:

- logins and active users;
- prompts or queries sent;
- documents or posts produced;
- staff trained;
- chatbot conversations started.

They can be useful context, but never the headline.

## What To Measure Instead

### 1. Time Saved On Specific Tasks

Choose real tasks, such as drafting a standard letter, summarising a report or producing a set of social posts. Estimate how long they took before. Then measure with AI, including time spent checking and correcting. Report the difference honestly, even if it's small.

### 2. Quality

Faster isn't better if quality drops. Ways to check quality:

- peer review of a sample of AI-assisted outputs;
- readability scores for public-facing text;
- the number of corrections needed before sign-off;
- feedback from the people who receive the output.

### 3. Resident Outcomes

Where AI touches public services, look for changes residents notice:

- response times for enquiries;
- repeat contacts about the same issue, which often signal unclear information;
- complaints about clarity or accuracy;
- satisfaction feedback.

### 4. Staff Experience

Do staff feel AI helps or adds pressure? A short survey every six months can track confidence, workload and concerns.

### 5. Risks And Errors

Track:

- errors caught before publication;
- errors that reached the public;
- data protection concerns raised;
- incidents involving AI tools.

A rising number of caught errors can be a good sign that checking works.

## Keep It Simple

Busy teams won't fill in complex forms. A simple approach:

- each department picks two or three tasks to measure;
- a quick log records time and corrections for a sample of those tasks;
- results are reviewed quarterly by the AI champions or a small working group;
- a short summary goes to senior managers and members.

## A Simple Quarterly Scorecard

A one-page scorecard could include:

- tasks measured and average time saved;
- quality checks and corrections;
- one resident outcome measure per relevant service;
- staff confidence score;
- risks and incidents;
- what we'll change next quarter.

## Context From Mid Ulster's Shape

A council covering several towns and a big rural area might also look at whether AI helps reach people more evenly. For example, does producing local versions of messages more quickly lead to more responses from rural areas? That's a meaningful outcome worth tracking.

## Spring And Summer 2027 Timeline

- **Early spring:** agree measures and baseline a few tasks.
- **Late spring:** first quarterly scorecard.
- **Summer:** second scorecard, with a review of the measures themselves.
- **Autumn:** report to members on what worked and what didn't.

AI tools will change during the year, which may change what's worth measuring. That's expected.

## A Made-Up Example

A made-up customer services team measures how long it takes to draft replies to five common enquiry types. Before AI, it averages fifteen minutes per reply. With an approved AI tool and checking, it averages nine. Corrections are needed on about a third of drafts, mainly local details. The team adds those details to a shared fact sheet, and corrections fall the following quarter. It's modest, honest and genuinely useful evidence.

## Reporting To Elected Members

Members will want a balanced picture. A good update is short and honest:

1. what we tried and why;
2. what changed, with numbers where they're reliable;
3. what went wrong and how it was handled;
4. what it cost in time and money;
5. what we'll do next, and when we'll review it.

Avoid overselling. If something saved little time but improved quality, say so. If something didn't work and was stopped, that's a good story about responsible management, not a failure.

## Measuring Business Support Too

If a council runs AI skills sessions for local firms, the same principle applies. Rather than counting attendees, ask businesses a few weeks later what they changed: a new website page, a faster quote process, a better social post routine. Those answers say far more about value than a sign-in sheet.

## Pitfalls To Avoid

- Reporting activity as impact.
- Ignoring checking time when calculating time saved.
- Hiding errors.
- Measuring so much that nobody has time to do the work.
- Never changing the measures as AI use matures.

## How Starien Can Help

Starien knows the difference between vanity metrics and real results: during six years leading Coleraine BID's digital delivery, we reported engagement growth of more than 40% alongside audience numbers, because outcomes mattered more than counts. Our [AI Coaching](/ai-coaching/) helps council teams set up simple, honest measurement alongside practical AI skills.

We also help Mid Ulster businesses track what matters on their websites and social media, rather than chasing likes. Owners can learn the basics with [Launchpad](/launchpad/), or [get in touch](/contact/).

## Frequently asked questions

### What are vanity metrics for AI?

Numbers that look impressive but don't show value, such as the number of AI logins, prompts sent, posts published or people trained, without any evidence of what changed as a result.

### How can a council measure time saved by AI?

Pick specific tasks, estimate or time how long they took before, then measure the same tasks with AI, including checking time. Small samples are fine if they're honest.

### Should councils measure AI risks too?

Yes. Track errors caught, complaints, data concerns and any incidents. A full picture includes the costs and risks as well as the benefits.
