GOV.UK is the digital front door to UK government services, and as a citizen, your journey in engaging with these services begins with search. Whether you’re looking for benefits, housing, or to renew your passport, being able to easily access this information quickly makes for a much smoother citizen experience.

At CGPN last year, Karl Hampson, Chief Technology Officer, Data and AI at Valtech and Kin + Carta delivered an engaging case study presentation on his work with GOV.UK and how they transformed the citizen experience with improved search.

The challenge

Millions of users visit GOV.UK every week, and with over 700,000 pages to scan through, it’s important that citizens can find the information they need quickly and easily. However, despite a wealth of information, GOV.UK’s traditional search capabilities struggled with relevancy, usability and future-proofing. That’s where Kin + Carta came in.

Collaborating closely with the Government Digital Service (GDS), Kin + Carta set out to solve these challenges with three clear objectives. As Karl explained, ‘One was to improve the relevancy. The second was to make the platform easier to manage. And the third was to future proof the technology and use an evidence-based approach to select a new piece of search technology.’

The timing of this project was also very interesting, as ChatGPT went public just days after Kin + Carta came onboard, marking a pivotal moment in the mainstream adoption of AI. ‘This put AI in the hands of us all and it made this really mystical technology tangible for absolutely everybody and it became number one topic in boardrooms around the world. And we knew that we were going to engage at a really special time in the history of AI with this piece of work.’

The process

Transforming GOV.UK’s search wasn’t just about implementing a new tool, it required careful strategic planning rooted in discovery, experimentation and data-backed decision-making. Here are four steps the team took before selecting their search engine.

• 1 – Discovery and requirements gathering

The first thing Karl and his team did was to discover what was going on with the existing search. ‘We looked at the pain points and the issues with relevancy. We then gathered the broader functional requirements and non-functional requirements from the GDS team and mapped these to what we call our CSPS framework.’

• 2 – Vendor shortlisting

With the CSPS framework in hand, the team then approached a list of vendors and shared their requirements for the project. ‘We used then our framework to score them and reduce them down to a shortlist.’ Each shortlisted vendor was then invited for a two-week proof of concept (POC) trial, taking the technology, indexing the 700,000 pages from GOV.UK, and optimising the selected products for relevancy to see how they did.

• 3 – Proof of concept (POC)

So how do you measure a search engine? To evaluate the performance of the vendors, Karl and his team developed a judgement list system. As Karl explained, ‘you have a list of search queries and you then you have a curated set of results that you would expect to see for those queries. You can either create that manually, or you can use technology to go and pull that from the telemetry from an existing search deployment, for example.’ He continued, ‘We use both of those techniques to create actually a range of judgement lists that address all of the different relevancy problems that we saw on GOV.UK.’

• 4 – Blind testing

To ensure unbiased results, Kin + Carta conducted a range of blind tests. ‘These four POCs that had been created were put side by side and we let the GDS team go and do their own testing, and we asked them to score which one they thought returned the best results.’

Luckily the automated, manual and CSPS results all pointed to one winner – Google’s Vertex AI Search.

Why Vertex AI?

But what exactly is Vertex AI, and why was it the right choice for GOV.UK? While Google needs no introduction to the world of search, what sets Vertex AI apart is its ability to bring together very traditional keyword search, word embeddings and a data powered approach. As Karl explained, ‘These three things together mean that you’ve got this relevancy, recall and ranking working brilliantly well, and it is absolutely a next generation from previous keyword technologies.’

After they selected the technology, the process didn’t end there. To validate their decision, the team implemented the new search engine on the live GOV.UK site and ran a large-scale A/B test – splitting real user traffic between the old system and the new one. The results spoke for themselves: Google’s Vertex AI clearly came out on top, delivering more relevant results and a better overall experience for users.

Real-world impact

The implementation of the new search has already delivered amazing results, and Kin + Carta have calculated that every year:

• 3,285,000 more users will find what they need on the first click
• 2,529,450 less users will need to refine their initial search.

In short, the upgrade has made search more accessible and intuitive, especially for those citizens who might not know the ‘right’ keyword to use.

Speaking about the results, Karl stated, ‘When you take a piece of contemporary search technology and then you add or you upgrade it to something new, you don’t expect to see this kind of uplift… so I think this is really telling us how much better this new technology is.’

Managing search

With the new system up and running, the next phase was about what comes next: support, iterate and improve this service.

Now you might be thinking, ‘how much maintenance does a search engine actually need?’ Well, let’s put it this way – every time you go to Google to search something, it just works. That’s no accident. And that’s because Google have got thousands of engineers looking at and testing their search engine every day.

As Karl put it, ‘It’s like not taking your car for a service and expecting it to not break down at some point.’ He continued, ‘A search management approach for this new platform was obviously super important. User behaviour changes, the content changes and of course for this use case, world events change.’

Join our future discussions

Thank you to Karl for his expertise and thought-provoking insights. If you’d like to take part in future discussions register your interest to join us at CGPN on 14-15 October!