AI content consequences
As reported by the BBC in January, Apple suspended an AI content feature that summarised news headlines after a number of complaints. AI generated summaries, including BBC news alerts sent to Apple users, were found to regularly contain mistakes.
As the BBC’s Technology editor, Zoe Kleinman explained:
“Not only was [Apple] inadvertently spreading misinformation by generating inaccurate summaries of news stories, it was also harming the reputation of news organisations like the BBC whose lifeblood is their trustworthiness, by displaying the false headlines next to their logos.”
This story made me think about the reported inaccuracies in AI content that is now prominent on Google search results pages, with people sharing example AI hallucinations on different social platforms – there was this example from Bluesky, where someone was excited to tell their family about the upcoming sequal to the Disney film Encanto, only for them to scroll down and learn that Google’s generative AI had completely made it up.
Google has been a trusted search engine for a long as most people can remember the internet. It’s arguably now doing irreparable damage to it’s reputation through the addition of AI content to it’s search results pages. This could even be dangerous given the context of what people might be searching for.
I’ve been thinking about how there is a useful comparison point here. There are definitely instances where AI content can be useful. A lot of shopping sites, like Amazon, now have an AI content summary of what might be hundreds of product reviews. These summaries might not be perfect, but alongside things like a star or scoring systems they can give customers a level of assurance about a product they’re browsing. And if a customer is wanting to research something further, they can dig deeper into user generated content, spending time reading the actual posted reviews.
As a design pattern this is becoming more and more common. For example, I often read hiking review summaries on AllTrails (AI content). I know that these are very generalised, but they give me a sense of the overall sentiment of multiple reviews of a hiking route.
What’s important here is the consequences of content being wrong. In the case of AllTrails, I always do a lot more research when planning a trip. This includes looking at maps (online and offline) and even watching YouTube videos or reading other hiking blogs. The consequences of one piece of content being inaccurate on one App is therefore low for me.
In comparison, if my NHS App created AI content that misled me about my health, or how to manage the symptoms of a health situation, that potentially has a very contrasting set of consequences. This is also why so much focus on content design and testing is central to government and health services. The consequences of any misinformation and misunderstanding are extremely high, with GOV.UK and NHS.UK brand associations providing assurance around the accuracy of information being provided.
The recognition of content context
So what we’re now starting to see is more recognition of the importance of content context.
If it’s important and the consequences of a scenario or use case are high, then we have to question any use of AI content. The most straightforward thing to evaluate is: how likely is someone to be looking at multiple sources of information? This could minimise the risk of misinformation if one source of content is inaccurate. But there are further considerations around understanding situations where people could be under time pressure, so may act upon the first information they’re able to access and take as accurate. Therefore, we might need to think about the next steps people will take, and how they might validate or use the information they’ve found. What will they do next?
We also need to consider how likely people are to associate the accuracy of AI content with a brand, and what that brand represents in terms of trust or validation of information. This takes us back to the BBC example, where once trust is broken it takes time to rebuild. This is where I think Google has gone so wrong, whereas Amazon can afford to use AI content as a feature when we already know that it’s vast marketplace where reviews don’t always tell us the truth. You’re still taking a level of risk when making a purchase, but with other types of insurance such as product returns.
Where content consequences are high, Google’s AI content could instead recommend the best places to look for information, along with a written commentary about links or considerations to make between multiple sources. That would better complement a traditional search results listing. It could even prove to be useful search ‘assistant’ without damaging the Google brand further.
Understanding content consequences is recognising that not all AI content is bad. It’s about the careful application of new features that use it, but designed in ways that fully understand context and how different brands validate and build trust in information.
This is my blog where I’ve been writing for 20 years. You can follow all of my posts by subscribing to this RSS feed. You can also find me on Bluesky and LinkedIn.