AI Snake Oil – book notes and reflections
I’ve been reading about AI and emerging technologies more widely again this year. AI Snake Oil: What Artificial Intelligence Can Do, What It Can’t, and How to Tell the Difference is a new book by Arvind Narayanan and Sayash Kapoor. They also run a blog and newsletter. My introduction to the book was this Tech Policy podcast.
AI Snake Oil is based on academic research and testing around the capabilities, claims and hype around AI. There’s a lot in there and I listened to the audiobook. I’m sharing a summary of the notes I made with some of my reflections.
This isn’t an in depth or comprehensive book review but reflects what I found useful and most interesting. My thoughts relate specifically to the work I’m involved in with the UK public and third sectors.
To structure my notes, I’ve started by using the definitions the book introduces for three distinct types of AI that we can consider:
- Predictive AI
- Content Moderation AI
- Generative AI
Predictive AI
AI Snake Oil strongly argues that Predictive AI is not that good. That it’s driven by hype and myths about the accuracy of predictive AI models. The book talks about the incentives of those evaluating these models, with much of this research controlled and owned by the tech companies developing and selling Predictive AI products.
There are a number of examples used to illustrate this (mostly US based companies), from the HireVue hiring platform to Epic’s Sepsis AI prediction solution.
The authors share data to show problems with the accuracy and effectiveness of these models and also describe why these types of solutions are so appealing. For example, with HR support and as discussed further in the Podcast I linked to in the introduction:
“…automation is so appealing in hiring because companies are getting hundreds, perhaps thousands of applications per position and that points out something that’s broken in the process, but then it seems appealing to try to filter through all of those candidates with AI. And even if AI in this context is not doing much, we think that a lot of these AI hiring tools are just elaborate random number generators from the perspective of an HR department that is swimming in the sea of applications. It’s done the job for them. It gives them some excuse to say we’ve gotten it down to these 10 candidates and so there are often underlying reasons why we think broken AI gets adopted.”
Content Moderation AI
AI Snake Oil is also not positive about AI as a moderation tool. It summarises that Content Moderation AI is bad at context and nuance, and that it is much worse than human moderators.
The book again highlights the power of tech companies as a key problem here and talks about the negative impacts or consequences people might face because of the algorithms and types of automation being applied to content moderation. This includes questions raised around how it becomes the role of social media companies to decide what is and isn’t acceptable:
“…as society we can’t agree about what we want out of content moderation.”
In summary, content moderation is a hard problem with unclear boundaries. The book talks about how people will always learn to work around content moderation and even use it in harmful ways – there’s an example shared of police officers playing copyrighted music so that people filming them can’t upload and share content as easily to social platforms.
Generative AI
Focussing on Generative AI, AI Snake Oil starts by talking about ChatGPT as “automated bullshit. It describes how ChatGPT hype came from its choice of benchmarks – that it was able to pass the legal bar and medical exams. The book argues that this tells us very little about how ChatGPT is suited to automating jobs in the real world.
There a useful description of real jobs as “bundles of tasks” – the book concludes that AI will potentially reshape those tasks, but this doesn’t mean it has to replace jobs.
My reflections here are that we’re starting to see the limits of Large Language Models (LLMs) being exposed. A good recent example is the AI summaries being added by Google at the top of search results – I’ve seen a number of examples shared where the AI generated content in Google search results is factually wrong. Arguably, Google are damaging their reputation and trust of ‘search’ with the way they’re now deploying GenAI features.
AI Snake Oil gives clear examples of ChatGPT hallucinating when used as a research or search tool while also being positive overall about the potential of LLMs. The book goes on to highlight how co-piloting with LLMs can provide powerful and valuable tools in the right context, especially where a level or error can be tolerated alongside human work and validation.
There’s an author interview at the end of the audiobook where using co-pilot tools is described, specifically how the authors use LLMs for writing code that supports their own work:
“We have to be careful about not over-relying on AI, but especially in scenarios where the outputs of the code are easier to verify rather than writing the code itself, AI makes a lot of a sense.” e.g. when you’re creating a website you can easily check if its behaving in the right way… “it’s a big productivity boost.”
Examples given as more of a positive future for Generative AI later in the book lean towards image recognition – describing tools like PlantNet (I’ve previously used a similar App called PictureThis) as having the potential to enhance real life experiences, such as helping you identify and talk about plants while on a walk.
The problem of broken institutions
Throughout, AI Snake Oil has a strong theme of how AI is appealing to broken institutions, explaining how the types of problems AI is being applied to are often something much deeper in our systems and organisations.
Along with other examples, the book talks about education as a sector that is “susceptible to quick fix solutions.” This is especially considering the current financial constraints and pressures on educational institutions. It describes how AI is being applied here but with a number of unintended consequences.
In describing how “flawed AI distracts from the core goals of institutions” an example given here is of colleges providing mental health support for students:
“…many colleges want to provide mental health support to students, but instead of building this institutional capability […] many colleges adopted a product called Social Sentinel to monitor students social media feeds.”
The book explains that the accuracy of Social Sentinel was really low and it was even used in harmful ways by some colleges, like for monitoring student protests and other student surveillance.”
Embracing randomness
I liked the ideas in Chapter 8 in a section titled ‘Embracing Randomness’. An interesting point made here is about how AI is often trying to optimise systems or processes in computational terms:
“The embrace of predictive AI comes out of the optimisation mindset, where one tries to formulate a decision on computational terms, in order to find an optional solution and achieve maximum efficiency. The failure of predictive AI is an inditement of this broader approach. When there are multiple valuable goals that can’t be accurately quantified relative to each other, optimisation can backfire badly.”
Building on the earlier hiring example, the book explains the idea of ‘partial lotteries’ which acknowledge the randomness that already exists in decisions, so, for example:
“Instead of trying to pick the top applicant […] who would receive a grant to get into a college, partial lotteries make randomness an explicit part of the decision making process. All applicants who satisfy a certain basic cut off are included in a pool and a random draw is used to see who gets in [and is shortlisted to interview].”
The argument is that some randomness is better than over engineering selection criteria. The book is clear that this approach won’t always be the most appropriate solution, but sets this out as an alternate to formulating decisions with computational logic or AI predictive models that introduce bias.
There’s also a small example included here about grant applications, which would be applicable to an area like funding allocation within UK Government and MHCLG. The book argues that partial lotteries may be effective at “countering rich get richer effects, where academics who have already received grants are more likely to get them in the future.” They could also help “reduce wasted time preparing applications.”
New approaches to efficiency and automation
AI Snake Oil goes on to talk about less risky approaches to finding efficiencies with AI:
“We can aim to find strategies, or policies, that achieve modest efficiency gains while being simple enough to understand, both for decision makers and decision subjects. Simplicity helps decision makers assure themselves that things can’t go catastrophically wrong and build trust with decision subjects.”
This section reminded me of one the themes in Richard Pope’s Platformland book. Richard talks about the need for “new ‘tools’ for understanding” and the need for future public services to be better at “revealing decisions, rules, data and have accountability to users.”
I think there’s a need to simplify how we apply AI and automation at a task based level, with technology that works with caseworkers and decision makers to achieve this. This is the idea of finding lots of small efficiency gains, made possible with the right tools and through targeted automation. My feeling is that this is the potential of work with smaller, closed AI or LLM models inside organisations and systems.
I was also reminded here of Richard’s point that “‘likely’ and ‘probably’ create a safer space for AI to operate in than full automation.”
Finally, with the points here about finding smaller interventions, there’s something important highlighted in AI Snake Oil about the ability to experiment with new technologies in smaller ways. As the book says:
“Efficiency is especially seductive to any organisations that are cash strapped. These organisations might also lack the capacity to experiment with AI and discard it if it doesn’t work out.”
Jevons paradox
When thinking specifically about efficiency, another useful quote is shared from the book Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass:
“Every time a new type of automation is introduced it takes over work previously done by humans, but also creates new types of needs for human labour.”
This reminded me of Jevons paradox, which is also mentioned later in the bonus section at the end of the audiobook. Jevons paradox in relation to automation is explained more in this excellent Tim Paul post.
Institutional incentives
A final quote I bookmarked makes an important point about institutional incentives when AI work is being driven by the need to meet increased demand at reduced costs:
“Even if all the AI companies that make false promises go out of business tomorrow, flawed institutions would turn to some other type of snake oil that promises a quick fix. The demand for AI snake oil here isn’t primarily about AI, it’s about misguided incentives and the failing institutions that adopt them.”
This is probably the most important point in the book. That we can’t see any technology or solution as a ‘silver bullet’ or single, easy fix to deeper institutional problems related to demand, complexity and cost.
AI Snake Oil concludes by saying that we should be far more worried about what people will do with AI, rather than what AI will do on its own. The book argues that we don’t have to accept technological determinism and should push back on the vision of the world being sold to us by AI and tech companies.
My feeling is that we can do a lot better with how these technologies are developed from within the public and third sectors. There are already great examples of UK government departments experimenting with large language models and sharing their learnings in the open. We still have the ability to shape different types of incentives and governance to make good choices about the uses of AI as part of the systems we design and deliver.
This is my blog where I’ve been writing for 20 years. You can follow all of my posts by subscribing to this RSS feed. You can also find me on Bluesky and LinkedIn.