In Love with AI
While getting a master’s degree at Utrecht University, I spent a year exploring human perception of Artificial Intelligence (AI), and the results are fascinating. I found out that people tend to hugely undervalue advice or prognoses if they’re coming from AI. The tendency of neglecting AI-made advice has its reasons rooted from the innovative character of the technology, and I outline potential countermeasures in the second half of the article. But first, let’s paint a big picture.
We live in times when everyone is talking about AI, ChatGPT writes your emails and (some) students’ thesis, and Midjourney has generated your social media profile picture. The progress in AI industry runs like a snowball reporting new and new capabilities. We (a collective we) started developing AI technology to be able to process large quantities of data quicker and with less errors, and the goal is largely achieved. Look at the AI efficiency chart below.
There’s evidence that various AI-based automations make fewer mistakes and produce more accurate advice and predictions than human experts. Reviewing academic literature on the subject I found examples from healthcare (AI is more accurate than an average doctor in certain diagnostic techniques), education (AI predicts students performance and does it better than a trained professional), weather forecasting and other domains.
Q: Does it mean that people happily line up for an AI-provided diagnosis?
A: Not quite so.
Let’s hold a hand of a user of the exciting AI technology for a moment. In my experiment, I asked participants a series of choice questions. They were given a prediction and a choice to take an action considering the risks that were stated (numerically) in the prediction. For example: “It was predicted that tomorrow the chance of hail is forty per cent. Would you plan a picnic outside?” Simple. But I added some serious questions about trying a new medicine with possible side effects and so on to see how the answers differ for high-risk cases. One group of people were told that prediction were made by human experts and the second — that they were made by AI. The results show that independently of the risk level subjects more often agreed to accept an option in the human-expert group. In the AI group people more often rejected a choice option. What does it tell us? No matter how light or risky a decision is, people prefer (and trust) a human adviser (not AI). But why?
The AI efficiency chart suggests that every rational person must trust capable AI rather than a human, but in reality the opposite takes place. As we saw in my experiment, people show higher trust levels towards a human compared to AI. That phenomenon is re-discovered for each new technology the world of people is introduced. First, people didn’t trust computers, then e-commerce, then internet-banking. You see the trend. Innovations bring new convenience and unknown risks. AI triggers suspicion and mistrust in a great magnitude due to invisible and mysterious nature of the technology. It’s invisible, black-box like (in the data-scientists jargon), it’s hard to control the quality of its output. AI’s creators themselves support the mysterious narrative by insisting that every self-learning (AI) model has a mind of its own. Ultimately, the user is left wondering about how an AI-advisor came to the conclusion it did.
From the human perception stand point, the areas where AI is losing to a person are interactivity and familiarity. Let’s begin with the latter. In general, we know what level of professional expertise to expect from a doctor (or any other specialist). We know that every doctor goes through many years of professional training. We can eyeball their competency by the amount of grey hair and deep circles under their eyes. And we expect that they ask colleagues for a second opinion in difficult or ambiguous cases. Human mistakes are known to take place (sometimes) but, hopefully, not in our situation. Plus we are likely to have prior experience of getting useful medical advice. Now, about AI. Who created it? On what dataset it was trained? What is the model accuracy? How many patients did this model serve and with what success rate? Commonly, the user isn’t aware of those facts, hence it’s difficult to have any expectations. And if we don’t know what to expect, we decide to take the pessimistic approach and massively undervalue received information.
Interactivity problem has similar roots. When hearing advice/prognosis from a doctor, we can ask for explanations. If a mistake is uncovered, we blame the doctor who made it. AI, on the other hand, doesn’t give additional explanation on demand. And if a mistake or miscalculation is made, there is no one to hold responsible and that adds to the pain.
Q: Are we doomed to be unsatisfied with AI?
A: That would be sad because AI-generated predictions often are more accurate than human-made.
The secret of success is better connection to the user (a generic “the user” that is represented by a crowd of people interacting with your product). One day CTOs and business executives will realise that every data science lab needs a cognitive psychologist (that often disguise themselves as UX Designers), so those labs can build something human digestible.
Human-AI interaction operates in the same human perception space as more traditional human-computer interaction. And problems that arise in that space can be solved by user research.
Approaches will differ per product, but we can list a couple of generic UX tips here.
1. Improve transparency of AI mechanics. Write the core principles of your AI model in a simple human language and clearly state its accuracy.
2. Talk to your user to find out if they are able to calibrate expectations of your AI performance with time. If in two to three months of interaction the user is still surprised by the generated outcome, something needs more work.
3. Collect feedback and feature requests from your users to plan how your AI can facilitate to more of their tasks in the future.
Starting user research as early on as possible and keeping a strong connection to our users is essential, if we want them to adopt our innovations.
There are more things you can and should do, but those depend on the specifics of the AI you are developing. If in doubt that some of given advice, for instance transparency, is beneficial to your business, stay tuned we will discuss that sensitive subject separately.
Let’s look at a practical example from my own work life. I witnessed a data team developing an AI for processing incoming documentation on a SaaS platform. The AI model did clustering sort of analysis to detect anomalies and potential fraud. The stated benefits of the model included automatic error-free processing of large volumes of data. The clients, on the other side, were interested in sifting the data through AI filter in order to spot inconsistencies and anything fraud-looking, so that the staff could investigate the found anomalies further.
I spoke to the users to explore the AI data processing features that satisfied their expectations and those that didn’t. What I saw was an odd picture. Because our users couldn’t understand the AI’s working principles they invented all kinds of workarounds to double check as many documents as they could. Since some of the indicated by AI anomalies were simple misinterpretations of the data, people doubted the effectiveness of AI. Consequently, the users weren’t utilising the technology as intended. In addition, my research participants expressed a wish to interact with the AI updating its knowledge of fraudulent schemes (fraud is morphing and changing shapes every so often).
In this example, we see a confirmation of transparency and interactivity request from technology consumers. Being left with a black-box, the user is forced to come up with time costly workarounds and the company that’s paying for automation don’t get full benefits.
Conclusion
People don’t always behave in purely rational ways, they are complex beings. Their motivation is driven by emotions and not immediately obvious. Starting user research as early on as possible and keeping a strong connection to our users is essential, if we want them to adopt our innovations.
