I was invited to help a company who was granted authority to test a new AI-enhanced intelligence platform. However, analysts were not using the new platform, preferring to use their old methods and tools. With contract renewal deadlines looming, the company needed solutions that would appeal to end users and senior stakeholders alike to ensure the longevity of the product.
Why weren’t users using the amazing AI?
The goal was to understand the barriers to adoption and to work with the delivery team and end users to find solutions that could be delivered within the remaining sprints and budget before contract renewal decisions were finalised.
The main challenge was to conduct a user study that included end users who were working in remote inaccessible places and could only communicate by email.
Despite receiving excellent feedback on the beta product in the lab, I heard that they were using their old methods in the field, not the new system at all.
How we uncovered barriers to adoption
Direct observation in the end users work setting was not possible and remote user testing over a network connection was difficult.
Users conducted a short diary study recording their expectations and impressions of how they used the system in the field. I compared their experiences with those of end users in the lab.
I also administered the system usability score (SUS) by email, which took minutes to complete and is known to reliably differentiate between usable and unusable products with a small sample size.
A comparison of scores between the lab and live phases, showed that users rated usability as excellent in for the lab (SUS 85), and good (SUS 70) for the live system, so usability was not the issue per se. I noticed that users of the live system gave very low scores to the question “I think that I would like to use this system frequently”.
Disconnect between lab and real world
During beta testing in the lab, the end users learned the value of working with AI to automate routine tasks and get jobs done quickly.
AI-generated outcomes included a 0-100% “confidence level” to show how close the AI gets to a correct prediction, which helped them to decide whether to trust the outcomes.
Users expressed excitement about the prospect of using AI in the field.
However, in the real-world, the diary study showed that sometimes the results showed a high confidence level, even though they could see the answer was wrong.
What was happening?
With my direction, machine-learning engineers discovered that analysts in the field were using data sets that were very different from those the AI had been trained on in the lab.
This led the AI to return inaccurate results with a high degree of confidence.
Since the end users didn’t know about the data drift, they assumed the AI was stupid or broken and didn’t trust any of the results.
Changing mindsets
Armed with these insights, I joined forces with the engineers to ideate solutions to increase transparency. Solutions including visualising various types of drift, allowing users to select between models that best fit the data, and options to automatically select the best model for the data.
I then designed interactions that modified users behaviour and enabled them to monitor performance of the AI models.
Product owner empowerment
The UX insights empowered the product owner to discuss end user needs with senior stakeholders and to agree priorities and co-create value propositions that met users core needs and stakeholders desires.
From users to promoters
The recommendations for transparent designs for explainable AI fostered trust development between end users and outputs from AI models – a prerequisite for adoption and reliance behaviour. This directly led to improvements in net promoter scores, which met the company’s key performance indicators.