Challenge

The team working on the Intelligent Assistant for a large financial institution had run into an hiccup.  Engineering and UX Design needed to decide which of two potential concepts to implement as a stop gap solution while the hiccup was fixed.  We needed to act fast while still mitigating the risk of a poor user experience in the stop gap turning users off of the using the full feature once it was fixed and implemented.  

Here’s when I step up as UX Research.  First, I ran a kickoff workshop with my Product, UX Design and Conversation Design stakeholders to align and prioritize the most important questions we needed to answer.  After the workshop but before conducting any studies, I reviewed our extant research again to determine what questions had already been answered.  This allowed me to focus the studies on collecting new insights. 

Solution 

In order to address all the questions my team was working through, I decided on a two prong approach utilizing unmoderated studies using Userzoom.

The first prong was to run 2 unmoderated usability tests (1 for each design) with 10-15 users each.  As the tests used the same tasks, scripts and collected a Single Ease Question after each task, they could be compared.  These tests showed us where users failed in the intended flow andwhat trouble shooting tactics they used.