Part 2 of Designing User-Centric AI/ML Products
To demonstrate the practical application of this framework, let’s explore a real-world example from Twilio. While I was at Twilio, the company was facing the challenge of simplifying onboarding experiences to improve customer retention. By thoroughly examining the market we served and identifying the most impactful opportunities, such as reducing deliverability failures and enhancing conversion rates, we were able to develop over 20 potential solutions. Then through heuristic experiments, we collected valuable data to validate and refine our top opportunities. Focusing on high-impact areas and leveraging the power of AI, my team effectively justified our investment while mitigating the risks associated with developing the wrong product.
Now, let’s dive deeper into the implementation of this framework with a practical example. While at Twilio my team had a goal to reduce customer conversion rates by 20%. Here’s how we applied the steps of the framework to achieve this objective:
1 Hypothesis
In this initial stage, we hypothesized that notifying customers to remove third-party URLs in SMS messages could significantly improve customer conversion rates. Third-party URLs are marked as spam, but the customers didn’t know that.
2 Research
To validate our hypothesis, we conducted thorough research. We discovered that third-party URLs only affected a minority of customers. However, we also found that failed message deliverability resulting from campaign mismatches were the most common user errors leading to reduced conversion rates.
Be diligent in choosing opportunities that are AI/ML problems. Not all problems are AI problems, and can better be solved heuristically. To avoid reinventing the wheel I referred extensively to the Google’s People + AI Guidebook for patterns that work best with AI.
3 AI/ML Vision
Armed with this information, we formulated a vision. In this example, our vision involved suggesting changes to the campaign type or SMS message. We did this by aggregating data from successful messages in the same campaign category from across the Twilioverse. By leveraging the power of AI, we aimed to improve SMS message and campaign matches, leading to an increase in conversion rates.
The greatest risk to an AI product’s success is an accurate model that supports our vision. By thinking about data sources, models, and model scalability at this stage, we determined which solutions had the potential to scale and utilize AI to solve customer problems.
4 Heuristic Validation
Next, we employed heuristic validation to test our solution. I designed a user interface where users receive keyword suggestions indicating the wrong campaign type. This proactive feedback prompted users to correct their language and prevented the wrong message from being sent, thus enhancing campaign relevance and increasing the likelihood of conversion.
5 Build an Accurate Model
To ensure the feasibility and maintain trust, our development teams focused on building an accurate model. This involved training and refining the AI model based on extensive data analysis and feedback loops. We continuously iterated to enhance the accuracy and reliability of our AI-powered solution. If maintaining trust proved to be a challenge, we remained flexible and pivoted to a fallback experiment, exploring alternative approaches that prioritize user confidence.
6 Continuous Improvement
Designing AI/ML products is an ongoing process. Once our solution is deployed, we continue to monitor its performance, collect user feedback, and iterate based on the insights gained. By embracing a culture of continuous improvement, we can enhance the user experience and optimize conversion rates over time.
By following the comprehensive framework outlined in this blog post, you will now have a clear strategy to approach the design and development of AI/ML products. This framework combines proven methodologies, user-centered design principles, and the power of experimentation to ensure that your product ideas are validated before the cost of implementation becomes unjustifiable.
Not only will this approach benefit your team by providing them with a systematic and user-centric process to follow, but it will also help them become more comfortable with testing and validating products in the AI/ML space. By leveraging familiar concepts from traditional product design, your team can navigate the unique challenges of AI product development with confidence and agility.
It is crucial to acknowledge the differences between AI products and traditional digital products. Having situational awareness and understanding the risks involved allows your team to make informed decisions and avoid potential pitfalls. Embracing this awareness will enable you to create AI products that truly address user needs and deliver exceptional experiences.
Furthermore, this framework encourages you to explore beyond the boundaries of conventional thinking. Don’t limit yourself to predefined ideas or solutions. Instead, be open to different approaches and innovative ways to create great AI products. Embrace creativity and out-of-the-box thinking to unlock the full potential of AI/ML.
Lastly, by investing in the design and development of AI/ML solutions using this framework, you can expect a high return on investment (ROI). With a user-centered approach, validated ideas, and a deep understanding of your target audience, you can be confident that your AI/ML solution will deliver superior performance and address user needs effectively.