047 - How Yelp Integrates Data Science, Engineering, UX, and Product Management when Creating AI Products with Yelp’s Justin Norman

In part one of an excellent series on AI product management, LinkedIn Research Scientist Peter Skomoroch and O’Reilly VP of Content Strategy Mike Loukides explained the importance of aligning AI products with your business plans and strategies. In other words, they have to deliver value, and they have to be delivered on time. Unfortunately, this is much easier said than done. I was curious to learn more about what goes into the complex AI product development process, and so for answers I turned to Yelp VP of Data Science Justin Norman, who collaborated with Peter and Mike in the O’Reilly series of articles. Justin is a career data professional and data science leader with experience in multiple companies and industries, having served as director of research and data science at Cloudera Fast Forward Labs, head of applied machine learning at Fitbit, head of Cisco’s enterprise data science office, and as a big data systems engineer with Booz Allen Hamilton. He also served as a Marine Corps Officer with a focus in systems analytics. We covered: Justin’s definition of a successful AI product The two key components behind AI products The lessons Justin learned building his first AI platform and what insights he applied when he went to Yelp. Why AI projects often fail early on, and how teams can better align themselves for success. Who or what Beaker and Bunsen are and how they enable Yelp to test over 700 experiments at any one time. What Justin learned at an airline about approaching problems from a ML standpoint vs. a user experience standpoint—and what the cross-functional team changed as a result. How Yelp incorporates designers, UX research, and product management with its technical teams Why companies should analyze the AI, ML and data science stack and form a strategy that aligns with their needs. The critical role of AI product management and what consideration Justin thinks is the most important when building a ML platform How Justin would approach AI development if he was starting all over at a brand new company Justin’s pros and cons about doing data science in the government vs. the private sector. Quotes from Today’s Episode “[My non-traditional background] gave me a really broad understanding of the full stack [...] from the physical layer all the way through delivering information to a decision-maker without a lot of time, maybe in an imperfect form, but really packaged for what we're all hoping to have, which is that value-add information to be able to do something with.” - Justin “It's very possible to create incredible data science products that are able to provide useful intelligence, but they may not be fast enough; they may not be [...] put together enough to be useful. They may not be easy enough to use by a layperson.” -Justin   “Just because we can do things in AI space, even if they're automated, doesn't mean that it's actually beneficial or a value-add.” - Justin “I think the most important thing to focus on there is to understand what you need to be able to test and deploy rapidly, and then build that framework.” - Justin “I think it's important to have a product management team that understands the maturity lifecycle of building out these capabilities and is able to interject and say, ‘Hey, it's time for us to make a different investment, either in parallel, once we've reached this milestone, or this next step in the product lifecycle.’” - Justin “...When we talk about product management, there are different audiences. I think [Yelp’s] internal AI product management role is really important because the same concepts of thinking about design, and how people are going to use the service, and making it useful — that can apply to employees just as much as it can to the digital experience that you put out to your end customers.” -Brian “You hear about these enterprise projects in particular, where the only thing that ever gets done is the infrastructure. And then by the time they get something read

Om Podcasten

Are you an enterprise data or product leader seeking to increase the user adoption and business value of your ML/AI and analytical data products? While it is easier than ever to create ML and analytics from a technology perspective, do you find that getting users to use, buyers to buy, and stakeholders to make informed decisions with data remains challenging? If you lead an enterprise data team, have you heard that a ”data product” approach can help—but you’re not sure what that means, or whether software product management and UX design principles can really change consumption of ML and analytics? My name is Brian T. O’Neill, and on Experiencing Data—one of the top 2% of podcasts in the world—I offer you a consulting product designer’s perspective on why simply creating ML models and analytics dashboards aren’t sufficient to routinely produce outcomes for your users, customers, and stakeholders. My goal is to help you design more useful, usable, and delightful data products by better understanding your users, customers, and business sponsor’s needs. After all, you can’t produce business value with data if the humans in the loop can’t or won’t use your solutions. Every 2 weeks, I release solo episodes and interviews with chief data officers, data product management leaders, and top UX design and research professionals working at the intersection of ML/AI, analytics, design and product—and now, I’m inviting you to join the #ExperiencingData listenership. Transcripts, 1-page summaries and quotes available at: https://designingforanalytics.com/ed ABOUT THE HOST Brian T. O’Neill is the Founder and Principal of Designing for Analytics, an independent consultancy helping technology leaders turn their data into valuable data products. He is also the founder of The Data Product Leadership Community. For over 25 years, he has worked with companies including DellEMC, Tripadvisor, Fidelity, NetApp, Roche, Abbvie, and several SAAS startups. He has spoken internationally, giving talks at O’Reilly Strata, Enterprise Data World, the International Institute for Analytics Symposium, Predictive Analytics World, and Boston College. Brian also hosts the highly-rated podcast Experiencing Data, advises students in MIT’s Sandbox Innovation Fund and has been published by O’Reilly Media. He is also a professional percussionist who has backed up artists like The Who and Donna Summer, and he’s graced the stages of Carnegie Hall and The Kennedy Center. Subscribe to Brian’s Insights mailing list at https://designingforanalytics.com/list.