A successful product has consistent behavior, meets or exceeds user-expectations, and significantly contributes to the top-line growth for the business. It is vital for a Product Manager to set and manage the expectations of users, gather quantifiable feedback regularly, communicate it rigorously to engineers, and make sure the product pragmatically evolves with the business and market transitions.

AI products, however, can differ significantly from traditional products. For example, in my prior experience as a Product Manager, success was measured through delivery of a ‘deterministic’ product that always delighted customers — a hardware product has the same behavior under the standard conditions, the same user actions in a software product results in the same expected response. An AI-driven product, however, may not always have a deterministic behavior and may in fact produce counter-intuitive results — a personalized recommender system may produce different results to a user action after learning additional preferences.

To successfully deliver a product using AI, a Product Manager needs a new mindset and a few additional skills in addition to those required for a regular PM. In this blog, I will address some of these shifts in mindset, particularly around product ideation, initial prototyping, and early release.

1. Track how the market is using AI technology

As per McKinsey’s Global institute (MGI), a review of more than 160 use cases for AI across a variety of industries found that only 12 percent had progressed beyond the experimental stage and the adoption has been limited outside the technology sector. One of the best-practices as noted in MGI’s report includes companies adopting an agile, test and learn approach. This includes setting up a cross-functional AI task force, which builds a prototype within weeks and tests with the business units before iterating further. Having decent market insights on applying AI and keeping abreast of relevant macro trends would help the Product Manager stay relevant. I find the AI market report from MGI, Gartner and those from CB Insights AI research very insightful and I regularly follow the updates on their Twitter handle.

2. Follow the trends in deep learning research

An AI Product Manager’s true competitive advantage comes not from expertise in the algorithms themselves but in his/her ability to shorten the time-to-market of products and services that apply those algorithms. Nevertheless, one needs to have a good handle on all the latest research trends and algorithmic advances.

Since most of the influencers in AI (like Peter Norvig from Google, Yann LeCun from Facebook AI Research, and Eric Horvitz from Microsoft) seem to believe in having their researchers share breakthroughs in AI algorithms and architectures, you can find their papers and methods on their company websites.

To keep tabs on the rest of the deep learning community outside of these companies, Andrej Karpathy’s site summarizing the most recent research in Machine Learning/ AI published on Arxiv is a good start.

3. Cut through the AI hype — focus on practical use cases

It is easy to get conditioned by the hype about the promises and threats surrounding an emerging or in the case of AI, a technology re-emerging from several busts. Peter Norvig, while interacting with the Fellows at Insight, recently noted that in many cases reporters in the popular media twist the words of influencers working in the field of AI or create hype to instill in readers a sense of FOMO and FUD.

An AI Product Manager needs critical thinking to separate the hype from real-world capabilities and have insights into the practical use cases of AI. The PM should understand the technologies in the realm of AI that are commoditized versus those still in research but look promising enough to be made part of the product roadmap. Also, the PM should clearly differentiate between those use cases where AI models can provide highest ROI and those where heuristic models perform better while shortening the time-to-market.

4. Be obsessed with customer-centric data

While being customer obsessed forms the basis for a successful product manager, an AI/ Data product manager needs an additional hat of data literacy while taking a product from ideation to launch. Customer obsession requires going beyond product features & benefits, and understanding the meaning for customer’s jobs, their purpose, motivations and the conscious choices they make.

Data obsession has two aspects: (1) being a champion of digitization while quantifying problems that customers care about and (2) being able to build comprehensive datasets for building quality AI models. The later aspect includes conceiving ways of fetching data that accurately reflects user’s jobs, behaviors, interaction patterns and pain-points. The data could be in the form of pixels, characters, numbers or bits (from various types of sensors).

Having a basic understanding of handling the data flows including data ingestion, data processing pipelines (including Extract-Transform-Load) and data visualization tools helps in setting up the stage for building AI solutions that create customer value.

5. Build a usable product with a simple model before exploring complicated AI models

Not all successful products using AI/ML have implemented intricate models and it is prudent not to become over obsessed with the complexity of AI models. This is because the accuracy of underlying models doesn’t always tie with a great user experience.

Some simple models result in accuracy that is good enough and increasing the complexity of the model only results in marginal improvement, thus validating the Pareto principle. However, in some scenarios accuracy is vital and the models need more iterations.

As discussed previously, it may be necessary to build a small pipeline with the simplest AI model to create a user experience and gather feedback. Jussi Pasanen’s Minimally Viable Product (MVP) pyramid model can be adapted to AI Products as follows:

6. Iteratively build use cases where AI directly impacts a metric

The PM needs to be familiar with any methodology that offers fast validated learning loops to quickly iterate with AI models. Building on the Lean Product methodologies of hypothesize-design-test-learn (by Dan Olsen), which is more relevant here than the build-measure-learn (from Eric Ries), the PM should have the ability to conceive simple use cases central to the core activities of business. Agile experiments should be run for these use cases building on small data sets. These experiments should be mapped to basic capabilities of machine learning such as classification (binary or multiclass), clustering, regression (prediction + forecasting), and the universal approximation capabilities of Deep Learning. After mapping the model metric to the business metric, the models can be iterated starting from a simple one.

The use cases should ultimately focus on creating significant value to the end user while improving earnings (EBIT) and should be tied to a small set of few metrics that matter to the customer. Optimizing an end-to-end AI model for multiple objectives at the same time could be challenging and make the system hard to debug. In a pragmatic implementation, a pipeline of AI models can be built, each optimized for specific metric(s). This would result in a system that can be explained well and is easy to maintain. For example, while building a text simplification application, the post-ingestion data processing system can consist of three AI models- a topic classifier, a sentence simplifier and a coherence checker each optimized for a unique metric.

7. Build Breadth-first (Data/Pipeline/Model) instead of Depth-first (AI model)

An AI Product Manager should have familiarity with the tools and techniques used to create an end-to-end product that leverages AI. This provides them the ability to influence:

AI Engineers and Data Scientists to utilize the right level of sophistication in their models, while still ensuring the ability to add complexity

Data Engineers to build robust systems and scale them appropriately

The entire team to leverage the appropriate cloud compute services and virtualization architectures.

This understanding includes a high-level awareness of the API ecosystem that help serve the end users, data ingestion tools such as Kafka, data processing systems such as Spark, and NoSQL DBMS such as Cassandra to work on Big Data. It’s also worth understanding their commercial alternatives on AWS and GCP as well as their trade-offs. Finally, It is also important to understand the cost structures of building various components and using the commercial alternatives.

It is better to avoid reinventing the wheel for commoditized AI techniques and utilize the services from popular cloud providers such as AWS, GCP, IBM and Azure.

8. Ensure your product fails gracefully

There are many ways large companies make their AI products gracefully handle low performance scenarios or failures. One simple method is providing a method for users to immediately relabel data to further improve the model.

The iPhone’s voicemail transcription service, for example, is transparent about its low confidence and offers the user an option to help Apple improve the transcription by submitting the voice recording.

In the text simplification product I worked on at Insight, one addition I considered was a coherence checker which could determine the coherence / grammar of the simplified sentence from the previous model. If it was below a threshold, the system could become transparent about its confidence and give an option for the user to provide a feedback on the result, or alternatively use the same input sentence as the output.

9. Insist on AI model explainability

An AI model built using deep learning is a black box and in some crucial applications involving high liability such as law, medicine and safety, the output requires a clear explanation for compliance purposes. This paper summarizes two approaches to explaining the predictions of deep learning models, one method that computes the sensitivity of the prediction with respect to changes in the input and one approach that meaningfully decomposes the decision in terms of the input variables. The Product Manager should leverage tools building on such approaches and also ensure that there are no biases in the models. Here are some recommendations from SAP design center to eliminate the bias.

10. Establish clear communication with your teams — know the fundamentals and language of data and ML

Last but not the least, an AI Product Manager should understand the language of AI researchers, data engineers, and data scientists. This allows the AI Product Manager to provide critical feedback and help the AI engineers tweak the models with the user experience in mind.

I found the Data Science Hierarchy of Needs pyramid (from Monica Rogati, one of Insight’s advisors) an excellent representation of the technology stack underlying AI products.

Also, AI engineers and data scientists usually come from a strong academic (PhD) background and derive tremendous intellectual gratification from novel academic projects. The Product Manager needs to mould their inclination to make a product that is marketable and user friendly.

Apart from a strong product sense, if you have a high level understanding of the various ML algorithms and AI models, along with their contexts, you have the chops to successfully manage a team of AI engineers and roll out AI Products.

Here are several resources which can help you get started with AI and ML technologies:

For a more in-depth covering of specific topics, you can watch the following video lectures:

Here’s a link to a good collection of AI, Deep Learning and Machine Learning cheat sheets.

Also, follow the social handles of influencers in AI (Andrew Ng, Peter Norvig, Geoffrey Hinton, Yoshua Bengio, Yann LeCun, Andrej Karpathy)

Conclusion

The current phase of AI is promising and there are several opportunities to make elegant products that create tremendous value, delight customers and significantly transform the business. An AI Product manager is a catalyst in this transformation and needs to be equipped with the right mindset and skill set.

Are you a company working data analytics, data infrastructure, or machine learning and would like to get involved in the Insight Data PM Fellows Program? Feel free to get in touch.