← Back
Blog
Struggling to see value from your analytics projects? Product management might be your answer!
January 1, 2021
.jpg)
Product management is a discipline that, when executed correctly, can yield incredible results. This discipline has only increased in popularity and necessity as market expectations of products have continued to rise. This blog will explore the benefits of product management, the differences between applying this framework to software and analytics products, and what happens when there is a void.
In order to discuss the difference between product management in software and analytics, we must first align on the current definition. I went out on a search to compile a set of skills and concepts that the industry currently recognizes as product management. This is a product manager job description sample from Monster.com:
“Develops products by identifying potential products; conducting market research; generating product requirements; determining specifications, production timetables, pricing, and time-integrated plans for product introduction; developing marketing strategies.”
Other skills mentioned:
Some descriptions include familiarity with Agile and Scrum methodologies. There are also some differences between a product manager that is overseeing an internal product vs an external product. For example, you may consider evaluating the cost savings of an internal product and the cost to build it. In this case, you would not necessarily be concerned about pricing like you would with an external product.
Product managers for technical products may need to have a basic understanding of the complexities associated with building it. They may need to familiarize themselves with the general process of developing the product to ensure that they are taking all requirements and complexities into consideration when evaluating potential solutions. An example of this would be a product that has potential security vulnerabilities. If you are not familiar with the process of getting a solution into a production system, you may introduce critical teams too late in the process causing delays.
Product management exists to ensure that products are being designed and built to serve the specific needs of its intended users, all while being cost effective. This discipline enforces a framework that efficiently coordinates teams to incorporate the many dimensions of successfully building and shipping a product. From researching market needs and opportunities to evaluating solutions to monitoring success metrics; the product manager is the quarterback of building meaningful products. This fact is no different in the world of analytics. Yet, for some reason, they have largely been left out of the conversation. The mentality might be that “this project is expensive enough as is, why would I add yet another resource?”. The project might be more expensive because you are not using a product manager.
Now that we understand how this role is defined in the industry today and its importance, what is the difference between applying these concepts to software vs. analytics? Let’s first explore the different characteristics between them.
Analytics and models have different requirements and processes to designing, building, deploying, and monitoring them than typical software. Here are some key differences:
Properly defining user requirements for analytics solutions can be extremely challenging. Going through the process to understand what information is needed, when, and why with key stakeholders and end-users will help refine the requirements and ensure that the solution addresses areas that are actionable. A question that you should ask yourself is “what information do you need and how can the solution display that without creating room for misinterpretation?” The key difference here is that users are more accustomed to software standards; clicking a button, clicking the drop down menu, filling in a text field. How seamlessly these interactions work together is the job of a UX designer. The interactions with a solution look different when the user is also faced with interpreting a model prediction or a chart of data and then doing something with it.
In both software and analytics, you must translate the user/system requirements into solutions and evaluate those solutions based on effort, value, complexity, effectiveness, and resource constraints. Assessing the true problem at hand and fully evaluating potential solutions could force you to come to a conclusion that a complex predictive model is not the most reasonable path forward. It is important to go through these exercises to ensure you are not over engineering the solution.
When designing an analytics solution, an analyst or data scientist must complete an exploratory analysis on the available data to determine if any valuable solution or conclusion is even feasible. This process can sometimes be very time consuming. An analyst can hit a lot of dead ends and explore the possibilities of “interesting” forever if they had the time and a lack of direction. Not to mention the technical challenges during the process of data acquisition and analysis. During this exploratory phase, the team may come to the conclusion that there is not enough data, context, or relevant information to arrive at a meaningful solution - this is often a hard pill to swallow. The uncertainty in this phase creates a challenge to accurately establish timelines and makes it even more difficult to calculate an investment estimate for stakeholders.
Beyond the analytical feasibility, a product manager should examine the characteristics of potential solutions that could drive additional design considerations. Some models or solutions need to be updated near real-time and others can be refreshed monthly. Some models require a large amount of feature generation and large training data sets. Some modeling techniques have low explainability and others are fairly simple to understand. In certain cases, there might be an additional regulatory requirement to be able to explain how the model produced each output. This can be nearly impossible with some techniques which emphasizes the need to gather these requirements prior to designing and building a solution.
All products and solutions should be monitored over time. Success metrics need to be defined and continuously tracked to ensure that the product is serving its intended purpose. In software, these metrics might include feature usage, downtime, latency, or conversions. Predictive models and analytics have additional metrics to track. For predictive models, you will want to track model accuracy, drift, and regulatory compliance to name a few. Your team will need to understand how the model is performing over time, if outputs are reasonable and acceptable, how it's being used, what decisions are being made because of it, and what downstream effects those decisions have. Tracking these metrics will allow you to determine when a model refresh or update is required, to generate reporting for regulatory audits, to ensure the model is being used as intended, and to help calculate the return on investment. There can be extreme consequences if these metrics are not captured and monitored properly depending on the context.
In the last 5 years, the industry has seen a flood of new analytics and data science software products. Don’t get me wrong, there are a lot of products out there for developing software as well. The difference here is that the industry has largely settled on best practices and standards for designing, building, deploying and monitoring different flavors of software. This is not to say that everyone follows those standards, but there has been a general consensus on the concepts of DevOps and CI/CD. The adoption has gone far enough to push companies to create and enforce policies around them. While it has been slower for analytics, it is not far behind. DataOps and MLOps have become more popularized in the last year. There is definitely still a battle of full end-to-end solutions and “Frankenstein” systems as organizations continue to experiment with their operational standards for analytics. A lack of clarity on standards and policies makes the job of adhering to them much more difficult.
According to Gartner, the primary barrier to delivering business value for organizations in 2018 is the difficulty of deploying analytics into business processes and applications.
Models and dashboards alone do not bring much value. It is when these products are fully integrated into operations that they make a meaningful difference. The process of successful integration requires the work of product managers. This is not even a separate conversation in the world of software. These requirements are taken into consideration during the design phase. What users are interacting with which features and when? Organizations have placed such an emphasis on the challenges of building models and dashboards, they forgot that these products need to be used and adopted! In some cases, users may be just fine with adding another dashboard to their daily process, but oftentimes it is not practical. Some solutions will need to be seamlessly integrated into existing systems. This idea of operational integration is usually an afterthought and can have a huge role in low utilization, redesigning solutions, and project delays. Without a full understanding of how users would interact with a model or a dashboard, you may find yourself right back at the design phase 3 weeks prior to the original go-live date.
Having a product manager for analytics solutions can ensure fewer iterations on user acceptance testing, proper monitoring and alerting in place prior to deployment, faster time to deployment, higher solution utilization and impact, clearer solution impact metrics and return on investment, and a more seamless integration into business operations.
All of these tasks can be very costly and time consuming. Lack of product management typically results in analytics solutions that are unclear, or worse, misused. On top of that, the amount of resources invested in creating these solutions can be very expensive. Having clear requirements can also reduce the time of data acquisition, validation of solution interpretability, resource utilization, and can ensure it passes organizational policies.
Right now, most teams expect data scientists and analysts to cover these responsibilities. The first problem is assuming data scientists are familiar and skilled in this discipline. They are typically inquisitive and can help pull out requirements along the way, but as we saw earlier, that is only a portion of what a product manager is responsible for. The second problem with this is that they are typically doing most of the problem understanding at the business stakeholder level, not at the user level. This means that models and analytical solutions are designed and built without a clear understanding of how they will be adopted or integrated into business operations. This typically leads to low utilization of the solution or a long journey to deploying and using the solution.
Product managers should have a general familiarity with technologies and tools to use when creating, deploying, visualizing, and monitoring analytics. Having a basic understanding of how models and analytics are built and evaluated can come in handy during the requirements gathering and solution evaluation phases. An understanding of internal promotion policies and regulatory compliance can ensure higher quality solutions and incorporation of these requirements early on in the design process. Two additional top challenges that exist when building successful analytics products are poor planning and unreasonable expectations (referencing Garter chart above). This is why it is important for analytics product managers to have a particularly strong ability to clearly communicate expectations and sift through ambiguity.
All in all, there are a lot of similarities between key responsibilities of product managers in software and analytics. There are some distinct differences that are important to highlight and be aware of when hiring or staffing a project. Regardless, the need for skilled product managers is clear. When we bring the concepts of data science out of the lab and into the industry, we must supplement it with the best practices of building usable products and solutions that are meaningful and provide value.
Interested in learning more about applying these product management concepts to your analytics projects? Set up a 30 minute call with us to discuss!
Let us show you a demo of AlignAI, discuss services, or create a custom program for your team.
