The 3 Qualities To Measure When Evaluating Product Operations
What to look for when assessing your Product Operations team
Product Operations is an emerging role within most modern Product organizations. At the same time, it’s an often misunderstood one. The lack of clarity in its role and responsibilities stems from a wide spectrum of definitions around the basic objectives of the Product Operations function.
But across all these definitions is a core set of skills that Product Operations roles can be measured by. These skills can be an effective way to evaluate candidates when hiring for Product Operations, or when evaluating existing Product Operations workers.
The role of Product Operations is often hotly debated and there are many thought leaders who offer their opinions of how the role should be defined. That’s not the focus of this article, but for context, my view of Product Operations is to act as a role which provides support for product leadership, product managers, and primarily anyone involved in designing or producing products.
As any business knows, getting products built and shipped on time is no easy task. If deployed and integrated properly, Product Operations acts as a force multiplier, helping the product teams accomplish more by freeing them up to focus on their key strengths. It can be challenging to find and develop Product Operations workers who can handle the busy workflow while motivating staff to perform at their best. And that’s where the 3 qualities framework comes in.
In this article you’ll find an in-depth walk through of my proposed framework as well as a link to my evaluation template.
The need for an objective framework
I’ve written before about the lack of consistency across the job descriptions of Product Operations roles (here and here). The problem this disparity creates is that there are so many different types of Product Ops roles, and they often require different skill sets. What’s more, the responsibilities can vary wildly from company to company.
As a result, when hiring for Product Ops, it can be difficult to determine the ideal candidate profile and interview questions. This is a huge problem for companies looking to hire, as well as people looking for their next role. The lack of specificity means that you’re likely not getting what you think you’re asking for when hiring Product Ops professionals. But with the right framework there are some common themes that you can use as a starting point when writing the job description.
But what if you’re not hiring? The same lack of specificity is often present for existing Product Ops employees. You need an objective framework for evaluating their competency and identifying growth opportunities for them as individuals.
Product teams need a way to evaluate their Product Ops team — regardless of their specific responsibilities. Just as you would evaluate your Product Managers, you should also evaluate the competency of your Product Ops team. To get the most value out of a Product Ops employee, you need an objective way of evaluating them. Ideally, you have a structured set of criteria that enables you and your team to evaluate them in a constructive way. These team members should understand where they are currently proficient, where they might be over-performing, and where they need improvement.
While searching for an evaluation framework for my own needs, I stumbled upon the Product Manager Competency Toolkit from Ravi Mehta.
As Ravi explains in his material:
A competency framework enables you to have more productive 1:1s, hire better candidates, make smarter promotion decisions, and build more balanced teams.
I was unable to find anything specific to the role of Product Ops that captured the ability to evaluate someone so simply and effectively as Ravi’s toolkit. So I took it upon myself to create my own version of a Product Ops Competency Assessment loosely modeled after Ravi’s approach.
The qualities you need to measure
Let’s pivot back to Product Managers for a second. It would be foolish to evaluate Product Managers solely on whether or not they’re shipping releases. Such a myopic view ignores how effective they are at motivating their team, resolving conflicts with stakeholders, leveraging data to make the best decisions, and all the other competencies that add up to a great Product Manager.
A Product Manager who always releases on time looks great on the surface but what if they’re burning out their team, mismanaging stakeholders, or missing important insights that could unlock more value? Would you still evaluate them as “met expectations” in their next review? Of course not.
The same logic applies to evaluating Product Ops. If all you care about is whether or not they’re checking the boxes on their job description, then you’re missing out on a huge opportunity to understand what’s happening in your Product organization. You’re overlooking where these individuals have growth opportunities as people.
As a result, the assessment I’ve developed focuses on 9 competencies categorized across 3 qualities:
- Operational Excellence
a. Team enablement
b. Complex relationship dynamics
c. Operational strategy - Project Management
a. Business outcome ownership
b. Project delivery
c. Data fluency - Influencing people
a. Stakeholder management
b. Communication
c. Managing up
Let’s dive into each of these in more detail…
Quality #1: Operational Excellence
The foundation for successful Product Ops is the ability to help your teams unlock efficiency and achieve results. There are many different ways to achieve operational excellence, but it all starts with having the right mindset. You need to be willing to constantly look for ways to improve and never be satisfied with the status quo. As a Product Ops team looking to achieve operational excellence, there are a few key areas to focus on.
Team Enablement
The ability to sniff out, prioritize, and resolve points of friction within the Product ecosystem that are holding teams back from delivering the best outcomes to their customers.
Team Enablement is where constantly looking for ways to improve and never being satisfied with the status quo is paramount. Companies, and the teams within them, are full of ever-changing variables. These variables can quickly take a high-performing team and reduce them to a sluggish group of under-performers if you’re not minding the details.
This gets especially difficult when you compound it by considering a team of teams — the most likely operating model for a Product development ecosystem in today’s best Product companies. An effective Product Ops professional should be in tune with the inner workings of the ecosystem and the intersection points so they can observe and call attention to problem areas before they overwhelm the team at large.
Complex Relationship Dynamics
The ability to identify, understand, and resolve problems stemming from individual and team dynamics within the complex network of PMs and other departments within the business.
Closely related to Team Enablement is the concept of the operational relationship dynamics within those teams. Operational relationships are the relationships between the different parts of a business. In other words, it’s the way that the different departments within a company interact with each other to get work done.
Operational relationships are important because they help to ensure that the different parts of a company are working together efficiently. When operational relationships are well managed, it can lead to better communication and coordination between departments, which can ultimately improve the overall performance of a company.
Effective product development is not possible without a tangled web of operational relationships across the business. It can be argued it’s through these relationships that product teams unlock their greatest points of leverage. But that leverage suffers when these relationships break down.
Product Ops professionals need to have the positioning and awareness to spot the early warning signs of a possible breakdown. They also need to be equipped with the skills to intervene and resolve any erosion that might damage the integrity of those relationships.
Operational Strategy
The ability to make observations, collect information, and synthesize your findings in order to make sound conclusions that drive recommendations for improvements to the tools, rituals, and processes used by the Product team; and the ability to adjust your plan as new insights are discovered.
The idea that Product Ops should be evaluated by their Operational Strategy skills shouldn’t come as a surprise. But in many product teams, operational strategy is often confined to product leadership. It can sometimes be difficult for product leadership to release their grip on this aspect of running their team. But that’s where Product Ops is an especially valuable partner to product leadership.
The operating model (i.e. fighting configuration) of the product team can produce outsized results if managed correctly. A Product Ops function should be adept at observing how the team is structured and capable of making recommendations to product teams and product leadership to maintain the efficiency and impact of the team.
Quality #2: Project Management
As a support function, it’s inevitable that Product Ops will be tasked with carrying out projects of their own. These projects might benefit only a small team or the entire product ecosystem as a whole. Either way, it’s imperative that Product Ops professionals be competent at the discipline of project management.
Business Outcome Ownership
The ability to connect project activities and define goals that drive meaningful outcomes for the business which achieve the strategic objectives of the Product team and the company overall.
The role of Business Outcome Ownership within the quality of Project Management is simply to ensure that the business outcomes are met. In this capacity, Product Ops works with the stakeholders to identify and prioritize the project’s intended business outcomes. They will also create a set of expected results for each outcome, including how they might be measured.
Ownership of business outcomes means that for the given project, Product Ops is responsible for ensuring that all stakeholders involved in the project understand their roles, responsibilities and expectations. Product Ops must also ensure all stakeholders have access to any information they need in order to fulfill their responsibilities or meet expectations.
Project Delivery
The ability to work closely with a project team to define project goals and quickly & effectively deliver a result that accomplishes those goals.
Project Delivery is a process of managing the delivery of a project. It is an essential part of Project Management as it ensures that the project meets the team’s expectations.
The Project Delivery process can include things like:
- Managing stakeholder expectations
- Executing on work packages
- Maintaining quality standards
- Planning for the delivery phase
- Ensuring timely completion of deliverables
If you’re going to be doing any amount of Project Management, Project Delivery skills are critical to your success.
Data Fluency
The ability to use and interpret data to create actionable insights and the ability to leverage those insights to achieve meaningful outcomes.
I want to caveat this skill by saying I’m not advocating for Product Ops to perform any amount of data analysis in lieu of the Product Managers. But that doesn’t absolve Product Ops from needing to be comfortable working with data.
The best Product Ops teams treat the product teams they serve as their product to manage. As the saying goes “What gets measured gets managed.” So when it comes to Product Ops, and the effectiveness of the product team at large, you’ve got to possess some amount of data fluency.
In fact, the approach I advocate for any Product Ops teams just starting out (also a useful exercise for established Product Ops teams) is to use data to obtain an objective view of where the product teams are falling short. A minimal amount of data fluency, an ability to draw conclusions, and present the data is required to conduct this exercise.
Quality #3: Influencing People
Like Product Management, Product Ops accomplishes most of what it achieves through the actions of others. As a result, Product Ops teams need to be comfortable leveraging their relationships to make an impact. It’s important that they have excellent communication skills, are highly organized, and are able to build solid relationships with their stakeholders in order to be successful.
Stakeholder Management
The ability to identify stakeholders impacted by the Product Ops Manager’s area of ownership and to include their requirements into project goals.
In order to influence people effectively, Product Ops must be able to identify stakeholders and their different needs/approaches early in the planning phases, and help to ensure they are involved in key decisions. This helps ensure their needs are included in the project scope, specifications, solution design and deployment. Getting in sync with all of the affected business units also helps ensure that you aren’t repeating work already done by another stakeholder or re-inventing something a group has already considered within its domain.
Ultimately, the ability to effectively manage stakeholders is an integral aspect of being successful at Product Ops. If you are proactive in identifying various stakeholders and developing effective ways to work with them, managing these individuals and their needs can (and should) go smoothly.
Communication
The ability to demonstrate promptness, active listening, empathy, and confidence when giving & receiving information to/from stakeholders.
Next in the skills required for effectively influencing people is Communication. You must keep in mind that Product Ops should be able to communicate frequently with stakeholders in order to garner buy-in. This communication can take place through ad-hoc syncs or meetings that are informational, project delivery focused, or regularly scheduled depending on the complexity of projects and the time required for deliverables. Formalizing these interactions will ensure all project plans are kept up to date and that the business needs of the stakeholders are being considered.
Product Ops often acts as the primary conduit of information between teams. Possessing good communication skills is essential to an effective Product Ops function.
Managing Up
The ability to leverage project sponsors, senior managers, and executive stakeholders to help achieve goals and deliver business outcomes that positively influence the strategic direction of the Product Management team and the larger organization.
Like any job, a Product Ops position comes with its own set of challenges that you must overcome in order to reach your goals, and managing up is one of them. It may not come naturally to everyone (it definitely wasn’t a natural fit for me), but it’s something that you need to work hard at if you want to succeed in your job and reach your career objectives. And the more effort you put towards building good relationships with those who are senior to you, the easier time you’ll have working towards success.
It’s an area that often gets overlooked in internal communications, but it has a huge impact on how well a Product Ops role functions. A lot of the focus for Product Ops will be on managing across — that is, working with stakeholders to get projects completed. But, they’ll also be working with people above, so it’s important that they feel they can trust and rely on your Product Ops team.
How to perform the evaluation
Now that you have an overview of each skill that should be evaluated, let’s talk about how to actually conduct an assessment.
This approach works whether you’re a Product Ops team of one trying to evaluate yourself, or if you’re a product leader responsible for a Product Ops team.
The rules
Most performance evaluations score performance using three, vague values: exceeded, met, or below expectations. This approach falls short because it doesn’t give the employee being evaluated a clear idea of what’s most important to work on. As a result the evaluation I’m advocating for here will use a slightly more detailed approach.
Score the individual by selecting from the following four scoring values. Choose from:
- Needs focus
- On track
- Over-performing
- Not a focus
The “Not a focus” scoring value is best for when a skill isn’t something the employee should focus on for this review cycle. It’s usually most applicable for junior roles who don’t yet need a certain skill to be developed. It can also be applicable for senior roles who no longer need to work on developing a particular skill given their senior position in the team.
This slightly more detailed scoring method gives the employee a sense of what to focus on now compared to what they can de-prioritize for the time being. This way, they can hone the skills that are most important for them to develop right now. It also gives product leaders a chance to set clear development priorities for the employee.
The constraint
The constraint to this approach is that you should select no more than (2) areas where the individual “needs focus” and no more than (2) areas where the individual is “over-performing.”
This is important because highlighting every gap is demotivating for the employee. The employee should be encouraged to grow their natural skills rather than fall into the trap of feeling like they need to address every shortcoming. The flip side is also true — you don’t want to identify every instance of over-performing because it doesn’t allow the individual to identify opportunities for growth.
Step 1: Self assessment
The first step is for the Product Ops worker to evaluate themselves using the same criteria. They should score themselves using the rules and constraints listed above. This gives you (and the individual) an indication of where they feel they perform across the framework.
Step 2: Peer assessment
The next step is for the individual’s peers to evaluate the employee. You can give them the same template and ask them to adhere to the same rules and constraints. Peers should be encouraged to make notes and provide feedback with specific examples if possible to further enrich their assessment. This gives product leaders a chance to go deeper and get at the root causes. This will be useful material for the individual’s manager to discuss with the individual during their review by providing the causal factors behind the peer assessments.
Step 3: Manager assessment
The third step is for the manager to conduct the same assessment of the employee using the same template, rules, and constraints as everyone else. The manager should also include any notes and specific examples that speak to the causal factors behind the scoring. This gives the manager a holistic view of the employee’s competencies rather than resorting to a single, subjective view of the individual. Managers can build on the employee’s reflective self assessment, the peer assessments, and the manager’s own perspective.
Step 4: Delivery & discussion
The last step is for the manager to combine the separate assessments into a single document that can be shared with the employee being evaluated. The final document should at a minimum include the scores from the self assessment, the peer assessments, and the manager’s assessment. It can also include any written examples collected from the peer assessments and any additional notes from the manager’s assessment. The goal is to compile a package of information that leads to a conversation rather than simply reading through the scoring comparisons.
The final document should be shared with the employee ahead of their review with their manager so they can consume the information beforehand. Otherwise, the employee is likely to be overwhelmed by trying to digest everything all at once and won’t be fully present in the discussion.
The template
I’m including a link here to the template I’ve developed. Feel free to make your own copy and tweak it as needed to fit your unique circumstances.
👉 👉 Product Operations Competencies Assessment 👈 👈
Takeaway
Everyone has their own perspectives on what makes a successful Product Ops worker, but in my view these qualities are must-haves for any member of that function. There are likely to be additional qualities and skills you may want to evaluate given the nuances of the Product Ops role in your organization. But at its core, Product Operations is a role that can be effectively evaluated using the approach I’ve laid out here.
Interested to hear more of my thoughts about Product Operations? Check out my other articles on this topic:
Want more templates?
There are no shortcuts in life… but a good template is pretty darn close! I’ve packaged up 10 of my favorite Product Ops templates for anyone who wants some help getting started or is already on their Product Ops journey. These are templates that I’ve used in my day-to-day work, and are designed to be easy to use, customize and update.
You can find the Product Ops templates here:
👉 👉 More Product Ops templates 👈 👈
