Defining Product Success: Metrics and Goals

Goals are important in defining and monitoring success. When a goal is in place, the destination is clear—even though the route may change. Goals help connect your mission to your strategy, roadmaps, initiatives and tactics by tying the single metric you care about most with a target and a time frame during which it can be achieved. More than anything else, this process will help your team define success. In this post, we’ll explain how to define success for your product by identifying the one key “metric that matters” and then, how to set the right goals.

DEFINING THE RIGHT METRIC

What is the one metric that matters most to the success of your company and that you can rally your team around? For Facebook, it is active users; for WhatsApp, it is number of sends; for eBay, it is gross merchandise; for PayPal, it is total payment volume. Once you identify this “top-line” metric, you can set success criteria around it, monitor it, understand what drives changes in it, obsessively push it in the right direction—and properly evaluate and manage the health of your product.

For growth, a simple top-line metric might be number of users: for engagement, time spent; for monetization, revenue or number of advertisers. Fundamentally, your choice of metric should be driven by your vision for your product and the mission of your company. If you dream years into the future and visualize your product and company, how would you qualitatively describe them?

A vision statement should be aspirational, inspiring and future-focused. For example, eBay’s vision for commerce is “enabled by people, powered by technology and open to everyone.” eBay’s mission is “to be the world’s favorite destination for discovering great value and unique selection.” Taken together, these two statements point toward eBay’s dream of a world where everyone can find whatever they want, however obscure, at a good price.

What top-line metric would best encapsulate this goal? It is not the number of active users shopping on the site; that metric doesn’t measure whether buyers are actually finding what they want, and at the right price. How about the number of active buyers? While that and similar buyer-side metrics can tell us whether users are finding what they want at the right price, they cannot address the “unique selection” criteria of eBay’s mission statement.

Maybe a seller-side metric would work—how about the number of sellers? This metric can be helpful, but it doesn’t measure the inventory sellers are listing. How about the number of listings? That metric doesn’t tell us whether the listings are unique or whether the inventory is selling.

Ideally, the metric we choose would measure the number of unique items sold. A large number would suggest buyers are likely getting what they want, and at the right price. Therefore, the “right metric” in this case would be the market share of the sale of unique selections. However, this metric is complex and difficult to define, measure and grow. Instead, eBay chose as their top-line metric gross merchandise volume (GMV), or the total value of merchandise sold over a period of time. While this metric measures value to users, it does not guarantee unique selection. Nevertheless, it captures much of the spirit of eBay’s vision and mission. Picking the right metric can be as much art as it is science.

Some additional guidance on choosing a top-line metric for your company:

• Do not pick more than one metric. A single “metric that matters” is unifying and will enable you to set priorities across your entire organization. While it may be tempting to track everything and choose multiple metrics, this isn’t wise. Many metrics correlate with one other; they may help move the top-line metric but can become unimportant and distracting when measured on their own. The more metric goals you have, the more complicated it is to weigh them all and make trade-offs against them. Keep it simple.
• Avoid vanity and non-actionable metrics. For example, the number of likes your company gets on social media generally isn’t correlated with business results or customer success.
• When choosing between multiple metrics, pick the simplest measurable metric you can move. For example, if your number of advertisers is correlated with your revenue, and the number of advertisers is easier to measure and move, choose the number of advertisers. You can always establish an exchange rate to determine the impact from one metric to another. Likewise, if you are ultimately interested in a metric that has a low sample size or takes a long time to measure, consider instead choosing a correlated metric to measure.
• Pick the metric that most closely represents the usage of your product. For a company like Facebook and Instagram, for example, the single most important metric is active users. To measure growth at such a company, we could pick one of several active user metrics, such as daily, weekly or monthly active users (DAU, WAU or MAU), all of which are typically correlated. Choose based on the expected usage of the product. For example, if you expect the product to be used once per day or more, select DAU as your top-level metric. If instead you think the product will be used only on a weekly basis (e.g., when searching for specific restaurants, businesses, etc.), then choose WAU. One of these three—DAU, WAU or MAU—is a top-line metric for most consumer companies.
• Do not be afraid to change the metric if you need to. This can lead to thrash, but it is better to change to a metric that accurately reflects your mission than to move the wrong metric. Ideally, you will put in the time and effort up front to make sure you begin with the right metric. But if you must change it, do so sooner rather than later.
• Choose a simple metric that connects to your drivers. Let’s say you are looking to increase new user acquisitions. You send emails to potential customers, and a fraction of them visit your landing page. A smaller group of users then sign up, and an even smaller group become active users. From these numbers, we can create a simple framework to think about the problem of user activation (as seen below). You can increase the total number of users who activate by increasing any of the four terms above. For example, if you see the largest drop-off among people who visit the site but do not sign up, it may make sense to set the percentage of site visitors who then sign up as your metric to move.

• Avoid ratios. If click-through rate is what you really care about, see if you can instead measure the number of clicks. However, this isn’t a hard-and-fast rule; there are many examples of companies that successfully used a ratio as the “metric that matters.”
• Consider counter-metrics if needed. In the eBay example above, useful counter-metrics include the number of unique items sold and unique inventory listed. If such counter-metrics remain flat or decrease, that may indicate you are drifting from your mission. To address this, you could set an explicit goal that these counter-metrics not decrease. In eBay’s case, the primary GMV metric would remain in place, but the counter-metric would allow the company to maintain checks and balances in a complex environment.
• Change the metric as your business evolves. Your top-line metric may need to change over time. For example, before the mobile age, users checked products such as Facebook less frequently because of lack of access and connectivity. As mobile use increased, these companies revised their primary metric from MAU to DAU. This evolution is also common when companies launch new products. For example, Amazon Video has likely increased overall visits to Amazon, which could warrant a change in the company’s top-line metric.

SETTING GOALS

Teams often think about metrics and goals simultaneously, as they cannot easily be separated. Once you have identified the right metric and goal, you will be prepared to define a strategy and roadmap against which your product team can execute. Goals should highlight what you hope to accomplish and are often stepping stones to accelerating business growth. They can unify your company around a common objective and hold your team accountable for its promises.

As an example, let’s assume you want to grow your number of active users. A goal statement could be “grow MAU to 10M by Q4 2018.” This goal connects the metric (MAU) to a target (10M) and a time frame (Q4 2018), clearly describing what the product team wants to achieve and providing a purpose for the organization. Goals should be simple, actionable, achievable and most important, easy to measure and track.

You can choose goals based on:

• Product or business aspirations: Most long-term goals are based on the company’s mission. For example, if your company is in the video space, you might aspire to have the fastest-growing share of time spent on video. If you want to achieve that goal in x years, you can then break it into chunks to determine the growth you’ll need over the next y months in order to stay on track.
• Product metrics: If your product has been around for a while, you can do a “bottom-up” forecasting exercise to determine your goal for your top-line metric over a given period of time. For example, a forecast of MAUs could consider historical data on seasonality, platform, country, penetration and product changes. (Future blog posts will offer in-depth guidance on forecasting.)
• New products: If your product is completely new, it will be useful to look at external benchmarks and set “top-down” goals. For example, if the product is a Messenger-style communication app, you may choose to study growth at similar companies and let that inform your goals. You may also consider postponing goal-setting for a completely new product for a couple of months, until you see how it performs.

Some general guidance on choosing goals:

• Pick goals that are time-bound. Set a time frame, such as hitting the goal by the beginning of Q1 or the end of the year. You cannot hold people accountable to a goal that isn’t time-bound.
• Set different goals for different time frames. Choose a long-term, aspirational goal that aligns with your vision and mission for the company, as well as more tangible, short-term goals needed to reach that point. How you frame the short-term goals should depend on how your product team operates, as well as on the time frames over which you evaluate business results. For example, some companies operate on a quarterly cycle, while others run on a half-yearly or even yearly cadence. Your goals should reflect these different time frames, and your team should execute primarily on the short-term goals.
• Set two goals: an 80-20 and a 50-50. 80-20 goals are the ones you have an 80 percent chance of achieving. These goals are attainable, and hitting them will motivate the team. However, they will not help the team stretch and perform at a higher level. Therefore, it’s also important to set 50-50 goals, which you have only a 50 percent chance of achieving. While these goals are more challenging, they are also far more satisfying to reach. Whatever you do, don’t “sandbag” by setting only goals you can easily achieve. Failing to set the bar high can lead to complacency on the team and a decline in the product. If you find you always reach or exceed your 80-20 or especially your 50-50 goals, it’s likely you have set your sights too low.
• Set specific goals. Make sure your goal is measurable—not something like “delight users.” Combine a time frame with your top-line metric to define a specific, quantitative goal.
• When writing your goals, don’t include the “how.” While specifics are important in planning your roadmap and initiatives and in setting goals from a bottom-up perspective, the goal statement itself should be simple and high-level.
• Setting a goal is as much science as it is art. You’ll often face many unknowns while setting a goal. For example, you may not understand how the market, competition, retention of your cohorts, expansion into new countries, etc., will affect your top-line metric. Accept the fact that, like choosing a metric, setting a goal will be part science and part art.

NEXT STEPS: STRATEGY AND BEYOND

Imagine a consumer company with a 50-50 goal to increase monthly active users 10 percent by Q4. This goal meets many of the criteria mentioned above: it is specific, quantitative, time-bound and includes an appropriate top-line growth metric. The next step of the product-building process is to establish a product strategy, such as “resurrect users,” that specifies how to achieve the goal. This strategy can then help drive a roadmap, such as “resurrect stale India users.” The company could generate specific initiatives within this roadmap that would contribute most toward moving their top-line metric. Such initiatives might include “improve SMS notifications in India,” and would in turn lead to engineering tasks such as “build SMS notifications for low-end Android phones.” This example shows how a company’s vision and mission can drive its goals, strategies, roadmaps, initiatives and tasks. Future blog posts will offer in-depth guidance on setting data-informed roadmaps and strategy.

TAKEAWAYS

• Find a single actionable, top-line metric that encapsulates the vision for your product. This metric should be easy to measure and connected to your business drivers.
• Goals define the success of your product and can be set by breaking down business aspirations into smaller chunks. For older products, try a bottom-up forecasting exercise to connect your goal to your top-line metric. Newer products should use a top-down approach that assesses performance relative to external benchmarks.
• Your goals should be measurable, time-bound and staggered across time frames. To stretch the team, define 50-50 goal as well as 80-20 goals. Keep your goals sufficiently high-level and avoid the “how.”

This work, which is a product of Sequoia Capital’s Data Science team, was previously published on siteadmin.peakxv.com here.