Chapter 20 – Test and Measure – Lather, Rinse, Repeat

Estimated Reading Time: 13 minutes

“You can’t manage what you can’t measure” is a favorite management maxim.  It is commonly attributed to the legendary manufacturing quality guru William Deming, although nobody is certain who really coined it.   Regardless of whether Deming actually said it, however, he would no doubt agree that in marketing, as in any process-oriented task, measurement is a natural precondition to improvement.  Professional marketers should take heed.  Marketing efficacy can be improved only through continual testing and measurement.

Marketers should utilize empirical tests for two fundamental purposes: to make informed decisions and to track progress over an extended period of time.  There is no shortage of marketing elements to test.  Among the most basic are the following:

• Email subject lines
• Ad headlines and copy
• Calls to action
• Offers
• Lead capture form length
• Landing page design
• Web page design

Put simply, the amount of marketing data that can be analyzed represents an embarrassment of riches.  Practically every buyer behavior – whether call, click, search, or scan – is at our disposal.

This chapter opens with a discussion of the culture of testing and measurement and how to shift an organization towards it.  It then describes specific test methods, followed by methods for measuring different marketing programs. It concludes with an overview of marketing benchmarks for comparing your marketing effectiveness with your competition and how to use these assessments to adjust your overall marketing mix.

Shifting to a Test and Measurement Culture

A major challenge for most marketing teams is to make testing and measurement a standard component of their culture.   Marketing is not manufacturing, after all. Testing and measurement cannot be performed on an ad hoc basis.  Rather, they have to be continuous, with the end goal of refining the marketing program over time.

In Benjamin Cheever’s novel The Plagiarist, a marketing executive becomes an industry legend by adding one word to shampoo bottles: Repeat. This single action doubles shampoo sales overnight.   You will find this advice on shampoo bottles today.  It has stuck in our lexicon as a humorous way of saying that a certain set of instructions should be repeated until an explicit goal is reached.   The same instructions apply to a test and measurement culture: Test, measure, repeat.

There are several good reasons for marketers to test as much data as they can, as frequently as they can. Here are just a few:

• To outperform the competition – If your competitors don’t conduct ongoing testing, they may be wasting money, producing ineffective marketing materials, or both.   For this reason, integrating testing into your marketing culture can provide your marketing team – and your company – with a key competitive advantage.
• To spend money more wisely – Understanding which ads, creatives, Web pages, or other elements appeal to customers will enable you to allocate more dollars to them.
• Avoid costly missteps – Imagine a scenario in which your CEO is in love with an ad concept, but it falls flat with your customers and prospects.  Unfortunately, you have already paid the agency to create and place the ad.  Testing in advance can help you to avoid these very expensive mistakes.
• Bring empirical proof to your peers – When marketing programs are not supported with test results, their quality and impact can be discussed only in a subjective way, where everyone’s opinion carries equal weight.  To avoid opinion-based debates, provide all involved parties with test results.  Which ads did the focus group prefer?  Which creatives pulled better?

How to Test

Testing in marketing is not rocket science.  In fact, as simple as the two primary test methodologies are, it’s surprising they are not employed more often.

A/B Testing

A/B testing – also known as split testing and bucket testing – is a very simple tool that marketers use to evaluate which of two versions of a marketing asset is more effective.    The process is straightforward: Create two variants of the asset you want to test, and then measure the response for each variant.  The variant with the higher relative performance is the winner and should be used.   If you are testing an online asset, chances are your marketing automation tool can help you measure views, clicks, opens, conversions, or whatever responses you are looking to assess.

In the case of direct mail, rather than simply splitting your list of potential respondents in half, start with a subset.  That way you are not sending 50% of your total list the losing variant, just 50% of the smaller subset. After measuring which version of the direct mail this group preferred, use that variant for the balance of your total list.  For example, select 100 contacts from a list of 1000, then send version A of the direct mail to 50 and version B of the direct mail to 50.  Measure the response. Send the winning version to the remaining 900.

Figure 1 below shows two banner ads created for a North Carolina based hospital.  The A version of the ad showed a nurse, and the B version showed a cancer survivor and her child. All other copy stayed the same. The version with the nurse outperformed the other with 63% more click throughs. The nurse showed compassion and confidence. Changing just one element of an ad can have a significant impact.

Effective marketing teams adopt an iterative approach, constantly tweaking and testing, using best practices and incorporating knowledge gleaned in previous tests into all future marketing work. Building on the direct mail example above, a second test comparing the winner of the first A/B test versus a modified variant of that winner should always be done. Significantly, although A/B testing is simple in concept and relatively easy to execute, many marketing teams lack the discipline to conduct it. Test, measure, repeat.

Focus Groups

Gauging opinion before investing significant sums in a marketing effort is always a good idea.  Though they sell to buyers located outside their walls, they make the mistake of talking only among themselves.  Remember: No one in your building is a proxy for a customer.  Talk to a real one!

The best  strategy for communicating directly to customers is via a focus group.  A focus group is a form of qualitative research in which the individuals conducting the exercise ask a group of people about their perceptions, opinions, beliefs, and attitudes towards a product, a service, a concept, an advertisement, or an idea.  The first focus group – then known as “the focused interview” – was conducted by sociologist Robert K. Merton at Columbia University’s Bureau of Applied Social Research.[1] In 1943, Merton and his colleague Paul Lazarsfeld conducted the first focused interviews to measure the effects of “mass persuasion” radio announcements aimed at selling war bonds, on what was to become Voice of America.[2]

Focus groups can take a number of forms. Typically they are small groups comprised of six to ten people, plus a facilitator.  The group setting helps spark conversation, and the facilitator keeps the conversation moving, prevents any single participant from dominating, and ensures that all of the questions or topics are covered.  A marketing or research team typically observes the session through a one-way mirror.

In recent years, online focus groups have become more popular. A common format is for the facilitator to ask questions or show visuals online – using an application similar to one used for webcasting – and for the participants to reply using online chat or voting tools.  Major benefits of online focus groups are lower costs for the marketers and greater convenience for the participants.  The major drawback is that the marketers observing the session lose the ability to observe body language, facial expressions, and other forms of nonverbal communication, which can be as revealing as spoken language.

The results gained from focus groups can be very powerful.  Companies can utilize the feedback to make decisions ranging from how to package an individual product to how to frame an entire campaign.  They can also get ideas for new products or features.  Importantly, focus groups bring the customer’s voice back into the company.  This ensures that the opinions of insiders – even powerful executives – do not overshadow those of the target customers.  Even obtaining feedback from a handful of existing customers via phone or email can have a powerful effect.

How to Measure

Web Measurement

Web site statistics are an indicator of customer behavior, though they generally need to be interpreted.  Most marketing teams use these data to reveal online awareness trends and optimize both online advertising spend and lead capture and online purchases.  Marketers have access to a number of tools – some free, like Google Analytics – that can measure and report on Web statistics.  The frequency of reporting will vary by business, with online businesses reporting daily or weekly, and other organizations reporting on a monthly basis.  Below we list some of the most common measures:

• Pageviews – the total number of pages viewed during the time period.
• Visitors – typically the total number of visits, unique visits, and the location of the visitors.
• Time on site – the amount of time a user spends on the site.  A high score can be a good sign — your site is so useful that people spend a long time on it — or a bad sign — it is difficult to navigate, so customers take a long time to find what they’re looking for.
• Bounce rate – the percentage of single-page visits, or visits in which the person left – “bounced” – from your site back to the page he or she came from.  Marketers use this metric to measure visit quality:  A high bounce rate generally indicates that site entrance pages were not relevant to your visitors.
• Form or shopping cart abandonment – the percentage of visitors who abandon a form while they are in the process of signing up for an offer, or who fail to complete a transaction and abandon their shopping cart.  The major reasons why visitors abandon forms is that they are too long or the offers they contain don’t stand out. Visitors abandon shopping carts for assorted reasons: Shipping costs are too high, visitors discover better offers through comparison shopping, the transaction requires too much personal information, or the process takes too long.  Sometimes buyers simply change their minds.  Ecommerce sites devote a great deal of time to analyzing their checkout process to reduce shopping cart abandonment.
• Referrals – a list of Web sites that refer people to your site.  The most common sources of referrals are search engines, online news sites, advertisements, blogs and partner sites.  Effective marketers use this information to shift their ad spending to sites that generate referrals, work on linking with more popular referers, or optimize content to align with likely referers.
• Popular pages and links – a ranked list of the most popular content on your site, whether a page or a link to an asset.  This information is a good indicator of which content may no longer be useful, which content may need to be moved or highlighted, and which content or products buyers are most interested in.
• Top site-search terms – a ranked list of the popular search terms visitors use once they are on your Web site.  This information can reveal which items the market is looking for, which search terms buyers are using, and which information the company should highlight, or uplevel, on its site so that prospects and customers can find it more easily.
• Backlinks – inbound links to your Web site or to a page on your site.  Having a large number of backlinks from high-quality sites increases the likelihood that search engines will direct visitors to your pages.  Tracking backlinks and working to build more of them is a common SEO strategy.
• Site rank – the ranking of the popularity of a Web site by a third-party service, such as Alexa or comScore.  Site rank is generally more important for companies whose Web presence is vital to their business, such as ecommerce, gaming, and news organizations.  It can also be used to gauge how an organization’s online presence compares to its competitors.

There are many, many more measurements.  Which ones you use will depend on your company, whether you sell online, and the level of detail you need.

Measuring Demand Generation

A company should measure all of its lead information in a standard fashion to identify which activities are contributing the most to the funnel.  What exactly should they measure?  The following list identifies the key variables.

• Source – the program or source of the lead
• Conversion rate – the percentage of inquiries that converted to marketing qualified leads (MQLs)
• Lead flow – the total number of MQLs
• Cost Per Lead – the amount each MQL cost to generate Recall from Chapter 9 that this is also known as cost per acquisition, or CPA.
• Velocity / time to conversion – the number of days from inquiry to opportunity

Used by Permission of Marketo, Inc.

Table 1: Reporting on effectiveness of marketing activities

Table 1 provides a good example of how a company can identify which marketing tactics are performing well and which ones are not.  In this example, the marketing team has set high, average and low performance thresholds for each measure, shown in green, black and red, respectively.

Total inquiries (Inquiries) and cost per lead (Cost) are shown in the second and third columns.  Right away, you can see, based on the color coding, that pay-per-click (PPC) ad costs (in red) are significantly higher than the $100 threshold for cost per lead, whereas third-party email blasts, trade shows and virtual trade shows (in green) are significantly lower than the $40 threshold (Note that these thresholds value are just examples; every company will have its own.).   The table does not display explicit costs for “run rate” activities –   Web site leads, and referral programs, or sales generated leads – since there are ongoing costs.

The percentage of Inquiries that convert to MQLs in the fourth column (% MQL) is similarly color coded, with less than 20% being low, 20% to 40% being average, and above 40% being high.  We can see that PPC, while costly, is effective: Nearly half of the prospects became MQLs.  In contrast, content syndication is not cost-effective, nor does it convert well.  Although there are more elaborate graphical models for illustrating program effectiveness, we prefer this color-coded tabular representation, which we became aware of via the work of Jon Miller at Marketo.

In addition to the above, Table 1 illustrates two other interesting metrics.  The first — velocity — represents the average number of days required to convert a lead  to an opportunity.  Understanding velocity is important when deciding when to employ various tactics.  For example, if sales management feels its pipeline is low and needs more quickly, then marketing should shift investment to tactics with a high velocity that will generate more pipeline quickly, even if they are more expensive.  In the example above, an organization could easily reallocate budget to PPC advertising, which has a velocity of 15 days.

The second metric – the MQL-to-opportunity index – indicates the likelihood of conversion relative to other programs.  In Table 1, virtual trade shows was selected as the baseline index, and normalized to equal 1.  The MQL-to-opportunity indexes for all of the other sources were then normalized using the same weighting. The idea is to pick a source and make its index equal 1 to make comparison among sources easier.  Relative to virtual trade shows, then, trade shows are 1.8 times more likely to convert, and leads from the Salesforce AppExhange are 2.4 times more likely to convert.  Social media and third-party email blasts have a lower than average likelihood of converting.

Marketers can use the combination of cost, velocity and lead-to-opportunity to make informed decisions about their marketing mix.  Rather than discuss tactics in the abstract, direct comparisons of cost and efficacy can be made.  Social media, while trendy, is not very cheap and not very effective, as shown in the example above. PPC advertising is very expensive, but also very effective (high conversion, high velocity, high MQL to Opportunity Index). Sharing these metrics with sales management is a great way to show how marketing is driving the business, and to avoid uninformed requests from sales for new marketing efforts.

Measuring PR and Social Media

Chapters 5 and 6 covered measurement of press and social media in detail.  To recap, the marketing team should measure the following media elements on a regular basis:

• Share of voice (SoV) and press sentiment
• Total press mentions
• Clicks on the press releases, blogs and Web pages linked to from press releases and blogs
• Social media stats – tweets, retweets, Facebook “likes,” social sentiment and others

Measuring Advertising

Depending on the type of advertising your company is running, you may already be measuring its effectiveness.  Referrals to your Web site may be coming from SEM keyword or banner ads.  You may also be measuring the conversion rates of direct response advertising in direct mail offers you send out – hopefully evaluating different subjects and offers in your A/B testing.

Below are the most common metrics, which were discussed in Chapter 13.  Unless your specific goal is to raise awareness of a new product or brand, you should focus on conversion rates, because they are the most precise measurement of qualified leads.

• Impressions, aka ad views – the total number of times an ad is available for view.
• Share of voice (SoV) – the percentage of available ad inventory your company purchased.  Measuring SoV online, sometimes called impression share, is easier.  The percentage of times an online ad unit appears compared to the number of times it could have been shown, based on the total available ad space inventory, is the SoV. Offline, companies calculate estimates based on the total amount of money spent; for example, the amount spent on print advertising in national newspapers.
• Click-through-rate – the percentage of viewers of an online ad who clicked on it.
• Conversion rate – the percentage of viewers who took the desired action to achieve the business outcome, whether subscribing, registering, or actually purchasing something.

Marketing Program Mix

Using the metrics discussed above, a head of marketing can measure each program’s effectiveness and return on investment.  Certain programs will stand out as winners, while others will drop to the bottom.    Analyzing the mix of programs on a regular basis will help marketing to optimize its spending and enhance its contributions to the top line.

Analyzing the marketing program mix will also help battle two common marketing problems – the “sacred cow” and the “budget rut.”  The sacred cow, or “we always do that show,” can be a problem in marketing.  People get attached to certain events, reports, tools — even vendors.  Program analysis can help break this counterproductive pattern by providing statistical evidence that these treasured programs are not as effective as others.

The budget rut occurs when individual marketing teams get comfortable with their annual budgets and expect to receive the same level of funding in the coming years.   Becoming accustomed to an annual budget – the rut – can squeeze other, potentially more successful tactics and programs.  Using statistics to demonstrate a lack of efficacy on the part of these programs can help justify budget shifts.

There will always be marketing activities that don’t contribute directly to a lead and therefore to revenue.   A head of marketing can assign a value to a program based on its contribution to a “marketing-assisted lead,” which contributes indirectly. Alternatively, he or she can set aside a certain amount of funding for “keeping-the-lights-on” functions, such as maintaining the Web site, as well as awareness and influence programs like PR and AR.

Marketing Benchmarks

Tracking performance over time will, of course, let you know if your marketing performance is improving or worsening.  In addition, using A/B testing identify the relative performance of assets, headlines, Web pages, and other tools.  The question remains, though: How will your organization know where it stands against industry norms and how much potential improvement is possible?  That’s where marketing benchmarks – norms collected from a number of marketing organizations –  come in.

There are a number of organizations that produce annual benchmark reports, including the Corporate Executive Board, Forrester Research, Marketing Sherpa, and Sirius Decisions.  These companies create benchmarks for a broad range of categories including lead conversion, time spent on Web sites, ad click-through-rates, direct marketing efficacy by media, and many others.  Some organizations-  and publications – also survey buyers on how they like to be marketed to, what they pay attention to, and how they find information about products they buy.

Marketing benchmarks are also very useful in the budgeting process.  Based on these data, you may decide, for example, to shift funds from an area that is performing above average to one that needs more help.  Benchmarks can help defend your overall marketing budget by demonstrating that your organization is more effective, and hopefully more cost effective, than your competition.

Learning More

• Web Site Measurement Hacks, Eric Peterson, O’Reilly, 2005