Mystery shopper

From CEOpedia
Mystery shopper
See also


Mystery shopper - is a person who poses as an ordinary customer in order to observe, experience and evaluate the service a company (or its competitor) delivers, and then reports back against a fixed set of criteria. ESOMAR describes mystery shoppers as people "trained and/or briefed to observe, experience and measure any customer service process by acting as a prospective customer and undertaking a series of pre-determined tasks to assess performance against specific criteria, reporting back on their experiences in a comparable and consistent way"[1]. The method itself is called mystery shopping, and in various countries and industries it goes by other names: mystery customer research, secret shopping, ghost shopping, or simply service evaluation.

The idea is old and simple. Managers can't stand behind every counter, and customers who fill in a survey remember the overall mood rather than whether the cashier offered a receipt. So you send in someone who looks like everybody else, give them a script and a checklist, and ask them to pay attention. What comes back is not opinion in the usual sense but a record of what happened: how long the wait was, whether the adviser asked about needs before pitching a product, whether the toilets were clean. Banks, retail chains, hotels, car dealers, restaurants, pharmacies, public offices and regulators all use it.

History

Practitioner accounts usually trace mystery shopping to the United States in the 1940s. At first it had little to do with smiles and greetings. Banks and retailers hired outside investigators to check employee honesty: did the cash go into the till, were prices charged correctly, were rules on handling money followed. These were integrity checks, closer to loss prevention than to marketing research.

The shift towards service measurement came gradually, as competition in retailing and financial services moved from price and location to the quality of contact with staff. By the 1990s the method had become an established tool of service management in Europe and North America, and academic researchers began to study it. Alan Wilson's work in the United Kingdom examined how organisations in several service sectors used mystery shopping to measure and manage service delivery[2].

The industry organised itself around the same time. The Mystery Shopping Providers Association (MSPA) was established in October 1998; within a year it was drafting a code of professional ethics and standards, campaigning against misleading adverts that promised shoppers easy money, and had been asked by ESOMAR to contribute to its own guidelines on mystery shopping for market research[3]. The association later renamed itself the Mystery Shopping Professionals Association and now operates through regional bodies. Nowadays a large share of visits are booked, reported and paid through online platforms, with photos, receipts and timestamps uploaded straight from a phone.

Types of mystery shopping

The method comes in several forms. ESOMAR lists mystery observation, mystery visits, mystery telephone calls, mystery mail or fax and mystery e-mail or website visits[1]. In practice the main distinctions are these.

By channel

  • In-person visits - the classic form. The shopper enters a branch, store or restaurant, interacts with staff and may make a purchase.
  • Telephone shopping - calls to a contact centre or a branch. Easy to record (where the law allows), cheap per contact, good for measuring answer times and script adherence.
  • Online and digital shopping - e-mail enquiries, web chat, online ordering, app journeys, social media messages. The checklist covers response time, accuracy and tone.
  • Mixed journeys - a shopper starts online, books an appointment and then visits, so the whole customer perception chain is tested, not one touchpoint.

By what is measured

  • Audit-type shopping - factual, yes/no checks against company standards: signage, cleanliness, price labels, opening hours, uniform, whether a mandatory disclosure was given. Little room for interpretation.
  • Experience-type shopping - the shopper also reports how the contact felt: was the adviser helpful, did the explanation make sense, would they come back. Richer data, more subjectivity.
  • Integrity and compliance shopping - checks for theft, age-restricted sales (alcohol, tobacco), mis-selling of financial products. Regulators use this form too.

By whose service is tested

  • the company's own outlets,
  • intermediaries such as franchisees, dealers, agents and distributors,
  • competitors, which turns mystery shopping into a benchmarking tool.

Process of mystery shopping

A programme that produces usable numbers follows roughly the same steps, whoever runs it.

1. Objectives. What decision will the results feed? Training needs, a bonus scheme, compliance monitoring, a comparison with competitors? The answer drives everything else, including (as discussed below) the ethical rules that apply.

2. Standards and criteria. The company's service standards are translated into observable behaviours. "Friendly service" can't be scored; "greeted within 30 seconds of entering" and "asked at least two questions about needs before recommending a product" can. Many firms map criteria onto service quality dimensions so results can be read alongside customer surveys.

3. Scenario design. The shopper needs a believable reason to be there: opening a savings account, buying a laptop for a child starting university, asking about a mortgage. Scenarios have to fit the outlet and the shopper's profile; a 22-year-old asking about pension annuities attracts attention. ESOMAR requires that scenarios be safe, so shoppers are never asked to do anything illegal or anything that puts them at physical risk[1].

4. Questionnaire and checklist. Typically a mix of yes/no items, timings, rating scales and open comments. Each item gets a weight reflecting its importance to the business.

5. Recruitment and briefing. Shoppers are matched to the target customer profile and briefed on the scenario and the checklist. Training matters a lot. A shopper who doesn't know what "needs discovery" means will score it inconsistently.

6. Fieldwork. Visits are spread across days of the week and times of day, because a branch at 10 a.m. on Tuesday isn't the same branch at 5 p.m. on Friday. Shoppers write up the report immediately after leaving; memory fades fast, and research on mystery customer accuracy points to encoding, storage and retrieval of information by assessors as the main sources of error[4].

7. Validation and scoring. The agency checks reports for consistency, completeness and evidence (receipts, timestamps, photos). Scores are calculated per item, section, visit, outlet and region.

8. Reporting. Results go to managers, often as dashboards with rankings, trends and verbatim comments.

9. Feedback and action. The step that's skipped most often. Results only change service when they reach the front line, get discussed, and lead to training, coaching or process changes. Wilson's research among companies using the method found that mystery shopping results were rarely combined with other measures of service performance[5], so the data often stayed in a silo.

Mystery shopping and service quality models

Mystery shopping measures what the firm delivers, not what the customer thinks of it. That distinction matters when the results are placed next to service quality models.

The SERVQUAL model of Parasuraman, Zeithaml and Berry describes service quality as the gap between customer expectations and perceptions across five dimensions: tangibles, reliability, responsiveness, assurance and empathy[6]. In the wider gap model, one of the gaps is between the service standards a company sets and the service actually delivered by staff. Mystery shopping is aimed straight at that gap. It answers the question "do our people do what our standards say?" Customer surveys and satisfaction ratings answer a different one: "do customers like what they get?"

The two can disagree. A branch might hit every standard and still disappoint customers, because the standards themselves are wrong. Or customers might be happy with a branch that ignores half the checklist. Good programmes look at both, and some firms test whether outlets with higher mystery shopping scores also have better customer loyalty or sales. Where they don't, the checklist probably measures the wrong things.

Ethics and professional standards

Mystery shopping involves deception. Staff serve someone they believe is a real customer, and they don't know they're being assessed. Wilson's 2001 paper looked at the method precisely as a form of deception used to measure service performance[7]. Professional codes deal with it in several ways.

The ESOMAR guideline on mystery shopping studies (last revised 2005) accepts that the people observed are normally not aware at the time, since awareness "might lead to atypical behaviour which would invalidate the findings of the study". In return it sets conditions[1]:

  • when a firm shops its own organisation, staff and any staff body (such as a works council) must be told in advance that shopping will take place, what its objectives are, whether individuals will be identified and whether recordings will be made;
  • the time taken from staff of competitors or other organisations must be reasonable and kept to a minimum, especially for self-employed people whose time costs them money;
  • bookings or reservations made as part of the scenario must be cancelled as quickly as possible;
  • recordings must not be kept longer than needed, and are passed to the client only with the consent of the people recorded.

ESOMAR also draws a line between two types of study. Mystery shopping research keeps the people observed anonymous and uses the data only for research; its findings must not be used to discipline anyone. If results are used to identify individual staff for training, sales reinforcement or a bonus system, the study must not be called research and is instead a mystery shopping project[1]. And if there's only one employee at the outlet, anonymity is impossible, so the research label can't be used either. Studies designed to fake consumer demand for a product (sending "shoppers" to ask for it so the retailer stocks it) must not be undertaken at all.

The MSPA, for its part, has a common code of professional standards and ethical conduct for its members; member firms describe it as stressing honesty, professionalism, fairness and confidentiality. Data protection law adds its own requirements, as do national rules on recording phone calls.

There's also an ethical issue on the shopper side. Mystery shoppers are usually independent contractors paid per visit, and fake "secret shopper" job offers are a well-known fraud pattern.

Example of a mystery shopping score

Hypothetical example. A retail bank runs a mystery shopping programme in four branches. Each shopper asks about opening a savings account. The checklist has five sections, each scored 0-100 and averaged over the visits made to that branch. Weights reflect what the bank cares about most.

Section Weight Branch A Branch B Branch C Branch D
Greeting and waiting time 15% 92 60 95 80
Needs discovery 25% 70 88 55 78
Product knowledge 25% 78 90 62 74
Disclosure and compliance 20% 95 70 90 85
Closing and follow-up 15% 62 78 50 70
Unweighted mean 79.4 77.2 70.4 77.4
Weighted score 79.1 79.2 69.0 77.5

The weighted score is

\[ S = \sum_{i=1}^{k} w_i \cdot s_i, \quad \sum_{i=1}^{k} w_i = 1 \]

For Branch A:

\[ S_A = 0.15 \cdot 92 + 0.25 \cdot 70 + 0.25 \cdot 78 + 0.20 \cdot 95 + 0.15 \cdot 62 = 13.8 + 17.5 + 19.5 + 19.0 + 9.3 = 79.1 \]

For Branch B:

\[ S_B = 0.15 \cdot 60 + 0.25 \cdot 88 + 0.25 \cdot 90 + 0.20 \cdot 70 + 0.15 \cdot 78 = 9.0 + 22.0 + 22.5 + 14.0 + 11.7 = 79.2 \]

Two things stand out. First, weighting changes the ranking. On a plain average Branch A leads (79.4 vs 77.2), but once needs discovery and product knowledge get more weight, Branch B edges ahead. Neither result is "correct"; the weights are a management choice and should be agreed before fieldwork, not after the league table is out. Second, the branches have very different profiles. Branch A is polite and compliant but weak at selling. Branch B sells well but leaves customers waiting and skips disclosures, which in a bank is a regulatory risk. Branch C has the same pattern as A, only worse. A single score hides all of that.

How reliable are the numbers? Suppose a branch's score comes from just three visits, scored 84, 77 and 62. The mean is 74.3 and the standard deviation 11.2 points. A rough 95% margin of error under a normal approximation is

\[ 1.96 \cdot \frac{11.2}{\sqrt{3}} \approx 12.7 \text{ points} \]

With so few visits this is optimistic: a t-based interval would be more than twice as wide. With 12 visits the normal-approximation margin shrinks to about 6.4 points. So with three visits per branch, even the 8.5-point gap between Branch C and Branch D may be partly noise. Paying bonuses on it would be hard to defend.

Advantages of mystery shopping

  • Measures actual behaviour against defined standards, not recalled impressions.
  • Gives detail that customer surveys can't: timings, exact wording, whether a specific step happened.
  • Works where customers can't judge quality themselves, for instance whether a financial adviser gave the required risk warning.
  • Allows comparison with competitors and between own outlets, regions and channels.
  • Shows which training topics are needed and lets firms check whether training actually changed behaviour.
  • Results are concrete enough for front-line staff to act on ("you didn't offer the brochure").
  • Supports service quality control in franchise and dealer networks where head office has little direct oversight.
  • Can be repeated in a consistent way, so trends over time are visible.

Limitations of mystery shopping

  • Small samples. A branch often gets one to three visits per month or per quarter. As the example shows, that's rarely enough to separate real differences from chance, a classic sampling error problem. Finn and Kayandé's psychometric assessment of mystery shopping is a reminder that outlet-level scores need to be tested for reliability before anyone treats them as facts[8].
  • Subjectivity and shopper error. Two shoppers in the same situation may score it differently. Memory distortion, a bad mood or a mismatch between shopper and target customer all add noise[4].
  • Snapshot only. One visit captures one employee at one moment. A rude reply on a chaotic Saturday may say little about normal service.
  • Cost. Each visit costs shopper fees, travel, purchases, agency margin and management time. Large samples get expensive fast, especially for complex scenarios like mortgage advice.
  • Staff reactions. Employees may feel spied on, and morale suffers if results are used mainly to punish. Staff who learn to spot shoppers may perform for them and treat everyone else differently, which distorts the data. Linking results to performance appraisal or pay raises the stakes and the tension.
  • Measures standards, not satisfaction. A high score shows compliance with the checklist; it doesn't prove customers are happy or loyal.
  • Ethical and legal limits. Deception, recording and identification of individuals are restricted by codes and data protection law.
  • Burden on others. Shopping competitors takes their staff's time for nothing, which the ESOMAR guideline asks researchers to keep to a minimum[1].

Related approaches

Mystery shopping is rarely used alone. Firms combine it with customer satisfaction surveys, complaint analysis, transaction data (conversion rates, waiting times from queue systems), employee surveys and direct observation by managers. Some use announced audits, where an inspector visits openly with the same checklist; less realistic, but cheaper and without deception. Others use customer panels or real-customer feedback collected right after a transaction. Service blueprints and journey maps help decide which touchpoints are worth shopping, and the service quality models help interpret what the results mean. In operations terms, mystery shopping is one input into service operations management and wider quality improvement work.

References

Footnotes

  1. ↑ ESOMAR, ESOMAR World Research Codes & Guidelines: Mystery Shopping Studies, 2005
  2. ↑ Wilson A.M., The use of mystery shopping in the measurement of service delivery, The Service Industries Journal, 1998, pp. 148-163
  3. ↑ Quirk's Media, Association creates new standards for mystery shopping industry, Quirk's Marketing Research Review, 1999
  4. ↑ Morrison L.J., Colman A.M., Preston C.C., Mystery customer research: cognitive processes affecting accuracy, Journal of the Market Research Society, 1997
  5. ↑ Wilson A.M., The role of mystery shopping in the measurement of service performance, Managing Service Quality, 1998, pp. 414-420
  6. ↑ Parasuraman A., Zeithaml V.A., Berry L.L., SERVQUAL: a multiple-item scale for measuring consumer perceptions of service quality, Journal of Retailing, Vol. 64, No. 1, 1988, pp. 12-40
  7. ↑ Wilson A.M., Mystery shopping: Using deception to measure service performance, Psychology & Marketing, 2001, pp. 721-734
  8. ↑ Finn A., Kayandé U., Unmasking a phantom: a psychometric assessment of mystery shopping, Journal of Retailing, 1999, pp. 195-217
Mystery shopper — recommended articles
Service quality model — SERVQUAL — Service quality dimensions — Customer satisfaction — Customer expectations — Service quality control — Benchmarking — Marketing research
Retrieved from CEOpedia