Have you ever noticed client service ratings that change substantially from year to year, or market research findings that just don’t seem to make sense? There’s a reason.
Historically, few buyers of market research for professional service firms have had formal training in research methodology or statistics. This has improved somewhat in
recent years, but serious expertise remains unusual. As a result, law firms and other professional service organizations often fall prey to vendors selling research services (and research-based products such as attorney ratings) with little or no methodological rigor. This causes firms to make important strategic decisions believing they are acting from sound information when in fact they probably would be better off making “judgement calls;” at least then they would recognize the fragile underpinnings of their decisions and not be misled into unjustified certainty by shaky research.
When the purveyors of these dubious research services started to gain serious traction more than a decade ago, the Legal Marketing Association wisely directed its Market Research Committee to draft a set of guidelines to use when commissioning such services. The principles of statistics and sound research methodology have not changed so these recommendations are still valid today.
If you are considering commissioning any sort of market research services, I urge you to read the complete document (LMA Research Guidelines). Everyone in marketing should be conversant with these Top 10 Recommendations:
- Go to great lengths to achieve very high survey participation rates. Otherwise you will have no basis for assuming that your findings represent the population of interest.
- Use only interviewers you will be proud to have representing you to your clients (client research).
- Do not treat qualitative research (e.g., focus groups) as conclusive. This is probably the most common mistake that law firms make.
- Demonstrate to clients the benefit to them of taking time to participate in the survey (client research).
- Make sure you know who will be conducting each stage of your work and their qualifications. Don’t let a research firm’s salesperson perform a “bait and switch.”
- Make sure you know what determines the purchase decisions of your market/clients, and that your questionnaires assess performance regarding those determinants.
- Pretest all questionnaires for logical flow and wording.
- Do not conduct “nice to know” research. Every question in each study should be included only if the results are likely to drive action by the firm.
- Interview the right people. Make sure your lists are accurate and include the persons who will substantially influence the decision to use your services.
- Include “implications” in your reports that will drive action by your firm. A tomb of numbers sitting on a shelf is of no value.
Misleading information is abundant. A prime example is when a market researcher reports they’ve interviewed 1,000 buyers and proceed to show numerous charts and tables WITHOUT indicating the number of respondents for each specific question or rating each firm. They may have interviewed a 1,000 people overall but only 6 answered a specific question. Demand that your market research company provide complete information and transparency. Chart and tables without the “n” indicating the number of responses for that specific question can’t be relied be upon. (And what if they emailed their survey to 10,000 people and only the 1000 most interested in the topic responded? How representative are they of the population of interest?)
The bottom line is that you should (and have every right to) demand complete methodological transparency from your vendor. If they will not happily answer every question you pose about respondent demographics, sample sizes, response rates and confidence ranges, do not engage their services. Credible vendors such as Acritas and Allison Market Research do not hesitate to provide such information.