Surveys

Surveys are frequently used to assess thoughts, behaviors, opinions, needs, of stakeholders either inside and outside the client’s organization. Contemporary survey methodologies take the form of questionnaires that lend themselves to computer-based statistical analysis. For example, closed-ended questions, where the respondent picks up an answer from a given number of options, can be analyzed using simple descriptive statistics. Open-ended questions in contrast might involve sophisticated qualitative predictive modeling, or be coded into a response scale that will subsequently be treated quantitatively. A survey generally contains both closed-ended and open-ended questions. The first type is easier to record and score but diminishes expressivity and spontaneity. Open-ended questions allow responders greater flexibility but are difficult to record and score.

A survey contains thousands of features that can potentially be improved, thus careful design and interpretation are pre-requisites. For example, to generalize the findings from a sample to a population, it is necessary to avoid the so-called selection bias (over-representation of a given gender, race, education level, etc). Random and stratified sampling are often used to minimize selection bias, where stratified sampling consists in dividing the population into sub-populations and drawing a random sample from each of these “strata”.

Another common challenge of statistical surveys are spurious relationships, or the correlation-causation fallacy, where an observed correlation between two variables can be explained by a third (often unknown) variable. The consultant might be tempted to develop a strategy/action plan based on the existence of causality between the two variables. But when variable 1 is not causing variable 2, the correlation between the two variables might change of rate, or even disappear, when the value of variable 3 falls into certain ranges.

Another challenge is the unreliability of generalizing past and present to predict the future. It is common in newspapers to read about “attributes” of a population inferred from a rigorous survey, when the attributes in questions are dynamical variables and thus their past trajectory can hardly predict their future. In a so-called cross-sectional study, the survey is carried-out at one particular time; in successive independent studies, a survey is carried out at multiple time points, attempting to draw samples that are equally representative of the target population; in longitudinal studies, a survey is also carried out at multiple time points but on the same random sample. Longitudinal studies are the most rigorous. But even a perfectly orchestrated longitudinal survey will not predict the future. Predicting a company’s bottom line in a year is akin to forecasting the weather in a month; the past trajectory is indicative of its future, but any forecast must come with an associated probability. The consultant should always state a level of confidence.

Method:

- Decide what kind of information should be collected, and between 1-time vs. longitudinal study

- Construct a first draft of questions. Questions should be simple and direct. Start with questions that catch the respondent attention and move demographics questions toward the end

- Pre-test the questionnaire with a pilot

- Edit the final questionnaire and specify clear instructions to the attention of respondents

- Decide upon classification criteria (age, gender, location, education, income, characteristic needs, jobs-to-be-done) and use these criteria to select a representative sample of a target population using techniques aimed at avoiding selection biased (see above)

- Develop an action plan to increase response rate and reliability. For instance: introduce the survey with friendly and instructive comments (length of the survey, goal, interviewer’s personal/organization name), send an advanced letter to assess cooperation beforehand, provide incentives (gift, lottery), test-retest to assess consistency of respondent’s responses, avoid interviewer-effects.

- Complement insights from verbal surveys with direct observation to determine with better accuracy the behaviors that customers really engage in, the attitudes they truly uphold. 

Back to methodologies