Driver importance analysis is used to help direct resources in order to achieve organisational goals. Driver importance analysis is also known as Importance analysis, Key Driver analysis and Preference regression.
The goal is to quantify the impact of drivers upon an outcome (e.g. satisfaction with offer, pricing, complaints process etc. upon overall satisfaction). The motivation for this in turn is the underlying assumption that keeping customers satisfied leads to either higher sales or retained sales (loyalty). There are a range of approaches to measure the impact, each with strengths and weaknesses, depending on the data type.
Outcomes could be customer satisfaction, advocacy, value for money, feature importance or appeal. Drivers could be statements, messages, features or events. Once importance is established, frequency of events may also have to be considered as more frequent events have a higher impact.
We’ll outline some of the more popular approaches and get to our conclusions. In the new tradition of the ‘inter-web’ the TLDR (too long; didn’t read) summary is:
In our view, if starting from scratch aim to use MaxDiff for more than 7 drivers. If 7 or fewer drivers, rankings give you the best value for money, but don’t lose sight of driver touchpoint frequency and correlations. If starting with ‘scored drivers’ and outcomes or survey time or method precludes MaxDiff, ‘Johnsons Relative Weights’ is the way to go, supported by correlations.
Onto the discussion …
The most straightforward approach is to simply ask for the importance of each driver on a common scale. This is easy to do for short lists and average scores are easily communicated. However, frequently the drivers have been selected because they are known to be important and this is reflected in the scores, with the averages all tending towards ‘important’. This may also be affected by ‘scale use bias’, where individuals only use part of the scale.
Ranking items would seem to address this. Again, some sort of averaging or statistical analysis is required (average rank, where low scores are better, or probability of being in first place, where high scores are better); these techniques produce similar results and clear leaders emerge. For practical psychological reasons, this should be restricted to short lists of around seven items or less (single words are easier than complete phrases).
In the 1980’s Dr Jordan Louviere proposed Best-Worst scaling to take advantage of people’s ability to recognise extremes. After evaluating the best and worst items from short sub-lists, the unevaluated middle ranks can be partly inferred. Across several questions the ‘gaps’ are filled in. This technique is also known as MaxDiff (Maximise differences) and can cope with longer lists of up to 30 items before more complex variants are required.
An example MaxDiff task is below:
When thinking about buying research which is most and least important to you?
| Most important | Item | Least important |
|---|---|---|
| o | Project timelines | o |
| ♦ | Friendly contacts | o |
| o | Analytic ability | o |
| o | Project cost | ♦ |
| o | Visualisations | o |
MaxDiff produces a ‘score’ which can be averaged. Re-coding the score at an arbitrary threshold (we usually use either the average score or one corresponding to top n items) allows the use of TURF (total unduplicated reach and frequency). This can find which combination of items make the most people happy (e.g. if you were running a restaurant, ‘steak and chips’ might on average be higher, but including ‘nut roast’ you could also attract vegetarians).
Moving back to ‘raw scores’ we can try a variety of statistical techniques, although they are strongly constrained by a ‘halo’ effect (a good experience will tend to inflate the scores of everything evaluated and a bad experience, vice versa, masking the impact of an individual driver).
The most straightforward statistic is to just average. Next are correlations which reveal relationships. By correlating drivers with outcomes we can obtain a score as to how each driver and the outcome tends to vary together. This quickly highlights that drivers also tend to co-vary together. This hidden structure can be revealed using Principal Components Analysis (PCA) or Exploratory Factor Analysis (EFA). As an aside, these are broadly the same technique with the main difference being that PCA has one solution, which may not be ‘interpretable’, but nevertheless reduces the complexity of the data; EFA solutions can be manipulated for additional clarity and interpretation. Confusion between them occurred and remains today, due to mis-labelling in software popular with social scientists.
Factor Analysis was developed with the assumption that it could help identify the hidden (latent) uncorrelated factors that are impossible to directly measure. Drivers that ‘load’ (correlate) onto the same factor are likely to be measuring aspects of the same thing. Factors which only have one significant driver mean that driver is representing something unique. Factor Analysis can inform which one of several correlated variables to use in modelling (each one can be used as a proxy for the others, but if used together in a regression you can get difficult to explain results).
Drivers are often thrown piecemeal into a linear regression. For this discussion, we’ll assume that we know, a priori, that relationships should be positive. Due to the ‘halo’ effect and underlying latent variables, throwing everything in will almost certainly result in some negative coefficients, implying that ‘more’ of a particular driver will have a negative impact on the outcome. This appears to make no sense until we consider the relationship between drivers and realise we have two or more similar measures (identified by loading on the same factor). One measure will be a better predictor and will ‘grab’ the positive impact, leaving only the effect of random negative differences for the ‘negative’ driver to be measured. We can use factor analysis to progressively remove drivers that ‘load’ more weakly on a factor and which display a negative coefficient (or aren’t statistically significant). This is an iterative process to curate an acceptable model.
Or we could just specify a non-negative least squares optimisation. However, this is mathematical, rather than statistical in nature. It severely violates assumptions about normal errors, it can’t guarantee a unique solution and the final model can’t be statistically assessed. On the plus side, it does meet the criteria of face validity. In combination with correlations between the excluded and included drivers, based on a factor analysis, this may be informative enough.
An alternative to multiple linear regression is ‘drivers importance analysis’. It has been reinvented several times and seems to be the best solution yet to a regression approach. This is known by many names (e.g. Shapley, Kruskal, Dominance analysis and Relative Importance analysis). Purporting to explore all possible models, based on systematically including and excluding drivers, this averages the impact of drivers across all models (the impact might be in the ‘wrong’ direction).
In practice, the number of models to investigate grows exponentially with the number of drivers, so mathematical shortcuts that closely approximate the true result have been developed. Relative Importance Analysis uses approximations to handle more drivers and produces very similar results. Again, PCA can inform which variables to include (all variants of outcome variables apart from the outcome of interest should be excluded). These techniques though are limited to multiple linear regression models; in market research we generally have scales with upper and lower boundaries.
Generalizing linear models (to be able to accommodate ordered categorical outcomes) and rotating data (principal components) to remove correlations is the realm of Johnsons Relative Weights. This again emulates exploring all possible models, just with a different outcome type. This produces similar results to Shapley (if applied to scale data), can measure more output types (scale, ordered categories etc..), is computationally faster, can accommodate statistical tests and one can include more predictors.
As the TLDR indicated, in our view, if starting from scratch aim to use MaxDiff for more than 7 drivers. If 7 or fewer drivers, rankings give you the best value for money, but don’t lose sight of driver touchpoint frequency and correlations. If starting with ‘scored drivers’ and outcomes, or survey time or method precludes MaxDiff, ‘Johnsons Relative Weights’ is the way to go, supported by correlations. There is rarely a perfect solution, as each project makes its own compromises and the data that we work with is derived from sample and ultimately, subjective humans.