I have used a simplified AHP ( i.e. weighted tree) for over fifteen years to facilitate clients making strategic and procurement decisions. I read with interest the article by Ian Alexander titled ‘Discovering requirements: from wish list to want list’ in the IET magazine E&T dated 10 Oct-23 Oct (http://kn.theiet.org/magazine/issues/0917/wish-list-to-want-list-0917.cfm).
Alexander criticises AHP and ‘naive weighting’ as adding ‘apples to oranges’. He then goes on to discuss the merits of PCA. This technique maps evaluation scores from the scoring dimensions (e.g. functionality, cost, etc) into a set of new dimensions using linear transformation. The first new dimension is chosen to maximise the spread the scores of the solutions being evaluated i.e. it shows the differences between the options in the greatest possible way.
From Wikipedia, (http://en.wikipedia.org/wiki/Principle_component_analysis), ‘PCA is mathematically defined as an orthogonal linear transformation that transforms the data to a new coordinate system such that the greatest variance by any projection of the data comes to lie on the first coordinate (called the first principal component), the second greatest variance on the second coordinate, and so on’. For a data matrix, XT, with zero empirical mean (the empirical mean of the distribution has been subtracted from the data set), where each row represents a different repetition of the experiment and each column gives the results from a particular probe, the PCA transformation is given by:
YT = XTW
The transformation matrix W is generated using the original scores and is based on eigenvectors and eigen values analysis. This transformation is a simple matrix multiplication where each element of YT is the sum of the appropriate elements of XT multiplied by a set of elements in W. Clearly this is adding apples to oranges, the very sin that Alexander attacked AHP for.
I am happy that PCA gives a useful view of the data but equally AHP gives a client view of the data; both methods add apples to oranges. Interestingly, one of the claims for AHP is that it enables you to add dissimilar selection criteria to create a single score.
My experience is that it is the client team going through the process that brings the important consensus and the simpler and more transparent the process the better. It is not the tool making the decision but the team, as it is they who have to implement the chosen solution.
During most of the selection work I have done you end up having to select from a short list of around three or four solutions with some dozen criteria. PCA uses eigenvector and eigenvalue techniques on the covariance matrix of the scores. With a short list of three or four the statistical significance of the solution is going to be extremely poor.
A case history weighted tree is shown below.
There are four strategic options being marked against eleven criteria. I normalised the scores to zero means and used ViSta to run a DCA. It refused to run with only four sets of marks; ViSta requires at least a square matrix before it will calculate results i.e. more options than parameters. I forced a calculation by adding dummy options with zero scores and it scored the options:
1 -1.9
2 -0.6
3 -0.1
4 1.1
Actually, the same order as the client weighted scores!?!
The PCA criteria weights were:
1.3 -.31
2.1 .35
2.2 .35
2.3 -.39
3.1 .32
3.2 -.01
3.3 -.39
3.4 .02
3.5 -.31
Comparing these weights with the row scores in the table above gives:
- Criteria with high spread of scores get a high PCA weight.
- Criteria with low spread of scores get a low PCA weight.
All as you would expect from PCA.
The central limit theorem comes to our rescue at with, say, ten options to score and statistically significant results should therefore be generated at this level.
However, I feel that DCA cannot be used in a typical short list situation because of the significance issue.
In most final selection processes we have:
- four/five options from which to choose a winner,
- around ten attributes to rate and
- around ten stakeholders willing to mark the bids.
At the long list stage, the number of options could be much higher and the selection process often uses a set of go/no go criteria. In truth, it is often a cut process rather than selection of the best.
When the final proposals have been marked, we have a ‘slab’ of data:
XK with dimensions I*JK*K where:
I is the number of options (say five)
K is the number of stakeholders (say ten) and
JK is the number of attributes marked.
Therefore, there are IK sets of marks for each option.
PCA is a technique that is usable on two-dimensional data; it cannot be sensibly used on three-dimensional data.
MFA (and Tucker3) were developed to handle ‘slab’ data and create PCA results.
The technique uses covariance calculations therefore the dimensions of the slab of data is important. In a major bid process you are only likely to have around five proposals to mark and the results are going to lack statistical significance. The central limit theorem comes to our rescue at ten and above when things start looking Gaussian, but it is not practical to try and get ten or more bids. MFA therefore suffers the same issues as PCA when a dimension of the data is small.
MFA will give a view on the data, but beware of the poor statistical significance of the result.
Of course, the ‘significance’ of the stakeholders holding a meeting and agreeing weights and score to select a winner could be challenged….
Ian Richmond
Email: 'About' and 'Email Me' link