Indices behaving badly

I have been thinking about indices in ecology and environmental science lately, partly because of concerns about the SAFE index. Indices are used for environmental monitoring, setting management priorities, and evaluating management actions. They are also found in basic ecology (see Table below). However, while some ecological indices are very helpful, others are built and used in dubious ways. Let me explain with an example.

A little while ago, I reviewed a manuscript that developed a particular index. The details are not important here, except to say that the authors hoped that the index correlated with a standard error¹. The index included some important aspects of the problem, but combined these in essentially an arbitrary way.

In reviewing the manuscript, I stated that indices should not be arbitrary combinations of variables. The authors responded along the lines of “But ecological indices are arbitrary combinations of variables!” They are correct, of course. In ecology, indices are commonly just that. They are typically ad hoc combinations of variables that the author thinks are important. But should this be the case? I think not.

An index should in some way approximate a quantity that we care about. In this particular case, the index should somehow approximate, or at least be correlated with, the standard error. At least in this case it is clear how we would evaluate the index – by seeing how strongly it correlated with the actual standard error.

In another sphere, the various indices that are used to measure vegetation condition or quality in Australia (e.g., Habitat Hectares, Biometric) are essentially arbitrary combinations of variables. They measure important attributes of vegetation to which particular animals respond (e.g., tree cover, occurrence of large trees, logs) and other attributes that inform the condition of the vegetation (e.g., presence of weeds, understory cover), assign points for each attribute, and then sum the points.

How do we know if such an index of vegetation quality is meaningful? Well, to answer that, we would need to answer ‘What is “vegetation quality”, and how is that measured?’ We might then simply plot, for a range of values of the index, a precise measure of vegetation quality versus the index, and examine the strength of the relationship.

But indices of vegetation quality have the inherent problem of not being developed as indices of a measurable quantity. It is not clear what quantity should be measured to evaluate them, because “vegetation quality” is not clearly defined. If we find that butterfly species richness does not correlate with the index, does that mean the index has failed, or simply that it is not meant to be related to butterflies? It is unclear how the indices could be improved with accumulation of knowledge. In fact, these indices cannot be proven to be poor or good, because it is not clear what they are meant to be indices of. This lack of clarity about these indices is alarming because decisions worth millions of dollars are based on them.

So, how would one go about developing an index of vegetation quality? To start, we would need to define what we want our index to measure. Let’s assume that we only cared about birds that depend on remnant vegetation. We might then relate the species richness of these selected birds to attributes of the vegetation. This could be done through expert elicitation, by using data and statistical modelling, or perhaps a combination of both. Regardless of how the index was developed, we would know immediately how it should be evaluated – by seeing how well it predicted species richness of these birds. And those evaluation data could be used to improve the index.

But, I hear you say, vegetation is important for more than birds. For example, it is also important for maintaining water quality in streams. Immediately we see another index – one that seeks to be related to water quality. Now we have two indices, or perhaps more if different aspects of water quality or biodiversity are considered. These indices could be combined into one, but doing so implicitly assumes that they are tradable (an increase in one compensates for a reduction in the other). A single index might be needed, but such a trade-off should be explicit, not hidden in an arbitrary combination of variables.

I suggest that indices used in environmental management should aim to approximate quantities that people care about: aspects of biodiversity, ability to influence water quality, aesthetics, etc. The particular components should be stated explicitly, so that we also know what it is not an index of. In short, indices should be models, with plausible links to processes. Otherwise, we are left with an index that is very difficult to evaluate or improve.

1. The standard error is a measure of how uncertain an estimate is. It is a standard statistical quantity, so at face value, the need for an index in this case is not clear when an actual measurement already exists. An index might be useful in this case if the standard error was difficult to calculate with the available data, while a useful index could be calculated.

Some examples of indices used in ecology and environmental management. You can decide for yourself whether they are indices of measurable quantities, and whether they can be evaluated and improved.

Indices	Notes
Indices for monitoring
Index of stream condition	A combination of five different sub-indices to represent river health.
SIGNAL score	A component of the index of stream condition, based on a weighted mean of scores for different taxonomic groups of freshwater macro invertebrates. The scores depend on the relative sensitivity of the taxa to degradation of water quality.
Living Planet Index	Relative geometric mean abundance of a set of monitored species.
Indices for prioritization and evaluation of management
PPP efficiency	An index that measures the expected benefit of undertaking a particular conservation project, based on the reduction in the risk of extinction within a specified time frame (Joseph et al. 2009).
Habitat hectares	An amalgamation of vegetation attributes that aims to describe vegetation “quality”.
Abundances of arboreal marsupials	The density of arboreal marsupials in Victoria’s tall eucalypt forests.
Indices in ecology
Shannon diversity index	The entropy of the species-abundance distribution in an ecological community.
All those landscape indices from FRAGSTATS	A diverse range of indices that are commonly used in landscape ecology. Few can be easily related directly to ecological processes, except through heuristic models.