The predictive accuracy of the Habitat Suitability (HS) map can be evaluated by a special cross-validation procedure: the Boyce index.

The Boyce index

Evaluation is done by a cross-validation process (Menu Evaluation/Cross-validation). It computes a confidence interval about the predictive accuracy of the HS model. The species locations are randomly partitioned into k mutually exclusive sets. k-1 partitions will be used to compute a HS model and the left-out partition will be used to validate it on independent data. This process is repeated k times, each time by leaving out a different partition. This process results in k more-or-less different HS maps. By comparing these maps and how they fluctuate, one can assess their predictive power.
By default, the partitions are set so that they do not overlap geographically. This makes the cross-validation more robust to spatial auto-correlation (e.g. conspecific attraction). This is actually controlled by the Randomness field. 0% means that partitioning is determined only by geography. A value of 100% indicates that assigning a point to a partition is determined half by its geographical position and half by random. 1000% indicates that randomness is 10 times more important than geography. And so on…
Note that each time you apply the cross-validation process, you will get the same partitioning, and therefore the same results. If you want another partitioning, change the Random seed value. Each value generate a different set of partitions.
BioMapper follows the method described by Boyce et al. (2002) and further developed in Hirzel et al. (2006). Each map is reclassified in b bins (by default, b=4). Each bin i covers some proportion of the map’s total area (Ei) and contains some proportion of the validation points (Pi) (validation points are those observation left out during the cross-validation process). One computes then the predicted to expected ratio P/E for each bin as Fi=Pi/Ei . If the HS map is completely random, one expects Fi=1 for all the bins. If the model is good, low HS should have a low F (below 1) and high HS a high F (above 1) with a monotonic increase in between. A way to measure the monotonicity of the curve is to compute a Spearman rank correlation on the Fi;, which we call the Boyce index B.
The main shortcoming of the Boyce index is its sensitivity to the number of bins b and to their boundaries. To fix this problem, we derived a new evaluator based on a “moving window” of width W (good values for W range between 10 and 20) instead of fixed classes. Computation starts with a first class covering the suitability range [0, W[ whose P/E ratio is plotted against the average suitability value of the class, W/2. Then, the moving window is shifted from a small amount upwards and the P/E is plotted again. This operation is repeated until the moving window reaches the last possible range, [100-W, 100]. This provides a smooth P/E curve, on which a “continuous Boyce index” Bcont(W) is computed by a Spearman rank correlation coefficient as above.
Practically, one often get a sigmoid curve, the F increasing exponentially and then stabilising and oscillating around a maximum value. Looking at the shape of this curve allows one to define where is the threshold between suitable and unsuitable habitat, from which point the model doesn’t add significant information, etc.. Also, the variance of the Fi among the cross-validated curves reflects the prediction power of the model.

Hirzel, A.H., Le Lay, G., Helfer, V., Randin, C., Guisan, A. (2006) Evaluating habitat suitability models with presence-only data. Ecological Modelling, in press.

Operations

Open the cross-validation (CV) dialog box (Habitat suitability/Area-adjusted frequency cross-validation)

This dialog box is similar to the Habitat Suitability one and should already be filled correctly if you have just computed your HS map. The only new field is entitled “k-fold cross-validation”. Here you can define the CV parameters.

The most important one is the number of partitions (k). You can also choose to keep the temporary HS maps for further analysis or compute confidence limits maps.

Click on the Compute button.

The cross-validation may take some time as k HS-maps must be computed. Once the process is completed, a new dialog box appears, entitled “area-adjusted frequency cross-validation”. The upper graph shows the Fi curves along with the Boyce indices. The panel just below the graphs allows the user to select various display options. The lower graph displays the bins. By default, there are four of them and they have an equal width. The histograms represent the average number of cross-validation points in each bin (in green) and the average area (or number of map cells) covered by each bin (in red). You can change the number of bins with the spin editor below the graph. You can also modify the sizes of the bins, either manually by drawing the histogram bar borders with the mouse, or by selecting one of the bin-partitioning algorithm in the lower panel. You can choose between equal width (all bins cover the same HS range), equal counts (BioMapper tries to set bin limits so as they all have about the same number of cross-validation points) or equal area (BioMapper tries to set bin limits so as they all cover about the same area). The equal counts is somewhat similar to the operation described in the paper by Boyce et al. and seems to give the best results. There is still some work going on this part too.
If you want to compare several models later, you may want to store the cross-validation data into a file. You can do so with the menu Cross-validation/Save CV data…. These data can be opened again later by the menu Evaluation/Load CV data…, which brings back the cross-validation dialog box. The CV-data file is in text format and can be opened in another program if you want to make your own analysis of them. You may also want to save the continuous P/E curves (Cross-validation/Save continuous P/E curves…) for display in another application.
Once you have chosen the relevant bin limits, you can reclassify the HS map to present only these new categories by clicking on the Reclassify HS map button.
You can also save the graphs (Cross-validation/Save graphs…) in WMF or EMF format, which you can use for instance in Word or Powerpoint.
Finally, by leaving the cross-validation box, you will find a summary of the cross-validation statistics, along with the bin limits and old validation indices (AVI and CVI), in BioMapper’s main result window. Remember that all presence-only evaluation indices (and probably presence/absence indices as well) are always somewhat sensitive to the study area.

Old evalutation indices

The Absolute Validation Index (AVI) and the Contrast Validation Index (CVI) were developed before Boyce indices and cross-validation processes. They are still provided in Biomapper for backward compatibility as a side results of the cross-validation procedure, the Boyce continuous index should be preferred.
The AVI is the proportion of validation cells that have a HS>50. It evaluates how good a job the model is doing (i.e. predicting high HS for cells where the species is present). However, an AVI of 1 (best value) could be obtained by a model predicting presence everywhere. In other words, the AVI cannot tell if a model is better than chance. The CVI does this:
If we call AVI0 the proportion of all cells that have a HS>50, then the CVI = AVI – AVI0. A model predicting presence everywhere would thus get a CVI of 0. Obviously, the CVI is always lower than the AVI.
Both indices are useful. A good model should have high values for both indices (say, AVI>0.75 and CVI>0.3). If your model has a low AVI (<0.5), it shouldn’t be trusted. However, if it gets a good AVI and a poor CVI, it doesn’t mean that your model is bad, just that is it not very different from random expectation. This could happen if your study area is closely fitting the focal species ecological requirements. Indeed, the larger the study area (or the more specialised the focal species), the higher the CVI.
Remember that all presence-only evaluation (and probably presence/absence indices as well) indices are always somewhat sensitive to the study area.

References

Hirzel, A.H., Arlettaz, R., 2003. Modelling habitat suitability for complex species distributions by the environmental-distance geometric mean. Environmental Management 32, 614-623.

Reutter, B.A., Helfer, V., Hirzel, A.H., Vogel, P., 2003. Modelling habitat-suitability using museum collections: an example with three sympatric Apodemus species from the Alps. J Biogeography 30, 581-590.

## Table of Contents

## EVALUATION

The predictive accuracy of the Habitat Suitability (HS) map can be evaluated by a special cross-validation procedure: theBoyce index.## The Boyce index

Evaluation is done by a cross-validation process (Menu Evaluation/Cross-validation). It computes a confidence interval about the predictive accuracy of the HS model. The species locations are randomly partitioned into k mutually exclusive sets. k-1 partitions will be used to compute a HS model and the left-out partition will be used to validate it on independent data. This process is repeated k times, each time by leaving out a different partition. This process results in k more-or-less different HS maps. By comparing these maps and how they fluctuate, one can assess their predictive power.By default, the partitions are set so that they do not overlap geographically. This makes the cross-validation more robust to spatial auto-correlation (e.g. conspecific attraction). This is actually controlled by the Randomness field. 0% means that partitioning is determined only by geography. A value of 100% indicates that assigning a point to a partition is determined half by its geographical position and half by random. 1000% indicates that randomness is 10 times more important than geography. And so on…

Note that each time you apply the cross-validation process, you will get the same partitioning, and therefore the same results. If you want another partitioning, change the Random seed value. Each value generate a different set of partitions.

BioMapper follows the method described by Boyce et al. (2002) and further developed in Hirzel et al. (2006). Each map is reclassified in b bins (by default,

b=4). Each bin i covers some proportion of the map’s total area (Ei) and contains some proportion of the validation points (Pi) (validation points are those observation left out during the cross-validation process). One computes then the predicted to expected ratioP/Efor each bin asFi=Pi/Ei. If the HS map is completely random, one expectsFi=1 for all the bins. If the model is good, low HS should have a low F (below 1) and high HS a highF(above 1) with a monotonic increase in between. A way to measure the monotonicity of the curve is to compute a Spearman rank correlation on the Fi;, which we call the Boyce indexB.The main shortcoming of the Boyce index is its sensitivity to the number of bins b and to their boundaries. To fix this problem, we derived a new evaluator based on a “moving window” of width W (good values for

Wrange between 10 and 20) instead of fixed classes. Computation starts with a first class covering the suitability range [0,W[ whoseP/Eratio is plotted against the average suitability value of the class,W/2. Then, the moving window is shifted from a small amount upwards and theP/Eis plotted again. This operation is repeated until the moving window reaches the last possible range, [100-W, 100]. This provides a smoothP/Ecurve, on which a “continuous Boyce index”Bcont(W) is computed by a Spearman rank correlation coefficient as above.Practically, one often get a sigmoid curve, the F increasing exponentially and then stabilising and oscillating around a maximum value. Looking at the shape of this curve allows one to define where is the threshold between suitable and unsuitable habitat, from which point the model doesn’t add significant information, etc.. Also, the variance of the

Fiamong the cross-validated curves reflects the prediction power of the model.## References

## Operations

k-fold cross-validation”. Here you can define the CV parameters.k). You can also choose to keep the temporary HS maps for further analysis or compute confidence limits maps.The cross-validation may take some time as k HS-maps must be computed. Once the process is completed, a new dialog box appears, entitled “area-adjusted frequency cross-validation”. The upper graph shows the Fi curves along with the Boyce indices. The panel just below the graphs allows the user to select various display options. The lower graph displays the bins. By default, there are four of them and they have an equal width. The histograms represent the average number of cross-validation points in each bin (in green) and the average area (or number of map cells) covered by each bin (in red). You can change the number of bins with the spin editor below the graph. You can also modify the sizes of the bins, either manually by drawing the histogram bar borders with the mouse, or by selecting one of the bin-partitioning algorithm in the lower panel. You can choose between equal width (all bins cover the same HS range), equal counts (BioMapper tries to set bin limits so as they all have about the same number of cross-validation points) or equal area (BioMapper tries to set bin limits so as they all cover about the same area). The equal counts is somewhat similar to the operation described in the paper by Boyce et al. and seems to give the best results. There is still some work going on this part too.

If you want to compare several models later, you may want to store the cross-validation data into a file. You can do so with the menu Cross-validation/Save CV data…. These data can be opened again later by the menu Evaluation/Load CV data…, which brings back the cross-validation dialog box. The CV-data file is in text format and can be opened in another program if you want to make your own analysis of them. You may also want to save the continuous P/E curves (Cross-validation/Save continuous

P/Ecurves…) for display in another application.Once you have chosen the relevant bin limits, you can reclassify the HS map to present only these new categories by clicking on the Reclassify HS map button.

You can also save the graphs (Cross-validation/Save graphs…) in WMF or EMF format, which you can use for instance in Word or Powerpoint.

Finally, by leaving the cross-validation box, you will find a summary of the cross-validation statistics, along with the bin limits and old validation indices (AVI and CVI), in BioMapper’s main result window. Remember that all presence-only evaluation indices (and probably presence/absence indices as well) are always somewhat sensitive to the study area.

## Old evalutation indices

The Absolute Validation Index (AVI) and the Contrast Validation Index (CVI) were developed before Boyce indices and cross-validation processes. They are still provided in Biomapper for backward compatibility as a side results of the cross-validation procedure, the Boyce continuous index should be preferred.

The AVI is the proportion of validation cells that have a HS>50. It evaluates how good a job the model is doing (i.e. predicting high HS for cells where the species is present). However, an AVI of 1 (best value) could be obtained by a model predicting presence everywhere. In other words, the AVI cannot tell if a model is better than chance. The CVI does this:

If we call AVI0 the proportion of all cells that have a HS>50, then the CVI = AVI – AVI0. A model predicting presence everywhere would thus get a CVI of 0. Obviously, the CVI is always lower than the AVI.

Both indices are useful. A good model should have high values for both indices (say, AVI>0.75 and CVI>0.3). If your model has a low AVI (<0.5), it shouldn’t be trusted. However, if it gets a good AVI and a poor CVI, it doesn’t mean that your model is bad, just that is it not very different from random expectation. This could happen if your study area is closely fitting the focal species ecological requirements. Indeed, the larger the study area (or the more specialised the focal species), the higher the CVI.

Remember that all presence-only evaluation (and probably presence/absence indices as well) indices are always somewhat sensitive to the study area.

## References

## Links