Eliciting expert opinion from experts – some lessons from ecology, psychology and reliability

22 - 24 September 2009

Expert opinion can provide a valuable resource for biosecurity sciences. In particular biosecurity surveillance relies on several expert estimates of probabilities such as:

• a species will enter and survive in transit (along a particular pathway in a given time period)
• if it does successfully enter an area, that it will establish, and
• if it does establish, where it will spread to.

Typically such probabilities are difficult to quantify, resulting in qualitative risk assessments, which are limited in their scope for further use. Quantitative estimates of these probabilities would not only improve precision of the risk analysis, but would also provide useful input to statistical designs for surveillance, and therefore of resource allocation for surveillance and monitoring. In this paper we summarise some lessons learnt about achieving more robust and repeatable expert elicitation in the fields of ecology, psychology and reliability. This includes some relatively simple tricks such as the wording and ordering of questions about probability. In addition some effort in formulating the goals and methodology of elicitation are important: considering what aspect of the scientific domain of interest that experts will most successfully quantify; and matching this to a statistical model to capture the expert opinions and their uncertainty. Current research is focusing on the challenging problem of calibrating multiple expert opinions so that their opinions can be combined, mathematically or otherwise. We also note that some software has recently proven useful for automating tedious aspects of the elicitation process, whilst providing instantaneous feedback to experts.