How accurate are individuals in judging uncertain quantities? Findings from a novel experimental approach.

Speaker: Marta Soares (University of York)

Title: How accurate are individuals in judging uncertain quantities? Findings from a novel experimental approach.

Soares MO, Jankovic D, Jackson C, Morton A, Claxton K, Bojke L, Sharples L

Abstract:

Judgements about uncertain quantities are required across all fields of knowledge, such as engineering, health, physics, or forecasting. These are particularly important in fields closely linked to decision making as, even in the absence of ‘hard’ evidence, decisions need to take consideration for the consequences of different courses of action and any uncertainty surrounding it. In support of such decisions, expert judgements are often elicited. Elicitation asks individuals to translate personal knowledge and beliefs into an explicit statistical distributions expressing uncertainty. Also, elicitation may involve multiple experts where interaction needs to be managed. An example of controlled interaction processes is the Delphi, which consists of multiple rounds of individual revision after summary feedback on the group’s distribution.
Although elicitation is widely used in crucial decision arenas, little is known about the accuracy of elicited individual estimates and the validity of the iterative feedback and revision to improve these. We here describe a novel experimental approach where the individual’s knowledge is determined by observations from a simulated process allowing for accuracy to be measured. Using such an approach, three experimental set-ups were implemented. The first established that the method of elicitation (i.e. the summaries individuals are asked to provide) affects the descriptions of uncertainty provided, although the magnitude of such an effect is small in relation to the large scale of between-individual variation observed. The second found no evidence of bias or significant misrepresentation of uncertainty when individuals were asked about a quantity that required some analytic reasoning in addition to observation. The final experimental set-up showed that the likelihood of revision in Delphi-type processes depends on the method of elicitation, but it identified no evidence that less accurate individuals are more likely to revise their initial estimates or that the extent of revision is larger. The findings of this last experiment cast doubts on the benefits of the iterative process widely used within Delphi processes.

Seminar recording available soon.