Breadcrumb

6 Receptor impact modelling workshop (stage 5)

Given the hydrological scenarios defined in the previous stage, the predicted response for the receptor impact variables is elicited from the external experts at the receptor impact modelling workshop. The expert education and preparation process leading up to and including the receptor impact modelling workshop is detailed in Section 6.1. The elicitation procedure used at the workshop to elicit expert judgement conditional on the hydrological scenarios, which were developed in the previous stage, is detailed in Section 6.2. Incorporating additional or alternative experts would likely lead to alternative receptor impact models. The quality of a constructed receptor impact model can only be assessed by evaluating the expert assessment against empirical data. The statistical nature of the receptor impact models (Section 5.1) and scenario construction (Section 5.2) explicitly enables this empirical evaluation of expert contributions by allowing incorporation of independent empirical data through the likelihood function within a generalised linear model. This section discusses the expert preparation, elicitation structure and probability assessment that ensure the potential coherent assimilation of the contributed expert assessments with empirical data, while accommodating for expert uncertainty in the relationship between hydrological response variables and the receptor impact variable.

6.1 Expert preparation

The protocol for eliciting receptor impact models is structured around each landscape class defined within each bioregion or subregion. The fundamental question ‘How might selected receptor impact variables change under various scenarios of change for the hydrological response variables?’ is addressed by the experts. A very important step in the process therefore is the formalisation and specification of the relevant impact and response variables and the relationships among these variables (Chapter 4).

The protocol methodology asks experts to provide estimates of the chosen receptor impact variable at a range of values of the relevant hydrological response variables. It then uses statistical modelling to build a relationship between the hydrological response variables and the receptor impact variables, where the former are treated as covariates and the latter as the response of a linear model with potentially non-linear basis functions (sensu Hosack et al., 2016). This approach provides a repeatable protocol to develop these models via experts.

The elicitation session is designed to tackle a challenging problem, and it depends on the collaboration and cooperation of the experts. For this session, experts are asked to contribute their knowledge and expertise in a small group setting, although sometimes individually when only a single expert is available. The group format permits experts to confer and seek a consensus opinion when responding to presented scenarios. A group approach not only allows for the entire group to contribute but also permits the opportunity for feedback and group learning while responding to hypothetical scenarios.

The elicitations take place at receptor impact modelling workshops, which are separately held for each bioregion and subregion. The attending experts represent a wide range of expertise and experience. At the workshop, parallel sessions are run with experts grouped by expertise. The groups may range in size from a single individual up to a small number of individuals. The expert preparation for these group elicitation sessions is as follows.

The following steps were involved prior to the receptor impact modelling workshop:

  1. Experts are selected for the elicitation process. The choice of experts and their invitation into the process ensures that appropriate expertise is identified and included in the process. Relevant experts are contacted by staff collaborating among the Office of Water, the Bureau of Meteorology and the BA ecology discipline teams for each bioregion or subregion. Ideally, experts will have previously attended the relevant qualitative modelling workshop for each bioregion or subregion. This allows experts to take advantage of their previous exposure to and discussion about the bioregion or subregion, the relevant landscape classes and the receptors and hydrological response variables identified in the qualitative models. This continuity helps experts smoothly transition into a quantitative assessment for one or more receptor impact variables, while understanding the context given to a particular receptor impact variable by its relationship with respect to the landscape class and ecosystem. As noted above, the qualitative models help experts identify both potential direct and indirect hydrological impacts. Also, experts that have attended the qualitative modelling workshop will have had previous exposure to the scope and objectives of the Bioregional Assessment Programme, which leads to further gains in efficiency.
  2. Prior to the first meeting the experts receive material outlining the Programme, the nature and purpose of the receptor impact models and a description of the landscape classes, hydrological response variables and potential receptor impact variables. It also describes how the experts’ responses are to be used to develop the receptor impact model.
  3. The quality of the group elicitation depends on a collegial, open-minded and focused atmosphere. The experts are provided information on the group elicitation approach. In particular, the following guidelines are provided:
    1. Respect alternative opinions. Everyone has different points of view and experience that will be called upon over the course of the session.
    2. Practice patience. The problem is new and challenging. It will take some thought and work to assess and talk about, which can take time. Occasionally, points will need to be considered again (and even again).
    3. Ask questions of the session facilitators. Assessing expert opinion is a communication exercise, and please ask if you are unsure about what is being asked of you. We much prefer to address any confusion as it arises.
    4. Ask questions of your fellow experts. Do not assume that ‘you know what they mean’, even if you have worked together closely in the past. Everyone has different ways for expressing their views, and clarifying questions can help resolve where points of view are both different and similar.
    5. Recognise the minority opinion. Please speak up! The group elicitation approach depends on everyone participating. If you are uncertain, that is ok, this is accommodated by the elicitation approach that will be used in this session.

At the receptor impact modelling workshop, the following steps are completed before beginning the elicitation sessions:

  1. The experts are convened and given a brief introduction about BAs, landscape class definitions and hydrological response variable definitions. They are then encouraged to ask questions about the process.
  2. Before undergoing an elicitation session, experts are first educated about subjective probability, trained on heuristics and biases commonly encountered in expert elicitation exercises (O’Hagan et al., 2006).
  3. The elicitation procedure is presented (Section 6.2) to the experts. Practice examples are worked through until experts are comfortable with the method before beginning the elicitation.
  4. The experts are reminded of the group elicitation approach and its accompanying challenges and guidelines for expected group behaviour, as documented above. The experts are then split into their parallel sessions by expertise.
  5. Within each parallel session, the facilitator uses their experience of the process to highlight challenges experts may have and strategies to overcome them. The choice of receptor impact variables is discussed to identify any fundamental conceptual modelling modifications from the preceding qualitative modelling workshop.

Throughout each parallel elicitation session, a BA contact is made available to respond to all queries from the experts about questions of hydrology, ecology or the BA process.

6.2 Conditional elicitation step

6.2.1 Structure of elicitation

The target of the elicitation is the unknown value of the receptor impact variable, y. The receptor impact variable y is defined such that it has a direct interpretation relative to potential observables and expert knowledge (Chapter 4). For example, in a Bernoulli response model considered at the ith hydrological scenario, a (hypothetical) observation z subscript i end subscript, corresponds to either a presence or absence, and y subscript i end subscript equals expected value of z subscript i end subscript is interpreted as the probability of presence. In a Poisson response model where z subscript i end subscript corresponds to a count, y subscript i end subscript is interpreted as an intensity that may relate to the annual average abundance over a defined spatial scope. The experts provide subjective probability distributions describing the receptor impact variable estimates conditional on a hydrological scenario summarised within the design point x subscript i end subscript superscript T(transpose) end superscript (see Section 5.2.2). The elicited subjective probability distributions are assumed independent conditional on the hydrological scenarios.

6.2.2 Elicitation of subjective conditional probabilities

Conditional on the design point x subscript i end subscript, the goal is to elicit a normal distribution for eta subscript i end subscript vertical line x subscript i end subscript with the mean and variance parameters summarised into the parameter vector phi subscript i end subscript equals open square bracket m subscript i end subscript comma v subscript i end subscript close square bracket to the power of T(transpose) end exponent. An elicitation of fractiles (equivalently, percentiles or quantiles) for the target y subscript i end subscript equals g to the power of negative 1 end exponent open parenthesis eta subscript i end subscript close parenthesis, given the monotonic link function, directly translates into fractiles for the conditional normal distribution of the linear predictor, g open parenthesis y subscript i end subscript vertical line eta subscript i end subscript close parenthesis equals eta subscript i end subscript vertical line x subscript i end subscript tilde N open parenthesis m subscript i end subscript comma v subscript i end subscript close parenthesis. The experts are asked for each design point to perform judgements of equal odds in three steps:

  1. What value do you believe gives a fifty per cent chance that the true receptor impact variable is lower? (This obtains a prediction of the median, f subscript 1 divided by 2 end subscript)
  2. Assume that the true value is really below f subscript 1 divided by 2 end subscript. Given this information, what value do you believe gives a fifty per cent chance of being above or below the true value of the receptor impact variable? (This is the first quartile, f subscript 1 divided by 4 end subscript)
  3. Assume that the true value is really above f subscript 1 divided by 2 end subscript. Given this information, what value do you believe gives a fifty per cent chance of being above or below the true value of the receptor impact variable? (This is the third quartile, f subscript 3 divided by 4 end subscript).

The parameters phi subscript i end subscript are then chosen to minimise the information lost by approximating the elicited fractiles by a normal distribution. The elicited fractiles are assembled into the vector, f equals open square bracket f subscript 0 divided by 4 end subscript comma dot dot dot comma f subscript j divided by 4 end subscript comma dot dot dot comma f subscript 4 divided by 4 end subscript close square bracket to the power of T(transpose) end exponent where j equals 0 comma dot dot dot comma 4. The extreme fractiles f subscript 0 divided by 4 end subscript and f subscript 4 divided by 4 end subscript (possibly infinite) are determined by the support of y subscript i end subscript. Let the probability intervals determined by these elicited fractiles be denoted by l subscript k end subscript equals f subscript k divided by 4 end subscript minus f subscript open parenthesis k minus 1 close parenthesis divided by 4 end subscript equals 0.25 for k equals 1 comma horizontal dot dot dot comma 4. Let P subscript e vertical line i end subscript denote the histogram constructed by these elicited fractiles.

Fitted probability intervals for the approximating normal distribution are obtained by:

z subscript k end subscript equals integral subscript f subscript open parenthesis k minus 1 close parenthesis divided by 4 end subscript end subscript superscript f subscript k divided by 4 end subscript end superscript d P subscript s end subscript open parenthesis eta subscript i end subscript vertical line phi subscript i end subscript close parenthesis equals P subscript s end subscript open parenthesis f subscript k divided by 4 end subscript vertical line phi subscript I end subscript close parenthesis minus P subscript s end subscript open parenthesis f subscript open parenthesis k minus 1 close parenthesis divided by 4 end subscript vertical line phi subscript i end subscript close parenthesis comma k equals 1 comma dot dot dot comma 4

(31)

where P subscript s end subscript open parenthesis eta subscript i end subscript vertical line phi subscript i end subscript close parenthesis is the subjective probability distribution function parametrised by phi subscript i end subscript.

The Kullback-Leibler divergence of P subscript s vertical line i end subscript from the elicited P subscript e vertical line i end subscript, where subscripts denote dependence on phi subscript i end subscript and x subscript i end subscript, is approximated by:

Kullback-Leibler divergence from P subscript s vertical line i end subscript to P subscript e vertical line i end subscript equals integral log fraction numerator dP subscript e vertical line i end subscript over denominator dP subscript straight s vertical line i end subscript dP subscript e vertical line i end subscript is almost equal to sum for k of log open parenthesis fraction numerator l subscript k end subscript over denominator Z subscript k end subscript close parenthesis l subscript k end subscript comma

(32)

where P subscript e vertical line i end subscript is absolutely continuous with P subscript s vertical line i end subscript. The parameters phi subscript i end subscript are chosen so as to minimise the approximate Kullback-Leibler divergence of P subscript s vertical line i end subscript from P subscript e vertical line i end subscript:

phi with hat on top subscript i end subscript equals arg space min with phi subscript i end superscript below sum for k of log open parenthesis fraction numerator l subscript k end subscript over denominator Z subscript k end subscript end fraction close parenthesis l subscript k end subscript

(33)

For each scenario, the values of the design point are portrayed and the group discusses the potential ecological response. The goal is to develop a probability distribution that is an acceptable representation of the experts’ beliefs. Of course, with unlimited time resources a suite of distribution families could be presented for consideration by experts. Resources in BA are not unlimited, however, and for consistency and efficiency the parametric distribution considered by the experts is defined by the models specified and described in Section 5.1. The following steps are then performed by the experts:

  1. Initial fractile assessments are elicited using the quartile method. These are plotted graphically as vertical dashed lines. Note that more fractiles than free parameters are elicited in an approach that is referred to as ‘overfitting’ (O’Hagan et al., 2006), which will permit feedback between the model representation and the group’s final probabilistic statement as detailed in the following steps.
  2. The corresponding fitted fractiles and density curve from the fitted probability density function p subscript s vertical line i end subscript open parenthesis y subscript i end subscript vertical line phi with hat on top subscript i end subscript close parenthesis are plotted as three vertical blue dashed lines and blue curve, respectively. The overfitting approach uses the parametric model to average across multiple probability statements. The parametric model is unlikely to exactly match the elicited fractiles from Step 1, and so this process encourages the group to evaluate the parametric model with respect to their beliefs. If the values for the matching fractiles are not acceptable, then the group returns to Step 1 and adjusts the elicited fractiles. These two steps are repeated as often as necessary until the group accepts the parametric model quantiles as acceptable.
  3. The extreme deciles from p subscript s vertical line i end subscript open parenthesis y subscript i end subscript vertical line phi with hat on top subscript i end subscript close parenthesis are plotted as dashed blue lines. The group considers these new predictions and returns to Step 1 if the predictions are unacceptable.
  4. The group is allowed to consider other fractiles or cumulative probabilities as predictions from the parametric model. The group returns to Step 1 if these predictions are unacceptable.
  5. After completing the above feedback steps, the group is allowed to accept the elicited subjective probability distribution p subscript s vertical line i end subscript open parenthesis y subscript i end subscript vertical line phi with hat on top subscript i end subscript close parenthesis as a reasonable assessment of the expert opinion.

This process thus elicits from the group a subjective probability distribution of ecological response given the covariate values that make up the defined scenario. Note that the elicitation focuses on the distribution, not the raw fractile assessments (e.g. Step 1 above). The raw fractiles are used by the experts as ‘parameters’ to iteratively build a probability distribution that is an acceptable representation of their beliefs. The probability predictions made by the elicited probability distribution provide the final products that are assessed by experts for either acceptance or rejection and further iteration until the experts accept the probability distribution as a reasonable model of their beliefs. This process is repeated for each design point.

Last updated:
18 October 2018