- Home
- Assessments
- Bioregional Assessment Program
- Methods
- Impacts and risks
- 6 Building on the impact and risk analysis
6.1 Overview
Bioregional assessments (BAs) seek to help governments, industry and the community make better-informed regulatory, water management and planning decisions.
A BA is an analysis at a particular point in time. Those components that are most likely to change are the human parts, particularly decisions around the coal resource development and around the list of community assets.
The coal resource development pathway (CRDP) is verified during the BA as the most likely future at the time of analysis, even though it may ultimately be implemented in different ways, such as changes to the timing and scale of some proposed developments. Additionally, particular coal resource developments may become more or less likely in response to a range of external economic, social or political factors. Despite the potential for the CRDP to change with time, it still provides a valuable indicative scenario as the basis for highlighting potential regional-scale changes to water resources and water-dependent assets. These may need to be considered further as part of local-scale assessments by proponents, or through future regulatory approval processes and government decision making at both national and state levels for particular coal resource developments. Equally as important, the impact and risk analysis indicates where impacts to water resources and water-dependent assets are unlikely to occur, which may help in ensuring that both regulators and proponents concentrate their focus on those aspects and areas that have greater potential to change.
The water-dependent assets are identified as features of ecological, economic or sociocultural value by the community and supplemented by key Commonwealth and state databases. The assets identified may change over time as values change and additional assets are included and others lessen in importance.
While the CRDP and asset register are date-stamped, BAs have been conceived and implemented in a modular fashion. This means that future updates or iterations to a BA do not have to revisit each component of work to the same level and intensity as done during the implementation. For example, if the CRDP were to change, adjustments to some components of work may be needed (e.g. incorporating new coal resource developments in the groundwater model) but may not affect many other components (e.g. the landscape classification). BAs have certainly been undertaken with the clear intention of updating the assessment at some future stage. While there is effort and expertise required in any update, the modular nature of the assessment means that effort is greatly reduced by the way a BA has been implemented.
It will be essential to identify gaps or opportunities to improve those components in the future. Given prediction is at the heart of the impact and risk analysis, focusing on those components that may reduce the predictive uncertainty should take priority. For example, that might include new data requirements to better characterise the hydraulic properties of important geological layers (e.g. coal-bearing units and water supply aquifers) and tighten the plausible range of associated parameters used in the groundwater models. It could include improved spatial resolution of groundwater-dependent ecosystem (GDE) mapping to reduce the spatial uncertainty and tighten the link between hydrological modelling and the ecosystem modelling; or it might involve additional expert elicitations to focus on ecosystem indicators that tie more directly to specific decision making and reduce some of the management uncertainty.
One of the key challenges for a BA is scale. BAs focus on regional cumulative analyses. These reflect the broad-scale hydrological and ecosystem changes related to impacts that may accumulate from multiple sites and types of coal resource development. Where changes are predicted, and particularly close to the mine or coal seam gas (CSG) operations, the Assessment team is confident in asserting that hydrological changes may occur, but less confident in the precise magnitude or extent of propagation of those changes from depth to the surface because of the dependence on local processes and operations. BAs are not a substitute for careful assessment of proposed coal mine or CSG extraction projects under Australian or state environmental law. Such assessments may use finer-scale surface water and groundwater models and consider impacts on matters other than water resources. However, the results from a BA should help inform the advice on proposed coal resource development projects from the Independent Expert Scientific Committee on Coal Seam Gas and Large Coal Mining Development (IESC), a federal government statutory authority established in 2012 under the Commonwealth’s Environment Protection and Biodiversity Conservation Act 1999, and state government regulators.
There is also a limited ability to isolate the impact of individual developments from the regional cumulative analyses. The baseline and CRDP may each consider a suite of developments, the potential impacts of which may overlap to varying degrees in both time and space. This allows an assessment to predict and understand the cumulative hydrological changes and potential impacts of those developments on surface water, groundwater and water-dependent assets. However, it does not, in general, allow the attribution of these effects to individual developments. In some cases the spatial or temporal alignment of certain coal resource developments may allow for some attribution, but that is the exception rather than the norm. To accurately isolate the contribution of any particular coal resource development would require the comparison of two futures – one with that coal resource development and one without it. The hydrological models are available as part of the Programme focus on transparency so it is possible with sufficient expertise to make an adjustment (e.g. remove a coal mine) to these models and re-run the analysis.
A BA provides important context to identify potential issues that may need to be addressed in local-scale environmental impact assessments of new coal resource developments. It should help project proponents to meet legislative requirements to describe the environmental values that may be affected by the exercise of underground water rights, and to adopt strategies to avoid, mitigate or manage the predicted impacts. These assessments do not investigate the broader social, economic or human health impacts of coal resource development, nor do they consider risks of fugitive gases and non-water-related impacts.
In comparing results under two different futures, factors such as climate change or land use are held constant through an assessment. Future assessments could look to include these and other stressors to more fully predict cumulative impacts on a landscape scale. Within any bioregion or subregion there will may be interest in building on the BA for particular assets or areas of interest. In such cases the BA outputs could feed into additional and focused impact and risk assessments surrounding that asset or area. For instance, this might occur by using the individual asset profile for the asset of interest, examining the range of hydrological changes that asset may experience and bringing additional knowledge about changes or thresholds that may be important to protecting values that derive from that asset.
A number of design choices have been made for the impact and risk analysis to achieve this objective while addressing the constraints imposed by the BA methodology (Barrett et al., 2013), complexity of the task and good practice in risk assessment. Ultimately these design choices need to be scrutinised, and particularly when considering if the BA outputs are seen to meet the needs of decision makers and the scientific quality criteria.
6.2 Data and information
The impact and risk analysis, and the companion products that underpin it, will produce a vast amount of data and analysis output. Only some of this is able to be summarised in product 3-4 (impact and risk analysis). The full suite of information, including information for individual assets, will be provided on www.bioregionalassessments.gov.au. A subset of that will be displayed on the BA Explorer interactive web mapping tool on www.bioregionalassessments.gov.au/explorer. For example, for the Gloucester subregion, users can explore detailed results for:
- the entire subregion at www.bioregionalassessments.gov.au/explorer/GLO
- hydrological changes at www.bioregionalassessments.gov.au/explorer/GLO/hydrologicalchanges
- impacts on landscape classes at www.bioregionalassessments.gov.au/explorer/GLO/landscapes
- impacts on assets at www.bioregionalassessments.gov.au/explorer/GLO/assets.
Much more information is provided as datasets at data.gov.au.
These underpinning datasets, including shapefiles of geographic data and modelling results, can assist decision makers at all levels to review the work undertaken to date, and to extend or update a BA if new models or data become available. This access also allows others to use those same data layers as part of tailored risk assessments about individual assets or areas of concern within the bioregion or subregion. In doing so, people are able to choose thresholds of impact that may threaten the specific values they are trying to protect and calculate the corresponding likelihood of occurrence.
The Bioregional Assessment Programme has adopted an extensive and rigorous approach to the management and publication of data. This is part of a commitment to making sure there is a clear understanding of the scientific process and that the data used and created along the way are accessible to the community. This approach is consistent with the Australian Government's principles of providing publicly accessible, transparent and responsibly managed public sector information.
6.3 Monitoring
6.3.1 Objectives and motivation
Most risk assessment frameworks identify the need for, and emphasise the importance of, post-assessment monitoring designed to test and (in)validate the predictions of the risk assessment (Hayes, 1997). Post-assessment monitoring is essential to complete the scientific method loop: hypothesis, prediction and observation. In the context of a risk assessment, and BAs more particularly, the hypothesis step is embodied within the conceptual modelling stages of the assessment, and the prediction step is embodied within the outputs of the surface water, groundwater and receptor impact models. A post-assessment monitoring strategy embodies the observation step. Without it the risk assessment is incomplete because it does not close the scientific loop of hypothesis, prediction and observation.
All monitoring programmes should begin with clear operational objectives, both for scientific and practical reasons. In this context, the objectives of the programme are to test and (invalidate) all of the risk assessment predictions. This includes surface water, groundwater and receptor impact model predictions. Monitoring may also be able to confirm or ‘rule out’ the existence of particular causal pathways and influence mitigation options.
The objectives of a post-risk assessment monitoring programme must speak directly to the predictions of the risk assessment and the management objectives that motivated the risk assessment. They should also provide additional details about what the monitoring programme and sampling protocol will do, and identify boundaries or limits of the monitoring programme by specifying particular areas, species or measures. An effective set of monitoring objectives should meet the test of being realistic, specific and measurable. The US National Park Service (2012) suggests the use of the following checklist of questions to determine if monitoring objectives meet the test:
- Are each of the monitoring objectives measurable?
- Are they achievable?
- Is the location and spatial bounds of the monitoring specified?
- Is the species or asset being monitored specified?
- Will the reader be able to anticipate and understand what the data will look like?
Alternative lines of evidence (e.g. from existing risk assessments and local analyses) may also complement the monitoring in validating (or invalidating) the predicted risk outcomes.
6.3.2 Design and implementation
Figure 21 provides a basic flow chart that illustrates the steps in designing and implementing a monitoring programme, beginning with a clear specification of the objectives, and ending with an analysis of the data that, in this context, is used to compare risk predictions with actual outcomes.
GW = groundwater; RIM = receptor impact model; SW = surface water
6.3.3 Existing monitoring programmes
Once the Programme objectives have been clearly enunciated, the next step in the process is to collate information on any existing monitoring programmes. There are a number of ways to go about compiling information on existing monitoring programmes. In some cases there may be existing reviews of monitoring programmes that could provide a sound basis to start this work. Another approach is to search metadata within institutional, or ideally national, data centres, such as the Australian Ocean Data Network (AODN, n.d.), Atlas of living Australia (ALA, n.d.) and the Terrestrial Ecosystem Research Network (TERN, 2009). Searches can be performed using keywords and/or by providing a bounding box around the assessment area to retrieve all records that intersect with this box.
Completing this step requires very little time and expertise if relevant metadata records are provided to central data repositories. Considerably more time and effort will be required to complete this work if there are no existing reviews of monitoring programmes and if existing monitoring programs do not publish metadata records for their monitoring data. If this is the case, the discovery, summary and analysis in these circumstances will need to rely on internet searches, supported by the experience, tenacity and networking skills of the analyst concerned.
6.3.4 Sample design considerations
Developing the overall sampling design for a post-risk assessment monitoring programme comprises two inter-related functions: (i) selecting variables to monitor and (ii) developing sampling design. A third integration step in this stage is necessary if there are existing monitoring programmes already in place. This third step must include an evaluation of the existing programmes and, if necessary, selection and integration of existing programmes outputs in the post-assessment programme.
In this context the monitoring variables are specified in advance by the previous steps in the risk assessment, specifically:
- the hydrological response variables used in the receptor impact models – targeting the associated predictions from the surface water and groundwater models
- the ecological receptor impact variables chosen through the qualitative mathematical modelling steps – targeting the associated predictions from the receptor impact models.
It is important that the monitoring programme seeks to measure hydrological response variables and receptor impact variables. In the event that the post-assessment observations do not agree with the risk assessment predictions it is important to distinguish between the situation where the hydrological response variables do not behave as predicted (indicative of surface water or groundwater modelling errors) and the situation where the receptor impact variables do not behave as predicted (indicative of incomplete system understanding and/or errors in the receptor impact models).
The sampling design for a post-assessment monitoring programme is shaped by a variety of factors including the monitoring variables; the existing monitoring legacy; advice from experts in sampling design and the constraints of budgets, resources and logistics.
The adequacy of sampling design for the selected monitoring programmes (i.e. existing, refined or proposed monitoring programmes) should be assessed before they are incorporated into the post-assessment programme. Opportunities to integrate sampling designs across monitoring programmes (e.g. co-location of sample sites for pressure and value monitoring, or complementary site selection of monitoring sites for the same type of monitoring to generate better insights from the collective monitoring effort) can also be considered at this point; this can produce benefits in both cost savings and data analysis.
The sampling design phase of a monitoring programme must address three critical questions: (i) what is an appropriate level of statistical power to inform decisions in a timely manner, (ii) how are sample sites to be selected, and (iii) how often should measurements be taken at these sites or subsets of sites? These three questions address the fundamental issues of where, and how often, samples should be collected.
Informally, statistical power is the probability of making the right decision when it matters most. Environmental managers face two options when presented with data from a monitoring programme: act upon the information or do nothing; this entails the possibility of two types of errors. The first (Type I error, with probability ) occurs if the manager acts in the belief that a significant trend or change is occurring, when in fact no such change is or has occurred. The second error (Type II error, with probability ) occurs when the manager fails to act in the erroneous belief that no significant change is occurring when in fact a change has or is occurring.
The question of appropriate statistical power has been traditionally approached using the ‘5-80’ convention, which fixes the Type I error rate to be 5% and seeks a sample size such that statistical power (1- ) is 80% (i.e. the Type II error rate is 20%). This approach, however, places the burden of proof disproportionately on those trying to demonstrate environmental change, and undermines the fundamental aim of many monitoring programmes, which is to ensure that real change is detected and acted upon as early as possible (Field et al., 2007).
Mapstone (1995) recommends that the relative weighting of the two error rates are set according to the costs associated with each, and in the absence of this information the two error rates should simply be set equal to each other. This is a sensible proposition. Importantly the selection, and desired ratio, of the two error rates provides a means to tailor the monitoring design to the priorities of management objectives, for example, selecting lower Type II error rates for higher priority objectives, and vice-versa.
There are two important challenges that must be met in order to answer the second critical question in the context of BAs:
- BAs take a regional, whole-of-system, perspective, which implies inference must be made at greater spatial scales and higher levels of ecological organisation (i.e. regional populations and communities), than that typically associated with impact assessments for individual coal resource developments.
- Large-scale monitoring programmes must try to integrate the existing monitoring legacy with any new initiatives in order to be cost efficient and generate the long time series of observations that are typically necessary to detect changes in ecological systems.
Stevens (1994) identifies two distinct approaches when deciding where to locate sample sites for the purposes of regional-scale evaluation of environmental status or trends. The first approach is judgmental sampling wherein sites are selected by their anticipated ability to reflect regional characteristics. The second approach is probability sampling characterised by three distinguishing features: (i) the population being sampled is explicitly described, (ii) every element of the population has some opportunity of actually being sampled, and (iii) the sample selection procedure includes an explicit random element.
Judgmental sampling has been applied for many decades to environmental and social problems, and has demonstrably failed on many occasions (Edwards, 1998). Although recent modelling approaches have been developed to help account for this complication, this requires additional effort and modelling assumptions. It is strongly recommended that this approach is avoided in a post-assessment monitoring programme. It is also important that existing monitoring programmes are evaluated to identify the basis for site selection and transparently clarify any assumptions of existing monitoring programmes based on judgmental sampling.
Examples of probability-based approaches to survey design include systematic sampling, simple random sampling, two-stage sampling, stratified random sampling (Gilbert, 1987), spatially balanced Quasi-Monte Carlo sampling and Generalised Random Tessellation Stratified (GRTS) sampling (Stevens and Olsen, 2003).
Monitoring programmes designed to meet the needs of a strategic assessment will typically seek to identify trends and change points in regional (rather than local) populations. This type of monitoring objective implies that sites will be re-surveyed with a specified periodicity that depends on the defined management need. In this context it is important to recognise that the ability to detect trends in regional populations is influenced by variability in populations, space, time and the way data are collected (Larsen et al., 1995; Urquhart et al., 1998).
The main sources of uncertainty that will be encountered in this context, and that will affect the ability of a monitoring programme to detect trends are:
- population variance – differences in observations across the members of a regional population or sub-populations (such as a receptor impact variable in a landscape class across the northern half of a large bioregion)
- temporal variance – the amount by which observation across all members of a population or sub-population are high or low in a particular time period (e.g. a year). Over time, the value of any observation will fluctuate around a trend, or in the absence of a trend, around a central value. This variance component measures the amount by which all members of the population are above or below a long-term trend line or curve, or central value. Larsen et al. (1995) call this a ‘year effect’
- space-time interaction (random) effects – the amount by which observations taken on an individual member of a population (e.g. at a single forest) fluctuate over time around a trend line, trend curve or central value. These fluctuations are caused by localised factors that operate at small scales, such as individual forest, or a localised group of forests
- index variation – a composite of several sources of variation, some natural and some introduced by the differences in the way data are collected. It includes sources such as differences caused by imprecise measuring devices and differences between survey teams. Standard operating procedures outlined in monitoring protocols are typically designed to minimise this source of variance
- spatial temporal dependence – objects near to each other in time and space will exhibit more similar responses than objects that are far apart.
In designing a post-assessment monitoring programme it is important to consider the effect of each of these sources of uncertainty on the analysis and the power of the program to detect trends at local and regional scales.
6.3.5 Monitoring protocols
In its minimalist form a monitoring protocol is a detailed document that provides operational instructions about how data are to be collected. It should provide operational instructions for the entire ‘data life cycle’ including how to collect, manage, analyse and report data in a consistent and comparable fashion over space and time. Monitoring protocols must be sufficiently well documented so that different people, or new programmes, can complete these procedures in exactly the same way.
Monitoring protocols are important for ensuring monitoring data are robust to changes in personnel, technology and management needs. They set minimum standards for issues such as observer training, data collection and storage, and are therefore a key component of quality assurance and quality control for integrated monitoring to support strategic assessment.
Oakley et al. (2003) provide generic guidance on developing monitoring protocols, and recommend that protocols include:
- a narrative that gives background information on why a particular component or process of the ecosystem was selected for monitoring, together with an overview of the various components of the monitoring protocol, including the objectives, the sampling design, field methodology, data analysis, data archival and reporting, personnel requirements, training procedures and operational requirements
- a set of standard operating procedures (SOPs) that provide detailed, step-by-step instructions on how each component of the protocol is to be completed, including instructions for how any of the SOPs are to be amended
- supplementary materials that provide additional guidance and support, and can include items such as reports, photographs and data analysis examples
- a conceptual model without too much detail that can guide monitoring programmes and provide a graphical narrative that can be updated with improved scientific understanding (Lindenmayer and Likens, 2010).
6.3.6 Data management
Australia has an established and developing national data infrastructure with the supporting processes and standards that could be used to meet the needs of regional monitoring programmes to support strategic assessments in terrestrial, coastal and marine regions. This includes national data stores and metadata stores to access data (e.g. Australian Ocean Data Network, Atlas of living Australia and the Terrestrial Ecosystem Research Network) and national standards for data management (e.g. ISO Standard and metadata profiles). Data standards are very important for discovery, storage and accessibility of data, particularly in decentralised systems where differences in vocabularies can create problems for discovery and access to data.
Data management for monitoring programmes all too often receives insufficient attention and support (Caughlan and Oakley, 2001; Lindenmayer and Likens, 2010). The costs of adequate data management systems to support monitoring are typically underestimated and can be expected to be about 20% to 30% of the total monitoring programme budget (Fancy et al., 2009; Lindenmayer and Likens, 2010).
Another important focus for data management is identifying the preferred model for discovering, storing and accessing monitoring data (i.e. the primary asset) generated from the selected programmes. A decentralised model may be attractive if selected monitoring programmes involve numerous institutions. It is also important to identify the existing data management infrastructure, processes and standards, and opportunities to establish the preferred model for data management. Acknowledgement and consideration should also be given to the relationship between data management processes and standards and monitoring protocols. Guidance on data management processes and standards needs to be embedded in monitoring protocols to ensure data are discoverable, stored securely and made accessible.
METHODOLOGY FINALISATION DATE
- 1 Introduction
- 2 Design choices for the impact and risk analysis
- 3 High-level logic and workflow
- 4 Impact and risk predictions for assets and landscape classes
- 5 Reporting and communicating impacts and risks
- 6 Building on the impact and risk analysis
- Appendix A Methods for structuring and processing data for bioregional assessment impact and risk analysis purposes
- References
- Datasets
- Citation
- Acknowledgements
- Contributors to the Technical Programme
- About this submethodology