Towards a Definition of Performance-Based Laboratory Methods

Towards a Definition of Performance-Based Laboratory Methods

A position paper developed by the Methods and Data Comparability Board (MDCB)

Jerry Diamond, Tetra Tech, Inc.
Andrew Eaton, Montgomery Watson Laboratories
Clifford Annis, Merck & Company, Inc.
Herb Brass, US Environmental Protection Agency
Larry Keith, Instant Reference Sources, Inc.

August 2001
Technical Report XX-XX

Table of Contents

Executive Summary …………………………………………………………………………………..……............1

Introduction …...............................................................................................................................................2

Why is a prescriptive methods approach still used by many organizations ………………………..……2

Disadvantages of a Prescriptive Methods Approach ………………………………………………………....3

What is a Performance-Based System ………………………………………………………………………..…4

Advantages of using a Performance Based System Approach …………………………………….…....…5

Examples of Programs that Use a Performance-Based System Approach …………………………..…..6

The Role of a Performance-Based System in Compliance and Ambient Monitoring………………..…..7

Requirements for Validating Protocols in a Performance-Based System …………………………….…..8

The Performance-Based System Process …………………………………………………………………..…..9

Performance-Based Systems in the Context of Method-Defined and Field-Measured Parameters…10

Next Steps for the MDCB………………………………………………………………………...........................12

Literature Cited …………………………………………………………………………………………………..….12

Executive Summary

With the evolution of different water quality monitoring and analysis methods among organizations, even within a single agency, there have been significant consequences in terms of our ability to assess the quality of our nation’s waters. In ambient water quality monitoring, many programs have used prescriptive methods without documenting associated method performance or data quality. This has resulted in unknown or poor quality data and uncertainty in the comparability of data collected across programs or organizations (ITFM 1995a). Consequently, monitoring groups outside the agency collecting the data do not know which information can be used with confidence, resulting in limited data sharing across organizations. This is a significant problem because: (a) assessments of aquatic resources on broad geographic scales (basins for example) or from state to state are not easily feasible and (b) opportunities for increased resource efficiency or for minimizing duplication of efforts are missed. Many organizations have recognized the limitations of a prescriptive methods approach and after much deliberation, the Interagency Task Force on Monitoring Water Quality (ITFM) recommended that a performance-based approach be used to address these limitations (ITFM 1995b).

The reliance on prescriptive methods, without appropriate method performance documentation, has also had significant consequences in compliance water quality monitoring. Due to current bureaucratic and administrative constraints, it is time consuming, resource intensive, and cumbersome to modify existing methods or add new improved methods to the Federal Register. The result is that more sensitive, less expensive, faster, or more modern methods, developed either by federal agencies or consensus organizations, have not been easily implemented or encouraged in compliance or ambient monitoring.

Consistent with one of the goals of the Clean Water Action Plan (USEPA 1998), the National Methods and Data Comparability Board (MDCB) under the National Water Quality Monitoring Council (NWQMC) endorses development of a performance based system as one of its top priorities. A performance-based system will provide a mechanism to: (a) enhance data comparability among various monitoring programs and databases to provide data of known quality, and (b) encourage the implementation of better or more cost-effective methods.

A performance-based approach permits the use of any scientifically appropriate method that demonstrates the ability to meet established method performance criteria (e.g., accuracy, sensitivity, bias, precision) and complies with specified data quality needs or requirements. Key aspects of the MDCB endorsement of performance-based system include; a) the need to establish concise measurement quality objectives (MQOs) or data quality objectives (DQOs) for each parameter reported; b) the need for demonstrated methods capable of meeting these MQOs or DQOs or regulatory limits; c) the need for adequate reference materials to assist laboratories in demonstrating the appropriateness of a given method (prescriptive or performance-based); d) the need for laboratories to adequately document method performance, and e) the successful completion of a pilot program to demonstrate the advantages and viability of a performance-based approach. In addition, several implementation issues need to be resolved such as real or perceived liability and confidentiality of patented methods or intellectual property. The MDCB recognizes that several types of performance-based systems have been proposed or used by different programs and that there is a need to develop a unified national approach. The MDCB endorses the need for validated methods and concise, achievable performance criteria. The MDCB also recognizes that there are outstanding issues concerning how such an approach would be implemented in compliance or ambient water quality monitoring. The training requirements to implement a performance-based system, and to reach some level of national comparability, are extensive due to the diversity of water quality monitoring programs and data requirements. The MDCB recognizes that adequate training and education of data generators, auditors, and users is central to the successful implementation of a performance-based system, particularly in compliance monitoring programs. The EPA Office of Water PBMS Implementation Plan, which uses reference methods and stated performance criteria, is recognized as an important step towards practically implementing PBMS. The MDCB and the National Council should act as the focal point for harmonizing existing or proposed performance-based systems so that a sound, unified approach can be achieved and implemented in an intelligent, fair manner on an inter-organizational basis.

Why is a prescriptive methods approach still used by many organizations?

Currently, most state and federal agencies require prescriptive methods in their monitoring or regulatory programs for several reasons, many of which are perhaps more pragmatic than scientifically-based. Major reasons that are cited for using prescriptive methods are:

. • they are generally well documented in terms of their performance characteristics (e.g., precision, bias, etc.), at least under certain known conditions or for certain matrices (often reagent water). Therefore, data can be evaluated with a similar matrix using a prescriptive approach.

. • they have generally been used by many laboratories and organizations and so are familiar to the personnel collecting and interpreting the results of the method

. • the agency requiring the data can have a relatively simple and clearly defined methodology structure and correspondingly, a less intensive and costly quality assurance program (i.e., fewer and simpler laboratory audits or data quality checks).

All of the above reasons have been used by state and federal agencies to defend relatively cost-effective (though narrowly defined) laboratory certification programs and straightforward data quality control programs.

Disadvantages of a Prescriptive Methods Approach

The disadvantages of prescriptive methods are due, in part, to some of the assumptions made concerning true performance characteristics. Prescriptive methods often only define a published detection level. These detection levels do not provide sufficient information on precision of the concentration estimates needed in determining compliance with regulatory requirements. Additionally both in the regulatory arena and in the scientific literature, there are many examples demonstrating that the performance characteristic of a given method varies when applied in the "real-world". Laboratory capabilities to detect and quantify analytes with known precision varies day to day within any given laboratory and more so among laboratories, particularly between research and production laboratories. Method performance in certain matrices (e.g. certain types of wastewater effluent, groundwater, leachates, or even some high dissolved solids drinking waters) may be far different (poorer) that those same method characteristics based on laboratory reagent water or other relatively simple matrices. However, newer methods issued by some organizations require validation in a variety of matrices to demonstrate performance. To compensate for uncertainties with the prescriptive method detection levels, many states have recently introduced into their water quality standards minimum quantitation levels for several chemicals (e.g., metals and some organic contaminants). For biological monitoring, prescriptive methods do not generally attempt to reduce sampling variability, a key element in the performance of collection gear.

Regarding laboratory protocols, unless a laboratory conducts rigorous quality control analyses on the matrix it is analyzing (which is now required in many newer compliance methods and should allow comparability to be assessed more easily), one can not assume that the performance characteristics reported for the method have been achieved. Thus, prescriptive methods, as currently implemented in many programs, could give a potentially false sense of known and acceptable data quality and may encourage less rigorous quality control programs (both within a laboratory and the agency requiring the data) than are actually needed.

Other disadvantages of prescriptive methods are:

. • different agencies or programs have often developed and published different prescriptive methods for the same parameter making it difficult to determine the degree of comparability in data among programs

. • there is less incentive for laboratories or manufacturers to design and evaluate potentially better methods; i.e., methods that are more sensitive, more reliable, cheaper, or faster unless they can be rapidly adopted by those doing the monitoring

. • there is a lack of extensive intra- and interlaboratory comparability between methodologies (both laboratory and field methods)

. • method improvements, even if well documented, are difficult to implement because of regulatory and administrative constraints associated with using a prescriptive method framework

. • actual method performance and associated data quality is often unknown, especially in some of the older established methods.

What is a Performance-Based System?

Previous work by the ITFM (1995b) and several agencies (e.g., EPA, NOAA, USGS, USACE) independently emphasized the need for data quality objectives (DQOs) or measurement quality objectives (MQOs) in performing assessments. Both concepts are central to a performance-based system approach. MQOs are statements that contain specific units of measure such as: percent recovery, percent relative standard deviation, standard deviation of X micrograms/L, or detection level of Y ppb. They should be thoroughly specified to allow specific comparisons of data to an MQO. DQOs are statements that define the confidence required in conclusions drawn from data produced by a project (USEPA 1994a). The USEPA’s DQO process is a 7-step strategic planning approach that is used to define what, how, when, and where data are collected and analyzed to ensure that the type, quantity, and quality of environmental data used in decision making will be appropriate for the intended application (USEPA 1994a). For example, the USEPA’s Office of Ground Water and Drinking Water used the DQO process to help ensure that water quality measurement data and engineering information gathered under their Information Collection Rule (ICR) are adequate to support development of a series of drinking water regulations regarding surface water treatment requirements and disinfectant and disinfection byproduct controls (RTI, 1995).

Several definitions of a PBMS have been proposed by different organizations and reviewed by the MDCB. Various distinctions have been made between a performance-based methods system and a performance-based measurement system. The former generally implies the use of reference methods and their associated performance criteria as the standard of comparison to other methods while the latter requires only stated performance criteria as the comparison standard. Each of these definitions share the concept that PBMS is a framework that permits the use of any appropriate sampling and analytical technology that demonstrates the ability to meet established performance criteria and complies with specified DQOs and MQOs of the project in which the sampling and analytic technology is employed.

The MDCB endorses the development of validated methods and concise, achievable performance criteria. To establish and preserve the credibility of performance-based systems, performance criteria, such as precision, bias, sensitivity, specificity, detection and quantitation levels, and rates of false positives and false negatives must be designated and a sample collection or sample-analysis method-validation process must be documented. Whether we call PBMS a “methods” system or a “measurement” system, the basic goals are the same to provide information of known quality that will satisfy user needs. It is generally agreed by the MDCB that the implementation of a PBMS, with corresponding required data qualifiers entered into a multi-user database, will allow divergent data from numerous environmental programs to be used for many purposes.

The MDCB proposes unifying the term to performance-based system to accentuate the fact that known quality data requires a systems approach whether it is based on method or measurement performance. The MDCB recognizes that there are real differences between a performance method and a performance measurement system and that either form of performance-based system may be appropriate depending on the specific application. In this paper, the MDCB acknowledges the popularity (and confusion) regarding the acronym PBMS and believes that any new acronyms would add further confusion at this time. Therefore, unless specified differently in this paper, the acronym PBMS is used in the more broad sense of a system approach and is neither exclusively a method or a measurement system.

For a PBMS to be successful, the following basic conditions must be met:

. • DQOs or MQOs must realistically define and measure the quality of data needed. These objectives must be compared to the attributes of the data to be used in the performance-based system.

. • Validated methods must be made available that meet these objectives, or objectives should be dependent on results of multiple measurements on known samples using different methods. A number of consensus organization and previously developed EPA methods are available for many frequently measured analytes that may serve as validated methods.

. • The performance of selected methods, in the hands of a qualified operator, must be adequate to meet the DQOs or MQOs and be well documented. Adequacy can be defined as meeting various performance goals or criteria including, but not limited to analytical precision, accuracy, sensitivity; applicability to the measurement target(s) within the applicable matrix; number and type of parameters addressed; and sample collection, preservation, and storage requirements

. • Reference materials covering a variety of relevant matrices, containing the analytes of interest, should be available either through preparation using known concentrations or through round robin testing of unknowns. The MDCB recognizes that this is currently a significant technical limitation for some analytes. Concentrations of the reference materials must be at or near expected quantitation levels or at levels expected in the environment, adding another level of complexity in terms of the availability of appropriate reference materials. It should be noted that the lack of availability of appropriate reference materials for some analytes and matrices is a limitation for both performance-based and prescriptive methods systems.

. • Method ruggedness must be demonstrated. Ruggedness is a measure of reproducibility of test results under normal, expected operational condition, from laboratory to laboratory and from analyst to analyst as well as normal, expected variations within one laboratory by one analyst. Ruggedness must be greater, i.e., the method must have less variability within and among laboratories and matrices if the method is intended for general use in different matrices.

. • For compliance-based programs, mechanisms for determining liability must be clarified.

. • For ambient monitoring programs, the comparability of data collected using different sampling methodologies must be determined.

Advantages of using a Performance-Based System Approach

There are two general advantages to implementing a performance-based rather than a prescriptive methods approach. One advantage pertains to enhancing method technology: development of better, faster, less expensive, or new methods to satisfy new or modified programs. Thus, encouraging a performance-based system ensures that (a) methodologies are appropriate for the matrix being tested, (b) new technologies are adopted much more readily than when using prescriptive methods, and (c) labs can readily modify methods where such modifications are documented as still being effective and reliable.

A second type of advantage pertains to interpretation of data already collected. Using a performance-based approach and documentation of data qualifiers in a database, data users can more easily decide which data can and should be used for their project needs. Thus, encouraging a performance-based system in this sense ensures that (a) data of known quality are reported, (b) data may be used appropriately for several different purposes or by several organizations, and (c) comparability of data collected by different programs can be determined.

In compliance monitoring, both the regulatory agency and the regulated facility need accurate information to ensure that correct environmental management decisions are made. Methods that yield more accurate data, or that attain more appropriate sensitivity given federal or state/tribal standards, should be preferred in compliance monitoring. In addition, the uncertainty or variability of a data point must be known to determine compliance with a regulatory endpoint such as a permit limit or water quality criterion/standard. Under a performance-based system, such method innovations or refinements are encouraged. This could be especially relevant in those cases in which a particular matrix demands certain modifications to a given method.

In ambient monitoring, the monitoring agency also needs to have accurate information to help prioritize and implement appropriate management strategies. More efficient, more accurate, or less costly collection and analysis methods for ambient samples should be preferred because the overall efficiency and quality of the monitoring program would benefit. Under a performance-based system, such method improvements could be explored and implemented relatively easily. Additionally this system offers users the option of using either lower cost, lower accuracy methods with intensive sampling or very sophisticated but expensive analyses with fewer samples, depending on needs and DQOs/MQOs.

Another area in which a performance-based system would benefit ambient monitoring is in determining the comparability and relative quality of data collected by different monitoring organizations across the country. Currently, each organization may have its own methods for certain types of parameters (e.g., biological assessments). The MDCB suggests that method intercomparability exercises be performed to allow for expansion of the spatial extent of programs conducted within the same waterbody by different states. It is difficult to determine which, if any, of these methods produce comparable data and to what extent regional or national status and trends can be discerned given that data are collected using different methods. Under a performance-based system, this situation could be improved because each monitoring entity would need to demonstrate the performance characteristics of their method and ensure that those characteristics are achieved routinely. As a result of this process, it would be possible to determine which methods are comparable, which data can be combined for status and trends analyses, and what data and measurement quality objectives could (and should) be required to address a given question. Another benefit of using a performance-based system for ambient monitoring is that different organizations may be able to use each other’s data, thereby reducing the number of sites that each one needs to sample. Alternatively, more sites could be sampled by the combined organizations for the same cost, increasing efficiency and the information database.

Examples of Programs that Use a Performance-Based System Approach

There are several examples where a performance-based system has been used successfully to generate environmental data of known quality. One example is the NOAA Status and Trends Program (Cantillo and Lauenstein 1998), which has been in operation for over 10 years. This program has been nonrestrictive in terms of methodologies, allowing participants to use any measurement system they felt was appropriate. However, this program ensures comparable data quality among laboratories and methods systems by having: a) well defined DQOs that were shown to be achievable using a variety of methods; and b) a continuing inter-comparison program using reference samples, which are representative of the matrix being evaluated, so that laboratories could demonstrate their ability to meet the DQOs. This program has been successful largely because of the inter-comparison data on reference samples, which has allowed both laboratories and data validators to assess the accuracy of the results from an individual laboratory and to determine whether methods were indeed adequately validated. Demonstrating that a method works on an unknown sample, with concentrations which are environmentally relevant (and not simply reagent water spiked with a high concentration of the test material), is the best demonstration of method adequacy.

Implementation of a performance-based system within EPA’s Office of Water (the primary regulatory office responsible for requiring water quality monitoring), is proceeding with a proposed process to reduce the regulatory burden of prescriptive methods required under the Clean Water Act and the Safe Drinking Water Act. This proposal (Federal Register 1997) applies to chemical and possibly some microbiological methods and would allow analysts to use professional judgment to modify and develop alternatives to established EPA methods. This process is very similar to the definition of PBMS proposed by the MDCB. Implementing PBMS in compliance monitoring programs is particularly challenging. The EPA Office of Water PBMS Implementation Plan, which uses reference methods and stated performance criteria, is recognized as an important step towards practically implementing PBMS.

In anticipation of an Agency-wide shift to a performance-based system, EPA’s solid waste program updated its SW-846 methods in a performance-based system format which included method performance criteria (USEPA 1998). An initial and continuing demonstration of method performance are required in this update and implementation of PBMS does not negate the need or use of standard or consensus methods; it only eliminates the mandate that they be used (however, the National Technology Transfer and Advancement Act (Federal Register 1998), which requires use of consensus methods when available, should be considered). Other examples where a performance-based system has been applied successfully include the EPA’s Pesticide Registration program under FIFRA and the Federal Drug Administration’s drug approval and pesticide analysis program.

The Role of a Performance-Based System in Compliance and Ambient Monitoring

There are some differences in the applicability of performance-based systems to regulatory versus ambient monitoring programs. Compliance monitoring is fundamentally different from ambient monitoring in that data collected for compliance purposes has legal ramifications and could be used by the regulatory agency to support enforcement actions such as violations, fines, law suits, and even facility closure. Thus, any methods approved for use in collecting compliance monitoring data must be: (a) reliable, (b) provide the desired sensitivity, accuracy, and precision required by the particular regulatory program, and (c) assure the identification of the contaminants or analytes being measured. Conversely ambient monitoring, by design, tends to focus on the reduction of error associated with a specific methodology so that related changes to the environment due to a pollution source will not be masked by method-related variability. Although both ambient and compliance monitoring equally require data of high quality, it is generally assumed that there is greater liability associated with compliance monitoring data. However, this may not necessarily be the case as inaccurate ambient monitoring data could lead to inaccurate assessments of waterbodies and an inappropriate selection of total maximum daily loads (TMDLs), resulting in broad and perhaps costly implications.

Liability is one of the major issues currently being debated in regards to implementing a performance-based system (see ELAB 1998). Whereas the EPA, other federal agencies, or states may have the chief liability for method performance and data quality within a prescriptive methods system, under a performance-based system it is less clear who encumbers the liability for the data generated. It is likely that the data generators (contract laboratories or permit-holder laboratories) and/or the data users (i.e., permittees who use a contract laboratory to analyze samples for their facility) could be liable for incorrect or poor quality data, even if the method was approved for use under a performance-based system. In some cases, the state or tribal agency may ultimately be liable for problems caused by poor data quality because ultimately, the state approved the use of the method under a performance-based system. Ideally the state should be reviewing the data and not only the method, but the appropriateness of the method for its use is equally important. Clearly, liability issues need to be resolved before any performance-based system is fully implemented, but it is beyond the purview of the MDCB to directly address them, as many of these issues are ultimately decided on a case-by-case basis.

A related legal issue associated with using a performance-based system in compliance monitoring is that courts of law may be reluctant to recognize results from an alternate or new method even if it has been shown to achieve the same or better level of performance as the prescriptive method previously required. It needs to be made clear who or what organization would be legally responsible for upholding or denying the use of a certain analytical approach in a court of law or who ultimately approves data under a PBMS. Under the current proposal by EPA’s Office of Water, pre-notification of regulatory bodies would be required for new but not modified methods. This approach does ensure that PBMS is used with some foreknowledge, but it does not address the liability issue. Pre-notification also does not address the need for education in dealing with PBMS especially for small facilities in which the level of expertise is typically insufficient to adequately evaluate data without clear guidelines. It remains to be seen how this issue will be resolved within EPA.

A final legal issue regards confidentiality of intellectual property. A laboratory that invests in significantly improving or developing and testing a method may want to retain rights and/or confidentiality regarding the new procedures. It does not appear that regulatory agencies have addressed the possibility of approving a performance-based system in which the details are requested to remain confidential. This issue should be discussed and resolved as part of the implementation of performance-based system.

Despite the current challenges in implementing a performance-based system, the MDCB believes that the potential gains in both compliance and ambient monitoring (e.g., better quality data, greater opportunity for sharing data across organizations, easier incorporation of more appropriate or innovative methods) far outweigh the logistic challenges. The MDCB is available to assist in finding solutions to legal implementation barriers of a performance-based system.

Requirements for Validating Protocols in a Performance-Based System

Ultimately, a performance-based system is no different from Good Laboratory Practice (GLP) in terms of the types of data that must be provided to ensure data quality and comparability. The biggest difference between a performance-based system and the use of prescriptive methods is the degree of prior validation of the methods, which makes it extremely important to have demonstrated the appropriateness of the performance-based approach on the matrix of choice. Additionally, many laboratories will still rely on existing reference methods because they do not have the resources to validate a performance-based protocol. However, even with validated methods, laboratories need to perform their own demonstration of proficiency, because outside groups cannot control how a method is implemented and thus the quality of the data that is generated. The trend towards increasing flexibility in existing monitoring methods, with a rigorous requirement for proficiency demonstration, is an important step towards ensuring that a performance-based system is workable.

In follow-up papers, the MDCB plans to evaluate and recommend the minimum data set required by a laboratory to document the validity of a performance-based protocol. The MDCB is reviewing data quality requirements used by many organizations including USGS, USACE, and NOAA. Both EPA’s Environmental Monitoring Management Council (Performance-based Measurement System checklist) and EPA’s Office of Water (Performance-based Method System checklists) have proposed method performance criteria. Both of these checklists are extensive and often require providing data that does not exist for many of the reference methods. These checklists must be evaluated critically to identify both roadblocks to, and opportunities for, the implementation of a performance-based system. The MDCB has identified techniques not often used by laboratories to evaluate data quality, such as evaluating standard curves with back calculations or rotating levels of check standards, or control charts on appropriate QC materials, which, if encouraged, may eliminate some of both EPA checklists’ requirements. Table 1 below indicates which data the MDCB currently feels should be provided by the laboratory to validate method performance and to ensure comparability of data.

The Performance-Based System Process

Defining the performance criteria of a method that meets MQOs is the first step in initiating a performance-based system. Statistically-based quality-control criteria for replicate measurements and calibrations should be established as a measure of required precision. Bias limits are typically determined by analyzing spiked samples, standard reference materials, and performance-evaluation samples. Long-term method detection limits are desirable to determine the application of a method to monitoring needs or regulatory requirements [USGS in prep]. The performance range of a method also should be determined. The method must not generate background “noise” or be sensitive to interference that will give unacceptable rates of false qualitative or quantitative information. If a method is considered to be applicable for multimedia, then documented evidence should be available to support this use.

Achieving these goals in all media requires training, the availability of matrix-specific performance-evaluation materials, the implementation of a laboratory-accreditation process, and the systematic audit of activities. The current stock of standard chemical and biological reference materials and performance evaluation samples is limited or, in some cases, nonexistent and needs to be developed or expanded to cover a wider range of constituents and media.

The training requirements to implement a performance-based system, and to reach some level of national comparability, are extensive because of the diversity of water quality monitoring programs and data requirements. The MDCB recognizes that adequate training and education of data generators, auditors, and users is central to the successful implementation of a performance-based system. A "National Curriculum" needs to be established and should include formal (training courses and/or published manuals) and informal components. The MDCB is available to assist in this process, assuming sufficient resources are available. Another alternative would be the establishment of Centers of Excellence in academic institutions, which would provide training and assistance in implementation of a performance-based system. This too would require substantial resources. The Methods Board also recognizes the need for laboratory accreditation, with periodic review of activities as an important element in a performance-based system. To this end, the MDCB supports laboratory accreditation efforts by the National Environmental Laboratory Program (NELAP) and other organizations. The MDCB, however, urges these organizations to ensure that they adopt recommendations that reflect the minimum necessary requirements to ensure good quality data and to not create unnecessary or burdensome laboratory requirements. Continued review of protocols by knowledgeable outside auditors will help to make a performance-based system sufficiently rugged when used broadly.

Performance-Based Systems in the Context of Method-Defined and Field-Measured Parameters

Certain parameters, such as BOD, Oil and Grease, and most biological, microbiological (somewhat less so), and field-measured parameters, are “method-defined” indicating that the results obtained are dependent on the particular method used. Unlike many chemical parameters, in which data accuracy and other elements shown in Table 1 can be verified in a number of objective ways, data derived from method-defined protocols cannot often be objectively verified outside the method itself. We are unable, for example, to conduct meaningful "matrix spikes" for biological assessments or toxicity methods. The current EPA Performance-based Method System proposal excludes such methods from a performance-based approach for this reason. However, the MDCB believes that these types of parameters can and should be subject to a performance-based approach; certain method performance characteristics such as sensitivity, precision, and performance range, can be quantified and compared for method-dependent or field methods.

A key factor in the implementation of a performance-based system for method-defined parameters is the use of defined reference conditions or field reference sites (ITFM 1995c; Diamond et al. 1996). Just as analytical methods depend on the availability of appropriate reference materials to verify and document certain method performance characteristics (notably accuracy, sensitivity, and bias), method-defined parameters require either field or synthetic reference samples in which at least the level of the parameters is thought to be known. For example, known clean, well-aerated streams or groundwater could be used to demonstrate method ‘blank’ performance for toxicity or BOD. Alternatively, there may be other ‘reference’ sites that could provide a quantifiable or consistent level of the desired parameter. A certain natural spring may produce a relatively consistent level of toxicity (due to the physicochemical characteristics of the water) or microbial density (due to the source of the water). Various types of method validation and comparability demonstrations using a performance-based approach have begun in several state programs for biological assessment methods (Yoder and Rankin 1995), EPA’s sediment toxicity testing methods (USEPA 1994b), and zooplankton enumeration and species richness methods for the Chesapeake Bay Program (CBMP 1998). Similar kinds of method characterization and documentation have also begun for microbiological pathogen methods within EPA’s Office of Drinking Water and Ground Water.

Table 1. Requirements for Validation of a PBMS Protocol

Item	Existing System	PBMS Used
Specific method or MQO/DQO being compared to.		X
Deviations from the method (with explanation) if modified method.	X	X
Specify matrix being tested		X
Method blank results.	X	X
Other appropriate blank results	X	X
Desired rate of false negatives and false positives		X
Required MQOs and/or DQOs		X
Reference sample results. (reference sample should mimic the matrix of interest and be chosen to test method modifications near the desired action level)	X	X
Spike, duplicate spike, and duplicate sample results, where appropriate	X	X
Surrogate results (if applicable).	X	X
Tuning results to meet method specifications (if applicable).	X	X
Calibration checks and calibration specifications. –multipoint or single point, as applicable	X	X
Any appropriate data qualifiers	X	X
Interference checks if applicable	X	X
Method detection levels and quantitation levels	X	X
Sampling and preservation, testing all relevant parameters	X	X
Project Decision Level, where appropriate	X	X

Note that many of the items suggested here are the same as those that should be available for prescriptive methods, and therefore the burden on an auditor or user to evaluate data is not necessarily significantly increased under this approach, as long as the auditor focuses on data review rather than method review per se. It will however require an increased level of expertise by auditors.

Next Steps for the MDCB

There are other important tasks required to bring PBMS issues to closure for the MDCB. These tasks may include:

. • Refine recommendations for the minimum data set required to document the validity of a measurement protocol under a performance-based system.

. • As applicable, establish quantitative and qualitative performance requirements for comparison of validated and performance-based methods.

. • Recommend reporting formats, consistent data qualifiers, and caveats for a performance-based system (e.g. the kinds of interferences often not checked adequately in environmental data) to maximize the likelihood that a performance-based system will be used effectively.

. • Coordinate efforts with ELAB under NELAC, and other organizations, in refining the implementation of a performance-based system in compliance monitoring.

. • Compile a list, and sources of, reference materials to assist labs in evaluating method modifications.

. • Conduct additional tests of the performance-based validation requirements listed in this paper, using different sets of analytes, to verify its completeness and to provide examples in a future document.