[link or image removed] indicates that a reference has been removed from this document in order to prevent the exposure of internal resources.

GW2017.03, SW2017.10, WQ2017.07 Procedures for Processing, Approving, Publishing, and Auditing Time-Series Records for Water Data

Purpose:

This memorandum defines the procedural steps for processing, approving, publishing, and auditing time-series records for water data. The steps are explicitly defined by data type as are the tasks required to accomplish them.

Introduction: The demand for timely surface-water, groundwater, and water-quality time- series data has increased and the technology used to collect, process, and deliver these data has evolved and improved. USGS Water Science Centers (WSCs) have adapted the workflow for continuous records processing and approval, as outlined in WRD Policy Memorandum No. [link or image removed]2010.02 [link or image removed]and clarified with joint 2017 Technical Memoranda Office of Groundwater (OGW) GW2017.01 [link or image removed], Office of Surface Water (OSW) SW2017.05 [link or image removed], and Office of Water Quality (OWQ) WQ2017.02, [link or image removed]to meet these demands. However, former procedural aspects of the workflow sometimes have been forced to fit the new demands and, similarly, former terminology has been forced to fit within new workflow procedures. As a result, the procedures followed by WSCs for processing and approving time-series records have become inconsistent and inefficient. The implementation of the new AQUARIUS software provides an opportunity to redefine the procedures for processing and publishing time-series records and to define a workflow that removes redundancy, provides efficient quality assurance, and leads to more timely delivery of approved data.

The goals of these procedures for processing, approving, and publishing time-series records are to: Policy:  Time-series processing and publication procedures consist of two distinct steps: analyze and approve. After data are approved, any additional examination will be considered an audit. Some data types will have required routine audits. Documentation of the analysis, approval, and audits is required to be done in the Water Mission Area supported time-series records management system. No time-series data may be published through the National Water Information System (NWIS) unless they have been processed following these procedures. When followed, this policy meets the requirements of the U.S. Geological Survey’s (USGS) Fundamental Science Practices for review and release of scientific data on the National Water Information System Web Interface (NWISWeb) as described in Survey Manual (SM) Chapter 502.8 [link or image removed].

The new internal data states of time-series data are working, analyzed, and approved. The data processing roles are Operator, Analyst, and Approver. The Operator is the person assigned to make site visits and collect field data and often serves as the Analyst. Records will be analyzed for discrete periods of time by a hydrographer (the Analyst); more than one Analyst may be needed in some situations. A second hydrographer (the Approver) will examine the analysis and either approve it, or return the analysis to the initial hydrographer informing them of what needs to be resolved in order for it to be approved. The WSC Director or designee will assign the appropriate individuals to the data processing roles based on experience in collecting and processing time-series water data. To ensure that the quality of published data remains high, data types that require routine audits will be audited at preset intervals.

Data states of time-series data (with corresponding NWISWeb display) and short definitions:

    1. Working (NWISWeb = Provisional): The working data state is defined as time-series data that are as accurate as possible following current best practices and in compliance with Technical Office and Water Mission Area policies (for example, OSW Technical [link or image removed] Memorandum 2014.08 [link or image removed]). The working data state is shown on NWISWeb as Provisional.
    2. Analyzed (NWISWeb = Provisional): The analyzed state is defined as fully processed time-series data with a complete station analysis. Analysis of a time period should be initiated as soon as practical after field data are collected. If, during the analysis of a period, it is determined that there are insufficient data to complete the analysis, the data state will remain as working until additional data are collected and issues are resolved. Data in the analyzed data state are shown as Provisional on NWISWeb.
    3. Approved (NWISWeb = Approved): The approved data state is defined as completely analyzed time-series data, including examination for errors and proper interpretation. Following full resolution of any problems identified, the record is set to Approved by a second hydrographer. Data are shown as Approved on NWISWeb and are considered “published”.
Additional information may become available after the data are approved that can be used to improve the published record. Time-series data types for which the acquisition of additional information can lead to improved published record, such as model-derived data types like streamflow or sediment concentrations, may be routinely audited. Routine audits are made at defined intervals, such as annually. Routine audits may also be required for a sample (specified percentage) of published record for other data types to ensure systematic quality assurance of all time-series data. Routine audits verify the effectiveness of the approval process and can be adapted as needed to meet the needs of specific data types.

Non-routine audits may occur any time and on any aspect of an approved record that needs to be re-examined. For example, an end user may have a question about the gage-height data for May and June two years ago. When the data are re-examined, the audit is documented with a description of what was examined, why it was examined, and the outcome of the examination. Another example of a non-routine audit would be examination of data quality as part of a triennial technical review. In this case, all aspects of a designated time period are examined and the documentation includes the notes or forms that were filled out by the technical reviewer.

The tracking of audits documents the level of review that the data have received and prevents unneeded re-examination and re-analysis of previously audited data. All audits should have attributes describing the "level" of audit that the data have received and the “results” of the audit.

Audit Levels:

Audit Results:

All audits should be logged and documented internally. The decision to make changes to approved time-series records due to the findings of an audit should be based upon existing revision criteria as documented in policy and procedures to identify and document revisions to USGS approved time-series and discrete water data as required by OGW (GW2017.02 [link or image removed]), OSW (SW2017.07 [link or image removed]), and OWQ (WQ2017.03).

Explicit guidance on the tasks required to achieve the analyzed and approved data states for each time-series data type, along with any required routine audits, are provided at https://water.usgs.gov/osw/time-series-guidance/. [link or image removed]

 

//signed//

William L. Cunningham Chief, Office of Groundwater

//signed//

Robert R. Mason, Jr.

Chief, Office of Surface Water

//signed//

Donna N. Myers

Chief, Office of Water Quality

 

REFERENCES

U.S. Geological Survey, 2010, Continuous Records Processing of all Water Time Series Data, Water Resources Discipline Policy Memorandum 2010.02 (available online at http://water.usgs.gov/admin/memo/policy/wrdpolicy10.02.html [link or image removed])

 

Office of Surface Water, Office of Groundwater, Office of Water Quality 2017. Clarification of Water Mission Area Policy on Continuous Records Processing of Time Series Data: Categorization of Groundwater, Surface Water, and Water-Quality Sites/Records (available online at https://water.usgs.gov/admin/memo/SW/sw17.05.pdf [link or image removed], https://water.usgs.gov/admin/mem [link or image removed] o/GW/gw2017.01.pdf [link or image removed], https://water.usgs.gov/admin/memo/QW/qw2017.02.pdf [link or image removed])

 

U.S. Geological Survey Manual Chapter 502.8 – Fundamental Science Practices: Review and Approval of Scientific Data for Release, updated 01/13/2017 (available online at https://www2.usgs.gov/usgs-manual/500/502-8.html [link or image removed])

 

Office of Surface Water, 2017, Policy for the timely correction of time-series data displayed on the NWISWeb, Office of Surface Water Technical Memorandum 2014.08 (available online at https://water.usgs.gov/admin/memo/SW/sw1408.pdf [link or image removed])

 

Office of Surface Water, Office of Groundwater, Office of Water Quality 2017. Procedures for Identifying and Documenting Revisions to USGS Water Data (available online at https://water.usgs.gov/admin/memo/SW/sw17.07.pdf [link or image removed], https://water.usgs.gov/admin/mem [link or image removed] o/GW/gw2017.02.pdf [link or image removed], https://water.usgs.gov/admin/memo/QW/qw2017.03.pdf [link or image removed])

 



Time-Series Guidance