In Reply Refer To:
Mail Stop 412

June 26, 2007

Office of Water Quality Technical Memorandum 2007.04

Subject: Controlling Flow of Water-Quality Data to NWISWeb

Major changes are forthcoming to the National Water Information System (NWIS) processes for compilation and display of water-quality data on the NWISWeb system (http://waterdata.usgs.gov/nwis/nwis). This memorandum provides detailed guidance to the U.S. Geological Survey (USGS) Water Science Centers (WSCs) on options for controlling the release of water-quality data based on the policies described in Office of Water Quality Technical Memorandum 2007.03. This memorandum supercedes Office of Water Quality Technical Memorandum 2000.10.

The flow of water-quality data into an aggregated national database and subsequent NWISWeb display are controlled by WSC personnel by controlling the aggregation frequency and the attributes of the stored data. The aggregated national database will be used internally for national program needs and NWIS redesign investigations. A subset of the aggregated national database will be used to serve data to internet users via NWISWeb.

Controlling the attributes of stored data:

There are NWIS attributes at the site, sample, and result levels that each provides restrictions on the release of data. The exclusion effects are cumulative. For example, restricting data for a site restricts all the sample and result data for that site, regardless of the sample- and result-level attributes. Similarly, restricting a sample restricts all the stored results associated with that sample.

• Site-level controls
  
  Site-web flag: This field contains the status and availability of a site record for display on NWISWeb. All sites will be aggregated nationally, but only records flagged with a ‘Y’ (which indicates that the record has been checked and is ready for web display) are displayed on NWISWeb. The site-web flag can be set by NWIS database administrators (DBAs) using programs described in the NWIS GWSI User's Manual.

• Sample-level controls
  
  District processing status code: In NWIS version 4.6 and earlier, the sample-level district processing status code (DSTAT) is one of two sample-level codes that can be used to control the release of samples and associated results. Samples with DSTAT code = Z [Complete, but do not transmit (Local-use data for internal use)] will not be displayed on NWISWeb. The DSTAT code has been eliminated in NWIS version 4.7, and specific elements of the code have been incorporated within a modified analysis status code domain as discussed below.
  
  Analysis status code: The sample-level analysis status code (ASTAT) can also be used to control the flow of samples and associated results to NWISWeb. In NWIS version 4.6 and earlier, the ASTAT code was used to identify the workflow progression of data entry, review, approval, and distribution. The ASTAT codes are listed in table 7 of Appendix A of the QWDATA User's Manual.
  
  In NWIS 4.7, the ASTAT domain will be revised and contain the following elements:
  
  U - Unrestricted
  I - Internal-use only
  P - Proprietary
  
  Proprietary data consist of information obtained by the USGS for which the Bureau’s rights of ownership are restricted in a manner that limits our ability to freely distribute the data (http://wwwnwis.er.usgs.gov/SECURITY/Privacy/Proprietary.data.Policy.html). Internal-use data, which include quality-control (QC) data, are generally not available to the public. Unrestricted data are available to the public, and the distribution of these data is not restricted in any way. Samples coded with an ASTAT of ‘P’ will not be aggregated nationally. Samples coded with an ASTAT of ‘I’ will be aggregated nationally, but not displayed on NWISWeb. Samples coded with an ASTAT of ‘U’ will be aggregated nationally and displayed on NWISWeb.

• Result-level controls
  
  Data quality indicator code: The data quality indicator code (DQI) is a result-level field used to designate the review status of the result, control the ability of a batch input program to overwrite the result, and affect the inclusion of the result in output or on the web. The DQI codes are listed in table 14 of Appendix A of the QWDATA User's Manual. Results coded with proprietary DQI codes P, O, and X will not be aggregated nationally. Results coded with DQI codes of I, Q, and U will be aggregated nationally, but not displayed on NWISWeb. Results coded with DQI codes of A, S, and R will be aggregated nationally and displayed on NWISWeb.
  
  Null-value qualifier code: Null-value qualifier codes identify a failed measurement due to field, lab, or shipment problems. Results coded with a null-value qualifier will be aggregated nationally, but not displayed on NWISWeb. The null-value qualifier codes are listed in table 15 of Appendix A of the QWDATA User's Manual.
  
  Parameter-public flag: In NWIS version 4.7, the parameter-public flag is a new attribute of the parameter reference table that allows or disallows the public data retrieval of a parameter code. Parameters with the parm_public_fg “on” will be available for public display and parameters with the parm_public_fg “off” will be aggregated nationally, but not displayed on NWISWeb. Examples of parameter codes with the parm_public_fg “off” are deactivated codes such as 99875 (special code reserved for USGS laboratory use only). Some parameter codes may indicate that water is used as a public supply (parameter codes 71995-71998 and parameter code = 72005 with specific values) and are not displayed on NWISWeb.
  
  The values of data attributes for NWIS version 4.7 that control the release of information from the local NWIS database system to the nationally aggregated database and to the internet on NWISWeb are summarized in table 1.
  
  Table 1. NWIS version 4.7 attributes that control flow of water-quality data to NWISWeb.
  
  

  Restriction Level	Data attribute	Values that allow the data flow to:
  		National aggregation	Display on NWISWeb
  Database	--	Not restricted	01
  Site	Site-web flag	Not restricted	Yes
  Sample	Analysis status code	U or I	Only U
  Result	Data-quality indicator code	Not P, O, or X	Among A, S, or R
  	Null-value qualifier code	Not restricted	Not displayed
  	Parameter-public flag	Not restricted	Yes
  	Parameter code in 71995-71998 and result value = 4941	Not restricted	Not displayed
  	Parameter code = 72005 and result value in 44, 46, 69, or 70	Not restricted	Not displayed

  
  Details on the site-, sample-, and result-level attributes are described in the NWIS User manuals for the GWSI and QWDATA programs (http://wwwnwis.er.usgs.gov/currentdocs/).

Controlling the aggregation frequency (future enhancement):

• NWISWeb refresh cron job: In the future, aggregation frequency will be controlled manually or automatically through the management of the NWISWeb refresh cron job. The cron will be used to create a file of data that is subsequently copied to the aggregation host. The default cron will be set to scan QWDATA once per day for changes, but can be set to operate on a monthly, daily, or more frequent basis depending on WSC needs. Instructions for setting up the cron are available at http://waterdata.usgs.gov/nwis/?nwisweb_manual_faq#change_cron_program_runtime. Instructions for the manual uploading of data to NWISWeb are available at http://phoenix.cr.usgs.gov/reload/nwisweb_qw_update.html.
  
  For further assistance with controlling the flow of water-quality data to NWISWeb, please send email to GS-W-Phoenix.

Walton H. Low /s/ for Timothy L. Miller
Chief, Office of Water Quality

This memorandum supersedes Office of Water Quality Technical Memorandum 2000.10.

Distribution: All WRD Employees