In Reply Refer To:
Mail Stop 412

August 25, 2008

Office of Water Quality Technical Memorandum 2008.05

Subject: Appropriate Data Storage in the National Water Information System (NWIS)

POLICY

All well-defined water-resource data produced by USGS activities that represent monitoring, investigations, and research collecting measurements of environmental conditions at any specific place and time should be recorded in the National Water Information System (NWIS).

PURPOSE

Water Science Centers (WSCs) frequently ask what data should be stored or even must be stored in the NWIS?  During WSC reviews, review teams find appropriate data are not always archived in NWIS.  This policy clarifies expectations for fulfilling the USGS mission responsibility to archive and make data widely available.  Archiving all environmental data collected to document, describe, and measure water resource conditions and trends is appropriate for NWIS.  Data from computer model results and laboratory experiments are not appropriate for NWIS archival as they do not represent measured environmental conditions.

BACKGROUND

This policy continues our longstanding practice: “All original data that are published or support published scientific analyses should be placed in archives.” (Hubbard, 1992)  Similarly, Water Resources Discipline (WRD) Memorandum 99.07 states that “in general, no restrictions will be placed on the timing or distribution of WRD information products.”  Placing as much of our data as possible in NWIS furthers both these goals.

The NWIS database serves two purposes.  First, it is a permanent archive of historical data for the National Archives and Records Administration, and second, it is a tool used in ongoing and future investigations.  As a result, NWIS is a valuable source for answering external inquiries as well.  NWIS documentation is available at http://wwwnwis.er.usgs.gov/currentdocs/index.html

NWIS provides archival data storage that is systematically maintained and preserved for future use.  The future value of environmental data is not fully known by those who collect it.  Few, if any, people will be able to use data not in common electronic archives, particularly after the original scientist’s career ends and personal contact is impossible.  Archiving data collected at public expense is part of the job in every USGS scientific activity.  Trends and changes in the environment can only be measured in reference to carefully recorded archival data.

NWIS provides tools for USGS staff to work efficiently and accurately.  As an internal transactional database, NWIS records original measurement results and their subsequent modification during review.  NWIS records both completed measurements and failed measurement attempts, in support of internal quality control.  Standard data formats allow for trustworthy data capture from many sources and standard analysis tools allow for data checking and statistical or graphical analysis.  NWIS associates a clear geospatial reference location with every environmental measurement, which facilitates geographical analyses.

NWIS provides a standard route to distribute most USGS water-resource data to the public through the NWISWeb system.  Data in NWISWeb are more likely to be used than data in other forms because the effort required to find and retrieve them is lower.  This ease of access fosters uses of the data, such as integration or comparison with other independently collected information for the same places or times.  Together, the internal NWIS and public NWISWeb databases maximize the value of information collected at public expense by both meeting the needs of the immediate project and facilitating data use in later USGS projects or in the research of others.

SPECIFIC REQUIREMENTS

Site identification:  All sites at which water-resource data are collected, even for a single visit as part of a synoptic or reconnaissance study, must be identified in the Ground-Water Site Inventory (GWSI) subsystem.  Minimum data elements for GWSI site identification have been established (Office of Ground Water Technical Memorandum 98.02), and State Water Science Centers (WSCs) may require additional minimum elements for each local NWIS database.

Time-series data:  Daily and more frequent time series data for sites operated for more than a brief period must be recorded and processed in the Automated Data Processing System (ADAPS) subsystem, and data series of any length may be put there.  Typical data series include stream stage, stream discharge, ground-water level, temperature, specific conductance, pH, dissolved oxygen, and turbidity.  Any water characteristic that is recorded automatically at intervals should be identified with a parameter and method code and stored in NWIS.  Policy differs by parameter whether the database should include only measured values or whether properly-identified estimates may also be included (Office of Surface Water Technical Memoranda 90.11 and 2005.07; Wagner and others, 2006).

Discrete-sample water-quality data:  Data that are produced using analytical methods, either approved or not approved by WRD, should be entered into NWIS (Office of Water Quality Technical Memorandum 2004.01).  Data from approved methods also may be released to the public through NWISWeb and data reports that do not fully explain the laboratory methods.  In addition, long-standing WRD policy requires that analytical laboratories be evaluated, approved, and regularly reviewed before storing the results in a database open to the public.  For the past several years, these requirements have been interpreted as permitting a wide variety of data sources in NWIS, while restricting NWISWeb (our public database) to data from approved methods and laboratories.

The process for reviewing laboratories has now been revised to extend the requirement for review to all laboratories used by WRD, to link the review closely to the data requirements of a specific project, and to focus the lab approval decision primarily on results of specific performance evaluation samples (Office of Water Quality Technical Memorandum 2007.01).

Analytical data stored in NWIS should be linked to metadata that identify the laboratory and method used, among other details.  The USGS Water-Quality Sample Data (QWDATA) subsystem, in particular, now allows storage of abundant metadata, such that any reasonably stable laboratory or field measurement process can be described clearly.  Links to published, peer-reviewed method documentation are preferred.  Data from research or unpublished methods must have appropriate flags to prevent their transfer to NWIS-Web.

Any laboratory or project producing data that must be in NWIS, as defined in the general policy above, also should format the data for electronic transfer to NWIS and include consistent metadata in the transfer.  This requirement is in addition to any other transfer format convenient to the project participants.

Data from non-USGS sources:  In general, it is unwise to add a static copy of data from one database to a second database because later modifications to the original data often fail to reach the copy.  A better practice is for the original database to maintain and revise the data as needed, while the second database points to the original and allows users to retrieve combined datasets from both.  As this ideal is often difficult to achieve, WSCs may store data from any source needed to meet project commitments, but are discouraged from frequently copying data from other active databases.  This guidance does not change the long-established procedures for managing streamflow, stage, ground-water level, and water-quality data provided to USGS by others.  USGS publishes and archives furnished records from others when the data contribute useful information on the water-resources of the Nation, are quality assured by USGS, and USGS can attest that the data are collected using methods creating data with accuracy comparable to USGS data accuracy standards.  The policy of USGS accepting furnished records has been recently restated as WRD Policy Memorandum 2008.01.

EXCEPTIONS

Current data that cannot be clearly characterized, or for which the data quality cannot be defined, should not be stored in NWIS.  These data can be recognized by inadequate metadata, such as an ambiguous definition of the variable to be reported or extremely large uncertainty.  However, historical data already in NWIS should not, in general, be removed for lack of currently acceptable metadata.

Data that result from laboratory experiments or transient experimental conditions in the field do not belong in NWIS as they do not represent environmental conditions comparable to typical environmental data collected by others.  Data that are primarily environmental conditions, but include experimental tracers can be stored in NWIS, provided that the tracer (e.g., bromide) results are clearly identified as such and the specific tracer results are blocked from transfer to NWISWeb.

Data that are too dense in space or time to be recorded clearly in NWIS must be maintained in other databases appropriate for the particular datasets.  For example, the Cape Cod research site of the Toxic Substances Hydrology Program has more than 10,000 ground-water observation points in an area roughly 100 by 200 meters, which overwhelms the NWIS site identification system.  Similarly, the ADAPS subsystem is presently capable of recording a maximum of 2880 observations of a variable per day, although data at a minimum time interval of 1 second can be recorded for part of a day.

ALTERNATIVE ARCHIVES

When projects generate significant quantities of data that, for whatever reason, are not archived in NWIS, WSCs and the principle investigators have a responsibility to ensure that the data are archived in ways to remain available for other researchers and to verify published results.  USGS has a public responsibility to provide unbiased and transparent results.  If published results cannot be verified by subsequent or external inquiry, the unbiased and transparent nature of USGS work is called into question.  Thus it is essential that archival responsibilities through NWIS or some other mechanism are successfully accomplished.

In addition, principle investigators are encouraged to establish at least one NWIS site as a pointer to the larger dataset.  For example, some well-characterized part of the data might be recorded in QWDATA along with a sample-level text comment giving a report citation, a brief description of the full dataset, or some other pointer.  Geographic searches of the database that encounter the site could then lead a later researcher to the full dataset.  Alternative archives are also discussed in Office of Surface Water Technical Memorandum 2005.08 for electronic discharge measurement data.
 

References:
 
Hubbard, E.F., 1992, Policy recommendations for management and retention of hydrologic data of the U.S. Geological Survey: U.S. Geological Survey Open-File Report 92-56, 32 p.

Office of Ground Water Technical Memorandum 98.02, Data Elements for Ground-Water Sites, http://water.usgs.gov/admin/memo/GW/gw98.02.html

Office of Surface Water Technical Memorandum 90.11, RECORDS--Policy on the Publication of Estimated Flows, http://water.usgs.gov/admin/memo/SW/sw90.11.html

Office of Surface Water Technical Memorandum 2005.07, Use of the Program HYDRA to Estimate or Modify Unit Values in ADAPS, http://water.usgs.gov/admin/memo/SW/sw05.07.html
(identical to Office of Ground Water TM 2005.03 and Office of Water Quality TM 2005.03)

Office of Surface Water Technical Memorandum 2005.08, Policy and Guidance for Archiving Electronic Discharge Measurement Data, http://water.usgs.gov/admin/memo/SW/sw05.08.html

Office of Water Quality Technical Memorandum 2004.01, Revised Policy for the Approval of U.S. Geological Survey (USGS) Water-Quality Analytical Methods, http://water.usgs.gov/admin/memo/QW/qw04.01.html

Office of Water Quality Technical Memorandum 2007.01, Policy for the Evaluation and Approval of Analytical Laboratories, http://water.usgs.gov/admin/memo/QW/qw07.01.html

Wagner, R.J., and others, 2006, Guidelines and standard procedures for continuous water-quality monitors—Station operation, record computation, and data reporting, U.S. Geological Survey Techniques and Methods Book 1 Chapter D3, 83 p., http://pubs.usgs.gov/tm/2006/tm1D3/      

Water Resources Division Memorandum 99.07, Policy on Release of Water Resources Division Information Products, http://water.usgs.gov/admin/memo/policy/wrdpolicy99.07.html  

Water Resources Discipline Memorandum 2008.01, Policy on Accepting Furnished Records, July 1, 2008. http://water.usgs.gov/admin/memo/policy/wrdpolicy08.01.html



                          Timothy L. Miller /s/
                          Chief, Office of Water Quality

This memorandum does not supercede any existing policy.

Distribution:  All WRD Employees