August 8, 2005
OFFICE OF SURFACE WATER TECHNICAL MEMORANDUM NO. 2005.07
OFFICE OF WATER QUALITY TECHNICAL MEMORANDUM NO. 2005.02
OFFICE OF GROUND WATER TECHNICAL MEMORANDUM NO. 2005.03
SUBJECT: Use of the Program HYDRA to Estimate or Modify Unit Values in ADAPS
THIS DOCUMENT SUPERSEDES:
OSW TECHNICAL MEMORANDUM NO. 2003.02
OWQ TECHNICAL MEMORANDUM NO. 2003.02
OGW TECHNICAL MEMORANDUM NO. 2003.01
The latest release of the National Water Information System (NWIS) Automated Data Processing System (ADAPS), version 4.5, for the first time allows the direct editing of “computed” unit values. In previous NWIS releases only “edited” unit values could be manipulated in HYDRA. Also in this release, the computed unit values changed or added in Hydra automatically will be assigned an “estimated” remark code (e) which in turn will cause corresponding daily values subsequently computed from an ADAPS primary computation to be flagged as estimated. The release of these features as part of NWIS 4.5 requires updating the previous policy on editing unit-value data using HYDRA.
It is now important to make a distinction between “edited” and “computed” unit values. “Edited” unit values are those that come directly from a field instrument but have undergone editing by a hydrographer. Such editing is limited and in general reflects the removal of bad data, application of remark codes, or combination of data from multiple data sources into one data set. “Computed” unit values are edited data that have had any necessary systematic data corrections applied. These corrections may be related to instrument drift, fouling, or corrections to a different datum, but in all cases are applied though the ADAPS primary processor rather than by hand. These types of unit values are commonly called computed “input” unit values. Computed unit values also may be values that are not directly recorded in the field but are calculated from another
parameter, such as discharge calculated from stage or salinity calculated from specific conductance. These types of unit values are commonly called computed “output” unit
values. Both computed input and output unit values reflect the final values of a given parameter, and are the only unit value data able to be displayed on NWISWeb.
These changes in the NWIS 4.5 release and the policy outlined in this memorandum are continuing steps toward making unit values the primary basic product of our data collection programs. However, there is nothing in this policy that requires the estimation of data at the unit value level at this time. For discharge and other time-series data for which estimating missing record is common (e.g., sediment load), it remains acceptable to estimate daily values. The new tools added to HYDRA allow and facilitate the estimation of data at the unit-value level where appropriate, as described below, and/or can be used to increase the efficiency of records processing.
The following guiding principles shall be used to judge the appropriate use of HYDRA to estimate or modify unit values:
1. Modifications to “edited” unit values should be limited to the application of appropriate remark codes, copying or hand entry of data from other sources, or removal of obvious spikes or other spurious data. Edited unit-value data should never be created if it did not previously exist on some recording device. Note that direct observation of a parameter and manual recording of this observation in field notes is considered a recording device.
2. “Computed” unit values may be estimated by editing existing data or adding new data as necessary, subject to limitations based on the type of data being collected (see Appropriate and Inappropriate Uses below). This refers to both computed input parameters (e.g. stage) and output parameters (e.g. discharge). This editing may be done using all the available tools in HYDRA including the graphical editor.
3. Otherwise correct input data, whether edited or computed, should not be changed simply to get a correct computed output result.
4. It is preferable to estimate output unit values such as discharge directly, while still recognizing that input data have value to the user community and are not just a surrogate for the output unit values.
5. Estimation of ground-water level and water-quality time series data are not acceptable.
1. It is appropriate to use the graphical editing option of HYDRA for computed unit values if there are stage data problems of any kind, including missing data. It is
preferable that the estimates be done to the discharge data directly and that the bad stage data be marked erroneous. However, the computed stage data may be estimated and the estimated discharge developed from the primary computation, if it is important to have a complete stage record.
2. It is appropriate to “paste” edited unit values from backup sources into the primary data source using the “select and paste” option in HYDRA. Backup measurement sources must record values at the same site. For example, stage may be measured at a site using a pressure transducer (primary sensor) and a float/counterweight (backup sensor); measurement of water-quality parameters may be made during the time that the water-quality sensors are removed for servicing.
3. It is appropriate to add an “erroneous” remark code (X) or to delete any spurious edited unit value data. It is preferable to mark such data “erroneous” as the decision is tracked more appropriately within the database. However where the data exhibit extreme fluctuations or spikes that will make graphical display of the dataset difficult due to scale issues, it is acceptable to delete the spurious edited unit value data.
1. HYDRA should NEVER be used to alter correctly recorded input unit values simply to generate correct output unit values. For example, input stage values that are correct, but are ice affected, should never be “adjusted” to create stage values which, when applied to a rating curve, yield the correct discharge unit values. Similarly, correctly recorded stream-stage values that are affected by backwater should not be edited to yield correct discharge unit-values. When these situations occur the discharge unit values should be estimated directly.
2. HYDRA should not be used to estimate missing ground-water levels or water-quality data. Only measured ground-water levels or water-quality values should be stored in NWIS and published in the Annual Data Report.
The version of HYDRA released as part of NWIS 4.5 has several changes that help enforce many of the above policy decisions. These include:
The direct importing of “computed” unit values into HYDRA and their graphical editing is now supported. Any editing of computed unit values will result in the automatic application of an “estimated” remark code and primary processing will pass that estimated remark code to any calculated unit and daily values.
Graphical editing of “edited” unit values has been turned off. This reflects the fact that such mass editing is actually data estimation and should be done to the computed unit values. The utility, nwts2rdb, has been modified to retrieve water levels from GWSI.
Stephen F. Blanchard /s/
Chief, Office of Surface Water
Timothy L. Miller /s/
Chief, Office of Water Quality
William M. Alley /s/
Chief, Office of Ground Water