In Reply Refer To:
Mail Stop 415

OFFICE OF SURFACE WATER TECHNICAL MEMORANDUM NO. 2006.03

SUBJECT: Availability of New Instantaneous Data Archive for Unit-Value Discharge Data

The purpose of this memorandum is to announce the availability of the Instantaneous Data Archive (IDA) for storing historical instantaneous (unit-value) discharge data. The web address for IDA is http://ida.water.usgs.gov.

Since 1889 the United States Geological Survey (USGS) has been collecting continuous stage, discharge, and other instantaneous time-series data on the Nation’s rivers and streams. The recorded time-intervals for these continuous data typically range from 5 to 60 minutes. These instantaneous data have been processed into and published as various daily values, such as the daily maximum, minimum, and/or mean. Because the published records are daily values, the original instantaneous data have not historically been officially approved, published, or made widely available. The IDA has been established to make available as much historical instantaneous discharge data from USGS streamgages as possible. Although this site currently serves instantaneous discharge (streamflow) data only, work is planned to extend it to other time-series parameters in the future.

As described above, the USGS procedure for processing and publishing time-series data has focused on daily values as the final product and not the instantaneous values. As a result, the instantaneous values were not corrected and processed to the same extent as the daily values. Because of these USGS procedures, the instantaneous discharge data provided through IDA should be considered predominately unreviewed and unapproved data.
In order to provide a basic level of review and quality assurance of these data, the instantaneous data loaded into IDA are recovered and compared against the published daily-mean discharge through the use of automated filtering and computational software. Thus the archive is being built around a process that compares the available instantaneous data to the published daily mean values. For each set of instantaneous data that exists for a given day, a new daily mean value is calculated and compared to the published daily mean value. The instantaneous data set that computes a value that most closely matches the published value is added to the archive for that day, along with an accuracy remark code explaining how closely it matched. As each daily mean discharge computed from that day’s instantaneous discharge values is compared to the published daily mean discharge value, that day’s instantaneous values are summarily categorized by one of the four accuracy remark codes described below or they are excluded if they do not meet a base accuracy criteria.

0 - A daily mean discharge calculated from the instantaneous data on this day is 0.01 cubic feet per second or less and the published daily mean is zero.
1 - A daily mean discharge calculated from the instantaneous data on this day matches the published daily mean within 1 percent.
2 - A daily mean discharge calculated from the instantaneous data on this day matches the published daily mean from 1 to 5 percent.
3 - A daily mean discharge calculated from the instantaneous values on this day matches the published daily mean from 5 to 10 percent.

If the daily mean discharge calculated from the instantaneous values on a given day is greater than 10 percent different from the published daily mean, those instantaneous values are excluded from the archive. In addition, if a published daily mean is an estimated value, the corresponding instantaneous data for that day are automatically excluded from the archive regardless of the results of the comparison.

It is important to note that the values available in the archive have not been individually reviewed and approved, nor are they described as such on the IDA web site. Rather they have been automatically compared against the published daily mean value and found to have no gross errors when used to compute a daily mean. Individual instantaneous values may still have significant error and may have been affected by backwater, instrumentation or transmission problems, or any number of other issues. Data users are, therefore, cautioned to consider carefully the provisional nature of the information before using it for decisions that concern personal or public safety or the conduct of business that involves substantial monetary or operational consequences. Instructions on the IDA web page strongly encourage IDA users to review all data prior to use.

The USGS converted to electronic storage and computer processing of instantaneous time series data in the 1980’s, however it was not until the late 1990’s that computer storage was sufficient to hold long-term periods of instantaneous data. Recovery of historical data from backup tapes into NWIS has been done in most locations, and thus stations loaded into IDA will typically have data starting in the late 1980’s. Earlier data are archived on paper charts and would have to be processed manually. Currently there are no resources to accomplish this monumental task, but it is hoped that the existence of IDA will spur further interest in historical instantaneous data that will lead to data recovery projects.

We acknowledge that populating this web site with available instantaneous discharge data will have to be done as resources become available. At this time, IDA is only available internally and only includes data from the beta test sites. IDA will be made public after more Water Science Centers (WSCs) process and load their data. Although use of IDA is voluntary, the experience of the beta test sites indicates that all WSCs will find the benefit well worth the time involved (approximately two days).

All WSCs are strongly encouraged to load their unit values discharge data into IDA as time and resources permit. Information on loading data to IDA can be found at: http://hexe.er.usgs.gov/usgs/adaps/uvarchive/index.html.

A cyber seminar introduction to IDA and further training on loading data will be announced shortly. For further information, please contact Joseph Nielsen (jnielsen@usgs.gov) or the IDA help group (GS-W Help IDA).

Stephen F. Blanchard
Chief, Office of Surface Water