Policy for the Collection and Archiving of Electronically Recorded Data In Reply Refer To: WGS-Mail Stop 460 SEP 18, 1987 WATER RESOURCES DIVISION MEMORANDUM NO. 87.85 Subject: PROGRAMS AND PLANS--Policy for the Collection and Archiving of Electronically Recorded Data The purpose of this memorandum is to state the Water Resources Division (WRD) policy regarding the collection and archiving of electronically recorded data. WRD policy permits the collection and storage of hydrologic data utilizing electronic devices such as satellite telemetry or magnetic recording devices located at the data-collection stations. Electronically recorded data can be utilized to generate records for publication, interpretive studies, research, and original data archives, provided that the data are handled in accordance with the updated policy guidelines presented in the attachment. (The guidelines were originally issued with WRD Memorandum No. 84.106, subject as above.) Adherence to the policy guidelines will also sustain the legal status of electronically recorded data. The Associate Solicitor, Energy and Resources, in a memorandum to the Director, Geological Survey, dated June 10, 1983, concluded that "Data originally prepared as microfilm or recorded directly into a magnetic tape, and processed through a computer to produce the first hard copy is properly admissible and will constitute valid evidence in a court proceeding." The guidelines in the attachment provide for the verification, transfer, transport, processing, and archiving of original data recorded on volatile or reusable media, such as magnetic disks and tapes, computer chips, or other solid-state memory devices. All personnel responsible for the collection, analysis, manipulation, and storage of data using these technologies should assure that the specified requirements are implemented. The guidelines require the archiving of original data, but do not describe the content, format, or procedures to be used. The Office of Surface Water will begin to develop these procedures in fiscal year 1988 as a part of a cooperative effort with the Southeastern Region and the Office of the Assistant Chief Hydrologist for Scientific Information Management. Additional information in this regard will be furnished as soon as it becomes available. Chief Hydrologist Attachment WRD Distribution: A, B, S, FO, P0 U.S. GEOLOGICAL SURVEY WATER RESOURCES DIVISION GUIDELINES FOR COLLECTION AND ARCHIVING OF ELECTRONICALLY RECORDED DATA The Water Resources Division (WRD) at present collects hydrologic data at a number of sites where there is no hard-copy archivable record of the basic sensor information generated at the site. The two types of sites are: (1) locations where the data are transmitted from a remote location by land line or radio (satellite or land based) telemetry to a central location where data are recorded on a medium such as magnetic tape, disk, or solid-state memory device; and (2) locations where data are recorded at a remote location on a medium such as magnetic tape, disk, or solid-state memory device. It is anticipated that significant increases in this type of data collection and recording will occur in the next decade. The purpose of this document is to establish policy guidelines for the collection and archiving of data which are recorded on electronic media. These guidelines follow: 1. Data verification.--Sensors will be read or checked independently of the recording system as appropriate or by using the recommended data-record verification procedure during field visits. Readings or verification information will be written or coded on a log with time and date and compared with the system readings as appropriate via some form of mandatory system output, such as a digital display. Written or electronic logs and calibration tests will be checked against data listings produced from the storage medium during office data reduction and will become part of the permanent record. 2. Data transfer. --The transfer of data from one electronic storage medium to another as a result of remote telemetry, onsite transfer (recorder to portable field unit), or office transfers to a computer will require thorough and, where possible, automated checking to ensure that the transfer was successful, or that as much data as possible have been recovered and errors identified. 3. Data transport.--The unstable nature of the medium necessitates the establishment and use of special procedures and equipment to prevent data errors or loss. Storage media, such as portable field computers, should be carried in protective containers from the field site to the office to protect the media from physical damage. Storage locations in field vehicles should be constructed and located so as to prevent physical damage and electromagnetic destruction of stored data. 4. Data processing.--The initial data processing in the office will be for the purpose of obtaining an archivable copy of the original data as explained in item 5. For this reason, the data are not to be manipulated by the field instrument or by a computer except for the purpose of converting recorded signals into data having commonly used units for hydrologic use or for the purpose of displaying the data in a convenient format. If data compaction schemes have been used, a key or other information must be supplied for reading the data. Where "intelligent" sampling schemes are used in the collection of data, the scheme and criteria for the "intelligent" sampling must be documented. 5. Data archival.--It is the policy of the WRD, as stated in WRD Memorandum No. 77.83 dated March 30, 1977, to indefinitely maintain and preserve all geologic and hydrologic raw-data records in their original form. In the case of electronically recorded data, a copy will be preserved on a nonvolatile medium such as paper or microfilm. Archivable paper copies can be obtained as computer printout of the raw data (unprocessed except as stipulated in consideration 4). An archivable copy of the data may also be obtained by computer output microfilm (COM) of the raw data. Silver halide based film is to be used for this purpose. Thorough checks must be made to ensure that the computer printout copy is consistent with the field recorded data. Paper or microfilm copies of data to be used for permanent storage must be identified by station identification numbers or other identifier, station name, and the period of record. Also, type of data recorded such as stage, rainfall, or specific conductivity must be clearly noted as well as other appropriate information which identifies the data. The storage of paper or microfilm records will be in accordance with the WRD Records Disposition Schedule which is the WRD policy on the handling and storage of data records. Following the creation of an archivable copy of the original data, updates may be made to account for known sensor, recorder, or telemetry malfunctions. 6. Reuse of electronic storage medium.--The field and transport medium used for storing sensor information will generally be erased and reused at some time. These temporarily stored data are not to be erased until an archivable copy of the data has been obtained and checked for accuracy. The storage medium for the field unit (onsite data collection and storage device) shall contain sufficient memory for storage of new data so as to allow time for data transport and the creation of an archivable copy before the retrieved data are overwritten in the field unit.