Data Archival Guidance

How to Archive NAWQA Data

Questions and comments are welcome to: jsstewar@usgs.gov

Contents

• Introduction
  • Purpose
• What needs to be archived?
  • Discussions
    • Why archive?
    • Static versus dynamic archive
    • When to archive multiple copies of the same data
    • What is the role of NWIS?
  • Categories of study-unit data
• Decisions to be made by study units
  • Select an archive custodian
  • Process data sets
    • Determine the fate of the data set
    • Designate a data archive custodian
    • Document the data set
    • Organize the data set
    • Catalogue the data set
  • Arrange the data sets
    • Physical records
    • Computerized records
      • Storage medium
      • Directory structure and naming conventions
  • Review and distribute the archive
    • Study-unit archive
    • National-synthesis archive
• Getting assistance
• Tables
  • Table 1.--NAWQA data archive review checklist categories.
  • Table 2.--Suggested file-name suffixes.

Why archive?

Archiving study-unit data is one of the primary work elements as NAWQA study units move from the High-Intensity Phase (HIP) to the Low-Intensity Phase (LIP). Records managers typically consider archival activities to consist of organizing and safeguarding "original records." Archiving in the context of this document refers not only to "original records," but also to plans, interpretive documents, and data sets that have been modified from their original form and content. In this sense, archiving by NAWQA study units is performed to ensure that a legacy remains for subsequent investigators' work without needless repetition or uncertainty about the purpose and extent of prior work.

Static Versus Dynamic Archive

In theory, earth-science observations are static values because once recorded, the values represent a measurement of some phenomenon at a particular place and time. However, this theory is belied by actual data-base experience.

• Data often are entered into the computer prior to quality assurance, thus data-entry errors are detected and corrected at some later time.
• Quality-assurance techniques frequently compare different data elements in the data base, which requires waiting until all records contain the desired data elements.
• Analysis of quality-control samples at the laboratory causes later revisions to method-detection levels, and subsequent alteration and retransmission of environmental-sample results.
• Computer software bugs can introduce systematic errors in the data.
• New data are being collected and (or) entered continuously.

Thus in actual practice, theoretically static data are dynamic, as ongoing quality-assurance processes improve the quality of the data base and new data are added. A dynamic data base poses a data-archival problem. When, if ever, is the correct time to create a "snapshot" of a dynamic data base? Should the archive simply consist of the dynamic data base?

The answers to these questions depend on the needs of the archive audience. There are several potential audiences that need a "snapshot:"

• Multiple analysts of the same data set need the same copy of the data, to avoid possibly embarrassing conflicting interpretations. ("How come A says there are 31 wells in glacial till and B says that there are 35?")
• A single analyst, performing multiple interpretations of the data would not benefit from a data set that varied from one analytical procedure to the next.
• Some future analyst, doubting how well the original data supported a previous analyst's interpretation would require an exact copy of the original data, including any erroneous values that were found and corrected between the investigations of the two analysts.

While a dynamic data base is beneficial because data quality and availability are constantly improving, a "snapshot" data base is needed for archiving. Creation of a "snapshot" data base entails establishing a "due date" for data-quality assurance. A "due date" provides an impetus to complete the data review in a timely manner, while at the same time stressing limited time and resources at the study unit. The national-synthesis and public audiences, in particular, would benefit from a data-base "snapshot." A "snapshot" is optional for meeting local study-unit needs; however a "snapshot" is necessary to meet national needs. Outreach and data dissemination could be achieved by handing out a NAWQA CD-ROM, for example; so the local Survey office could benefit from a "snapshot" product.

A data-base "snapshot" does not eliminate the need to correct errors and add new data to the dynamic data base. Relatively painless procedures are required to make the snapshot data base from the dynamic data base. If done consistently among study units, the documentation requirements could be drastically reduced, as "cookie-cutter" documents would suffice.

When to archive multiple copies of the same data

Data sets typically occur in three different stages of evolution: (1) the "raw" stage, where no attempt at error detection or correction has been made; (2) the "quality-controlled" stage, where the data have been carefully reviewed and all known errors have been corrected; and (3) the "analysis" stage, where records have been selected that suit a particular analysis. Often several "analysis" data sets are created from a single "quality-controlled" data set, to meet different analytical goals.

If only one copy of a data set is archived for NAWQA, it should be the "quality-controlled" data set. Documentation of the changes made during the evolutionary cycle of the data set also should be provided, in the form of narrative text, computer programs that make the necessary changes, "transactions" showing the changes, and (or) copies of the "raw" and "analytical" data-set stages.

Often good reasons for keeping two or three copies of a data set in the different stages of development exist. A "raw" copy of a data set obtained from another agency's computer system could be retained, because retrieving the original data set may be impossible in the future since computerized data bases are dynamic. Keeping a copy of the "analysis" data set is advised, if the results of the analysis are likely to be contended or controversial, or if the "analysis" data set is difficult or time consuming to recreate.

What is the role of NWIS?

The use of NWIS alone as the NAWQA data archive is inadvisable because the data-base system does not currently offer the attributes needed for a NAWQA data archive. Therefore, an additional storage location is needed.

Several observations about the NWIS data-management system are pertinent.

1. NWIS provides many software capabilities for entering, editing, validating, tabling, and exporting data.
2. Data from the National Water-Quality Laboratory (NWQL) are transmitted to the study units via NWIS.
3. Most of the site-descriptive information is stored in NWIS.
4. Special programs for handling time-series data exist in NWIS, and most, if not all, these data are stored there.
5. RDB-based computer programs have been developed to easily process NWIS data.

The above considerations (as well as WRD policy) suggest that all NAWQA data that fit the NWIS data model should be stored in NWIS. NWIS should contain the "quality-control" evolutionary stage of the data, because others outside the study-unit team will have access to much of these data through NWISWeb.

An archival checklist was developed to help study units plan,conduct, and document data archival. The checklist should be used as a guide and will serve as the primary documentation for study unit archive reviews. The archival categories listed in Table 2 are included on the archive review checklist.

Table 1.--NAWQA archive review checklist categories. 

------------------------------------------------------------------------------
Site History and Identifiers 
         Location narrative and directions
         Maps with locations plotted
         Data Networks, description, and Site IDs
         Site reconnaissance notes and sketches 
         Land owner, local contacts, and permission forms 
         Sampling/measurement location sketches 
         Photographs, slides and video
         Site header data 
         Well inventories 
Continuous Data Records 
         Raw media (ADR tapes, datalogger files (Hobo temp files),strip charts) 
         Preliminary and processed computations for stage, temperature, 
           conductance, pH, DO from on-site monitoring equipment. 
         Computed unit-values and daily-values data
Field Notes 
         Measurement notes (Discharge, leveling, light penetration,etc.)
         Water-quality field sheets 
         Field-analysis record sheets (titrations, immunoassay, Hydrolab, etc.
         Site inspection notes 
         Analytical Services Request forms (ASRs)
         '91 Land use / land cover forms
         '94/'97 New Land use / land cover field forms
Chemical Analyses
         Water, including suspended sediment
         Tissue
         Bed sediment
         Atmospheric deposition
         Other environmental matrices
         Quality-control samples
         Chemical analysis review
         Analytical rerun requests
Physical Measurements
         Meter calibration records
         Atmosphere (meteorlogic)
         Habitat
         Soil
Biological Analyses(Taxonomic)
         Biota (fish)
         Biota (invertebrates)
         Biota (algae)
Specimens 
         Organisms 
         Water
         Sediment cores or Drill-stem cores 
         Drill cuttings 
Spatial Data and Imagery 
         Aerial photographs and satellite imagery 
         Agricultural activities:  crop patterns, chemical, and fertilizer usage
         Census (population, housing, commerce)
         Climate: precip, runoff, temperature 
         Commercial, industrial, and mining activities
         Elevation
         Bedrock geology
         Surficial geology 
         Known environmental-contamination incidents
         Land ownership
         Land use and land cover 
         Physiography and ecoregions
         Point sources (NPDES, TRI, CERCLA, RCRA)
         Public-land survey boundaries (PLSS)
         Sewered areas
         Soils
         Water use
         Wetlands 
         Basin boundaries
         Study boundaries
         Base Maps
Project Management and Related Information
         Annual Study-unit workplans
         Email pertinent to study planning activities
         NAWQA Spinoff studies
         Purpose, procedure, and conclusions of topical quality-assurance studies
         Equipment purchases
         Equipment inventory
         Instrumentation information
         Interagency coordination
Outreach and Related Information
         Contacts data:  Name, Organization, Purpose of contact, etc.
         Liaison committee meeting information
         Presentations
Study Unit Library and Reports
         Reports published by Study Unit
         Graphics and Illustration files for reports
         Data-analysis files for each report
         News clippings
         Bibliography and reports relevant to study unit
Other (The above list serves as a guide. Study units may have addition items to archive.)

• Securing the physical and electronic records from damage, alteration, and misplacement.

• Answering information requests from the National Synthesis Teams and the public.

• Updating the archive with information not included during the current HIP.

• Evaluating the physical integrity of the archive for continued long-term storage.

• Considering (with National coordination) alternative approaches or technologies for preserving the archive.

• Reviewing archive contents during the next HIP, carefully considering which unneeded or obsolete items could be removed from the archive.

Determine fate of the data set

A data set may either be deleted or preserved after the current HIP. The audience(s) of a preserved data set must be determined. Many data sets will be preserved only for the study-unit audience. A data set may be important to the next HIP team because of the data content, effort expended for quality-assurance, and (or) the usefulness of the data arrangement. The National-Synthesis Teams have prepared a list of their data requirements.

Data sets designated for public release should conform to the requirements of the U.S.G.S. Manual section 500.24 governing characteristics of data bases that are released to the public. These public data-base requirements specify as a minimum that: (1) the data must be reviewed, and (2) the data base must be documented. These requirements are independent of the distribution method: traditional paper data report, CD-ROM, floppy disk, or World-Wide Web document.

Public release of digital spatial data sets (GIS data sets) requires some special considerations. Guidelines are available.

Designate a Data Archive Custodian

The data archive custodian will be responsible for the timely documenting, organizing and cataloging of the data set. Periodically update the checklist to show the progress of archiving the data sets.

Document the data set

For any number of reasons, the raw data are not self-explanatory. One must explain where the data originated, what alterations were made during analysis, and what has been learned about the data.

Proper documentation is the key to public release of data. Each data set must be documented to be useful to future users. Although similar in content, the documentation methods for tabular and GIS data differ. Data set documentation is commonly referred to as metadata ("data about data").

Documentation describes the utility of the data set for any future (unknown) analytical purpose. The following kinds of information are helpful.

• Identification and indexing information
  • Narrative description of data
  • Keywords
• Source information
  • Collection purpose
  • Collection procedures
  • Local customization of NAWQA protocols
  • Data-processing procedures
• Modifications
  • Quality-assurance procedures
  • Errors identified and corrective actions taken
  • Interpretive computations
• Known limitations
  • Suitability for particular interpretations
  • Uncorrected errors
  • Anomalies
  • Biases
  • Comparisons with other similar data sources for particular interpretations.
• Coding schemes
  • Tables of code values and the corresponding meanings, such as parameter codes, fixed-value codes, GWSI coded fields, and taxonomic codes
  • Explanation of how coding rules were applied to various data types, especially QC-sample data
• Data Processing and Analysis
  • Programs or code fragments used to perform data analysis
  • Description or documentation of software used
  • Code that depends on proprietary software languages is fine, as long as comments are included

The format of documentation varies, depending upon the kind of data being documented.

• Tabular data set documentation

  A simple text file is used to document tabular data or ascii text files. This file should be attached to the beginning of RDB data files and can be saved as a README file in the same directory as the file that is being documented. A documentation template should be used to ensure that all appropriate topics are included and organized in a consistent manner.

• GIS data set documentation

  The Federal Geographic Data Committee (FGDC) has developed a standard for GIS metadata. All Federal agencies are required by Executive Order 12906 to use the FGDC metadata standard. Additional metadata guidance that may be helpful to USGS users is maintained by the Enterprise GIS office.

• Data-base applications of metadata

  Many data sets internally provide information in the form of codes. Data-base field names are often codes that refer to complete data-element definitions. Other times, the contents of the fields contain coded values.

  Repeated usage of a consistent coding scheme using formal metadata tables avoids continually documenting the same information. Several metadata files can be used directly from the National-Synthesis Data Base.

Organize the data set

Much of the information to be archived is (or will be) computerized, and the tasks related to organizing computerized information differ substantially from the tasks for organizing the more traditional physical records that are the source of much of the computerized data. The tasks associated with physical records (paper, film, specimens, etc.) are discussed first and computerized records are discussed next.

Collating and verifying physical records: Paper records include field forms and notes, lab reports, calibration logs, and any other records important to identification or interpretation of data or samples collected.

All records and artifacts kept in the archive should have all essential information completely documented. Elements such as station identifiers, dates, cross-reference information, and field observations and notes should be complete on all data records. All file folders and containers should be clearly labeled. The archive should include documents such as equipment manuals (if equipment is archived) and maintenance and calibration logs. Sediment samples, any kind of cores, drill cuttings, stable isotope water samples, and any other physical samples must be clearly and completely identified and stored in spill-proof containers.

Certain items associated with data collection and management should be maintained as part of the Study Unit archive. For example, sample collection or measurement equipment unique to NAWQA should be available to duplicate procedures during future High-Intensity Phases. In some cases. it may be sufficient to keep detailed plans for constructing equipment rather than the actual equipment. Some expensive equipment might be shared among study units during the NAWQA cycles, but accountability must be assured so that equipment can rotate back to the High-Intensity Phase study unit.

Entering and formatting computerized data: Data should be entered into NWIS, for all data that fit. Sample coding guidelines are provided by the NAWQA Data Synthesis Team.

• Water-quality Sample-coding Outline

Guidance is also provided for entering and storing data that do not fit into NWIS. These data include station network files, biological data and ancillary GIS data. Station network files are rdb files that are maintained by study units and are retrieved from time to time during national aggregations. Guidance for station-network files is maintained as part of the NAWQA data dictionary. Ecological data that do not fit into NWIS are entered into the NAWQA Biological Transactional Database (BIO-TDB) . All NWIS data, station-network files, and some ancillary GIS data are also maintained in the NAWQA Data Warehouse.

Other Tabular Data: RDB files are recommended for their ease of use and wide compatibility with many computer hardware and software systems. The files are a tab-delimited, ASCII format and are easily created, processed, and exchanged with other software packages, even on different computing platforms. The data documentation can be stored within the RDB file. RDB format is described well by the RDB man page, which has a few literature references about the RDB system, but not about the file format.

The only other format seriously considered was fixed-format ASCII files, but these files: (1) are more work to create and process, and (2) do not have any ready method for internal documentation.

A Documentation Template should be used to ensure that all appropriate topics are included and organized in a consistent manner. A completed documentation template is provided below as an example.

• Long documentation template
• Short documentation template
• An example completed documentation template

Many training materials for learning about RDB files have been prepared. Data in proprietary formats (SAS libraries, STATIT workspaces, Excel spreadsheets, etc.) can and should be converted to a non-proprietary format, because: (1) there is no guarantee that future revisions of proprietary software will be able read the current format, and (2) there is no assurance that the NAWQA Program will have the software or expertise for working with data in a vendor-specific format in the future. Documented software exists for converting data to RDB format from most other data formats used by WRD.

Use the public-domain gzip software to compress large files when appropriate. Gzip recognizes .gz,.tgz,.taz file name extensions, among others (see the man page).

GIS Data: ESRI data sets are complex collections of geographic features, attribute tables, and support files. ESRI software is used by all the NAWQA study units, and future GIS contracts will support the translation of ESRI format data sets. Although guidance has been prepared to describe how to release USGS digital spatial data to the public, few of these guidelines are official.

Publishing GIS data sets as a USGS product or report is a good thing to do from an archival standpoint as the data then can be maintained virtually forever as part of the USGS public web publication server (pubs.water.usgs.gov), included in the National Spatial Data Infrastructure, and shared without restrictions. Many Water Discipline GIS data sets have been published in this way over the years, mostly as OFR and WRI reports, and a few as part of the USGS Digital Data Series.

The following steps should be taken to archive GIS data:

1. Document all ESRI data sets using ArcCatalog. Other GIS formats and other files can also be documented using ArcCatalog or alternatively using FGDC text metadata files. (ArcCatalog stores the data in XML format, which allows it to easily be exported and viewed many other forms, including the traditional FGDC text format.)

   Metadata guidance is available from USGS's Enterprise GIS team

2. Perform a final check to make sure all geographic data sets and related data files are documented and free of inconsistencies and potential problems. An AML program (for Workstation ArcInfo) is available (CHECKDATA.AML) has been written to do a cursory check of an ArcInfo workspace.

   Data stored in the newer ArcGIS geodatabase format can be verified using many different kinds of topology checks (overlapping polygons, etc.) by building a topology on the geodatabase.

3. Save vector and raster data to SDTS format. (Public release and critical data sets only)
   
4. Optionally convert data to additional formats. This may be done to
   1. Ensure that the data will be available in the future (if one format is unreadable, another may work)
   2. Provide data to the public and cooperators using an alternative format.
   3. Provide the data internally within USGS
   For example:
   
   • Convert a point coverage to an RDB file

     Arc: ADDXY cover
     % qwsy (convert INFO table cover.PAT to RDB text format)

   • Convert a grid to a compressed TIFF image

     Arc: GRIDIMAGE grid grid.cmp grid.tif TIFF COMPRESSION

   • Convert a native-format ArcInfo data set (grid or coverage) to a workspace tarfile. Two AMLs (ARCTAR/TARARC) are available that make this as easy as using EXPORT.

   Arc: arctar grid (creates file grid.tgz

An HTML-format metadata report may be generated easily from metadata in XML or text format using either ArcCatalog or the USGS metadata parser mp program. For more details see the metadata link on the USGS Enterprise GIS page.

Text and graphics:

FrameMaker files and Microsoft Word documents or other word-processor proprietary formats should be translated to one or more of the following formats: ASCII text, html, PDF.

This document incorporates the efforts of many people,including:
  Jim Bisese      (jabisese) - SCTX study unit
  Gary Buell      (grbuell)  - ACFB study unit
  Gary Fisher     (gtfisher) - MD-DE-DC mgmt.(also POTO & DLMV study units)
  Kerie Hitt      (kjhitt)   - Nutrients National Synthesis
  Mike McNally    (mmcnally) - Biology National Synthesis
  Curtis Price    (cprice)   - VOC National Synthesis
  Sarah Ryker     (sjryker)  - CCPT study unit
  Jon Scott       (jon)      - National Synthesis (also COKL pilot study unit)
  Mike Yurewicz   (mcyurewi) - Northeast Region Ass't Regional Hyd. for NAWQA

Updates to this document are made by:
  Jana Stewart    (jsstewar) - NAWQA Data Archival Technical Advisor