action-allord.html Annual Water-Data Reports issues USGS Water Resources
INTERNAL WRD ACCESS ONLY

Annual Water-Data Reports Issues


*Probable items of interest to CPAC
This document contains notes on all aspects of the Annual Water-Data Reports. The portions of possible interest to CPAC are marked. Preparing a description of all elements allows this to be developed into a work plan and puts the entire effort into perspective.

Overall coordination

The responsibility for the data reports coordination was assigned to Chuck Novak until he retired in 1995. At that time his full-time duties were distributed to others within the Office of Water Information. No one person has since given the reports the dedicated attention to detail that Chuck provided. It is my opinion that we have gotten away with this for a number of water years but the decay is becoming apparent and that it is time to focus on all aspects of these reports and update materials and procedures.

Different efforts are underway in the last half of FY98 to resolve some of the more pressing issues. Beginning, October 1, 1998 staff from the Office of Information will again have time dedicated to this series as part of their job descriptions. The anticipated level of effort will be approximately 0.4 person-years in contrast to the approximately 1.5 person years assigned to this project prior to 1995. This does not include outside, contracted work.

Guidelines
The major reference material available is "WRD Data Report Preparation Guidelines, 1985 Edition" and WRD Memorandum 92.-4, "Revised Annual Water-Data Report Format". While much of this material is current, it needs to be reviewed for relevancy and accuracy. Most of the material included in these documents are examples and it is not limited to the guidelines themselves. This extraneous material will be removed.

Proposed solution: A web site for the Office of Information, Publications Management and Support Program will be constructed and a major section will be dedicated to the data reports. This web site will have current guidelines posted and available for the Districts by October 1, 1998. This is the beginning of the next water year and will be the first date when records will be processed for the next reports. These guidelines will be initially reviewed by Greg Allord (OI) and Barbara Balthrop (TN) and will be sent to the Discipline Offices for comment.

CONTENT - INTRODUCTORY MATERIALS
This section of the report is the narrative section describing the data. The listing below are the typical portions of this material.

    List of surface-water stations, in downstream order
    List of ground-water wells, by county
    List of discontinued surface-water discharge or stage-only stations
    List of discontinued surface-water-quality stations
    Introduction
    Cooperation
    Summary of Hydrologic conditions
    * Definition of terms
    Downstream order and station number
    Numbering system for wells and miscellaneous sites
    * Special networks and Programs
    Explanation of stage- and water-discharge records
    Explanation of water-quality records
    Explanation of ground-water level records
    Publications on Techniques of Water-Resource Investigations
    Gaging-station records (this is the major portion of the book)
    Ground-water records
    Index
* The definition of terms needs a complete review. Some errors from the initial list prepared years ago have been reported. These terms will be updated and reviewed by the appropriate Discipline Office.

* The special networks and programs also needs to be up-dated. WATSTORE has been removed but the other programs descriptions need to be reviewed yearly, beginning this year. The proposed owner and program descriptions are:

    Office of Water Quality
    • Hydrologic Benchmark Network
    • National Stream-Quality Accounting Network (NASQAN)
    • The National Atmospheric Deposition Program/National Trends Network

    NAWQA

    • The National Water-Quality Assessment Program
Other program descriptions will be included as they are identified. For instance, should there be a NWIS description?

I would like to thank Kathy Fitzgerald for her assistance with these each year. I believe it is necessary to have the Discipline Office formally own these program descriptions. This is not an effort that will take more than a day each year but needs to be done on a continual and planned basis.

SURFACE-WATER DISCHARGE RECORDS
While a number of efforts have begun under NWIS2 and several Districts no templates were ever finalized and released to all Districts. The initiatives that I am aware of were the templates team for NWIS2 (coordinated by Terry Reinitz, DIS) and recent programming by David Yancey, California District. The NWIS2 effort was not completed and the team did not finalize their work and the programming in California is described as being very sophisticated and goes all the way through to generating HTML code for discharge data. Merritt Blalock, on a volunteer basis and mostly on his own time, has also taken the initiative to write scripts.

A meeting was held with the Texas District Data Section on April 21, 1998 to discuss data report publications options. This meeting was initiated by Susan Gandra because of the problems Texas had for their four volumes in water year 1997 – the first year they attempted to use FrameMaker. Attending from Texas were: Susan Gandra, Ruth Jones, Willard Gibbons, Jim Bisese, Peter Bush, and Gloria Smith. Greg Allord and Merritt Blalock represented OI.

The following are notes from this meeting.

Texas in water year 1997 made their first attempt at using FrameMaker for producing the annual water-data report. The Data Section took the lead of processing the data and running FrameMaker. The Publications Group was not used and the Data Section now feels this was a mistake as they would have benefitted from the Pubs Section experience.

A major objective was to decide what is the best tool to use. This could be FrameMaker or it could be another choice. Texas has prior experience with having programs for tabling the data in ASCII form and this was a viable method but would require rewriting all the programs.

Susan queried many District regarding their current methods for preparing the data reports. She has an extensive collection of notes. Her conclusion is that:

  • here is not a focused effort to support the data report
  • CA is only district h\that did a good job of gathering all existing templates
  • NWIS2 had a templates group (Terry Reinitz, Lynn Osburn, Patsy Mixon) but this died
  • Terry has tried to pull this together several times but does not have the time available
  • consistency across the country in regards to naming conventions or scripts
  • There have been no computer programming resources available to the data sections to solve this issue. Texas would like support of this kind


    DECIDED

    The Office of Information will support the least common denominator approach to supporting programming for the annual water data reports. At this time I do not feel that it is wise to support a very sophisticated approach for data extraction for NWIS, tabling, inclusion of graphics, and Web publishing. Something is needed now and must be available by October 1, 1998 for the District to use. I am defining least common denominator as a set of scripts that meets the minimum needs of producing material for the data reports. If a District wants to make the material more sophisticated they can alter the source code to their own needs. The programming should allow minimal maintenance and basic training for installation.

    The web release of this information needs to be a factor in this process as this is an eye to the future. However, there is no foreseeable end to the need for printed copies. The WWW is needed as an alternative distribution mechanism.

    There was strong support for the idea of processing data in Ascii and not having it in a binary form.

    FrameMaker, version 4, is the targeted release level. Most District data sections will be running these scripts on Data General systems and it is important to support version 4.0. The 4.0 material can be imported to release 5.5 because of the structure of the files and the font selected.

    The introductory pages are a completely different issue from the data tables. The front could arguably to be an interpretative report. I propose that all aspects of this part of the book be done in FrameMaker.

    Merritt demonstrated his scripts and there was an extensive discussion about specific aspects of his programming. He is programming in AWK. After a discussion about AWK vs PERL vs other options is was decided that AWK is fine as this can be deployed widely and meets the goal of least common denominator. Ascii allowed parsing of the tables and formatting for publication.

    The scripts extract information from NWIS through a menu-driven system and creates Ascii files that are processed to FrameMaker MIF format. The final files have correct typographic controls (superscripts, etc) and have any plot files from G2 included in the MIF files.

    One file per station is created. These individual files can be used by FrameMaker's book building function to assemble and print the entire report.

    This is a very abbreviated description of the scripts Merritt has written. Examples are attached at the end of this document. Refer to the appendix for a listing and samples of the materials produced by the script.

POSSIBLE ENHANCEMENTS:
The scripts read existing files for the station description information (location, drainage area, period of record, gage, remarks, extremes) and these header files are merged with the tabled data to produce the finished page. I believe the local, duplicate files can create inconsistencies and also requires manually updating each year. A part of the NWIS design should require a station manuscript be created and stored in NWIS. This allows the station manuscript to be available when a station comes on-line for WWW display of data. NWIS storage of these files eliminates many individual files that have no naming consistency from District to District.

A flag could be added to NWIS records indicating a station is a DATA REPORT station and this flag can be turned on to show that the data is final. A final data flag will print the data and otherwise the data cannot be printed from NWIS.

Jim offered the NWIS group can assist with some of this programming. It was decided that Merritt will continue his programming efforts and that the NWIS team in Illinois will be kept informed of these scripts. Any work to pull these scripts into NWIS, to run internally, will benefit the process. However, I do not want to derail either Merritt and NWIS and decided to only support Merritt's programming at this time. John Briggs is aware of these scripts and a full demonstration will be given when they are ready.

    PLANS:

    Programming - Merritt is finished programming the SW records retrievals. He is finishing the documentation (12 pages written to date) and wants time to do an editing and revision cycle. Estimated completion date for finished documentation is June 30, 1998. At that time Merritt will begin writing scripts and documentation for QW data. Estimated completion date for Q. portion is September 21, 1998. It was decided that the GW tabling may not have to be done at all and will be done, if required, in FY99. OI will provide support in editing and formatting documentation.

    FrameMaker book building - After the individual station files have been created, a procedure for use FrameMaker to generate the entire book, including page numbers and index, is needed. Texas District (Peter Bush and Gloria Smith) has been funded by OI to write any scripts and documentation necessary. A review will be requested from DIS staff (Terry Reinitz).

    Beta testing - Texas District (Susan Gandra and Ruth Jones) volunteered to do beta testing using the SW scripts. Testing is planned to begin about the week of May 18, 1998.

    Maryland and Maine Districts have also seen alpha versions of the scripts and have expressed an interest in testing these scripts. They will be officially asked to do beta testing when the SW documentation is complete. One other District, possibly Wisconsin, may also be asked to evaluate the scripts.

    Implementation - The software will be announced and be available for use by October 1, 1998. The documentation will cover the organization, installation and use. A tar file will be available for download to local servers. The important issue will be getting the word of these scripts to the staff actually preparing the data reports.

    It is possible that some mini-sessions may be requested on a District or Regional level. OI will need to continue to support Merritt's travel (the one trip to Texas was paid so far) for him to train users on these scripts. This is the fallback position if offices want training instead of using the documentation. Texas District should also be considered as a source of trainers if more are required. Funding will be needed to support whatever staff takes on this assignment.

    Follow-up - In the longer run, Merritt will provide support for bug fixes, I will coordinate any suggested changes or enhancements, and the NWIS team in Illinois may elect to incorporate some of this functionality into NWIS.

    By way of these notes, I will also let the previous NWIS2 templates team know what we are planning. This information will be sent to Terry Reinitz for her to distribute.

RELEASE OF REPORT DATA AND MEDIA TYPES:
The printing contract with Gray Graphics, Philadelphia was canceled effective March 30, 1998. A replacement contract for one year, with 4 option years, was placed with Braceland Brothers, Atlanta, effective May 1, 1998. We have had extensive, previous experience with Braceland and I am glad that they are the printing contractor. We should not have the delivery and quality problems that we had with the previous contractor. In the future the need of printed copies needs to be evaluated. I would anticipate that the number of printed copies will eventually fall to the level where print-on-demand is a viable option. The existence of the scripts or the following paragraph about CD-ROM also adds to this viability.

The Illinois District has an approved project to eventually replace the printed copies with a CD-ROM. This is a three year project to develop the ability to serve current and historical data on one CD-ROM. This project is to design and program the a multi-media CD-ROM to improve the availability of USGS data. Illinois' major cooperator would like this format to ease the need of making repeated calls for specific data. Data retrieval options and basic statistical and graphic packages will eventually be included on the CD. Programming will be done by University of Illinois Computer Science Department students. The CD will include the introductory material, the statistical and graphic software, and the data itself. The long-term plan is to be the first paperless annual data report.

I see the Illinois project tying in with the release of the information the WWW. Again, the scripts being developed now could be used for preparing extracting and tabling the information. A major point will be that the scripts show how the data should be extracted and presented. The NWIS web staff can work on the process of presenting this data on the web. I propose that any WWW development be coordinated between the Pubs and NWIS units of OI. It is enirely possible that will dramatically alter the requirement for printed copies.

OTHER ITEMS TO BE ADDED BY CPAC:
U.S. Department of the Interior, U.S. Geological Survey
Maintainer: CPAC Webmaster Team
Last modified: Thu May 28 14:14:36 1998