Clarification of policy for using non-U.S. Geological Survey computer programs in ground-water projects

To: "Albert T Rutledge, Ground Water Specialist, Reston, VA ",
        "Robert E Faye, Reg GW Spec, Norcross, GA ",
        "Alan W Burns, Reg GW Specialist, Denver, CO ",
        "Keith R Prince, Hydrologist, Menlo Park, CA ",
        "William M Alley, Chief, OGW, Reston, VA ",
        "Velvie E Stockdale, Office Automation Assistant, Reston, VA "
cc: " , WRD Archive File, Reston, VA "
Subject: OGW Technical Memorandum No. 97.01
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Date: Wed, 02 Oct 1996 15:46:49 -0400
From: "Velvie E Stockdale, Office Automation Assistant, Reston, VA "

In Reply Refer To:                                      October 2, 1996
Mail Stop 411


Subject:  Clarification of policy for using non-U.S. Geological 
          Survey computer programs in ground-water projects

A number of people have asked questions about the policy for 
using non-U.S. Geological Survey (USGS) computer programs 
(software) in ground-water projects as stated in Office of 
Ground Water (OGW) Technical Memorandum No. 91.04.  Some feel 
that the memorandum appears to discourage the use of non-USGS 
software in ground-water studies, and others are uncertain about 
how to comply with the testing requirements. The purpose of this 
memorandum is to: (1) clarify that the use of non-USGS ground-
water software is encouraged, (2) further explain why there are 
some constraints on such use, and (3) refine the policy on 
testing requirements.

In today's computing environment, there are many sources of 
software for scientific applications.  It makes sense to take 
advantage of appropriate software to minimize the effort it 
takes to complete a project and to improve the quality of our 
work.  The only concern with using non-USGS ground-water 
software comes from a quality-assurance perspective.

USGS software must be documented in published reports prior to 
use in USGS projects.  The typical USGS software documentation 
report describes methods, implementation details, and user 
instructions.  Considerable testing usually has been done to 
help provide confidence that the software works as designed.

There is no guarantee that similar quality-assurance mechanisms 
have been applied to ground-water software that has been 
developed outside of USGS.  Thus, a user may need to perform 
some testing of non-USGS software.  This testing is similar in 
concept to the process of evaluating a report for correctness 
prior to using the methods described in it.  In order to refine 
the policy on testing requirements, a distinction is made 
between documented and undocumented testing.

Documented testing is testing that compares program results to 
known results and is documented either in a report that can be 
referenced or in a USGS project file.  The documentation of 
testing must include the input and output from the program and 
an evaluation showing that the output is correct.  However, 
documented testing need be done only once for any particular 
program.  That is, if adequate testing is contained in a 
software manual, a published report, or a USGS project file; 
then no further documented testing is required prior to using 
the software in a new project.

Documented testing is required if a program is used to perform 
significant analysis; the user is charged with applying common 
sense to determine if "significant analysis" is performed and 
the amount of testing that is required if significant analysis 
is performed. Flexibility in determining testing requirements is 
allowed in order to avoid setting requirements that could be 
inappropriate for many situations.  One generalization can be 
readily made: all ground-water model programs perform 
significant analysis.

As one example of ground-water software that performs 
significant analysis, consider an aquifer-test analysis package 
that includes several methods of analysis.  Further consider 
that example problems are included in the software documentation 
that demonstrate the ability to correctly perform some, but not 
all, the analysis methods that are incorporated.  If the 
software will be used to perform only the methods that were 
tested, then no additional documented testing is required. While 
it is expected that a user would run one or more of the supplied 
test problems to make sure that the software has been installed 
properly, it would not be necessary to save those test results 
in the project files.  If one of the untested methods will be 
used, however, then a documented test should be made to gain 
confidence that the method has been implemented as described.

Undocumented testing is any other testing.  As already 
mentioned, previously documented tests are typically rerun to 
gain confidence that a program is installed properly.  It is 
also customary to perform some undocumented testing of programs 
that do not perform significant analysis.  It is usually fairly 
easy to make some tests to see that the appropriate output is 
produced in a variety of situations.  An example is checking 
that a special-purpose program that prepares model input data 
(commonly called a preprocessor) produces files containing the 
expected data.

In summary, although one cannot prove that software is error 
free, testing provides confidence that the software performs 
analysis as stated in the documentation.  The policy for testing 
is intentionally kept flexible with the recognition that 
uncertainty is better than making a rigid policy that is likely 
to be inappropriate in many situations.  Contact your Regional 
Ground-Water Specialist or OGW if you have any questions about 
the use of non-USGS software.  As always, it is better to deal 
with questions early in a project rather than late.  In addition 
to helping resolve specific questions about what documented 
testing is required, OGW will attempt to keep current with which 
programs have been successfully tested so that duplication of 
effort can be avoided. 

                        /s/ William M. Alley
                        Chief, Office of Ground Water

Distribution: A, B, S, FO, PO