Clarification of policy for using non-U.S. Geological Survey computer programs in ground-water projects
To: "Albert T Rutledge, Ground Water Specialist, Reston, VA ",
"Robert E Faye, Reg GW Spec, Norcross, GA ",
"Alan W Burns, Reg GW Specialist, Denver, CO ",
"Keith R Prince, Hydrologist, Menlo Park, CA ",
"William M Alley, Chief, OGW, Reston, VA ",
"Velvie E Stockdale, Office Automation Assistant, Reston, VA "
cc: " , WRD Archive File, Reston, VA "
Subject: OGW Technical Memorandum No. 97.01
Mime-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Date: Wed, 02 Oct 1996 15:46:49 -0400
From: "Velvie E Stockdale, Office Automation Assistant, Reston, VA "
In Reply Refer To: October 2, 1996
Mail Stop 411
OFFICE OF GROUND WATER TECHNICAL MEMORANDUM NO. 97.01
Subject: Clarification of policy for using non-U.S. Geological
Survey computer programs in ground-water projects
A number of people have asked questions about the policy for
using non-U.S. Geological Survey (USGS) computer programs
(software) in ground-water projects as stated in Office of
Ground Water (OGW) Technical Memorandum No. 91.04. Some feel
that the memorandum appears to discourage the use of non-USGS
software in ground-water studies, and others are uncertain about
how to comply with the testing requirements. The purpose of this
memorandum is to: (1) clarify that the use of non-USGS ground-
water software is encouraged, (2) further explain why there are
some constraints on such use, and (3) refine the policy on
testing requirements.
In today's computing environment, there are many sources of
software for scientific applications. It makes sense to take
advantage of appropriate software to minimize the effort it
takes to complete a project and to improve the quality of our
work. The only concern with using non-USGS ground-water
software comes from a quality-assurance perspective.
USGS software must be documented in published reports prior to
use in USGS projects. The typical USGS software documentation
report describes methods, implementation details, and user
instructions. Considerable testing usually has been done to
help provide confidence that the software works as designed.
There is no guarantee that similar quality-assurance mechanisms
have been applied to ground-water software that has been
developed outside of USGS. Thus, a user may need to perform
some testing of non-USGS software. This testing is similar in
concept to the process of evaluating a report for correctness
prior to using the methods described in it. In order to refine
the policy on testing requirements, a distinction is made
between documented and undocumented testing.
Documented testing is testing that compares program results to
known results and is documented either in a report that can be
referenced or in a USGS project file. The documentation of
testing must include the input and output from the program and
an evaluation showing that the output is correct. However,
documented testing need be done only once for any particular
program. That is, if adequate testing is contained in a
software manual, a published report, or a USGS project file;
then no further documented testing is required prior to using
the software in a new project.
Documented testing is required if a program is used to perform
significant analysis; the user is charged with applying common
sense to determine if "significant analysis" is performed and
the amount of testing that is required if significant analysis
is performed. Flexibility in determining testing requirements is
allowed in order to avoid setting requirements that could be
inappropriate for many situations. One generalization can be
readily made: all ground-water model programs perform
significant analysis.
As one example of ground-water software that performs
significant analysis, consider an aquifer-test analysis package
that includes several methods of analysis. Further consider
that example problems are included in the software documentation
that demonstrate the ability to correctly perform some, but not
all, the analysis methods that are incorporated. If the
software will be used to perform only the methods that were
tested, then no additional documented testing is required. While
it is expected that a user would run one or more of the supplied
test problems to make sure that the software has been installed
properly, it would not be necessary to save those test results
in the project files. If one of the untested methods will be
used, however, then a documented test should be made to gain
confidence that the method has been implemented as described.
Undocumented testing is any other testing. As already
mentioned, previously documented tests are typically rerun to
gain confidence that a program is installed properly. It is
also customary to perform some undocumented testing of programs
that do not perform significant analysis. It is usually fairly
easy to make some tests to see that the appropriate output is
produced in a variety of situations. An example is checking
that a special-purpose program that prepares model input data
(commonly called a preprocessor) produces files containing the
expected data.
In summary, although one cannot prove that software is error
free, testing provides confidence that the software performs
analysis as stated in the documentation. The policy for testing
is intentionally kept flexible with the recognition that
uncertainty is better than making a rigid policy that is likely
to be inappropriate in many situations. Contact your Regional
Ground-Water Specialist or OGW if you have any questions about
the use of non-USGS software. As always, it is better to deal
with questions early in a project rather than late. In addition
to helping resolve specific questions about what documented
testing is required, OGW will attempt to keep current with which
programs have been successfully tested so that duplication of
effort can be avoided.
/s/ William M. Alley
Chief, Office of Ground Water
Distribution: A, B, S, FO, PO