Water Resources Division CPAC Home Page

A Method for Ranking WRD Corporate Software Support Activities

The purpose of this document is to describe a method for ranking software packages deployed nationally by the WRD to determine the relative importance of software-support activities. A pair of indices are described for ranking the importance of centrally provided national: (1) installation support, and (2) user support. The goal of the index computations is to provide a more thorough and objective basis for determining which software packages receive various levels of support.

The two indices do not provide guidance on several other pertinent issues related to software deployment and support. The decisions related to these other issues might be influenced by the results of the index computations, however the indices are not designed to rank packages on the basis of these issues. The additional issues pertain to decisions on whether to: (1) nationally procure (or deploy) any particular package, (2) support a package on more than one hardware/software environment, or (3) develop software to enhance, interface, or integrate the package.

The index score for each package is computed by summing the scores for several factors. The factor scores are determined by characterizing a package for pairs of support issues and finding the appropriate factor score in a factor matrix. After index scores have been determined for all nationally deployed packages, the packages are sorted and ranked by the index scores to determine the relative importance of providing software support among the packages.

Installation-Support Index

Installation support is designed to help system administrators to successfully install a software package for a particular hardware and software environment. Different levels of support can be provided: (1) collation of the software installation files with other software packages to minimize the number software-installation activities, (2) preparation of guidance documentation regarding installation of the software package, (3) preparation of installation scripts to copy (or link) files to appropriate directories, (4) configuration for particular hardware environments, and (5) preparation (or integration) of software user documentation into the on-line "help" system.

The relative need for installation support is determined by the installation-support index, which is computed from several factors. The factors considered are described below.

• EXTERNAL SUPPORT QUALITY:  The quality and availability of external sources for installation support. The support might be provided by the software vendor or another contractual entity. External support is characterized as "Unavailable", "Poor", or "Good".
• VERSION CONTROL:  The importance of version control and patches among the various installations. If data will be exchanged among the users of the software from different installations, it may be important that all users employ the same version of the software (e.g.: SAS). Other times, the output files of the software are version independent (e.g.: emacs). Version control is characterized as: "Important", or "Unimportant".
• INSTALLATION COMPLEXITY:  The number of options and actions required to successfully install the software package. With more decisions and actions required to install a package, it is more likely that benefits will be obtained from additional guidance and prepackaging. Software-installation complexity is characterized as: "Simple", "Average", or "Complex". Support received directly from the vendor should be considered.
• NUMBER OF INSTALLERS:  The number of persons performing the package installation. Greater efficiency and standardization may be obtained by prepackaging software. With more persons puzzling through the installation, it is more likely that benefits will be accrued from some prepackaging. The number of installers is somewhat arbitrarily characterized as: "Less than 25", or "25 or more".

These four factors are paired in factor matrices, for the purpose of determining factor scores, as shown below. Characterizations for each of the four factors are determined, then a pair of factor scores are determined from the DELIVERY and SCOPE factor matrices. The two factor scores are added to compute the installation-support index.

Installation Support Index

DELIVERY FACTOR MATRIX		EXTERNAL SUPPORT QUALITY    Unavailable                   Poor                       Good
VERSION CONTROL	Important ------------------- Not Important	3 ------------------- 2	2 ------------------- 1	1 ------------------- 0

Figure 1.   Delivery-factor matrix for the installation-support index.

SCORE FACTOR MATRIX		INSTALLATION COMPLEXITY      Simple                     Average                   Complex
NUMBER OF INSTALLERS	<25 ------------------- >=25	0 ------------------- 1	1 ------------------- 2	2 ------------------- 3

Figure 2.   Scope-factor matrix for the installation-support index.

User Support Index

A process analogous to the installation-support index is used to prepare a user-support index. However, the levels of potential user support, and the factors determining the need for user support are different. User support can be provided by: (1) on-line help files, (2) tutorials, (3) lists of Frequently Asked Questions (FAQ) and answers, (4) establishment of a newsgroup designated for discussion of the package, (5) maintenance of e-mail mailing lists of package experts (who agree to answer questions), or (6) centrally funded package experts who participate in newsgroups, answer e-mail queries, and provide telephone consultation.

The relative need for user support is determined from user-support index, computed from several factors. The factors considered are described below.

• EXTERNAL SUPPORT QUALITY: The timeliness and usefulness of user support provided by an external entity. Similar to one of the factors for installation support, this factor measures the quality of external support available to package users, and is characterized as "Unavailable", "Poor", or "Good".
• COST RATIO: The relative cost of obtaining the external support versus providing software support using internal expertise. The cost of user support may be significant, and savings may (or may not) be obtained by providing internal expertise. All costs associated with internal support (including salary, benefits, expert training, and overhead) are computed for the expected package usage timeframe, and divided by the costs expected for obtaining external support for the same timeframe. A ratio of internal versus external user-support costs is thus determined. The ratio is characterized as "Half or less", "Between half and double", or "Twice as much or more". However, given the intrinsically hydrologic mission of the organization and the pressure to use scarce manpower to support directly the mission, the external cost is deprecated in determination of the factor score (see figure 3).
• USE COMPLEXITY: The number of options, features, navigation paths, and interfaces of a software package. The more decisions a user must make regarding software use, the more likely it is that the user will require assistance. Software-use complexity is characterized as: "Simple", "Average", or "Complex".
• USER ABILITY: The computing proficiency of the expected user community. The more experienced and capable computer users will need less assistance than other less proficient users. User ability is characterized as: "Below Average", "Average", or "Above Average".
• NUMBER OF USERS: The size of the supported user community. As the number of supported package users increases, the variability of user abilities and occurrences of unusual software problems increases. The number of package users is rather arbitrarily characterized as: "Less than 100", "Between 100 and 500", and "500 or more".
• SCOPE OF FAILURE: The organizational scope of the consequences for the complete failure of the software to function at an installation. Software is redundantly installed in the organization's distributed computing environment. Each of these installations may be configured and maintained in different ways, with some of the installations incorrect in some catastrophic manner. This factor attempts to capture the consequences for the organization of the complete failure of the software package at one installation on the network. Scope of failure is characterized as: "Project", "Office", or "Division".

"Project" consequences occur if only a few projects located at the installation are adversely affected by software failure (such as a ground-water model). "Office" consequences occur if most or all of one office is adversely affected (most of the currently supported software is in this category, e.g.: spreadsheet, GIS, graphics, electronic publishing). "Division" consequences occur if the package failure causes significant consequences beyond the office where the software is installed. Examples of software with potential divisional consequences are: NWIS, AIS, computer security, LRGS, and e-mail.

These six factors are paired in factor matrices, for the purpose of determining factor scores, as shown below. Characterizations for each of the six factors are determined, then factor scores are determined from the MARKET, FRUSTRATION, and IMPACT factor matrices. The three factor scores are added to compute the user-support index.

User Support Index

MARKET FACTOR MATRIX		EXTERNAL SUPPORT QUALITY    Unavailable             Poor               Good
COST RATIO	<= 0.5 ------------- 0.5 < x < 2.0 ------------- >= 2.0	3 -------------- 3 -------------- 3	3 -------------- 2 -------------- 1	2 -------------- 1 -------------- 0

Figure 3.   Market-factor matrix for the user-support index.

FRUSTRATION FACTOR MATRIX		USE COMPLEXITY    Simple             Average               Complex
USER ABILITY	Below Average ------------- Average ------------- Above Average	1 -------------- 0 -------------- 0	2 -------------- 1 -------------- 0	3 -------------- 2 -------------- 1

Figure 4.   Frustration-factor matrix for the user-support index.

IMPACT FACTOR MATRIX		NUMBER OF USERS < 100             100-500               500
SCOPE OF FAILURE	Project ------------- Office ------------- Division ------------- Outside	0 -------------- 1 -------------- 2 -------------- 2	1 -------------- 2 -------------- 3 -------------- 3	2 -------------- 3 -------------- 3 -------------- 3

Figure 5.   Impact-factor matrix for the SA user-support index.

Product Support Index

A process to prepare a product support index is basically a yes or no answer to the type of resource needing support.

• Internal Need to Provide

The internal need to provide is based on whether there is any outside support available and if so, if used.

• Resource

The type of resource for which internal support is or is not needed is in the form of technical support, updates and releases of software, training, and if there is a need for investigation of new tools.

This Scope of Support Factor Matrix is used to compute the product support index.

Product Support Index

SCOPE OF SUPPORT FACTOR MATRIX		INTERNAL NEED TO PROVIDE YES            NO
RESOURCE	Technical ------------- Updates & Release ------------- Training ------------- Investigation of New Tools	1 -------------- 1 -------------- 1 -------------- 1	0 -------------- 0 -------------- 0 -------------- 0	--------------   --------------   --------------

Figure 6.   Scope of support factor matrix for for product-support index.

Limitations to this Method

The index approach to software-support prioritization is not a panacea whereby anyone can objectively and reproducibly determine the relative needs of the Division. The approach has several limitations.

First, continuous factor variables have been simplified as categorical variables. This truncation results in loss of precision in the computed index score. Therefore, judgment must be used when ranking similar or equal index scores. However, software packages with grossly different index scores should probably receive different levels of national support.

Second, different weighting factors within the factor matrices could be assigned. Those provided herein incorporate the author's judgments and biases. However, computation of an index score does require the consistent evaluation of multiple factors when determining how much support is needed for different software packages.

Third, the characterization of the various factors is, at times, a subjective process. The biases resulting from factor characterization can be mitigated by having several individuals evaluate the factors and compute index scores. The factor scores or the index scores can be averaged from multiple responses to provide a more balanced perspective.

A Method for Ranking WRD Corporate Hardware Support Activities

The purpose of this document is to describe a method for ranking hardware systems (platforms) deployed nationally by the WRD to determine the relative importance of hardware-support activities. A pair of indices are described for ranking the importance of centrally provided national: (1) installation support, and (2) user support. The goal of the index computations is to provide a more thorough and objective basis for determining which hardware systems receive various levels of support.

The two indices do not provide guidance on several other pertinent issues related to hardware deployment and support. The decisions related to these other issues might be influenced by the results of the index computations, however the indices are not designed to rank hardware on the basis of these issues. The additional issues pertain to decisions on whether to: (1) nationally procure (or deploy) any particular system, (2) support a system, or (3) develop software to enhance, interface, or integrate the systems.

The index score for each system is computed by summing the scores for several factors. The factor scores are determined by characterizing a package for pairs of support issues and finding the appropriate factor score in a factor matrix. After index scores have been determined for all systems used in WRD, the systems are sorted and ranked by the index scores to determine the relative importance of providing hardware support among the systems.

Installation-Support Index

Installation support is designed to help system administrators to successfully install an operating system for a particular hardware environment. Different levels of support can be provided: (1) collation of the hardware installation files and service patches to minimize the number of installation activities, (2) preparation of guidance documentation regarding installation of the operating software package, (3) preparation of installation scripts to copy (or link) files to appropriate directories, (4) configuration for particular hardware environments, and (5) preparation (or integration) of hardware system user documentation into the on-line "help" system.

The relative need for installation support is determined by the installation-support index, which is computed from several factors. The factors considered are described below.

• EXTERNAL SUPPORT QUALITY:  The quality and availability of external sources for installation support. The support might be provided by the hardware vendor or another contractual entity. External support is characterized as "Unavailable", "Poor", or "Good".
• VERSION CONTROL:  The importance of version control and patches among the various installations. If data will be exchanged among the users of the operating systems from different installations, it may be important that all users employ the same version of operating system. Version control is characterized as:   "Important", or "Unimportant".
• INSTALLATION COMPLEXITY:  The number of options and actions required to successfully install the operating system software package. With more decisions and actions required to install a package, it is more likely that benefits will be obtained from additional guidance and prepackaging operating systems. Software-installation complexity is characterized as: "Simple", "Average", or "Complex". Support received directly from the vendor should be considered.
• NUMBER OF INSTALLERS:  The number of persons performing the package installation. Greater efficiency and standardization may be obtained by prepackaging operating system software. With more persons puzzling through the installation, it is more likely that benefits will be accrued from some prepackaging. The number of installers is somewhat arbitrarily characterized as: "Less than 25", or "25 or more".

These four factors are paired in factor matrices, for the purpose of determining factor scores, as shown below. Characterizations for each of the four factors are determined, then a pair of factor scores are determined from the DELIVERY and SCOPE factor matrices. The two factor scores are added to compute the installation-support index.

Installation-Support Index

DELIVERY FACTOR MATRIX		EXTERNAL SUPPORT QUALITY Unavailable             Poor               Good
VERSION CONTROL	Important ------------- Not Important	3 -------------- 2	2 -------------- 1	1 -------------- 0

Figure 7.   Delivery-factor matrix for the installation-support index.

SCOPE FACTOR MATRIX		INSTALLATION COMPLEXITY Simple             Average               Complex
VERSION CONTROL	< 25 ------------- >= 25	0 -------------- 1	1 -------------- 2	2 -------------- 3

Figure 8.   Scope-factor matrix for the installation-support index.

System Administration/User Support Index

A process analogous to the installation-support index is used to prepare a user-support index. However, the levels of potential System Administrator (SA) user support, and the factors determining the need for user support are different. User support can be provided by: (1) on-line help files, (2) tutorials, (3) lists of Frequently Asked Questions (FAQ) and answers, (4) establishment of a newsgroup designated for discussion of the package, (5) maintenance of e-mail mailing lists of package experts (who agree to answer questions), or (6) centrally funded package experts who participate in newsgroups, answer e-mail queries, and provide telephone consultation.

The relative need for SA user support is determined from user-support index, computed from several factors. The factors considered are described below.

• EXTERNAL SUPPORT QUALITY:  The timeliness and usefulness of SA user support provided by an external entity. Similar to one of the factors for installation support, this factor measures the quality of external support available to package users, and is characterized as "Unavailable", "Poor", or "Good".


• COST RATIO:  The relative cost of obtaining the external support versus providing hardware support using internal expertise. The cost of user support may be significant, and savings may (or may not) be obtained by providing internal expertise. All costs associated with internal support (including salary, benefits, expert training, and overhead) are computed for the expected package usage timeframe, and divided by the costs expected for obtaining external support for the same timeframe. A ratio of internal versus external user-support costs is thus determined. The ratio is characterized as "Half or less", "Between half and double", or "Twice as much or more". However, given the intrinsically hydrologic mission of the organization and the pressure to use scarce manpower to support directly the mission, the external cost is deprecated in determination of the factor score (see figure 9).


• USE COMPLEXITY:  The number of options, features, navigation paths, and interfaces of a software package. The more decisions a user must make regarding software use, the more likely it is that the user will require assistance. Software-use complexity is characterized as: "Simple", "Average", or "Complex".


• USER ABILITY:  The computing proficiency of the expected user community. The more experienced and capable computer users will need less assistance than other less proficient users. User ability is characterized as:   "Below Average", "Average", or "Above Average".


• NUMBER OF USERS:  The size of the supported user community. As the number of supported package users increases, the variability of user abilities and occurrences of unusual software problems increases. The number of package users is rather arbitrarily characterized as: "Less than 100", "Between 100 and 500", and "500 or more".


• SCOPE OF FAILURE:  The organizational scope of the consequences for the complete failure of the software to function at an installation. Software is redundantly installed in the organization's distributed computing environment. Each of these installations may be configured and maintained in different ways, with some of the installations incorrect in some catastrophic manner. This factor attempts to capture the consequences for the organization of the complete failure of the software package at one installation on the network. Scope of failure is characterized as: "Project", "Office", or "Division".

"Project" consequences occur if only a few projects located at the installation are adversely affected by software failure (such as a ground-water model). "Office" consequences occur if most or all of one office is adversely affected (most of the currently supported software is in this category, e.g.: spreadsheet, GIS, graphics, electronic publishing). "Division" consequences occur if the package failure causes significant consequences beyond the office where the software is installed. Examples of software with potential divisional consequences are:   NWIS, AIS, computer security, LRGS, and e-mail.

These six factors are paired in factor matrices, for the purpose of determining factor scores, as shown below. Characterizations for each of the six factors are determined, then factor scores are determined from the MARKET, FRUSTRATION, and IMPACT factor matrices. The three factor scores are added to compute the user-support index.

System Administration/User Support Index

MARKET FACTOR MATRIX		EXTERNAL SUPPORT QUALITY    Unavailable             Poor               Good
COST RATIO	<= 0.5 ------------- 0.5 < x < 2.0 ------------- >= 2.0	3 -------------- 3 -------------- 3	3 -------------- 2 -------------- 1	2 -------------- 1 -------------- 0

Figure 9.   Market-factor matrix for the user-support index.

FRUSTRATION FACTOR MATRIX		USE COMPLEXITY    Simple             Average               Complex
USER ABILITY	Below Average ------------- Average ------------- Above Average	1 -------------- 0 -------------- 0	2 -------------- 1 -------------- 0	3 -------------- 2 -------------- 1

Figure 10.   Frustration-factor matrix for the user-support index.

IMPACT FACTOR MATRIX		NUMBER OF USERS < 100             100-500               500
SCOPE OF FAILURE	Project ------------- Office ------------- Division	0 -------------- 1 -------------- 3	1 -------------- 2 -------------- 3	2 -------------- 2 -------------- 3

Figure 11.   Impact-factor matrix for the SA user-support index.

Product Support Index

A process to prepare a product support index is basically a yes or no answer to the type of resource needing support.

• Vendor Support Quality
Determining the level of vendor support quality for the need.
• Need
Determine the need for cost to buy, maintenance costs, peripherals available, performance, and reliability.
• Internal Need to Provide
The internal need to provide is based on whether there is any outside support available and if so, if used.
• Resource
The type of resource for which internal support is or is not needed is in the form of technical support, updates and releases of software, training, and if there is a need for investigation of new tools.

The Hardware Systems Factor Matrix and the Scope of Support Factor Matrix are used to compute the Product Support Index.

Product Support Index

HARDWARE SYSTEMS FACTOR MATRIX		VENDOR SUPPLY QUALITY LOW             MEDIUM               HIGH
NEED	Cost to Buy ------------- Maint. Costs ------------- Peripherals Available ------------- Performance ------------- Reliability	3 -------------- 3 -------------- 3 -------------- 3 -------------- 3	2 -------------- 2 -------------- 2 -------------- 2 -------------- 2	1 -------------- 1 -------------- 1 -------------- 1 -------------- 1

Figure 12.   Hardware systems factor matrix for product-support index.




SCOPE OF SUPPORT FACTOR MATRIX		INTERNAL NEED TO PROVIDE YES             NO
RESOURCE	Technical ------------- Updates & Release ------------- Training ------------- Investigation of new Tools	1 -------------- 1 -------------- 1 -------------- 1	0 -------------- 0 -------------- 0 -------------- 0	--------------   --------------   --------------

Figure 13.   Hardware systems factor matrix for product-support index.

Limitations To This Method

The index approach to hardware systems prioritization is not a panacea whereby anyone can objectively and reproducibly determine the relative needs of the Division. The approach has several limitations.

First, continuous factor variables have been simplified as categorical variables. This truncation results in loss of precision in the computed index score. Therefore, judgment must be used when ranking similar or equal index scores. However, hardware systems with grossly different index scores should probably receive different levels of national support.

Second, different weighting factors within the factor matrices could be assigned. Those provided herein incorporate the authors' judgments and biases. However, computation of an index score does require the consistent evaluation of multiple factors when determining how much support is needed for different hardware systems.

Third, the characterization of the various factors is, at times, a subjective process. The biases resulting from factor characterization can be mitigated by having several individuals evaluate the factors and compute index scores. The factor scores or the index scores can be averaged from multiple responses to provide a more balanced perspective.

FUTCOM3 Software Table (attached)

http://wwwrcamnl.wr.usgs.gov/uo/f3/sftw_tbl.html