Ask the Scholar
Document scope · 1 page
Scholar
Ask about this object, its catalog metadata, its source description, or the page inventory.
For page-specific OCR and visual context, open one of the page chats.
Scholar Source Context
Document identity
localId
351086345
label
"A Benefit-Cost Approach to the Prioritization of Rehabilitation Research" [1980]
core
doc
dtoType
document
citationUrl
pageCount
1
Source metadata
id
351086345
contentType
document
title
"A Benefit-Cost Approach to the Prioritization of Rehabilitation Research" [1980]
citationUrl
collections
Lex Frieden Collection: Records on Disability Rights
Printed Materials
imageCount
1
hasImages
yes
source
import
hasTranscription
no
Source extras
naId
351086345
levelOfDescription
fileUnit
recordType
description
ocrSource
nara-archive
Single page context
seq
1
pageIndex
0
type
document
mediaId
5fd8e595df97dd81
ocrText
Originally Processed With FOIA(s):
FOIA Number:
S
S
FOIA
MARKER
This is not a textual record. This is used as an
administrative marker by the George Bush Presidential
Library Staff.
Record Group/Collection:
Donated Historical Materials
Collection/Office of Origin: Frieden, Lex, Collection
Series:
Printed Materials
Subseries:
Papers/Books
OA/ID Number:
52110
Folder ID Number:
52110-009
Folder Title:
"A Benefit-Cost Approach to the Prioritization of Rehabilitation Research" [1980]
Stack:
Row:
Section:
Shelf:
Position:
A BENEFIT-COST APPROACH TO THE
PRIORITIZATION OF REHABILITATION RESEARCH
Report
Grant HEW 12-P-59036/6-03
The Institute for Rehabilitation and Research
Baylor College of Medicine
September 25, 1980
Houston, Texas
A BENEFIT-COST APPROACH TO THE
PRIORITIZATION OF REHABILITATION RESEARCH
Report
Grant HEW 12-P-59036/6-03
Prepared by:
David Cardús, M.D., Project Director
Marcus J. Fuhrer, Ph.D.
Robert M. Thrall, Ph.D.
With the Assistance of:
John H. Boynton, M.S.
David S. Bunch, M.S.
Sarah Taylor, M.P.H.
September 25, 1980
Houston, Texas
CONTENTS
Page
PREFACE - EXECUTIVE SUMMARY
1
PART I - BACKGROUND AND THEORY
I.1 A PROCESS FOR ALLOCATING REHABILITATION RESEARCH FUNDS
.....
8
I.1.1 Formulation and Prioritization of Rehabilitation Issues/Problems
I.1.2 Delineation and Analysis of Researchable Components
I.1.3 Development of a Research Agenda
I.1.4 Formulation of Project Concepts
I.1.5 Prioritization of Project Concepts
I.1.6 Choice of Project Formulation
I.1.7 Prioritization of Competing Proposals and Selection for Funding
I.1.8 Role of Budget Constraints
I.2 THE BENEFIT-COST MODEL
16
I.3 BENEFITS AND COSTS OF REHABILITATION RESEARCH
18
I.3.1 Identification of Benefits
I.3.2 Identification of Costs
I.4 OPERATIONALIZING THE TERMS OF THE BENEFIT-COST MODEL
23
I.4.1 Assessment of Benefits
I.4.2 Target Population
I.4.3 Probability of Success
I.4.4 Probability of Utilization
I.4.5 Introducing Costs into the Benefit-Cost Ratio
I.5 IMPLEMENTING THE MODEL
30
I.5.1 The Monetary Submodel
I.5.2 The Non-Monetary Submodel
I.5.3 Combining Monetary and Non-Monetary Benefits
I.5.4 Using the Benefit-Cost Ratio to Prioritize Projects
I.6 COMPUTER PROGRAM FOR MODEL APPLICATIONS
44
PART II - PROCEDURES
II.1 ESTIMATION OF TARGET POPULATION SIZE
45
II.2 ESTIMATION OF MONETARY BENEFITS FOR INDIVIDUALS
50
II.3 SCALING OF NON-MONETARY BENEFIT DIMENSIONS
53
II.3.1 Procedure for Quantifying Individual Non-Monetary Benefits
II.3.2 Procedure for Quantifying Non-Individual Non-Monetary Benefits
II.4 ESTIMATION OF THE PROBABILITY OF SUCCESS
58
II.5 ESTIMATION OF THE PROBABILITY OF UTILIZATION
60
II.6 WEIGHTING BENEFIT DIMENSIONS
63
ii
CONTENTS (Continued)
PART II - PROCEDURES (Cont.)
Page
II.7 ARPCOM USER'S GUIDE
66
II.7.1 Accessing ARPCOM
II.7.2 ARPCOM Instructions
II.7.3 Using ARPCOM
II.7.4 Data Inputting
II.7.5 Conclusions
GLOSSARY
76
REFERENCE LIST
77
APPENDIX 1 - THE ROLE OF THE BENEFIT-COST RATIO IN THE SELECTION OF
ALTERNATIVE COURSES OF ACTION
78
1.1 The Expected net Benefit of a Research Proposal
1.2 Benefit-Cost Analysis which Considers Constraints
1.3 The Role of the Denominator in the Benefit-Cost Ratio
1.4 Illustrative Numerical Examples
APPENDIX 2 - BENEFIT CLUSTERING
87
iii
PREFACE
EXECUTIVE SUMMARY
:
It is virtually inevitable that the National Institute for Handi-
capped Research (NIHR) will be confronted with far more worthwhile pro-
posals for rehabilitation research than can be funded. This situation
will underscore the need for an approach to prioritizing proposed re-
search that is reasonable, systematic, timely, relatively economical
and open to the scrutiny of the rehabilitation community, the Admini-
stration, and the Congress.
This executive summary serves to introduce the scope of this
report and describes briefly an approach to the benefit-cost analysis
of proposed rehabilitation research, which is the primary thrust of the
main body of the report. Particular emphasis is directed to the evaluation
of research project concepts which, at a minimum, are understood as speci-
fying: (1) a designated rehabilitation issue or problem, (2) problem-
related gaps in knowledge, (3) an overall research strategy, (4) applicable
research methodologies, and (5) estimates of benefits and costs. It is
expected that research project concepts of this kind are likely to loom
large in the formulation and implementation of NIHR's long-range plan
for rehabilitation research.
In 1971, a team of multi-disciplinary professionals was formed and
funded by the Social and Rehabilitation Service to conduct a study entitled
"Analytic Aids for Research Proposal Selection (AARPS)." The intent of
this group, which includes specialists in rehabilitation research and
operations research, was to construct a mathematical evaluation model that
1
would take into account both the monetary and the non-monetary benefits of
rehabilitation research. This goal made it necessary to develop means of
combining monetary and non-monetary benefits in the mathematical analysis,
a goal that has been successfully achieved in the AARPS benefit-cost model.
This executive summary serves therefore as an introduction to the
model and to its specific administrative applications. A computer program
has been proposed and partially developed to perform the routine calculations
required by the model so as to facilitate its use. This program is more
fully described in a later section of this report.
The AARPS Benefit-Cost Model
Following a lengthy theoretical analysis and through several math-
ematical developments, the following expression for benefits which might be
expected from a particular research project was constructed:
The probability of success, Ps, is defined as the likelihood that the objec-
tives of the project will be achieved, and this parameter varies between
zero and one. The probability of utilization, PU, is the likelihood that one
or more individual benefits will be utilized by those in the target popula-
tion (N), either collectively or individually. The parameter N represents
the size of the target population, which includes all of those individuals
(or defineable groups, institutions, or other single units) which are seen
to benefit directly from the project. The individual expected net benefit,
B₁, is a measure of those advantages or conditional improvements which are
expected to accrue to the individuals or units within the specified target
population, less the costs of achieving those benefits. The indirect benefit,
Bs, provides an accounting of those benefits and costs that depend upon
project success but are not proportional to the size of the target popu-
2
lation. Similarly, the indirect benefit, BF, accounts for those benefits
and costs which occur irrespective of project success or any target group.
Rehabilitation professionals were surveyed systematically to deter-
mine the possible benefits to be used in the model. The result was a list
of 243 separate benefit factors which was subsequently reduced in number
to 46 and then to 18 in an attempt to achieve a manageable number for
the benefit-cost model. These 18 benefit factors were ultimately trans-
formed into seven benefit dimensions, which are as follows:
Monetary Benefits
All benefits that can be represented in monetary units which accrue to
either an individual, a group of individuals, one or more institutions,
or society as a whole.
Enhanced Quality of Services
Those benefits exemplified by improved access to services, improved
individualization of services, improved coordination among services
or improved continuity of services.
Improved Individual Client Outcomes
Those benefits exemplified by minimization of functional limitations
and personal disability, the encouragement of greater individual social
participation, or improved vocational status and material well-being.
Improved Administrative Bases for Service Provision
Those benefits exemplified by improved management of information systems
yielding timely and relevant administrative decision support, identifying
operational constraints on operations and implementation, establishing
more explicit procedures for program prioritization, and more effective
sequencing of program development.
Improved Policy Bases for Rehabilitation
Those benefits exemplified by improved legislative impact and coordination
of government entities; the development and communication of policies,
plans, and procedures; and the facilitation of societal change.
Indirect Benefits, Given Project Success
Those benefits exemplified by expanded knowledge bases, the identification
of new areas of research, and the spinoffs of technology and/or procedures.
Indirect Benefits, Regardless of Project Success
Those benefits exemplified by an enhanced public awareness of an issue
or problem or a sustained effort focussed on a research problem.
3
To perform a benefit-cost analysis, all research project costs must be
accounted for in using the above benefit equation. Two quantities can be defined
to show how costs enter the evaluation process. The first quantity is the
"expected net benefit," B, which results when all costs necessary to realize
a particular benefit have been subtracted from the value of that benefit.
All of the benefit terms (B₁, Bs, and BF) in the benefit expression are
expected net benefits. An example of this quantity would be an environmental
control device that might ultimately save the quadriplegic consumer $1000
but costs $200 to purchase and maintain. The expected net benefit would be $800.
The second quantity is the "benefit-cost ratio," B/CR, and is the
value of the total expected net benefits, B, divided by the first-year cost
of research, CR. The cost of research is included, of course, in the total
cost necessary to realize the total expected benefits (i.e. C*). Costs,
therefore, can be seen to figure into the benefit-cost analysis as follows:
total expected net benefits, B, are equal to total expected benefits, B*,
less total expected benefit costs, C*, or simply B = B* - C*; and the
benefit-cost ratio, B/CR, becomes the total expected net benefits divided
by the first-year cost of research.
Features of the Benefit-Cost Model
Several unique features of the AARPS model are available when it is
used for administrative planning and evaluation. The most significant is the
ability to combine both monetary and non-monetary benefits. In the term Ps,
full consideration is taken of the likelihood of failure to which research
is vulnerable. For such assessments, peer-review is deemed to be critical.
Both the direct and indirect benefits considered by the model are viewed as
being contingent upon the estimate for probability of success.
4
When several different target populations are expected as beneficiaries
of a project, it is also possible that the benefits differ from one population
to another. With the separate term (N) in the model, the capability exists for
considering these population differences, i.e., each separate benefit can be
applied differentially to each specific population.
In the same context, not all members of a target population will use, or
take advantage of, each benefit to the same extent. Provision for the probabil-
ity of utilization, PU, accounts for this estimated variation in usage and
is the remaining parameter in the individual benefit product (PUNB₁).
Historically, most approaches to prioritizing proposed research have
only considered benefits accruing to persons to whom the research was
directed. It was apparent to the AARPS group, however, that many research
project concepts may benefit others besides the obvious beneficiaries, so the
Bs term was added to the model and made contingent upon project success.
The final net benefit, BF, was included in the model to account for
benefits and costs that are independent of project success. An example
is the cost of the research itself, since this is a "negative benefit,"
or benefit cost, which must be paid regardless of the success of the project.
For an administratively more meaningful analysis of benefits and costs,
certain "weighting" and "scaling" factors can be applied to one or more
terms of the model. These apply not only to estimates for the probabilities
and target populations, but also importantly to the combination of monetary
and non-monetary benefits. The importance of the weighting factors is that
the values and perspectives of decision-makers can be taken into account.
These, and other mathematical concepts related to using the model, are ex-
plained in much greater detail in the remainder of this document.
5
Administrative Aids to Using the Model
A parallel activity has been the generation of a basic computer program
which has the potential of being developed into a more sophisticated computer
program that employs all possible features of the benefit-cost model. The
present computer program ARPCOM, together with an available user's guide,
can be used to compute the total expected net benefits according to the seven
benefit dimensions. In other words, ARPCOM provides for the combination of
monetary and non-monetary benefits. In addition, research costs are intro-
duced and corresponding benefit-cost ratios are calculated. The framework
for a more elaborate computer program, ARPSIN, is only envisioned at this
writing, but if developed later by the AARPS group, it will become a supplement
to this report. In conducting this project, the AARPS group has stressed the
importance of first achieving a fully developed, idealized conceptual model
and then considering the kinds of adaptations necessary for it to be introduced
into administrative practice. This approach contrasts with developmental
efforts that embrace practical expedience at the onset. The virtue of the AARPS
group's approach is that, as adaptations are made to meet practical constraints,
their significance, both conceptually and operationally, can be better assessed.
Some of these adaptations are discussed in the body of this report; others
have been envisioned by the AARPS group and could be refined after future
interactions with the NIHR staff.
Conclusions
As an aid to the management task of prioritizing research project
concepts and competing research project proposals, the AARPS group has developed
a mathematical benefit-cost model that features multidimensional benefit
parameters and has the capability for considering both monetary and non-monetary
6
benefits. The model provides for the consideratation of probabilities of project
success and benefit-utilization, target populations and subpopulations for
each benefit, and for direct and indirect benefits. The model also allows
for benefits which may be realized even if the project is unsuccessful.
Properly applied, the model can be a powerful tool for the administrator who
must consider a multitude of worthwhile research projects, all competing
for limited available funds.
7
PART I
BACKGROUND AND THEORY
I.1 A PROCESS FOR ALLOCATING REHABILITATION RESEARCH FUNDS
It is generally recognized that a benefit-cost (B-C) analysis is a
useful adjunct to decision making.
A prerequisite to the application of benefit-cost analysis is the defin-
ition of decisions to be made. The B-C model developed. by the AARPS group
was intended to support the rehabilitation research management process and
in particular to provide workable concepts and practical procedures for
evaluating proposed rehabilitation research. Implicit in this work is an
overall approach to research management which has grown out of a close working
relationship with the RSA research management staff. The AARPS group believes
that the proposed benefit-cost model and the management concepts to which it
relates apply equally well to the National Institute of Handicapped Research
(NIHR).
In the following sections, the prioritization of rehabilitation research
is considered from two key perspectives:
(1) The array of management tasks that are inherent in the process of
allocating research funds, regardless of the administrative mechanisms used,
(2) The approach to research management which has been implicit in the
AARPS work to date and which, we believe, bears upon NIHR's approach to the
allocation of research funds.
The component tasks of the research allocation process are discussed
separately in the following sections and are duplicated in their sequential
order in figure 1. A glossary of certain non-standard terms used in this
report is presented at the conclusion of Part II.
8
FORMULATE
PRIORITIZE
ANALYZE/DELINEATE
REHABILITATION
REHABILITATION
RESEARCHABLE
ISSUES/PROBLEMS
ISSUES/PROBLEMS
COMPONENTS
DEVELOP A
FORMULATE
PRIORITIZE
CHOOSE
RESEARCH
RESEARCH
RESEARCH
PROJECT
AGENDA
PROJECT CONCEPTS
PROJECT CONCEPTS
FORMULATIONS
BUDGET
CONSTRAINTS
PRIORITIZE
SELECT
ALLOCATE/
COMPETING
OPTIMAL
AUTHORIZE
RESEARCH
PROJECT(S)
RESEARCH
PROPOSALS
FOR FUNDING
FUNDS
FIGURE 1. - PROCESS FOR ALLOCATING REHABILITATION RESEARCH FUNDS
9
I.1.1 Formulation and Prioritization of Rehabilitation Issues/Problems
By legislative mandate, the research to be supported by the NIHR is
to be strongly mission oriented. According to its own statement of functions,
that mission is to " improve the lives of people of all ages with physical
and mental handicaps, especially the severely disabled." The legislation
is mute, however, regarding the full range of specific issues and problems
that might be resolved by research in order to improve the lives of
handicapped people and to support the rehabilitation service system.
The long-range plan for rehabilitation research doubtlessly will
provide the principal basis for issue identification. Hopefully, it
will cover the domain of rehabilitation issues in a relatively balanced
and comprehensive manner, indicating what are the problems and unmet needs.
No less important will be the effort to specify the relative importance
and urgency of the problems that are surfaced.
I.1.2 Delineation and Analysis of Researchable Components
Some of the concerns and issues that are identified will not be
solvable by the provision of additional knowledge but rather by the
redeployment of funds, new policy decisions, or the intensification of
program efforts. Consequently, each high priority issue or problem must
be submitted to competent analysis to identify components reflecting
knowledge gaps that may be filled by systematic investigation. Some
issues may be characterized by one predominant gap; others may consist
of several different knowledge gaps. When performed systematically, this
kind of assessment frequently takes the form of a state-of-the-art review
specifying what needs to be known, what is known, and what is likely
to be knowable given current knowledge and the existing investigative
capabilities. As a number of such reviews becomes available, they can be
10
studied together to identify research possibilities common to two or
more issues. Such possibilities for research are more attractive candi-
dates for development than those pertaining to single issues.
I.1.3 Development of a Research Agenda
If performed adequately, the state-of-the-art review provides
the critical groundwork for constructing a research agenda to acquire
the needed knowledge. The agenda provides some notion of the order in
which sub-problems must be addressed, i.e., which problems presuppose
the solutions of others. An attempt is made to specify the alternative
outcomes of component studies and to consider their implications for
the design of succeeding studies. At each juncture an effort is made to
specify the kinds of investigative resources that are required. The
preparation of research agendas and their critical review for soundness
and promise by duly qualified experts were not visible parts of the RSA
approach to rehabilitation research management. Efforts by NIHR to
establish a capability for research agenda planning should include an
attempt to become familiar with similar efforts in other federal agencies
(NCI, NET and NIMH in particular), and means should be explored of
coordinating efforts of NIHR staff members and rehabilitation researchers
throughout the country in agenda construction.
I.1.4 Formulation of Project Concepts
Having broadly identified how investigation is to proceed to
fill the knowledge gap associated with a high priority issue or concern,
it is possible to plan a specific research effort that implements the
agenda. The resulting project concept may take a variety of forms,
but at a minimum, it is likely to identify the rehabilitation issue or
11
problem to which it pertains, highlight an issue-related gap in knowledge,
articulate an overall research strategy, characterize applicable research meth-
odologies, describe how the knowledge to be gained can be disseminated and its
use encouraged, and supply a cost estimate for conducting the research.
I.1.5 Prioritization of Project Concepts
The aggregate costs of the total array of project concepts
will inevitably surpass the funds available to NIHR. The difficult
decision must be faced of selecting a subset of project concepts that have
two characteristics: (1) they are particularly meritorius, and (2) their
aggregate costs conform to the total funds available. The AARPS benefit-
cost model has been developed to facilitate this selection process because
of the key role it plays in the overall management strategy. It should
be noted, however, that only modest revisions of the operational procedures
are required to assess investigator-initiated research proposals as well.
I.1.6 Choice of Project Formulation
Project concepts which have been chosen for funding must be
developed further in the form of specific proposals for research. The
proposal may be prepared by the agency staff in the form of a contract
for which bidders will be sought nationally or in the form of an agency-
solicited request for proposals addressed to a particular project concept.
Some attention also must be directed to investigator-initiated
proposals because of the prominent role these play in the funding allocation
strategy of other agencies, including NIH and NSF. Were NIHR fully committed
to the management strategy that has been outlined, investigator-initiated
12
proposals might still be condoned (but effectively discouraged) by requiring
that these achieve priority for funding solely in terms of the extent of
their relationship with high priority project concepts. Alternatively,
a portion of the available funds for research and demonstration
support can be laid aside specifically for investigator-initialted proposals
and a prioritization process can be developed to choose those to be funded.
In the present context, it was noted before that the AARPS benefit-
cost model can, with only a moderate procedural revision, be made applicable
to evaluating investigator-initiated proposals versus project concepts.
The reason for this applicability is that investigator-initiated proposals can
be viewed as containing two principal components: (1) the direct equivalent
of a project concept (i.e., identification of a rehabilitation issue,
specification of an issue-related knowledge gap, characterization of an over-
all research strategy, etc.), and (2) the formulation of a specific research
proposal aimed at addressing the project concept. The evaluation of such
project proposals, as well as contract proposals and responses to agency-
solicited proposals, is discussed in the following sections.
I.1.7 Prioritization of Competing Proposals and Selection for Funding
The AARPS model for evaluating competing project concepts has been
adapted to assessing competing project proposals. In the related pro-
cedures, a prominent role is assigned to peer review.
The points discussed previously are essential components of
a process for allocating rehabilitation research funds. Some of the
linkages in this process can immediately be perceived as sequential
lines of action. Others can be conceived as feedback loops which
impart more accurate direction toward the desired goals of the
whole process, as shown in figure 1.
13
I.1.8 Role of Budget Constraints
The allocation of research support by NIHR must be managed with
sensitivity to opportunities for research or demonstration that are
delineated explicitly in the authorizing legislation (PL 95-602). That
legislation authorizes, for example, research bearing on the rehabilitative
needs of handicapped children, individuals aged 60 or over, rural populations,
native americans, etc. Though line-item appropriations have not been
associated with each problem area, the agency may well wish to assure
that some effort is supported in most, if not all, such areas.
One step that could be taken to foster the desired effort is to
encourage the development of one or more well-formulated project concepts
in each legislatively specified area. This development could be aided by
an appropriate choice of staff members and external consultants who would
be responsible for developing the concepts. However, if the benefit-cost
model alone is used for funding decisions, it cannot be assumed that the
project concepts in each and every area would compete successfully against
the entire array of proposals. This situation requires the use of the
benefit-cost model be coupled with a management strategy in which both
maximum and minimum funding limits are assigned to each legislatively
designated area. The project concepts for each would be formulated with
cognizance of these limits, and at the same time provision could be made
for their modification if it became clear that the required research
costs must exceed the upper funding limit. In connection with this approach,
the benefit-cost model might have two uses, one for the case in which two
or more competing project concepts had been developed in the same designated
area, thus necessitating prioritization; the other use would be to evaluate
the project concepts in each designated area against the entire array of
14
proposals to estimate the degree to which the pre-assignment of funding
levels was at the expense of achieving maximum benefit-cost outcomes.
The following sections are particularly addressed to the development of
project concepts and to the application of the model to prioritization. The
development of the model and the operational procedures for evaluating project
concepts has been the focus of the AARPS group's effort. See GLOSSARY,
which follows Part II, for definitions of key words used in this report.
15
I.2 THE BENEFIT-COST MODEL
The benefit-cost (B-C) model developed by the AARPS group to priori-
tize rehabilitation research was derived from a theoretical analysis.
The general formulation for the benefit portion of the model is:
[1]
where B = total benefits, Ps and PU = probabilities of success and of
utilization, respectively; N = size of the target population; B₁ = benefits directly
accruing to individuals or units; Bs = indirect benefits of successful
research; BF = indirect benefits of funding. Equation [1 represents both
the one-dimensional and multi-dimensional cases for each of the parameters
shown. In the multi-dimensional case, each of the benefit terms (B₁, Bs, and BF)
is a vector. The specific connotation of these terms as vectors, as well as
a full explanation concerning the multi-dimensional aspects of the model, can
be found in sec. I.3, Benefits and Costs of Rehabilitation Research.
The complete form of the model includes, of course, the costs. There are
various ways in which the costs may be included. For our purposes, the
benefit-cost ratio we will use is:
R = B/CR
[2]
where, for application to research prioritization, CR is the cost of research.
Equation [2]is formally correct in the one-dimensional case only. In the
multi-dimensional case, B is replaced by V (see section I.5.3).
The application of benefit-cost analysis requires a careful consideration
of the concepts of benefits and costs. When the benefit-cost ratio is used,
it is important that there be a clear understanding of what goes to the
numerator and what to the denominator.
A benefit is generally defined as a specific advantage that one or more
persons may realize as a result of some action. The action involved in this
particular case is the performance of rehabilitation research. The benefits
16
gained by the utilization of the results of rehabilitation research may be
increased personal earning power, reduction of expenses for medical care or
less dependency on others. Such benefits can be measured in monetary terms.
Benefits can also be expressed in terms of improved "quality of life,"
a greater sense of "well-being," or a more positive "role in society."
These less tangible benefits are difficult to express in monetary terms but,
nonetheless, exist and are taken into account in the benefit-cost model
that the AARPS group has developed. As such, the model is an important
advance over conventional benefit-cost models which ignore non-monetary
benefits.
17
I.3 BENEFITS AND COSTS OF REHABILITATION RESEARCH
In the AARPS benefit-cost model, three types of benefits are posited:
(1) those serving individuals or institutions directly if the research is
successful and utilized (B₁); (2) those serving individuals, groups and
institutions or society-at-large, but indirectly and only if the research
is successful (Bs); and (3) those serving any person or group of persons
regardless of research success (BF).
I.3.1 Identification of Benefits
When the AARPS study began, an attempt was made to develop an
exhaustive list of potential benefits that may be derived from rehabili-
tation research. To accomplish this search, the administrative staff of
the appropriate Federal agency (SRS) was asked to list all benefits they
would consider important. After eliminating obviously redundant items,
a list of 243 potential benefits was generated. Although the survey was
certainly effective in regard to achieving comprehensiveness, the list
was far too lengthy for direct use in the model. It was imperative,
therefore, to reduce the potential benefit list. This reduction was
accomplished by conducting a second stage of data collection, which was
submitted to a cluster analysis. The benefit list was thus reduced to 23
personal and 23 non-personal benefits.
Such a list makes it evident that each of the three types of benefits
(B₁, Bs, and BF) is a multi-dimensionalterm in the benefit-cost model.
The practicality of any benefit-cost analysis decreases substantially with
an increase in the number of dimensions being considered. With a potential
of 23 benefit factors for each type of benefit in the equation, the task of
rating a group of projects could become prohibitive. The AARPS group
sought, therefore, to reduce further the list of potential benefits into a
18
more manageable number of dimensions. A subsequent two-stage reduction
analysis (Fuhrer, Cardus, and Rossi, 1979) resulted in the final
list of benefit dimensions shown in table I. This benefit clustering process,
together with a comprehensive chart of all benefit factors, is presented
in Appendix 2. The reader is reminded that each benefit dimension in table I
includes both benefits and costs (which may be considered negative benefits).
As was described in sec. I.2, the benefit portion of the AARPS model is:
B = Pₛ(P₁NB₁+Bₛ) + + + BF
[1]
This equation represents both the one-dimensional and the multi-dimensional
case for each of the benefit terms, with the multi-dimensional condition
meaning that all of the B terms are vectors. This concept is more readily
understood when the vectors are broken down into their constituent components,
each component collectively represented by variable subscripts:
[3]
In this expression, Bi is the expected net benefit for the i-th dimension.
If all of the Bi's are expressed in the same units, they can be added together
to get a single number (a value V) for the total expected net benefit.
In the AARPS model, each of the seven benefit dimensions (table I) has:
different units, and a slightly modified notation can be used to avoid confusion:
[4]
Thus, aseven-dimensionalvector, X, is generated by the model and is
then used to produce another vector, X(M), in which all seven components are
expressed in the same units (normally dollars). The procedure is described
in section I.5.3.
Equation [4]indicates there are twenty-four terms in the model:
However, it is clear from an examination of table I that some of the benefit
19
dimensions are defined in a manner that would make several of the twenty-
four terms have a zero value. For example, Improved Individual Client Outcomes
(benefit dimension 3) would have a numerical value for B1,3, but would have
zero values for Bs,3 and BF,3, by definition. In fact, Monetary Benefits
(benefit dimension 1) is the only dimension that can have contributions from
all three benefit categories (B₁, Bs, and BF). Each of the six remaining
benefit dimensions has a contribution from only one out of the three.
This orthogonality of benefits is an important aspect of the model and
results from the benefit clustering process described in the previous section
and in Appendix 2. The table below illustrates this important aspect and clarifies
the correspondence between the vectors BI, Bs, and BF and the seven
benefit dimensions (table I). The nature of the table below and of table I
are described in Appendix 2.
Benefit
Dimension
1
2
3
4
5
6
7
Model BI BI,1 B₁,₂ BI,3, BI,4 B₁,5 - -
Benefit
Bs
Bs,1
-
-
-
-
BS,6
-
Term BF BF,1 - -
-
-
- BF,7
20
TABLE I
LIST OF BENEFIT DIMENSIONS AND BRIEF DESCRIPTIONS
1. Monetary Benefits-
All benefits that can be represented in monetary units which accrue
to either an individual, a group of individuals, one or more insti-
tutions, or to society as a whole.
2. Enhanced Quality of Services-
Those benefits exemplified by improved access to services, improved
individualization of services, improved coordination among services
or improved continuity of services.
3. Improved Individual Client Outcomes-
Those benefits exemplified by minimization of functional limitations
and personal disability, the encouragement of greater individual
social participation, or from improved vocational status and
material well-being.
4. Improved Administrative Bases for Service Provision-
Those benefits exemplified by improved management of information
systems yielding timely and relevant administrative decision support,
identifying operational constraints on operations and implementation,
establishing more explicit procedures for program prioritization, and
more effective sequencing of program development.
5. Improved Policy Bases for Rehabilitation-
Those benefits exemplified by improved legislative impact and coordi-
nation of government entities, the development and communication of
policies, plans and procedures, and the facilitation of societal change.
6. Indirect Benefits, Given Project Success-
Those benefits exemplified by expanded knowledge bases, the identifi-
cation of new areas of research, and the spinoffs of technology
and/or procedures.
7. Indirect Benefits, Regardless of Project Success-
Those benefits exemplified by an enhanced public awareness of an issue
or problem, or a sustained effort focussed on a research problem.
21
I.3.2 Identification of Costs
In order to achieve a maximal total expected net benefit
for a given research budget, the benefit-cost ratio can be employed.
To achieve this goal, the traditional benefit-cost ratio of total
expected benefits to total expected costs (both with future terms
discounted to present values) is not suitable. Instead, we use the
more correct ratio of expected net benefits (total expected benefits
minus total expected costs) to current-year research costs. Details
of this modified ratio are given in section I.4.5 and Appendix 1.
The essence of the distinction between the two is that
what is significant to the decision-maker is the amount of the
scarce item, i.e., current research appropriations, that is required
for a given project or set of projects.
22
I.4 OPERATIONALIZING THE TERMS OF THE BENEFIT-COST MODEL
The terms of the model include three types of benefits (B₁, Bs and
BF), the target population size (N), two probabilities (Ps and PU) and the
costs. The following discussion contains recommended guidelines for operational-
izing each term of the model.
I.4.1 Assessment of Benefits
As was described in sec. I.3 all non-monetary individual benefits
and costs (B₁,i for i=2,3,4, & 5) are subsumed in benefit dimensions
2, 3, 4, and 5. Non-monetary indirect benefits and costs, given project
success (Bₛ,6), are subsumed in benefit dimension 6. Non-monetary in-
direct benefits and costs, regardless of project success (BF,7), are
subsumed in benefit dimension 7. All monetary benefits and costs (BI,1,
Bs,1, and BF,1) are subsumed in benefit dimension 1.
Of the seven benefit dimensions described, it is often easiest to
estimate monetary benefits (see procedure in sec. II.2) since they can be
totalled in identical units (dollars).
When benefits are non-monetary, the measurement process becomes
less conventional. To quantify a non-monetary benefit, some kind
of utility scale must be devised which represents value, in a defined
framework or from a professional viewpoint, of that benefit to those
who would realize a gain from it (an individual client, a population of clients,
an institution, etc.). Although this judgement of value is not as straight-
forward as for monetary benefits, there are nonetheless several satisfactory
procedures for estimating such values. These procedures are discussed
in greater detail in the following paragraphs.
Rehabilitation professionals can rate project concepts with respect
to each benefit dimension, based on their knowledge and experience.
Several subjective scaling methods have been developed which
23
could be used in assigning values to the non-monetary benefit dimensions.
These methods include paired comparisons, ratio estimation (constant
sum method), rank ordering (order of merit), equivalence, rating scales
and the method of magnitude estimation. In selecting the method to be
recommended, the following criteria have been applied: (1) the method
should be readily comprehensible to raters who may not be particularly
skilled in psychometric measurement; (2) written instructions should
be sufficient to explain the method to the judges; (3) the entire judgement
process should be able to be carried out by any one rater in a reasonable
amount of time (e.g., a few hours or so); and (4) the method should be
capable of producing a ratio scale. Using these criteria, the method of
magnitude estimation was selected as the most appropriate for the purpose
of scaling non-monetary benefits. Details about the procedure are given
in section II.3.
I.4.2 Target Population
Each project concept, either explicitly or implicitly, specifies
a set of individuals who are the intended potential beneficiaries. This
set of persons is called the target population of the project concept.
The World Health Organization (WHO) has proposed a system of class-
ification on three levels that is very useful in describing target populations.
The first level is that of impairment: "a permanent or transitory psycho-
logical, physiological or anatomical loss and/or abnormality." Impairments
frequently relate to clinical diagnoses, e.g amputation, paralysis due to
polio, hypertension, etc. The second level consists of functional limitations:
"the partial or total inability to perform (underlines added) those activities
necessary for motor, sensory, or mental functions within the range and manner
24
of which a human being is normally capable, such as walking, counting, taking
an interest in and making contact with surroundings." It is noted further that
such limitations may last varying lengths of time, be permanent or rever-
sible, progressive or regressive, but "should be quantifiable wherever
possible." The third level consists of disabilities, described as follows:
"Disability in which functional limitation and/or impairment is a causative
factor, is defined as an existing difficulty in performing one or more
activities, which in accordance with the subject's age, sex and normative
social role, are generally accepted as essential, basic components of daily
living, such as self-care, social relations, and economic activity. (under-
lines added)."
In terms of the WHO distinctions, target populations are frequently
specified in terms of multiple criteria, i.e., in terms of several impair-
ments or of a combination of impairments, functional limitations and dis-
abilities. Thus, the target group for a new orthotic device to improve a
stroke patient's gait might be defined in terms of a number of functional
limitations, e.g. weakness of leg muscles or range of motion of the ankle
and knee joint, as well as a number of functional abilities, e.g., mental
ability to understand instructions required for use of the device and ade-
quate visual/spatial perception.
Seldom will survey data be available for the target populations
of proposed rehabilitation research. The reason is that the available
data are based upon very broad characterizations of populations, e.g.,
an inability to work or the occurence of injury to the spinal cord. Such
general descriptors do not accord satisfactorily with the restrictive
manner in which target populations of specified research proposals need to
be defined.
25
A number of approaches are available for estimating a target popu-
lation if the pertinent survey data are unavailable. One approach is to
obtain global estimates from a number of experts familiar with the popula-
tion in question. A Delphi-type process could be used to refine such
judgements. Alternatively, the AARPS group has described an explicit,
analytic process for generating target-population size estimates. As
described in greater detail in section II.1, this process requires that an
estimate be made of a parent population, i.e., one containing the entire
target population as a proper subset or subpopulation. The relationship
between estimates of parent and target populations is explained next.
Estimation of target population size: Estimates of target populations can be
generated if two requirements are met. First, national prevalence estimates
concerning the pertinent parent populations should be available. Second,
authoritative estimates based upon local experience must be available con-
cerning the size of specific target groups relative to (e.g., a percentage of)
the appropriate parent populations. With availability of these two estimates,
it is possible to estimate the national prevalence of the target population.
Consider, for example, a proposal regarding an innovative medication
to eliminate urinary tract infections in spinal cord injured individuals.
The two prerequisites would then be: (1) an estimate of the nationwide
prevalence of spinal cord injury, and (2) an authoritative estimate based
on local program experience of the proportion of such persons who have,
or who are at risk of having, such infections. Variance of estimation for
target groups can be assessed by obtaining information from multiple sources
(see section II.1).
26
Implementation of this approach to target population estimation
would place considerable emphasis upon NIHR becoming a repository for
prevalence data bearing upon the various kinds of disabilities and upon
functional limitations and impairments associated with disabilities. In
addition, substantial responsibility would be borne by research proposers
to seek out the best expertise available for estimating the size of relevant
target populations.
I.4.3 Probability of Success
In evaluating project concepts, some means are necessary for esti-
mating the extent to which each objective will be met. The estimated prob-
ability of success of some project concepts may be quite low. It may be the
case, however, that these are precisely the ones which yield the most substantial
potential net benefits. This observation underlines the importance of benefits
and probability of success being treated as independent values that are assessed
separately. With this provision, high-risk projects can score higher than
low-risk ones because the potential benefits are larger or because a larger
target population is affected.
Estimation of probability of success: It is critical that the judgements of
the probability of success be rendered by individuals with expertise and demon-
strated competency in the problem area being evaluated. Because the focus is
on project concepts, however, the applicable criteria must differ from those
that ordinarily pertain to the peer review of proposed research projects.
For example, the peer review of project proposals often involves close scrutiny
of the experimental design, data collection methods, and the investigator's
experience in conducting such research. Specific information of this kind will
27
ordinarily be unavailable for project concepts. Instead, the investigative
methodology will be characterized only in very general terms, e.g., the state
of the art regarding supporting knowledge or technology, and the availability
of experienced investigators and supportive facilities needed to address
the problem. Operationally, two or more qualified judges can be asked to
apply such criteria to each objective of a specific project concept. Each
evaluator then would be asked to provide a single combined judgement for
that project concept on a rating scale ( sec. II.4 has recommended procedure).
I.4.4 Probability of Utilization
As previously defined, the probability of utilization appearing in the
AARPS benefit-cost model is the likelihood that a given benefit will accrue
to the target population or subpopulation of either individuals or organi-
zations. An important feature of the model is that the probability of utili-
zation can be assessed in a benefit-cost analysis for each specific benefit
and for each target population or subpopulation.
Estimation of Probability of Utilization: Estimates of probability of util-
ization may be obtained in a manner similar to that described for the prob-
ability of success. However, the professionals who are called upon to pro-
vide these judgements might well be of a different expertise. Whereas
judgements provided by administrators at the institutional level could be
instructive, a survey of practitioners (e.g., medical rehabilitation pro-
fessionals or vocational counselors) or representative members of the target
population might be more meaningful. A rating scale similar to that proposed
28
for estimating probability of success may be employed, with judgements being
guided by criteria that include: relevance to agency goals, ease of under-
standing, ease of implementation, accessibility to the outcomes of the
research, the relative advantage of those outcomes compared to what is
currently available, the compatibility of the research outcomes with existing
values and norms, etc. (See section II.5 for a recommended procedure).
I.4.5 Introducing Costs into the Benefit-Cost Ratio
Costs are introduced into the benefit-cost model as follows:
a. One calculates, and designates by B*, the total expected value
of all benefits (see Assessment of Benefits, sec . I.4.1).
b. One calculates, and designates by C*, the total expected
value of all costs.
c. One calculates, and designates by B, the difference of expected
benefits minus expected costs. The project is considered, in principle,
worthwhile if B is positive.
d. One determines, and designates by CR, the current research
funds requested for conducting the project.
e. One calculates the benefit-cost ratio, R, using:
R = B/CR = (B* - C*)/CR
[5]
bearing in mind that CR is part of C*, as discussed in Appendix 1.
This ratio is the one used to rank the various projects according to
merit and is therefore used by the administrator to make funding
decisions in the presence of specified constraints, a process explained
more fully in Appendix 1.
29
I.5 IMPLEMENTING THE MODEL
The mathematical structure of the benefit-cost model can be better
described by considering separately its principal parts. These are:
the monetary submodel, the non-monetary submodel, the combination of
monetary and non-monetary benefit dimensions and the derivations and use
of the benefit-cost ratio.
I.5.1 The Monetary Submodel
The AARPS model requires that each project be evaluated from the
standpoint of each of the seven benefit dimensions listed. in table I.
(section I.3.1). Then, each project is represented by a set of values,
constituting a vector X, whose components are
x1 Monetary benefits
x₂ Enhanced quality of services
x3 Improved individual client outcomes
x4 Improved service administration
x5 Improved service policy bases
x6 Indirect benefits, given project success
X7 Indirect benefits, regardless of project success
In this section, the procedure for obtaining x1 is explained. In
section I.5.2, the procedure for dealing with the remaining benefit dimen-
sions is discussed in detail.
According to the small table in section I.3.1, B₁, Bs, and BF all con-
tribute to the new parameter x1 (benefit-in this case monetary-expressed as a vector):
x1 = PSPUNBI,1 + PsBs,1 + BF,1
[6]
The term containing BI,1 represents the monetary gain realized by those
individuals or units which have been designated as beneficiaries (target
population) of the project. For this monetary gain to be realized, the
30
research must be successful and the results must be utilized.
In general, the target population, N, must be partitioned into sub-
populations T1, T₂, ,Tₖ overeach of which the expected net benefit and
the probability of utilization per individual are approximately constant,
so that the PSPUNBI,1 term in [6 Jis replaced by a sum of similar terms,
one for each of the k subpopulations. The extent of the population refinement
should be determined as a compromise between the precision desired and the
costs of the analysis.
As already mentioned, the benefits discussed above are all expected net
benefits; that is, they have been aggregated over a number of years, the
costs have been subtracted and the discount rates have been factored in.
The calculations of individual monetary benefits can now be illustrated
through equation [6]and its generalization with two appropriate examples.
Example 1
Suppose that for the possible action of developing an improved wheel-
chair, the estimated target population is 1000 and is composed of the follow-
ing subpopulations differentiated by four levels of functional limitations:
SI = those able to walk with some aid (cane, walker, etc.)
S2 = those partially dependent on a wheelchair (e.g. during times of
fatigue, pain, stress, etc.)
S3 = those who are totally dependent on a wheelchair for mobility
S4 = those who are immobile (i.e. bedridden).
The monetary benefits of this action are attributed to restoration of
earning power and more effective self-care. Some direct and indirect costs
to be considered are: purchase and maintenance of the equipment, job training
expense, and other employment-related expenses, such as new transportation costs.
31
To assess earnings over a lifetime, the target population should be
further refined into age groups. Table II lists the number of persons in
each functional limitation-age group (chosen in decades from 10 to 60).
TABLE II
DISTRIBUTION OF 1000 INDIVIDUALS IN FUNCTIONAL LIMITATION-AGE GROUPS(N&)
Age Range, yr
10-20
21-30
31-40
41-50
51-60
SI
65
50
41
56
60
S2
31
45
52
98
126
S3
21
37
53
40
66
S4
5
18
29
44
64
To illustrate the estimation of the monetary portion of B₁, a sample
beneficiary aged 35 from group S3 can be taken. His or her benefit would
be a sum of terms, each of which is a product of three terms: (1) Bj,
expected net benefits to age j, (2) Qj, probability of survival at age j,
and (3) a discount factor (1+r)⁻ⁱ⁺³⁵. The sum runs from age 35 to age 80,
with Qj = probability of survival* to age j =(0.98)
Also assume that:
Job training cost
$1000
Ej
Expected earnings
$10,000/yr
Savings in self-care
$1000/yr
Cj
Maintenance fee
$100/yr
Other employment expenses
$400/yr
Price of wheelchair
$2500
*This formula is for illustration only, since it underestimates survival to
intermediate ages and overestimates survival at the larger ones.
32
Then Bj = Ej - Cj, where:
Ej
10,000 + 1000
35 <j <65
1000
j >65
2500 + 1000 + 100 + 400
j = 35
Cj
100 + 400
35 <j < 65
100
j > 65
Adding the terms together, one gets BI = $90,000 as an average figure
for the S3 and 31-to-40 age group. Another sample is taken for age 25 from
S3 and the 21-to-30 age group, and so forth. The 20 values for individual
monetary benefits are summarized in table III.
TABLE III
CALCULATED MONETARY BENEFITS FOR EACH FUNCTIONAL LIMITATION-AGE GROUP
B₁, α , Monetary Benefits: in $1000
Age Group, yr
10 - 20
21 - 30
31 40
41 - 50
51 60
S1
125
130
120
100
90
S2
110
115
105
90
80
S3
98
100
90
80
70
S4
9
10
8
5
2
Table IV presents hypothetical probabilities of utilization for each
subpopulation. Actual numbers could be based on estimates such as the per-
centage of people who would actually buy these improved wheelchairs, how
much they depend on this improvement for the purpose of employment, for
self-care, etc.
33
TABLE IV
WHEEL-CHAIR PROBABILITY OF UTILIZATION BY EACH FUNCTIONAL LIMITATION-AGE GROUP
PU,α
Values
Age Group, yr
10-20
21-30
31-40
41-50
51-60
S₁
0.01
0.01
0.01
0.005
0.005
S₂
0.02
0.25
0.25
0.200
0.200
S₃
0.40
0.45
0.45
0.400
0.300
S4
0.02
0.02
0.01
0.005
0.001
Suppose that Ps = 0.3. Substituting this value for Ps and using
values for Nα,B₁ₓ and from Tables II, III and IV in the expression
PsPU,α BI,1, , followed by adding the 20 products, yields an approximate
value of PSPUBI,1 = $4,400,000. This value is the first component (x1) of
the vector X for this project.
The twenty population-group breakdown is unworkable in practice
since it requires twenty assessments of target population size, probabilities of
utilization and individual benefits. The work is extremely tedious, if not
impossible, when a similar partitioning technique is applied to a large number
of project concepts. Once again, some compromise between precision and
practicality is called for. One modification under consideration is to reduce
the number of subpopulations and use a single average number for individual
benefi ts for all subsets sharing the same functional limitation.
Having obtained a dollar value for the first item in equation [6],
the remaining two terms must now be considered. The term Bs,1 contains
monetary benefits and costs that accrue indirectly and only if the project
is successful. Such benefits and costs are independent of target popu-
lation size.
34
One example of the cost included in Bs,1 is the development costs
associated with implementing the new wheelchair. It is assumed that
the government will pay for such development in order to facilitate the
availability of the expected new product. A manufacturer must be hired to
do the necessary engineering and testing, to build a prototype and pro-
duction-model wheelchair, and to perform the manufacturing design and
tooling for full-scale production. Such activity could easily approach a
cost of $100,000. In addition, a training program for wheelchair users,
complete with a curriculum and any support materials, must be developed
(with an estimated cost of $15,000).
The final term, BF, contains the benefits and costs of funding
the project regardless of project success. An obvious cost included in this
term is the research cost in funding the project over a 3-year period.
Assuming a yearly budget of $300,000 and a discount rate of 10 percent:
BF,1 = - $300,000 - (300,000)/(1 + 0.1) - (300,000)/(1 + 0.1)²
= - $820,000.
Using all previous information for the three terms in eq. [6]
the estimated net monetary benefits for the wheelchair are obtained:
x1 = $4,400,000 - 0.3($115,000) - $820,000 = $3,550,000.
Example 2
A project is proposed to develop a rehabilitation management
system which would use new computer techniques. If applied to a state
agency, the new system would eliminate 30 percent of the paperwork and ten
percent of the administrative staff. This reduction in operational workload
would have a significant budgetary impact over a ten-year span, but would also
require a sizeable initial investment.
35
In the numerical representation of table V, the fifty state agencies
(the target population for the project) are partitioned into subclasses.
For political or financial reasons, the project is not applicable to 17 of
the states. Values for Co (initial cost setup), Ej (annual savings); and Cj
(annual maintenance costs) are listed for each subclass. The approach for
developing the system has already been established as very reliable; therefore,
Ps can be assumed to be one and, for convenience, the discount rate can be
assumed to be zero. Table V contains the calculations for the individual
monetary benefits.
The total expected net benefits associated with the B₁,₁ term is
$25.26 million. It is assumed that Bs,1 = 0 and that BF,1 = -CR, the cost of
research, which is estimated at $260,000. Then X1 for this project woud be:
X1= N 7 PSPU,aNαBI,1,a + = $25 million
TABLE V
Monetary Parameters ($ millions)
Subclass
Co
Ej
Cj
=
=10(Ej-Cj)-Co
{α}
1
-
-
-
-
0
17
-
2
0.17
0.15
0.05
0.83
0.3
10
2.49
3
0.17
0.15 0.05
0.83
0.7
6
3.49
4
0.27
0.20 0.05
1.23
0.5
4
2.46
5
0.35
0.25 0.05
1.65
0.5
7
5.78
6
0.35
0.25 0.05
1.65
0.8
4
5.28
7
0.50
0.42 0.05
3.20
0.9
2
5.76
Total
25.26
36
1.5.2 The Non-Monetary Submodel
An approach to constructing utility scales for estimating non-
monetary benefits is now described. Along with the analytic description
a hypothetical example to illustrate the procedure is given. Suppose
there are three competing project concepts (PCj) to be prioritized.
PC1 is the action of developing an improved wheelchair (example 1,
section I.5.1)
PC2 is the development of a new computer system to be used in
three state agencies (example 2, section I.5.1)
PC3 a project that fulfills a congressional mandate.
Here, the population breakdown is 20 populations for PC1, 3 populations for
PC2, and one population for PC3. The size of each subgroup may be denoted
by NJ, where the superscript j identifies the project and the subscript
identifies the subpopulation.
Then, for each benefit dimension (i=2,3,4, and 5):
X is a sum of products one such product for
each subpopulation),
and
Now, a set of judges (assume four) is asked to assess all of the
non-monetary benefits for each subpopulation by the method proposed in
section II.3.
For example, in table VI judge 1 assesses the achievability of x₂, or
"enhanced service quality," of the three projects (refer to table II for
the population breakdown of PC1).
37
TABLE VI
RATINGS GIVEN BY JUDGE 1 ON THE BENEFIT DIMENSION
"ENHANCED SERVICE QUALITY" FOR THREE PROJECT CONCEPTS
1
1
1
2
2
1
1
1
2
2
PC1
2
2
2
3
3
3
3
3
4
4
PC2
55
55
55
PC3
70
The normalized values for judge 1 are given in table VII. Each entry
in table VI is divided by 278 (the sum of all entries) to get the
corresponding entry in table VII.
TABLE VII
NORMALIZATION OF RATINGS GIVEN BY JUDGE 1 (Table VI)
0.0036
0.0036
0.0036
0.0072
0.0072
0.0036
0.0036
0.0036
0.0072
0.0072
PC1
0.0072
0.0072
0.0072
0.0108
0.0108
0.0108
0.0108
0.0108
0.0144
0.0144
PC2
0.19784
0.19784
0.19784
PC3
0.2518
The purpose of the normalization is to give equal total weight to each
judge while preserving his/her relative values.
38
I.5.3 Combining Monetary and Non-Monetary Benefits
Once monetary and non-monetary benefits have been estimated and
each project is expressed by a vector X, a procedure must be used to merge
monetary and non-monetary benefits so that all benefit dimensions can be
converted to the same unit (dollars).
The process is as follows:
Step 1. The j-th competing research project is represented by a
vector X whose seven components are ratings on the respective
benefit dimensions.
Step 2. Each benefit dimension is assigned a weight which is an estima-
tion of its relative importance as perceived by the responsible
decision-maker. These weights define a vector 1, , as explained
in section II.6.
Step 3. Another vector Y is selected to represent all competing projects.
The components of vector Y are the maximal values for each of
the seven benefit dimensions of the project vectors xj.
Step 4. The representative vector Y and the vector expressing the
relative importance of the benefits ()) are compared and
proportionality coefficients (mᵢ) are obtained to allow the
conversion of all non-monetary benefit dimensions into dollar
equivalents.
An illustrative example is presented next:
Step 1. Suppose we have five project concepts and suppose the five
X vectors rating them are:
39
1
X
x²
x3
X4
x5
x1
4,400,000
1,000,000,000
26,550,000
850,000
1,100,000
X₂
8,000
2,400
65,000
12,800
3,500
x₃
53,200
25,000
3,333
41,350
9,900
x4
14,000
10,880
78,000
353'00
106,000
X5
828
1,100
3,000
9,500
1,500
x6
8
1
32
90
45
x7
11
0
18
55
88
2. Each benefit dimension is given a weight of relative importance
(vector ). Let us suppose
1
0.50
X2
0.15
}3
0.20
A =
14
=
0.05
J
0.05
¹₆
0.03
⁷₇
0.02
Step 3. We select a vector Y whose i-th component yᵢ (for each i) is
the largest of the i-th components of the vectors xj, with Y given by:
1,000,000,000
65,000
53,200
Y
=
106,000
9,500
90
88
40
Step 4. We determine multipliers m1, m₂ m₇ SO that the products
m1y1, m2y2,....,m7y7 are proportional to the relative importance
weights Since the first dimension already
has a well-defined unit of measurement (the dollar)
it is taken as basic, and by setting m1 = 1, we can interpret
each mᵢ as the induced dollar value of one unit in dimension i.
(the formula for mi is
Next, each vector X is transformed into a vector X³(M) defined by:
[m1x1j
XJ(M) =
m7x7j
For example, for project 1:
4,400,000
39,923,200
400,000,000
X¹(M) =
13,207,600
8,715,528
5,333,330
5,000,000
The components of this vector are the dollar values contributed by the
seven benefit dimensions. Thus, the sum, v¹, of the components of x1 (M) is:
v1 = 4.7358 X 10⁸
41
and is the computed dollar value of project 1. The same procedure is followed
to obtain total dollar values (vi) for the remaining projects.
I.5.4 Using the Benefit-Cost Ratio to Prioritize Projects
The problem of choosing projects among PC¹, PCⁿ for grant awards
is obviously under the constraint of limited funds. Suppose the research
cost for PCi is CRJ and the total amount of funds available is S. With the
calculated data, vi, and the benefit-cost ratios Rj = VJ/CRi, we can formu-
late the process of selection as an integer programming problem.
max
v¹z¹⁺
subject to
CR¹ z¹⁺
+. CRⁿ Zn, S
where
zj = 1, if PCi is to be funded, or
zj = 0, if PCi is to be rejected.
One intuitive solution (see Appendix 1) is to arrange the PC! in order
according to their benefit-cost ratios beginning with the largest RJ and
then implement projects in order until the total resource S is allocated.
Suppose, for example, that S = $1,750,000 and that the values
vj, CR and RJ for the previous five project concepts are those listed:
PC¹
PC²
PC³
PC⁴
PC⁵
vj
4.7x10⁸
1.22x10⁹
4.863x10⁸
5.89x10⁸
2.755x10⁸
CRJ
3x10⁵
1.05x10⁶
8x10⁵
1.05x10⁵
6.5x10⁵
Rj
157.9
97.7
60.8
393
42.7
Thus, R⁴ > R¹> R². > R3 > R⁵, and
CR 4 + CR¹ + = 1,500,000 < S
CR 4 + CR 1 + CR² + CR³ = 2,300,000 > S
Therefore, the final decision is to accept projects 1, 2, and 4.
42
There is an apparent defect in this "order and choose" rule.
Suppose, for instance, S = $1,300,000. Then CR⁴ + CR¹ + CR² > S and
hence only two projects, PC⁴ and PC¹ , should be funded, However, the
implementation of PC⁴, PC¹, , and PC³ is within the range of total funds
and implies more benefit than the previous selection of only two projects.
This problem may be solved in either of two ways: (a) apply a computer
algorithm to solve the so-called "knapsack" programming problem, or
(b) allow the funds S or the research funding CRJ to be somewhat flexible
SO that the project listed on the margin of total budget is also included
in the funded set.
43
I.6 COMPUTER PROGRAM FOR MODEL APPLICATIONS
Sections I.4 and I.5 describe the rationale and operationalization
of the AARPS benefit-cost model and its special application to the
prioritization of research concepts. By necessity, the model must have a
certain degree of complexity. Since this complexity could be a deterrent
to its utilization, a computer program can be used to facilitate its
application. The computer program is being generated in two stages.
In the first stage, a computer subroutine (ARPCOM) was developed
which accepts as input data the benefit assessments (X vectors), the
research cost for each project concept, and the relative-importance weights
for the benefit dimensions. ARPCOM performs all of the necessary mathe-
matical calculations and places in rank order the resulting benefit-cost
ratios. This subroutine is capable of doing for large sets the same
thing that was done manually for the small set of data shown in
the illustration in section I.5.3. Instructions for use of ARPCOM are
given in section II.7. For the future, a much more elaborate program
(ARPSIN) is envisioned that will use as input data the estimations of
probability of utilization, probability of success, target population, and
benefits. The ARPSIN output will be the benefit vectors which are needed
as input data for ARPCOM. ARPSIN development, pending funding, will be
completed once experience in collecting and processing input data has
been acquired.
44
PART II
PROCEDURES
The procedures included in this part are the instructions and
methods intended for use by judges in attaining numbers for the terms
of the AARPS model (i.e., Bₗ, Bs, BF, N, Ps, and PU) and the weights
for benefit dimensions (¹ₛ) and to rate the judges, if desired. Judges are
asked to give ratings only, which are used as input to a special computer
program (developed in two stages, as discussed in sect. II.7).
II.1 ESTIMATION OF TARGET POPULATION SIZE
Figure 2 displays the successive steps in a process that begins
with attention to the project concept's or research proposal's stated
objectives. These objectives imply the target population of the research
and therefore provide the basis for constructing this specification.
As such, a target population is simply the total group that potentially
falls within the focus of the project. Included are all individuals to whom
the findings might apply. In addition to being consistent with the project's
objectives, the target population must also be designated in satisfactorily
operational language. For example, defining the target population as "handi-
capped people" is hardly adequate unless the proposer can specify the opera-
tions by which "handicapped" persons are to be identified.
Having delineated the target population, the question becomes
whether or not the national prevalence of the population and of its
significant subpopulations have been established (Step 2, figure 2).
For the overwhelming proportion of research proposals, satisfactory
prevalence data will not be available. The reason for this void is that
available survey data deal with broadly defined populations, such as "per-
45
STEP 1.
STEP 2.
STEP 3.
ADOPT,
DELINEATE
SPECIFY
DETERMINE WHETHER
YES
INDICATING
PROJECT
TARGET
SATISFACTORY TARGET
UNCERTAINTIES
OBJECTIVES
GROUP(S)
POPULATION PREVALENCE
OF ESTIMATING
ESTIMATES ARE AVAILABLE
NO
STEP 4.
STEP 5.
STEP 6.
IDENTIFY CRITICAL
IDENTIFY
ENUMERATE TARGET
PARENT POPULATION
ACCESSIBLE
GROUP MEMBERS AND
& ITS PREVALENCE
POPULATION
EXPRESS AS PROPORTION
OF ACCESSIBLE POPULATION
STEP 7.
OBTAIN ESTIMATED TARGET POPULATION PREVALENCE
BY MULTIPLYING OUTCOMES OF STEPS 4 & 6,
ADJUSTING FOR INFERRED BIASES AND INDICATING
UNCERTAINTIES OF ESTIMATING
FIGURE 2. - SEQUENCE FOR OBTAINING TARGET POPULATION PREVALENCE VALUES
46
sons unable to maintain employment" or "spinal-cord injured individuals."
On the other hand, the target populations of specific research proposals
are of necessity defined in relatively restrictive terms.
If an estimate of a target population needs to be generated anew,
a critical parent population (CPP) must be designated (Step 4). The notion
of a parent population is critical to understanding the CPP. A parent
popluation is one that contains the entire target population as a proper
subset, i.e., as one of several subpopulations. In principle, a given target
population may have a large number of different parent populations, some
hierarchically organized and others completely independent of one another.
A target population of mobility-impaired rheumatoid arthritics has as
parent populations rheumatoid arthritics generally (both with and without
mobility impairments) as well as arthritics generally (those with both the
rheumatoid and non-rheumatoid forms of the disease). Yet another parent
population is "mobility-impaired persons," including those with impair-
ments due to rheumatoid arthritis as well as to other impairments.
The CPP is a parent population with two additional characteristics:
(1) reasonable estimates of its national prevalence are available, and (2)
it approximates an accessible population that can be studied to determine the
proportion of individuals fulfilling the restrictive definition of the target
population.
The interplay of these two essential characteristics of the CPP can
be illustrated by considering an example of proposed research with a target
population consisting of persons with spinal cord injury who are considered
to be candidates for receiving training in independent living skills.
Assume, further, that the target population is defined specifically as
47
consisting of individuals who are wheelchair users, are from 16 to 50
years of age, have completed a comprehensive inpatient rehabilitation program
and have expressed an interest in such training.
The CPP for this kind of hypothetical target population might well
be "persons with traumatic spinal cord injury," since a credible prevalence
estimate of this population has been provided by the National Center for
Health Statistics. Ideally, estimation of the target population's prevalence
would proceed by drawing a statistically valid, random sample from this CPP.
It is hardly likely that resources would be authorized to mount a properly
designed national study of this kind. It might be realistic, however, to
acquire data from a sample drawn from an accessible population of "persons
with traumatic spinal cord injury" (Step 5, figure 2).
In some instances, the accessible population will be presented by
clinical records that can be studied retrospectively. The data may repre-
sent the experience of a specific program or possibly the aggregate experience
of several similar programs. For the traumatic spinal cord injury population,
for example, aggregate data are being compiled by the National Spinal Cord
Injury Data Research Center. The available record file may be exhaustively
surveyed or randomly sampled with the purpose of determining the proportion of
individuals conforming to the definition of the target population (Step 6).
Some target populations are defined by attributes that are not routinely
reported in clinical records. In such instances, the data must be acquired
directly from a random sample of subjects drawn from an accessible population.
This procedure might be necessary in the example being considered, since
an expression of interest in receiving training in independent living
skills is not likely to be documented routinely in clinical records. The
feasibility of acquiring the needed data will depend substantially upon the
48
requirements for detecting or measuring the attributes in question. For the
present example, a relatively economical telephone survey of appropriately
selected respondents would probably suffice.
The number of target population members in the sample may be expressed
as a proportion of the accessible population, and that proportion may be
applied to the prevalence estimate for the CPP to obtain the needed target
population size estimates (Step 7, figure 2). It is important that the
estimate be adjusted upward or downward depending upon judgements of the
degree to which the accessible population represents the CPP. Degrees of
CPP representation may arise, for example, because of differences in
operational definitions.
In the example being considered, traumatic spinal cord injury may
have been identified in the CPP by means of health status interviews, but
specified in the accessible population by means of a formal neurological
examination. It is also useful to compare descriptive statistics bearing
upon the CPP and accessible population. Of particular concern are differ-
ences between the populations on variables that may interact with those
critical to defining the target population. For the hypothetical project
being considered, interest in receiving the independent living skills
training may be correlated with the level of average personal income.
Thus, evidence that the accessible population and CPP differ markedly in
average income might be used as a basis for modifying the estimate based
upon data from the accessible population.
49
II.2 ESTIMATION OF MONETARY BENEFITS FOR INDIVIDUALS
In section I.5, it was noted that equation [3]applies to each
subclass (subpopulation) of the parent population. We are now concerned
with evaluation of the monetary component of the expected net benefit, BI,1'
for an "average" individual member of the subclass. Noble (1977) sum-
marizes benefits and costs of rehabilitation in table VIII (table 6 of that
reference). Not all of Noble's listed items will apply to any given project.
A well formulated project concept will indicate which benefits and
costs are considered to be relevant to it and should also give estimates in
dollars of their magnitude for the average individual. Recall that one
criterion for subclass selection is approximate constancy of these dollar
amounts from individual to individual within the subclass.
Our definition of B₁,₁ includes use of an approved discount rate
to bring all future benefits and costs back to present values. Thus,
selection of a discount rate to be used in all benefit and cost analyses
is a critical step in the whole evaluation process. Setting this rate is
an important management prerogative. and responsibility.
As the discount rate increases, an individual's age becomes less
important. For example, for a 10 percent discount rate, values twenty
years hence are divided by 7, and for 15 percent, by 16. With such rates,
it becomes reasonable to use some average age in the calculation and avoid
refinement of the target population into age groups as was done in the illus-
trative example in section I.5.1 (see tables I, II, and III).
Suppose that the target subpopulation includes a large current
population as well as expected annual increments. Then, one discounts
values for these incremental groups from their date of entry into the
target population back to the present time. If the current target popu-
50
lation is N, if the annual increments all have approximately the same size
(denoted by n), if the discount rate is r, and if the discounted expected
net benefit for the average individual in the current target population is
BI,1, then the total net benefit for the overall target subpopulation
(present and future) is:
(N +n/r)B1,1
[7]
and for this subpopulation, the contribution to the economic component X 1
will be
PsPu(N + n/r)B₁,1
[8]
Use of equation [7] permits a substantial reduction in the extent of refine-
ment that will be needed for the entire target population.
The interactive computer program , ARPSIN, is planned to relieve the
evaluator of the burden of the numerical calculations inherent in our benefit-
cost model. The monetary component clearly involves more of both arithmetic
and value judgements than are needed in the non-monetary components.
Similar considerations apply to the other monetary terms, Bs,1 and BF,1
51
TABLE VIII
BENEFITS AND COSTS OF REHABILITATION BY ANALYTIC PERSPECTIVES *
Analytic Perspective
Benefits & Costs
Indiv./Families Employers/Pvt. Sector Gov't Society
A. Benefit Increases
1. Earnings
X
a. .Net of taxes
X
b.Taxes
X
2. Homemaker Services
X
X
3. Unpaid Work
X
X
4. Life Satisfaction
X
X
5. Family Member Earnings
a.Net of taxes
X
b.Taxes
X
6. Decreased Nursing, Med-
ical & Custodial Costs
(a + b + c = 1)
aX
bX
cX
X
7. Lower Turnover in the
Labor Markets
X
X
B. Costs
1. Case Service Expendi-
tures (a + b + c = 1)
aX
bX
cX
X
2. Administrative & Over-
head Costs (a + b = 1)
aX
bX
X
3. Income Loss & Foregone
Earnings in the Program
a. Net of taxes
X
b. Taxes
X
4. Research, Training, and
Facility Costs (a + b = 1)
aX
bX
X
*From J. H. Noble, Jr., 1977 (p. 352).
52
II.3 SCALING OF NON-MONETARY BENEFIT DIMENSIONS
The forms in this section provide raters with magnitude estimation
procedures for rating a project concept according to each of the seven
benefit dimensions given in table I. With slight adaptation, the first
type of form is appropriate for rating each of the individual non-monetary
benefit dimensions (numbers 2, 3, 4 and 5). The second type of form is
designed for rating dimensions 6 and 7, which are non-individual and
usually non-monetary in character.
To rate the individual benefit dimensions, cognizance must be taken
of a project's target populations, i.e., all individuals to whom the findings
potentially apply. Since not all target population members may benefit to
the same extent, provision is made for dividing target populations into
subclasses that are relatively homogeneous in degree of expected benefit.
The subclasses are termed Project-Target Populations (PTP's). The PTP
pertaining to each project concept will be explicitly designated for raters.
It should be noted that dimensions 6 and 7 are rated in terms of
the overall project, not each of the PTP's.
53
II.3.1 Procedure for Quantifying Individual Non-Monetary Benefits (2 to 5)
DIRECTIONS: The projects are to be evaluated in a specific sequence, so
please do not rearrange them. You are being asked to judge the ex-
tent to which a typical member of a Project-Target Population stands
to benefit in terms of the benefit dimension specified below, assuming
the project is successful and the results used. Identify PTP's in col.(2)
1. Identify yourself in the space marked "judge."
2. To proceed, rank order the Project-Target Populations (PTP's)
associated with the first project. Assign the number "1" to the
PTP that stands to benefit the most. Rank the remaining PTP's accord-
ing to the extent to which you feel they stand to benefit. Write
the rank you assign each PTP in column (3) below marked RANK. Go
on to the next project, again assigning a rank of "1" to its top-
most PTP, and ranking the remaining PTP's as you feel appropriate.
3. After you have ranked the PTP's within all of the projects, it
is necessary to assign the PTP's points on a ratio scale. To do
this, go back to the first project and give its topmost PTP (i.e.,
the PTP ranked "1") a rating of 100. Write the rating you assign
in column (4) below marked RATING (100). Assign a rating of zero
to all PTP's of this project to which this benefit does not seem
to apply. Then take a second ranked PTP (assuming it has not been
assigned a zero) and decide how much a typical individual would
benefit relative to an individual in the PTP that has been assigned
the score of 100. For example, if the benefit would be half as much,
assign a rating of 50; if two-thirds as much, assign a rating of 66,
etc. Do this for all PTP's associated with the project and go on
to the next project.
4. Having done this rating for all projects, it is now necessary to
compare the top-rated PTP across project concepts, i.e., those that
received a rating of 100. First, find among all project concepts the
top-rated PTP which you consider would benefit more than any other
top-rated one. Assign this PTP a rating of 1000. Write the rating
54
you assign in column (5) below. Now search for the next most
valued PTP that has been assigned a 100, and assign it a new
rating between zero and 1000 that is proportional to the expected
benefit of the PTP rated as 1000. As before, if you feel the
benefits will be half or three-fourths as much (or any other
proportion), assign the appropriate numerical value less than
1000 that reflects this judgement (e.g. 500 or 750). Do this
rating for all the 100-rated PTP's. The scaling process is
now complete (the computer program ARPSIN will establish the
ratings of the remaining non-top-rated PTP's using the ratings
you have provided once this program becomes available).
EXAMPLE: Benefit Dimension No. 2
Enhanced Service Quality-Improved access to services, improved
individualization of services, better coordination of ser-
vices, improved continuity of services
Judge
PROJECT NO.
PTP NO.
RANK
RATING (100)
RATING (1000)
(1)
(2)
(3)
(4)
(5)
55
II.3.2 Procedure for Quantifying Non-Individual, Non-Monetary Benefits (6&7)
DIRECTIONS:
1. Identify yourself in the space marked "judge."
2. Rank all projects according to the extent to which each would
provide benefits of the kind described by the factor specified below,
assuming each project was completed successfully and the results were
used. Assign the number "1" to the highest ranked project. Rank
the remaining projects according to the extent to which you feel they
would provide the benefit dimension described below. Write the ranks
you assign in the column below marked RANK.
3. Next, it is necessary to assign the projects to a ratio scale.
Assign the project ranked 1 a score of 1000. Assign a rating of
zero to all projects to which this benefit does not seem to apply.
Then take the second ranked project (assuming it has not been
assigned a zero) and decide how much it would provide the benefit
described below relative to the project that has been assigned to
the rating of 1000. For example, if you feel it would yield only
three quarters of the benefit provided by the first, then assign
it a rating of 750. If you feel, instead, that it would provide only
half the benefits of the first, assign a rating of 500. Write the
ratings you assign in the column below marked RATING (1000) for
all projects considered in the rating exercise.
56
EXAMPLE: Benefit Dimension No. 6
Indirect Benefits Given Project Success - Expanded knowledge bases,
identification of new areas of research, spinoffs of technology/procedures
Judge
PROJECT NO.
RANK
RATING(1000)
PROJECT NO. RANK RATING(1000)
1.
13.
2.
14.
3.
15.
4.
16.
5.
17.
6.
18.
7.
19.
8.
20.
9.
21.
10.
22.
11.
23.
12.
24.
57
II.4 ESTIMATION OF THE PROBABILITY OF SUCCESS
The following forms represent an initial effort to provide judges
with a procedure for rating probability of success for project concepts
on an absolute scale.
The probability of success should be judged by individuals with
research experience and demonstrated competency in the problem area
being evaluated. The following criteria should be used in rating : the
probability of success.
a. The state of the art regarding supporting knowledge
b. The state of the art regarding technical knowledge
c. Availability of qualified manpower and resources.
The format suggested for quantifying the probability of success is
presented on the following page.
58
Estimation of the Probability of Success
DIRECTIONS: You are asked to rate the probability of success of the
project concept based on the information provided by the project proposer.
First, identify yourself and the project concept you are rating in
the spaces provided below. Now, using the scale below, please indicate your
assessment of the likelihood of successful completion of the project concept.
Certain
10
Highly Probable
9
Quite Probable
8
Judge
Somewhat Probable
7
Project Concept
Slightly Probable
6
Equally Probable
5
Slightly Improbable
4
Somewhat Improbable
3
Quite Improbable
2
Highly Improbable
1
Impossible
0
59
II.5 ESTIMATION OF THE PROBABILITY OF UTILIZATION
The accompanying form is an initial effort to provide judges
with a procedure for rating the probability of utilization on an absolute
scale.
PU should be judged by administrators or staff within the
agency and by potential users of the information generated by research
(e.g., patients, practitioners, researchers, engineers, administrators,
policy makers, and legislators).
The following criteria were developed to be used in determining PU.
These criteria are variably applicable depending upon the concreteness
and specificity with which the outcomes of the research can be envisioned.
Relevance: Importance of proposed research in relation to agency goals.
Payoff: In relation to users in terms of improved efficiency/effectiveness.
Ease of understanding: Degree to which an innovation is easily under-
standable by prospective users.
Ease of implementation: Degree to which an innovation is easily
implementable by prospective users.
Compatibility: With existing values, norms, past experience, and pro-
cedures and facilities available to users.
Accessibility: Degree to which an innovation is available to users.
Observability: Opportunity for users to see a convincing and practical
demonstration of the usefulness of the innovation.
Trialability: Ease with which it can be tried out on a small scale
before major or irreversible commitments are made.
Credibility: Arising from evident validity of the findings and degree
to which the researcher has won the trust of potential users
Relative advantage: Over existing practices that will more than offset
the effort of changing to something new.
Utilization plan: Provision made to identify users, needs and to dis-
seminate and promote utilization of research results.
60
To rate the probability of utilization, cognizance must be taken of
a Project-Target Population (PTP), i.e., all individuals to whom the findings
potentially apply as described in project benefits (section II.3). Since
not all target population members may utilize the findings to the same
extent, provision is made for dividing target populations into subclasses
that are relatively homogeneous in degree of utilization. The subclasses
are termed PTP's. The PTP pertaining to each project concept will be
explicitly designated for the judges.
The format suggested for rating the probability of utili-
zation is presented on the following page,
61
Quantifying the Probability of Utilization
DIRECTIONS: You are asked to rate the probability that the findings generated
by the project concept will be utilized by each of the PTP's within each project
concept.
First, identify yourself and specify which project concept and PTP you
are rating in the spaces below. Now, using the scale below, indicate your
assessment of the likelihood that the specified target population will utilize
project concept findings.
Certain
10
Highly Probable
9
Judge
Quite Probable
8
Project Concept No
Somewhat Probable
7
Project-Target Population
Slightly Probable
6
Equally Probable
5
Slightly Improbable
4
Somewhat Improbable
3
Quite Improbable
2
Highly Improbable
1
Impossible
0
62
II.6 WEIGHTING BENEFIT DIMENSIONS
In choosing the most appropriate procedure to use in weighting
benefit dimensions, it was decided that the method should allow for both
numerical and graphical conceptualizations of weighting factors. It was
believed that, although some raters may feel more comfortable using
numbers, others might be more comfortable using graphs. A distinct advan-
tage of using both methods is that discrepant judgements yielded by the
two can be called to the judge's attention, thus affording the chance to
resolve the differences. In the following example, weights are generated
both numerically and graphically. The estimation part of the procedure
is included in steps 1 to 4. The rater could then finish the procedure
using a simple pocket calculator for computational steps 5 to 8.
Alternatively, the rater could conclude the weighting analysis (steps 5 to
8) using the ARPCOM computer program (see section II.7), particularly as
the number of weighting judgements increases.
The suggested procedure for weighting benefit dimensions is
presented on the following two pages (which may be removed or
copied, as with other formats in this book, for evaluative use).
63
Weighting Benefit Dimensions
DIRECTIONS: 1. Identify yourself as "judge."
2. Assign a number between 0 and 100 to each of the seven
benefit dimensions below, indicating how much weight should
be given to each benefit dimension compared to how much weight
would be given to the other benefit dimensions. The actual value
chosen does not matter. Only the value relative to the other
numbers counts. For example, if you feel all benefits should be
weighted equally, assign the same number (such as 1) to each.
Or, if you feel that benefits of one kind are worth as much as
all of the others combined, assign a 6 to that benefit and a 1
to each of the other six.
Judge
Benefit Dimension (see table I, p. 21)
Weight
1. Monetary Benefits
2. Enhanced Quality of Services
3. Improved Individual Client Outcomes
4. Improved Administrative Bases for Service Provision
5. Improved Policy Bases for Rehabilitation
6. Indirect Benefits, Given Project Success
7. Indirect Benefits, Regardless of Project Success
3. Without looking at the numbers you have assigned, graph what you
feel would be appropriate relative weights for these benefits. Do
not estimate numbers, simply graph your judgments:
EXAMPLE:
Weight
1
2
3
4
5
6
7
Benefit Dimension
So as to minimize distraction from your previous estimations,
it is suggested you rate the benefit dimensions in a different order of
numbers from those you assigned just above the graph (for example,
4-3-5-2-6-7-1). Although you may choose to rate these values
in a different order, the numerical assignment of the benefit dimensions
(i.e., numbers 1 to 7) must remain consistent. Now turn to the two
scales on the next page (step 4).
64
GRAPH A
GRAPH B
15
1.0
10
0.5
5
0
01234567
01234567
4. Now write in numbers on the scale at the left on Graph A.
assume that the bottom of the scale is zero and note
that the top of the scale. is fifteen (with intermediate points 5 & 10).
Determine the number value for each column (bar) in your graph
by using the scale at the left. Write the determined value above
each column (bar).
5. Add the numbers which you have written above each column.
Then divide each number by the total score and plot the results
on Graph B above. Do this for all seven benefit dimensions.
6. Now, referring to the preceeding page, add together the scores
assigned to the seven benefit dimensions. Next, divide each number
by the total score and plot the results, again on Graph B. It is
suggested that, if you used a pencil to plot in step 5, use a pen
for step six, or vice versa.
7. If the two graphs on Graph B do not agree, decide what each
value should be. Repeat any of the above steps if it's helpful.
8. Write your final weights for each benefit dimension in the appro-
priate space below. These values should add up to 1.0 exactly.
1
2
3
4
5
6
7
The arithmetic for steps 5 to 8 is done by computer as part of ARPCOM.
65
II.7 ARPCOM USER'S GUIDE
ARPCOM is an interactive computer program that calculates bene-
fit-cost ratios for project concepts. The program obtains data from the user,
performs calculations using the AARPS benefit-cost model, sorts projects
in order of decreasing benefit-cost ratio, and prints the results.
One purpose of ARPCOM is to demonstrate the decision-making
utility of the AARPS mathematical model. The project-concept benefit assess-
ments used as data in this program are generated by another, more-complex pro-
gram (ARPSIN). ARPCOM illustrates how the decision-maker may generate
administratively useful numbers, i.e., benefit-cost ratios, once the raw
benefit data are available.
II.7.1 Accessing ARPCOM
ARPCOM is stored in the memory of the General Electric Mark III
Foreground computer system under the ID of the AARPS Research Group.
The computer is accessed by dialing the local GE computer network tele-
phone number, and then coupling the phone to a terminal via a modem. The
following modes are used:
Modem - Full duplex
Terminal - Half duplex
Parity - Odd
Baud - 1200 for CRT, 300 for regular terminal.
II.7.2 ARPCOM Instructions
Within five seconds of coupling the telephone to the modem-type
"HHH," push the "return" button. The computer responds by printing
the characters "U#=." Type in the user number, password, and project
identification (separated by commas); then push return. The computer
66
prints "USED 2.32 UNITS" (the number may vary). The computer is now
waiting for a command. To run ARPCOM, type "RUN ARPCOM" and push return.
See Table IX, a sample ARPCOM RUN, at the end of this section.
To escape from ARPCOM in the middle of a run, push the "break"
button. This command aborts the program and returns control to the
computer, which prints "READY" and waits for instructions. To
run ARPCOM again, type "RUN ARPCOM." To sign off, type "BYE."
II.7.3 Using ARPCOM
ARPCOM is an interactive computer program. This term means that
there is a constant "dialogue" between the computer and the program
user, via a computer terminal. When the computer wants data or some kind
of response from the user, it will prompt the user with a question mark (?).
For example:
ENTER THE NUMBER OF PROJECTS, NOT TO
EXCEED FIFTY.
?
The user types in the appropriate number, and then pushes RETURN:
ENTER THE NUMBER OF PROJECTS, NOT TO
EXCEED FIFTY.
? 12
R
Note: In this user's guide, all user responses are underlined to
differentiate them from characters printed by the computer. The symbol R
means that the user is to push the "RETURN" button. In actual use,
no underlining or R's will appear.
When entering more than one item of data, all values are typed
in consecutively on one or more lines, separated by commas:
ENTER THE SEVEN BENEFIT ASSESSMENTS, AND THE ESTIMATED COST
FOR EACH PROJECT.
PROJECT 1 ?4400000, 839, 1140, 23, 5000, 90, 200, 2500000
R
67
In this case, the computer is looking for eight numbers. If too few
numbers are entered, the computer will keep prompting until all data
are entered:
PROJECT 1 ?4400000, 839, 1140, 23, 5000 R
?90,200
R
?2500000
R
If too many data are entered, the computer will use the first
eight numbers and ignore the remainder. It should be noted that, since
commas are used to separate data, they cannot be used in the conventional
sense of signifying thousands. For example, 4.4 million is written
"4400000" instead of "4,400,000."
One important feature of ARPCOM is that it allows the user to
correct his mistakes before proceeding to another part of the program, e.g.:
IF ALL DATA ARE CORRECT, PUSH RETURN.
IF NOT, ENTER THE NUMBER OF A PROJECT CONTAINING
INCORRECT DATA ? 12
R
RE-ENTER DATA FOR PROJECT 12 ?
Once the above principles are understood, using ARPCOM becomes
straightforward. The computer asks the user for the data it needs
(see Data Inputting, next) and gives the user opportunities to correct any
mistakes made in previous steps. The computer then calculates and prints the
final results. For an example, see sample of ARPCOM RUN (table IX).
The user may then change the Lambda data (next section), and rerun the
68
calculation step. This procedure enables the decision-maker to assess how
sensitive the final results are to changes in Lambda (see sample ARPCOM RUN).
II.7.4 Data Inputting
ARPCOM uses the following input data:
1. Number of projects, which must not exceed fifty.
2. Project concept assessments, up to and including eight entries per project.
The first seven entries are benefit assessments and the eighth is the
project cost. The seven benefit dimensions are listed in table I and
must be entered in the same corresponding order as in the table. In the
future, the entire data set will be generated by ARPSIN for access by ARPCOM.
3. Lambda data, the understanding of which is crucial to using ARPCOM.
The Lambda values are normalized weights indicating the relative impor-
tance of the seven benefit dimensions. Thus, a "Lambda" is a normalized
set of seven numbers. Here is an example:
Benefit Dimension: 1234567
[100 50 60 30 15 25 5 = Benefit Weights
This set of numbers was generated by the rater who was asked to
indicate the relative importance of the seven benefit dimensions. In this
example, and those that follow, the graphical technique for estimation des-
cribed in section II.6 was not used. This rater believed that the economic
benefits (dimension 1) were the most important and therefore arbitrarily
assigned B1 a value of 100. B₂ was about one-half as important as B1, so
the rater assigned B₂ a value of 50, continuing until each had a weighted value.
While this procedure is a convenient method for assigning weights,
these numbers cannot be used as they are. They must be "normalized," i.e.,
adjusted so that their sum totals to one. The result of normalizing the
69
above benefit dimension weights is referred to as "Lambda":
[0.35 0.17 0.21 0.11 0.05 0.09 0.02]= Lambda
Lambda is generated by adding up all seven numbers to get a total T
and then dividing each of the seven numbers by that total (T).
NOTE: The ARPCOM program performs all normalizations for the user,
thus allowing input of benefit weights in the form initially shown.
ARPCOM also allows the user to generate a composite Lambda. A com-
posite Lambda is a value obtained by averaging a set of two or more indiv-
idual Lambdas. This procedure allows the decision-maker to solicit bene-
fit weights from a group of "experts" and then put them together to get
a final, composite value representing all raters. The decision-maker may
even wish to assign "weights" to each of the experts, depending on their
perceived-expertise, as in the example below:
Rater
Benefit Dimension Weights
Rater's
Weights
1
[ 100
50
60
30
15
25
5]
100
2
[1000
300
500
200
200
100
100]
80
3
[
5
10
2
2
3
1
5]
10
Lambdas
1
[ 0.35
0.17
0.21
0.11
0.05
0.09
0.02]
0.53
2
[ 0.40
0.12
0.20
0.08
0.08
0.04
0.08]
0.42
3
[ 0.18
0.36
0.07
0.07
0.10
0.04
0.18]
0.05
The composite Lambda = 1(0.53) + 2(0.42) + 3(0.05)
= [0.36 0.16 0.20 0.09 0.07 0.07 0.05]
70
The previous example illustrates some important points. First,
it is obvious why normalization is important in this instance. If the
benefit dimension weights are not normalized prior to the final calcu-
lation, Rater 2's weights would have exerted a disproportionate effect
on the final results. Second, the example shows the importance of the
decision-maker's role.
Raters 1 and 2 believe that B₁ is more important than B₂, whereas
Rater 3 thinks that B₂ is more important than B₁, yet the latter was
assigned a lower weight by the decision-maker than the first two.
The final result reflects this intention: the opinions of Raters 1 and 2
completely dominate those of Rater 3.
ARPCOM gives the decision-maker both Lambda options: the
rater may enter a Lambda of choice or may generate a composite Lambda
from a set of individual Lambdas, along with the respective weighting
values. This latter process is illustrated in the sample ARPCOM RUN
at the end of this section (table IX).
II.7.5 Conclusions
ARPCOM is not a practical program in and of itself but is designed
for demonstration purposes. In the future, project concept data for ARPCOM
will be generated by another, more-complex computer program, ARPSIN
(yet to be developed). In actual day-to-day use, entering all project
data, as is required by ARPCOM, would be cumbersome indeed. A future
version of ARPCOM will include an option to access a data file created by
ARPSIN, yet the interactive portion of ARPCOM, which allows entering dif-
ferent Lambdas, would remain essentially the same.
71
RUN ARPCOM
ARPCOM
09:08EDT
07/01/80
WELCOME TO ARPCOM
THIS INTERACTIVE COMPUTER PROGRAM CALCULATES BENEFIT - COST
RATIOS FOR PROJECT CONCEPTS. PLEASE SEE THE ARPCOM USER
GUIDE FOR COMPLETE OPERATING INSTRUCTIONS.
ENTER THE NUMBER OF PROJECTS TO BE PROCESSED, NOT
TO EXCEED FIFTY.
75
ENTER THE BENEFIT AND COST DATA USING THE FOLLOWING FORMAT:
PROJECT x7x1,x2.x3,x4,X5.X6.x7.PROJECT COST
(PUSH RETURN)
WHERE
X1 = MONETARY BENEFITS
X2 = ENHANCED QUALITY OF SERVICES
X3 = IMPROVED INDIVIDUAL CLIENT OUTCOMES
X4 = IMPROVED ADMIN. BASES FOR SERVICE PROVISION
X5 = IMPROVED POLICY BASES FOR REHABILITATION
X6 = INDIRECT BENEFITS, GIVEN PROJECT SUCCESS
X7 = INDIRECT BENEFITS, REGARDLESS OF PROJECT SUCCESS
PLEASE LIMIT EACH ENTRY TO EIGHT DIGITS.
PROJECT 17100000,22500,1000.500,250,250,300x350000
PROJECT 27500000,3000,1500,600,1000,300,200,475000
PROJECT 37267000,500,500.167.167.233,267.500000
PROJECT 47255000,2500,2750.5000,500250,0y600000
PROJECT 57325000,250.500,100.2000,45.500x9999
PROJECT DATA
P#
X1
X2
X3
X4
X5
X6
X7 PROJ COST
1
100000.
22500.
1000.
500.
250.
250.
300.
350000.
2
500000.
3000.
1500.
600.
1000.
300.
200.
475000.
3
267000.
500.
500.
167.
167.
233.
267.
500000.
4
255000.
2500.
2750.
5000.
500.
250.
0.
600000.
01
325000.
250.
500.
100.
2000.
45.
500.
9999.
FIGURE 3. SAMPLE ARPCOM COMPUTER PROGRAM RUN
72
RUN ARPCOM
ARPCOM
09:08EDT
07/01/80
WELCOME TO ARPCOM
THIS INTERACTIVE COMPUTER PROGRAM CALCULATES BENEFIT - COST
RATIOS FOR PROJECT CONCEPTS. PLEASE SEE THE ARPCOM USER
GUIDE FOR COMPLETE OPERATING INSTRUCTIONS.
ENTER THE NUMBER OF PROJECTS TO BE PROCESSED, NOT
TO EXCEED FIFTY.
75
ENTER THE BENEFIT AND COST DATA USING THE FOLLOWING FORMAT:
PROJECT x7x1.x2.x3,x4.x5.X6.X7.PROJECT COST
(PUSH RETURN)
WHERE
X1 = MONETARY BENEFITS
X2 = ENHANCED QUALITY OF SERVICES
X3 = IMPROVED INDIVIDUAL CLIENT OUTCOMES
X4 = IMPROVED ADMIN. BASES FOR SERVICE PROVISION
X5 = IMPROVED POLICY BASES FOR REHABILITATION
X6 = INDIRECT BENEFITS, GIVEN PROJECT SUCCESS
X7 = INDIRECT BENEFITS, REGARDLESS OF PROJECT SUCCESS
PLEASE LIMIT EACH ENTRY TO EIGHT DIGITS.
PROJECT 17100000,22500,1000,500,250,250,300,350000
PROJECT 27500000,3000,1500.600,1000,300.200:475000
PROJECT 37267000,500,500,167.167:233.267:500000
PROJECT 47255000,2500,2750.5000,500,250.0,600000
PROJECT 57325000,250.500,100,2000,45x500,9999
PROJECT DATA
P#
X1
X2
X3
X4
X5
X6
X7 PROJ COST
1
100000.
22500.
1000.
500.
250.
250.
300.
350000.
2
500000.
3000.
1500.
600.
1000.
300.
200.
475000.
3
267000.
500.
500.
167.
167.
233.
267.
500000.
4
255000.
2500.
2750.
5000.
500.
250.
0.
600000.
01
325000.
250.
500.
100.
2000.
45.
500.
9999.
FIGURE 3. - SAMPLE ARPCOM COMPUTER PROGRAM RUN
72
IF ALL DATA ARE CORRECT, PUSH RETURN.
IF NOT, ENTER THE NUMBER OF A PROJECT CONTAINING
INCORRECT DATA. 75
RE-ENTER DATA FOR PROJECT .73250,500,250,1000,2000.45.500.300000
IF ALL DATA ARE CORRECT, PUSH RETURN.
IF NOT, ENTER THE NUMBER OF A PROJECT CONTAINING
INCORRECT DATA.?
PROJECT DATA
P#
X1
X2
X3
X4
X5
X6
X7 PROJ COST
1
100000.
22500.
1000.
500.
250.
250.
300.
350000.
2
500000.
3000.
1500.
600.
1000.
300.
200.
475000.
3
267000.
500.
500.
167.
167.
233.
267.
500000.
4
255000.
2500.
2750.
5000.
500.
250.
0.
600000.
5
325000.
500.
250.
1000.
2000.
45.
500.
300000.
IF ALL DATA ARE CORRECT, PUSH RETURN.
IF NOT, ENTER THE NUMBER OF A PROJECT CONTAINING
INCORRECT DATA. ?
IF LAMBDA IS COMPOSITE, ENTER THE NUMBER OF COMPONENTS, NOT
TO EXCEED EIGHT. IF LAMBDA IS NOT COMPOSITE, ENTER 1. ?1
ENTER THE SEVEN BENEFIT WEIGHTS.7100,40.50.30.20.10.10
1.000E+02 4.000E+01 5.000E+01 3.000E+01 2.000E+01 1.000E+01 1.000E+01
IF THE BENEFIT WEIGHTS ARE CORRECT, PUSH RETURN.
TO RE-ENTER THE WEIGHTS, ENTER 1.7
LAMBDA:
0.385
0.154
0.192
0.115
0.077
0.038
0.038
PROJECT BENEFIT DATA IN DOLLAR EQUIVALENTS
1 1.000E+05 2.000E+05 9.091E+04 1.500E+04 1.250E+04 4.167E+04 3.000E+04
2 5.000E+05 2.667E+04 1.364E+05 1.800E+04 5.000E+04 5.000E+04 2.000E+04
3 2.670E+05 4.444E+03 4.545E+04 5.010E+03 3.350E+03 3.883E+04 2.670E+04
4 2.550E+05 2.222E+04 2.500E+05 1.500E+05 2.500E+04 4.167E+04 0.
5 3.250E+05 4.444E+03 2.273E+04 3.000E+04 1.000E+05 7.500E+03 5.000E+04
PROJECTS IN ORDER OF DECREASING BENEFIT-COST RATIO
PROJECT
TOTAL BEN.
COST
B-C RATIO
5
5.3967E+05
3.0000E+05
1.799
2
8.0103E+05
4.7500E+05
1.686
1
4.9008E+05
3.5000E+05
1.400
4
7.4389E+05
6.0000E+05
1.240
3
3.9579E+05
5.0000E+05
0.792
FIGURE 3A. - SAMPLE ARPCOM RUN (Continued)
73
TO RE-RUN ARPCOM WITH NEW LAMBDA DATA, ENTER 1.
IF NOT, PUSH RETURN. 71
IF LAMBDA IS COMPOSITE, ENTER THE NUMBER OF COMPONENTS, NOT
TO EXCEED EIGHT. IF LAMBDA IS NOT COMPOSITE, ENTER 1. 72
ENTER THE SEVEN BENEFIT WEIGHTS FROM RATER 1.7100,50,3020101
????
?11111,111,1111
1.000E+02 5.000E+01 3.020E+06 7.770E+02 1.111E+04 1.110E+02 1.111E+03
ENTER THE SEVEN BENEFIT WEIGHTS FROM RATER 2.730,30,100,80,10,10,10
3.000E+01 3.000E+01 1.000E+02 3.000E+01 1.000E+01 1.000E+01 1.000E+01
IF ALL DATA ARE CORRECT, PUSH RETURN.
IF NOT, ENTER THE NUMBER OF THE INCORRECT COMPONENT.?1
ENTER THE SEVEN BENEFIT WEIGHTS FROM RATER 1.7100,40,50,30,20,10,10
1.000E+02 4.000E+01 5.000E+01 3.000E+01 2.000E+01 1.000E+01 1.000E+01
IF ALL DATA ARE CORRECT. PUSH RETURN.
IF NOT, ENTER THE NUMBER OF THE INCORRECT COMPONENT.?
LAMBDA 1:
0.385
0.154
0.192
0.115
0.077
0.038
0.038
LAMBDA 2:
0.111
0.111
0.370
0.296
0.037
0.037
0.037
ENTER THE RATER WEIGHTS.750,50
5.000E+01 5.000E+01
IF THE WEIGHTS ARE CORRECT, PUSH RETURN.
IF NOT, ENTER 1.
?
NORMALIZED RATER WEIGHTS:
0.500
0.500
COMPOSITE LAMBDA:
0.248
0.132
0.281
0.206
0.057
0.038
0.038
PROJECT BENEFIT DATA IN DOLLAR EQUIVALENTS
1 1.000E+05 2.672E+05 2.064E+05 4.152E+04 1.437E+04 6.346E+04 4.569E+04
2 5.000E+05 3.563E+04 3.096E+05 4.983E+04 5.747E+04 7.615E+04 3.046E+04
3 2.670E+05 5.939E+03 1.032E+05 1.387E+04 9.598E+03 5.914E+04 4.066E+04
4 2.550E+05 2.969E+04 5.675E+05 4.152E+05 2.874E+04 6.346E+04 0.
5 3.250E+05 5.939E+03 5.159E+04 8.305E+04 1.149E+05 1.142E+04 7.615E+04
FIGURE 3B. - SAMPLE ARPCOM RUN (Continued)
74
PROJECTS IN ORDER OF DECREASING BENEFIT-COST RATIO
PROJECT
TOTAL BEN.
COST
B-C RATIO
4
1.3596E+06
6.0000E+05
2.266
2
1.0591E+06
4.7500E+05
2.230
5
6.6809E+05
3.0000E+05
2.227
1
7.3865E+05
3.5000E+05
2.110
3
4.9940E+05
5.0000E+05
0.999
TO RE-RUN ARPCOM WITH NEW LAMBDA DATA, ENTER 1.
IF NOT, PUSH RETURN. ?
TO RE-RUN ARPCOM, ENTER 1. IF NOT, PUSH RETURN.?
PROGRAM STOP AT 1950
USED
18.67 UNITS
BYE
00036.72 CRU
0000.43 TCH
0007.29 KC
OFF AT 09:27EDT 07/01/80
FIGURE 3C. - SAMPLE ARPCOM RUN (Concluded)
75
GLOSSARY
Benefits - Direct or indirect advantages accruing to a defined set of persons
which include discounted monetary benefits and costs (negative benefits)
Benefit Cluster - A galaxy of benefit factors, each related in a defined manner
to a rather general system of benefit-oriented values
Benefit-Cost Analysis - An analytic technique for evaluating entities with
respect to their potential benefits and costs
Benefit-Cost Ratio - Net expected benefits divided by the sum of critical costs
Benefit Dimension - A general grouping of several related benefits or benefit
factors which then allows the research evaluator or administrator to select
specific benefits within the grouping for emphasis in a research evaluation
Benefit Factor - A class of benefits from research which can be further broken
down into specific and realizable benefits to individuals or groups
Critical Cost - The limiting portion of total costs which come from the
decision-maker's resources, such as a budget line-item for research
Discount Rate - The percentage reduction in future monetary benefits and costs
to make them compatible with current values
Direct Benefits - Advantages from a project which accrue directly to persons
Expected Net Benefits - Expected benefits, expressed numerically, with all
discounted costs subtracted from discounted benefits on a probable distribution
Indirect Benefits - Advantages from a project which accrue indirectly to persons
Monetary Benefits - Advantages from a project which can be expressed in money units
Multidimensional Parameter - A variable in an analysis which can be measured or
considered on more than one value scale and therefore of one or more dimensions
Non-Monetary Benefits - Advantages which accrue to either persons or groups but
which are measured or estimated on a non-monetary value scale
Parent Population - A group of individuals large enough to include all of one or
more smaller target populations or subpopulations
Prioritization - The process of ranking issues or projects according to relative merit
Probability of Success - The likelihood that a project will fulfill its stated
objectives
Probability of Utilization - The likelihood that project benefits will be used
as intended
Project Concept - A planning proposal which identifies research issues, highlights
research gaps, articulates research strategy, addresses information dis-
semination, and embraces cost estimations where research funding is limited
Ratio Scale - A scale where the zero point is prescribed and where only ratios
are meaningful
Research Project - An area of investigation that if funded may provide information
or technology which apparently fills a void in the state-of-the-art
Subpopulation - A subset of persons in a larger population (target population)
defined by identifiable common characteristics
Target Population - The total array of persons about which knowledge is to be
developed by a research project or demonstration program
Weighting Factor - A relative multiplying factor which indicates a chosen
assignment of relative importance for one parameter or value with respect
to others in the set. Usually, all weighting factors are made to sum to one.
76
REFERENCE LIST
Allen, K.H. First Findings of the 1972 Survey of the Disabled: General Characteristics.
Office of Research Statistics, Soc. Sec. Adm., 1977.
Cardus, D. & Thrall, R.M. An Overview: Health and Planning of Health Care Systems.
Preventive Med. 6:134-142, 1977.
Cardús, D., Hammons, D.B. & Thrall, R.M. Multiple Objective Benefit-Cost Modeling for
Decision Makers. Decision Information, Academic Press, pp. 73-93, 1977.
Cardus, D. & Thrall, R.M. The Concept of Positive Health & Planning of Health Care
Systems. "Health System Modeling and the Information System for the Coordination
of Research in Oncology." Proceedings of the HASA Biomedical Conf., Dec. 8-12, 1975.
D. D. Venedicton, Ed., pp. 211-233, 1977.
Cardús, D., Fuhrer, M.J. & Thrall, R.M. Quality of Life in Benefit-Cost Analysis of Rehab-
ilitation Research. Arch. Physical Med. & Rehab. In press.
Dudek, R.A. Human Rehabilitation Techniques: A Technology Assessment. Texas Tech.
Univ., pp. 1-56, 1975.
Fuhrer, M.J., Cardus, D., and Rossi, D. Judgements of the Potential Benefits of Rehabil-
itation Research. Arch. Phys. Med. & Rehab. 60:239-246, 1979.
Fuhrer, M.J., Cardús, D., & Thrall, R.M. Estimating Target Population Size for Proposed
Rehabilitation Research and Demonstration Projects. Arch. Phys. Med. & Rehab. In press.
Nagi, S.Z. An Epidemiology of Disability Among Adults in the U.S. Milbank Memorial
Fund Quarterly, 54:439-467, 1976.
National Center for Health Statistics: Impairments Due to Injury, U.S., 1971; Vital and
Health Statistics Series 10, No. 87; DHEW Publ. HRA 74-1514, Dec. 1973.
National Center for Health Statistics: Limitations of Activity and Mobility due to Chronic
Condition, U.S., 1972; Vital and Health Statistics Series 10, No. 96: DHEW Publ.
HRA 75-1523, Nov. 1974.
National Spinal Cord Injury Data Res. Center. Model Systems SCI Digest, Good Samaritan
Hospital, Phoenix, Ariz., Spring, 1979.
Noble, John H. Rehabilitating the Severely Disabled: The Foreign Experience. J. Health,
Politics, Policy & Law. 14:221-249, 1979.
Thrall, R.M. & Cardús, D. Benefit-Cost and Cost Effectiveness Analysis in Rehabilitation
Research Programs. Methods & Information in Med. 13:147- , 1974.
Thrall, R.M. Benefit-Cost Estimation, Alternative Requirements, Advantages and Disadvan-
tages. Computer Applications in Health Care Delivery. Symposia Specialists, Miami,
Fla., pp. 27-35, 1976.
Thrall, R.M. & Cardús, D. Benefit-Cost Modeling in the Presence of Multiple Decision
Criteria. "Health System Modeling & the Information System for the Coord. of
Research in Oncology." Proceedings of the IIASA Biomedical Conf., Dec. 8-12, 1975.
D. D. Venedicton, Ed., pp. 225-237, 1977.
Thrall, R.M., Cardús, D., & Fuhrer, M.J. Multicriterion Decision Analysis. Am. Assoc. for
Advancement of Science. In publication.
Urban Institute: Report of the Comprehensive Service Needs Study. Wash., June 23, 1975.
77
APPENDIX 1
THE ROLE OF THE BENEFIT-COST RATIO IN
THE SELECTION OF ALTERNATIVE COURSES OF ACTION
The concepts "benefit-cost" and "cost/effectiveness" have led
to considerable controversy in areas to which either or both have been
applied, and the health care area is no exception. In the sense that a
decision-maker weighs expected advantages and disadvantages before under-
taking a course of action, benefit-cost assessments are almost universally accepted,
as is the consideration of the most economical (cost/effective) way
to implement a decision once it is made. The controversy relates to the
acceptability of methods for measuring benefits, costs, and, in some appli-
cations, "effectiveness."
1.1 THE EXPECTED NET BENEFIT OF A RESEARCH PROPOSAL
The following discussion is concerned with: (1) the formal definition
of the expected net benefits (ENB) of a research proposal, (2) the uses of
benefit-cost ratios in the presence of budget constraints, and (3) the dis-
tinctive features of the AARPS benefit-cost model concerning the selection of
costs which appear in the denominator of the benefit-cost ratio.
To set the stage for benefit-cost comparisons, we first consider a
single research project P. If the research is successful and the results
implemented, there is an associated train of benefits and costs sometimes
running far into the future. Some of the costs are operational in nature
and can be directly associated with one or more of the benefits; such costs
can be interpreted as "negative" benefits. Other costs relate to funding
of the initial research itself.
78
If a discount rate of r has been selected, then the total benefit
B*(P) associated with P will have the form:
B*(P) = B₁*(P) +(1/1 +r)B₂*(P)+
+ [1.1]
where Bt*(P) is the total dollar value of benefits expected in year t and
N is the total time span of the project (in years). Sometimes, "r" is
called the social discount rate. Similarly, the total cost, C*(P), is:
C*(P) = C₁*(P) + (1/1 +t)C₂*(P)+
+ .+[1/(1 + r)N⁻¹]CN*(P) [1.2]
The difference
B(P) = B*(P) - C*(P)
[1.3]
is the expected net benefit (ENB) of the research project P.
It is difficult to justify undertaking a research project P unless
its ENB is positive. If we recognize that the measurement of benefits and
costs may not be precise, we might set some safety threshold T and require
B(P) > T, instead of merely B(P) >0. We may be faced with a set of
several alternatives (e.g., multiple research project choices) Pᵣ
,Pₙ
and be asked to select that or those which are best. If we renumber the
set in order of decreasing ENB SO that
B(P₁) ≥ B(P₂) ≥ ≥B(Pₙ)
[14]
then, except for a tie, the choice of P1 is clearly optimal.
More generally, if we could undertake some or all of the alterna-
tives, it seems reasonable to reject any for which the ENB is negative
(or below a selected threshold T) and in the absence of constraints
(other than operational costs) to consider favorably the acceptance and
implementation of all that remain.
79
1.2 BENEFIT-COST ANALYSIS WHICH CONSIDERS CONSTRAINTS
In the presence of additional constraints, the situation is different.
Suppose that the only constraint is a limitation on the number, S, of
alternatives to be accepted. If the top "s" alternatives are all accep-
table, we would maximize the total ENB by selecting P₁,..,P₃. Otherwise,
we would select P₁,...,P r' where r is chosen so that Pr is acceptable but
Pr + 1 is not.
A more common and important type of constraint is provided by a limi-
tation of one or more resources needed for implementation of the proposed
courses of action. For example, if P₁,...,Pₙ are research anddemonstra-
tion projects, then the total funds available to the decision-maker for
implementing such projects may be a binding constraint.
Suppose that to implement Pi, the amount of scarce resource needed
can be measured in cost terms as CR(Pi), which is part of the total cost
C*(Pᵢ). The constraint imposed by the scarce resources takes the form:
[1.5]
where C(S) measures the total amount of the scarce resource which is avail-
able and where, for each i (i=1,...,n) :
Zi=1 =
if alternative Pi is implemented, and
zi=0 =
if alternative Pi is rejected.
[1.6]
The decision problem is then to choose the Zi so as to maximize
the sum of the ENB's.
B(P₁)z₁+B(P₂)z₂ +...+B(Pn)Zn
[1.7]
subject to equations [1.5]and [1.6]. Strictly speaking, this conceptualization
is the integer programming exercise known as the "knapsack" problem* If
the bound C(S) is slightly flexible, the solution is to (1) arrange the Pi
in order according to the benefit cost-ratios:
R(Pᵢ) = B(Pᵢ)/CR(P₁)
[1.8]
"Danzig, D.B. Linear Programming and Extensions. Princeton Univ.Press,pp.517-520,196
80
beginning with the largest ratio, and (2) implement projects in order until
the total resource C(S) has been allocated.
In more detail, suppose that the alternatives have been numbered
so that:
R(P₁) R(P₂)≥ ≥R(Pₙ)
[1.9]
Let "k" be the largest integer for which
CR(P₁) + + CR(Pₖ) C(S),
[110]
thenset z₁ = ... = Zh, ²h+1 = ... = Zn = 0, where either h = k
or is k + 1 (i.e., small overruns are acceptable). This method is known as
"selection by the benefit-cost ratio."
In the situation where C(S) is inflexible (e.g., costs incurred in
emergency life-saving equipment), an exact solution of the maximization
problem can be found by using more sophisticated mathematical algorithms
(e.g., dynamic programming), but the naive selection of h = k gives a
feasible solution, which under most circumstances might be acceptable.
If there are other constraints, in addition to the budget limita-
tion, selection via the benefit-cost ratio may need to be replaced by more
comprehensive mathematical programming algorithms. However, if the addi-
tional constraints involve some lower bounds to funding of special classes
of research projects, the ratio can still be used with the following changes:
a. Arrange the projects in each special class in descending order of
their benefit-cost ratios
b. Select from the top of the list just enough projects to meet the
lower bound(s) of the additional constraint(s)
c. Return to the full set of projects and delete those already funded
d. Select from the top of those remaining until C(S) is exhausted.
81
1.3 THE ROLE OF THE DENOMINATOR IN THE BENEFIT-COST RATIO
There is a number of alternative formulations of the benefit-cost
ratio, and these are far from equivalent. The major source of variation
is in the selection of the cost which appears in the denominator. For a
research project P we would write
C*(P) = CO(P) + CR(P)
[1.11]
Here, CR(P) denotes the "cost of research" and is defined as the amount
of the current research budget C(S) that is required to fund P. The term
CO(P) refers to "other costs." This term includes set-up costs, opera-
tional costs and also downstream research funding for P. The traditional
benefit-cost ratio discussed in most economics texts is the quotient
B*(P)/C*(P)
[1.12]
of total expected benefits divided by total expected costs. This ratio
exceeds one if, and only if, B(P) (the ENB) is positive. The equation:
B(P)/C*(P) = [B*(P) - C*(P)/C*(P) = [B*(P)/C*(P)] - 1 [1.13]
shows that, for comparing projects, the ratio, ENB/total cost, is equiv-
alent to the traditional benefit-cost ratio. The ratio we have used before:
B(Pᵢ)/CR(Pᵢ)
[1.8 ]
has a much smaller denominator and hence a greater value. We now provide
some numerical examples to illustrate why[1.8]is considered superior to
[1.12]or [1.13]
Clearly, if two projects P₁ and P₂ differ only in some inessential
details, they should be assigned the same (or nearly the same) value
measurement. To illustrate this concept in more detail, suppose that
some proposed project P₁ has total expected benefits B*(P₁) and costs C*(P₁).
Let P₂ differ from P₁ only in that it includes (a) borrowing D dollars for
one year at interest rate "r" equal to the social discount rate (see [1.1] above)
and (b) investing those D dollars with a guaranteed return rate, also equal
82
to r. We may describe (a) and (b) as "wash items." Clearly, projects
P₁ and P₂ are of identical merit.
Now, suppose that B*(P1) = $1,000,000; CR(P1) = C*(P₁) = $200,000;
and C = $10,000,000. Then, since the interest rate is equal to the social
discount rate, the total expected benefit for P₂ is B*(P₂) = $11,000,000
and its cost is C*(P₂) = $10,200,000.
The ratios then are B*(P₁)/C*(P₁) = 5 and B*(P₂)/C*(P₂) = 1.078.
Clearly, in this case, the ratio of all benefits to all costs is not an
appropriate measure for comparing the merits of the two projects.
By contrast, for the ENB's we have B(P₂) = $800,000 and
CR(P1) = CR(P₂) = $200,000 so that R(P1) = R(P₂) = 4, which agrees with
our intuitive assessment that P₁ and P₂ have equal merit.
However, there is more to the story. Consider a third project P₃
with B*(P₃) = $1,300,000 and CR(P₃) = C*(P₃) = $500,000. Now, B(P1) =
B(P₃) = $800,000, so that if one used ENB as the sole criterion of merit,
projects P₁ and P₃ would rate equally. Yet, to achieve this common ENB,
project P₃ uses over twice as much of the research budget as does P₁.
This fact is reflected in the benefit-cost ratios, since R(P1) = 4,
whereas R(P₃) = 1.6. In this case, the traditional ratios 5 and 2.6
would also reflect the fact that, per dollar of research funding, P1
is more productive than is P3.
Our use of the ratio ENB is proper if the overall objective is to
use the total available current research funds to maximize the total
ENB. The traditional benefit-cost ratio does not do this.
83
1.4 ILLUSTRATIVE NUMERICAL EXAMPLES
Table 1.1 contains data for examples which further illustrate
various features of the benefit-cost ratio. The first three cases have
already been discussed.
Projects P₄ and P5 differ only in the distribution of costs
between CO and CR. The larger ratio R for P5 is responsive to its
lower research budget.
Let P₆ be a research project and P₇ a project modification
based on the same research but involving additional downstream benefits
and costs. For example, the addtional feature could be individual
purchase and use of a newly designed wheelchair where the total added
benefit ($600,000) was 1.5 times the added cost ($400,000). The tradi-
tional ratio B*/C* penalizes P₇ and obscures its higher ENB per research
dollar.
Projects P₈, P9, and P₁₀ all have the same research costs but
differ in the other costs and the resulting benefits. Project P₉ comes
out on top with either benefit-cost ratio but they interchange the posi-
tions of P₈ and P₁₀.
Projects P₁₂ toP 15 are all variants of P₁₁; their rank under B/CR is
P₁₃, P₁₅,P₁₁,P₁₄,P₁₂
and under B*/C* is
P₁₃P₁₄₁,P₁₁,P₁₂,P₁₅
the low position of P 15 under B*/C* once again illustrates the fact that
the traditional ratio not only penalizes projects with wash items but also
those where additional downstream expenditures can result in desire-
able downstream benefits.
The last two projects, P₁₆ and P₁₇, in table 1.1 illustrate a
situation in which neither ratio is completely satsifactory.
84
P₁₆ is a project which lasts only one year, whereas P₁₇
is the same project spread over two years (in this example we assume
that the discount rate is zero, with half of the research funding in each
year). The deferred half of the research costs are accounted for as
"other costs." The result is that the one-year project is heavily penal-
ized under the ratio B/CR. On the other hand, the ratio B*/C* gives no
credit for deferring some of the costs until the following year. It is not
difficult to modify the definition of CR so that it represents the present
value of all research costs for a project. One would then use total (dis-
counted) research costs for determining the ratios for new research
starts and first-year costs to determine the cut-off point.
If the funding process consists of first providing for continuing
projects and then taking for C(S) what remains from the total research
appropriation, then the use of current-year research costs for CR permits
maximization of the total ENB for projects funded in the current year.
However, if the funding process is done under a longer range view which
has as its objective to maximize ENB over a period of years, then the in-
clusion of total research costs in CR may be preferred. The choice among
the definitions of CR is a policy decision which should not be made by
the modeler. If, in fact, almost all projects have three-year durations,
then it makes no essential difference which alternative is followed. We
could use current year costs for CR and take care of a handful of one-
year projects by giving their ratios an enhancing multiplier.
It is important to recognize that the benefit-cost ratio
principle is flexible enough to model whichever variant of research
cost accounting turns out to be most appropriate, and if, as time
progresses, changes are needed, they can be readily incorporated.
85
TABLE 1.1
BENEFIT-COST DATA FOR NUMERICAL EXAMPLES
Project
B*
*
B
CR
B/CR
B*/C*
P₁
1,000,000
200,000
800,000
200,000
4
5
P₂
11,000,000
10,200,000
800,000
200,000
4
1.078
P₃
1,300,000
500,000
800,000
500,000
1.6
2.6
P₄
100,000
50,000
50,000
30,000
1.67
2
P5
100,000
50,000
50,000
20,000
2.50
2
P6
600,000
200,000
400,000
100,000
4
3
P₇
1,200,000
600,000
600,000
100,000
6
.2
P₈
400,000
200,000
200,000
100,000
2
2
P₉
600,000
250,000
350,000
100,000
3.5
2.4
P₁₀
600,000
350,000
250,000
100,000
2.5
1.71
2,000,000
800,000
1,200,000
480,000
2.5
2.5
P₁₁
2.100,000
850,000
1,250,000
530,000
2.36
2.47
P₁₂
2,250,000
850,000
1,400,000
530,000
2.64
2.65
P₁₃
2,150,000
850,000
1,300,000
530,000
2.45
2.53
P₁₄
2,350,000
1,000,000
1,350,000
530,000
2.55
2.35
P₁₅
500,000
200,000
300,000
100,000
3
2.5
P₁₆
500,000
200,000
300,000
50,000
6
2.5
P₁₇
86
APPENDIX 2
BENEFIT CLUSTERING
As stated in the body of this report, an original comprehensive list
of benefits numbering 243 separate items was obtained from administrators
of the Social and Rehabilitation Services (SRS) agency in 1973. This list
was obtained at the beginning of a Delphi-like study in which 50 individuals
were asked to participate. Of those 50, 42 (84 %) responded to the initial
query. The result of the preliminary benefit query produced 119 personal
benefits and 124 non-personal benefits, as shown in figures 2.1 and 2.2.
Clearly, comprehensive as they were, the two lists contained an
unwieldy number of items to incorporate into a benefit-cost analysis di-
rectly, so a data-reduction process was undertaken in a later stage of the
Delphi query. In what was the beginning of a "clustering" process, the
participants identified similarities among the items within each list.
The similarity judgements were then submitted to Johnson's hierarchical
cluster analysis to identify groups of similar benefits within each list.
The process starts with considering each element as a cluster and then
merges those elements which are closest according to the predefined
metric. The next closest elements or clusters are then identified and
merged to form new clusters. The process can be stopped at any stage and
continued until all elements or clusters have been merged into a single
cluster containing all of the original elements.
In the initial Delphi-type study, the analysis reduced the original
lists (119 and 124) to a total of 46 clusters, 23 each for personal and
non-personal benefits. These two sets of 23 "benefit factors" are
also shown in figures 2.1 and 2.2.
In a later Delphi-type study, participants were, first asked to
judge each of the 46 benefit factors to determine if any should be
87
deleted as superfluous or not as meaningful as others on the lists.
Only two items were removed as being of less importance (Child
Welfare and Ameliorating Societal Disturbance) in preparation for
the final clustering (which yielded nine clusters for each category,
personal and non-personal). These 18 clusters represent the far-
right columns in figures 2.1 and 2.2. They were considered as
"second order" by the AARPS team, whereas the "first-order"
benefits were the total of 46 benefit factors described earlier.
A final query consisted of ordering the 18 second-order
benefit clusters according to relative importance. Table 2.1 shows
the final order produced from this query and also shows the mean
ratings and standard deviations produced by 96 participants during
this phase.
Because the eventual list of 18 benefit clusters was still
impractical in size to be used with facility in a benefit-cost analysis
using the AARPS model, the final clustering results were reassessed
using an alternative clustering criterion, one yielding fewer, more
generic clusters. Figure 2.3 shows how the 18 clusters were finally
reduced to five benefit categories, all of which applied to either of
the three benefit terms in the model (B₁, Bs, and BF). Table 2.II
indicates which of these final five categories applies to which of
the three benefit terms.
It was recognized that all of the 18 clusters mentioned
previously could have monetary benefits under certain circumstances,
so lines of connection are drawn in figure 2.3 between the items in
the list of 18 clusters and the monetary benefit dimension (no. 1,
shown second at the far right). Since the BF and Bs terms are
unique in the model, two benefit dimensions were created to address
these terms separately, thereby reducing the final list of what could
be considered 15 (five for each of the three benefit terms in the model)
to a total of seven. This total of seven make up the contents of table I
in the body of this report (page 21), and they are the seven benefit
dimensions ultimately intended by the AARPS team to be used with its
benefit-cost model (eq. [1]). Thus, the completion of a rather elaborate
88
clustering process over a period of several years resulted in reducing
a list of 243 very specific potential rehabilitation-research benefits
to a more practical number of seven benefit dimensions to facilitate
addressing each of the multi-dimensional benefit terms in the AARPS
benefit-cost analysis model.
89
TABLE 2.1
IMPORTANCE RATINGS FOR SECOND-ORDER BENEFITS *
Consensus
Second-Order Benefit Factors
Mean
Standard
Order
Rating
Deviation
1.
Enhancing quality and accessibility of services
86.80
+18.87
2.
Enhancing individual coping skills
79.22
19.45
3.
Minimizing functional limitations and personal
disability
78.15
21.36
4.
Improving personal vocational status and
material well-being
76.33
21.62
5.
Enhancing effectiveness of service providers
76.21
20.77
6.
Improving program development and evaluation
71.98
19.83
7.
Expanding knowledge base
69.49
22.19
8.
Encouraging individual's social participation
66.65
23.65
9.
Improving program performance and performance
measures on berefit-cost criteria
65.51
26.41
10.
Fostering consumer involvement
65.54
23.94
11.
Facilitating societal change
59.42
26.84
12.
Improving legislative impact and coordination
of government entities
57.49
27.35
13.
Improving physical environment
57.44
26.91
14.
Developing and communicating policies, plans
and procedures
56.73
25.20
15.
Facilitating administrative flexibility and
improvement
54.77
26.71
16.
Containing personal costs and need for services
52.83
23.01
17.
Promoting generalizability of services
50.26
26.39
18.
Containing institutionalization
49.93
26.38
Mean value of all scores
65.26
22.25
*From Fuhrer, Cardus, & Rossi (1979).
90
TABLE 2.II.- BENEFIT DIMENSION APPLICABILITY TO TERMS OF MODEL
Benefit-Model Term
Benefit Dimension
B1
Bs
BF
1. Enhancing Service Quality*
X
-
-
2. Monetary Benefits to Individual*
X
X
X
3. Improving Individual Client Outcomes
X
-
-
4. Improving Administration Bases for
Service Provision
X
-
-
5. Improving Public Policy Bases for Rehab-
ilitation
X
-
-
6. Indirect Benefits, Given Project Success
-
X
-
7. Indirect Benefits, Regardless of Success
-
I
X
*NOTE: The benefit dimension, monetary benefits to individuals, was later
made more general to include all three benefit terms and is now referred
to as simply Monetary Benefits, applying to any benefits which can be
expressed in monetary units. Because this benefit dimension was seen to
apply to all three terms of the model, it was interposed with item 1 above
in the final list (as presented in table I in the text) so that monetary
benefits would be listed first.
91
DECREASED NUMBER OF CHILDREN IN FOSTER CARE
MORE EARLY ADOPTION PLACEMENTS
FIGURE 2.1-PERSONAL BENEFIT FACTORS
FEWER SCHOOL DROP-OUTS
CHILD FARE
IMPROVED WELFARE OF CHILDREN
EARLIER RETURN OF CHILDREN TO OWN HOME
MORE REALISTIC EXPECTATION OF SERVICE EFFECTIVENESS
PERCEIVED EQUITY OF QUALITY OF SERVICES AVAILABLE
IMPROVED USAGE OF PREVENTIVE SERVICES
IENT EXPECTATION
SKILLS FOR OBTAINING SERVICES
OF SERVICES
KNOWLEDGE FOR OBTAINING SERVICES
IMPROVED UNDERSTANDING OF VALUE OF SERVICES
FOSTERING CONSUMER
INCREASED AWARENESS OF AVAILABILITY OF SERVICES
INVOLVEMENT
INCREASED UNDERSTANDING OF RIGHTS/CELIGATIONS
UNDERSTANDING RIGHTS
AND ORI IGATIONS
MORE CHANCES FOR CLIENT PROGRAM PARTICIPATION
INCREASED AWARENESS OF OPPORTUNITIES FOR FEEDBACK
CONSUMER PARTICIPATION
LESS INCONVENIENCE IN ESTABLISHING ELIGIBILITY,
IMPROVED ACCESS TO FAMILY COUNSELING
LESS PERSONALLY DISRUPTIVE SERVICE
BETTER COORDINATION OF SERVICES
AVAILABILITY OF NEW SERVICES
IMPROVED QUALITY OF SERVICES
MORE HUMANE SERVICES
QUAL ITY OF SERVICE
REFERRAL COUNSELING
JVERY
BETTER QUALITY PROVIDERS
IMPROVED ACCESS TO SERVICES
IMPROVED CONTINUITY OF SERVICES
IMPROVED DURABILITY OF SERVICE RESULT
ENHANCING CUM ITY AND
IMPROVED INDIVIDUALIZATION OF SERVICES
ACCESSIBILITY SERVICES
INCREASED COMMUNITY AMARENESS OF SERVICE PROGRAMS
INCREASED ACCESS TO EDUCATION
COUNSEL ING. EDUCATION
INCREASED ACCESS TO TRAINING
AND TRAINING
IMPROVED ACCESS TO COUNSELING
REHABILITATION OF INDIVIDUALS NOT NOW SUITABLE
EXPANDING BENEFITS OF
IMPROVED POTENTIAL FOR REHABILITATION
ABILITY TO PERCEIVE/ACT ON CUES FOR DYSFUNCTIONS
SERVICES
PREVENTION OF INSTITUTIONAL PLACEMENT
CONTAINMENT OF
CONTAINING
POSSIBILITY OF DE-INSTITUTIONALIZATION
INSTITUTIONAL IZATION
INSTITUTIONALIZATION
REDUCED NEED FOR PUBLIC SUPPORT
DECREASED NEED FOR SERVICE
SERVICE CONTAINMENT
CONTAINING PERSONAL COSTS
DECREASED TIME IN SERVICE PROGRAM
AND NEED FOR SERVICES
DECREASED COST OF SERVICE
CONTAINMENT OF
CLAIMS PAID MORE RAPIDLY
PERSONAL COST
REDUCTION IN PERSONAL COST OF ONGOING TREATMENT
FREE OTHER HOUSEHOLD MEMBERS FOR OTHER PURPOSES
IMPROVED ABILITY TO CARE FOR CHILDREN
IMPROVED PARENTAL FUNCTIONING
BETTER FAMILY RELATIONSHIPS
FAMILY FUNCTIONING
FAMILY BETTERMENT
AND STABILITY
IMPROVED FAMILY FUNCTIONING
BETTER MARITAL RELATIONSHIP
PREVENTION OF FAMILY BREAK-UPS
GREATER NUMBER OF PERSONAL SOCIAL CONTACTS
IDENTIFICATION WITH ADVOCACY GROUP
IMPROVED INTERPERSONAL RELATIONS
IMPROVED GROUP RELATIONS
ENCOURAGING INDIVIDUAL'S
MORE SOCIALLY OUTGOING
SOCIAL INVOLVEMENT
SOCIAL PARTICIPATION
INCREASED SOCIAL COMPETENCE
INCREASED SOCIAL MOBILITY-
MORE COMPENITY PARTICIPATION
GREATER VARIETY OF PERSONAL SOCIAL CONTACTS
.CCESS TO DESIRED ROLES (MARRIAGE,CHILDREN,ETC)
PERSONAL -SOCIAL
FULFILLMENT
BETTER HOUSING
BETTER TRANSPORTATION
BETTER PHYSICAL
IMPROVING PHYSICAL
ENVIRONMENT
IMPROVED SATISFACTION WITH LIFE
ENVIRONMENT
INCREASED SELF-RESPECT
FEELING OF WELL-SEING
GREATER CONTENTMENT
IMPROVED SELF-IMAGE
INCREASED HAPPINESS
SELF-ACTUALIZATION
PERSONAL WELL-BEING
SELP-UNDERSTANDING
AND IDENTITY
INCREASED FREEDOM
MORE CONTROL OVER FUTURE
COMFIDENCE TO PARTICIPATE
MORE ENJOYMENT OF LEISURE
SENSE OF NOT BEING ANONYMOUS
INCREASED EMOTIONAL STABILITY
INCREASED POTENTIAL FOR SELF-DEVELOPMENT
CLIENT SELF-PROPELLING
CLIENT INITIATIVE
ENHANCING INDIVIDUAL
ABILITY TO CHANGE ENVIRONMENT
COPING SKILLS
INCREASED ABILITY TO PLAN EFFECTIVELY
ADAPTIVE BEHAVIOR
RELIEF FROM STRESSES IMPAIRING SUCCESSFUL FUNCTIONING
BETTER INSIGHT INTO IMPROVED BEHAVIOR
INSIGHTS AND BEHAVIOR BETTER LINKED
IMPROVED COGNITIVE FUNCTIONING
COPING BEHAVIOR
BETTER INTELLECTUAL SKILLS
INCREASED ABILITY FOR PERSONAL PRODUCTIVITY
IMPROVED CAPABILITY FOR SOLVING ONGOING PROBLEMS
PERSONAL USAGE OF EFFECTIVE REGIMENS FOR SELF-CARE
INCREASED ABILITY TO PERFORM HOMEMAKING SERVICES
ABILITY FOR DEAF TO USE TELEPHONE
INCREASED PHYSICAL MOBILITY
PERSONAL FUNCTION OF
RESTORATION OF FUNCTION
ABILITY TO WORK AGAIN-
HANDICAPPED
ABILITY TO USE HANDS AGAIN
IMPROVED ABILITY FOR SELF-CARE
MINIMIZING FUNCTIONAL
ABILITY TO COMLETE SCHOOL FOR HANDICAPPED
INITATIONS
IMPROVED PERSONAL PRACTICES FOR HEALTH MAINTENANCE
PERSONAL DISABILITY
DECREASED DISABILITY DAYS
DECREASED ACCIDENTS
CONTAINMENT OF
IMPROVED HEALTH-
PERSONAL ILLNESS
DECREASED DAYS OF ILLNESS
DECREASED PAIN AND ANGUISH
SAVINGS IN THE BANK (OUT OF DEBT)
INCREASED INCOME
PERSONAL ECONOMIC
SELF-SUPPORT
IMPROVEMENT
ASSURANCE OF CONTINUATION OF SELF-SUPPORT
IMPROVED WORK AND VOCATIONAL SKILLS
INCREASED ABILITY TO WORK
IMPROVING PERSONAL VOCA-
INCREASED JOB MOBILITY
INDIVIDUAL VOCATIONAL
TOTAL STATUS AND
INCREASED JOB SATISFACTION
SUFFICIENCY
MATERIAL WELL-BEING
INCREASED MOTIVATION TO WORK
NOREASED EMPLOYMENT OPPORTUNITIES
MPROVED QUALITY OF LIFE FOR HANDICAPPED
MPROVED STANDARD OF LIVING
IMPROVED QUALITY OF LIFE
MATERIAL CLIN ITY OF
IMPROVED LIVING CONDITIONS
LIFE
IMPROVED ACHIEVEMENT OF INDEPENDENT LIVING GOALS
92
TRANSFERABILITY TO OTHER TARGET GROUPS
PREVENTION OF COMMUNITY DISINTEGRATION
FIGURE 2.2-NON-PERSONAL BENEFIT FACTORS
IMPROVED STATUS OF POPULATION GROUPS
IMPROVED URBAN/RURAL LIFE
IMPROVED ENVIRONMENT
REDUCED CRIME RATE
REDUCED ALIENATION
ENHANCE SOCIAL JUSTICE
INCREASED SOCIAL MIXING
AMELIORATING SOCIETAL
REDUCE SOCIAL DISRUPTIONS
DISTURBANCES
REMOVE BARRIERS TO ADOPTION
MORE EFFECTIVE USE OF HUMAN POTENTIAL
REDUCE DEPENDENCY OF POPULATION GROUPS
LESS ADVERSE SOCIAL PRESSURE ON RECIPIENTS
IMPROVED PROGRAM RELEVANCE TO TARGET POPULATIONS
TRAIN AND INFORM STAFF RESPONSIBLE FOR PROGRAMS
BROADER STAFF UNDERSTANDING
BETTER MIDDLE MANAGEMENT SELECTION AND TRAINING
IMPROVED STAFF TRAINING
INFORMATION FOR MANPOWER NEEDS, DISTRIBUTION, UTILIZATION
IMPROVED EFFICACY OF WHAT PRACTITIONERS/PROFESSIONALS DO
ENHANCING EFFECTIVENESS OF SERVICE PROVIDERS
ESTABLISH REQUIREMENTS FOR UTILIZATION OF NEW PROCEDURES
IMPROVED EFFICIENCY OF PRACTITIONERS/PROFESSIONALS
IMPROVED PROVIDER'S
IMPACT OF PROGRAM ON INSTITUTIONAL AND INDIVIDUAL PROVIDERS
EFFECTIVENESS
ESTABLISH REQUIREMENTS FOR USE OF NEW DEVICES BY PRACTITIONERS
ADVANCING THE FRONTIERS OF SCIENTIFIC KNOWLEDGE
STIMULATION OF FURTHER NEEDS FOR RESEARCH
IMPROVED RESEARCH CAPABILITY-
ADAPTION OF NEW SOCIAL SCIENCE THEORY
EXPANDED R&D POTENTIAL
METHODOLOGY FOR MEASURING SOCIAL CHANGE
SERENDIPITOUS REVELATIONS OF NEW BENEFITS
EXPANDED DEMONSTRATION OF VALUE OF RESEARCH
CONTRIBUTIONS TO DEVELOPMENT OF SOCIAL SCIENCES,DATA PROCEDURES
INFORMATION SUGGESTING CHANGES NEEDED TO REALIZE GOALS
INCREASED DATA BASE FOR PROGRAM ANALYSIS
EXPANDING KNOWLEDGE BASE
IMPROVED INFORMATION
INFORMATION FOR PROGRAM MODIFICATION
SYSTEMS
BETTER MANAGEMENT INFORMATION SYSTEM YIELDING TIMELY
AND RELEVANT ADMINISTRATIVE DECISION SUPPORTS
UTILIZATION OF COMPUTER TECHNOLOGY
INCREASE QUALITY OF STATISTICS
IMPROVED DATA QUALITY
SUGGEST NEW ORIENTATIONS TO ISSUE ANALYSIS
VERIFICATION/DETERMINATION OF BENEFIT-ASSUMPTION VALIDITY
BASIS FOR ASSESSING POTENTIAL SOCIETAL IMPACT OF PROGRAMS
VALIDATION OF BENEFITS
INCREASED COST/EFFECTIVENESS OF PROGRAMS
BETTER TECHNIQUES FOR DETERMINING COSTS/BENEFITS OF PROGRAMS
COST/EFFECTIVENESS AND
BENEFIT/COST
INCREASED EFFICIENCY IN PAYMENT OF BILLS
IMPROVING PROGRAM PERFORMANCE AND PERFORMANCE
MORE EFFICIENT UTILIZATION OF RESOURCES
MEASURES ON BENEFIT-COST CRITERIA
USING TAX DOLLARS MORE EFFECTIVELY
IMPROVED FINANCIAL MANAGEMENT
IMPROVED PRODUCTIVITY OF
SAVING TAX DOLLARS
SPENT TAXES
FEWER INSTANCES OF ABUSE AND WASTE
REDUCTION IN COSTS PER UNIT OF SERVICE
IMPROVED TECHNOLOGY TO MINIMIZE EXPENDITURES/MAXIMIZE SERVICES
IMPROVED TECHNIQUES FOR THERAPEUTIC/SERVICE INTERVENTION
IMPROVED ORGANIZATION OF SRS SERVICE DELIVERY
INNOVATIONS IN SERVICE DELIVERY-
SERVICE PROCESS REFINEMENT
REDUCED TIME FOR SERVICE DELIVERY
PROVIDE FOLLOW-UP SERVICES FOR DEVELOPPENTALLY DISABLED
ELIMINATE NON-EFFECTIVE PROGRAMS
DETERMINE PROGRAM MALFUNCTION
PROGRAM CONTAINMENT
EVENTUAL REDUCTION OF PROGRAMS NEEDS
BETTER MEASURES TO EVALUATE SERVICE EFFECTIVENESS
IMPROVED TECHNICAL EVALUATION OF PROGRAMS
BETTER MEASURES OF PROGRAM OUTPUT
IMPROVED EVALUATION OF
MEASURES OF DIFFERENTIAL CASE DIFFICULTY
PROGRAMS
IMPROVING PROGRAM DEVELOPMENT & EVALUATION
ESTABLISHMENT OF MEASURES OR INDICATORS
COMPREHENDING IMPLICATION OF EXISTING/POTENTIAL PROGRAMS
IMPROVED UNDERSTANDING OF PROGRAM OPTIONS
USEFUL TESTING OF ALTERNATIVE APPROACHES
PROGRAM IMPROVEMENT
BETTER JUSTIFICATION OF RELATIVE PROGRAMS
STRATEGIES
ESTABLISH MORE EXPLICITLY PROCEDURES FOR PROGRAM PRIORITIES
CLARIFY THE MAGNITUDE AND SOLVABILITY OF SRS PROGRAM ISSUES
KNOWLEDGE OPTIMIZING SERVICE PROGRAMS IN DIFFERENT SETTINGS/CONDITIONS
UNDERSTANDING HOW TO CONSTRUCT MORE EQUITABLE PROGRAMS
MORE EFFECTIVE SEQUENCING OF PROGRAM DEVELOPMENT
IDENTIFY APPROPRIATE SCOPES OF PROGRAMS
PROGRAM IMPROVEMENT TACTICS
MORE FUTURE-ORIENTED PROGRAM DEVELOPMENT
RESOURCES REQUIRED FOR EFFECTIVE PROGRAM IMPLEMENTATION
BETTER UNDERSTANDING OF MOTIVATION AND GROUP PROCESSES
IDENTIFY WAYS OF DEALING WITH SOCIAL PROBLEMS
IDENTIFY FACTORS IN SOCIAL CHANGE
MEDIATING SOCIAL CHANGE
IDENTIFY CAUSES OF SOCIAL PROBLEMS
BETTER INTEGRATION OF COMMUNITY RESOURCES
FACILITATING SOCIETAL CHANGE
BETTER PUBLIC UNDERSTANDING OF PROGRAMS
IMPROVED IMAGE OF PROGRAMS
INCREASED CONSUMER PARTICIPATION
PUBLIC ACCEPTANCE
OBTAIN COMMUNITY SUPPORT OF PROGRAMS
KNOWLEDGE OF HOW DIFFERENT POPULATIONS REACT TO VARIOUS SERVICES
GENERAL IZABILITY OF SERVICES
PROMOTING GENERALIZABILITY OF SERVICES
TRANSFERABILITY TO OTHER TARGET POPULATIONS AND SUBPOPULATIONS
EFFECTIVE COMMUNICATION OF PROGRAM OUTCOMES TO POLICY MAKERS
INFORMATION DISSEMINATION
EXTENSION OF KNOWLEDGE TO INDIVIDUALS/AGENCIES NOT COVERED IN PROGRAMS
FEEDBACK INFORMATION TO ASSIST IN PLANNING
IMPROVED LONG-RANGE PLANNING-
PLANNING PROCESSES
IMPROVED CAPACITY FOR BUILDING, PLANNING, ETC.
DEVELOPING & COMMUNICATING POLICIES. PLANS.
AND PROCEDURES
KNOWLEDGE ABOUT BENEFITS OF ALTERNATIVE POLICY CHANGES
INFORMATION ON AND METHODS TO IMPLEMENT POLICY CHANGES
ESTABLISH POLICY RECOMMENDATIONS FOR IMPROVING SERVICES
POLICY REFINEMENT
ESTABLISH POLICY RECOMMENDATIONS FOR IDENTIFYING TARGET GROUPS
INCREASED AGENCY OPENNESS TO SELF-EVALUATION
INCREASED AGENCY OPENNES TO CHANGE
SIMPLIFY ADMINISTRATION
BETTER ADMINISTRATION
ADMINISTRATIVE FLEXIBILITY
CACILITATING AGMINISTRATIVE FLEXIBILITY
REDUCE PROCEDURAL COMPLEXITY
AND IMPROVEMENT
AND IMPROVEMENT
MANAGEMENT DECISION-MAKING SUPPORT
IMPROVED EXECUTIVE UNDERSTANDING OF PROGRAMS
ESTABLISH LEGAL BASIS FOR HAHDICAPPED'S NEEDS(HOUSTNG,RECREATION.ETC)
LEGISLATIVE BASIS FOR PROGRAM IMPLEMENTATION ON A NATIONAL SCALE
IMPROVED CONGRESSIONAL UNDERSTANDING OF PROGRAMS
IMPROVED JUDICIAL UNDERSTANDING OF PROGRAMS
NEW LEGISLATION TO ALTER OLD PROGRAMS
NEW LEGISLATION TO ALTER NEW PROGRAMS
ADVOCACY AND PROTECTIVE LEGISLATION
REVISE OBSOLETE LEGISLATION
ELIMINATE LEGISLATIVE GAPS/LOOPHOLES
EGISLATIVE IMPACT
SIMPLIFICATION OF CRITICAL LEGISLATION
CONSOLIDATE MAJOR LEGISLATIVE AUTHORITY
INFORMATION AND METHODS TO IMPLEMENT LEGISLATION
INTERACTIONAL EFFECTS AMONG/BETWEEN LANS/POLICIES
RATIONAL INTEGRATION OF POLICY/LEGISLATION WITH RELATED PROGRAMS
IMPROVING LEGISLATIVE IMPACT & COORDINATION
BETTER MEANS TO STRUCTURE LEGISLATION BASED ON MEASURED NEEDS/BEMEFITS
OF COVERNMENT ENTITIES
RELATIVE EFFECTIVENESS OF STATE ADMINISTRATIVE ORGANIZATIONS
ESTABLISH STANDARDS FOR STATE AGENCIES
FEDERAL LINKS WITH
PROVIDING GUIDANCE TO OTHER COUNTRIES
OTHER GOVERNMENTS
BETTER FEDERAL-STATE,STATE-LOCAL,FEDERAL-LOCALETC.RELATICNS
DIFFERENTIATE AMONG FEDERAL, FEDERALYSTATE, AND STATE RESPONSIBILITIES
93
FIGURE 2.3-FINAL BENEFIT CLUSTERING
FOSTERING CONSUMER INVOLVEMENT
ENHANCING QUALITY AND
II. ENHANCED QUALITY OF SERVICES
ACCESSIBILITY OF SERVICES
CONTAINING INSTITUTIONALIZATION
I. .MONETARY BENEFITS
CONTAINING PERSONAL COSTS AND
NEED FOR SERVICES
Personal Benefit Factors
ENCOURAGING INDIVIDUAL'S
SOCIAL PARTICIPATION
IMPROVING PHYSICAL ENVIRONMENT
ENHANCING INDIVIDUAL COPING SKILLS
III. IMPROVED INDIVIDUAL CLIENT OUTCOMES
MINIMIZING FUNCTIONAL LIMITATIONS
AND PERSONAL DISABILITY
I
IMPROVING VOCATIONAL STATUS
AND MATERIAL WELL-BEING
ENHANCING EFFECTIVENESS OF
SERVICE. PROVIDERS
-
EXPANDING KNOWLEDGE BASE
I
IMPROVING PROGRAM PERFORMANCE AND
PERFORMANCE MEASURES ON B-C CRITERIA
IV. IMPROVED ADMINISTRATIVE BASES
FOR SERVICE PROVISION
IMPROVING PROGRAM DEVELOPMENT
Non-Personal Benefit Factors
AND EVALUATION
FACILITATING SOCIETAL CHANGE
PROMOTING GENERALIZABILITY
OF SERVICES
DEVELOPING AND COMMUNICATING
POLICIES, PLANS AND PROCEDURES
V. IMPROVED POLICY BASES FOR
REHABILITATION
FACILITATING ADMINISTRATIVE
FLEXIBILITY AND IMPROVEMENT
IMPROVING LEGISLATIVE IMPACT AND
COORDINATION OF GOVERNMENTAL ENTITIES
VI. INDIRECT BENEFITS, GIVEN
PROJECT SUCCESS
VII. INDIRECT BENEFITS, REGARDLESS
OF PROJECT SUCCESS
94