Ask the Scholar
Document scope · 1 page
Scholar
Ask about this object, its catalog metadata, its source description, or the page inventory.
For page-specific OCR and visual context, open one of the page chats.
Scholar Source Context
Document identity
localId
351002347
label
"A Resource Handbook for Vocational Rehabilitation" [1974]
core
doc
dtoType
document
citationUrl
pageCount
1
Source metadata
id
351002347
contentType
document
title
"A Resource Handbook for Vocational Rehabilitation" [1974]
citationUrl
collections
Lex Frieden Collection: Records on Disability Rights
Printed Materials
imageCount
1
hasImages
yes
source
import
hasTranscription
no
Source extras
naId
351002347
levelOfDescription
fileUnit
recordType
description
ocrSource
nara-archive
Single page context
seq
1
pageIndex
0
type
document
mediaId
6be042126fff800d
ocrText
Originally Processed With FOIA(s):
FOIA Number:
S
S
FOIA
MARKER
This is not a textual record. This is used as an
administrative marker by the George Bush Presidential
Library Staff.
Record Group/Collection: Donated Historical Materials
Collection/Office of Origin: Frieden, Lex, Collection
Series:
Printed Materials
Subseries:
Manuals
OA/ID Number:
52104
Folder ID Number:
52104-010
Folder Title:
"A Resource Handbook for Vocational Rehabilitation" [1974]
Stack:
Row:
Section:
Shelf:
Position:
A RESOURCE HANDBOOK FOR
VOCATIONAL REHABILITATION
ICD
Laboratory
PROGRAM EVALUATION:
A Resource Handbook For
Vocational Rehabilitation
Eleanor Carol Bennett
Resource Analyst
Research Utilization Laboratory
ICD Rehabilitation and Research Center
Marvin Weisinger
Associate Project Supervisor
Research Utilization Laboratory
ICD Rehabilitation and Research Center
ICD Rehabiliation and Research Center
New York, N.Y. 10010
Copyright © 1974 ICD Rehabilitation and Research Center
except for portions copyrighted by others, permission for
the reprinting of which herein has been obtained from the
respective copyright owners.
Portions of this publication, except for those portions
copyrighted by others, may be reprinted provided per-
mission to do SO is obtained in writing from the ICD
Rehabilitation and Research Center or from the Social and
Rehabilitation Service, U.S. Department of Health, Educa-
tion, and Welfare with whose support the project resulting
in this publication was conducted.
The development of this publication was supported, in part, by Research Grant
No. SRS 22-P-55060/2 from the Division of Research and Demonstration
Grants, Social and Rehabilitation Service, Department of Health, Education and
Welfare, Washington, D.C. 20201.
TABLE OF CONTENTS
Page
ACKNOWLEDGEMENTS
V
PREFACE
vi
I.
PROGRAM ASSESSMENT
1
OBJECTIVITY
2
A PERSPECTIVE
5
A SUGGESTED MODEL
7
Classes of Measures
7
Program Level
9
A Word on Indices
11
Some Additional Considerations
11
SUMMARY
15
II.
PROGRAM EVALUATION DESIGN
17
STATEMENT OF PROGRAM EVALUATION PROBLEM 18
Operationalization
19
SPECIFICATION OF METHODOLOGY
21
Approaches to Evaluation
21
Research Approach
22
Systems Approach
24
Reliability, Bias, Validity
27
Selection of the Sample
28
COLLECTION OF DATA
29
ANALYSIS OF DATA
32
Levels of Statistical Measures
32
Data Analysis Techniques
33
Research Approach
33
Systems Approach
40
INTERPRETATION AND CONCLUSIONS
43
III. THE NINTH PANACEA: PROGRAM EVALUATION
45
By Robert A. Walker
IV. COMPENDIUM OF MEASURES
61
PROGRAM
63
Effort
63
Effectiveness
64
Efficiency
70
Quality
71
SERVICE DELIVERY
72
Counselor Performance
73
Evaluator, Job Trainer, Placement Officer
75
PROGRAM MANAGEMENT
76
Administrative Functions
77
Community Relations
78
Staffing
81
Inservice Training
82
Fiscal Administration
84
WEIGHTED CLOSURES
85
CASE FLOW (Monitoring)
86
.
V.
EXAMPLES OF PROGRAM EVALUATIONS
89
PLANS
90
RESEARCH APPROACH
95
SYSTEMS APPROACH
99
COST BENEFIT ANALYSIS
104
OTHER FIELD EXAMPLES
109
GLOSSARY
111
READING LIST
117
REFERENCE LIST
139
APPENDIX I. Other Classes of Measures Used
in Program Evaluation
149
APPENDIX II. Standards
150
ACKNOWLEDGEMENTS
Compilation of this handbook reflects the support and suggestions of many
people. Special acknowledgment must go to the staff of the Research Utilization
Laboratory whose aid and encouragement was instrumental in its completion-
Dr. Isabel Robinault, Project Supervisor and Susan Ahrenhold, Information
Specialist. Dr. Salvatore G. DiMichael, Project Director and Director of ICD
Rehabilitation and Research Center provided valuable assistance to this effort by
placing the Center's resources at our disposal. Helen Stonehill, Center Librarian,
generously supplied us with much of the esoteric literature in the field of
program evaluation. In addition, we must express our great appreciation for the
fortitude and patience of our secretarial staff, Helen Carr and Christine Hicks,
who typed numerous drafts of this document. Finally, we wish to thank the
many program evaluation professionals whose substantial consultant effort is
critical to enhancing the Handbook's value for the field. Any errors in the
interpretations and compilations based on the consultant's input for the program
evaluation examples and compedium of measures are ours alone. We particularly
would like to acknowledge the efforts of the following:
Edward C. Baumheier, Ph.D.
Leonard Miller, Ph.D.
Janet M. Derr
Division of Counselor Education
Lalitha Raichur
College of Education
Social Work Research Institute
University of Iowa
University of Denver
Iowa City, Iowa 52242
Denver, Colorado 80210
John Muthard, Ph.D.
James Bitter, Ph.D.
Regional Rehabilitation Research Institute
Regional Rehabilitation Research Institute
University of Florida
University of Northern Colorado
Gainesville, Florida 32601
Greeley, Colorado 80631
Robert Philbeck
Gabriel Cifor
Assistant Director for Planning & Evaluation
Robert Struthers, Ph.D.
Division of Vocational Rehabilitation
Department of Education/
North Carolina State Department of Human
Vocational Rehabilitation Services
Resources
Lansing, Michigan 48904
Raleigh, North Carolina 27602
Larry Dickerson, Ph.D.
Sidney Stein
Research & Training Center
Program Evaluation
University of Wisconsin-Stout
Division of Vocational Rehabilitation
Menomonie, Wisconsin 54751
Texas Rehabilitation Commission
Austin, Texas 78705
William Hills, Ph.D.
Barbara Chesney, Ph.D.
Adam Zawada
Regional Rehabilitation Research Institute
Program Planning and Research
The University of Oklahoma
Division of Vocational Rehabilitation
Norman, Oklahoma 73069
Tallahassee, Florida 32304
Adrian J. Marinelli
Research Utilization Specialist
New Jersey Rehabilitation Commission
Trenton, New Jersey 08625
V
PREFACE
This handbook on program evaluation is a compilation of approaches,
methods, and techniques culled from over 250 sources. It is an attempt to bring
together a large array of information and organize it into a unified conceptual
framework.
The evaluator, whether novice or experienced, often faces a bewildering
mountain of information when he attempts to examine the literature of program
evaluation. There seems to be no one source where information on indices can
be obtained. Indeed, from publication to publication, there seems to be little or
no agreement even on definition of terms. Therefore, one of the major efforts of
this manual was to review the diverse literature and opinion on program
evaluation, to organize it, and reduce it to a useable form. However, it should be
clearly understood that while the scope of the Handbook is comprehensive, we
have not attempted to write the "magnum opus." Program Evaluation: A
Resource Handbook is in no way a substitute for a text book, background
training or experience. Some of the defined approaches and techniques require a
good deal of technical sophistication to be properly applied, interpreted, and
used in making program decisions for change. Some techniques and approaches
are relatively simple to apply and this is indicated. Most important, we have tried
to be cautionary where appropriate.
The contents of this manual are directed to three audiences. One audience is
comprised of the new evaluators. We assume they have been exposed to a course
in statistics and a course in research methods, have read several program
evaluations and are now wondering how to put it all together. The second
audience consists of experienced evaluators who might be looking for some
specific measures or techniques to solve an immediate problem. For the former,
we hope this manual will serve as a major adjunct to his training and give him
some important leads, cues and cautions on how to organize and proceed with a
program evaluation. For the latter, the annotated list of indices in Chapter IV
and the Bibliography and Reading List could be most helpful. The third
audience consists of administrators. For them, we hope this handbook will serve
as an aid to communication with program evaluation staff.
Since program evaluation is in its relative infancy in some of the social
services and since our review of the literature indicated little consensus among
authors, we expect certain honest differences of opinion regarding the material
contained in this handbook. We hope this document will be read, discussed and,
perhaps, even argued over.
New York
Eleanor Carol Bennett
June, 1974
Marvin Weisinger
vi
PROGRAM EVALUATION:
A Resource Handbook For
Vocational Rehabilitation
PROGRAM
ASSESSMENT
Today, program assessment is a key concern of management-both govern-
mental and private. Controversy surrounds it and published contradictions are
rampant. Some of the reasons for this controversy and lack of understanding is
the relatively recent use of program assessment in the management processes of
human services where its purpose may seem antithetical to these services. In
addition, the personnel involved in assessing programs and those utilizing the
results of such assessments often have conflicting philosophies and differing
goals. Consequently, the information-decision-making value of program assess-
ment can get lost or may be interpreted as a spectre capable of damaging worthy
programs, productive jobs and creditable reputations.
Part of this difficulty lies in the differing backgrounds (and thus differing
philosophies) of the evaluator and the practitioner-administrator. In many cases,
the administrator who received his primary training as a practitioner in the
helping services can be quite astute in systematically analyzing the complex
problems of clients and recommending appropriate actions. Yet, when he is
with systematically analyzing the complex problems of a program and recom-
mending actions, he is somewhat at a loss. In addition, the training of a
1
practitioner emphasizes the solution of client problems with all due speed,
without too much regard for economy.
On the other hand, the evaluator's background stresses precision and control.
He is often at a loss to understand why the boundaries for acceptable perform-
ance for program activities are set SO wide.
Communication between evaluator and administrator is often at cross pur-
poses, even though they may be discussing the same things, they use different
words. One purpose of this handbook is to provide an administrator with a
conceptual framework which will aid him to effectively communicate with
evaluation staff and to aid him in analyzing his program with the same acumen
he analyzes client problems.
It must be clearly understood that the final outcome of any evaluation
process is change. Evaluation basically is a systematic and objective method of
looking at where a program presently stands and estimating how a program can
do better. Change is an integral part of this examination-change to serve the
clients better and change in program economy. Only through an orderly process
of documented program evaluation can justifiable alternatives for action be
developed for the evolution of more efficacious and economical programs.
OBJECTIVITY
What sort of information should be used to document current program
activity and to justify future change? The answer that comes quickly to mind is,
"objective information". Two questions can then be asked, "How objective can
an assessment be?" and "How does one objectively assess a program?"
A program assessment can be conducted on one of two major levels of
objectivity. The least objective, Level I, is the opinion of experts. That is, it is an
assessment based upon the review of project documents or based on an onsite
visit by a person recognized as "expert" in the field. The review may be
superficial or in depth, but it relies on the previous experience of the reviewer
for the ultimate statement of program worth. It is quite conceivable that two
experts, reviewing the same program, could present two very different sets of
findings. Therefore, choice of this level of objectivity should be avoided when-
ever possible.
The more objective approach, Level II, is the systematic collection and
analysis of hard data. If this approach is followed, it should produce findings
that are reproducible no matter who undertakes the assessment.
Who administers the program assessment depends on the organization of the
agency, the finances available for the assessment and the source of the request
for assessment. Usually, there are three categories of people assessing programs:
consultants or consulting firms, panels of knowledgeable people and depart-
ments within the organization.
2
Exactly who administers the appraisal many times influences the impact of
the assessment. For instance, an expert visiting the program on a site visit, or an
outside researcher collecting "relevant" data to generate "meaningful" compar-
isons can be open to attack on such dubious grounds as "lack of familiarity with
local conditions" or the researcher's "Ivory Tower" approach. On the other
hand, an "in-house" assessment may be accused of a "white wash" or bias. There
is no simple solution to the selection of the evaluator. However, the use of a
rigorous objective approach that is replicable can usually be defended against
such attacks no matter who administers the assessment.
The answer to the second question "How does one objectively assess a
program?" is dependent on a number of factors: the type of information
available, the precision required from the conclusions, and the funds available to
undertake this assessment. There are three basic means of program assessment,
that is, a program may be monitored, evaluated, or be the subject of a special
study.
Monitoring is ongoing feedback of information about professional perform-
ance and program activities for the purpose of controlling those activities.
This type of assessment serves as an indicator of disruption in program
activities, but does not necessarily define the nature of the difficulty. No
value judgement is offered concerning the "goodness" or "badness" of the
program, although application of externally or internally derived standards
may offer some kind of statement concerning professional activities.
Examples of usual monitoring methods are caseflow information and case
review. In some instances, ongoing outcome measures are examined to
indicate disruption of activities.
Program Evaluation usually examines annual program information and can
offer a value judgement on an entire program's worth. It can identify
causes of problems or reasons for success. Usually this judgement is based
on program outcomes, or the outcomes of program components.
Special studies offer a judgement from the examination of one facet of a
program. That facet may be universal, in that it pervades many portions of
the program, or it may be highly circumscribed. The final impact of that
assessment depends upon what aspect of the program has been examined.
For instance, to examine "the effectiveness of counselor performance
when serving the culturally disadvantaged" is a pervasive problem
encompassing attitudes and program policy. On the other hand, to examine
"the effectiveness of white counselor performance in dealing with aged,
black alcoholics" is specific to the administration of one program and, as
such, is so circumscribed as to be of doubtful utility.
3
Figure I illustrates some of the benefits and limitations of the combinations
of level of objectivity and level of assessment. Since the methods, dynamics and
value of monitoring are well understood as necessary mechanisms for admin-
istrative control, our discussion of this approach is limited to a presentation of
indices for monitoring in Chapter IV.
It can be seen from the above discussion that objectivity is partially a matter
of the degree of refinement of information from which judgements are to be
made. The remainder of this monograph is addressed to that process which is
based on the analysis of hard data and is used to evaluate programs. This
technique, program evaluation, is just now emerging as a program-planning and
decision-making tool. "Special Studies" may be considered as a limited or
circumscribed case of the broader "Program Evaluation". Therefore, any of the
following discussions which refer to Program Evaluation may be considered as
perfectly applicable to the topic of Special Studies.
FIGURE I
BENEFITS AND LIMITATIONS OF
PROGRAM ASSESSMENTS
Level
Level of Assessment
of
Objectivity
Monitoring
Special Studies
Evaluation
Benefits
Benefits
Benefits
I. Expert's
Inexpensive
Inexpensive
Inexpensive
Opinion
Not time consuming
Not time consuming
Not time consuming
from
May identify problems
May be perceptive
May be perceptive
A. Site Visit
B. Annual
Limitations
Limitations
Limitations
Report
Not replicable
Not replicable
Not replicable
May not identify problems May be imprecise
May be imprecise
May not identify cause
Lack of adequate firm
Lack of adequate
data
firm data
May over-emphasize
minor detail
Benefits
Benefits
Benefits
II. Analysis of
Ongoing
Precise
Precise
Hard Data
Identifies problems
Replicable
Replicable
Replicable
Identifies cause
Identifies cause
Broad in scope
Limitations
Limitations
Limitations
Does not identify cause
Expensive
Expensive
More expensive than
May be time consuming
Time consuming
Level I
Limited in scope
4
A PERSPECTIVE
In reviewing the program evaluation literature, several problems confront the
reader. A great deal of literature is directed to extolling program evaluation's
virtues as an aid in management, and an equally large collection decries the
damage it can and does do, if perspective and methodology are not sound. A
smaller body of representative literature discusses technique and methodology.
However, these methodology articles are written from what appears to be many
conflicting points of view. Actually, close examination reveals only two or three
points of view.
The first point of view is that of administrators who have learned to analyze
their programs by using several different models. Probably the most popular
model speaks of input analysis, process evaluation and outcome evaluation. A
second model refers to levels of program responsibility or control and mentions
the institution (program), managerial functions, and the technical core (service
delivery). (95)
An administrator, when considering his program, may view it as a system to
be controlled and regulated. He then speaks of specific parts such as inputs,
process, outputs, and some sort of a "feedback loop". The analyses of these
parts may be defined as follows:
Input Analysis-the - study of the resources (men, money and materials)
used by the program to accomplish its goal.
Process Evaluation-the study of a combination of program operations and
techniques used by a program to accomplish its goal. Criteria usually are
addressed to questions of conformity to, and expectations of the respec-
tive professions involved. These criteria are usually called Quality. Many
times the statistics used to monitor programs are composed of these types
of data.
Outcome Evaluation-the study of the results of a program's activities.
Usually, the criterion used is the degree to which programs have attained
goals. Outcome evaluation is frequently obtained through follow-up
studies.
The second model relates to specific levels of program responsibility
(responsibility model)-the institutional level, managerial level and technical
level. This breakdown may be considered as a reduction to program level, that is:
* *Program monitoring and evaluation are part of the feedback loop.
5
Institution-(a program) is supported by a larger social system which
legitimatizes the program's goals. Stated another way, we can say society
expects a vocational rehabilitation program to assist the disabled to
become meaningfully employed.
Technical-(service delivery) is a sub-organization or level of program. Its
function is to provide the services to clients and SO accomplish the
program's goal.
Managerial-(program management) is another sub-organization or level of
program. Its function is to mediate between services, and between services
and clients. In addition, it procures resources and aids in assuring appro-
priate outputs.
A second point of view is that of the evaluator who usually addresses his
perspective to methodology. This perspective can be divided into two camps-
the researcher who speaks of comparisons and goal attainment, and the systems
analyst-operations researcher who speaks of effectiveness and survival models.
(34) For the purposes of practical illustration, the popular goal-attainment
model may be considered as an inductive approach, while the systems models
usually are deductive in nature. Research constructs hypothesis about the way
programs work, and the systems approach builds models describing program
structure, functions and goals. Below are our definition and distinctions of the
two approaches:
Research Approaches seek to evaluate programs through making certain
comparisons between variables. The basic question is, has the program
caused a change in a variable? For example, has the introduction of a
PA/VR program caused more PA recipients to leave the welfare rolls?
These comparisons may be: before VS. after introduction of a program,
between programs operating in differing localities, between clients in a
special program and clients who did not have the benefit of the special
program, between client behavior before entering the program and after
leaving the program, etc. Of course, certain steps must be taken to assure
comparability of the data (See Chapter II).
A special case of the research approach is the Goal Attainment Model. In
this case, in order to determine program effectiveness, the broad objectives
of the program are translated into measurable criteria and a comparison is
made between these criteria and actual program performance, i.e., has the
program attained its goal? Standard Setting or Comparison of Planned vs.
Actual Performance are special cases of the Goal Attainment Model in
which the goals are set by some formula or by professional judgement and
6
are not necessarily one-to-one reflections of total program objectives. (See
Chapter III.)
Systems Approaches-seek to evaluate programs by describing the
organizational structure of a program, the stated functions of the program,
and the process by which that organizational structure with those func-
tions accomplishes the program's objective. That is, the analytic process
tries to build a model of the program, uses the model to stimulate the
program, and then offers more efficient and/or effective alternatives of
action (optimization). An enormous amount of information is needed for
this type of evaluation, together with a great deal of data processing effort.
(See Chapter II.)
A SUGGESTED MODEL
"The purpose of the evaluation process is to provide policymakers with the
basic data necessary for them to make decisions wisely. Impact evaluations of
programs should provide five essential sets of information. First, they should
provide all of the data necessary to determine if a particular program should be
continued. Second, they should determine which of the alternative programs
achieve greatest gains for a given cost. Third, evaluations should present informa-
tion on the components of each program and the mixes of components which
are most effective for a given expenditure SO that maximum operating efficiency
can be achieved. Fourth, evaluation should provide the first three types of
information for persons with different characteristics SO that a decision-maker
may determine which individuals are best served by each program. Finally, in the
course of evaluating existing programs, data should be gathered which will
suggest new methods to attack problems. To date, no evaluation of programs has
provided all this information." (11)
In short, the primary goal of program evaluation is to ask "Which programs
are worthwhile?" or "Which programs work best?" and then state "Why they are
worthwhile" or "Why they work best." These findings may produce information
for program decision-making that may result in the implementation of a change.
Suggested below is a model that approaches some of the issues for the
systematic evaluation of programs.
Classes of Measures
Basically, there are five classes of measures or criteria that are used to
estimate various aspects of program worthiness. The first class of measures
enumerates the resources used by the program (effort). The second class ex-
7
amines the results of program service in terms of broad social change (impact).
The third class attempts to estimate specific client outcomes or changes
(effectiveness). The purpose of the fourth class is to determine the economics of
program operation in comparison with program accomplishment (efficiency).
The final class of measures, Quality, has come to mean different things to
different people. One accepted global definition is that quality is a statement of
program "goodness." A second definition used equates quality with effective-
ness. That is, it uses process measures (see appendix I) to represent program
effectiveness. The perspective taken in this handbook is that Quality estimates
the conformance of program activity and professional activity to accepted
standards of the day. It is one reading of ongoing program process. These five
measures are derived from the administrator's system breakdown of input,
process, and output, where effort represents input, quality is equivalent to
process, impact and effectiveness represents output, and efficiency is an input-
output ratio. More specifically:
Effort is a statement of fact about the "amount and kinds of program
activities used to reach program objectives." It refers to staff time, activity
and commitment, and to the allocation and use of material resources
(funds, space, equipment, etc.). In addition, it takes into account ancillary
resources used (outside consultation, media, public relations, etc.). It
answers the question "What did I do?" (94,97)
Impact is a statement based on how the program was able to affect a broad
social change together with the utility of this program technique. Usually,
ost/benefit or cost/effectiveness is used for this type of evaluation. (11)
Effectiveness is a statement of fact about how well the objectives of the
program were met, or a statement of fact about how well the program
components functioned to meet the program objectives or the program
components' objectives. Effectiveness measures answer the question "Has
Vocational Rehabilitation program accomplished its objective?" or "Have
clients, who are rehabilitated by Vocational Rehabilitation, gotten jobs?"
In other words, "How has the program changed clients behavior or
performance?" (27,41,94,97)
*Impact is the same as effectiveness, but on a much broader scale; that is, it can
be thought of as a long-range socially pervasive change, whereas, effectiveness is
usually thought of as simply attaining goals or a circumscribed change. In many
cases, impact criteria are very difficult to establish and data very costly to
collect. Therefore, this monograph only considers effectiveness.
8
Efficiency is a rate that is produced when effort (inputs, staff time,
activities, salaries, physical facilities and purchased service) expended by a
program is compared to the program's effectiveness (output). It answers
the question "Can the same program results (output) be achieved by either
reducing the effort expended or by choosing other less costly alternatives
of action?" (26,49,97)
Quality is a judgemental statement made about professional competence,
acceptability of services to the client and the setting of minimum perform-
ance standards by legislative and regulatory bodies. It answers the question
"How well did you do what you did?" or "How well did you use the
knowledge you had?" (2,29,97)
Program Level
Having established and defined proxy measures for the system elements
(input, process, outcome) the "responsibility model" (program, program
management and service delivery), was selected as a suitable outline for analysis.
The goals of each organizational element are defined below.
Program-goals are to meet society's expectation of a Vocational Rehabili-
tation organization.
Program Management-goals are to secure and organize inputs for the
system (funds, staff, facilities, referrals) and then assure appropriate out-
puts (closures).
Service Delivery-(technical component)-goals are to perform the process
of rehabilitating the disabled.
Figure II visually represents the conceptual outline for a program evaluation
model that will include input analysis, process analysis, and outcome analysis
(goal attainment) or measures of effort, effectiveness, efficiency and quality
broken down by program level.
9
FIGURE II
PROPOSED MODEL FOR
PROGRAM EVALUATION IN
VOCATIONAL REHABILITATION
MEASURE