|
A multidimensional
approach to the structure of consumer satisfaction
with vocational rehabilitation services.
John F. Kosciulek.
Rehabilitation Counseling Bulletin,
Wntr 2003 v46 i2 p92(6)
Full Text: COPYRIGHT 2003 Pro-Ed
The purpose of this investigation was to examine the dimensions
underlying consumer satisfaction with vocational rehabilitation
services. Participants were 223 consumers who received services from a
state rehabilitation agency in a Midwestern state. Data were gathered
via mail survey through participant completion of a self-report
questionnaire. A two-dimensional configuration of 14 consumer
satisfaction stimuli yielded the following dimensions: (a) satisfaction
related to case management versus employment and (b) satisfaction
related to consumer choice versus customer service. Findings suggest
that consumers view satisfaction as a multidimensional construct.
Implications for the measurement of consumer satisfaction and use of
consumer satisfaction data in vocational rehabilitation program
evaluations are discussed.
Due to the significant lack of progress in promoting the employment of
persons with disabilities during a period of unprecedented economic
growth in the 1990s, current public disability policy and accreditation
mandates have greatly increased the requirements of vocational
rehabilitation (VR) service providers and funders to demonstrate
positive outcomes (O'Day, 1999; Seelman, 2000). For instance, the
Rehabilitation Act amendments of 1992 and 1998 increased program
evaluation documentation requirements for state rehabilitation agencies
(Council of State Administrators of Vocational Rehabilitation [CSAVR],
1998). Similarly, recent standards established by the Commission on the
Accreditation of Rehabilitation Facilities (CARF, 2000) now require
community rehabilitation programs to document program effectiveness more
thoroughly through ongoing outcome assessments. Furthermore, adopting a
market-based approach to reimbursement for VR service provision, the
Ticket to Work and Work Incentives Improvement Act of 1999 (TWWIIA)
connects payment for services to quality employment outcomes for service
recipients (Hoff, Vamey, Marrone, Butterworth, & Silverstein, 2001;
Kosciulek, 2000).
One area of outcome evaluation that has received significant attention
in terms of VR program effectiveness is consumer satisfaction. Koch and
Merz (1995) aptly described how the importance of measuring consumer
satisfaction has grown during the past decade as a result of federal
legislation and the emergence of consumer choice approaches in
vocational rehabilitation. Richard (2000) reported that effective
consumer satisfaction assessment practices were necessary to promote
consumer empowerment philosophies. Similarly, Kosciulek, Prozonic, and
Bell (1995) emphasized that consumer involvement and the increased
demand for accountability has resulted, in a need for VR programs to
demonstrate high levels of consumer satisfaction with services. Further,
Schwab and Fenoglio (1992) stated that a high level of consumer
satisfaction was one of the key features of a quality rehabilitation
system. Finally, a major conclusion of the CSAVR (1998) Committee on
Client Services Consumer Satisfaction Report was that consumer
satisfaction is one of the primary variables for evaluating the
effectiveness of services and documenting positive rehabilitation
outcomes (CSAVR, 1998). In this report, the CSAVR described the critical
roles consumer satisfaction data play in state VR agency program
implementation. The functions of consumer satisfaction data identified
in this report included the following: (a) service improvement, (b)
program planning and revision, (c) in-service training, (d) counselor
performance appraisal, and (e) agency performance appraisal in relation
to state and federal regulations. Thus, the literature has indicated
that consumer satisfaction is an essential component to consider when
assessing the efficacy of VR services.
Given its significance to the assessment of VR programs, accurate and
comprehensive evaluation of consumer satisfaction is imperative.
However, one of the criticisms of consumer satisfaction program
evaluation research has been that there is a lack of sophisticated and
holistic measurement of the construct. Numerous researchers have
reported a dearth of empirical literature that captures the potential
complexity of the construct (e.g., Janikowski; Bordieri, & Musgrave,
1991; Koch & Merz, 1995; Kosciulek, Vessell, Rosenthal, Accardo, & Merz,
1997; Richard, 2000). Janikowski et al. (1991) succinctly concluded,
"Very little research has been conducted examining the varying
dimensions of client satisfaction with rehabilitation programming" (p.
43). Similarly, Richard (2000) stated that VR researchers and program
evaluation specialists too often have restricted themselves to using
single global measures of consumer satisfaction. Thus, the need exists
for reliable and valid data on the multiple cognitive constructs that VR
service recipients use to arrive at their opinions of satisfaction or
dissatisfaction.
Identifying and measuring multiple dimensions of consumer satisfaction
would provide VR personnel with a comprehensive and accurate view of
consumer perceptions of the services they receive (Richard, 2000). For
example, from a pragmatic perspective in response to regulatory
mandates, the CSAVR (1998) reported that consumer satisfaction surveys
must address all the major aspects of the VR service delivery process,
including interface with the VR system, the specific services received,
service providers, consumer choice, and rehabilitation outcomes. More
precise consumer feedback regarding satisfaction with the counselor's
role as a case manager, informed choice opportunities, and the quality
of employment outcomes would yield specific data useful for improving VR
services.
The purpose of this investigation therefore was to respond to the need
for additional information on the multiple dimensions that encompass
consumer satisfaction. The specific intent of the study was to identify
the dimensions that underlie consumer satisfaction with VR services. It
was planned a priori for this study to be an initial step in identifying
consumers' perceptions of satisfaction and to encourage researchers and
program evaluation specialists to move toward the development of
multi-dimensional measures of consumer satisfaction with VR services. It
is anticipated that as specific dimensions that make up the construct of
consumer satisfaction are identified, defined, and reliably and validly
measured, useful information will be obtained to guide service provision
modification that would both
increase satisfaction levels and enhance employment outcomes for
consumers with disabilities.
METHOD
The purpose of descriptive research is to make assertions about
particular aspects of populations or samples (Cozby, 1993). As an
inquiry in the area of consumer satisfaction with VR services, the
current study attempted to elicit salient features of consumer
satisfaction. A descriptive research design was necessary to depict
dimensions of satisfaction among a sample of consumers who had received
VR services.
Procedure
Participants were recruited from a state rehabilitation agency in a
midwestern state that serves individuals with visual impairments. All
consumers (N = 812) whose cases were closed by the agency during Fiscal
Year (FY) 1999 were mailed a Demographic Information Sheet (DIS), the
14-item Consumer Satisfaction Survey (CSS), and a cover letter
describing the study and ensuring confidentiality and anonymity. To
maximize the overall response rate, a second set of study materials was
mailed to individuals who had not returned the original set of materials
within 2 weeks of the first mailing.
Because the selected state agency served individuals with visual
impairments, a significant effort was made to make study materials and
processes accessible to all consumers. First, as recommended by the
agency's Rehabilitation Advisory Council, study materials were mailed in
large-print format. Second, consumers were given the option of
completing the survey via Braille format or through a telephone
interview. Of the 812 study packets originally mailed, 82 were returned
as not deliverable. Thus, 730 packets were delivered to the most current
consumer addresses available.
Three consumers requested, were provided, completed, and returned study
materials in Braille format. Five consumers requested and completed the
14-item CSS and the DIS via telephone interview. An additional 261 study
packets were returned to the investigator. Of these packets, 26 CSSs
were returned with no data and 20 were returned with missing data (i.e.,
not all 14 items completed). These 46 survey sets were not included in
the final data analysis. The 20 CSSs with missing data were not included
due to the full data matrix requirement of the statistical technique
used in this study. As a result, full data sets from a total of 223
consumers, or 30.5% of the 730 possible participants, were used to
identify the dimensions that underlie consumer satisfaction with VR
services.
Participants
The population of interest in this study was consumers with disabilities
who received VR services. The sample was drawn from consumers whose
cases were closed by a state rehabilitation agency in a midwestern state
during FY 1999. The participants were 223 consumers who completed and
returned a survey of their level of satisfaction with the VR services
provided by the state rehabilitation agency. Of the participants, 127
(57%) were women and 96 (43%) were men. In terms of race, 167 (75%)
participants were ,White, 49 (22%) were African American, and 7 (3%)
were Asian American. One hundred thirteen (51%) participants had never
been married, 74 (33%) were married, 32 (14%) were divorced, and 4 (2%)
were widowed. The participants' ages ranged from 18 to 56 years, with a
mean of 29.25 years and a standard deviation of 4.31 years. Participants
averaged 12.14 years of education completed, with a standard deviation
of 2.79 years.
Instruments
The primary data source in this investigation was the 14-item CSS used
by a state rehabilitation agency to evaluate the level of consumer
satisfaction with VR services. (The 14 survey items are included in
Figure 1.) In addition, the DIS was developed by the investigator to
collect demographic information (e.g., age, marital status) from
participants. Data from the DIS were used to describe participant
characteristics.
Participants were asked to rate each of the 14 satisfaction items on a
4-point Likert-type scale (1 = strongly agree, 4 = strongly disagree),
indicating their level of satisfaction with each service delivery
aspect. It is important to note that the Rehabilitation Advisory Council
of the state rehabilitation agency developed the 14 items used as the
stimuli for data analysis. Given the input of consumers and direct
service providers (e.g., rehabilitation counselor) into their
development, the satisfaction items related directly to the context of
the VR services provided by the state agency. Such an item development
process is consistent with the recommendations of Kosciulek (1999) and
Zimmerman (1995), who both emphasized that open-ended constructs such as
consumer satisfaction must be measured in context in order yield valid
and useful results. The internal consistency reliability (Cronbach's
alpha) of the CSS for the current study was .94. According to Power
(2000) and Thorndike (2001), this finding indicates that the CSS had a
high level of reliability in its application with the sample in the
present study.
Data Analysis
In accordance with the purpose of the investigation and the nature of
the descriptive research design, multidimensional scaling (MDS) was used
to evaluate the structure of consumer satisfaction with VR services. MDS
is useful for obtaining information about the structure of a data set
(Davison, 1992; Forgas, 1979). Given that the research literature
emphasizes the complex nature of consumer satisfaction (CSAVR, 1998;
Koch & Merz, 1995; Richard, 2000), MDS was identified as the most
appropriate procedure to examine the data obtained in this study. The
use of MDS enabled the identification of underlying dimensions that
might not be apparent in the data with more conventional methodologies.
MDS is a mathematical procedure that systematizes data by representing
the similarities of objects on a spatial map, or configuration, in which
stimuli judged to be similar are points close to each other on the map
and stimuli judged dissimilar are points distant from one another (Schiffman,
Reynolds, & Young, 1981). MDS spatial representations differ from
one-dimensional scales in that the former are able to portray the
complexity (i.e., the underlying structure) of the data. Specifically,
MDS uses iterative numerical methods to estimate the spatial coordinates
for each of the stimuli in space. The resulting interpoint distances
will match as closely as possible to the participants' response data and
approximate a "good fit" to the model (Davison, 1992).
The coordinate axes that emerge represent perceptual dimensions that can
be identified and defined by inspecting the properties of the stimuli
positioned at the extremes of the axes. Theoretically, the positions the
stimuli hold in the overall spatial configuration correspond to the
salient attributes on which they are being perceived and can be
conceived of as representing the respondents' "perceptual space" or
"cognitive map." The relationships among the points on the map reveal
how the respondents perceive the stimuli. In other words, the map or
configuration uncovers the "hidden structure" in the data, thereby
making it easier to comprehend (Kruskal & Wish, 1978). In relation to
the current study, if distinct and interpretable satisfaction patterns
can be identified from the consumer's perspective, more relevant and
useful VR consumer satisfaction program evaluation processes can be
developed.
The SPSS 10 for Windows (SPSS, 1999) nonmetric MDS ALSCAL procedure,
using the symmetric, matrix conditional, and ordinal options, was
applied to the ratings of the 14 consumer satisfaction items completed
by the 223 participants. The Euclidean distance scaling model was used
to transform the rating data into the dissimilarity matrix necessary for
MDS. Solutions were specified in two through four dimensions. This
number of potential dimensions is within the bounds of MDS stability
considerations for 14 stimuli (Kruskal & Wish, 1978; Schiffman et al.,
1981). As recommended by Young and Hamer (1987), measures of stress,
fit, and interpretability were used to determine the number of
dimensions that best represented the structure of the consumer
satisfaction stimuli.
RESULTS
As illustrated in Figure 1, the results of the ALSCAL analysis revealed
the location of each of the 14 consumer satisfaction stimuli in relation
to each other within a two-dimensional space. This two-dimensional MDS
solution was obtained with a Kruskal's stress value of .091 and an
[R.sup.2] of .93. Stress is a lack-of-fit index; values of the stress
index higher than .10 may indicate a poor solution and a lack of fit of
the MDS model to the data (Kruskal & Wish, 1978). [R.sup.2] is an index
of fit that indicates how well the MDS model corresponds to the input
data (Hair, Anderson, & Tatham, 1987). The Kruskal's stress value and
[R.sup.2] index indicate an adequate fit and high level of
correspondence of the MDS model to the data. A three-dimensional
solution, even though it had slightly lower stress and higher fit, was
rejected as being not readily interpretable. The two-dimensional
solution thus presents an accurate and reliable visual depiction of the
data obtained from consumers regarding their satisfaction with VR
services.
As recommended by Davison (1992), interpretation of the underlying
consumer satisfaction patterns was conducted by examining the
configurations and recalling from the existing consumer satisfaction
literature what is known about the stimuli positioned at extremes of
each of the coordinate axes. As shown in Figure 1, the two-dimensional
MDS configuration indicated a case management (left) versus employment
(right) satisfaction pattern (Dimension 1--horizontal). Results also
indicated a consumer satisfaction pattern along Dimension 2 (vertical)
of consumer choice (top) versus customer service (bottom).
DISCUSSION
The purpose of this investigation was to examine the dimensions
underlying consumer satisfaction with VR services. Prior to discussing
results, caveats regarding methodological limitations must be noted. The
first is that the validity of the findings in this study is threatened
by the 14 consumer satisfaction stimuli used in this investigation (as
opposed to another group of items), the use of level of agreement (e.g.,
strongly agree) as the measure of consumer satisfaction, and a
quantitative rather than qualitative approach to assessing consumer
satisfaction. Numerous researchers (e.g., Janikowski et al., 1991; Koch
& Merz, 1995; Kosciulek et al., 1997) have affirmed that consumer
satisfaction with VR
services involves many facets that can be assessed through various
methods.
The second caveat relates to the degree to which the findings of this
investigation can be generalized across persons, times, and settings
(Parker, 1993). As per guidelines presented by Wilkinson (1999) and the
Task Force on Statistical Inference of the American Psychological
Association, because the sampling procedure was non-random, study
results cannot be generalized to all consumers who receive services from
state VR agencies. Similarly, the external validity of results in this
study is threatened by the 30% response rate that limits generalization
to the overall sampling frame and population of consumers with
disabilities who receive VR services. Thus, the cross-sectional nature
of the data
collection procedure allows only for interpretation of results
concerning the sample used at the time of the study (Heppner, Kivlighan,
& Wampold, 1999).
The results of the MDS of the 14 consumer satisfaction stimuli suggest
two dimensions along which consumers perceive satisfaction with VR
services. The first dimension identified in this study represented
satisfaction related to case management versus employment. As
illustrated in Figure 1, satisfaction stimuli at the left end of this
dimension (e.g., "I received services from the agency without excessive
waiting") entailed the processes of coordinating, the VR program. At the
right end of Dimension 1 were stimuli related to consumer perception of
satisfaction with employment outcomes resulting from VR services (e.g.,
"As a result of the services provided by the agency, my present work
situation is better than it was before I began the program"). Thus,
consistent with previous quantitative reports (CSAVR, 1998; Janikowski
et al., 1991)and conceptual papers (Koch & Merz, 1995; Richard, 2000) on
consumer satisfaction with VR services, a major dimension underlying
satisfaction as perceived by participants in this study focused on case
management and employment aspects of the VR process.
The second dimension of the MDS solution was interpreted as satisfaction
related to consumer choice versus customer service. Satisfaction stimuli
at the top end of this dimension centered on consumers' perceived
satisfaction with the availability of choice in the VR process (e.g., "I
had the final say in the selection of a vocational goal"), whereas
behaviors at the bottom end of Dimension 2 related to customer service
aspects of the VR process (e.g., "The agency staff returned my phone
calls promptly"). Consistent with the CSAVR (1998) Committee on Client'
Services Consumer Satisfaction Report, this finding indicates that
consumers perceive choice and customer service as distinct components of
satisfaction to consider when assessing the efficacy of VR services.
The two dimensions underlying consumer satisfaction with VR services
identified in this investigation suggest that consumers consider
satisfaction a complex, multifaceted construct. The data gathered in
this study indicate that consumers' cognitive map of satisfaction
consists of multiple factors. An important practice implication of this
finding is that state rehabilitation agencies may need to expand the
brief survey format currently used in most states (CSAVR, 1998) in order
to obtain comprehensive, reliable, and valid measures of consumer
satisfaction. Various methods, including standardized quantitative
instruments, qualitative focus groups, and individual interviews, are
necessary to adequately capture the diverse aspects of satisfaction in
program evaluation activities conducted by state VR agencies. In
addition, no reports exist in the consumer satisfaction literature
relative to the long-term assessment of consumer satisfaction in
relation to employment outcomes following vocational rehabilitation.
Studies assessing satisfaction levels during service provision, at case
closure, following 90 days of successful employment, at 6 months post
closure, and at 1 year post-closure would begin to shed light on the
true efficacy of VR services in relation to consumer standard of living
and quality of life.
The findings in this study also demonstrate that even with a relatively
small number of stimuli, consumer satisfaction studies can yield
distinct, interpretable, and stable dimensions. Results thus provide
evidence of the complexity of consumer satisfaction and need for further
study of the construct. Further elucidation of the construct of consumer
satisfaction through similar studies is warranted. Additional research
providing evidence that consumers view satisfaction as a
multidimensional construct suggests the need for the development of a
psychometrically sound, multifactorial instrument that assesses consumer
satisfaction in the context of vocational rehabilitation. In this
manner, if discrete satisfaction patterns can be identified from the
consumer's perspective, more pertinent VR consumer satisfaction program
evaluation processes can be developed.
Future research also should attempt to identify the patterns,
dimensions, and themes of consumer satisfaction for consumer population
subgroups (e.g., closure status, cost of services, disability type). In
addition, it is necessary to conduct longitudinal studies that assess
the relation of satisfaction to long-term employment outcomes (i.e.,
beyond 90 days of employment). A comparison of the satisfaction of
consumers using agencies serving the visually impaired to that of
consumers using general agencies would also be instructive. State VR
agencies can use the dimension anchors of case management, employment,
choice, and customer service to focus satisfaction survey development
and processes, as well as to compare consumer population subgroups.
Furthermore, given that response rates in most consumer satisfaction
surveys conducted by state VR agencies are very low (CSAVR, 1998),
valuable information would be gained by comparing demographic
characteristics and case service processes and outcomes of those
consumers who do and do not respond to satisfaction surveys. Finally,
regarding measurement approaches, in order to yield data useful for
program improvement, context-specific satisfaction instruments that
address the VR service process as provided by state rehabilitation
agencies and community rehabilitation programs are preferable to generic
human service satisfaction assessment approaches, such as SERVQUAL (Zeithaml,
Parasuraman, & Berry, 1990), which do not specifically address the VR
process.
REFERENCES
Commission on the Accreditation of Rehabilitation Facilities. (2000,
December). Believing in consumers: CARF practices what it preaches. CARF
Connection, 1-3.
Council of State Administrators of Vocational Rehabilitation [CSAVR].
(1998). CSAVR committee on client services consumer satisfaction report.
Washington, DC: Author.
Cozby, P. C. (1993). Methods in behavioral research (5th ed.). Mountain
View, CA: Mayfield.
Davison, M. L. (1992). Multidimensional scaling. New York: Wiley.
Forgas, J. P. (1979). Social episodes: The study of interaction
routines. New York: Academic Press.
Hair, J. E, Anderson, R. E., & Tatham, R. L. (1987). Multivariate data
analysis with readings (3nd ed.). New York: Macmillan.
Heppner, E E, Kivlighan, D. M., & Wampold, B. E. (1999). Research design
in counseling (2nd ed.). Belmont, CA: Wadsworth.
Hoff, D., Varney, E., Marrone, J., Butterworth, J., & Silverstein, R.
(2001). The proposed Ticket to Work regulations. Tools for Inclusion:
Family and Consumer Series, 9(1), 1-10.
Janikowski, T. P., Bordieri, J. E., & Musgrave, J. (1991, Summer).
Dimensions of client satisfaction with vocational evaluation services.
Vocational Evaluation and Work Adjustment Bulletin, 24, 43-48.
Koch, L. C., & Merz, M. A. (1995). Assessing client satisfaction in
vocational rehabilitation program evaluation: A review of
instrumentation. Journal of Rehabilitation, 61 (4), 24-30.
Kosciulek. J. E (1999). Consumer direction in disability policy
formulation and rehabilitation service delivery. Journal of
Rehabilitation, 65(2), 4-9.
Kosciulek, J. E (2000, November). Implications of the Workforce
Investment Act of 1998 and Ticket to Work and Work Incentives
Improvement Act of 1999 for vocational rehabilitation. Paper presented
at the Mid-South Professional Counselors and Case Managers Symposium,
Memphis, TN.
Kosciulek, J. E, Prozonic, L., & Bell, D. (1995). On the congruence of
evaluation, training, and placement. Journal of Rehabilitation, 61 (4),
20-23.
Kosciulek, J. E, Vessell, R., Rosenthal, D. A., Accardo, C., & Merz, M.
A. (1997). Consumer satisfaction with vocational rehabilitation
services. Journal of
Rehabilitation, 63(2), 5-9.
Kruskal, J. B., & Wish, M. (1978). Multidimensional scaling. Newbury
Park, CA: Sage.
O'Day, B. (1999). Policy barriers for people who want to work. American
Rehabilitation, 25(1), 8-15.
Parker, R. M. (1993). Threats to the validity of research.
Rehabilitation Counseling Bulletin, 36, 130-138.
Power, E W. (2000). A guide to vocational assessment (3rd ed.). Austin,
TX: PRO-ED.
Richard, M. A. (2000). A discrepancy model for measuring consumer
satisfaction with rehabilitation services. Journal of Rehabilitation,
66(4), 37-43.
Schiffman, S. S., Reynolds, W. E., & Young, E W. (1981). Introduction to
multidimensional scaling: Theory, methods, and applications. Orlando,
FL: Academic Press.
Schwab, A. J., & Fenoglio, J. A. (1992). The Texas Rehabilitation
Commission system for measuring quality rehabilitation. Journal of
Applied Rehabilitation Counseling, 23(2), 21-25.
Seelman, K. D. (2000, July). Employment of individuals with
disabilities: Opportunities and challenges. Paper presented at the
Cornell University Employment and Disability Policy Summer Institute,
Ithaca, NY.
SPSS. (1999). SPSS for Windows (Version 10.0) [Computer software].
Chicago: Author.
Thorndike, R. M. (2001). Reliability. In B. E Bolton (Ed.), Handbook of
measurement and evaluation in rehabilitation (3rd ed., pp. 29-48).
Gaithersburg, MD: Aspen.
Wilkinson, L. (1999). Statistical methods in psychology journals:
Guidelines and Young, E W., & Hamer, R. M. (1987).
Multidimensional scaling: History, theory, and application. Hillsdale,
NJ: Erlbaum.
Zeithaml, V. A., Parasuraman, A., & Berry, L. L. (1990). Delivering
quality service: Balancing customer perceptions and expectations. New
York: The Free Press.
Zimmerman, M. A. (1995). Psychological empowerment: Issues and
illustrations. American Journal of Community Psychology, 23, 581-599.
John F. Kosciulek, PhD, is an associate professor in the Office of
Rehabilitation and
Disability Studies at Michigan State University. His current research
and teaching
interests include research methodology, outcome assessment, Web-based
learning,
and family adaptation to brain injury. Address: John F. Kosciulek, 237
Erickson
Hall, Michigan State University, East Lansing, MI 48824; e-mail:
jkosciul@msu.edu
|
Proed Article 
