Validity of the Supported Employment Fidelity Scale

Discriminant Validity

Predictive Validity

Transcript

A second type of validity is sometimes referred to as discriminate validity. It asks the question, “Does the scale differentiate between programs that exemplify the evidence-based practice and those providing usual services?” For the Supported Employment Fidelity Scale, the answer is “Yes,” as illustrated by a study of 123 employment programs conducted by Bond and others (2001). In this study, 10 programs developed with the assistance of Debbie Becker and other experts in the Individual Placement and Support model were compared to 106 programs that had implemented supported employment without this training and support and to 7 programs that were following other types of vocational models such as clubhouses and sheltered workshops. This study showed that programs developed to follow the IPS model of supported employment score higher on the Supported Employment Fidelity Scale than do programs following other vocational models, including programs that have implemented what they call supported employment, but which have not systematically followed supported employment principles. The main factor that differentiates IPS programs from other supported employment programs is integration of employment services with treatment.

Since this study was completed, further surveys have replicated the basic finding that the Supported Employment Fidelity Scale differentiates evidence-based supported employment from other types of vocational programs. These additional studies have shown that simply calling a program supported employment does not ensure that the program follows the principles. This finding validates the core of the reason for monitoring fidelity.

A third type of validity is predictive validity. Predictive validity is pivotal to the theory of fidelity measurement, stated earlier in the hypothesis: “Programs that faithfully implement evidence-based practices were found to have better outcomes. Simply put: “Do higher fidelity programs have better outcomes?” Findings from a study of 10 employment programs in Vermont, conducted by Becker and colleagues were published in 2001. The results suggest that the SE Fidelity Scale does have predictive validity. The correlation for the total scale was .76 with employment outcomes, which is a very strong relationship. This is the only published study on the predictive validity of the Supported Employment Fidelity Scale, although fidelity scales for other evidence-based practices have demonstrated predictive validity. Please note that regarding predictive validity, many factors in addition to fidelity of implementation influence outcome. The skill level of practitioners, organizational support, leadership, selection of consumers to receive the services, financial support for the program, and adequacy of the clinical case management services, are just a few of the factors that also influence program effectiveness.

Is the Supported Employment Fidelity Scale a useful monitoring tool? Over the past 5 years we have accumulated a considerable wealth of information about the usefulness of the SE Fidelity Scale. There is substantial information available about its reliability, validity, and utility. The SE Fidelity Scale has been found to be adequately reliable and valid in determining the extent to which a program is implementing supported employment according to the 6 evidence-based principles presented earlier. There are other resources posted in the course that you can use to find additional information on this topic.