Is the SE Fidelity Scale a Useful Monitoring Tool?

1. Reliability
2. Validity
   

Transcript

Over the past 5 years, a considerable wealth of information has been published about the usefulness of the Supported Employment Fidelity Scale in the areas of reliability, validity, and utility. Reliability refers to the accuracy of measurement. In other words, the degree to which we have measured the scale in a dependable fashion that could be replicated by others.

The internal consistency of the SE Fidelity Scale (that is, how well the items “fit” together) is adequate, according to one large study conducted by Bond and his colleagues in 2001. Other studies also indicate that fidelity assessors agree on item ratings. Vogler completed the most systematic reliability study of the SE Fidelity Scale in 1998. She found fairly good agreement of ratings based on interviews with team leaders compared to employment specialists.

One type of reliability for which there is no published data is test-retest reliability. Because programs use information about fidelity to make improvements, we would not necessarily expect high test-retest reliability over any extended period of time. More research on the reliability of the Supported Employment Fidelity Scale is needed. However, the reliability is adequate for many practical applications, as long as users recognize the limits of the scale.

Validity refers to the credibility of the scale in measuring what it is intended to measure. The validity context for the Supported Employment Fidelity Scale is whether it actually measures the extent to which a program is implementing supported employment according to supported employment principles. Three aspects of validity are content validity, discriminant validity, and predictive validity.

Content validity refers to whether the items included in a scale correspond to the construct being measured. Thus the question at hand is, “Do the Supported Employment Fidelity Scale items follow from the principles of supported employment?” The short answer to this is, Yes, because the items were based on the work of Becker and Drake, who developed the Individual Placement and Support (IPS) model of supported employment, which is now synonymous with evidence-based supported employment.

A second, more rigorous standard for establishing content validity is to ask, “Does the research evidence support the inclusion of each of the individual items?” The criterion for including each item is ideally that the item has been shown to positively affect outcome. This is a very high standard that is rarely fully met for any measure. However, as briefly outlined above, this standard has been partially met, at least for some principles on which the SE Fidelity Scale items are based.