Some of the practical uses of fidelity scales include...

• defining supported employment standards at program start-up,
• tracking progress over time,
• documenting services for vocational rehabilitation and other funders, and
• making comparisons between programs and with norms for established programs.
  

Transcript

One common use of fidelity scales is to provide concrete standards for new teams starting out to determine their staffing, their organizational structure, and the types of services that they provide. The SE Fidelity Scale explains clearly what is expected of the team. By looking at the criteria for full implementation on each of the 15 items on the scale, program planners, program leaders, and practitioners have a set of operational standards for supported employment. The standards do not substitute for a practice how-to manual, but they do offer some concrete benchmarks for performance at the program and staff levels. Let’s use the evidence-based practice of a rapid job search as an example. The explicit benchmark for a rapid job search is.... “On average, make first contact with employer within 1 month after admission”

Conversely, there is ample evidence that the absence of clear cut criteria leads to variable implementation. In other words, there are consequences of not defining SE standards. A frequently heard comment by staff in vocational programs is: “We’re already doing supported employment.” This comment is often made even when it is obvious that the program is not following the principles of supported employment.

Another key use of fidelity scales is for monitoring programs over the course of their development and even after they are fully established. Routine use of fidelity scales provides an objective, structured way to give feedback about program development. Experience also suggests that this is an excellent method to diagnose program weaknesses, as well as to clarify strengths for providing positive feedback on program development. For instance, a program could look at individual item ratings and compare them to norms for established programs. This may help to pinpoint what needs changing.

Programs implementing an evidence-based practice should graph their fidelity ratings over time, using their total fidelity score to see their program change. For instance, the Capital Region Mental Health Center in Hartford, Connecticut with Kim Mueser as the Principal Investigator started a supported employment team. Fidelity assessments were made quarterly for almost two years. After a brief start-up period, the team showed consistently good fidelity, as indicated by scores over 70 for every time period after the first. For the SE Fidelity Scale, a score in the range of 66 - 75 represents good implementation.

In addition, using the SE Fidelity Scale is a way to demonstrate improvement to the Vocational Rehabilitation Agency, state mental health agency, and other funding entities as well as to the program’s Board, Chief Executive Officer, other administrators and to the program’s employment team! These uses take on value in the context of audits, accreditation, licensing, and similar review processes.

Fidelity scales also provide a comparative framework for evaluating statewide implementation of an evidence-based practice and allow the evaluator to identify statewide trends and outliers. Increasingly, states are demanding accountability. Some states, such as New York, have explicitly adopted fidelity scales for evidence-based practices, to determine eligibility for state funding. States such as Vermont and Kansas have used the supported employment Fidelity Scale for statewide quality improvement. The strategic use of repeated evaluations of programs using fidelity scales, either on an individual program or statewide level, is based on the general principle that whatever is paid attention to is more likely to be improved.