Members:

Description:

A state-level working committee responsible for conducting project evaluation to determine 1) the project’s effectiveness in meeting the specific goals and objectives of the grant 2) measuring the client and systems outcomes.

Goals:

Development and execution of an evaluation workplan that focuses on three major areas specified in the grant: implementation fidelity, process evaluation, and outcome evaluation.

Implementation fidelity -Implementation fidelity focuses on how closely implementation matches the grant plans, where deviations in planned activities occur, the reasons for these deviations, and the impact of deviations on the grant objectives. Deviations are noted, discussed and possible impact assessed so that potential problems can be minimized. Documentation of project activities are provided to the evaluation staff to document implementation fidelity.

Process evaluation -Process evaluation focuses on what specific project activities have been supported by COSIG funding and who has participated or received these services. Each component of the project is evaluated to assess the quality and efficiency of the various activities. The primary focus is on whether the process is functioning well enough so it has a reasonable chance of achieving desired outcomes. It is very important to note that a continuous improvement approach is used in this portion of the evaluation in order to provide feedback as quickly as possible so changes can be made as needed to improve processes.
A variety of data collection techniques may be employed here including (but not limited to) examination of meeting notes and minutes, evaluator observation of selected activities, peer review of clinical processes, focus groups with appropriate groups, quantitative evaluation of training and conference activities, and quantitative measures to assess satisfaction with the consensus building process (more described below).

Outcome evaluation -Outcome evaluation focuses on the impact of grant activities on the service delivery system and the clients, including measurement of outcome of treatments on service participants, factors associated with these outcomes, and sustainability of these outcomes. For our purposes, we further divide this portion of the evaluation into two sub-groups: output evaluation and true outcome evaluation.

Outputs - Outputs provide short term data that may be a leading indicator of future outcomes and thus are important to evaluate. R regular examination of these as well as process measures enhances the likelihood that the project will achieve longer term outcomes. A key output of the project will the training outcomes. Training outcomes are usually assessed with at least three levels of outcomes: participant perceptions, learning, and performance outcomes. Another key output measure is consensus quality measures to assess the consensus building process. Simply put, the consensus building process should result in a reasonable level of consensus or improved treatment outcomes are not as likely to occur.

True Outcomes - Much of the focus of the Evaluation Steering Committee is expected to be on designing true outcome measures that reflect improvements in patient outcomes. This is a critical part of the infrastructure that will emerge from this grant. Participants from OAD, OMH and other consultants and stakeholders examine performance indicators and measures available and determine measures to be used in the evaluation. Performance indicators will be selected from the following: SAMHSA performance indicators; CMHS evaluation tools and guidelines; OMH and OAD performance indicators currently tracked.

Publications:

  Fidelity Assessment Executive Overview (.PDF)

  Dual Diagnosis in Addiction Treatment (DDCAT) Fidelity Tool (.PDF)

  Dual Diagnosis in Addiction Treatment (DDCAT) Manual (.PDF)

  Dual Diagnosis in Addiction Treatment (DDCAT) Fidelity Tool - Mental Health Version (.PDF)

  Dual Diagnosis in Addiction Treatment (DDCAT) Manual - Mental Health Version (.PDF)

  COSIG – Program Evaluation (PE) Committee Workplan (.PDF)

     View All Provider Publications

Home  |  Contact  |  Events Calendar  |  F.A.Q.  |  Feedback  |  Links  |  LITS Org Chart  |
| Go to The Clients & Family Site |