Supportive Supervision and Resiliency
The QIC-WD evaluation was conducted with the support of the Ohio Department of Job and Family Services to determine if a Supportive Supervision and Resiliency intervention, known as Coach Ohio, was effective in improving workforce and child welfare outcomes.
Research Questions
The site-level evaluation for Ohio was designed to understand implementation of and outcomes related to the Coach Ohio intervention and its component parts: (1) the ACCWIC Coaching Model for managers and supervisors to introduce the key principles of supportive supervision and (2) Resilience Alliance (RA) groups for supervisor and frontline workers to enhance coping in the face of exposure to trauma.
Relationships among intervention components, outputs, and outcomes were mapped out in Ohio’s logic model. Initial research questions of interest included how well the components of the intervention were executed with fidelity, in the form of dosage, adherence, quality, and participant engagement. The efficacy evaluation addressed a substantial number of research questions about short- and medium-term outcomes and included:
- To what extent did the intervention lead to a workforce with higher levels of: (1) emotional regulation, (2) coping, (3) optimism, (4) resilience, (5) work-life balance, and (6) perceived social support among those in the intervention vs. comparison condition? And to what extent did the intervention lead to changes in these variables pre to post intervention?
- To what extent did the intervention lead to a workforce with lower levels of: (1) secondary trauma and (2) burnout? And to what extend did the intervention lead to changes in these variables pre to post intervention?
- To what extent did the intervention lead to a workforce with higher levels of: (1) job satisfaction and (2) intent to stay and lower levels of: (3) intent to leave and related leaving behaviors, and (4) absenteeism among those in the intervention vs. comparison condition? And to what extent did the intervention lead to changes in these variables pre to post intervention period? To what extent did the intervention affect organizational culture and climate, particularly in the targeted areas of rigidity, resistance, engagement and stress?
- How did uptake of Coach Ohio and changes in the attitudes and behaviors described above vary by demographic and personality characteristics?
The evaluation also addressed longer-term research questions about the effect of the Coach Ohio intervention compared to those in the comparison condition on turnover and child and family engagement and permanency/safety outcomes.
Evaluation Design
The process evaluation was descriptive in nature and assessed: (1) intervention fidelity of the ACCWIC Coaching Model training through observations and facilitator self-assessments and (2) RA intervention fidelity including dosage (e.g., participant attendance rates across the 24 RA sessions), adherence to the content of each RA session by facilitators (using periodic observations and facilitator self-ratings of each session), and participant satisfaction with the content of RA and quality of facilitation of RA sessions (every three weeks).
A quasi-experimental design was employed to evaluate the effectiveness of Coach Ohio (see diagram). Staff in four counties and half of the staff in a large urban county received the intervention while staff in three other counties and the second half of staff in the large urban county did not and served as part of the comparison group (see the Site Overview for a list of counties). Both survey and administrative data from child welfare and human resources systems were collected for intervention and comparison participants.
Measures included personality and attitude scales that might mediate uptake or outcomes as well as all short- and medium-term outcomes of perceived work stress, secondary traumatic stress (STS) and burnout, as well as emotional regulation, coping behaviors, optimism, resilience, perceived social support and organizational culture and climate. Long-term outcomes of workforce turnover and case-based family engagement as well as child safety and permanency were measured using child welfare and human resource administrative data extracts. Survey data were collected either in-person or using a secure online platform. Survey data were linked together and with administrative data for mediation analyses. All data were stored and managed in a secure location, accessible only to the QIC-WD evaluation team and the site data coordinator.
