Addressing Work-related Traumatic Stress

The QIC-WD evaluation was conducted with the support of the Nebraska Division of Child and Family Services to determine if an intervention to Address Work-related Traumatic Stress, known as CFS Strong, was effective in improving workforce and child welfare outcomes.

Research Questions

The site-level evaluation for Nebraska was designed to understand implementation of and outcomes related to CFS Strong and its component parts, including Resilience Alliance (RA), Peer Support Groups (PSG) and Restoring Resiliency Response© (RRR). Relationships among intervention components, outputs, and outcomes were mapped out in Nebraska’s logic model. Initial research questions of interest included how much participants were satisfied with the RA, PSG, and RRR sessions they attended. Furthermore, for RA, the evaluation measured how much facilitators adhered to the manualized RA program, whether group dynamics (e.g., engagement and conflict) were positive or negative, and how much participants believed they could transfer learning to their professional work.

The evaluation addressed a substantial number of research questions about short- and medium-term outcomes. Questions were posed and answered separately for the RA intervention alone (with outcomes measured after six months) and for the combination of RA plus PSG (with outcomes measured after 12 months). These questions included: 

• To what extent did the intervention lead to a workforce with higher levels of: (1) emotional regulation, (2) coping, (3) optimism, (4) job satisfaction?
• To what extent did the intervention lead to a workforce with lower levels of: (1) secondary trauma and (2) burnout?
• To what extent did the intervention lead to changes in: (1) withdrawal cognitions, (2) absenteeism, (3) intent to search for a job/intent to quit? And were such changes mediated by changes in job satisfaction and burnout?
• To what extent did the intervention lead to enhanced perception of organizational commitment to address secondary traumatic stress (STS)?
• How did uptake of RA and changes in the attitudes and behaviors described above vary by demographic and personality characteristics?

The evaluation also addressed long-term research questions about the combined effect of RA and PSG. These included: to what extent did these interventions lead to reductions in actual turnover and, was this change mediated by changes in intent to quit, withdrawal cognitions, absenteeism, or intent to search? Finally, to what extent did RA and PSG lead to enhanced child and family safety and permanency outcomes. (And were such changes mediated by reductions in turnover?)

Finally, regarding the RRR intervention, the primary research question was to what extent does participation in RRR change staff perceptions of the organization’s commitment to addressing secondary traumatic stress as an occupational hazard? However, the ability to answer outcomes research questions for RRR was limited by the fact that it started 20 months into the data collection timeline period and was offered as a pilot program in only select offices. 

Evaluation Design

The process evaluation was ongoing throughout all phases of the intervention. It was descriptive in nature, assessing participant attendance rates, the degree to which interventions were implemented with fidelity, and participant reactions, comments, and satisfaction. This allowed the site team to identify problems and address such problems during implementation, as much as possible in the context of a randomized trial. For example, when attendance logs indicated declining participation, the site team checked in with facilitators and/or supervisors.  When several participant comments indicated a lack of ability to engage in RA sessions, additional coaching for facilitators was provided.

Data for the process evaluation were collected from a variety of sources, including CFS Strong participants, facilitators, and session observers. Data were collected after each CFS Strong session (weekly for the RA phase and monthly for the PSG phase) and after each scheduled RRR session. Data collected from participants and facilitators included attendance logs, facilitator reflection sheets, and participant reaction/satisfaction surveys. The evaluation team collected session adherence data (for both RA and PSG) by observation. Data to assess RA group dynamics (e.g., conflict, engagement) and participants’ confidence in transferring skills learned to their casework were collected halfway through RA and when all sessions had finished. 

Regarding outcomes, a cluster randomized controlled trial (cRCT) design was employed to evaluate the effectiveness of the RA and subsequent PSG intervention components. The unit of randomization was an entire supervisory group (supervisor + workers) with supervisory groups stratified by the five Nebraska service areas and assigning equal numbers of groups to intervention and control. Supervisory units assigned to the intervention were then combined into groups of 10-15 employees to form RA/PSG groups within service areas. A few workers were not part of the defined target population, including hotline workers, adult protective service workers, and the trainee group within the Eastern Service Area. Both survey data and administrative data from child welfare and human resources systems were collected for treatment and control participants.

All employees assigned to either the intervention or control group completed baseline (pre-intervention) measures, as well as follow-up measures at six, 12 and 24 months. Measures included short- and long-term outcomes of perceived work stress, job satisfaction, intent to stay, intent to leave, resilience, coping behaviors, emotional regulation, optimism, STS and agency commitment to addressing STS. Additional long-term outcomes of workforce turnover and case-based child safety and permanency were measured using child welfare and human resource administrative data extracts. Survey data were collected using a secure online platform. Survey data were linked with administrative data for mediation analyses. All data were stored and managed in a secure location, accessible only to the QIC-WD evaluation team and the site data coordinator.

Timelines

Randomization and baseline data collection occurred in May 2019. The six-month follow-up survey was administered in December 2019 when RA groups ended; the 12-month survey in June 2020 when PSGs were scheduled to end.  (Some PSGs extended beyond June 2020 due to Covid-19 pandemic disruptions). The RRR pilot test began in January 2021 and ran for six months. Finally, the 24-month survey was distributed in May 2021.  Administrative data were transmitted to the evaluation team on an on-going basis throughout the data collection period.