Onboarding Program

The QIC-WD evaluation was conducted with the support of the Eastern Band of Cherokee Indians Family Safety Program (EBCI FSP) to determine if an Onboarding intervention was effective in improving workforce outcomes.

Research Questions

The evaluation of the newly developed onboarding program for the Eastern Band of Cherokee Indians Family Safety Program (EBCI FSP) was designed to understand both implementation and early outcomes. Examples of primary implementation questions from the new employee’s perspective included:

• To what degree was the new employee’s workspace ready on their first day?
• Did the new employee have individual meetings with their supervisor in weeks 1-5?
• To what extent did the new employee feel welcomed during their first days on the job?
• What were the overall pros and cons of the onboarding program?

Additional implementation questions were also considered from the perspective of each new employee’s immediate supervisor:

• How could the handoff from Human Resources (HR) have been improved?
• How did the supervisor feel about the time required to manage the new employee's onboarding experience?
• How did the supervisor feel about the time required for the new employee to complete onboarding activities?

Key outcome evaluation research questions included:

• What onboarding practices were in place at the beginning of the QIC-WD project and how did they change over time?
• What were the onboarding experiences of EBCI FSP employees hired before the new onboarding program was implemented?
• After the onboarding intervention was implemented, what were the experiences of staff who received it?
  • How do onboarding experiences differ between staff who received the intervention and staff who were hired before the intervention was implemented?
• What understanding did staff have of historical trauma and its implications for their work with FSP?
• To what extent did staff who receive the onboarding intervention report higher newcomer socialization (e.g., social connections; understanding of organizational goals) compared to staff hired before the intervention was implemented?

The evaluation also aimed to evaluate longer-term outcomes such as turnover, job performance, continuity of case practice, and child and family outcomes, see Logic Model. However, the ability to assess these outcomes was limited by later than anticipated implementation and the size of the EBCI FSP.

Evaluation Design

The evaluation for the EBCI FSP onboarding intervention employed a case study research approach. Process evaluation for the EBCI FSP onboarding intervention was informed by surveys for new employees during onboarding weeks 2 and 5; surveys for new employees’ supervisors during onboarding weeks 2 and 5; and discussions between the local site implementation manager and new employees and supervisors at various points throughout implementation. Results of the surveys were analyzed and disseminated to site leaders on three occasions during implementation.

Three evaluation activities were undertaken to establish a baseline prior to intervention implementation for the outcome evaluation.

1. Archival records of previous and current onboarding procedures were analyzed. This involved collecting and reviewing documents such as orientation notes, checklists for new employees, meeting minutes, information relayed by existing FSP workers, and other relevant activities. The result of this activity was a description of the onboarding process prior to intervention implementation.
2. FSP workers participated in qualitative interviews to better understand their experiences with and understanding of three primary onboarding domains: (1) Cherokee history, culture, and historical trauma; (2) role clarity and policies and procedures; and (3) connection to other FSP workers and the FSP overall. Interviews were professionally transcribed. Using thematic analysis, two analysts identified and reported patterns within the data to identify frequent and significant themes connected to reoccurring details, explanations, and descriptions. Using Dedoose as an organizational and analytic tool, the researchers coded the transcripts using the themes as code families. The analysts took an inter-coder reliability test and scored a pooled Cohen’s kappa statistic of .84 which is considered to be “excellent” agreement.
3. Baseline and follow-up surveys were administered to FSP workers and supervisors at multiple time points throughout the intervention. The survey assessed various factors associated with job satisfaction and turnover (e.g., stress, supervisor support, resilience) relevant for the cross-site evaluation, as well as measures tailored specifically to the EBCI intervention (e.g., organizational socialization).

After baseline interviews were coded, qualitative and quantitative data were integrated using Dedoose. Triangulation and integration of both data sources facilitated a more complete understanding of how the onboarding intervention impacted the employee socialization process.

Timeline

Archival documents related to pre-intervention onboarding practices were reviewed throughout the project. Baseline qualitative interviews and the baseline survey were conducted in August 2019. Following implementation of the onboarding program in July 2020, all newly hired employees completed the 5-week onboarding program. Implementation surveys were administered to these new employees and their supervisors two and five weeks after the new employee’s first day of employment. Additionally, qualitative interviews with new employees were conducted on a rolling basis approximately 90 days after their first day of employment. A follow-up survey involving all FSP staff was conducted in October 2020, with a final survey in October 2021.