Psychosocial Intervention Psychosocial Intervention
Psychosocial Intervention 26 (2017) 147-54 - Vol. 26 No.3 DOI: 10.1016/j.psi.2016.12.002
Efectos de la fidelidad de la implementación sobre los resultados de una intervención preventiva en salud mental escolar: un análisis multinivel
Effects of implementation fidelity of the results of a preventive intervention in school mental health: A multilevel analysis
Rodrigo Rojas-Andradea,, , Loreto Leiva-Bahamondesb, Ana María Belén Vargasc, Ana María Squicciarini-Navarroc
a Escuela de Psicología, Universidad Academia de Humanismo Cristiano, Santiago, Chile
b Departamento de Psicología, Universidad de Chile, Santiago, Chile
c Junta Nacional de Auxilio Escolar y Becas (JUNAEB), Santiago, Chile
Received 14 November 2016, Accepted 15 December 2016
Resumen

El propósito de esta investigación fue examinar los efectos de la fidelidad de la implementación sobre los resultados de una intervención en salud mental escolar. A través de un diseño de investigación pretest/postest se evaluó la desadaptación escolar a 1,414 estudiantes chilenos de 510 escuelas participantes de un taller preventivo selectivo realizado por 94 equipos psicosociales del programa Habilidades para la Vida. Por su parte, la fidelidad (dosis recibidas por los padres, profesores y estudiantes; pertinencia de la intervención, compromiso directivo, adherencia y satisfacción laboral) fue medida a través de los productos permanentes completados por los equipos durante el período de ejecución del taller y relacionada con los resultados mediante un análisis multinivel. Se encontró una alta fidelidad en los componentes implementados, a excepción de la adherencia (29.8%) y la dosis recibida por los padres (50.1%). Se logró evidenciar que los resultados varían principalmente a nivel del estudiante y que están relacionados con la cantidad de sesiones a las que asisten y con la pertinencia de la intervención para ellos. Se discute la importancia de considerar el contexto relacional en la implementación de intervenciones en el mundo real hispanoamericano.

Keywords
  • Fidelidad de la implementación
  • Intervención psicosocial
  • Salud mental escolar
  • Ciencia de la implementación
Abstract

The purpose of this research was to examine the effects of implementation fidelity on the results of a school mental health intervention. Through a pre-test/post-test research design, school inadaptation was assessed on 1,414 Chilean students from 510 participating schools of a selective preventive workshop conducted by 94 psychosocial teams from the Skills for Life program. Fidelity (doses received by parents, teachers and students, intervention relevance, management commitment, adherence and job satisfaction) was measured through the permanent products completed by the teams during the workshop period and related to the results through multilevel regression analysis. A high fidelity was found in the implemented components, with the exception of adherence (29.8%) and the dose received by parents (50.1%). It was shown that the results vary mainly at the student level and are related to the number of sessions they attend and the relevance of the intervention for them. The importance of considering the relational context in the implementation of interventions in the Spanish-American real world is discussed.

Keywords
  • Implementation fidelity
  • Psychosocial intervention
  • School mental health
  • Implementation science
NOTICE Undefined offset: 1 (includes_ws/librerias/utilidades.php[294])
NOTICE Undefined index: url (includes_ws/librerias/utilidades.php[295])
NOTICE Undefined index: pii (includes_ws/librerias/utilidades.php[310])
Bibliografía
Anello et al., 2016
V. Anello,M. Weist,L. Eber,S. Barrett,J. Cashman,M. Rosser
Readiness for positive behavioral interventions and supports and school mental health interconnection: Preliminary development of a stakeholder survey
Journal of Emotional and Behavioral Disorders, (2016), http://dx.doi.org/10.1177/1063426616630536
Ato et al., 2013
M. Ato,J.J. López,A. Benavente
Un sistema de clasificación de los diseños de investigación en psicología
Anales de Psicología, 29 (2013), pp. 1038-1059
Barnett et al., 2014
D. Barnett,R. Hawkins,D. McCoy,E. Wahl,A. Shier,H. Denune
Methods used to document procedural fidelity in school-based intervention research
Journal of Behavioral Education, 23 (2014), pp. 89-107
Berry et al., 2015
V. Berry,N. Axford,S. Blower,R.S. Taylor,R.T. Edwards,K. Tobin
The effectiveness and micro-costing analysis of a universal, school-based, socia-emotional learning programme in the UK: A cluster-randomised controlled trial
School Mental Health, 8 (2015), pp. 1-19
Blase et al., 2012
K. Blase,M. Van Dyke,D. Fixsen,F. Wallace
Key concepts, themes, and evidence for practitioners in educational psychology
Handbook of Implementation Science for psychology in Education, pp. 13-34
Bond et al., 2000
G.R. Bond,L. Evans,M.P. Salyers,J. Williams,H.-W.W. Kim
Measurement of fidelity in psychiatric rehabilitation
Mental Health Services Research, 2 (2000), pp. 75-87
Bradshaw et al., 2009
C.P. Bradshaw,J.H. Zmuda,S.G. Kellam,N.S. Ialongo
Longitudinal impact of two universal preventive interventions in first grade on educational outcomes in high school
Journal of Educational Psychology, 101 (2009), pp. 926-937 http://dx.doi.org/10.1037/a0016586
Burns et al., 1995
B.J. Burns,E.J. Costello,a. Angold,D. Tweed,D. Stangl,E.M. Farmer
Children's mental health service use across service sectors
Health Affairs, 14 (1995), pp. 147-159
Christl et al., 2010
B. Christl,M.F. Harris,U.W. Jayasinghe,J. Proudfoot,J. Taggart,J. Tan
Readiness for organisational change among general practice staff
Quality & Safety in Health Care, 19 (2010), pp. e12 http://dx.doi.org/10.1111/1471-0528.15015
Codding y Lane, 2015
R.S. Codding,K.L. Lane
A spotlight on treatment intensity: An important and often overlooked component of intervention inquiry
Journal of Behavioral Education, 24 (2015), pp. 1-10
Dane y Schneider, 1998
A.V. Dane,B.H. Schneider
Program integrity in primary and early secondary prevention: Are implementation effects out of control?
Clinical Psychology Review, 18 (1998), pp. 23-45
Domitrovich et al., 2010
C.E. Domitrovich,S.D. Gest,D. Jones,S. Gill,R.M. Sanford deRousie
Implementation quality: Lessons learned in the context of the Head Start REDI trial
Early Childhood Research Quarterly, 25 (2010), pp. 284-298 http://dx.doi.org/10.1016/j.ecresq.2010.04.001
DuBois et al., 2002
D.L. DuBois,B.E. Holloway,J.C. Valentine,H. Cooper
Effectiveness of mentoring programs for youth: A meta analytic review
American Journal of Community Psychology, 30 (2002), pp. 157-198 http://dx.doi.org/10.1023/A:1014628810714
Dupaul, 2009
G.J. Dupaul
Assessing integrity of intervention implementation: Critical factors and future directions
School Mental Health, 1 (2009), pp. 154-157
Durlak, 2013
J.A. Durlak
The importance of quality implementation for research, practice, and policy
ASPE Research Brief, (2013), pp. 1-16
February
Durlak y DuPre, 2008
J.A. Durlak,E.P. DuPre
Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation
American Journal of Community Psychology, 41 (2008), pp. 327-350 http://dx.doi.org/10.1007/s10464-008-9165-0
Durlak et al., 2011
J.A. Durlak,R.P. Weissberg,A.B. Dymnicki,R.D. Taylor,K.B. Schellinger
The impact of enhancing students’ social and emotional learning: A meta-analysis of school-based universal interventions
Child Development, 82 (2011), pp. 405-432 http://dx.doi.org/10.1111/j.1467-8624.2010.01564.x
Eccles y Mittman, 2006
M.P. Eccles,B.S. Mittman
Welcome to implementation science
Implementation Science, 3 (2006), pp. 1-3 http://dx.doi.org/10.1186/1748-5908-3-1
Faragher, 2005
E.B. Faragher
The relationship between job satisfaction and health: A meta-analysis
Occupational and Environmental Medicine, 62 (2005), pp. 105-112 http://dx.doi.org/10.1136/oem.2002.006734
Fixsen et al., 2009
D.L. Fixsen,K.A. Blase,S.F. Naoom
Core implementation components
Research on Social Work Practice, 19 (2009), pp. 531-540
Flannery et al., 2014
K.B. Flannery,P. Fenning,M.M. Kato,K. McIntosh
Effects of school-wide positive behavioral interventions and supports and fidelity of implementation on problem behavior in high schools
School Psychology Quarterly, 29 (2014), pp. 111-124 http://dx.doi.org/10.1037/spq0000039
Forman et al., 2009
S.G. Forman,S.S. Olin,K.E. Hoagwood,M. Crowe,N. Saka
Evidence-based interventions in schools: Developers’ views of implementation barriers and facilitators
School Mental Health, 1 (2009), pp. 26-36
Forman et al., 2013
S.G. Forman,E.S. Shapiro,R.S. Codding,J.E. Gonzales,L.A. Reddy,S.A. Rosenfield
Implementation science and school psychology
School Psychology Quarterly, 28 (2013), pp. 77-100 http://dx.doi.org/10.1037/spq0000019
Fredricks et al., 2004
J.A. Fredricks,P.C. Blumenfeld,A.H. Paris
School engagement: Potential of the concept, state of the evidence
Review of Educational Research, 74 (2004), pp. 59-109
Gelman y Hill, 2007
A. Gelman,J. Hill
Data analysis using regression and multilevel/hierarchical models
Cambridge University Press, (2007)pp. 1-651
George et al., 2004
M. George,A.M. Squicciarini,R. Zapata,M.P. Guzmán,M. Hartley,C. Silva
Detección precoz de factores de riesgo de salud mental en escolares
Revista de Psicología, 12 (2004), pp. 9-20
Gregory et al., 2007
A. Gregory,D.B. Henry,M.E. Schoeny,L. Eron,N. Guerra,D. Henry
School climate and implementation of a preventive intervention
American Journal of Community Psychology, 40 (2007), pp. 250-260 http://dx.doi.org/10.1007/s10464-007-9142-z
Gresham, 2009
F.M. Gresham
Evolution of the treatment integrity concept: Current status and future directions
School Psychology Review, 38 (2009), pp. 533-540
Gresham et al., 1993
F.M. Gresham,K.A. Gansle,G.H. Noell,S. Cohen,S. Rosenblum
Treatment integrity of school based behavioral intervention studies
School Psychology Review, 22 (1993), pp. 254-272
Guzmán et al., 2015
J. Guzmán,R.C. Kessler,A.M. Squicciarini,M. George,L. Baer,K.M. Canenguez
Evidence for the effectiveness of a national school-based mental health program in Chile
Journal of the American Academy of Child and Adolescent Psychiatry, 54 (2015), pp. 799-807 http://dx.doi.org/10.1016/j.jaac.2015.07.005
Hagermoser-Sanetti y Kratochwill, 2009
L. Hagermoser-Sanetti,T.R. Kratochwill
Toward developing a science of treatment integrity: Introduction to the special series
School Psychology Review, 38 (2009), pp. 445-459
Humphrey, 2013
N. Humphrey
Social and emotional learning a critical appraisal
Sage Publications, (2013)
Judge et al., 2001
T.A. Judge,C.J. Thoresen,J.E. Bono,G.K. Patton
The job satisfaction-job performance relationship: A qualitative and quantitative review
Psychological Bulletin, 127 (2001), pp. 376-407
Klassen y Chiu, 2010
R.M. Klassen,M.M. Chiu
Effects on teachers’ self-efficacy and job satisfaction: Teacher gender, years of experience, and job stress
Journal of Educational Psychology, 102 (2010), pp. 741-756
Koth et al., 2009
C.W. Koth,C.P. Bradshaw,P.J. Leaf
Teacher observation of classroom adaptation-checklist: Development and factor structure
Measurement and Evaluation in Counseling and Development, 42 (2009), pp. 5-30
Laker et al., 2014
C. Laker,F. Callard,C. Flach,P. Williams,J. Sayer,T. Wykes
The challenge of change in acute mental health services: Measuring staff perceptions of barriers to change and their relationship to job status and satisfaction using a new measure (VOCALISE)
Implementation Science: IS, 9 (2014), pp. 23 http://dx.doi.org/10.1186/1748-5908-9-23
Langley et al., 2010
A.K. Langley,E. Nadeem,S.H. Kataoka,B.D. Stein,L.H. Jaycox
Evidence-based mental health programs in schools: Barriers and facilitators of successful implementation
School Mental Health, 2 (2010), pp. 105-113 http://dx.doi.org/10.1007/s12310-010-9038-1
Leff et al., 2009
S.S. Leff,J.A. Hoffman,R.L. Gullan
Intervention integrity: New paradigms and applications
School Mental Health, 1 (2009), pp. 103-106 http://dx.doi.org/10.1007/s12310-009-9013-x
Leiva et al., 2015a
L. Leiva,M. George,A. Antivilo,A. Simonsohn,A. Antivilo,A.M. Squicciarini
Salud mental escolar: logros de una intervención preventiva en salud mental en niños y niñas del primer ciclo de enseñanza básica
Psicoperspectivas, 14 (2015), pp. 31-41
Disponible en: http://www.psicoperspectivas.cl/index.php/psicoperspectivas/article/view/508/394
Leiva et al., 2015b
L. Leiva,M. George,A.M. Squicciarini,A. Simonsohn,J. Guzmán
Intervención preventiva de salud mental escolar en adolescentes: desafíos para un programa público en comunidades educativas
Universitas Psychologica, 14 (2015), pp. 1285-1298
Low et al., 2016
S. Low,K. Smolkowski,C. Cook
What constitutes high-quality implementation of SEL programs? A latent class analysis of Second Step® implementation
Prevention Science, 17 (2016), pp. 981-991 http://dx.doi.org/10.1007/s11121-016-0670-3
Low et al., 2014
S. Low,M.J. van Ryzin,E.C. Brown,B.H. Smith,K.P. Haggerty
Engagement matters: Lessons from assessing classroom implementation of steps to respect: a bullying prevention program over a one-year period
Prevention Science, 15 (2014), pp. 165-176 http://dx.doi.org/10.1007/s11121-012-0359-1
Marks, 2000
H.M. Marks
Student engagement in instructional activity: Patterns in the elementary, middle, and high school years
American Educational Research Journal, 37 (2000), pp. 153-184
McGinty et al., 2011
A.S. McGinty,A. Breit-Smith,X. Fan,L.M. Justice,J.N. Kaderavek
Does intensity matter? Preschoolers’ print knowledge development within a classroom-based intervention
Early Childhood Research Quarterly, 26 (2011), pp. 255-267
Mendive et al., 2016
S. Mendive,C. Weiland,H. Yoshikawa,C. Snow
Opening the black box: Intervention fidelity in a randomized trial of a preschool teacher professional development program
Journal of Educational Psychology, 108 (2016), pp. 130-145
Murillo, 2008
J. Murillo
Los modelos multinivel como herramienta para la investigación educativa
Magis. Revista Internacional de Investigación en Educación, 1 (2008), pp. 45-62
Patel et al., 2007
V. Patel,A.J. Flisher,S. Hetrick,P. McGorry
Mental health of young people: A global public-health challenge
Perepletchikova, 2011
F. Perepletchikova
On the topic of treatment integrity
Clinical Psychology: Science and Practice, 18 (2011), pp. 148-153
Perepletchikova et al., 2009
F. Perepletchikova,L.M. Hilt,E. Chereji,A.E. Kazdin
Barriers to implementing treatment integrity procedures: Survey of treatment outcome researchers
Journal of Consulting and Clinical Psychology, 77 (2009), pp. 212-218 http://dx.doi.org/10.1037/a0015232
Pettigrew et al., 2015
J. Pettigrew,J. Graham,M. Miller-Day,M. Hecth,J. Krieger,Y. Ju
Adherence and delivery: Implementation quality and program outcomes for the 7th grade keepin’ it REAL program
Prevention Science, 16 (2015), pp. 90-99 http://dx.doi.org/10.1007/s11121-014-0459-1
Proctor et al., 2011
E. Proctor,H. Silmere,R. Raghavan,P. Hovmand,G. Aarons,A. Bunger
Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda
Administration and Policy in Mental Health and Mental Health Services Research, 38 (2011), pp. 65-76 http://dx.doi.org/10.1007/s10488-010-0319-7
Reiss, 2013
F. Reiss
Socioeconomic inequalities and mental health problems in children and adolescents: A systematic review
Social Science & Medicine, 90 (2013), pp. 24-31 http://dx.doi.org/10.17226/24744
Rodrigo, 2016
M.J. Rodrigo
Quality of implementation in evidence-based positive parenting programs in Spain: Introduction to the special issue
Psychosocial Intervention, 25 (2016), pp. 63-68
Rosenblatt y Elias, 2008
J.L. Rosenblatt,M.J. Elias
Dosage effects of a preventive social-emotional learning intervention on achievement loss associated with middle school transition
Journal of Primary Prevention, 29 (2008), pp. 535-555 http://dx.doi.org/10.1007/s10935-008-0153-9
Sarno et al., 2014
J. Sarno,A.R. Lyon,N.E. Brandt,C.M. Warner,E. Nadeem,C. Spiel
Implementation science in school mental health: Key constructs in a developing research agenda
School Mental Health, 6 (2014), pp. 99-111 http://dx.doi.org/10.1007/s12310-013-9115-3
Schoenwald y Garland, 2013
S.K. Schoenwald,A.F. Garland
A review of treatment adherence measurement methods
Psychological Assessment, 25 (2013), pp. 146-156 http://dx.doi.org/10.1037/a0029715
Schulte et al., 2009
A.C. Schulte,J.E. Easton,J. Parker
Advances in treatment integrity research: Multidisciplinary perspectives on the conceptualization, measurement, and enhancement of treatment integrity
School Psychology Review, 38 (2009), pp. 541-546
Sklad et al., 2012
M. Sklad,R. Diekstra,M.De Ritter,J. Ben,C. Gravesteijn
Effectiveness of school-based universal social, emotional, and behavioral programs: Do they enhance students’ development in the area of skill, behavior, and adjustment?
Psychology in the Schools, 49 (2012), pp. 892-909
Smith et al., 2004
J.D. Smith,B.H. Schneider,P.K. Smith,K. Ananiadou
The effectiveness of whole-school antibullying programs: A synthesis of evaluation research
School Psychology Review, 33 (2004), pp. 547-560
Tobler, 1986
N.S. Tobler
Meta-analysis of 143 adolescent drug prevention programs: Quantitative outcome results of program participants compared to a control or comparison group
Journal of Drug Issues, 16 (1986), pp. 537-567
Vargas y Peña, 2016
B. Vargas,F. Peña
Manual de talleres preventivos Habilidades para la Vida I
Santiago de Chile, (2016)
Weare y Nind, 2011
K. Weare,M. Nind
Mental health promotion and problem prevention in schools: What does the evidence say?
Health Promotion International, 26 (2011), pp. i29-i69 http://dx.doi.org/10.1093/heapro/dar075
Wilson et al., 2003
S.J. Wilson,M.W. Lipsey,J.H. Derzon
The effects of school-based intervention programs on aggressive behavior: A meta-analysis
Journal of Consulting and Clinical Psychology, 71 (2003), pp. 136-149
Zins et al., 2007
J.E. Zins,M.R. Bloodworth,R.P. Weissberg,H.J. Walberg
The scientific base linking social and emotional learning to school success
Journal of Educational and Psychological Consultation, 17 (2007), pp. 191-210
Zvoch, 2012
K. Zvoch
How does fidelity of implementation matter? Using multilevel models to detect relationships between participant outcomes and the delivery and receipt of treatment
American Journal of Evaluation, 33 (2012), pp. 547-565
WARNING Invalid argument supplied for foreach() (includes_ws/librerias/html/item.php[1199])
Copyright © 2017. Colegio Oficial de Psicólogos de Madrid
Psychosocial Intervention 26 (2017) 147-54 - Vol. 26 No.3 DOI: 10.1016/j.psi.2016.12.002