New Mexico State Improvement Grant: Measuring Up

Authors
Cathy Jones, Statewide Service Coordinator for NMPBS
Carlos Romero, Evaluator for NMPBS

Background

For the past four years New Mexico’s Public Education Department has used a portion of its State Improvement Grant (SIG) to support implementation of a Positive Behavior Support (PBS) framework to assist schools in establishing the social culture and behavioral supports needed to provide an effective learning environment for all students. NMPBS promotes a team-based, schoolwide system approach with the following features:

• Behavioral Expectations are defined: A small number of clearly defined behavioral expectations are defined in positive, simple, rules.
• Behavioral Expectations are taught. The behavioral expectations are taught to all students in the building and are taught in real contexts. The goals of the teaching are to take broad expectations and provide specific behavioral examples. Teaching occurs across the curriculum and throughout the school year.
• Appropriate Behaviors are acknowledged. Once appropriate behaviors have been defined and taught, they are acknowledged on a regular basis. Schools developed schoolwide systems that reinforce appropriate behavior across all settings and with all students.
• Consequences are clear and predictable. When students violate behavioral expectations, clear procedures are established for providing information to them that their behavior was unacceptable and for reinforcing/teaching appropriate expectations.
• Use of data for active decision-making. Teams actively engage in using meaning, current and relevant data to make effective decisions about school-wide, classroom, and individual student systems.
  
  

Evaluation Design, Measures and Results

The evaluation design is based on a simple logic model, illustrated below. The table that follows identifies the measures that are used to examine implementation processes, level and fidelity of implementation, and outcomes. Results can be discussed in terms of infrastructure, implementation, and impact.

Infrastructure

NMPBS has successfully built a broad and sound infrastructure to sustain and scale-up implementation of PBS. This has been accomplished in a relatively short period of time. Participation in PBS has grown steadily each year with 44 districts and 134 schools (including those from Juvenile Justice) by 2005-2006 with representation from all regions of the state. Of the 44 districts participating, 17 are considered district-wide initiatives.

Participating Districts and Number of Schools Participating by Grade Level 2002-2006

New Mexico has established the capacity to support initial training and ongoing support with 52 coaches, 15 SWIS facilitators, and 15 trainers. NMPBS features a statewide leadership team made up of key state, regional, district and school level stakeholders. The leadership team has completed regular self-assessments and developed a long range plan for sustaining and scaling-up PBS. The Leadership Team has identified exemplar schools that will serve as demonstration sites. In transitioning from a model o f high dependence on University of Oregon consultants to one that relies primarily by New Mexico resources, NMPBS has continued to increase participation, maintain a high standard of quality for training (evidenced by satisfaction and feedback evaluations), and exhibit high levels of implementation.

PBS Schools in Participating Districts (44 of 89 Districts)

Implementation

NMPBS monitors the level and fidelity of implementation using the Team Implementation Checklist (TIC), which measures the extent to which key features of implementation are in place. Schools submit checklist data three times per year and it is reported to school teams, trainers/coaches, and the statewide leadership team. Data are disaggregated by cohort and grade level. The charts that follow illustrate how data are communicated by overall percentage of implementation and by each sub-scale of the checklist. Data are used to inform training and coaching decisions. TIC data allow for comparison of outcomes based on degree of implementation. University of Oregon research has determined that when schools reach an implementation level of 80% on the TIC, they show improved student outcomes. Over one-third of schools that have implemented for two years or more have met criterion.

Impact

Ultimately, NMPBS revolves around student-centered outcomes in three areas: improved behavior measured by discipline data, increased school safety measured by prevalence of risk and protective factors, and higher levels of academic achievement measured by percentages of students meeting state proficiency standards. A number of New Mexico schools have shown progress in improving behavior based on decreases in Office Discipline Referrals (ODR). The following charts illustrate the inverse relationship between implementation and ODR in three sample schools. As implementation increases, behavioral referrals decrease. ODR is calculated per day per 100 students to provide a standard, comparable measure.

Sample NMPBS Schools: Implementation and Office Discipline Referrals
Elementary School	Middle School	High School

In addition to the success stories above, NMPBS looks at behavior on a broader scale. The charts below show ODR (standardized per day per 100 students) for all elementary, middle and high schools that are using the Schoolwide Information Systems (SWIS). The data show some improvement. The number of New Mexico schools using SWIS has increased each year, accounting in part for the increases in level of discipline referrals, particularly for high schools. It is important to note that the national SWIS data represents a high standard given that these schools are actively implementing PBS. SWIS has been essential in providing timely and detailed behavior data for schools to identify areas of concern and monitor results and to monitor progress at the state level.

National Mean for all SWIS schools (1,668 schools, 838,184 students)

School safety is measured based on prevalence of risk and protective factors as determined through the University of Oregon’s School Safety Survey. Results from last year show that NMPBS schools are decreasing risk factors and increasing protective factors.  Those schools that are implementing with fidelity are showing better results.  Teams have used results as part of their needs assessment and action planning.

School Safety Survey:  Implementation and Risk/Protective Factors

Risk Factor

Protective Factor

NMPBS is just beginning to examine academic performance data on a broad scale.  This is explained, in part, by the fact that comparable data is just becoming available.  Because the state was transitioning from a norm-referenced to criterion referenced statewide assessment, 2005-2006 was the first time the same test was administered to the same grade-levels for two consecutive years.  In addition, while improving behavior can have a direct impact on academic achievement, NMPBS has not focused intently on academic interventions.  The tables that follow show no consistent or significant differences in the percentages of students who meet proficiency requirements between high implementing and low implementing schools.  On average, NMPBS schools seem to keep pace with state requirements.  However, this masks the variation between schools and within school sub-groups and explains why 66% of NMPBS schools did not meet AYP requirements.

NMPBS Elementary Schools: Average Percentage of Students Proficient

NMPBS Middle Schools: Average Percentage of Students Proficient

NMPBS High Schools: Average Percentage of Students Proficient

Next Steps: Scaling-Up
NMPBS was the basis for New Mexico’s State Personnel Development Grant (SPDG) entitled Scaling-Up a Response to Intervention Framework for Behavior and Literacy.   In making the case for scaling up, the infrastructure and capacity established through NMPBS was emphasized. SPDG will serve to align initiatives around State Performance Plan indicators using Response to Intervention as a common framework to braid behavior and literacy and a Professional Learning Community model to guide the work of district and school-based teams.