Past Reports

Indiana State Improvement Grant: Measuring Up

The Indiana State Improvement Grant (IN SIG) is currently in its third year of existence. The first year of the project was dedicated to organization and program development at the state and local level. Particular emphasis was placed on creating avenues that would allow creativity and support school improvement/student achievement efforts and “system change” outcomes for the six LEA partnership sites invited to participate in the project. Part of the planning process for IN SIG included the development of a comprehensive evaluation model to assess multiple inputs concerning state and LEA activities supported by the project. The IN SIG evaluation model was designed to collect data from three distinct sources: a) Indiana Department of Education evaluation data; b) School District Self Study Needs Assessment data; c) School District Action Item Initiatives data. Triangulation of data and longitudinal trend analysis are two essential components of the IN SIG evaluation model. The intent of IN SIG evaluation is to provide a rich pool of quantitative and qualitative data to US DOE, state and LEA stakeholders. The IN SIG has gone through two cycles of data collection (school years 2004-05 and 2005-06).

Indiana State Improvement Grant (IN SIG) Evaluation Model

Data reporting elements for each phase of the IN SIG Evaluation Model include the following components:

1) Indiana Department of Education evaluation data consisting of two elements

a) IN DOE Accountability System for Academic Progress as reported by the Indiana State Test of Educational Proficiency (ISTEP+) and the Graduation Qualifying Exam (GQE) given annually as progress indicators for NCLB and Indiana's PL 221 School Improvement Process (the state's version of NCLB implemented in 1997). Data sources include test results for 3 rd , 6 th , 8 th , and 10 th graders in Language Arts and Mathematics.

b) IN DOE Division of Exceptional Learners' Continuous Improvement Monitoring (CIM) indicators that include: a) overall percentage of student population placed in special education; b) special education placement patterns by percentage placed in LRE pre-K through 12; c) high school graduation rates; and d) suspension and expulsion rates.

These data represent broad-based benchmarking indicators reported as trend analysis by IN SIG LEA partnership site and state percentages.

2) School District Self Study Needs Assessment data consists of an electronic version of the IN SIG Needs Assessment Survey distributed by Email using Ball State University's INQSIT.

The survey includes multiple parallel forms designed for specific stakeholder groups by position and school level distributed to LEA personnel. Survey sections include: a) demographic information; b) access to the general education curriculum; c) progress in the general education curriculum; d) outcomes for students with disabilities; e) technology usage; f) instruction – curriculum alignment/training/classroom teaching; g) behavior; h) family-student involvement; i) secondary transition services; j) state-based resources; and k) professional development.

Data are analyzed using descriptive and inferential statistics. Item ratings using a five point Likert-type scale are analyzed using one-way analysis of variance (ANOVA) and Kruskal-Wallis procedures. Data are reported by school level (e.g. Elementary) and personnel classification (e.g. K-12 General Educator) for all survey sections by each specific item. Technical reports include tables, figures, narrative analysis, and open-ended question responses concerning suggested school improvement by survey section. Reports range from 75 to 125 pages in length including 29 tables and 8 figures for each IN SIG site. Summary reports and a compiled IN SIG site comparison report are distributed with the technical report for IN SIG coordinators and LEA management teams' consideration. This data is used in developing School District Action Items and LEA Action Plans for IN SIG improvement activities.

3) School District Action Item Initiatives data include qualitative indicators and continuous feedback sources designed to capture site specific data concerning school improvement efforts. Data are reported using two general methods:

a) School district and building-based IN SIG action plans with case study narrative analysis and site specific data sources related to the efficacy of specified targeted initiatives. Data reported here could include evaluation indicators from sources such as the school's PL 221 School Improvement Plan, national testing data, local assessment information, and/or informal assessments.

b) Site specific IN SIG initiative reporting is accomplished as part of an on-going process of continuous feedback as part of the school improvement cycle. The catalyst for this phase of local data analysis is highlighted in the Year-End Synopsis Report. This report is a comprehensive consistent approach to data reporting across IN SIG sites at the LEA level.

The Year-End Synopsis Report includes initiatives/interventions reporting as data statements for a) areas of concern in student achievement that these initiatives/interventions will address; b) what change in practice is expected as a result of the initiatives/interventions; c) what activities have been incorporated to develop the necessary skills to implement the new practices; d) demonstrate that the new practices are being implemented with integrity; e) show evidence of the impact on student achievement; f) how have school, families, and community partnerships been used to support the initiatives/interventions; and g) next steps.

Many of the current initiatives/interventions being implemented at the LEA level include literacy development, positive behavior supports, inclusion and access “best practices”, differentiated instruction, and co-teaching/collaboration.

Indiana State Improvement Grant (IN SIG) Data Collection and Evaluation Components for Academic Years 2004-05 and 2005-06

Next Steps for IN SIG Evaluation

Data management, analysis, and triangulation are current projects underway in the IN SIG evaluation model. We currently have the second cycle of IN SIG site technical reports in draft form and are developing a composite analysis of first and second cycle Needs Assessment Survey results for the first iteration of longitudinal analysis. These data will establish benchmarking trend lines for the project. Additionally, Indiana DOE data concerning trend line comparisons have been (i.e. ISTEP+/GQE) and/or are being compiled (i.e. CIM indicators). As mentioned, the Year-End Synopsis Report is part of an on-going process of continuous feedback concerning the IN SIG school improvement cycle. Results of this phase of the evaluation model will be essential as IN SIG identifies initiatives that have merit concerning future scaling up efforts. These data, cross-referenced with state and district self assessment data, will form the basis for future scaling up initiatives and activities under the IN SIG. The current intent of the IN SIG project is to identify several scaling up potential initiatives/interventions and to provide the state and Indiana's school districts with a decision-making process to adopt, implement, and institutionalize evidence-based practices with LEA expertise linking to a potential “train the trainer” model for implementation. Indiana's Scaling Up Model will be an outgrowth of the current IN SIG projects and a future process for moving school improvement from theory to practice.

Indiana and all the IN SIG stakeholders have dedicated efforts under this project to develop a comprehensive model of system change predicated on strong evaluation from multiple sources, triangulated to validate school improvement targeted at initiatives that affect student achievement.

For more information concerning the Indiana State Improvement Grant or the program evaluation model please contact:

Dr. Brett E. Bollinger, IN SIG Co-Director at bebollin@indiana.edu

Dr. Michael W. Harvey, IN SIG Evaluation Consultant at mwharvey@bsu.edu