How We Measure Program Impact
Click on the image to download the data trends sheet on our Certificate in College Teaching & Learning programs.
Systematic Inquiry Projects (SIP) by HSI Faculty
Click here to see the SIP of Dr. Melissa McAlexander (STEM Success Seminar, Notre Dame de Namur University, Belmont CA)
Click here to see the SIP of Dr. Khameeka Kitt-Hopper (Anatomy I, Notre Dame de Namur University, Belmont CA) Click here to see the SIP of Dr. Matthew Steffenson (Introductory Biology, Adams State University, Alamosa CO) |
During 2013-2014, we conducted pilot programs with two community colleges: UNM -Taos (Taos, NM) and Imperial Valley College (El Centro, CA), and one four-year open enrollment institution: New Mexico Highlands University (Las Vegas, NM). We found that using empirical data, collected and analyzed by faculty members, was the best method to drive implementation of evidence-based practices, and sustain changes to teaching behaviors.
We use the following 3 data collection techniques in our programs to measure our programs' effectiveness. Instrument #1: Teaching Observation Protocols to Measure Student Engagement Faculty use a video observation protocol called the TOPSE* (Teaching Observation Protocol for Student Engagement) to collect time-stamped data and code two 45 minute teaching videos for instances of specific evidence-based student and teacher behaviors. They compare their TOPSE data at the beginning and end of a single semester and use it to reflect on specific observable changes in their instruction and student engagement. We use the data to determine what practices had the highest impact on student learning. *Many elements of the TOPSE are based on the COPUS protocol, a validated instrument used in assessments of undergraduate STEM teaching (Smith et al.2013). Instrument #2: Achievement of a Data-Driven "Planning for Learning" Mindset To measure non-observable behaviors such as the increased knowledge of cognitive science to drive teaching behaviors and ability to use student formative assessments to drive teaching, faculty complete an Implementation of Highly Effective Practices (IHEP) rubric at the beginning and end of our semester-long programs. We use this ordinal data as a qualitative measure of our impact on attitude shifts that are correlated with published research on faculty-student interactions and minority student engagement. Instrument #3: Implementation of a Systematic Research Project (SIP) This project is designed and conducted while HSI faculty are in our Certificate program. The project is aligned with the tenets of the Scholarship of Teaching and Learning (SoTL) and helps faculty to develop a solid knowledge of the research behind the teaching change they are examining. In this project, faculty design an experimental study of a teaching change, and collect student learning data to evaluate its impact during a single semester. It serves as a pilot project for a larger scale quasi-experimental study that can be conducted in the next semester. Instrument #4: Student Learning Measures (under development) Faculty have used Pass/Fail rates and grade distributions as ways to determine impact of their SIP project. We find these to be crude measures of student learning and are looking in 2016 to find university partners to test the use of a pre/post survey on students that investigates development of study skills, memory retention techniques, and metacognition. A validated and reliable survey instrument will allow faculty to correlate specific teaching strategies with learning. |
