"But even if they are not valid, they do tell you something…."

Remember, “validity” means “they measure what you think they measure.” “Data driven” can also mean driven right off the side of the road.

From Inside Higher Ed

Zero Correlation Between Evaluations and Learning

New study adds to evidence that student reviews of professors have limited validity.
September 21, 2016 By Colleen Flaherty


A number of studies suggest that student evaluations of teaching are unreliable due to various kinds of biases against instructors. (Here’s one addressing gender.) Yet conventional wisdom remains that students learn best from highly rated instructors; tenure cases have even hinged on it.
What if the data backing up conventional wisdom were off? A new study suggests that past analyses linking student achievement to high student teaching evaluation ratings are flawed, a mere “artifact of small sample sized studies and publication bias.”
“Whereas the small sample sized studies showed large and moderate correlation, the large sample sized studies showed no or only minimal correlation between [student evaluations of teaching, or SET] ratings and learning,” reads the study, in press with Studies in Educational Evaluation. “Our up-to-date meta-analysis of all multi-section studies revealed no significant correlations between [evaluation] ratings and learning.”

Evaluating and Assessing Short Intensive Courses

Two articles on the topic of assessing and evaluating short, intensive courses.  Most of the results appear positive in terms of learning outcomes, but there are a number of factors associated with variations in outcomes that appear worth paying attention to.

Using a database of over 45,000 observations from Fall, Spring, and Summer semesters, we investigate the link between course length and student learning. We find that, after controlling for student demographics and other characteristics, intensive courses do result in higher grades than traditional 16 week semester length courses and that this benefit peaks at about 4 weeks. By looking at future performance we are also able to show that the higher grades reflect a real increase in knowledge and are not the result of a “lowering of the bar” during summer. We discuss some of the policy implications of our findings.


Altogether, we found roughly 100 publications that, in varying degrees, addressed intensive courses. After reviewing the collective literature, we identified four major lines of related inquiry: 1) time and learning studies; 2) studies of educational outcomes comparing intensive and traditional formats; 3) studies comparing course requirements and practices between intensive and traditional 

Scott and Conrad finish their literature review with several sets of open research questions suggested by their research:


  1. How do course requirements and faculty expectations of students compare between intensive and traditional formats and, if different, how does this affect the learning environment and student learning outcomes?
  2. How do student’s study patterns compare between intensive and traditional length courses?

Learning Outcomes

  1. How do pedagogical approaches compare between intensive and traditional length courses and, if different, how do these variations affect learning?
  2. How does the amount of time-on-task (i.e., productive class time) compare between intensive and traditional-length courses?
  3. How do stress and fatigue affect learning in intensive courses?
  4. Are intensive courses intrinsically rewarding and if so, how does that affect the classroom experience and learning outcomes?
  5. How do the immediate (short-term) and long-term learning outcomes compare between intensive and traditional-length courses?
  6. How do different student groups compare in their ability to learn under intensive conditions? For example, do older and younger students learn equally well in intensive courses?
  7. How does the degree of intensity influence student achievement? Do three week courses yield equivalent results to eight-week courses?
  8. How does the subject matter influence outcomes in intensive courses?
  9. Which kinds and levels of learning are appropriate for intensive formats?
  10. How do course withdrawals and degree completion rates compare between students who enroll in intensive versus traditional courses?
  11. How do intensive courses influence a student’s attitude toward learning?

Optimizing Factors and Conditions

  1. What disciplines and types of courses are best suited for intensive formats?
  2. What type of students are best suited for intensive formats?
  3. What types of pedagogical styles and instructional practices are best suited for intensive formats? Must teaching strategies change for intensive courses to be effective?
  4. Can certain instructional practices optimize learning?
  5. Do learning strategies differ between intensive and traditional-length courses and if so, can students effectively “learn how to learn” in time compressed formats? In other words, can students be taught effective learning strategies for intensive courses that would enhance achievement outcomes?

See Also

John V. Kucsera & Dawn M. Zimmaro Comparing the Effectiveness of Intensive and Traditional CoursesCollege Teaching Volume 58, Issue 2, 2010, pages 62-68