"But even if they are not valid, they do tell you something…."

Remember, “validity” means “they measure what you think they measure.” “Data driven” can also mean driven right off the side of the road.

From Inside Higher Ed

Zero Correlation Between Evaluations and Learning

New study adds to evidence that student reviews of professors have limited validity.
September 21, 2016 By Colleen Flaherty

 

A number of studies suggest that student evaluations of teaching are unreliable due to various kinds of biases against instructors. (Here’s one addressing gender.) Yet conventional wisdom remains that students learn best from highly rated instructors; tenure cases have even hinged on it.
What if the data backing up conventional wisdom were off? A new study suggests that past analyses linking student achievement to high student teaching evaluation ratings are flawed, a mere “artifact of small sample sized studies and publication bias.”
“Whereas the small sample sized studies showed large and moderate correlation, the large sample sized studies showed no or only minimal correlation between [student evaluations of teaching, or SET] ratings and learning,” reads the study, in press with Studies in Educational Evaluation. “Our up-to-date meta-analysis of all multi-section studies revealed no significant correlations between [evaluation] ratings and learning.”

Taking Note of Taking Notes on a Laptop

From Harvard Business Review

BUSINESS WRITING

What You Miss When You Take Notes on Your Laptop

Maggy McGloin

JULY 31, 2015

Even in my relatively short foray into office life, I notice that few people bring a pen and notebook to meetings. I’ve been told that over the years, the spiral notebooks and pens once prevalent during weekly meetings have been replaced with laptops and slim, touch-screen tablets.

I suppose it makes sense. In a demanding new age of technology, we are expected to send links, access online materials, and conduct virtual chats while a meeting is taking place. We want instant gratification, and sending things after the meeting when you’re back at your desk feels like too long to wait. It seems that digital note-taking is just more convenient.

But is longhand dead? Should you be embarrassed bringing a pen and paper to your meetings? To answer these questions, I did a little digging and found that the answer is no, according to a study conducted by Princeton’s Pam A. Mueller and UCLA’s Daniel M. Oppenheimer. Their research shows that when you only use a laptop to take notes, you don’t absorb new materials as well, largely because typing notes encourages verbatim, mindless transcription.

Mueller and Oppenheimer conducted three different studies, each addressing the question: Is laptop note taking detrimental to overall conceptual understanding and retention of new information?

For the first study, the researchers presented a series of TED talk films to a room of Princeton University students. The participants “were instructed to use their usual classroom note-taking strategy,” whether digitally or longhand, during the lecture. Later on, the participants “responded to both factual-recall questions and conceptual-application questions” about the film.

The students’ scores differed immensely between longhand and laptop note takers. While participants using laptops were found to take lengthier “transcription-like” notes during the film, results showed that longhand note takers still scored significantly higher on conceptually-based questions. Mueller and Oppenheimer predicted that the decrease in retention appeared to be due to “verbatim transcription.”

Read rest of article at HBR

Real "Competencies" for the 21st Century

Music to my ears. Sarah Lawrence, long known for its innovative approach to liberal arts education (still using narrative evaluations – something we could adopt at Mills to great effect IMHO), crafts a simple response to assessment madness and places it where it should be: at the student-advisor nexus.

Imagine: six goals that are about skill not ideological content; evaluated every semester in every course; tracked over time by student and advisor. Throw all the rest of the baroque apparatus away and get on with educating.

H/T to Mark Henderson

Play audio at MarketPlace Education

At Sarah Lawrence College in Bronxville, N.Y., about ten students — all women but one — sit at a round table discussing Jane Austen’s “Northanger Abbey.”

The 88-year-old college has a reputation for doing things differently. Most classes are small seminars like this one. There are no majors. Students do a lot of independent projects. And grades aren’t as important as the long written evaluations professors give every student at the end of every semester. It’s no surprise, then, that professor James Horowitz is skeptical of any uniform college rating system, like the one being proposed by the Obama administration.

“The goals that we are trying to achieve in instructing our students might be very different from what the University of Chicago or many other schools or a state school or a community college might be striving to achieve,” Horowitz says.

The Obama administration is due out this spring with details of its controversial plan to rate colleges on measures like value and affordability. The idea is that if students can compare schools on cost, graduation rates and even how much money students earn after they graduate — colleges might have to step up their game. Especially if, as proposed, poor performers risk losing access to federal financial aid.

All that, naturally, makes colleges just a bit nervous. Sarah Lawrence is fighting back with its own way of measuring value. The faculty came up with six abilities they think every Sarah Lawrence graduate should have. They include the ability to write and communicate effectively, to think analytically, and to accept and act on critique.

“We don’t believe that there’s like 100 things you should know when you graduate,” says computer science professor Michael Siff, who helped develop the tool. “It’s much more about are you a good learner? Do you know how to enter into a new domain and attack it with an open mind, but also an organized mind?”

Faculty advisors can use the results to track students’ progress over time and help them address any weaknesses. A student who’s struggling with communication could take class with a lot of oral presentations, for example, or make an appointment at the campus writing center.

But Siff says the tool is also about figuring out what the college can do better.
“This tool will allow us to assess ourselves as an institution,” he says. “Are we imparting what we believe to be these critical abilities?”

So how is the school doing? So far there are only data for two semesters, but on every measure seniors do better than juniors. Sophomores do better than freshmen.

Starting next fall, advisors will meet with their students at the beginning of each semester to talk over their progress. In sort of a trial run, Siff goes over the results so far with one of his advisees, junior Zachary Doege.

On a scale from “not yet developed” to “excellent,” he’s mostly at the top end. Doege says he likes seeing his own growth.

“I think the thing I like the most about this is just the fact that I can look back at how I was doing in previous semesters and sort of chart my own progress,” he says. “Not comparing me towards other students—just me to myself.”

That’s a different measure of the value of an education than, say, student loan debt or earnings after graduation — the sorts of things the Obama administration is considering as part of its ratings plan. Students and parents are right to ask if they’re getting their money’s worth, says the college’s president, Karen Lawrence. After financial aid, the average cost of a Sarah Lawrence education is almost $43,000 a year.

“People are worried about cost,” Lawrence says. “We understand that.”

And they’re worried about getting jobs after graduation. But she says the abilities that the new assessment measures—critical thinking and innovation and collaboration—are the same ones employers say they’re looking for.

“We think these are abilities that students are going to need both right after graduation and in the future, and so it could be an interesting model.”

One she hopes other schools will take a look at as they figure out how to answer the national debate about the value of college.