Thoughts on Teaching and Learning

Thoughts on Teaching and Learning

TLE Bloggers

  • Gender Biases and Teaching Evaluations

    A recent article in the New York Times, Is the Professor Bossy or Brilliant? Much Depends on Gender, highlights the advantage I, and my male colleagues, have over my fellow female instructors when it comes time to course evaluations. The article discusses work done by Benjamin Schmidt who has gone through data from over 14 million postings on Rate My Professors and created an interactive graphic that allows users to compare evaluations of male and female faculty members for particular descriptors. These comparisons between male and female faculty are broken down by discipline, so one can see that as a teacher of psychology, I am more likely to be described as funny, smart, and effective as compared to my female colleagues, though I am less likely to be described as organized. My fellow female instructors are more likely to be rated as rude, terrible or incompetent. As is discussed in the article, our unconscious biases serve as a lens through which we view other's behavior, and these biases have implications beyond the classroom.

    These data naturally raise concerns about how institutions use student feedback in the merit review process and how we judge a facutly member's success as an instructor, and I should also note that these data say nothing about any unconscious biases regarding race that no doubt impact how minority faculty are evaluatied relative to their majority peers. As a statistician, it seems obvious that some sort of correction needs to be introduced either in the form of computing a standardized course evaluation score wherein a faculty member's evaluation is scored relative to the population of other same sex or some regression based approach where gender and other variables like age and ethnicity are used to adjust an instrutor's evaluations. It's certainly something to chew on.


  • Technology Changing How Students Learn, Teachers Say

    I had a meeting with my TA this afternoon, and we were discussing questions that students had problems with on the last exam. One of the questions asked students how Baddley's model of memory differed from the traditional information processing model. The correct answer is that Baddley argues that information goes from sensory memory to long-term memory and then into short-term or working memory. It turns out that the students she talked with during her office hours had problems with the question because the lecture slide showed information from sensory memory going into implicit and explicit memory, which are two types of long-term memories, but the slide didn't have the words long-term memory written on it. So although I stated more than once and in more than one class that Baddley's model proposes that information from sensory memory goes first into long-term memory, they did not process that fact because it wasn't written on a slide. I find myself incredibly frustrated with this expectation that the only information necessary to attend to is that which is printed on the slide. I only put bullet points on my slides to serve as prompts for my lecture notes, and I emphasize from the very start of the semester that there is more to notetaking than just bullet points. In fact, I stop speaking at one point during my first lecture and ask students to look at their notes and compare what they have written down with my version of what I would have written had I been listening to the same lecture, and yet here we are halfway through the semester with that lesson unlearned.

    I then sat down with the New York Times and came across this recent article:

    There is a widespread belief among teachers that students’ constant use of digital technology is hampering their attention spans and ability to persevere in the face of challenging tasks, according to two surveys of teachers being released on Thursday. The researchers note that their findings represent the subjective views of teachers and should not be seen as definitive proof that widespread use of computers, phones and video games affects students’ capability to focus. Even so, the researchers who performed the studies, as well as scholars who study technology’s impact on behavior and the brain, say the studies are significant because of the vantage points of teachers, who spend hours a day observing students.

    The timing of the studies, from two well-regarded research organizations, appears to be coincidental. One was conducted by the Pew Internet Project, a division of the Pew Research Centerthat focuses on technology-related research. The other comes from Common Sense Media, a nonprofit organization in San Francisco that advises parents on media use by children. It was conducted by Vicky Rideout, a researcher who has previously shown that media use among children and teenagers ages 8 to 18 has grown so fast that they on average spend twice as much time with screens each year as they spend in school. (more)


    Comments: 1
  • Technology in Education: Revolution or Evolution?

    For all of the technology that I use in the conjunction with my teaching: PowerPoint, clickers, podcasts, and online resources, I don't believe that technology is sort of indispensible teaching tool. A good teacher is a good teacher with or without technology, and no amount of technological gadgets are going to make a bad teacher any better. This may be heresy to say aloud, but I don't believe that technology is the key to student success. What is most important is student engagement and effort both in and out of class, and a good instructor can nurture that engagement and increase student motivation. Adam Falk, the president of Williams College, spoke to these issues in an address this past April. He has allowed me to share his comments on the TLE site.

    Presented at “The Future of the Liberal Arts” Conference
    Lafayette College
    April 10, 2012

    It seems that you can’t pick up a newspaper or magazine–the New York Times education section, the Chronicle of Higher Education, the EDUCAUSE Review–without encountering the passionate assertion that information technology has changed everything about our students and how we must educate them. Streaming video! Chat rooms! Laptops! iPods (Remember when Duke gave every freshman an iPod, right when these devices first appeared?)! Course management systems! iPads! The list goes on, and it will continue to go on, as a seemingly endless stream of new technologies arrives on the scene.

    There is no doubt that all this technology is fun, and interesting, and presents new and innovative ways to do our work of preparing young women and men to engage the world as educated and thoughtful adults. But there are those who would make a stronger statement, that these new technologies have changed, or will soon change, the very fundamentals of our profession. Even at a place as seemingly secure as Williams, with our 229 years of history and our enduring metaphor of a faculty member on one end of a log and a student on the other, I hear concerns about our coming obsolescence in the face of the computer and internet revolutions. Will Williams still matter? Will a liberal arts education, offered in the mountains of rural New England, be irrelevant to the Twittering students of the 21st century? Too slow, too stodgy, too boring? Do we at Williams, and all of us, at liberal arts colleges, need to become something completely different if we are to survive?

    My response to this question is unambiguous: notwithstanding the very real changes that technology has brought, the core fundamentals of education, both the education that we offer at Williams and education as a larger practice, remain intact. Moreover, we should fiercely resist the reflexive conclusion that because our students come to Williams with different modes of encountering and absorbing information (multitasking, multimedia, instant access, short attention spans) we must as a consequence become like them if we are to reach them and educate them. Rather, I believe our task to be the opposite one, namely to understand both the advantages and the deficits that this new world of continuous information flow leaves them with, and use the brief opportunity of their time in college to reinforce the capacity and disposition for slow, reflective, and difficult engagement with material. In fact, our students are, more than ever, hungry for just this sort of experience.

    Our current situation is hardly a novel one. The invention of the printing press itself might have been thought to presage the end of the university (“Why bring all these students to Oxford when we can just send them all the books by horseman? That would be much cheaper and more efficient, and they could study at their leisure at home, when most convenient.”), but no such thing occurred. Quite the contrary, of course. And there are more recent examples of the same sort, times where innovative uses of technology didn’t end education as we know it. Let me recall three from my own experience.

    When I was seven years old, my favorite Saturday morning activity was to get up early and watch reruns of Gilligan’s Island. But if I got up too early, in those days where there were only four channels, I had to put up with the tedium of both the Farm Report and Sunrise Semester, which ran at 6:00 a.m. on CBS. Sunrise Semester, which lasted from 1957 to 1982, was New York University’s first experiment in distance education. Real courses were offered, with NYU faculty broadcasting from a studio in New York. According to the NYU web site, the first course offered was “Comparative Literature 10: From Stendhal to Hemingway,” taught by Prof. Floyd Zulli. Students could receive college credit by paying $75: 700 applied, 177 completed the course, and 120,000 followed on television without signing up. (It is amusing to note that these numbers not all that different from those for “Machine Learning”, offered in the spring of 2012 by a Stanford professor under the auspices of Udacity.)

    What’s my point? Sunrise Semester was a great success. It ran for a quarter of a century, won an Emmy Award, and was viewed by literally millions of people. It began in 1957, at the dawn of the modern era in which television sets became ubiquitous in American homes. Certainly the pioneers of Sunrise Semester must have entertained the idea that with access to the best lectures for every student in every home, the days of the expensive residential college would soon be at an end. But why did this exciting new technology of television not have this effect? Because, of course, college education isn’t simply about the most efficient, or most engaging, means of transmitting information. It’s about the creation and nurturing of a community of students, in a particular kind of social and physical environment. It’s about learning things together, not just alone at home in your bathrobe and fuzzy slippers.

    But of course, we do sometimes learn things on our own. When I was a freshman in high school, I was bored in my regular mathematics class and my teacher gave me what was called a “programmed” book on probability and statistics. The book worked by having a series of fold-out pages. On each page was a discussion of a concept, and on the fold-out flap was a series of questions. The answers were on the back of the flap. You were meant to go on to the next page only once you’d answered the questions correctly. The whole thing was very engaging, and very effective. It was the only course in probability I ever took, and it served me well through a professional career in theoretical physics. Working in an entirely self-directed way, I really learned the material, and I loved it. In its rigor and its appeal, I have to imagine it was the equal of any self-paced text you can find for your iPad today.

    This was a very efficient and cost-effective way for me to learn probability, and I learned it. But the world of mathematics teaching was never taken over by programmed textbooks. We didn’t do away with all the math classes, hand the students these books, and send them to the library. These independent methods always had their place, of course, but why wasn’t education transformed when the day came that programmed texts could be produced cheaply? Because, I would maintain, it’s hard for even the best students to learn everything on their own, no matter how good the materials. It’s more fun, and far more effective, to learn as part of a community of students, supported by real human interactions.

    Finally, when I was in college, I taught calculus by mail for Duke University’s Talent Identification Program. The girl I taught (her name was Jane) was in the ninth grade and had no opportunity to learn calculus in her high school. I sent her the text and the problem sets. She sent the worked problems back to me every week, and I would correct them and return them to her. We handled exams the same way. Jane did marvelously, earning a 5 on the AP Calculus BC exam (and an 800 on the SAT math test, back when that meant something). A brilliant kid, for sure, and the TIP Math By Mail program was just what this student needed in a school that couldn’t meet her needs. And yet, once again, the existence of high quality distance education programs, more than a quarter of a century ago, didn’t keep schools and parents from thinking that the more expensive option of offering calculus in the classroom was actually the best way to enrich their math curricula. Today, almost all good American high schools offer calculus to their students.

    The point of these anecdotes is that, in very real ways, effective (and cost-effective) technologies to support distance education and self-paced learning have been with us for many decades. Printing, television and the postal service are remarkable tools. And, in fact, they have been used since their inventions to enhance and deepen education. What none of them has done is change the fundamental fact that at its heart education is a social activity that takes its highest form within a real community of students and faculty. Neither books nor video nor chat rooms have made colleges obsolete.

    Nonetheless, there are many who argue that our present moment is different from all those that have come before. Perhaps they are right. Perhaps the new tools of information technology provide something deeper than just new modes of content delivery. Perhaps our students are so accustomed to online interaction that the virtual communities erected within modern course management systems are all that they need to be connected to their fellow students. Perhaps their brains really are wired differently now.

    Perhaps. Personally, I doubt it. But even if some or all of this has happened, and we really have entered a Brave New World, I would argue that some simple, interrelated, principles remain:

    1. Education is still as much about human interaction as about delivering content. More so, in fact.
    2. New technology is expensive, especially technology on the cutting (or bleeding) edge. And immediate obsolescence is a perpetual danger.
    3. The introduction of technology does not generally drive down overall costs; in fact, it is likely to increase them.

    I’ll illustrate what I mean with an example from a program with which I have personal experience, Johns Hopkins University’s Advanced Academic Programs (AAP). These are professional master’s programs in a variety of fields, offered largely by part-time faculty who work in the relevant professions. About half the 2,000 students are in a variety of programs in biotechnology, many of them working in the biotech industry hub in Montgomery County along the I-270 corridor. Initially, AAP offered these courses, extensively and successfully, in a modern glass-and-steel facility in the heart of “DNA Alley.” One could not imagine a better location, and the programs, which were of high quality and high relevance to the target audience, flourished.

    And yet, by 2005 it became clear that there were good reasons to consider offering some of these courses, and perhaps entire degrees, on line. Certainly, the combined brand of Johns Hopkins and biotechnology was a powerful and global one, and there were excellent opportunities to offer these courses to students outside the Washington area. (Since the AAP programs generated net revenue, extending their reach was desirable.) But equally, the lives of the local students, battling Beltway traffic while balancing demanding jobs and family responsibilities, presented challenges that could be met by presenting the courses in an asynchronous and nonlocal format. That is, on line, using a course management system. (Even today, a significant majority of the online learners actually live within fifty miles of that glass-and-steel building, which turns out to be entirely typical.)

    In those days when much of this was new, AAP was concerned to demonstrate that the delivery of courses on line was the equivalent of what was referred to as “on site.” So they did a controlled study, in particular, with the offerings in bioinformatics. A faculty member who was to teach on line needed first to have taught the same course in the conventional classroom. The content of the online and on site courses was identical, and students were given the same final exams. Both student satisfaction and student learning outcomes were assessed and compared.

    The study showed that an online course is every bit as effective as an on-site course, even, perhaps, slightly more so. But this happy outcome is the direct result of a material investment in the online course that is no less than the investment in teaching on site. A core component of the online course, both educationally and in terms of student satisfaction, is the extensive interaction between the faculty member and the students, both in moderated chat rooms and on an individual basis. Sustaining such virtual conversations, at a high level, is enormously time consuming for the professor. As a consequence, the student-faculty ratio in the online course must be approximately the same as (or, if anything, lower than) in the conventional course. There are no simple economies of scale in teaching on line, if it is to be done well. By a similar token, the preparation time for faculty who teach on line is higher than for those who teach face to face; although the preparation is by far the heaviest the first time the course is taught, online courses must be updated as much as on site courses, and faculty turnover is just as common. It is no easier to “package” online courses so they may be taught by “anyone,” than it is for face-to-face education.

    Yet although there were no savings of money or faculty time in teaching on line, there were great benefits to AAP and its students in adopting this model. By 2010, half the enrollments in AAP’s biotechnology program were in online courses. The flexibility in time and place afforded by online delivery was of critical value, both to the local population of working adults, and to the developing global student body. (When the chat room is asynchronous, and you live in India, you don’t have to get up at 4:00 a.m. to be part of the discussion.)

    And although they were not inexpensive to deliver, the courses were educationally successful because, among other things, they made it a core priority to create and nurture a virtual community of deeply engaged students. The critical importance of this aspect of the online courses was clear in student satisfaction surveys and in interviews.

    So what are the lessons in all this for Williams and other liberal arts colleges? Our core educational mission, and our defining structure, is to educate students in a personal, intimate, collaborative environment. Young women and men come to us because they want to be in a place where they will know the faculty and their fellow students, and the community will know them. We know, and they know, that for the great majority of them, this is the sort of education that will prepare them best to be purposeful in their lives and effective in the world. I would argue that the great potential of the new technologies is not to upend these core values, but to allow us to fulfill our existing educational mission more effectively, especially by giving us new strategies to transcend our limitations of scale and location.

    In particular, distance education holds the promise, if thoughtfully deployed, of extending our curricula into areas not covered by our relatively small faculties. For example, in our new multi-polar world, there are many more languages that students will want to learn than we can possibly offer in our classrooms. French, Spanish, German, Russian, Japanese and Chinese simply aren’t enough, anymore. Internet-based instruction could be a vital solution to this challenge. In a similar vein, we can now use efficient, high quality, low cost video conferencing technologies to bring our students into truly meaningful, collaborative contact with students around the world, enhancing the traditional international dimensions of our curriculum, such as study abroad and courses on global topics, with substantive virtual experiences. (At Williams, we have a longstanding relationship with the American University of Cairo, by which we use videoconferencing to teach a course jointly each year. During the Arab Spring of 2011, this was powerful educational experience for our students, indeed!) New technologies will certainly bring us richer, multimedia teaching materials, even if they won’t be any cheaper than conventional textbooks. Some of those materials may even allow us to change, in exciting ways, the very modes in which we teach. And we may even find that computerized, programmed delivery of some standard, elementary topics (Calculus, perhaps?) becomes of such high quality that we can realize some modest efficiencies while maintaining our educational standards.

    Educational innovation comes in many forms, and not all of them rely primarily on integrated circuits. If you visit the Williams campus today, I will proudly show you our new library coming out the ground. This new facility will bring together our extensive book collections in the humanities and social sciences, our wonderful rare book library, and our Center for Media Initiatives, and surround them with dedicated spaces for interdisciplinary and group work. Our plan is to create a vibrant academic hub that will excite students’ passions, nurture their curiosity, and above all else, bring them together in space, not cyberspace. Academic work has changed over the years: it is more collaborative, more disciplinarily fluid, more eclectic in its sources and methods. A modern library is no longer simply a box for books and carrels; rather it is the crossroads of the campus both physically and metaphorically, where students and faculty meet, and where technologies both new and old come into conversation. Great academic architecture brings people into contact with each other, with tools, and with ideas. Information technology is surely a part of this story, and our new library will support it as never before, but it is only one strand among many in the tapestry we are weaving at Williams.

    This brings me back to the other question with which I began. Are students now so fundamentally different that they can only learn, or at least learn best, via the new modalities implied by our current technological revolutions? There’s no question that a teenager raised in a world of Wikipedia and ubiquitous multimedia stimulation doesn’t encounter the world, or have the same relationship to information, as did today’s college leaders when we were that age. But it simply does not follow that the only education that will seem relevant to these students, that they will be able to absorb and willing to embrace, is one that foregrounds the most disjunctive and hyperkinetic features of the modern world. Speeding along the Information Superhighway can be cognitively exhausting, and for many students it is ultimately unsatisfying. If their experiences before college have not encouraged them to slow down and think carefully about a coherent set of information, rather than surf a relentless wave of disconnected facts, then college is certainly the time to start. This deeper connection to ideas, I believe, is what they are really hungry for.

    Educational innovation, as I have said, need not necessarily involve semiconductors. At Williams, we offer some courses in a “tutorial” format, in which a faculty member sits with two students at a time, weekly, and facilitates a discussion in which the students alternately critique each other’s work and defend their own. Over the past decade, we chose to make significant investments to deepen this program, to make tutorials available to students early in their careers, to extend the model to the sciences, to provide this opportunity for any student who wants to have it. Tutorials are now offered in almost every department of the college, most students graduate from Williams having taken a tutorial, and many students take them often. The expansion of this program required a very significant investment by the college, an investment in people that seemed to some critics to represent a retrograde commitment to a pedagogy of the past, during an era of great technological advances. After all, tutorials represent a deeply personal, interactive, and challenging mode of learning, and with no particular emphasis on technological sophistication.

    Yet student satisfaction with the tutorial program is, by far, the highest of any part of our formal curriculum; and the faculty reports that student learning in tutorials is the deepest, as well. We consider this program one of our great successes, a hallmark of a Williams education, and an expression of our fundamental, and enduring, educational values.

    I have spent decades asking alumni, both old and young, about what mattered the most to them in their college educations. Fundamentally, the answer has never changed. Students and alumni point to a small number of critical relationships with individual faculty, faculty who taught them something deep and important, faculty who mentored them in theses and research, faculty who helped them personally, faculty who woke them from their slumbers. Upon reflection, alumni express little preference for those who were, in their time, the flashiest professors, or those who were the closest to the cutting edge. Bringing faculty and students together and giving them the space and time to interact is what we, at liberal arts colleges, do best. That is our core purpose. Technological evolution will make it possible to do this better, in all sorts of important ways. But if the printing press and television didn’t cause a revolution that brought us down, the iPad and the Internet won’t do that either.

    posted with permission of the author, Professor Adam Falk


  • In Defense of the Living, Breathing Professor

    As we move into the 21st century, there has been an increased demand for college educated workers and for a more informed and engaged citizenry. There is also an expectation that those educated workers and citizens have higher levels of learning and knowledge, as well as new and different skills and abilities. While our institution and others across the country face declining federal and state financial support for higher education, there have been increased calls for accountability from lawmakers, parents and students as these groups look for evidence that an education matters. 

    If all we wanted of our students at the end of a college education was a collection of facts, we could give them access to wikipedia and call it quits. However, learning is something that goes beyond the retention of information, its important for students to be able to apply that knowledge and to demonstrate the application of that knowledge. 

    In a recent opinion piece in the Wall Street Journal, the president of Williams College had this to say about the value of education and I wanted to share his comments with the UW Madison teaching community. Professor Falk has kindly allowed me to republish his remarks below, and I have added emphasis to several important points that he raises. 

    Published in the Wall Street Journal (Aug. 28 online and Aug. 29 print edition)

    ‘Crowd-sourcing’ the grading of an essay online is no substitute for thoughtful evaluation by a trained educator.

    As classes resume on our nation’s campuses, amid anxiety about high tuition, student debt and other concerns, it’s worth examining what we value in college education. The question warrants consideration, in particular, following a recent recommendation by distinguished economists, appointed by the National Academy of Sciences, proposing to define the “output” of higher education as a combination of credit hours awarded and degrees earned.

    That reduces the work of colleges to counting how many students they push through the system—a bit like defining a movie studio’s output as the number of feet of raw footage shot, with no consideration of whether the resulting movies are any good.

    Most of us in higher education take the long view about the value of what we do. Sure, students graduate with plenty of facts in their heads. But the transmission of information is merely the starting point, a critical tool through which we engage the higher faculties of the mind.

    What really matters is the set of deeper abilities—to write effectively, argue persuasively, solve problems creatively, adapt and learn independently—that students develop while in college and use for the rest of their lives.

    At Williams College, where I work, we’ve analyzed which educational inputs best predict progress in these deeper aspects of student learning. The answer is unambiguous: By far, the factor that correlates most highly with gains in these skills is the amount of personal contact a student has with professors. Not virtual contact, but interaction with real, live human beings, whether in the classroom, or in faculty offices, or in the dining halls. Nothing else—not the details of the curriculum, not the choice of major, not the student’s GPA—predicts self-reported gains in these critical capacities nearly as well as how much time a student spent with professors.

    What follows from this finding is obvious, but apparently in need of saying these days: What we do is expensive—and worth it—because these rich, human interactions can’t be replaced by any magical application of technology.

    Technology has and will continue to improve how we teach. But what it cannot do is remove human beings from the equation. Coursera, one of the new purveyors of massive, open online courses, proposes to crowd-source the grading of essays, as if averaging letter grades assigned by five random peers were the educational equivalent of a highly trained professor providing thoughtful evaluation and detailed response. To pretend that this is so is to deny the most significant purposes of education, and to forfeit its true value.

    It follows that standard economic models that define productivity as total output per unit of labor are ill-suited to analyzing the activities of colleges. Understandable concerns about the cost of college have spawned a spate of high-stakes considerations of productivity; in Texas, “low-productivity” programs at flagship universities have been targeted for elimination. Yet the only way to achieve higher productivity, as the National Academy would define it, is to reduce each student’s time with the faculty. We know that while such approaches may allow us to deliver some facts to some students more efficiently in the short run, the approaches will undermine the fundamental purpose of education in the long run.

    Equally misguided is the common practice of judging a school’s success by measuring the net worth of its alumni. Is a Williams graduate who is teaching elementary school less successful, less influential, less transformed than she would be if she had become a banker? There’s no reason to think so, and anyone assessing colleges or setting public policy on that assumption is being mischievous.

    Education is not a commodity. It’s a social process, and its value, including its economic value, both to the graduate and to society is unquestionable. It is equally true that this value cannot be reduced to a single number, however much the measurers of productivity—or those who rank us in magazines—may wish it otherwise.

    Professor Falk is president of and professor of physics at Williams College.

    Copyright 2012 Dow Jones & Company, Inc. All Rights Reserved


  • Trying to Find a Measure for How Well Colleges Do

    A recent article in the New York Times regarding the increased pressure on colleges and universities to demonstrate their effectiveness. The interest in accountability asks us as educators to think about what the goals are of a college education.  I think that many of us would say that college is more than just the accumulation of knowledge, but how do we measure achievement of those goals?

    How well does a college teach, and what do its students learn? Rankings based on the credentials of entering freshmen are not hard to find, but how can students, parents and policy makers assess how well a college builds on that foundation? What information exists has often been hidden from public view. But that may be changing.

    In the wake of the No Child Left Behind federal education law, students in elementary, middle and high schools take standardized tests whose results are made public, inviting anyone to assess, however imperfectly, a school’s performance. There is no comparable trove of public data for judging and comparing colleges.

    Pieces of such a system may be taking shape, however, with several kinds of national assessments — including, most controversially, standardized tests — gaining traction in recent years. More than 1,000 colleges may be using at least one of them.

    “There’s a real shift in attitudes under way,” said David C. Paris, executive director of the New Leadership Alliance for Student Learning and Accountability, a coalition of higher education groups. “We used to hear a lot more of, ‘The value of college can’t be measured,’ and now we hear more of, ‘Let’s talk about how we can measure.’ ”

    In January, the New Leadership Alliance released guidelines calling on colleges to systematically “gather evidence of student learning” — though not explicitly advocating standardized tests — and release the results. The report was endorsed by several major organizations of colleges and universities.

    Advocates say the point is not to measure how each college’s students perform after four years, which depends heavily on the caliber of students it enrolls in the first place, but to see how much they improve along the way. The concern is less about measuring knowledge of chemistry or literature than about harder to define skills like critical thinking and problem-solving.

    That vision still faces major obstacles. Colleges that use standardized tests vary widely in what they test, how and when. And many of them that use those tests or national surveys keep the results to themselves.

    “I’d love for all the data to be public,” said Jennifer Carney, director of program evaluation at the Jack Kent Cooke Foundation, which conducts education research. But, she added, that would inevitably lead to some colleges manipulating the figures in pursuit of a higher standing, just as some have done with existing ranking systems.

    In the best-known college rankings, by U.S. News & World Report, up to 40 percent of a college’s score is based on its reputation among educators and its selectivity in admitting students. Other factors include several indirect indicators of what happens in classrooms, like student retention, graduation rates and class sizes, but no direct measures of learning.

    Critics of standardized tests say they are too narrow and simplistic.

    “I’m not sure any standardized test can effectively measure what students gain in problem-solving, or the ability to work collaboratively,” said Alice P. Gast, president of Lehigh University.

    In 2008, the Consortium on Financing Higher Education, a group of some of the nation’s most prestigious colleges and universities — including all of the Ivy League — issued alengthy manifesto saying that what its students learn becomes evident over decades and warning against a “focus on what is easily measured.”

    Many of those same colleges participate in the National Survey of Student Engagement, which measures many factors that educators say are good, though indirect, indicators of learning, like how many hours students spend studying and how much they interact with professors. The survey, which began with a handful of colleges in 2000, had more than 700 participate last year. Each college can see its own results and those of a group of comparable colleges, but the scores are not made public.

    The view from state-supported colleges has been shaped in part by pressure from policy makers to show what taxpayer dollars are accomplishing, through standardized tests. Texas, a leader in the movement, has required its state colleges to administer tests since 2004, and it makes the outcomes public.

    Testing advocates have also gained ammunition from books calling into question the quality of American colleges, notably “Academically Adrift,” by Richard Arum and Josipa Roksa, published last year. It was based on a study showing that more than one-third of students show no significant gain in critical thinking skills after four years of college.

     But the concept of universal assessment got its biggest boost in 2006, from the findings of a commission appointed by Margaret Spellings, then the education secretary. The report said that learning “must be measured by institutions on a ‘value added’ basis that takes into account students’ academic baseline,” and that the results must be made available to everyone “to measure the relative effectiveness of different colleges and universities.”

    That prompted talk that the federal government might mandate standardized testing, as it did for public schools with No Child Left Behind in 2001.

    “That’s what gave this issue urgency,” said Christine M. Keller, executive director of theVoluntary System of Accountability, an alliance of more than 300 state colleges that was created in response to the Spellings Commission. “No one wanted the government imposing a standard.”

    Her group has approved three competing tests and asks public colleges to use them and post scores on the group’s Web site.

    They are the ETS Proficiency Profile, from Educational Testing Service; the Collegiate Assessment of Academic Proficiency, produced by ACT; and the Collegiate Learning Assessment (used in the “Academically Adrift” study), from the Council for Aid to Education, a research group.

    “These instruments are not without controversy,” Ms. Keller said. “It’s still very much a work in progress.”

    While more blanks are filled in each year, the “college portraits” remain spotty. Some member colleges have not posted test scores so far, while others first posted scores from two or three years ago and have not updated them. Typically, only a small sample of students takes a test, and the ways the samples are chosen can vary, making comparison harder.

    Some major systems, like the University of California, do not participate in the organization, while others, like the State University of New York, are represented by only a few campuses.

    Selective colleges, public and private, complain of a “ceiling effect” in standardized testing, making it especially hard for their students to show improvement.

    “It does not, in my opinion, measure value added very well for our kind of institution,” said Neal E. Armstrong, a vice provost of the University of Texas at Austin. “Our freshmen come in with very high aptitude and critical thinking skills.”

    Even so, the use of the three tests in the voluntary system has boomed, with each company claiming several hundred client colleges, most of which do not make their scores public. Test authors acknowledge that colleges often use them in less than optimal ways, like skipping some sections, testing only once every few years (often to satisfy an accreditation agency) and not measuring student growth over time.

    If officials and experts in higher education can reach a broad consensus about how to measure learning, it will become routine, with results being made public “only if consumers demand it,” said Ms. Carney, of the Cooke Foundation.