The key may be a growing movement for peer review of Web sites and online teaching materials
By JEFFREY R. YOUNG
Don't waste time teaching online or laboring over electronic course enhancements unless you've already climbed to the top of the tenure-and-promotion ladder. Review committees may not take technology work seriously, so stick to traditional academic activities, like publishing journal articles.
That's the warning Gary Bradshaw heard from colleagues when he told them about his work on ePsych, an online resource to help introductory psychology students. He already had tenure when he started the online project, but he found out the hard way that the advice was sound. He was twice turned down for a promotion to full professor at Mississippi State University, where he is an associate professor of psychology. (The chairman of the psychology department, Stephen B. Klein, said he could not discuss personnel issues.)
Mr. Bradshaw feels that work creating online teaching materials -- such as sophisticated Web sites or multimedia tools designed to help students -- slips through the cracks of the three traditional categories used in promotion: teaching, research, and service.
"They don't count this work," he says. "In so many words, they told me, That's not a research project -- it counts sort of like service, but it's sort of a hobby."
"And that is crazy," adds Mr. Bradshaw, whose Web project employs a team of eight students and staff members and has won $400,000 in grants from the National Science Foundation (http://epsych.msstate.edu).
Many faculty members across the country say the tenure-and-promotion system fails to recognize teaching with technology -- even though more and more colleges seem eager for professors to use technology in the classroom or to develop online courses. But that might be changing, as a growing number of institutions are working to include digital creations in the tenure folders that form the core of a candidate's professional portfolio.
At issue are the time-intensive creations undertaken by many professors, rather than the basic use of off-the-shelf products that many colleges have purchased. And arguments about whether technology in teaching counts seem to be more pronounced at research universities than at institutions that are more teaching-oriented.
"This is a major struggle now in higher education," says R. Eugene Rice, director of the Forum on Faculty Roles & Rewards for the American Association for Higher Education. "Institutions across the country are thinking about building this into the tenure and promotion guidelines."
But including online materials in the academic-promotion process raises tough questions. How should a review committee judge the merit of multimedia teaching tools or online courses? Should scholars really be spending their valuable time dabbling in multimedia, or should they hand off the bulk of that work to technical experts provided by colleges? And should institutions -- especially research universities -- work harder to consider teaching activities of all kinds, not just digital ones?
Meanwhile, leaders of a nationwide project called Merlot say they have a solution: Apply the same kind of peer-review system used for journal articles to the evaluation of electronic teaching materials. The project, which began three years ago, is supported by 22 college consortia and systems (http://www.merlot.org).
Merlot seems to be gaining momentum, thanks to an aggressive recruiting and marketing strategy. But some education-technology experts say that online teaching tools should be judged by whether students learn from them, rather than by whether other professors like them. Meanwhile, the Merlot's future remains uncertain, and some participants worry that the group won't be able to continue operating without resorting to selling services or forging corporate partnerships that could threaten its independence.
Hard to Measure
Marcy Bauman says she discovered the difficulties of getting credit for online teaching when she came up for tenure three years ago.
Ms. Bauman, who was a junior faculty member in composition and rhetoric at the University of Michigan at Dearborn, created one of the first online courses at the university, and she did most of her teaching in electronic classrooms.
But Ms. Bauman was denied tenure, and she believes that her online teaching worked against her. "Everybody recognized that my expertise with technology was valuable," she says. "They just weren't going to call it academic. That's a real sort of schizophrenia in the profession as a whole."
What's worse, she says, some members of the committee used the detailed information on her course Web sites to discredit her. "What I had done, functionally, was I had allowed everyone to come to my class for the entire semester," she says. That means reviewers had access to a note she posted one day saying she was canceling a session to stay home with a sick child, among other details.
"It was an incredibly frustrating experience," she says. "It was really angering."
J. Randal Woodland, an associate professor of composition and rhetoric at the university, was on the committee that reviewed Ms. Bauman's case. Although he refuses to discuss specifics of her situation, he says that colleges are struggling with cases like hers.
Institutions are "eager" to encourage professors to use technology in instruction and even giving professional credit for it, he says. "But sometimes," he adds, "they have difficulty doing it when they see it."
Wary of such scenarios, many professors simply avoid virtual teaching or course-tool creation until they have secured tenure.
Eighty to 90 percent of the professors who have signed up to teach for the University of Texas System's TeleCampus already have tenure, for instance, says Darcy W. Hardy, director of the TeleCampus and assistant vice chancellor for academic affairs for the University of Texas System. "We've had assistant professors who have been approached to teach online and have balked because they need the time to work on their research for tenure," she says.
Sarah Horton, an instructional-technology specialist at Dartmouth College, says she has noticed a similar wariness among junior faculty members there.
"Some professors won't put the time in with the technology endeavors because they don't think it will have any influence," says Ms. Horton. "And sometimes there's concern that they will have a negative impact." But Ms. Horton thinks that the professors' fears are unfounded, because Dartmouth does recognize technology and teaching, she says.
In fact, the college recently revised an annual review form for faculty members so that it invites them to list technical innovations in teaching and research.
Many observers say that the climate varies greatly among institutions. "It depends on the kind of institution you're in," says A. Frank Mayadas, director of the Alfred P. Sloan Foundation's Asynchronous Learning Networks, noting that a professor at a community college would probably get more credit for teaching with technology than a professor at a research institution.
Cynthia L. Selfe, a professor of humanities at Michigan Technological University, says that the climate can be different even at similar types of institutions, however. "Know the tenure-and-promotion criteria at the institution that you're working at," she says. "Have regular talks with your chair, and keep your chair informed about what you're doing and why."
To help raise awareness of the issue, Ms. Selfe recently led an online discussion through the National Council of Teachers of English's Committee on Computers in Composition. Using a Web site, the group distributed five fictional tenure-and-promotion cases in which composition professors made heavy use of technology in their teaching and research. Then, real deans and faculty members described how they would handle the cases (http://www.hu.mtu.edu/~cyselfe/P&TStuff/ P&TWeb/Introduction.htm).
One of the fictional cases was set at a major university, where a junior professor named Sherry Richer helped graduate students learn how to teach online, moderated an e-mail list for her discipline, and developed bulletin-board software for use in online classes. But she didn't publish as much as she had hoped -- only an online article and a book chapter.
Most of the deans and faculty members who responded said that the fictional Ms. Richer would not be given tenure because she had not done much research. But some respondents stressed that the department chair should have done more to advise the professor of what the department expected from her to win tenure.
"In most of those cases the problem is with communication," says Ms. Selfe.
Adopting Guidelines: Some colleges, scholarly societies, and individual departments have devised their own ways of making sure that online teaching activities are part of the tenure equation.
Last year, the Association for History and Computing adopted "suggested guidelines for evaluating digital media activities in tenure, review, and promotion" that were endorsed last month by the American Historical Association. The Modern Language Association has also adopted similar guidelines in the past two years.
The guidelines assert that technology work by professors in teaching and research should be recognized, and that institutions should look for ways to "engage qualified reviewers" to judge the quality of technology projects. They also call on institutions and even individual departments to adopt guidelines.
At Indiana University-Purdue University at Indianapolis, the tenure policy has been changed over the past few years to add consideration of multimedia teaching materials, says William M. Plater, executive vice chancellor and dean of the faculties at the university.
The Pennsylvania State University system is among the institutions that have introduced student evaluations for online courses. At many other colleges that routinely conduct student evaluations of traditional courses, no student evaluation forms are distributed in online courses. "At Penn State University, teaching online is teaching, and it certainly does get counted," says Gary E. Miller, associate vice president of distance education and executive director of its World Campus.
At some institutions, individual departments have changed guidelines or issued statements that multimedia work counts toward tenure. It is difficult to determine how many departments have done so, however.
But not everyone thinks that colleges should go out of their way to consider online materials. Some administrators say colleges should set up technical-support systems that ensure professors don't shift their time from research to Web programming.
"There should be a minimal amount of effort from the professor," says Kyung I. Han, a consultant and co-project manager of the OpenCourseWare project at the Massachusetts Institute of Technology. The project is working to create detailed Web pages for nearly all of MIT's courses. "We don't want professors to put an inordinate amount of time to do this because their main focus at the university environment is teaching and research."
Some professors do get seduced by the technology and spend too much time creating online-course materials, says Barbara E. Walvoord, an English professor at the University of Notre Dame who is also director of its Kaneb Center for Teaching and Learning.
"Some faculty members get really hooked on the computer, and they love to spend hours and hours of their time producing these elaborate Web sites," says Ms. Walvoord. As a result, she adds, "sometimes the faculty members spend a lot of time on something that students don't use and don't find particularly useful."
A Taste of Merlot
Merlot is perhaps the most ambitious attempt to create a standard format for reviewing online-course components. The group, whose full name is the Multimedia Educational Resource for Learning and Online Teaching, has assembled a directory that lists more than 6,000 online-teaching materials, and it has developed a system to start evaluating those items. Only about 370 of the materials have been reviewed so far, however.
To build awareness of the project and to recruit more volunteer reviewers, Merlot's leaders have coached its members to pitch the project aggressively at disciplinary conferences and on their campuses. Participants who attend national meetings are asked to wear stickers that read "Ask Me About Merlot."
"I joke that it's like Amway," said Gerard L. Hanley, director of the Merlot project, during the group's meeting last month in San Diego. "You get one person who then goes and sells to five other people, who each go and sell to five other people. As the math guys know, that's exponential -- that's good."
The project has set up teams of reviewers in 13 disciplines. The teams focus their attention on pieces of course Web sites, rather than on sites as a whole, on the theory that professors might want to incorporate those pieces into their own courses. Although anyone can nominate course materials for inclusion in Merlot's directory of course materials, the reviewers use a process of "triage" to cut out inaccurate materials and select which items will get a full peer review.
The peer-review process is modeled after that of peer review for academic journals. Generally, two or three reviewers participate in a Merlot review, and they consider three major factors: quality of the content, potential effectiveness as a teaching-and-learning tool, and ease of use.
The reviewers then write comments about the site, including suggestions for how it could be improved, and give the site a rating of one to five stars for each of the three areas.
Most of the reviews are favorable -- few of the sites get less than four stars. Merlot editors say that is appropriate, however, because they select only the most promising sites for a full review, just as academic journals only review the best articles that are submitted.
After an item is reviewed, Merlot notifies the professor by e-mail. The group also offers to send a letter to the professor's dean or department head certifying that the item was reviewed and including the results. The letter is intended to be included in tenure folders.
"We put the reviews in a form that tenure-and-promotion committees are familiar with," says Mr. Hanley.
If Merlot succeeds, professors may someday routinely review course Web sites as part of their professional duties, just as they now judge and edit journal articles. And coding a widely used multimedia tool might become as important to winning professional advancement as writing an influential article.
Seeking Endorsements
So far, however, most professors have probably never heard of Merlot, and its leaders say that they will need more than word-of-mouth promotion to win widespread acceptance. So the group is also working to get major disciplinary organizations to endorse the project.
Merlot is also focusing on securing its financial future -- which some of the group's leaders are concerned about in a time of shrinking education budgets. So Merlot is looking for new ways to bring in money, such as a proposal to develop a commercial service that would allow publishers and companies to advertise their e-books or online products. But its leaders are split on how much corporate involvement the project should have. This month the project's chief executive officer, Ed Cooper, resigned to go back to his former job as a business professor at Regis University.
And some education-technology experts find Merlot's peer-review concept problematic. "I would rather see an evaluation of how the learning objects improve student learning," says Carol A. Twigg, executive director of the Center for Academic Transformation at Rensselaer Polytechnic Institute. She says that a better way to judge a course Web site would be to do research on how students perform when the site is used.
"Most faculty members are not experts in learning," she says. "They're experts in content."
It's the Teaching, Stupid
Many professors say the lack of consideration given to teachingwith technology at some institutions is simply the latest example of alongstanding neglect of teaching in professional evaluation. Youdon't hear "Teach well or perish" very often.
"Teaching doesn't get the respect it deserves when it comes timefor tenure and promotion," says Rachel Hendrickson, coordinatorof higher education for the National Education Association. "Withdistance education, it just takes that much more time [to developand teach], and I don't think that's recognized either."
Some professors and administrators hope that technology mightactually help win more recognition of the work that goes intoteaching, even in the traditional classroom.
"Two decades ago, teaching was often asserted but rarely proved,"says Mr. Plater, dean of the faculties at Indiana University-PurdueUniversity at Indianapolis. "Technology, I think, is enabling manyfaculty to show their work [in the classroom] to people around thecountry and around the world."
And some experts say they already see a shift in attitude among younger professors toward online teaching. "They're starting to feel like it's in their benefit to have some experience in using technology in teaching," says Ms. Hardy, of the University of Texas TeleCampus.
When it comes to hiring new faculty members, many departments are looking for candidates who have experience teaching with technology, says David G. Brown, vice president and dean of the International Center for Computer Enhanced Learning at Wake Forest University. Soon, he predicts, those skills will "become part of the essence of what it means to be a professional in the field."
Gordon E. McCray, an associate professor of information systems at Wake Forest, is an example of a professor who won tenure in part based on his work with online courses. But even Mr. McCray says he would tell colleagues to be cautious if they want to to follow his example.
"What about someone who is one publication short because of a lot of technology work?" he says. "Then, I don't know."
"I would advise any of my colleagues who are tenure-track to be careful," he adds. "The one thing that's quantifiable in our game is research."
HOW WORK WITH TECHNOLOGY HELPED 5 PROFESSORS' CAREERS (OR DIDN'T)
The following are five professors who recently came up for tenure, and how they think their work with technology affected their bids. Officials at the institutions declined to comment on the cases.
Professor: Marcy Bauman, full-time media consultant at the University of Michigan at Ann Arbor
What happened: Was denied tenure when she was an assistant professor of composition and rhetoric at the University of Michigan at Dearborn. She believes that her online teaching worked against her.
Comment: "Everybody recognized that my expertise with technology was valuable. They just weren't going to call it academic. That's a real sort of schizophrenia in the profession as a whole."
***
Professor: Gary Bradshaw, associate professor of psychology at Mississippi State University
What happened: Has twice been turned down for promotion to a full professor. He feels that his extensive work on ePsych, an online resource to help introductory psychology students, was not considered in his promotion review.
Comment: Colleagues told him, "That's not a research project -- it counts sort of like service, but it's sort of a hobby."
***
Professor: Kathryn Green, formerly an assistant professor of history at California State University at San Bernardino
What happened: Was denied tenure in August. She says she spent much of her time working on a Web site to track the troubles of research archives in Africa. She also worked to incorporate technology into her teaching and was active in an online discussion list in her discipline. She says the work with technology was not counted in her tenure review.
Comment: "There's a lot of dinosaur thinking in history, and a lot of older professors who can barely turn their computers on. ... It's a generational issue -- most full professors in any field have come into the computer world late, so they don't understand it."
***
Professor: Gordon E. McCray, associate professor of information systems at Wake Forest University
What happened: Experimented extensively with creating multimedia versions of his lectures, which he calls "cybershows." He won tenure, but he says that young professors should be cautious about spending too much time on technology and not enough on research.
Comment: "If I had done nothing with technology, I think I would have gotten tenure here [because of strong research]. I would advise any of my colleagues who are tenure-track to be careful."
***
Professor: Margaret A. Syverson, associate professor of rhetoric and communication and director of the Computer Writing and Research Lab at the University of Texas at Austin
What happened: Won tenure after doing most of her teaching online. Feels that her department values her technology skills.
Comment: "I was not nervous at all because the kinds of things I know how to do are in demand."
Section: Information Technology
Copyright © 2002 by The Chronicle of Higher Education