Viewpoints Viewpoint: Time For Computer Science to Grow Up

Credit: Flickr

Unlike every other academic field, computer science uses conferences rather than journals as the main publication venue. While this made sense for a young discipline, our field has matured and the conference model has fractured the discipline and skewered it toward short-term, deadline-driven research. Computer science should refocus the conference system on its primary purpose of bringing researchers together. We should use archive sites as the main method of quick paper dissemination and the journal system as the vehicle for advancing researchers' reputations.

In his May 2009 Communications Editor's Letter,2 Moshe Vardi challenged the computer science community to rethink the major publication role conferences play in computer science. Here, I continue that discussion and strongly argue that the computer science field, now more than a half-century old, needs to adapt to a conference and journal model that has worked well for all other academic fields.

Why do we hold conferences?

To rate publications and researchers.

To disseminate new research results and ideas.

To network, gossip, and recruit.

To discuss controversial issues in the community.

The de facto main role of computer science conferences is the first item: rating papers and people. When we judge candidates for an academic position, we first check the quality and quantity of the conferences where their work has appeared. The current climate of conferences and program committees often leads to rather arbitrary decisions even though these choices can have a great impact particularly on researchers early in their academic careers.

But even worse, the focus on using conferences to rate papers has led to a great growth in the number of meetings. Most researchers don't have the time and/or money to travel to conferences where they do not have a paper. This greatly affects the other roles, as conferences no longer bring the community together and thus we are only disseminating, networking, and discussing with a tiny subset of the community. Other academic fields leave rating papers and researchers to academic journals, where one can have a more lengthy and detailed reviews of submissions. This leaves conferences to act as a broad forum and bring their communities together.

Back to Top

A Short History of CS Conferences

The growth of computers in the 1950s led nearly every major university to develop a strong computer science discipline over the next few decades. As a new field, computer science was free to experiment with novel approaches to publication not hampered by long traditions in more established scientific and engineering communities. Computer science came of age in the jet age where the time spent traveling to a conference no longer dominated the time spent at the conference itself. The quick development of this new field required rapid review and distribution of results. So the conference system quickly developed, serving the multiple purposes of the distribution of papers through proceedings, presentations, a stamp of approval, and bringing the community together.

With the possible exception of Journal of the ACM, journals in computer science have not received the prestige levels that conferences do. Only a fraction of conference papers eventually get published in polished and extended form in a journal. Some universities insist on journal papers for promotion and tenure but for the most part researchers feel they have little incentive for the effort of a journal submission.

As the field went through dramatic growth in the 1980s we started to see a shift in conferences. The major CS conferences could no longer accept most qualified research papers. Not only did these conferences raise the bar on acceptance but for the papers on the margin a preference for certain subareas emerged. Researchers from the top CS departments dominated the program committees and, not necessarily consciously, helped set the agenda with areas that helped their faculty, students, and graduates. Over the years these biases became part of the system and unofficially accepted behavior in the community.

As CS grew the major conferences became even more selective and could not accept all the quality papers in any specialized area. Many new specialized conferences and workshops arose and grew to capture these papers. We currently have approximately a dozen U.S.-based conferences in theoretical computer science alone. The large number of conferences has splintered our communities. Because of limitations of money and time, very few conferences draw many attendees beyond the authors of accepted papers. Conferences now serve the journal role of other fields, leaving nothing to serve the proper role of conferences.

Other disciplines have started to recognize the basic importance of computation and we have seen strong connections between CS and physics, biology, economics, mathematics, education, medicine and many other fields. Having different publication procedures discourages proper collaboration between researchers in CS and other fields.

Back to Top

The Current Situation

Most CS researchers would balk at paying significant page charges for a journal but think nothing of committing well over $1,000 for travel and registration fees for a conference if their paper were accepted (not to mention the time to attend the conference). What does that monetary commitment buy the author? A not particularly fair review process.

With the tremendous almost continual growth in computer science over the past half-century combined with the desire of each conference to remain small and "competitive," even with the increase in the number of conferences we simply have too many papers chasing too few conference slots. Each conference has a program committee that examines submissions and makes decisions on which papers will appear at a conference and which will not. The great papers almost always are accepted and the worst papers mostly get rejected. The problem occurs for the vast majority of solid papers landing in the middle. Conferences cannot accept all of these papers and still maintain their high-quality reputations.

Even if the best decisions are made, several good papers will not make the cut. A variety of factors make the process imperfect at best:

Being on a program committee (PC) requires a large time commitment because of the number of papers involved. With the increase in conferences, many researchers, particularly senior scientists, cannot serve on many of these committees, leaving these important decisions mostly to those with less experience.

As our research areas continue to become more specialized a few to none of the PC members can properly judge the importance of most results.

These specialized areas have a small number of researchers, meaning the appropriate PC members know the authors involved and personal feelings can influence decision making.

PC members tend to favor papers in their own areas.

The most difficult decisions are made by consensus. This leads to an emphasis on safe papers (incremental and technical) versus those that explore new models and research directions outside the established core areas of the conference.

No or limited discussions between authors and the PC means papers often get rejected for simple misunderstandings.

Various conferences have implemented a number of innovative and sometimes controversial ideas to try to make the process more fair (author information removed from papers, author responses to initial reviews, multilevel program committees, separate tracks for areas and quality, higher/lower acceptance ratios) but none can truly avoid most of the problems I've outlined here.

In the extreme many of the best scientific papers slip through the cracks. For example, nearly half of the Gödel Prize winners (given to the best CS theory papers after they've appeared in journals) were initially rejected or didn't appear at all in the top theoretical computer science conferences.

We end up living in a deadline-driven world, submitting a paper when we reach an appropriate conference deadline instead of when the research has been properly fleshed out. Many also just publish "least-publishable units," doing just enough to get accepted into a conference.

Back to Top

The Road Ahead

How do we move a field mired in a long tradition of conference publications to a more journal-based system? Computer science lacks a single strong central organization that can by itself break the inertia in our system.

The Computing Research Association, in its 1999 tenure policy memo,1 specifically puts conference publications above journals: "The reason conference publication is preferred to journal publication, at least for experimentalists, is the shorter time to print (7 months vs. 12 years), the opportunity to describe the work before one's peers at a public presentation, and the more complete level of review (4-5 evaluations per paper compared to 2-3 for an archival journal). Publication in the prestige conferences is inferior to the prestige journals only in having significant page limitations and little time to polish the paper. In those dimensions that count most, conferences are superior."

A decade later the CRA should acknowledge that the growth in computer science and advances in technology changes the calculus of this argument. Quick dissemination via the Web makes time to print less relevant and two or three careful journal referee reports give a much more detailed level of review than four or five rushed evaluations of conference reviewers. The CRA needs to make a new statement that the current conference system no longer fully meets the needs of the computer science community and support the growth of a strong journal publication system. This will also encourage chairs and deans to base hiring and promotion more on journal publications as it should be.

Many of the strongest computer science conferences in the U.S. are sponsored by ACM and the IEEE Computer Society. These organizations need to allow special interest groups and technical committees to restructure or perhaps eliminate some conferences even if it hurts their publication portfolio and finances in the short term.

How do we move a field mired in a long tradition of conference publications to a more journal-based system?

But most importantly, leaders of major conferences must make the first move, holding their conferences less frequently and accepting every reasonable paper for presentation without proceedings. By de-emphasizing their publication role, conferences can once again play their most important role: Bringing the community together.

Back to Top

Conclusion

Our conference system forces researchers to focus too heavily on quick, technical, and safe papers instead of considering broader and newer ideas. Meanwhile, we have devoted much of our time and money to conferences where we can present our research that we can rarely attend conferences and workshops to work and socialize with our colleagues.

Computer science has grown to become a mature field where no major university can survive without a strong CS department. It is time for computer science to grow up and publish in a way that represents the major discipline it has become.

Back to Top

References 1. Evaluating computer scientists and engineers for promotion and tenure; http://www.cra.org/reports/tenure_review.html. 2. Vardi, M. Conferences vs. journals in computing research. Commun. ACM 52, 5 (May 2009), 5.

Back to Top

Author Lance Fortnow (fortnow@eecs.northwestern.edu) is a professor of electrical engineering and computer science at Northwestern University, Evanston, IL.

Back to Top

Copyright held by author.

The Digital Library is published by the Association for Computing Machinery. Copyright © 2009 ACM, Inc.

Comments

Filippo Menczer

I could not agree more, as I already mentioned in response to Moshe Vardi's editor's letter.

As program chair of an ACM conference I can attest to the many flaws

of the conference review process, in spite of everyone's best efforts.

Time constraints, concentrated load, difficulty of holding a relaxed

rebuttal phase, competition among authors and referees, pressure to go

with average ratings, financial concerns (attendance), and other

factors --- all bias and weaken the process in important ways.

And as an interdisciplinary researcher, I experience first-hand how

our conference-driven publication practices hurt us in terms of

impact, reach, and visibility. Computing journals have very low impact

factors and very long dissemination times, compared to other science

disciplines. Conferences hurt the viability of journals because top

researchers are busy refereeing conference papers. The time

constraints of conferences make us submit papers in a rush before they

are ready, or impose long delays. And competition for the top

conferences, as has been noted by others, means that many good papers

do not get the attention they deserve.

I propose a simple solution: abolish conference proceedings. Then

papers will be submitted to journals instead. Journals will receive

more and better papers. Refereeing resources will shift naturally from

conferences to journals. As a result, journals will gain impact,

improve quality, and speed up their processes. With our full

attention, they will become viable once again and the review process

will be more rigorous, effective and timely. Deadlines will no longer

be concentrated and we can submit better work, revise it until it is

ready, and profit immediately and directly from reviewer's feedback --

the same referee can judge improvements to a paper. I could go on but

it seems the many advantages are obvious.

We would still hold conference, of course. In many cases where

conferences and journals are nicely aligned, presentations can be

selected and invited among the best papers published in the previous

year. For newer areas and groundbreaking work, a conference or

workshop can still accept submissions --- but it should not publish

proceedings; publication is the job of journals.

ACM should take the lead in such a transition because it publishes the

proceedings of most top computing conferences, as well as many of the

top computing journals. Therefore ACM has everything to gain from

leading the way. ACM is also the only body that could successfully

shepherd such an undertaking.

The switch would not be easy, but with careful planning we could

manage a phased transition over a few years and catch up to the rest

of the scientific community.

Filippo Menczer

Program Chair, Hypertext 2009

Barry Jay

I also agree. Recently a journal editor asked me to expand an otherwise acceptable paper since "it could have been published in a conference". As a first step, it would be good if journals were to encourage original submissions, as well as those that have survived conferences.

My own, light-hearted, take on the difficulties of reviewing to a deadline are in "How to reject a paper you don't like, but don't know why" available at www-staff.it.uts.edu.au/~cbj/Publications/paper_rejection.pdf.

Lance Fortnow

Thanks for the support. The discussion continues on my weblog, at http://blog.computationalcomplexity.org/

Daniel Tunkelang

It's nice to see this piece joining a growing chorus questioning the way we conflate the distinct concerns of disseminating knowledge, establishing professional reputation, and building community. This problem is not unique to computer science, but we are certainly in a position to lead by example in addressing it.

In an age where distribution is nearly free, I agree that we should move the filtering role from content publishers to content consumers. There's no economic reason today why scholarship (or purported scholarship) shouldn't be published online. Of course, the ability to publish digital content for free (or close to free) does not imply anyone will (or should) read what you write. The blogosphere offers an instructive example: the overwhelming majority of blogs attract few (if any) readers. I suspect that the same holds true for arXiv.org. Of course, peer-reviewed content may not fare that much better, particularly given the proliferation of peer-reviewed venues. Regardless, it makes no sense for publishers to act as filters in an age of nearly-free digital distribution.

That brings us to the question of how researchers should establish their professional reputation--and, in the case of academics, obtain tenure and promotion. Today, they have to publish in peer-reviewed journals and conferences. Even if we accept the weaknesses of the current peer-review regime, we should be able to separate content assessment from distribution. The peer-review process (and review processes in general) should serve to endorse content--and ideally even to improve it--rather than to filter it.

Finally, conferences should primarily serve to build community. I find the main value of conferences and workshops to be face-to-face interaction, and I've heard many people express similar sentiments. Part of the problem is that so few presenters at conferences invest in (or have the skills for) delivering strong presentations. But more fundamentally it's not even clear that the presentations are the point of a conference--after all, an author's main motive for submitting an article to a conference seems to be getting it into the proceedings.

Here are some questions I'd like to suggest we consider as a community:

What if presentation at a conference were optional, and an author's decision to present had no effect on inclusion in the proceedings? Would there be significantly fewer presentations? Would those fewer presentation be of higher quality?

What if the process of peer-reviewing conference submissions required the submission of presentation materials rather than (or in addition to) a paper? Would the accepted presentations be of higher quality? Would researchers invest more in presentation skills? What would happen to strong researchers without such skills?

Can we update the traditional conference format to foster more productive interaction among researchers? For example, should we have more poster sessions and fewer paper presentations?

I'd love to see the computer science community take the lead in evolving what increasingly feel like dated procedures for disseminating knowledge, establishing professional reputation, and building community. I've tried to do my small part, co-organizing workshops on Human-Computer Interaction and Information Retrieval (HCIR) that emphasize face-to-face interaction and organizing the SIGIR 2009 Industry Track as a series of invited talks and panels from strong presenters. But I'm encouraged to see "establishment" types like Moshe and Lance leading the charge to question the status quo.

David Karger

You've described a problem that is dauntingly large and hard to solve. We can make progress by tackling pieces.

The database community has one piece of the answer: their premier conference, VLDB, is now driven by a journal submission model (http://www.jdmr.org/). Authors submit to the journal, and a year's worth of journal submissions are presented at each year's conference. The above url links to an excellent discussion of and rationale for their procedure.

You also highlight the way the proliferation of conferences has led to a breakdown in their value as tools for networking and drawing the community together. For these, I propose a simple solution: colocation. Take all those little conferences, and hold them all at the same time under one roof. Let them keep their independence, but share their coffeebreaks and banquets. Most importantly, allow anyone paid up at one conference to attend all of them---after all, no matter how many conferences someone "attends" at the same time, they can only consume one human-being's worth of resources. The federated computing conference does this in large, but it would be equally affective for subfields of computer science to bring together all their sub-subfields.

Anonymous

I really like David Karger's suggestion -- this way I have one place to send my students to where they will hear a large collection of interesting talks and hopefully deliver talks on their own research as well. Right now, if a student's paper is rejected from a "top" conference, I have to pay for them to go there to listen to interesting results and also pay for them to go to another conference where they might be presenting a paper.

Displaying all 6 comments