To Rate the Editors, Rate Their Publications





Magazines Anthologies · Analog · Asimov’s · Clarkesworld · F&SF · Lightspeed · Tor.com · Meeting Infinity · Mission: Tomorrow · Old Venus · Operation Arcana · The End Has Come · Twelve Tomorrows 2016

Of the six professional magazines, five of them have an editor-in-chief who makes the call on what to print, and that's clearly the person in contention for this award. Tor.com, however, assigns different editors to different stories, and two of those met the bar.



Picking nominees for Best Editor (Short Form) amounts to evaluating those two Tor.com editors plus the ten individuals who edited the other eleven publications.



To get started, lets look at the data. We rated editors based on the recommendations their stories received. The idea is that good editors publish lots of stories that get good reviews, and they don't publish a lot of stories that get bad reviews. Many publications got few or no reviews of any sort, and that (plus the Hugo eligibility requirements) ruled out all but six professional magazines and six anthologies for our data-driven analysis. ( See the details .)Of the six professional magazines, five of them have an editor-in-chief who makes the call on what to print, and that's clearly the person in contention for this award. Tor.com, however, assigns different editors to different stories, and two of those met the bar.To get started, lets look at the data.

Quantity: The Magazines Dominate





It treats the Tor.com editors Ellen Datlow and Patrick Nielsen Hayden as if they had each written an anthology consisting of the stories they edited for Tor.com in 2015.



Here's how we're going to rate the editors: We're going to look at a variety of measures (e.g. "most original fiction published") and see which publications are in the top 5 (again, treating Datlow and Nielsen Hayden as "publications"). We'll give every publication one "point" every time it appears in the top-5 list according to a particular measure. Then we'll see which publications earned the most points, and finally we'll rank their editors accordingly.



The first measure we'll use is total quantity of original fiction (regardless of quality). Using this measure, we give one point each to Analog, Asimov's, Clarkesworld, F&SF, and Lightspeed for being the five most prolific SFF publishers in 2015.



For the second measure, we'll look only at Recommended word counts (just the blue part of the bars). Old Venus edges out Clarkesworld, but otherwise it's the same list as the first. So we give one additional point to Analog, Asimov's, F&SF, and Lightspeed, and we give one point to Old Venus as well for being the top-5 publishers of Recommended content.

Quality: A Different Story

This chart shows the categories as percentages of the total word count for each publication. It lets us judge quality regardless of quantity.





F&SF (5)

(5) Asimov's (3)

(3) Old Venus (3)

(3) Analog (2)

(2) Lightspeed (2)

(2) Ellen Datlow (2)

Patrick Nielsen Hayden (2)

Clarkesworld (1)

(1) Meeting Infinity (1)

(1) Operation Arcana (1)

Notice that F&SF tops the final list, even though it was not number one by any of the five measures. This is a good thing because we wanted a measure of overall excellence. F&SF earned the most points because, although it's never at the top, it's the only publication that's always near the top, no matter which measure we use.



This gives a ranking for publications. Now we need to turn that into a ranking for editors.

Putting it together

The Naïve List

Lightspeed and Operation Arcana. Since neither was ever in the same list, we simply combine his points together. George R.R. Martin and Gardner Dozois coedited Old Venus, but



John Joseph Adams edited bothand. Since neither was ever in the same list, we simply combine his points together. George R.R. Martin and Gardner Dozois coedited, but Martin has asked people to give Dozois all the credit for the purpose of this award.

Editor Publication Points C.C. Finlay F&SF 5 John Joseph Adams Lightspeed and Operation Arcana 3 Gardner Dozois Old Venus 3 Sheila Williams Asimov's 3 Ellen Datlow Tor.com 2 Patrick Nielsen Hayden Tor.com 2 Trevor Quachri Analog 2 Neil Clarke Clarkesworld 1 Jonathan Strahan Meeting Infinity 1

Other Factors

Here are examples of things that arguably should count towards an editor's rating but which the charts above do not include. You should take a look and see if any of these is important to you.

Development: Did the editor develop new writers? How many writers eligible for the Campbell award did he/she publish? (We'll try to track this in the future.)

Pushing the Envelope: Did the editor publish anything special? (E.g. Clarkesworld 's translated stories or Lightspeed 's Women/Queers Destroy SF issues.)

's translated stories or 's Women/Queers Destroy SF issues.) Editorial Stance: Did the editor use his/her position to try to effect positive change in the SF community, either through editorials, interviews, or speeches?

Reprints: Reviewers only recommend original fiction, but many readers probably enjoy the reprints just as much. Clarkesworld and Lightspeed publish reprint stories every month, and some of these editors did extensive work on reprint anthologies, but the naïve list doesn't give them any credit for that.

and publish reprint stories every month, and some of these editors did extensive work on reprint anthologies, but the naïve list doesn't give them any credit for that. Art: Some magazines have far better cover art and illustrations than others.

Nonfiction: All the magazines have content ranging from science articles to book/movie reviews to author interviews to editorials. They take time and effort to produce, and they further separate magazines from anthologies. We didn't count them here, but you might want to reward an editor for quality nonfiction articles.

Author Service: Things like how long it takes for a magazine to accept or reject a manuscript, how promptly they pay authors, whether their contracts are reasonable, etc.

Publication Quality: Are there formatting errors in the print or electronic formats? Do they produce podcasts? Do they include word counts or some other way to identify the short fiction category? Can you buy back issues easily? Is the web site easy to navigate?

Blurbs: Do the blurbs before the stories give too much away?

Advancement: Did the editor elevate his/her publication from semi-prozine to professional this year?

Management: Attracted well-known film critics, book critics, science writers, etc. for nonfiction columns.

Drop one or more of our categories. (E.g. perhaps you don't agree with our ratings for Not Recommended stories. Or perhaps you don't mind the occasional story that is Not SFF .)

stories. Or perhaps you don't mind the occasional story that is .) Change the per-category list size from 5.

Give different scores based on the position in each list. Examples

Where Did This Data Come From?

So where did we come up with all these numbers, and what do they really mean? From here down, we'll discuss the decisions we made and we'll offer some of the raw data.

What does an editor do?





The exception is Tor.com, which assigns (and credits) different editors to different stories.



Given that, judging them by the quality of the stories that were printed in their magazines and anthologies makes good sense. The Hugo Award for Best Editor (Short Form) is for the people who select the stories that go into magazines and/or anthologies. These people generally have the title "editor-in-chief," as a little time spent reading their editorials and blog posts makes clear. (They constantly say things like "I accepted this story" or "I saved you from stories like that.") The award has nothing to do with copy editors, who correct spelling and grammar.The exception is Tor.com, which assigns (and credits) different editors to different stories.Given that, judging them by the quality of the stories that were printed in their magazines and anthologies makes good sense.

Who is eligible?

Future Visions, edited by Jennifer Henshaw and Allison Linn

Stories for Chip, edited by Nisi Shawl and Bill Campbell

War Stories from the Future, edited by August Cole

Why exclude the smaller publications?

How do you evaluate an editor?

A. Story-based Metrics

Ultimately editors are gatekeepers responsible for bringing us good stories. We should credit them for good stories and penalize them for bad ones. To do this, we defined four quality levels.

1. Not SFF





It's worth discussing why we chose to ignore the other reviewers on this one. The truth is, almost all of the non-SFF stories were excellent, well-written stories. The trouble is, it isn't fair to include them with SFF. Once you drop the requirement that a story have a speculative element, it becomes far easier to write. You eliminate the need for infodumps. You greatly reduce the amount of suspension of disbelief you require from the reader. You make it easier for the reader to identify with your characters, and you can use the time and words you saved to focus on making the reader care about the characters. These are all stories that RSR rated as not being science fiction or fantasy, regardless of whether anyone else liked them ( see list ).It's worth discussing why we chose to ignore the other reviewers on this one. The truth is, almost all of the non-SFF stories were excellent, well-written stories. The trouble is, it isn't fair to include them with SFF. Once you drop the requirement that a story have a speculative element, it becomes far easier to write. You eliminate the need for infodumps. You greatly reduce the amount of suspension of disbelief you require from the reader. You make it easier for the reader to identify with your characters, and you can use the time and words you saved to focus on making the reader care about the characters.

A good editor should understand this and not get so excited about the quality of a story that he/she fails to notice that it's not an SFF story at all.

2. Not Recommended

3. Recommended

4. Ordinary

B. Other Accomplishments

From reading editorials and doing a few Google searches, we know of a few special things that the editors did personally that ought to figure into any ranking. In no way is this complete. We'll happily amend this page to add new information that people bring to us.

1. John Joseph Adams (Lightspeed, Operation Arcana, and The End Has Come)

Lightspeed from a Lightspeed, he also edited the anthologies Operation Arcana and The End Has Come as well as one magazine we didn't review (Nightmare) and four more anthologies. He also produced the "Lightspeed this year, which was a first. In 2015, he turnedfrom a semi-prozine into a professional magazine . In addition to editing, he also edited the anthologiesandas well as one magazine we didn't review () and four more anthologies. He also produced the " Queers Destroy Science Fiction " issue ofthis year, which was a first.

2. Neil Clarke (Clarkesworld)

In addition to editing Clarkesworld, he also edited Forever Magazine and the anthology "Clarkesworld: Year Seven." A major accomplishment this year was the large number of translated science fiction stories he published. Another accomplishment was that in addition to 4 original short stories per issue he now publishes an original novelette.

3. Gardner Dozois (Old Venus)

Gardner edits the reprints for Clarkesworld, and he edited "The Year's Best Science Fiction: Thirty-Second Annual Collection."

4. C.C. Finley (The Magazine of Fantasy & Science Fiction)

As editor of F&SF he changed their policy against accepting electronic submissions and apparently set up the submission system himself (it's hosted on his personal domain).

5. Jonathan Strahan (Meeting Infinity)

6. Sheila Williams (Asimov's Science Fiction Magazine)

Other Opinons

Richard Horton, in his article, , in his article, More on the Best Editor, Short Form, Hugo , makes strong arguments for nominating Jonathan Strahan, John Jospeh Adams, and Sheila Williams.





George R.R. Martin, in , in A Rocket For The Editor, Part Two , makes arguments for a variety of editors and discusses the category in broad terms. (As well as eliminating himself from the list.)





What Do You Think? What's the Best Way to Help People Make Best Editor (short form) Nominations?

Best Editor is a notoriously difficult category to nominate for, and while a number of people like our approach, it has also drawn some criticism. (See the comments relating to item #10 on File770's Pixel Scroll 1/14/16 I’m Not A Pixel, I’m A Free Scroll .) We're definitely interested in other ways to attack the problem, so please share your thoughts in the comments below.

To help SFF fans make nominations for the Best Editor (Short Form) category in the 2016 Hugo Awards , we've analyzed all the recommendations by eight prolific reviewers of 560 original works of short fiction from the top publications of 2015.This post presents that analysis as a baseline for Hugo Award voters to customize and produce their own list of nominees.Update February 9, 2016: George R.R. Martin asked to not be considered for this award.In the third measure, Ellen Datlow had the highest percentage of stories, followed by, and. These are the five publications with the highestoffiction, and we give each one point.In the fourth measure, Ellen was the only editor to have zero stories. The next best in that regard wereand. Again, each gets one point for being the best 5 (i.e. having the lowest percentage) forFor our final measure, accepting stories that arewasn't a big problem for most publications, but F&SF deserves credit for being the only magazine that managed to publish none like that. We give it and Patrick Nielsen Hayden one point. (We see this as less challenging for anthologies, which have a theme, so we don't credit them for this.)These are the final point totals:This gives us a list of four names: John Joseph Adams, C.C. Finlay, Gardner Dozois, and Sheila Williams. It's the same result as a subjective analysis that said "andare the best magazines,was the best anthology, and Adams deserves credit for having both a top magazine () and a top anthology ()."The only problem is it doesn't take into account any of thethings that editors do (and we need to find a fifth name somehow).You can also analyze the data differently. Here are a few suggestions:We'll offer three examples, based on different ideas a person might have.: "Giving Dozois three points foris unfair. It should just be 1.5 points because Martin did half the work. And Neil Clarke deserves at least one extra point for all the innovative work he did bringing foreign-language translations to." In that case, you'll end up with John Joseph Adams, C.C. Finlay, and Sheila Williams and a 4-way tie between Datlow, Nielsen Hayden, Quachri, and Clarke. You might drop Nielsen Hayden (because he had zerostories) and Quachri (because you're mad thatcan't fix the formatting of their Kindle stories) and so you'd nominate John Joseph Adams, Neil Clarke, Ellen Datlow, C.C. Finlay, and Sheila Williams.: "The Tor.com ones shouldn't count because they didn't have to run a whole magazine. I don't agree with RSR's notion of 'bad' stories, and I'm fine with an occasional story that's." In that case, you'd drop Ellen Datlow and Patrick Nielsen Hayden entirely, subtract one point from Operation Arcana and Old Venus, and two points from F&SF. That would give you John Joseph Adams, Gardner Dozois, C.C. Finlay, Trevor Quachri, and Sheila Williams.: "I'm going to nominate Niall Harrison (editor of), even though he's not on the list, because he accepted my manuscript and was so helpful to me. Otherwise, I don't think anything matters but total quantity of stories that weren't." In that case, you'll have Neil Clarke, C.C. Finlay, Niall Harrison, Trevor Quachri, and Sheila Williams.We've made a partial list of extra accomplishments by the editors to help with this.Good luck!You can read the official definitions of the Hugo categories , but the short answer is anyone who edited short speculative fiction published in 2015, provided they produced the equivalent of at least one magazine issue or anthology during 2015, and had a lifetime total of four times that much. It doesn't matter if that work was a fanzine or a professional magazine or a big anthology. By taking the average across 12 months of, and, we decided that "the equivalent of one magazine issue" amounts to 45,000 words of original fiction.Only three of the Tor.com editors edited that much fiction in 2015, and only two meet the lifetime requirement. Of the editors of the nine anthologies we reviewed, only six met the lifetime requirement. Here are the three anthologies that were not eligible.Since we didn't read a lot beyond the professional publications, we didn't really have a lot of choice, but we can offer evidence that none of the editors of semiprozines or fanzines would have made the list anyway.Even though RSR didn't read the semiprozines, many of the 7 reviewers we follow did, so let's look at how many stories they recommended across all publications.(Click to enlarge.)The orange bar onrepresents reviews by K. Tempest Bradford, who seems to have an unusual fondness for the publication. We're treating that as an outlier.This shows that 82% of all recommended stories were in sources RSR reviewed, and 57% of recommended stories were inoralone.andall published about the same volume of fiction in 2015 asdid whilepublished about half that much, yet they had only a quarter as many recommended stories. Therefore they wouldn't be competitive either in quantity or in quality, at least not the way we've chosen to measure them.What this tells us is that the professionals really are different--very different--from the semiprofessionals, and therefore it really does make sense to consider only the editors of professional publications for the Best Editor (short form) award. That's consistent with the oft-heard claim that Best Editor (short form) is a proxy for "best prozine."A slightly different measure of an editor is how many bad stories they print. Bad stories range from things that never should have left the slushpile ( 1-star stories ) to stories that make it hard for readers to suspend disbelief or which don't seem to have plots ( 2-star stories ).Currently, RSR is the only reviewer that tries to identify bad stories. In cases where we gave a story 1 or 2 stars but any of the other reviewers recommended it, we counted that story as ordinary: neither recommended nor not recommended.These are stories that were recommended by any of the eight reviewers we surveyed ( see list ). These include both RSR 4 and 5-star stories. As mentioned above, when RSR recommended against a story that anyone else recommended, we treated it as ordinary.These are all the rest of the stories. Either RSR gave them 3 stars and no one else recommended them ( see list ), or else someone did recommend them but RSR gave them 1 or 2 stars ( see list ).Every story falls into one and only one of these four categories; there is no overlap.In 2015 he edited "Grand Crusades The Early Jack Vance Volume 5" and "The Best Science Fiction and Fantasy of the Year: Volume 9."Supervised a redesign of the Asimovs.com website.