How important is it for readers of game reviews to understand the conditions under which a review was written? Would it change things if you knew all the early reviews of a game were written based on play time given in one location, under a managed schedule? What if the publisher paid for airfare and hotel rooms for the press? If you read an early review of a certain AAA game, chances are the circumstances behind the review weren't mentioned, but they may change how you perceive the review.

The title was Modern Warfare 2, and Activision paid for both the airfare and the hotel rooms to get publications to cover the game. If you wanted early coverage, this was the only way to get it. Joystiq added an addendum to its review, expressing its issues with the practice, pointing out the event was organized and paid for by Activision. "As this was Joystiq's only opportunity to review the game in advance of its release, we willingly deviated from our standard policy of not accepting accommodations and used the room," the site explained in its disclaimer. "We did so because we felt that participating in this event best served the interest of our readers."

Garnett Lee wrote about his own misgivings in the Shacknews story about the game, and because of the circumstances he refused to call his thoughts a review. "A couple weeks ago Activision took a group drawn from members of the videogame media up to a resort in Santa Barbara for an intensive couple of days playing the freshly completed Modern Warfare 2. Extravagant? Sure, but that goes with the territory for a game that's become one of the biggest entertainment franchises in pop culture," he wrote. "Such an ideal setup also felt a little too removed from the real world to base a fair review on. So take this as impressions then from playing the full game."

Most outlets didn't disclose the circumstances of the review. We tracked down a writer who was willing to speak about the event under the condition of anonymity. Let's take a look behind the scenes, and see how the sausage is made.

"We weren't interfered with"

Reviewers were given a hotel room with a high-definition display, a surround-sound system, and a headset. The game was delivered on a hard drive, and progress was saved on memory cards that were collected after the event. The game was played in marathon sessions so reviewers could finish the single-player campaign.

"We weren't interfered with by PR at all while we played. No knocks on the hotel room door to see how things were going, no taps on the shoulder during the common hall review sessions either," our source explained. "They were available to answer queries if asked, but mainly kept to themselves and weren't breathing down our necks." He pointed out that Activision tends to be very respectful of the boundaries between PR and writers.

He did share some reservations about the time spent playing the multiplayer portion of the game. "Playing in a room full of friendly developers and your games press colleagues with perfect connections is undoubtedly much more fun than gaming online with some of the legendarily obnoxious Xbox Live players," he told Ars. "You'd have to be a pretty na�ve reviewer to think there's no difference—and if you let that experience form the basis of your multiplayer assessment without qualification, you're giving too much credit to online gamers' behavior."

The reviewers had two days of managed time to try the single-player, the Spec-Ops missions, and the multiplayer. The reviews seemed to be limited to the Xbox 360 version of the game, with reviewers given no option to try the PC version. Does this affect how reviewers think about the game? Our writer friend thinks so.

"Yes, to the extent I wouldn't have plowed through the single-player campaign in one sitting. But I'd suggest also that doesn't work in Activision or IW's favor—put most gamers in front of anything for a five- to seven-hour stretch and it will seem old." He also points out the importance of disclosure. "We referenced the review event conditions in our review at points where we thought it would potentially be different from at home, and obviously we consider that adequate," he told Ars. "However I readily concede that, too, is open to subjective analysis, especially when gamers erroneously suspect publishers are stuffing money into reviewer's pockets left, right, and center at the best of times."

This is becoming more common

It's not rare for publishers to set up reviews in this way. We took part in a Microsoft event to review the single-player content in Halo 3, and disclosed it in the review. We did not, however, accept paid airfare or accommodations. The multiplayer was played on our own time, under normal conditions. Another publisher invited us to play through an upcoming game in their offices more recently, and put up the airfare and accommodations. We declined, and told them to simply send the retail code when it was available.

The problem is that the gaming press that isn't local to these event is put in a difficult position: either break their ethical policies to ensure early coverage, or deliver late reviews. It's possible that some sites could pay for their own airfare and hotel room, but that's a huge budgetary decision for a single review. Presenting the game's online play in such ideal conditions also means that no one is shown a clear picture of what play will be like for the actual consumer, and that should be disclosed during the review.

The reviewers at this sort of event are told how long they can play, they are locked down to a specific version of the game—in this case the 360—and by accepting the airline ticket and a stay in "a resort in Santa Barbara," the ethical situation becomes muddy.

Would knowing when and how a review takes place change the way you think about the final score? What's clear is that readers should know the circumstances surrounding how the game was played, and how controlled the situation was. Reviewing the game at home is one thing, reviewing it in a remote location, surrounded by other enthusiasts and the game's creators is another. There is no reason Activision couldn't ship writers prerelease copies of the game: it has been finished and packaged for some time, and the leaks had already spread across the Internet.

Instead the company decided to pay for the gaming press to come to a specific location, stay in company-provided rooms, play the game a specific way... and all this came at a substantial cost to Activision Blizzard.

What value did they get for that money? We asked Activision and have yet to receive a response.