Imagine if you could spend a morning climbing a skyscraper, running with the bulls in Spain, or even walking on Pluto. As news outlets experiment with immersive content like virtual reality and 360-degree video, a smartphone and simple cardboard headset can make that happen.

According to Graham Roberts, director of Immersive Platforms Storytelling at the New York Times, the goal of journalism produced using these new tools is the same as it has always been: to transport readers into the story. But as The Times and other publications experiment with virtual reality to transport viewers into less innocuous situations — say, to the middle of a war zone or a jail cell in solitary confinement — new ethical considerations arise, not only about how these stories are produced, but also about the ways in which audiences experience and remember them.

The concerns accompanied The Times’ famous roll-out two years ago of a virtual reality experience delivered directly to readers’ homes, complete with a cardboard viewer. While praise for the foray was widespread (the immersive video took readers inside the plight of refugee children), some journalists immediately raised pointed questions. The Times was experimenting with a process “that will often be based on tricks and deceptions by photographers/cameramen,” Robert Kaiser, the former managing editor of The Washington Post, complained to the paper’s public editor at the time.

In the two years since, many other publications have begun experimenting with virtual reality and immersive storytelling — and the debate over where to draw lines continues. Roberts likens the rise of virtual reality journalism to the period of time between the invention of movies and movie theaters: The technology is there, but we don’t yet know what the rules and conventions of the medium should be.

Meanwhile, the Society of Professional Journalists, which publishes a Code of Ethics that journalists are encouraged to follow, doesn’t provide any specific guidelines for reporting in virtual reality. “It’s difficult because within SPJ, we’re resistant to creating different standards for different media,” says Andrew Seaman, the chair of the ethics committee within SPJ. “The standards that you apply to print, online, broadcast should be the same regardless of the medium.”

But how do the principles of fair and accurate reporting apply when the camera setup doesn’t simply depict an event, but exploits human perception to place audiences directly in the scene? How are editing choices amplified in such contexts? How to shoot 360-degrees without capturing the journalist and his or her equipment in the middle of it — and are efforts to remove them fair? And how does experiencing a story in virtual reality affect how viewers remember and respond to the story at hand?

One potential risk, according to Jennifer Talarico, a psychologist at Lafayette College who studies the many flaws to our memories, comes down to something called source-monitoring error. Source memory, she explains, is our way of remembering how it is that we came to know something. “This can be very general — did I read this, did something tell me this?” says Talarico. “Did I take my vitamin or did I just think about it? Did that really happen or was it just a dream?” Beyond that, source monitoring is our ability to remember whether we lived an experience or if it we adapted it as our own from a story someone told us.

“One would imagine that the more similar the narrative context is to an experiential context, the more likely someone is to make a source monitoring error,” says Talarico. She suspects that an immersive virtual reality experience could make it easier to mistake a fictional narrative experience with something from the audience member’s real life.

Even without source error, Talarico points out that an emotionally-potent, immersive experience can really stick with people. Without proper care, this can give people the wrong idea. “To the extent that any of these forms are more memorable than others, and to the extent that all media are adapted for good and evil, they have a greater potential to either mislead or educate,” says Talarico. One oft-cited example along these lines: a 2011 study out of Stanford suggesting that subjects who experienced cutting down a redwood tree in a virtual, immersive environment used 20 percent fewer napkins to clean up a planned, post-experiment “accident” with a glass of water than subjects who only read about cutting down trees.

To Robert Hernandez, an associate professor at the University of Southern California’s School for Communication and Journalism and a pioneer in the realm of virtual and immersive media and a self-proclaimed “mad scientist” of journalism, that’s the power and allure of virtual reality. “I can put you in a situation where your brain knows it’s not real, but it still feels like it’s real,” he says. “That can invoke trauma, that can invoke bad memories. And as a journalist I have to ask what’s my job? Is my job to hurt you? It’s to inform you, and sometimes it’s a little in the middle. I need you to understand what that little refugee kid is going through.”

Hernandez explains that this narrative — that a new medium will inevitably lead to misinformation — has emerged every time there’s a new way to tell stories. “You don’t cease being ethical as a journalist just because there’s a new piece of technology,” he says, pointing out that any good story, photo, or video will invoke an emotional reaction that some would decry as manipulation. “You have to think about it, you have to test it.”

Roberts says his newsroom at The New York Times hasn’t formally weighed the psychological impacts of stories shared this way, but noted that there are rules journalists can lay out in advance to make sure that virtual reality is used in a responsible, ethical manner. Namely, they need to remind the audience of exactly what they are seeing and how it fits in the context of the broader story. The Times’ virtual reality film “Fight for Falluja,” for example, which brings viewers to the front lines of a battle between American troops and ISIS, was embedded within a broader story about the same conflict — though there’s no guarantee that people will actually read it.

In that sense, providing context within the virtual-reality, or VR, experience may also be beneficial. The broader the scope, the less likely those people are to incorrectly assume that they’ve seen the whole picture. “What’s great about VR is you can put text, you can add subtitles,” says the SPJ’s Seaman. “I think there are opportunities to blunt the force of VR.”

Still, even simple discussions of context raise issues of what some critics would call manipulation, Hernandez says. One basic example: Although a 360-degree camera can capture virtually every bit of a scene — this would normally includes the journalists themselves and, inevitably, the tripod or other rigging on which the camera itself is mounted. Publications have long disagreed on whether and how to edit these production artifacts, which a viewer can plainly see by “looking” or panning around the immersive environment.

It gets much more ethically complicated than editing out the tripod. The Times’ 2015 refugee film, called “The Displaced,” received some criticism when viewers realized that certain scenes, including one in which a young boy pedals his bicycle down a sunlit street — a moment of joy in an otherwise stark exploration of refugee life — was at least partly an artificial construction. “Obviously, he doesn’t spontaneously ride down the street with a VR rig on his bicycle every afternoon,” Jake Silverstein, the editor of The New York Times Magazine, told The Poynter Institute, the journalism ethics nonprofit, last year. “He had to wait to let us mount the VR rig on his bicycle, which was complicated and difficult and required a little trial and error. But once he did, he was riding his own bike down the street that he typically rides his bike down.”

Jeffery DelViscio, director of multimedia and creative at STAT News, a digital publication covering biotechnology, health, and medicine, says it’s important to think about these sorts of situations ahead of time. “Once you put the headset on, it’s really meant to be like you’re the one standing there and you’re the camera,” he said, “so you have to be careful with how you handle them.”

Hernandez noted that the Associated Press the is among those media organizations that have opted to leave the image of the tripod intact, to avoid any charge that they’ve manipulated the experience. For his part, though, Hernandez sees no problem with this sort of tweaking to improve the immersive experience — something technology is making it increasingly easy to do. “We’ve produced this experience, so we want to optimize it,” he says.

Of course, just how far such optimization can be pushed without violating basic journalistic principles — if you can subtract elements, you can also add them, after all — is something that the industry continues to actively debate. Using sound journalistic judgment is a good starting point, Hernandez suggests, though he also hopes that immersive storytelling will be greeted by an audience that lends a critical eye towards the media it consumes — and the technology used to create it.

“It could be used for good or ill,” Hernandez says, “but that statement could be applied to any media type in our history.”

Dan Robitszki is a neuroscientist-turned-journalist based in New York. He is a recent graduate of the Science, Health, and Environmental Reporting Program (SHERP) at NYU, and his freelance work has appeared in Scientific American and ScienceLine, among other publications.