More than 30 years ago, the pioneering urbanist William Whyte was charged by the city of New York with the task of unraveling the mysteries of public space. Why do some such spaces attract crowds of happy visitors whilst others sit barren and empty?

Whyte's research programme, conducted with stopwatches, time-lapse videography, and lots of simple paper charts, was a spectacular success. Based on his findings, he made a series of simple and easily implemented recommendations that the city soon codified into its municipal construction codes.

Today, any visitor to New York might find any number of things to complain about but the wide availability and attractive human affordances of the city's many public spaces is not likely to be among them. Whyte's epiphany was that the way to answer important questions about how to build a commodious and psychologically healthy city lay in careful observation, collection of data and the creative ability to lay aside preconceptions and view a streetscape with a "beginner mind".

Whyte's book The Social Life of Small Urban Spaces, and the short film based on this work, are as fresh and insightful now as the day that they appeared, and are required reading and viewing for any student of urban behaviour.

Fast forward a few decades, and many things have changed, but the fundamentals remain the same. If we want to know how to make a better city, the place to start is at ground level, using observation and measurement, and applying what is known of the human sciences to those measurements to build a psychologically grounded view of the relationship between the physical design of a city and what happens there.

What has changed dramatically is the set of tools that are available to those who would understand the detailed workings of the urban realm. Now we can go well beyond simple observations of the overt behaviour of city dwellers. We can look inside the bodies and minds of those who inhabit urban spaces. We can measure their gaze, their beating hearts, and the state of their autonomic nervous systems as they react to arousing and stressful events.

We can also measure their brainwaves. We can, if we are willing to carry specially designed apps on our mobile phones, record our location and movements, but also our moods, interests, and our patterns of thought. New affordable devices for measurement of a host of brain and body variables are reaching the consumer marketplace every day, and many of them can be paired with phones.

There is even a recipe for a DIY brainwave headset that consumers can print for themselves with a 3D printer and use with openly available software. The wide availability of sensors that measure our physiology, powerful, mobile, computing platforms, and information networks that can connect us to research laboratories with interests in urban behaviour has vaulted the methodologies used by Whyte into a toolbox filled with unprecedented and exciting opportunities to harness huge amounts of psychological data to make sense of how a city works.

Rush hour in Shibuya, Tokyo, the basis for one of the Relive models. Photograph: Rudy Sulgan/Corbis

For a fully realised science of urban psychology though, it isn't enough to have a powerful array of methods for the collection of data. We also need theory and experiment. For a truly scientific approach to the problem of the city, we need to be able to test hypotheses, and compare alternative urban realities. But how can we do this when our subject matter consists of vast constructions of concrete, glass and metal? We can't tear down and rearrange city blocks to see what works. In tandem with our arsenal of tools for measurements, we need a way to build hypothetical city spaces. How do we develop an experimental science of urban design?

In the research laboratory for immersive virtual environments (Relive) at the University of Waterloo, we have turned to simulation methods to help build such a science. Participants are placed into highly immersive simulations of city spaces using sophisticated head-mounted displays and precise motion tracking. They are able to walk freely through photo-realistic simulations of urban spaces that are replete with depth, colour, and motion. We can monitor their gaze and their movements along with their physiology using a set of unobtrusive sensors while they do so. One of our environments is based on Shibuya, a hectic and seemingly chaotic part of Tokyo with its famous scramble crossing – one that allows pedestrians to cross from all directions.

The use of 3D visualisations by architects and planners is not particularly new – design professionals often use simulations to explore the look and feel of a construction. What is new is the capability and will to explore our reactions to such simulations at a fine-grained level of analysis using sophisticated methods of data collection and analysis. Using such methods, we can explore the behaviour of a visitor to a virtual urban setting whose design, because it is built only of pixels, is entirely under our control and can be arranged and rearranged with a few keystrokes on a computer.

For example, in one recent set of experiments, conducted by Kevin Barton of Relive, we designed a stark-looking industrial setting consisting of about 30 city blocks. The streetscapes were so designed to explore the impact of different types of layouts on how we find our way and how the shape of an environment influences attention, cognitive effort, and urban stress. One layout was very orderly, like a Manhattan, Washington or Canberra, while another was more organic and meandering, like New Orleans or London. We discovered that the manner in which such cityscapes were explored, and the psychological reactions to their design, varied strongly with their overall "grammar".

Journeys through the systematic spaces were stereotyped and efficient and accompanied by low levels of arousal and attention. Journeys through the more chaotic spaces were longer, filled with more hesitations, arousal and effortful attention. The value of such findings is that they give us a set of powerful methods by which to predict the psychological effects of an urban design before anything is built.

William Whyte on 5th Avenue, New York, in 1989. Whyte is considered the godfather of urban psychology. Photograph: William Foley/Time & Life Pictures/Getty

Although there is no doubting the power of a virtual reality simulation to unearth relationships between the organisation of the built environment and the operation of our minds, we still need to establish that our simulations are close enough to the real thing that findings from our laboratory generalise to the lived-in environment of a city filled with a hubbub of people, noise, smells, and traffic. For this, there is no substitute for experimentation at street level. Our approach has been to lead participants on walks through city spaces while wearing gear that allows us to measure their cognitive and emotional responses to what they experience.

In one such study, conducted as a part of the travelling BMW-Guggenheim Laboratory, we were able to show that views of green space not only caused people to become happier but also changed their physiology – their autonomic nervous systems showed strong signs of relaxation responses. Though such a finding provides good ammunition for those who advocate for the importance of refreshing oases of nature in dense cities, it was hardly a surprise to us. What is more interesting is that we can reproduce these effects in simulations in our laboratory, meaning that we are able to pinpoint exactly what aspects of natural views produce such restoration.

In one such study, conducted by Deltcho Valtchanov of Relive, we were able to produce physiological relaxation using views of virtual nature that was just as dramatic as that seen in visitors to real-world green spaces. In more recent work, we have shown that a part of the relaxation response to natural scenes hinges on specific properties of visual scenes that can be defined mathematically (the relative proportions of finely detailed contours as opposed to the more coarse "blobby" contours that are present in the image).

What is exciting about this finding is both that it makes a good match with what is known of the preferences of brain areas known to be involved in reward and environmental preference, and that it provides a tool with which to predict the restorative potential of an urban vista on the basis of its visual properties, regardless of whether it contains trees and grass. This might point the way to methods that could optimise such effects in crowded cities where the space available for parkland is severely limited.

The examples I have described suggest that a marriage of laboratory-based virtual reality simulations with real-world observations using smartphones and physiological sensors could form the basis of a new and powerful discipline of experimental urban design based on sound principles of psychology and neuroscience. As we move into an exciting new era of city design in which engaged citizens have never been more interested in how to make cities better, and in which they can be provided with good tools to contribute to the efforts to do so, we are poised to move into high gear.

We have ambitious plans to move beyond these initial steps to build more comprehensive models of both existing and hypothetical city spaces in our lab, and to provide more tools to determine how to grow great cities. As we face an era of daunting problems including population growth, changing energy balances, urban densification, and climate change, the need for solutions has never been more pressing. But on the positive side, our approach provides a strong window of opportunity to entertain and implement important changes.

Colin Ellard is an experimental psychologist at the University of Waterloo, Canada