How can you get CCTV cameras to not only record, but to recognise patterns of behaviour - and potentially prevent serious crimes?

Dima Damen is one of hundreds of PhD students around the world working hard on a challenging problem: how do you make CCTV smart? Later this year, she'll finish her studies at the University of Leeds having created prototype computer vision applications to detect bicycle theft and carried objects. Her research work is a small step towards building the smart CCTV systems of the future.

Nobody would like to spend their time staring at a CCTV image of a bicycle rack and trying to spot a thief. Prompted by her PhD supervisor, Professor David Hogg, Damen realised that a computer vision application could help detect suspicious activity. "I thought that whatever vision could do would certainly be better than the current situation," she says.

The need was clear - half a million bicycles are stolen each year - but how could it be done? "In computer vision, there are techniques to track people when they're walking in front of the camera," says Damen.

She then realised that if a smart CCTV system tracked people dropping off and picking up their bicycles, perhaps any differences could be noted. In computer vision terms, this meant comparing moving blocks of pixels from different times. The initial tests - where 11 out of 13 simulated thefts were successfully detected - involved using clothes' colouring to compare individuals and raising a theft warning when they looked different.

People-watching

"The hard bit, actually, is to link the drop at the beginning of the day to the pickup at the end of the day," says Damen.

Another part of her PhD work at Leeds is to detect carried objects based on shape and colour. "The application that we developed is capable of highlighting all the objects that have been carried," says Damen. "Now, we are at the stage of doing the final step, which is people going into buildings with objects and leaving without them."

Dr James Orwell works in the Digital Imaging Research Centre at Kingston University London, and appreciates the wider issues.

"If you wanted to try and cover everywhere, you'd need a whole army of people watching CCTV," says Orwell. "That's essentially the driver for this technology - it's automation, it's doing things more efficiently."

Like other researchers in the field, Orwell is driven by the intellectual excitement of solving computer vision problems, which, invariably, rely on complex mathematics. The biggest challenge today? Researchers in Australia are aiming to provide robust recognition of faces from low-resolution CCTV footage.

Tracking people effectively in cluttered conditions - such as a busy street - is another goal and a Kingston PhD student, Alberto Colombo, is working on just such a project. To do this with smart CCTV, Orwell says you look for "invariants" in the image data such as shape, size, or clothing colour. "You can track people and you can even track people between cameras."

It's not just people in the frame for smart CCTV. Norbert Buch, another PhD student at Kingston, is working on motion silhouettes to distinguish between vehicle types for urban traffic analysis. His project is funded by Transport for London.

So what would we like smart CCTV to do in future? Orwell talks about the "gold standard" of a human operator who, intuitively, can spot anomalies. For example, a person wearing a bright pink ski jacket on a railway station platform may stand out - but more significant is somebody who doesn't get on to three successive trains.

Smart CCTV finds meaning in what you see and that, in computer vision terms, requires high-quality data from more cameras, better definition, and a faster frame rate. Your digital camera with built-in face detection illustrates simple video analytics - the technology that analyses video for objects, people, or behaviour.

Computer vision academics are already trying their new algorithms on hundreds of gigabytes of Home Office CCTV test data. This research involves looking at abandoned baggage, parked vehicles, doorway surveillance, sterile zone object tracking, and multiple camera tracking.

"There are hundreds of PhD students in the world chipping away at these problems. Everybody agrees on the basics," says Orwell. Today, face recognition is achievable in controlled environments with face detection possible in uncontrolled environments. "I think in five years' time you'll see usable results in face recognition in uncontrolled environments," Orwell adds.

Another notable project is being carried out by Professor James Davis and PhD student Karthik Sankaranarayanan at Ohio State University. They are developing a computerised surveillance system that attempts to recognise whether a person is acting suspiciously.

By analysing and modelling behaviour patterns, the system learns what's normal and what's not. "If you're doing something strange, we want to be able to detect that, and figure out what's going on," says Davis. The system may spot people who stop in an unusual place, wander suspiciously around car parks, or leave a bag behind.

Big Brother Britain

Should we worry about smart CCTV's growing abilities? It concerns Liberty, AKA the National Council for Civil Liberties, which points out that Britain is currently the most-watched nation in the world.

"Technology in this area has come a long way from when CCTV videotapes first came on the scene. Unfortunately legislation has not kept up," says Isabella Sankey, Liberty's director of policy. "Sophisticated systems that can identify and track individuals need proper regulation so that appropriate safeguards can be built in." Being intelligent about using smart CCTV will be important too.