I blogged recently about video analytics, the attempt to build intelligence into video surveillance so that cameras can not only record our every move in public, but also in some respects understand what they are seeing. Now comes word of the latest twist in this effort: “product recognition.” As Technology Review reports, a startup called Graymatics

has developed software that can automatically identify specific products in visual media—a pair of Ray-Ban sunglasses in a music video, say, or the Banana Republic shirt your friend is wearing in a holiday photo…. the company's executives showed off software that can quickly identify items in videos and photos and then match these with the same or a similar product for sale through various online retailers and marketplaces. As one example, the software matched the sunglasses worn by Brad Pitt and Angelina Jolie in an image accompanying a news article about the movie stars to similar pairs available on Amazon. A reader online who hovered a cursor over the object would see the tagged link for the product…. The software learns by scanning product images available on the Web. It then recognizes objects in images or videos and uses algorithms to break them down into almost two dozen attributes like color, shape, and even texture. Finally, it finds the closest matches in a database of products, such as a feed of Amazon or eBay's stock, or on specific retailers' sites.

Discussion of this technology here is centered on advertising as in the above scenario, but it’s not hard to imagine other applications for product recognition.

The technology could be used not just to let us get more information about products, but to allow companies to get more information about us. If the technology proves effective and reliable at detecting the brand of clothing one is wearing, that could reveal a lot about a person. Human beings have been differentiating themselves by clothing for thousands of years; as this marketing document suggests, clothing and product choices reveal things about our gender, age, geography, lifestyle, and behavior. Not only that, but with modern big data techniques, it’s likely that our clothing choices could reveal many more correlations, such as our income, education, ethnicity, and preferences of all kinds.

Who would apply these analyses? Social networks like Facebook possess enormous stores of photographs posted by their users. The technology could also be applied of course to any other photographs appearing online (blogs, Flickr, news stories, etc.). And perhaps even more significantly, it could also be applied to surveillance cameras.

Imagine this scenario: you walk into a furniture store. A surveillance camera captures your image and applies product recognition. Based on the price and brand of your purse, briefcase, pants, jacket, and shoes, a computer decides in a flash that you are most likely white, middle aged, with a medium-low income and education. It makes guesses not only about your buying power but also about your lifestyle and tastes in music and much else. Right or wrong, this information is passed along to salesmen, security guards and managers and determines how you are treated. Face recognition is also applied so that if you return on another day, their product-based profile can be refined based on subsequent outfits. The store might even enter into a partnership with other establishments to capture your image across different contexts on different days to really pin you down. (Or more likely, a third-party company might emerge that will do this.) Soon, the reality of this system begins to sink in and people start to become very aware and self-conscious about what they wear and the products that they carry.

Fanciful? Perhaps, but in the years I've been at the ACLU I've seen many “far-fetched” surveillance scenarios become real. If there’s money to be made doing this, a company will do it. In fact, Graymatics, the company behind this technology, already offers a product on its site called “InterestInsights™” that promises “deep understanding of user interests via content associations.” I don’t know what that means but it sure sounds like something close to what I’m talking about.

If this kind of system doesn't come about, it will be for one of two reasons: because the economics don’t make sense—or because it's been rendered unnecessary because we're tracked through so many other means…

Whether or not this particular product succeeds, the technology is just another reminder that computers’ ability to understand what they are seeing is surely going to rise to amazing levels in coming years. We are facing a tsunami of artificial intelligence techniques that will be applied to tracking and sifting and sorting us in coming years, and as a society we need to decide whether we’re going to sit back, or act to preserve our privacy values through good public policies.