DRIVERS buying insurance from Progressive, an American insurer, get a choice: they can either supply a few bits of information about themselves and receive a quote based on the behaviour of similar people, or they can install a small gadget in their car. The device monitors their driving and adjusts the rate they pay accordingly. Those who refrain from braking sharply and stay off the roads at night can earn a discount of as much as 30% on the generic premium. For those who drive relatively little, Metromile, an insurer based in San Francisco, simply charges by the mile.

Discovery, a South African health insurer that has expanded to Europe and Asia, has 3m policyholders who have opted for a similar scheme. They can earn discounts by showing that they are looking after themselves, for example by wearing a device that monitors their fitness or by joining a gym. Oscar, a health insurer in New York, gives all its policyholders a fitness tracker; whenever they hit a set goal (walking 10,000 steps in a day, say) they get a refund of a dollar.

Insurers typically rely on blunter proxies to assess risk—age, sex and marital status, for instance. But assuming that all young, single, male drivers are reckless, for example, and that middle-aged, married, female ones are cautious is often inaccurate. It also involves unfair cross-subsidies: prudent and responsible young men help to pay for lead-footed mums.

Modern technology enables insurers to gauge individual risk much more precisely. Monitoring devices provide a wealth of data, as do social media, credit-card histories and other digital records. A pilot scheme in America from Aviva, a British insurer which has since sold its American business, found that analysis of a potential customer’s less conventional data, such as online behaviour and spending habits, was as effective in identifying potential health risks as a medical examination including blood and urine tests.

In a similar vein, Michal Kosinski of Stanford University and colleagues at Cambridge University recently found that computers which are fed a person’s Facebook “likes” are better than a human analyst at predicting whether they smoke or take drugs. Liking “Big Momma” films, a series of comedies in which a detective disguises himself as a fat, flatulent grandmother, is correlated with drug use; a love of curly french fries is a strong indicator of intelligence; fans of Honda are unlikely to smoke. Such prying is just the beginning: insurers speak with straight faces about a time when sensors in customers’ homes will alert plumbers to weak pipes before they burst, and glucose meters in contact lenses will keep a record of how healthily they are eating.

All of which calls into question the basic logic of the insurance industry—that it is impossible to predict who will be hit by what misfortune when, and that people should therefore pool their risks. “Cherry-picking” low-risk customers and spurning those who will prove liabilities is becoming much easier. In the process, insurers may transform themselves from distant, cheque-writing uncles into ever-present and interfering helicopter parents. The prize for the nimblest will be huge: the industry manages more than $30 trillion, nearly as much as the $36 trillion held by pension funds; last year it made $338 billion in profits.

Data mining and monitoring not only allow insurers to price policies more accurately, but also enable them to modify customers’ behaviour. “I think of us as Big Mother,” says Brian Vannoni of Guidewire, a firm that analyses data for insurers. Last year Guidewire helped predict the path of a hurricane in Australia, allowing a home insurer to narrow the group of customers whose houses would need extra weatherproofing. Kimberly Harris-Ferrante, an insurance analyst, tells of a client whose database singles out unsafe drivers so that agents can visit them at home in an effort to persuade them to change their habits. Kaiser Permanente, an American health insurer, does something similar with its most at-risk policyholders.

Progressive tells customers who use its monitors where they tend to drive unsafely and what their foibles are (overly sharp turns, for example). Those receiving such advice crash less. AllLife, which insures people with manageable diseases such as HIV and diabetes, offers them free monthly check-ups. If these show that they are not sticking to their doctors’ advice, their premiums go up. Happily enough, people tend to respond by looking after themselves better rather than pay more. Discovery policyholders went to hospital less and ran up lower bills after joining its health-monitoring scheme.

As a result of such schemes, risk pools will become smaller, says Jon Hocking of Morgan Stanley, a bank. Together with the Boston Consulting Group it recently produced a report that predicts that damage to insured homes will fall by 40-60% if all the latest technology is adopted. The risk pools for home and car insurance might shrink by as much as $109 billion, the report speculates. It also found that insurers using the latest techniques to cherry-pick the best drivers would receive just a ninth of the claims of the most hidebound firms.

Demand for Progressive’s prying car insurance grew by 28% last year alone. It now accounts for over $2 billion in premiums. This is part of a wider growth in such monitoring schemes, albeit from a low base (see chart). A natural consequence of more precise individual underwriting and pricing is that some risks may stand revealed as being so high that they become uninsurable. In some instances, this seems right: the world’s worst drivers may need to be discouraged from taking to the road. In other cases, however, such outcomes would raise hard questions: if people looking for medical insurance have a genetic predisposition to a deadly disease, there is less they can do about it. Likewise, those who do not want to let insurers look into their private lives risk being penalised. Honeywell, an American multinational, was recently sued, unsuccessfully, by America’s Equal Employment Opportunity Commission (EEOC), for asking employees to provide biometric data in exchange for a discount on health policies. The EEOC claimed the discount was so big that it was tantamount to coercion. Insurers are nonetheless pressing ahead, in part for fear that if they do not embrace the new technologies, others will. A survey by Gartner, a consultancy, showed that nearly half of British consumers would be happy to buy insurance from data-rich companies like Amazon or Google; less than 20% ruled the idea out flatly. Such firms, observers note, have intimate knowledge of their customers and have earned their trust, leaving them much better placed to play the role of Big Mother. Insurers, in contrast, are in touch with most of their customers once a year. “I’m far more concerned about the Silicon giants than about the AXAs and Generalis,” admits one director at a big European insurer.

Google has started to dip its toe in the water. In 2011 it bought a website called BeatThatQuote.com, which allows users to compare insurance quotes, among other things. This month it launched a comparison site for car insurance in California and has licences for many more states.

In response, insurers are busy trying to make themselves more like tech firms. Aviva last year hired a Progressive executive, Adam Kornick, as its first “global insight chief”. He will help set up a “digital garage” in a trendy part of London, where employees can think big thoughts. Allianz, a German insurer, is spending $500m a year to update its digital capacities. Much of this will go on the creation of its own data cloud, supported by five data centres around the world, so that it does not have to entrust its data to anyone else.

Partnerships with non-insurers are another way for conventional insurers to smarten up their act. State Farm, one of America’s biggest insurers, has teamed up with ADT, a home-security company which allows people to monitor their homes from a distance. FitBit, which makes watch-like fitness trackers, now works with a number of insurers and employers who want to keep tabs on their policyholders or employees. Andrew Rosenthal of JawBone, a competitor, says American health insurers are getting in touch because they “all want to know how to build relations with their customers”.

If any group ought to be worried by these changes to insurance, it is not the customers. Underwriters claim theirs is a unique skill that cannot be easily copied. But any decent health monitor would be flashing frantic warnings by this stage.