In the glorious sunshine of Andrew Lloyd Webber's stud farm Watership Down, overlooked by rolling pastures, retired millionaire Dr Philip Brown is inspecting his investment.

Her name is Darysina and even to an uneducated eye she is something special. Standing quietly next to a paddock gate she is sleek, muscular and even tempered. She oozes quality.

But what has Brown so excited isn't simply the four-year-old mare's classy pedigree, her record on the field or her cost – €800,000 (£640,000) at auction in December. No. What makes Darysina so precious is the foal she is carrying. In March, after her DNA was comprehensively analysed, she was introduced to the charms of Frankel – the greatest and most successful thoroughbred in modern racing history. Frankel is the superstar of horses. He was unbeaten in his 14-race career, is worth more than £100m and his genes are so valuable that his owners charge £125,000 for a quick roll in the hay.

The first Frankel foal sold at public auction for £1.5m. Photograph: Dan Abraham/racingfotos.com/Rex Features

Last week, his first foal to be sold at public auction was snapped up for £1.15m. Any wealthy owner aiming to produce a quality thoroughbred would consider Frankel as a suitor, but Brown didn't want to leave anything to chance. His decision to use him only came after he had consulted a British company that screens equine genes to find the best possible match.

Looking at the DNA of horses to find good breeding stock may seem obvious, but it goes against centuries of tradition in this most conservative of sports where suspicion of applied molecular genetics is rife. The Jockey Club insists on gene tests to confirm parentage, but bans the use of cloned horses – even though in 2012 the International Federation for Equestrian Sports announced that clones would be allowed to compete in future Olympic Games. Indeed, breeding decisions are made almost entirely by studying pedigree – the records of bloodlines and race results that go back generations.

But Philip Brown isn't a typical owner. A former science correspondent of the Daily Express in the 1960s, he made his fortune in science publishing. He is sometimes called the wealthiest ex-journalist in Britain and is fascinated, and knowledgeable, about molecular genetics. He was aware that pedigree can be a poor guide to quality: an ancestor five generations back contributes 3% of an animal's DNA.

The Green Monkey is a classic example of the dangers of relying on bloodlines. In 2006 the bay two-year-old colt with an impeccable pedigree sold at auction for $16m – the highest price of a publicly auctioned thoroughbred. It ran just four times and failed to win once. "I'm a scientist and I have a belief in science," says Brown. "So if you say there's a genetic reality to all this, you ought to apply the rules of genetics. It makes sense."

Over the last three years, Brown has worked closely with Dr Stephen Harrison, a geneticist whose Canterbury-based company, Thoroughbred Genetics Limited, creates genetic profiles of horses. The company, founded in 2000, was the first in the world to offer DNA screening for racehorse performance. Over the last decade, Harrison and colleagues have used whole genome analysis from thousands of animals to identify markers linked to equine stamina, strength, respiratory system and energy use. He claims Thoroughbred Genetics' techniques are 75% better than conventional non-genetic methods of choosing winners from a group of horses. Its methods give animals a performance profile based on an analysis of 750 genetic markers. The tests generate cluster graphs that indicate whether or not a horse is best suited to be a sprinter, a long-distance athlete or something in between. By comparing the analyses of mares and stallions, it is possible to get a clue where any progeny will fall in the chart.

The tests also show the level of inbreeding in a horse. We're used to thinking of inbreeding as a problem. After all, it led to blood disorders in the royal families of Europe in the early 20th century and hip problems in German shepherd dogs. But sometimes a degree of inbreeding can be beneficial. One of the most striking examples of this is the 90 Chillingham cattle of Northumbria, thought to be descendants of medieval cattle. They are so inbred that calves are almost genetically identical to their parents. Yet despite the narrow gene pool, the Chillingham cattle are thriving. In 2001 a study in Nature showed that no other cattle had joined the herd for more than 300 years – yet the creatures are fit and healthy. All harmful recessive genes have been purged from the group through a combination of inbreeding and selection.

The Chillingham cattle are an extreme example, but racehorses, too, are bred from a relatively narrow genetic pool. Almost all of the world's half a million thoroughbreds are descendants from just 28 healthy, fit and fast ancestors born in the 18th and 19th centuries. And up to 95% can be traced to just one stallion – the Darley Arabian, born in 1700. Centuries of selection for good hearts, healthy respiratory systems and muscle strength has created a genetic pool that is shallow, but not unhealthily so. Harrison's studies have shown that more inbred horses tend to have the speed and strength that makes them good sprinters. However, high levels of inbreeding in thoroughbreds doesn't suit all race types. Outbred "mongrels" tend to be more robust and better at withstanding the tough training regimes needed for longer courses.

It was partly on the advice of Harrison that Brown bought Darysina and mated her with Frankel.

"From within a roughly appropriate economic group, we gave Philip a list of stallions that we thought would suit Darysina," says Harrison. "Frankel actually came out top because he most suited the profile required for the inbreeding level of Darysina and which might most likely enhance 'genetic fixation' within the correct stamina zone and reduce mongrelism."

Australian Olympic gold medallist and his mare Panamera in a HIDEZ compression suit ahead of London's Olympic Games 2012. Photograph: Aude Reubrecht Tinney Eventi/Rex Features

Decisions about breeding matter because racing is big business. It's the second most popular spectator sport with more than six million racegoers each year. According to the British Horse Industry Confederation, it employs about 90,000 people directly and indirectly and generates £3.7bn a year for the economy. Perhaps surprisingly, some horses that never win a race can produce exceptional offspring – if they are bred with the right stock.

One of Harrison's greatest successes is Sacred Choice, bred by Ken Williams's Tarcoola Stud in Victoria, Australia, which had nine wins from 37 starts including the Group One Doncaster Handicap at Randwick in Sydney and the Group One Myer Classic at Flemington in Melbourne – two of Australia's big one-mile races. Yet on paper its mother, Sacred Habit, wasn't up to much. "Sacred Habit was sold because it was a rusty animal," says Harrison. "And yet it bred this multiple group-one winner."

The genetic profile suggested that the mother had potential but was too inbred. On Harrison's advice, she was bred with a stallion called Choisir to produce an offspring with the right balance of strength and stamina and increased mongrelism. The outbreeding did the trick and Sacred Choice became one of the great Australian milers of recent times.

When talking to enthusiastic equine geneticists such a Harrison, it's easy to forget that theirs is a brand new branch of the horseracing industry. The horse genome's 32 pairs of chromosomes, written in 2.7 billion base pairs of DNA, were sequenced and published only in 2009 and discoveries are being made all the time about genetic links to performance.

One of the leading lights in equine genetics, Dr Emmeline Hill at University College Dublin, was involved in unravelling the genome. In 2010 she discovered that a single gene was crucial in determining the most favourable racing distance for a horse. It was a remarkable discovery because single gene disorders or physiological traits are unusual. While the ability to roll your tongue is caused by a single gene, as is a widow's peak, the stickiness of your ear wax, and having a big toe that's shorter than your second toe, more complex characteristics such as intelligence, prowess at sport, musical ability or ability to hurdle are the result of complex interactions between dozens of genes.

Yet despite initial scepticism, repeated studies published in scientific journals have supported Hill's findings. Today her spin-out company, Dublin-based Equinome, offers three tests to owners and trainers, including one based on the single "speed" gene. Hill's big discovery was that variations in the thoroughbred horse mysostatin gene (MSTN), which encodes for a protein that regulates muscle development and muscle fibre type, determines the type of race that would most suit an animal and whether or not it will be an early developer.

DNA is written in four chemical "letters": A, C, T and G. A crucial point of the speed gene contains either the DNA letter C or the letter T. Every horse has two copies of the MSTN gene – one inherited from the mother, the other from the father – so there are three possible genetic combinations: C/C, C/T or T/T. Hill showed that the C/C gene variant leads to greater muscle development when horses first go into training. Creatures with this version were more suited to sprint distances, with 98% having a best race distance of one mile or less. They are the equine equivalent of Usain Bolt and typically have 7% more muscle mass than the other two types at age two. In contrast, animals with the T/T variant produce more of the inhibiting MSTN protein and tend to be leaner and less muscular, making them better suited to longer distance races such as the Derby or Ascot Gold Cup. Those with the C/T version of the gene are the most versatile and are best suited to middle distance races, she showed. "It was the first time that anyone had identified a single gene to an athletic trait in a thoroughbred," says Donal Ryan, managing director of Equinome. "It's quite astonishing that a single gene has such significance, but it does."

The company offers a broader "elite performance" test that looks at clusters of genetic markers, including those involved in the regulation of glucose and the respiratory system, while another single gene test gives an indicator of height. The company says this is accurate to within one inch 70% of the time.

Thilo Pfau, a computer scientist who has developed an inertial sensor-based gait analysis system to check for lameness in racehorses. Photograph: Richard Saker for the Observer

Like its British competitor Thoroughbred Genetics, Equinome faced scepticism in the industry from owners and trainers who favoured conventional approaches to breeding. So Hill looked at the price paid for 200 yearlings and compared it to their performance. She found that buyers were paying too much for poor quality horses and that Equinome's single gene and multi-gene tests were better at predicting success.

"Looking back too far in the pedigree becomes of limited value," says Ryan. "But what the direct parents did is of most importance. However, what an individual will inherit can still be a genetic lottery and so you try to load the dice by having more information about the parents' DNA."

Despite the enthusiasm of equine geneticists, genetically perfect animals can still run like donkeys. Training, nutrition, the jockey and the track conditions can be more important on race day; geneticists estimate that DNA probably accounts for 30-35% of a horse's performance. So to help load the dice a little more, trainers are turning to people such as Dr Thilo Pfau, a computer scientist and expert in horse gait and lameness at the Royal Veterinary College. A former expert in speech recognition, Pfau makes an unlikely horse boffin. But his skills of identifying patterns have proved useful in spotting the causes of lameness. Racehorses put extraordinary pressures on their legs and muscles. When galloping, each limb has to support the equivalent of two and half times the animal's body weight. In a 500kg animal, that's 1.25 tonnes with every stride.

With those demands, and with rigorous training regimes, lameness is an industry-wide problem and one that is notoriously tricky to treat. A trainer or vet will watch a trotting horse to try to find the cause of lameness. That may sound straightforward, but it's not. "The human eye is pretty bad at seeing these mild asymmetries we are looking for," says Pfau.

High-speed cameras may help to identify the problems but they are expensive and you need a lot to monitor a horse in different situations. Pfau's solution was to take motion sensors used in Hollywood animated movies to mimic human behaviour and attach them to the head, pelvis and hips of horses. The matchbox-sized sensors fed data wirelessly to a computer that analysed the gait. The software didn't just identify the cause of lameness; it also revealed if it was getting better with treatment.

That can be a real problem in veterinary medicine. Horses don't experience the placebo effect. But vets do. Studies have shown that vets are likely to see an improvement in an animal after treatment even if the behaviour and symptom are the same. Sensors and software don't suffer from the same bias.

Australia's Commonwealth Scientific and Industrial Research Organisation (CSIRO) has created a set of bespoke titanium horseshoes for a Melbourne race horse using 3D printing. Photograph: CSIRO

The next phase of Pfau's work is to detect problems before they become serious. Working with the Singapore Turf Club, his team is starting a project to put sensors on 200 racehorses over 18 months. "We want to spot things going wrong early on," he says. "We can build up a database and we know that a lot will develop injuries over time. If a horse does, we can go back and see what was happening and if there are any early signs."

The sensors are also being used to train jockeys. The biggest single improvement in racing occurred not with the advent of modern medicine, or nutrition, or transport or improved quality of courses, but when jockeys started to stand up. Until the start of the 20th century, British jockeys sat upright on their steeds. But in America, jockeys were experimenting. They discovered that if they put in a bit more effort themselves, and stood in their stirrups, bending forward and moving with the horse, they went faster. In the last decade of the 19th century, the speed of horses shot up by 5-7%. By 1910, British jockeys were doing the same.

To investigate why this "crouch" had such a huge effect, Pfau put his sensors on jockeys' belts and on the front of saddles. In a 2009 paper in Science he showed that a horse's back went up and down around six inches with each stride – and around four inches at the fore and aft. Crouching jockeys lifted up and down around 2.3 inches. That lift made a huge difference.

A jockey makes up around 13% of the total load a horse must carry, so by absorbing some of the movement of the horse by standing in the stirrups, jockeys save the horse work. Pfau is now working with the British Racing School in Newmarket to attach sensors on trainee and experienced jockeys to contrast their riding styles. "The feedback from the sensors should lead to programmes that tell jockeys what they are doing wrong. It's halfway through a two-year project."

Back in the sunshine of Watership Down, where rabbits loiter in the paddocks, Brown is contented with the progress of his thoroughbred's pregnancy. Horse owners are philosophical creatures and most are used to losing. They recognise that failure is more common than victory. The stud's general manager, Simon Marsh, a smartly spoken horse guru who has worked for the Lloyd Webber family since 1992, retains a healthy scepticism about genetics and technology when it comes to breeding. "The complete imponderable here is that you are dealing with livehreestock – with living, breathing animals," he says, as he drives Brown away from his inspection of Darysina in a 4x4.

"You can predict, if you are on the front row of the grid in a Formula One race that you are probably going to win the grand prix. But if you have the greatest stallion in the world and the best mare in the world, there's no reason why their progeny won't be beaten by something that cost just 10% of the price."