In the public mind, forensic science means cutting-edge technology: well-equipped professionals performing complex experiments in glossy laboratories, as in CSI or Silent Witness. In fact, the real story of forensics is full of courtroom disasters, eccentric pioneers, crowd-pleasing showmen and dangerous (sometimes fatal) research.

For years, authorities and individuals have known that there was more to crime than just testimony: that the scene of the crime, or a murder weapon, or even a few drops of blood, could also bear witness to the truth. The first recorded use of forensics in the solution of a crime comes from a Chinese handbook for coroners called The Washing Away of Wrongs, produced in 1247. One of the many case studies it contains follows the investigation of a roadside stabbing. The coroner examined the slashes on the victim's body, then tested an assortment of blades on a cow carcass. He concluded that the murder weapon was a sickle. But knowing what caused the wounds was a long way from identifying whose hand had wielded the blade, so he turned to possible motives. The victim's possessions were intact, which ruled out robbery. According to his widow, he had no enemies. The best lead was the revelation that the victim hadn't been able to satisfy a man who had recently demanded the repayment of a debt.

The coroner accused the moneylender, who denied the charge. But, tenacious as any TV detective, he ordered all 70 adults in the neighbourhood to stand in a line, their sickles at their feet. There were no visible traces of blood on any of the sickles. But within seconds a fly landed enthusiastically on the moneylender's blade, attracted by minute traces of blood. A second fly followed, then another. When confronted again by the coroner, the moneylender gave a full confession. He'd tried to clean his blade, but his attempt at concealment had been foiled by the insect informers humming quietly at his feet.

Buck Ruxton

The Washing Away of Wrongs was being carried to crime scenes by Chinese officials as recently as the last century. The west took longer to recognise the forensic benefits of entomology. But in Britain in 1935, scientists would rely on insects brought to the crime scene to help solve one of the most notorious murders in British criminal history. The case was a sensation, filling column after column of newsprint.

On 29 September, two women were walking across a bridge over a ravine near Moffat on the road from Carlisle to Edinburgh when, horrified, they realised they were looking at a human arm sticking up from the bank of the stream below. When the police arrived on the scene they found 30 bloody packages containing body parts wrapped in newspaper; 70 parts were eventually recovered, from two different corpses. They had almost certainly been butchered to prevent identification – the fingertips had been cut off – and the job had been done by someone who knew about human anatomy.

Some maggots were found feeding on the decomposing parts, and were sent off to the University of Edinburgh. There, the entomologists identified them as a particular kind of blowfly, and used the insect evidence to narrow down the time since the body parts had been dumped to between 10 and 12 days. The time span suggested a possible identity for the corpses. Isabella Ruxton, the wife of Buck Ruxton, a popular local doctor, had recently disappeared, along with their 19-year-old maid, Mary Rogerson. Ruxton maintained Isabella had run off with a lover, and that it had not been a happy household: Ruxton constantly accused his wife of infidelity, provoking blazing rows.

Some of the body parts had been wrapped in a special pullout section of the Sunday Graphic newspaper, distributed only in the Lancaster/Morecambe area. Some were wrapped in clothes belonging to the Ruxton children. Then, a cyclist came forward: he'd been knocked down by a car on the same day, and had scribbled down the number plate. It belonged to Ruxton's car. And the stream had flooded on 19 September, carrying some of the body parts high up on to the banks: so the packages must have been dumped before then.

Ruxton was found guilty of the so-called "Jigsaw Murders" and was hanged at Strangeways prison in Manchester; it is most likely that he strangled his wife. The maid died from having her throat cut, probably to silence her after she discovered his crime. The insect evidence was just one tile in a mosaic that spelled out the murderer's guilt. But the success of the combination of methods used in the case led to increased public and professional trust in the capabilities of forensic science. Even if Ruxton had wrapped the diced parts of his victims in bags rather than sections of the local paper, even if his car hadn't hit a bicycle, even if the stream hadn't burst its banks, the maggots would still have pointed to him.

Leutgart Murder Case

When they were first developed, techniques such as fingerprinting and forensic anthropology were often regarded as dangerously unreliable pseudoscience. It often took a major court case for them to prove their worth. For forensic anthropology – now part of crime-scene investigations of everything from paedophile rings to genocides – the break came in the form of the Leutgert Murder Case.

Adolph Leutgert had emigrated to Chicago from Germany as a penniless 21-year-old in 1866. For 15 years he worked odd jobs at tanneries and removal companies; eventually he raised enough money to build a factory and he set up the AL Leutgert Sausage & Packing Company. Sausages from the factory were soon distributed all over the city and beyond, earning Leutgert the title of "The Sausage King of Chicago".

On 1 May 1897 he went out for a stroll with his wife, the petite and attractive Louisa. But though an eyewitness told them that he'd seen them entering the sausage factory at 10.30, only Adolph returned home. Alerted by Louisa's family, the police soon arrived at the grounds, where they noticed a peculiar smell coming from a large vat used for steaming sausages. Peering into the vat, the officers noticed sludge at the bottom, which one described as having a "very sickening" smell, "something dead around it'. In the sludge, they found a wedding ring, and another ring engraved with "LL". Inside a nearby furnace, they found some small pieces of what looked like bone, and a piece of burned corset.

As the media went wild and sausage sales plummeted across America, Leutgert was put on trial at the Cook County courthouse in the summer of the same year. And an anthropologist – George Dorsey – was called in to testify that the bones found in the furnace were human, and that they included bones from the foot, finger, ribcage, toe and skull of a woman.

The first trial produced a hung jury; the jurors were so far from agreement that they nearly came to blows in the deliberation room. The following year a retrial was held. Dorsey testified again. And this time Leutgert was found guilty of murdering his wife. Away from the courtroom, Dorsey faced criticism from other anatomists, who jeered at his "identifying a woman from four fragments of bone the size of peas" - that he abandoned forensics. But the press coverage had put forensic anthropology on the map.

Mary Ann Cotton

The 19th-century fear of arsenic poisoning first brought toxicology to the forefront of forensic investigation. The French had such trouble with arsenic that they dubbed it poudre de succession (inheritance powder). In England and Wales, there were 98 criminal poisoning trials between 1840 and 1850. Although a reliable test – the Marsh test – had been established in 1838, arsenic was still often used. It was odourless, virtually tasteless – some said it had a slightly sweetish taste – and cheaply available from all manner of shops. The body cannot excrete it, so the heavy, metallic element builds up in the victim's system, mimicking the slow deterioration of a natural disease. Those who digest it suffer a range of symptoms with varying degrees of severity: hypersalivation, abdominal pain, vomiting, diarrhoea, dehydration and jaundice can all be a result of arsenical poisoning, often leading doctors to diagnose cholera, dysentery and gastric fever. Intelligent arsenic killers went down the chronic rather than the acute path of administration.

When she was 19, Mary Ann Cotton fell pregnant to a miner called William Mowbray and together they travelled the country looking for work. She gave birth to five children during this time, but four of them died, possibly from natural causes. In 1856, the couple moved back north, where she had three more children, all of whom died of diarrhoea. Her grief didn't prevent her from claiming on the life insurance policies she'd taken out on each of them. Then Mowbray injured his foot in a pit accident and had to convalesce at home. Soon he became ill and was diagnosed with "gastric fever"; he died in January 1865. Mary Ann went down to the office of the Prudential Insurance Company and collected the £30 policy which she had recently encouraged him to take out.

Over the next dozen years, Cotton became the most prolific female serial killer in British history. In 1872, she set her sights on Richard Quick-Mann, a wealthy customs and excise officer. Only her seven-year-old stepson, Charles Cotton, stood in the way. She tried fostering him with one of his uncles but failed. Then she took him to the local workhouse; the superintendent refused him entry unless Cotton accompanied him. All other options having failed, she poisoned Charles. The workhouse superintendent heard about his death and went to the police. The doctor who had attended Charles before he died carried out an autopsy and found no evidence of poison. So the coroner ruled death by natural causes. But the doctor had kept Charles's stomach and intestines and, when he tested them, he discovered the lethal poison.

The bodies of Mary Ann's most recent victims were exhumed and found to contain high levels of arsenic. Under the weight of this evidence and other witness statements, she was found guilty of murder and sentenced to death. The story ran in the newspapers for months, and a rhyme was coined that began "Mary Ann Cotton – she's dead and she's rotten". The press knew that their Victorian readers were fascinated by the figure of the female poisoner, exuding loveliness and sweetness, offering her husband a second spoonful of sugar for his tea and then making it a lethal one. More than 90% of convicted spouse murderers in 19th-century Britain were men. But men were far more likely to stab or strangle their wives; twice as many wives as husbands stood trial for poisoning.

Brides in the Bath

By the mid-20th century, forensic scientists had a much higher profile, and some were almost celebrities in their own right. Sir Bernard Spilsbury, the eminent pathologist, was held in reverent esteem by the police and courts. Arthur Conan Doyle, who was somewhat suspicious of Spilsbury, remarked of "the more than papal infallibility with which Sir Bernard is readily being invested by juries".

Spilsbury's most famous case was that of Dr Crippen, the London pharmacist accused of poisoning his wife Cora and burying her torso in the cellar of his home. But his unconventional methods were also vital in another extraordinary case that neither DNA testing nor any other modern forensic technique could have helped him solve.

On 3 January 1915, Charles Burnham, a Buckinghamshire fruit grower, sat down with a mug of tea and opened his copy of the News of the World. On page three, he found a headline "Dead in Bath: Bride's Tragic Fate on Day after Wedding" and a short report explaining that one Margaret Lloyd had been found dead at her flat in north London. Charles Burnham's daughter Alice had also died in a bath, shortly after her wedding almost exactly a year before. Burnham contacted the police, and discovered that Margaret Lloyd's husband was George Joseph Smith, who had previously been married to Alice Burnham.

The police called in Spilsbury to perform an autopsy on the exhumed body of Margaret. He then travelled to Blackpool to autopsy Alice. Following this, the police uncovered details of a third woman, Bessie Williams, who had married George Smith and died in similar circumstances at home in Kent in 1912. When the police investigated anew, they discovered that Smith had benefited financially from all of his wives' deaths, the largest amount from Williams, who had left him £2,500 in trust money (worth around £190,000 today). A pattern was emerging, and the police arrested Smith.

From the bodies of Margaret and Alice, Spilsbury could find no signs of violence, poison or heart attack, though the GP who had first seen Bessie's body noted that she had been clutching a bar of soap. He had all three bath tubs brought to Kentish Town police station, where he lined them up together and examined them minutely. He was particularly puzzled by the case of Bessie Williams. Shortly before her death Smith had taken her to see the doctor about epilepsy symptoms. Smith had told her that she was suffering from fits, even though she couldn't remember having them and had no epilepsy in her family. But Bessie was 5ft 7in tall and obese. The bath tub she had died in measured just five feet at its longest, and sloped at the head end. Spilsbury knew that the first phase of an epileptic fit causes complete rigidity of the body and that, given her size and the shape of the bath, such a fit would have raised Bessie's head above the water rather than brought it below.

Spilsbury researched further and learned that an extremely sudden rush of water into the nose and throat can inhibit a vital cranial nerve, the vagus nerve, and cause sudden loss of consciousness, swiftly followed by death. A common subsidiary result is instant rigor mortis – which Spilsbury thought explained the bar of soap clenched in Bessie's fist. Armed with this knowledge, the investigating officer, Detective Inspector Neil, decided to carry out a series of experiments before the trial. The first volunteer stepped into a full bath, clad in a bathing costume, and managed to grip the side of the bath and struggle. But when Neil grasped her ankles and abruptly pulled her legs up, she slid under the water and lost consciousness. It took a doctor several minutes to restore her to consciousness; she was lucky to live. But the police had their answer.

Smith, a predatory conman with a penchant for gold rings and brightly coloured bow ties, was tried for the murder of Williams. At the trial, Spilsbury spoke with great authority. The jury deliberated for 20 minutes before finding Smith guilty. Public interest in the "Brides in the Bath Murders" was intense. Scores of journalists, hungry for a "scientist foils serial killer" headline, doorstepped Spilsbury throughout the investigation.

The Phantom of Heilbronn

Although Smith was almost certainly guilty, Spilsbury's intense self-belief – and the implicit faith invested in him by the legal system – also led to several miscarriages of justice. Although juries these days might be less swayed by the word of a charismatic expert witness, they can still place much faith in disciplines that are subject to interpretation – such as fingerprinting. Similarly, recent developments have shaken the foundations of DNA profiling.

The strange case of the "Phantom of Heilbronn" involved a seemingly superhuman female serial killer whose DNA was found at the scene of robberies and murders across Austria, France and Germany in the 1990s and 2000s. Mitochondrial analysis of the DNA suggested it came from a woman of Russian or eastern European extraction, but she seemed to be involved in a bewildering variety of criminal activity, and to leave no other trace. In 2009, when the DNA appeared on the burned body of a male asylum seeker in Germany, the authorities concluded that the "phantom" was simply the result of laboratory contamination: the cotton swabs used for DNA collection were not certified for the purpose, and were eventually traced to the same factory, which employed several Eastern European women who fitted the DNA profile of the "phantom".

One expert I interviewed said: "DNA doesn't lie. It's an exceptionally good lead and exceptionally strong evidence, but there is human interaction in the process [of profiling]. So the error rate is exceptionally low but it's not zero … DNA shouldn't be a lazy way to not do an investigation."

Suzanne Pilley

The largest strides in forensics in recent years have been digital. One case that hinged on such evidence was the murder of Suzanne Pilley, who was last seen as she set off on her way to her job as a bookkeeper for a financial services company in central Edinburgh. Suspicion quickly fell on David Gilroy, a colleague of Suzanne's with whom she had recently ended an affair. The police interviewed Gilroy and noticed a cut on his forehead, subtle bruising on his chest and curved scratches on his hands, wrists and forearms. Gilroy said he had scratched himself while gardening. Forensic pathologist Nathaniel Cary would later examine photographs of these injuries and testify that they could have been made by another person's fingernails, possibly in a struggle

The police seized Gilroy's mobile phone and car, and specially trained cadaver dogs detected human remains or blood in the boot and in the footwell, though there had clearly been an attempt to clean the car. Police also found vegetation underneath the car and a damaged suspension. It was suspicious, but they needed more to make the case. A forensic digital analyst went to work on Gilroy's phone. When you turn your phone off, it keeps a record of the last phone mast it was connected to, so it can find it again quickly when you start using your phone again. On the day that they thought he had disposed of Suzanne's body, Gilroy had switched off his phone as he travelled between Stirling and Inveraray. Police suspected he had done this to avoid being tracked as he searched for a good place to dispose of Suzanne's body in the dense woodland. On his way back, Gilroy again switched off his mobile phone between Stirling and Inveraray. This, the police believed, was when he dumped the body.

Suzanne's body was never found. Nevertheless in March 2012, Gilroy was found guilty of murder and conspiracy to defeat the ends of justice. It's rare for murderers to be convicted in the absence of their victim's body. In the Gilroy case there was no DNA. The scratches on his arm would not have been enough. He was convicted because of unusual mobile phone activity, CCTV video and images from road-side cameras.

• This is an extract from Val McDermid's Forensics: The Anatomy of Crime is published by Profile at £18.99 on 2 October. To order a copy for £14.99, visit guardianbookshop.co.uk or call 0330 333 6846. A major forensics exhibition will be held at the Wellcome Collection, London NW1, in February 2015.