In the National Portrait Gallery in London at the start of this month, at the awards ceremony for the Taylor Wessing prize, a woman was standing with a tiny baby. That in itself was not unusual – there were probably three or four babies dotted around, and she was cradling it in the normal way, as if to support its head and not wake it. But it somehow didn't look right; it looked, in my peripheral vision, as if it wasn't moving enough. Anyway, while I Englishly darted looks at it without approaching, my friend did approach, and it wasn't real. Phew. Not ill, just inanimate.

It belonged to the photographer Rebecca Martinez, who had used it, and many others like it, for her project preTenders. And while I went over to look at it, and laughed, I felt resentful at being tricked. It had stirred some panic in me, something similar to that impotent distress you experience when you hear about a child being killed by an act of violence. Later, when telling me about the four years she has spent photographing people with these dolls, their collectors, their creators, her friends, a whole variety of subjects, Martinez said, "If I go out and I hold this doll in any way other than you would a real baby, people get mad. I cannot just hold it casually, like by one arm or whatever, because people will go, 'It's not right, you can't do that.' They go crazy. Even though the rational self knows it's a doll." But I'm with the mad people, because you don't start off knowing it's a doll; you start off thinking it's a baby. You can be disabused of your mistake but you can't, immediately, be disabused of your anger.

Reborns occupy a place that I think is unique in culture: to the artists who make them, they are works of art, and the artistry is undeniable. To some collectors, they are dolls, and to other collectors, they are something else altogether, a memory of a child or a substitute for a child. But it's possible to fall into neither of those categories, to be neither appreciating them as art, nor finding them cute as dolls, but nevertheless to respond to them in some profound way.

This is a relatively new phenomenon, springing up over the past six to seven years and spawning in its wake an entire industry that goes way beyond the making and selling of the dolls themselves into web forums, conferences, global export; generating ancillary industries, such as the provision of bespoke babywear. The dolls arrive as kits: vinyl "sculpts" made by specific people – some of them, such as Donna Rubert and Denise Pratt, are now big names in the business. Individual artists will then build on the basic structure, using layers of oil paint and various methods for the hair (a doll with painted hair will take a week to make, whereas a doll with real hair will take a month, since each strand needs to be individually rooted). They are weighted so they feel exactly like holding a baby, except that they're not warm. You can get quite crude ones on eBay for £100 but at their most expensive they can stretch to thousands of pounds (one was sold recently in the UK for around £11,000).

Martinez is full of stories about the way people react to a Reborn doll – the people who get freaked out and won't touch them, the people who seem to feel neutral towards them and yet start rocking them as if they were real, the men who play pranks with them. But before we consider the reactions of bystanders, the experiences of people who make and buy them are fascinating.

Claire Hughes and Min Li, two UK-based Reborn creators, are very upbeat and straightforward that this is an act of craft, with a burgeoning and busy market. Hughes remarks on the power of the dolls, but the vignettes she describes seem to underscore the fact that it's illusory: "My mum works in a care home with old people. If I take one of the dolls in, they love it. They think it's real, it calms them right down. The manager can't even look at them." She likens it to eccentric male hobbies – playing with train sets, or sitting for three hours by a riverbank, waiting to catch a fish.

Min Li has three boys, real ones, and started making baby girl dolls for her own enjoyment; she has since built up a market in China. "Most western babies have very thin hair and Chinese babies have lots of hair. They like that [thick-haired] kind of baby. So that's why I started doing it. Most people favour boys in their actual families," a hangover from the one-child policy, she says, "but," she adds feelingly, "people love girls."

The American artists I speak to, Cher Simnitt, Diana Mosquera and Gia Heath, seem less abashed, less inclined to forge an ironic distance. They describe the people who buy their dolls as more emotionally involved. Some people want a doll because what they really want is more children, but for practical or physical reasons can't have them; some want a doll made of their toddler, as the real child grows up, and they miss that physical sensation of the newborn; a family might commission a doll of their newborn to give to a grandparent, then, when the grandparent dies, it will pass back to the family "as a beautiful heirloom", Simnitt says. One woman who couldn't have children came to Heath and said, "Here's a picture of me, here's a picture of my husband, do you think you could make a baby that would look like us?" There's a story I find inexplicably moving about a wife who commissioned a doll of her husband, as a baby, then gave it to her mother-in-law. (What's the female for "uxorious"? And is there even a word for loving your mother-in-law that much?)

Then there are "portrait" or "memorial" babies, in which someone who has lost a child gets a doll commissioned in its image. Simnitt was, at one point, a midwife and a doll creator, and remembers, "I helped a woman who was 16 weeks pregnant. She came in and we got no heartbeat and she went on to miscarry. And she wanted to know what the baby looked like, but she was afraid to see it. So I had a model and I said, 'This is exactly what your baby looked like.' She carried that model for three weeks. And she said to me, 'I needed to grieve and hold something physical, and just work through it, and now I can let it go.' That's kind of drastic, I realise, but whatever gets you through."

What is more striking than these commemorative dolls, which are very rare, is the similarities between the artists. Before they started Reborn-creating full-time, they were often engaged in an intensely nurturing business, whether that was midwifery or art teaching for home-schooled kids; they had all been intensely nurturing people from a very young age – Simnitt cared for her mother, who was disabled by childhood polio, then went on to foster 125 drug-addicted babies and toddlers. Heath has adopted children; Mosquera had a typical experience as the oldest child of a large family: "I always took care of my sisters. When we had pets, I used to help with the breeding of the pets – there was always something being birthed around me." More importantly, both Simnitt and Heath suffered a tremendous loss just before they started making the dolls – Simnitt lost her mother, to whom she had been so close that, "literally, for 12 years, I was her body. When we ate, we had one plate, I took a bite, she took a bite, we bathed together. When someone passes away after having had a relationship like that, it's like something has been amputated from you. I would look at my hands and go, 'I don't know what to do. I don't know what to do with myself.' "

Heath, meanwhile, lost her baby daughter who was two months old, and says in a straightforward manner, "If I were to have a real daughter, I would love to have a daughter with green eyes and dark red hair and alabaster skin and freckles. I have my ideas, but when you go and put that on a doll, that's too much." It's almost as if they achieved this uncanny attention to detail as a product of their grief; that concentrating on something is a salve, but the focus of your concentration has to be a tightrope act, between reality and fantasy.

Martinez has observed the reactions these dolls get in many different scenarios, with friends and strangers, in different countries and cultures. "People say they want to hold the baby, then they get surprised, because the baby is made to feel as real as possible. Often, they'll start rocking the baby and cooing at it. And they'll realise what they're doing and they'll get embarrassed. They know on one level it's not real, and sometimes they're ashamed that they feel like that, that they've been fooled. It's something very deep and biological in people, something instinctive we have, that they're automatically comforting their baby. Some people are just delighted; they'll kiss the baby and not want to give it back. One time I had a man and he grabbed it and his body just tensed up, and he threw it on the ground. And I was upset, I said, 'Hey, that's a very expensive item, how dare you do that?' And he was so into what he was doing, he was so stiff, he wouldn't move for several minutes. He was trembling."

Martinez has observed wryly the stark differences between men's reactions in America and in Mexico – where American men will try to play some prank, to get a shocking reaction, Mexican men are much more nurturing and will kiss it and tend it, openly. She tells an extraordinary story about a time when she was burgled, in San Francisco: the boot of her car had been forced open, but nothing was stolen – she and the police surmised that the criminals had taken one look at the Reborns she had in there, concluded that they were real dead babies, and taken off. What was interesting was what happened next. "One of the officers said, 'I want you to photograph me with the baby.' So I said, 'What's your idea?' And he said, 'I want you to photograph me pointing a gun to the baby's head.' Even though it scared me a little – I'm afraid of guns – I thought, what a great photo this would be. I went to get a baby and in the couple of minutes I was gone, he was obviously talked out of it by his partner. So instead I have a photograph of him nurturing the baby. A few months later I was in New York and I walked past two police officers posing with tourists. So I went up and started talking, and one of the officers said, 'I have an idea' and said exactly the same thing, 'I want to be pointing a gun at the baby's head.' It was fascinating to me that these two police officers, 3,000 miles apart, both had the same idea."

It's funny because it's the grand impact images, the ones that fuel revulsion, that shock me the least; I can imagine how someone could look at a perfect simulacrum of a newborn and say, "I know, I'll pretend to eat it, or blow smoke on it, or smash its head against a wall." The Reborn-as-art is provocative, and you feel as if you should meet the provocation, that otherwise you're not up to its subversive standards. What I find compellingly mysterious, but simultaneously totally understandable, is the way people love them.