I have two different children, with two different fathers, neither of whom we live with and to both of whom we are close. I am not the typical single mother, but then there is no typical single mother any more than there is a typical mother. In fact, it's our unbridled fantasies and crude stereotypes of this "typical single mother" (overweight, short-tempered, popping out babies so that she can snare a council flat) that get in the way of our truly apprehending the richness and variety of thriving families.

The structure of my household is messy, bohemian, warm. If there is anything that currently oppresses the children, it is the idea of the way families are "supposed to be", an idea pushed in picture books, in classrooms, in adults' casual conversation, on children at a surprisingly early age, with surprising aggressiveness.

When I was pregnant, someone who was trying to persuade me not to have the baby said I should wait and have a "regular baby". What he meant, of course, was that I should wait and have a baby in more regular circumstances. But by this point I had seen the feet of the baby on a sonogram, and while he was pacing through my living room making this point, I was thinking: "This is a regular baby." His comment stayed with me, though. It evoked the word "bastard" – "Something that is spurious, irregular, inferior or of questionable origin."

Someone said something similar to a friend of mine when she found out that she was pregnant with the child of a man married to someone else. He said she should wait and have a "real baby". And someone else referred to the children her baby's father had with his wife as his "real children". As if her baby were unreal, a figment of her imagination, as if they could wish him away.

Small word choices, you might say. How could they possibly matter to any halfway healthy person? But it is in these casual remarks, these throwaway comments, these accidental bursts of honesty and flashes of discomfort that we create a cultural climate; it is here that the judgments persist and reproduce themselves, here that one feels the resistance, the static, the pent-up, residual, pervasive conservatism to which we do not generally own up. Nathaniel Hawthorne called it "the alchemy of quiet malice, by which [we] can concoct a subtle poison from ordinary trifles".

At lunch one day, I am talking to an editor about how I am thinking of writing about single mothers and the subtle and not-so-subtle forms our moralism towards them takes. He says, "That's a good idea. And I say that as a guy who looks at single women and thinks, 'What's wrong with her? How did she screw up?'"

One day, one of my colleagues, noticing that I was pregnant with my second child, ducked into my office and said, "You really do whatever you want." He meant this as some variety of compliment, and I took it as such, but I was beginning to get the sense that other people were looking at me and thinking the same thing: it seemed to some as if I were getting away with something, as if I were not paying the usual price, and if the usual price was takeaway Thai food and a video with your husband on a Saturday night, then I was not, in fact, paying that price. James Baldwin wrote, "He can face in your life only what he can face in his own." And I imagine, if you are feeling restless or thwarted in your marriage, if you have created an orderly, warm home for your child at a certain slight cost to your own freedom or momentum, you might look at me, or someone else like me, and think I am not making the usual sacrifices. (I may be making other sacrifices, but that is not part of this calculation or judgment.)

I am quite prepared to believe that in a house with two parents, there is generally a little more balance, a healthy divide between adult culture and child culture, a comfortable diffusion of affection. On my son Leo's first birthday, his seven-year-old sister Violet wrote him a poem that ended with the lines, "Even if you get a wife, I'll always be the love of your life." And when her father left when she was a little under three, she said, "Mama, it's like you and I are married." This would fairly accurately reflect the atmospherics of our house: a little too much love, you might tactfully say.

But I have to confess that I like the crazy intensity, the fierceness of the attachment, the too-muchness of it. In my heart of hearts, I don't really think "healthy" is better. I think there are some rogue advantages to the unhealthy, unbalanced environment, to the other way of doing things.

Which is not to gloss over the fact that being the only adult in a house with children can be really, really hard. There were times in the first few years of Leo's life where I wished the world would stop spinning on its axis, so I could step off and take a rest.

There is no doubt that single motherhood can be more difficult than other kinds of motherhood. In France, the response to that added difficulty is to give single mothers preferential access to excellent daycare. In the UK, the response seems to be to make alterations to the benefits, tax rates and childcare credits that compound that difficulty; and in the US, the response is moralism disguised as concern, and sometimes just plain moralism.

At the Republican convention last year, former presidential candidate Mitt Romney thought it would be an accurate assessment of reality to blame the rise in violent crime on single mothers. I would be tempted to think of this as a shimmering manifestation of American puritanism, but I notice it in the UK, too. Earlier this month, a study by the Centre for Social Justice, a rightwing thinktank, warned of a "tsunami" of family breakdown when it found that more than 1m children in the UK are growing up without a father at home. Then there are the furious attacks in newspapers on women who have children by more than one father; women such as Ulrika Jonsson, who was nicknamed "4 x 4" a few years ago, and Kate Winslet, criticised this month for announcing her third pregnancy with a third different father. The Telegraph berated her for "disastrous choices", asking, "Has bitter experience (your offspring's, if not yours) taught you nothing?" And continued: "The fallout for the little human beings you've brought into the world is too awful to contemplate." A cooler mind might wonder how the writer could possibly know what goes on in Winslet's children's minds, and whether they are not, in fact, thriving, but cooler minds don't worry as much about other people's private lives.

JK Rowling, one of the world's most spectacularly productive single mothers, addressed this sort of thinking in an article for the Times in 2010: "Women like me... were, according to popular myth, a prime cause of social breakdown, and in it for all we could get: free money, state-funded accommodation, an easy life." She went on to say, "Between 1993 and 1997, I did the job of two parents, qualified and then worked as a secondary school teacher, wrote one and a half novels and did the planning for a further five. For a while, I was clinically depressed. To be told, over and over again, that I was feckless, lazy – even immoral – did not help."

'People like to hear that you did not arrange your life in this chaotic way on purpose, that you're not enjoying yourself too much.' Photograph: Graham MacIndoe/The Guardian

The idea of "single mothers" may itself be the convenient fiction of a fundamentally conservative society. In fact, like Rowling, women move in and out of singleness, married parents break apart, couples live together without marrying, parents die, romantic attachments form and dissolve. Which is to say, the "us" and "them" tenor of the cultural conversation arises from prevailing fantasies of family life that bear no relation to life on the ground.

In spite of our exquisite tolerance for all kinds of lifestyles, we have a wildly outdated but strangely pervasive idea that single motherhood is worse for children, somehow a compromise, a flawed venture, a grave psychological blow to be overcome, our enlightened modern version of shame. It malingers, this idea; it affects us still.

I have noticed that single mothers, or mothers with children from different fathers, seem to do an awful lot of apologising ("I didn't do it on purpose"; "I thought he might stay"; "I think the baby is doing OK. Obviously it would be better if there were a more stable family situation…"). There is a sense that you have to explain yourself in a way that almost no one has to any more, because even the progressive world is operating on a pretty appalling, almost unthinking level of prejudice on this one particular issue. Specifically, people would like to hear that you did not arrange your life in this chaotic manner on purpose, and that you are not enjoying yourself too much, and that you realise your way of doing things is far inferior to the conventional way and possibly blazingly destructive to your children.

In America, a recent Pew poll on attitudes toward family structure showed that there is higher tolerance for gay couples raising children than there is for single mothers, with nearly seven in 10 Americans calling single motherhood a "bad thing for society". This in spite of the fact that two of the most popular presidents in recent memory, Obama and Clinton, were the sons of single mothers. And the fact that currently in America 53% of babies born to women under the age of 30 are born to single mothers; which is to say that most babies born to women under 30 are "bad for society". Our ideas about these things, to say the least, have not caught up with the way we are actually living.

To support the basic notion that single mothers are irresponsible and dangerous to the general order of things, people often like to refer darkly to "studies". To me, these sorts of studies are suspect because they tend to collapse the nuance of true, lived experience and because people lie to themselves and others. (One of these studies, for instance, in order to measure emotional distress, asks teenagers to record how many times in a week "you felt lonely". Is there a teenager on Earth who is a reliable narrator of her inner life? And can anyone of any age quantify how many times in a week they have felt lonely?)

However, studies such as those done by the Princeton sociologist Sara McLanahan, who is one of the foremost authorities on single motherhood and its effect on children, make the case that conditions such as poverty and instability, which frequently accompany single-mother households, increase the chances that the children involved will experience various troubles later in life. But there is no evidence that, without those conditions, the pure, pared-down state of single motherhood is itself harmful to children.

McLanahan's studies, and many like them, reveal that the main risks associated with single motherhood arise from financial insecurity, and to a lesser extent particular romantic patterns of the mother – namely introducing lots of boyfriends into children's lives. What the studies very clearly don't show is that longing for a married father at the breakfast table injures children.

And, of course, what these oft-quoted studies don't measure is what happens when there is simmering anger in the home, or unhappy or airless marriages, relationships wilting or faltering, subterranean tensions, what happens when everyone is bored.

In fact, as I learned when I talked to her, McLanahan's findings suggest that a two-parent, financially stable home with stress and conflict would be more destructive to children than a one-parent, financially stable home without stress and conflict. In other words, our notion that "studies show" a single-parent home is categorically worse for children is wrong.

By now, I have spent so long outside conventional family life that sometimes when I spend an afternoon with married friends and their children, their way of life seems exotic to me. The best way I can describe this is the feeling of being in a foreign country where you notice the bread is good and the coffee is excellent but you are not exactly thinking of giving it all up and living there.

When my son was two, he referred to his sister's father as "my Harry". He would say, "My Harry is coming!" It seems to me that this exuberant, unorthodox use of pronoun gets at the conjuring, the act of creation, the interesting magic trick at the centre of the whole venture: his family will be what he makes it.

I notice people often find little ways of telling me that this is not the real thing. But is it necessarily worse than the real thing? Is the physical presence of a man in the home truly as transfiguring, as magical, as necessary as people seem to think? One could argue that a well-loved child is a well-loved child. Many people have said to me over the years some variation of, "He needs a man in the house." But does he? It seems to me a little narrow-minded or overly literal to think that love has to come from two parents under one roof, like water from hot and cold taps.

When it comes time for Leo's class to study "families", I worry about how his three-year-old mind will process his family. I don't want him to feel like an outsider in a pre-school of married, heterosexual families. We've talked about how there are all different kinds of families, but his world does not reflect that conversation.

When it's time to put cutout silhouettes of family members on the wall, the other children in Leo's pre-school class put two parents and two, sometimes three children. Leo puts cutouts of himself, his sister, me, his father, his sister's father and his beloved babysitter, who has been with us since his sister was born 10 years ago. His teacher told me that when he did this, the other children started clamouring, "Wait, my babysitter is my family, too." "What about my grandfather? He takes care of me twice a week."

The wall got cluttered with rogue silhouettes, in bright colours, and I thought, we'll take our progress, paper cutout by paper cutout.

• In Praise Of Messy Lives, by Katie Roiphe, is published next month by Cannongate at £12.99. To pre-order a copy for £10.39, including UK mainland p&p, call 0330 333 6846 or go to guardian.co.uk/bookshop.