A startling painting in this terrific survey of the American artist Dorothea Tanning (1910-2012) shows dinner time in some conventional midwestern home. The paterfamilias, with blinding spectacles and a hideous orange tie, towers above the proceedings. His daughter is much smaller; tinier still is a cook bearing a dish, who is not much bigger than the family dog. Each is portrayed according to their power. The daughter shoots us an urgent glance: home life really is as bad as it looks.

This tragicomic tableau is one of many surprises unleashed at Tate Modern. If all one knows of Tanning is the conventional tale – the girl from Illinois who made it to Paris, married Max Ernst and became the last surrealist, still at work in the 21st century – the show will be nonstop revelation. It gives us the artist whole, from the early self-portraits to the marvellous late visions of unbridled joy, and reveals her as extraordinarily original, quizzical and humorous, with a profound talent for probing the quick of our psyches.

Take her most famous work, in this country at least: Tate Modern’s own Eine Kleine Nachtmusik from 1943, with its peppily ironic title. Two little girls in a hotel corridor by night are engaged in some frightening skirmish with a gigantic sunflower. One has given up, exhausted, a single petal in her hand; the other is still standing, hair rushing up to the roof as if electrified. She can’t get down the stairs for this tentacular monster, nor to the open bedroom door with its promising light. Whatever else the painting stirs, not least is this appallingly familiar sense of entanglement.

Family Portrait, 1954 by Dorothea Tanning. Photograph: DACS, 2018

Doors open on to doors in the 1942 self-portrait Birthday, where Tanning appears bare-breasted in a skirt of tangled roots, a chimerical critter at her feet. She may be lingering on the threshold, hand on doorknob, but you know the artist is already well into Wonderland (Lewis Carroll was a childhood hero), since she has conjured this vision. Yet the scene pivots upon her expression of deadpan determination in the face of so much uncertainty: who or what is beyond all these doors?

The title suggests a brand new self, but even when Tanning depicts herself she is effectively portraying others. Two years later, after moving to Arizona with Ernst, she painted a woman in a bathing costume standing on a cliff, dwarfed by the immense desert before her. You know it’s Tanning from the trademark updo, even though she is seen from behind; and yet it is every viewer too, vulnerable, speechless before life and landscape, overwhelmed with oppressive awe.

The bathing costume is dissonant, comical; the female figure Tanning’s lifelong subject. She painted women running, twisting, fighting, giving birth, taking up the tango, making love. It is hard to think of another American painter before Tanning who so much wanted to free women from the prison house of depiction. And this aim was enormously advanced by her sudden decision to give up the meticulous, overdetermined house style of surrealism, in the 1950s, for what she called her “Insomnia” paintings – gorgeously fluent canvases in which bodies, faces and curious biomorphic forms teem in diaphanously cloud-like spaces.

Two sisters are locked in some kind of love-hate tumult. A girl fights back despite tumbling between musical chairs. A trio of grey and white presences, huddled around a table, are turning into whispering ghosts. Most sinister, and yet delicate, is a painting of a couple apparently airborne with ecstasy, floating round the stem of a giant white poppy in their sightless embrace: narcotised with love or drugs?

And then Tanning changes again, giving up paint for fabric in the 60s. Halfway through this show, out of nowhere, comes one of her soft sculptures – a reclining nude, all abject and hunched, made out of pink stuffed cloth. Another nude, arching backwards, is hugged by a brown fur form. Go wild, or remain unfree; that seems to be a central imperative. And Tanning’s art is filled with dreams of escape: from the constraints of marriage, motherhood, convention, the tyranny of the dinner table. Emma, her staggering take on Flaubert’s Madame Bovary, is reduced to nothing more than a pregnant cushion of soft pink belly weighed down with a heavy fringe of Victorian lace.

Birthday, 1942 by Dorothea Tanning. Photograph: DACS, 2018

These sculptures were without precedent. Louise Bourgeois was not yet working with fabric, and it would be another 30 years before Sarah Lucas began making her own stuffed nudes. And they have their climax in another pioneering work – an alarming set-cum-sculpture called Hôtel du Pavot, Chambre 202, a very early instance of installation art dating back to 1970.

The hotel room is disturbing enough, with its filthy subfusc gloom and its naked lightbulb. But bursting through the wallpaper, and out of the fireplace, are bulging female forms – flailing legs and rounded stomachs, like the relics of some Walter Sickert nightmare. It seems to be pure gothic horror, except for that characteristic humour which adds a quirky new limb to a battered chair and makes a tragic heroine of a humdrum table.

All of these works turn out to be connected in the end – the theatre of doors, rooms and tables; the independent girls with their fantastical adventures; the scenes of mystery, entrapment and joy, particularly in the late paintings of flowers, radiant and flagrantly vast. They all come from an imagination so rebellious and unbound it lifts the viewer straight into catharsis.

Dorothea Tanning has all the inventiveness Tracey Emin lacks. A Fortnight of Tears – typically miserabilist title – is an enormous yet flimsy exhibition, a monotone lament in 100 works. It opens with magnified iPhone shots of an insomniac Emin – exhausted, with a cold sore, with plastic surgery scars, and, it is to be supposed, enduring all the emotions replayed throughout the rest of the show in paintings, neons and gigantic sculptures: the agony of solitude, childlessness, of being herself without love.

The many large painted nudes in this show, frontal, legs splayed, face scrambled, appear both slick and unfulfilled; inflations of smaller paintings and drawings from the past. At 55, Emin has been running this particular hybrid of expressionism and ab ex, Munch to De Kooning, for too long. Her titles are a forte – I Don’t Know Who I Love, In the Dead Dark of Night I Wanted You – but it sometimes feels as if she might just as well miss out the image. Which indeed she does; with the neon slogans, of course, but also some of the paintings. It’s a Crime Not to Love the One You’re With consists of nothing but that tendentious claim lettered in dripping paint across the canvas.

Sometimes There is No Reason, 2018, by Tracey Emin. Photograph: © Tracey Emin/Courtesy White Cube

There are some compelling new departures – tiny painted figures, ghostly and timeless, like latter-day figments out of Goya. They have a life of their own, delicate but fierce, characters in some evolving fable. But they expose, all too clearly, the mistake of scaling up small figures into the muckle bronze nudes that are by far the worst works in this show. The sincerity of Emin’s misery may not be in doubt, yet it is undermined these days by her relentless insistence upon it. No variation, or exploration, only repetitive and grandiloquent enlargement.

Star ratings (out of five):

Dorothea Tanning ★★★★★

Tracey Emin ★★

• Dorothea Tanning is at Tate Modern, London, until 9 June

• Tracey Emin: A Fortnight of Tears is at White Cube Bermondsey, London, until 7 April