There was this period in the ’70s when the comedian and actor Andy Kaufman always kept a tape recorder close at hand. He’d record the moments of strange, often confrontational improvised street theater into which he liked to draw unsuspecting strangers. He’d record phone conversations with women he was seeing, often antagonizing them just to see what they’d say. He’d record himself talking to his grandmother. Sometimes he’d just set the recorder down in a crowded room, on the off chance that it would pick up something interesting, and capture long stretches of wordless ambience instead. When people asked him what he was doing, he told them he was making a record, and that they’d be on it. He amassed around 80 hours of taped material, but never got around to doing anything with it.

By the end of the ’70s, he was into his second season playing the immigrant mechanic Latka Gravas on Taxi. Latka was based on “Foreign Man,” a character he’d developed years earlier in his nightclub act. Foreign Man, who hailed from an island in the Caspian Sea, spoke with a thick, unplaceable accent and did jokes and impressions so ineptly you began to feel bad for him. Then he would announce that he wanted to im-ee-tehhht de Elvis Presley, and he’d turn around and do a costume change, and when he faced the crowd again, Kaufman would do his extraordinary Elvis impersonation, an Elvis impersonation even Elvis himself was said to have praised, and people who’d never seen the act before felt amazement and relief. Kaufman’s transformation into Elvis revealed that Foreign Man’s sad, sweaty failure, his abjection, had been just part of the show all along; Elvis released the audience from the discomfort Foreign Man had created, because if Elvis was the reality then Foreign Man was just a mask, and this was somehow comforting.

Latka was a version of Foreign Man that NBC ABC owned, and he never turned into Elvis, but sometimes he turned into a smooth-talking ladies’ man named Vic Ferrari. Kaufman stopped doing Foreign Man once he became Latka. He also stopped walking around with his tape recorder. Presumably the show’s success made it harder for him to mess with people in public without being recognized. He had to find new ways to provoke and confuse while incorporating his growing fame into the gag.

And he did. After he was famous, he took a night job as a busboy at the Posh Bagel restaurant on Santa Monica Boulevard, just to see what would happen. And in the last few years of his life he poured most of his creative energy into a quixotic second career as a bad-guy pro wrestler. It was about him feeling hemmed-in by Latka and by the show, which was a good sitcom but also a constrictive space for a restless talent. “Because he had embraced a [mainstream comedy] world with so much sugar in it,” Lorne Michaels once suggested in a GQ oral history of Kaufman’s life, “his comedy became much more astringent. And, towards the end, sour.” He wrestled mostly female volunteers, because he couldn’t wrestle that well, and because he liked wrestling women. He’d taunt them with sexist trash talk first. He’d say women belonged in the kitchen, that they had nothing but oatmeal in their heads.

Eventually, having awarded himself the title of Inter-Gender Wrestling Champion of the World, he embarked on an elaborate feud with Jerry Lawler, a goateed Memphis grappler who happened to also be the greatest straight man a performer like Kaufman could ask for. Kaufman had experimented with willful audience-alienation before, via his alter ego, the sleazy and abrasive nightclub entertainer Tony Clifton. But playing Tony involved putting on makeup and padding and a pink tux jacket. In his wrestling career, Kaufman was playing himself, as a nasty, preening heel. He insulted his opponents, but he also insulted the mostly Southern audiences in front of which he wrestled. “I’m from Hollywood, where they make movies and TV shows,” he would say, acting as if they were lucky to have him there, this famous television comedian marching around the ring in grubby sweats, calling them dumb rednecks.

And Kaufman did all this without ever breaking character, so it was hard to tell if the wrestling was the crowning achievement of his life as a performer or the symptom of a mental breakdown. (There were rumors he had a brain tumor.) People thought he was crazy, and after Taxi went off the air in 1983 no one else in show business wanted to hire him. Which was fine. He talked about giving up comedy entirely to manage wrestlers full-time. Before that could happen, though, he was diagnosed with a rare form of lung cancer. He died in a West Hollywood hospital on May 16, 1984, although theories to the contrary persist, which is what happens when you spend your whole life putting people on.

“I think the latest is he’s living in Albuquerque,” Lynne Margulies Osgood told me last week. “Andy would love that. He’d be so happy the rumors are still out there.”

Osgood was Kaufman’s last girlfriend. They met in 1982 when Osgood, as Lynne Margulies, played a small role in the low-budget feature My Breakfast With Blassie, a mostly improvised My Dinner With Andre parody in which Kaufman eats and talks with the pro wrestler “Classy” Freddie Blassie at a Sambo’s coffee shop in Los Angeles.

They lived together, and after Kaufman died, Osgood — now an artist and teacher who lives on the Oregon coast — held on to his stuff, including the tapes he’d made in the ’70s. In 2009, she published a book of letters written to Kaufman by women who wanted to wrestle him, titled Dear Andy Kaufman: I Hate Your Guts!; through that book’s publisher, Process Media’s Jodi Wille, she met Dan Koretzy, cofounder of the Chicago indie-rock label Drag City. Osgood sat with Koretzky at a Starbucks in Los Angeles and played him some of the tapes. This week, Drag City and Process Media jointly released the first-ever Andy Kaufman comedy album, Andy and His Grandmother, a collection of bits culled from Kaufman’s cassette archives by writer/producer Vernon Chatman and Rodney Ascher, the director of the Stanley Kubrick conspiracy-theory documentary Room 237. The plummy, solemn Bill Kurtis–esque narration is by Saturday Night Live alum Bill Hader; Kaufman’s friend and creative coconspirator Bob Zmuda contributes liner notes.

Posthumously assembled albums of any kind tend to be a crapshoot, even with confidants and superfans in the mix, and comedy albums don’t always capture that which is remarkable about the comics who make them. Plus, pure audio doesn’t seem like the optimal delivery system for a performer like Kaufman, whose act was so visual and televisual and depended so much on gestures and the look on Kaufman’s placid David Berkowitz face. And yet Andy and His Grandmother is a landmark. It passes the basic comedy-album test in that it’s often quite funny. At one point, Andy chats up some hookers from his car; when they offer him a date, he suggests bowling or roller skating, and when they realize he’s just goofing around and start to walk away, he calls after them, “What kind of work do you do?” But as always with Kaufman’s work, the jokes aren’t the most important thing about it. The most important thing about it is Kaufman. You don’t come away from the record feeling like you know him, necessarily, but you feel like you’ve actually met him for the first time. Turns out he’s weird.

In 1992, R.E.M. released their eighth album, Automatic for the People, featuring “Man on the Moon,” a bouncy, maudlin elegy for Kaufman that peaked at no. 4 on the Billboard rock charts the following year. They made a video for it, cross-fading footage of Kaufman in the ring with shots of Michael Stipe wearing a cowboy hat. I knew who Kaufman was by the time this song came out; as a devoted fan of Nick at Nite’s abbreviated early-season Saturday Night Live reruns, I’d probably seen him do the Mighty Mouse bit 15 times without really processing what was funny about it. I wanted to see the Nerds or the Samurai and learn the meaning of the catchphrases I’d heard repeated so many times. You couldn’t quote Kaufman’s stuff at recess. “Man on the Moon” was my first clue regarding the existence and foundations of the Kaufman mythos, although if I’m being honest I didn’t even key into that until later. I was 14 and the idea that sad songs could be about something other than girls not liking me was something I was still wrapping my head around. I also didn’t recognize back then that the best famous-person song on Automatic for the People is the one about Montgomery Clift.

When director Milos Forman made a rote Kaufman biopic in 1999, it was also called Man on the Moon. That same year saw the publication of Esquire writer Bill Zehme’s propulsive, zingy, skitteringly insightful Lost in the Funhouse: The Life and Mind of Andy Kaufman as well as Zmuda’s own tell-some memoir, Andy Kaufman Revealed: Best Friend Tells All!, both of which were more nuanced portrayals of their slippery subject. But big-screen biopics tend to create notions of their subjects that are tough to shake, especially if their protagonists aren’t alive to shake them. The movie saw the glint of calculated mania in Andy’s eyes and mistook it for a twinkle. Jim Carrey was part of the problem. He simulated Kaufman’s greatest hits with an almost eerie virtuosity, but you were still watching one of the neediest comic actors in film impersonating one of the least-needy comics of all time.

Between “Man on the Moon” and Man on the Moon, Kaufman began settling into the memory of the culture as a too-good-for-this-world meta-martyr, a spacecake genius man-child. Even if it’s still unclear what exactly he was, it’s clear he was more than that. Andy and His Grandmother reopens Kaufman’s case, giving him back his ability to discomfit and befuddle. He seems like a real person again, and real people are discomfiting and confusing.

One of the first things Vernon Chatman bought on eBay, years ago, was a bootleg VHS compilation of Andy Kaufman’s appearances on shows like The Midnight Special and The Merv Griffin Show. To this day, he says, “I can watch him sing ‘I Trusted You’ on YouTube over and over again, and it’s so goddamn funny. He just repeats [the title phrase] over and over again with such fury and passion. This was, like, the late ’70s, early ’80s, and it’s one of the most punk-rock things ever done on TV.”

Chatman went on to write for Conan O’Brien, The Chris Rock Show, and South Park; he cocreated the depraved children’s-television send-up Wonder Showzen for MTV2, as well as Adult Swim’s Xavier: Renegade Angel and The Heart, She Holler. He’s also the voice of South Park‘s Towelie, the towel that smokes weed.

After listening to every tape in the box, so as to let all 80-plus hours of it percolate in his brain, Chatman set about structuring the material into something that played like a standard comedy album. Kaufman staged even his most subversive performances within a traditional format, hanging his eccentricities on the formal scaffolding of the variety-show guest spot, the nightclub song-and-dance routine, the kiddie-playhouse show. “I got the sense that he wanted [the record] to be presented pretty much in the form of the popular comedy albums of the day,” Chatman says. “That was a legit format in the late ’70s — Steve Martin had massive, massive hits. It was a medium that had a cultural relevance that I think he was trying to play with.”

So we hear Kaufman onstage, committing beyond all reason to a hambone nonsense song about going to Birmingham and then admitting to the audience that it sounded funnier that morning, when he’d just woken up. We hear him picking a screaming fight with a movie-theater usher who won’t let him stay in the theater to watch the credits. “Andy Can Talk to Animals” is a Dr. Doolittle routine that turns into an argument with what sounds like a shrieking pterodactyl; the psychedelic six-minute sound collage “Sleep Comedy” is presented as a routine created expressly for sleeping audiences, and begins with Hader instructing us to “please fall into a deep, comfortable state of sleep, and then have a friend or associate play the audio … Please refrain from listening to this track while you’re awake.” The audio is rougher, the tone is more extemporaneous, but we’re not far, in these moments, from the self-reflexive shtick of classic mid-’70s comedy albums like Martin’s Let’s Get Small or Albert Brooks’s great A Star Is Bought.

Around the fifth track, though, Andy and His Grandmother becomes something entirely different. We’ve already heard a string of snippets of women angrily telling Andy where he can stick his tape recorder. Now, at the beginning of Track 5, we hear him testing the volume, then saying, “It’s a slice of life. We just screwed, and here’s the afterward conversation.”

Skeptically, a woman says, “Yeah?”

“Yeah,” Kaufman says. “It’s something that everyone does, something that should be captured. Something that’s never really captured, and it’s only captured by actors and actresses acting out a part. I’m interested in hearing how it sounds, how an actual conversation sounds. Now you know that it’s on, but you should make believe that it’s not on.”

Still a little wary, the woman says “OK. Yeah.” And then Kaufman launches right into it. “So,” he says, “it didn’t look to me like you were enjoying it very much.”

There’s nothing accusatory or confrontational about his tone. He could be asking her a question about badminton or traffic. Just making conversation. “I was!” the woman says, and Kaufman says, “You were? But it looked like you were just lying there. You wanted it to be over soon.”

She gets defensive. She insists that she did enjoy it. Kaufman starts asking technical questions. Why didn’t she put her legs up? “I, personally, you know what I like?” he says. He describes a sexual position, matter-of-factly. “Most girls don’t like that,” he says. He keeps pushing, asking her questions about what she likes; eventually she asks him nicely to turn the recorder off.

“Why is it,” he says, “that nobody understands that these, the kind of conversations nobody wants me to tape, are the kind of conversations that should be taped?”

“Anyway,” she says, “I enjoyed it. I waited a long time.”

“Well, if I hadn’t’a come back from New York so diseased, it would have happened a long time ago,” Kaufman says. Then he asks her what she’d do if she got pregnant; she says she’d hop a plane to Toronto, get a little abortion — that’s how she says it, “hop a plane,” “a little abortion.” “Well, if you get pregnant,” he says, “let me know. Maybe we could work out something. Wanna get married?” He doesn’t mean it. She knows he doesn’t mean it. She tells him not to joke about things like that.

The woman’s identity is none of our business. Kaufman is either being brutally, borderline-autistically honest with her or cavalier with her feelings. It’s pretty clear that she likes him more than he likes her. Yet there’s an innocence and a guilelessness about him even as he pushes her and pushes her, looking for the line in order to cross it. It’s unmistakably Kaufmanesque mischief but it’s also Kaufman having a frank, adult conversation with a woman about sex and relationships.

He was interested in these things — or in one of them, anyway. From Zehme’s book, we know that Kaufman called his parents “Mommy” and “Daddy” as long as he lived, that he gave up drugs for Transcendental Meditation in the late ’60s but consumed cookies and ice cream in mass quantities. But we also know about his struggle to balance his libidinous impulses with the principles of TM, his epic visits to the Mustang Ranch brothel outside Las Vegas, the penchant for enthusiastic wanking that often made him late for appointments.

This part of Kaufman hasn’t been fully present in any of his work before. He’s operating, decades ahead of schedule, in the territory of the TMI blog post, the confessional podcast, the verité discomfort zone. He talks candidly about sleeping with groupies on tour; near the end of the record, there’s an extended segment in which he goads two women (a girlfriend and an ex-girlfriend, it sounds like) into having a blowout argument about him over the phone. It’s weird and manipulative and totally compelling to listen to. It makes Kaufman more interesting, not less. Suddenly the extravagant misogyny of the wrestling era seems less like a pose and more like Kaufman either critiquing his own sexism or letting it off the leash.

Chatman says he didn’t think twice about including a representative sample of this material, pointing out that Andy always seemed not just willing but eager to render himself unlikable if a comedic conceit required it. “The last thing I’d want to do,” he says, “is to make something that was safe. That didn’t seem like it would be in the spirit of what he wanted. I think it’d be really betraying his very clear intentions, which he states on the tapes when he’s doing these things. He’ll be driving somebody crazy, and then they get really mad, and he goes, ‘Oh, that’s a great reaction. That’s going to be so great on the album,’ and that makes them madder, and he laughs, and he’s like, ‘Oh, this is so great,’ and it snowballs.”

“It never slipped into something that wasn’t fueled by at least an impish glee,” Chatman says. “There was never anything that was really malicious, you know? And there was never any evidence on the tapes that any of the women that he was dealing with wanted to stop being his friend. Or the men. But he definitely went after the women more, and he definitely just got off on it, you know? He got off on fucking with people, but he definitely got off on fucking with women in a very particular way, just like he got off on wrestling with women. And it would be crazy to divorce that side of it … Now you could describe a lot of what he was doing as trolling, y’know?”

When they met, Kaufman and Osgood hit it off immediately. They liked the same movies. Kaufman was almost nocturnal, and she could hang with that. “You’d stay up all night, sleep all day, get up, have dinner, go back out, and it would be dark,” she says. And like him, she enjoyed seeing people made to feel uncomfortable. She became a player in the little private Andy Kaufman shows Kaufman was always putting on for whoever happened to be around.

“You’d be walking down the street with him,” Osgood says, “and he would start yelling at you, and that would be your cue that you guys were supposed to get in a fight when you were walking down the street, or we’d be driving in a car, and some car would pull up next to us, and he would start choking me, just for the benefit of the people in the car next to us.”

All the recordings on Andy and His Grandmother were made before they met. “He never taped anything when I knew him,” Osgood says. I ask if he’d found other ways to mess with people, and Osgood says “Yeah — it’s called wrestling.”

“He couldn’t wrestle; he didn’t have the physique for it, so he became just the biggest jerk,” Chatman says. “He just wore long johns and shitty shorts and said, ‘I’m Andy Kaufman, and I’m better than you idiots,’ and he insulted the people directly. He played with this level of theatrics that was there by walking in and pretending like he wasn’t playing their game at all, and that he really thought they were idiots and jerks. Here’s this ready-made world for him to walk into and have people be all charged up. A weird kind of stage that a performer like him had never used before.”

Like any other show, the wrestling matches were a context that came with a set of assumptions he could play off of. If a lot of what Kaufman did on all those different stages in the ’70s and ’80s doesn’t look quite as subversive today, it’s because a lot of his deconstructive finishing moves have become part of the standard comedy toolkit. It’s much easier to find a space in which to do your performance-art character piece disguised as a terrible stand-up act, or an outlet that will air or upload your sneakily self-aware sitcom or your stream-of-consciousness sketch show; it’s much harder to find an audience innocent enough to not see every twist coming.

In a sense Kaufman was the last comedian ever who could do the kinds of things he did without people seeing it as someone “doing an Andy Kaufman.” We know the Kaufman spirit when we see it. We feel it enter the room whenever a comedian sabotages the tension/release physics of the comedy act. It was there, a little bit, in Zach Galifianakis’s old stand-up act, when he’d tell dumb jokes in a solemn monotone while accompanying himself on piano or take the stage as his imaginary brother Seth or bring in a gospel choir to sing his punch lines. There’s a little bit of it in Colbert and Tim and Eric Awesome Show, Great Job! and the more conceptual bits of Jackass and any part of Craig Ferguson’s show in which Ferguson banters with a robot. But all of these things were alternative comedy presented to alternative-type people who knew exactly what they were in for. “Kaufmanesque” is now a format as formalized as the ones Kaufman used to booby-trap.

Traces of the truly anarchic, wool-pulling parts of the Kaufman sensibility survive at the margins, in the work of reality-TV hoaxster Ken Tarr or on the feeds of so-called “Weird Twitter” personalities like @KatWillFerrell, who tweets endless broken-record riffs on the same dumb joke about Steve Jobs, Johnny Cash, and Bob Hope. Each episode of Adult Swim’s The Eric Andre Show begins with Andre in a state of berserker rage, smashing the cheap-looking burlap-colored set of his talk show, assaulting the house band, stripping naked, and so on. Then he calmly takes a seat at a desk on a once-again-intact version of that same set and proceeds to do a talk show that unfolds like an old Conan episode Sweded by people high on lead-paint fumes and angel dust. The talk show is destroyed, refuses to die, eats its own tail. But nothing has actually been subverted, because this is a 15-minute show that airs at weird hours inside the Cartoon Network’s de facto recreational-marijuana-user programming block; the subversion is hipster kabuki. The key metaphor of the Andre opening isn’t the smashing of the set but the set restoring itself. We’re too sophisticated and cynical to imagine a comedian breaking the machine, so the frustrating unbreakability of the machine becomes a bleak punch line.

Joaquin Phoenix’s TMZ-baiting bearded-train-wreck period is probably the most legitimately Kaufmanesque stunt any performer’s attempted in the past few years. But what did he really risk, in the end? He stopped shaving, announced he’d quit acting, did that stoned and surly Letterman spot, performed a few shitty rap shows, and confessed in time to market 2010’s I’m Still Here as a brilliant mockumentary (which it certainly was). He and director/accomplice Casey Affleck had made their point, and it turned out to be kind of a small one: The tabloid media are easily fooled, and because they revel ghoulishly in the travails of real people with real substance-abuse problems, they probably deserve to be. Apart from maybe Harvey Levin, who would argue? Phoenix’s career recovered quickly, if it was ever in real jeopardy. He returned to Letterman’s couch, clean-shaven and chastened, and to acting, garnering an Oscar nomination for The Master.

Steve Martin’s longest-running character is “Steve Martin,” frequent Saturday Night Live host and self-satisfied showbiz prick; “Steve Martin” exists in part to discourage us from wondering if the real Steve Martin is a self-satisfied showbiz prick, because a real showbiz prick wouldn’t let us see that side of himself. The premise of Phoenix’s “Crazy Joaquin” act was that Joaquin Phoenix was suddenly doing a bunch of things we’d never imagined Joaquin Phoenix would do; by admitting that his Joaqa Flocka Flame heel turn was a put-on, Phoenix subtly reaffirmed our vision of him as a thoughtful and serious artist who is dismayed by certain aspects of celebrity culture. There were a million reasons why Phoenix couldn’t be Kaufman that had nothing to do with his ego, but his ego was always there in the way.

Throughout the wrestling period, to the very end, Osgood says, “Andy never winked. He was a purist, and he didn’t care if it ruined his career — and it actually was ruining his career. He didn’t care. There was no way he was ever going to say, ‘Hey, I was kidding.’ He was being a bad-guy wrestler, and the bad-guy wrestlers he’d seen growing up never winked and said, ‘Hey, I’m really a good guy,’ you know? So Andy was just doing that. It was a performance, and he loved it. You can say, OK, he was obsessed and crazy, or you can say he was just doing an act. He was just doing it. It made him happy.”

“There was just this crystal-clear wonder and purity to what he did that didn’t have any cynicism,” Chatman says. “It definitely had ego, but it had, like, the ego of a 3-year-old.”