“Predictable” was never a word you’d use to describe R.E.M.’s career. Their discography was full of genre experiments and creative left turns, but nothing captures the band’s spontaneity like some of the shockingly good cover songs they recorded over the years. Released on albums, as B-sides and on their annual fan club Christmas singles, these recordings gave a bump to some of their under-appreciated contemporaries, celebrated the band’s influences, and — above all — were just plain fun.

Here are 20 officially released covers that any good R.E.M. fan should keep in regular rotation.

20. “#9 Dream” (John Lennon)

Recorded ten years after his retirement, R.E.M.’s cover of “#9 Dream” for the Amnesty International Instant Karma compilation got founding drummer Bill Berry behind the kit for one last session with his old bandmates.

19. “Wicked Game” (Chris Isaak)

R.E.M. played “Wicked Game” a handful of times on the Monster tour, but fan club members got to add it to their collections when it got an official release on the band’s 1995 Christmas single.

18. “Ghost Rider” (Suicide)

Is it loyal to the original? Nope! But it was a bold move for R.E.M. to put a Suicide cover as a b-side to a big single like “Orange Crush” during their commercial peak.

R.E.M.’s 1991 MTV Unplugged taping

17. “Love Is All Around” (The Troggs)

A fixture of the band’s acoustic Out of Time-era performances, the Mike Mills-led cover of “Love Is All Around” was released as a b-side not once, but twice — on the“Radio Song” and “How the West was Won and Where It Got Us” singles.

16. “First We Take Manhattan” (Leonard Cohen)

R.E.M.’s version of “First We Take Manhattan” is a gritty, funky take on Leonard Cohen’s apocalyptic 1988 classic, recorded for the I’ll Be Your Fan tribute compilation. Their paths would (sort of) cross again when they gave Cohen a songwriting credit on “Hope” because of its similarities to “Suzanne.”

15. “Sponge” (Vic Chesnutt)

Michael Stipe became a Vic Chesnutt fan after seeing the singer-songwriter live at the 40 Watt Club in Athens, quickly becoming a close friend and even producing his first two albums. Their brooding cover of “Sponge” was released on the Sweet Relief II benefit album, which featured artists like Garbage, Smashing Pumpkins and Sparklehorse performing their favorite Chesnutt songs.

14. “Pale Blue Eyes” (The Velvet Underground)

While the studio version on the “So. Central Rain” single is a little bit sloppy, live versions of “Pale Blue Eyes” sound just as good as any R.E.M. original.

13. “Toys in the Attic” (Aerosmith)

R.E.M. covering Aerosmith seems weird at first, but Peter Buck pointed out that rock music fans in the seventies couldn’t help but be into the Bad Boys from Boston. Officially released as a b-side to “Fall on Me”, it was played more than 50 times on the Fables of the Reconstruction and Lifes Rich Pageant tours.

12. “Strange” (Wire)

One of only two covers to appear on an R.E.M. studio album (we’ll get to the other one shortly), the band’s take on ‘Strange” doesn’t stray too far from the original on Wire’s legendary 1977 debut album, Pink Flag.

11. “Funtime” (Iggy Pop)

Michael Stipe didn’t wanna be Iggy Pop, but if that’s what it takes to pull off this cover… The band released two versions of “Funtime”, a studio version that served as b-side to “Get Up” and a killer live performance from the 40 Watt Club that ended up on the “Strange Currencies” single.

10. “Crazy” (Pylon)

R.E.M. started covering “Crazy” almost immediately after it was released in 1983, eventually recording a studio version during the Fables of the Reconstruction sessions that ended up on the “Driver 8” single. Peter Buck says he heard it on the radio the same day Chronic Town came out and was “instantly depressed by how much better it sounded than our record.”

9. “Wichita Lineman” (Jimmy Webb)

Recorded live in Houston during the 1995 tour, R.E.M.’s cover of “Wichita Lineman” does an amazing job of capturing the narrator’s loneliness. I suppose 100+ arena shows in a year will do that to you…

8. “Arms of Love” (Robyn Hitchcock)

R.E.M.’s stripped down, acoustic take on “Arms of Love” is a vast improvement on the original’s icy synthesizers and drum machines. A seriously overlooked gem in the band’s catalog, this one’s required listening for fans of their direction on Out of Time and Automatic for the People.

7. “Academy Fight Song” (Mission of Burma)

By the time they recorded it for their 1989 fan club single, R.E.M. had been road testing their cover of “Academy Fight Song” on the Green Tour. A fitting tip of the hat to their influential peers in Mission of Burma, you can hear Michael give frontman Roger Miller a shoutout after the very last chorus.

6. “Favorite Writer” (Magnapop)

While R.E.M.’s version was released as a b-side to “Bad Day” in 2003, Michael Stipe had a hand in producing the original on Magnapop’s self-titled 1992 debut.

5. “See No Evil” (Television)

Taken from their first fan club Christmas single, this classic Television song sounds like it could’ve been an R.E.M. original. Maybe that’s why they played it live more than popular songs like “Perfect Circle”, “Stand” and “Daysleeper.”

4. “Wall of Death” (Richard and Linda Thompson)

Richard Thompson got an overdue critical reassessment with the 1994 tribute album Beat the Retreat, which featured covers of his songs by modern rock artists like X, Dinosaur Jr. and Bob Mould. While their contributions were all great, it was R.E.M.’s sunny, alt-country take on 1982’s “Wall of Death” that really stole the show. Featuring pedal steel from John Keane, it got a bit of a second life when it was included on the “E-Bow The Letter” single.

Photo by Sandra Lee Philips/I.R.S. Records

3. “There She Goes Again” (The Velvet Underground)

Peter Buck had fond memories of recording this perfect cover for the “Radio Free Europe” single. “It was recorded live to two track with Mitch Easter playing acoustic guitar into the same mic as myself. I remember that I had to step forward to play the solo, just like Bill Monroe.”

2. “Indian Summer” (Beat Happening)

One of two Beat Happening covers from the Accelerate era. Michael Stipe’s gravelly voice stands in stark contrast to Calvin Johnson’s innocent, childlike lyrics and the band clearly enjoys creating a hazy, drone-like sonic landscape behind him.

1. “Superman” (The Clique)

The obvious choice for #1, right? Mike Mills steals the spotlight again on a track that includes Godzilla toys, wild background yelps from Stipe and everything else you could want in a grand finale to an album like Lifes Rich Pageant.

The author’s favorite R.E.M. album is New-Adventures in Hi-Fi, but Lifes Rich Pageant is a very close second. You can follow him on Twitter.