The first African-Americans on Martha's Vineyard were indentured servants, runaway slaves, and whalers. They came for the oil. "You have to understand," says Skip Finley, a columnist for the Vineyard Gazette. "New Bedford, Nantucket, and Martha's Vineyard were pretty much the Middle East in those days." But by the mid–19th century whale oil was being displaced by more easily obtainable energy sources, and the island was reborn as a resort. A group of Methodists built a tabernacle in Wesleyan Grove for open air revivals, and rows of gingerbread cottages sprang up around it. The residential development was called Cottage City until it was reincorporated as Oak Bluffs, in 1907.

Dorothy West, "the Kid" of the Harlem Renaissance, grew up going there every summer. Adam Clayton Powell Jr. came to stay at Shearer Cottage, a seasonal inn catering to African-Americans—when very few places did—and beat the drum in his newspaper, the People's Voice, urging his readers to visit. Many did, and a number of them became serial renters or homeowners. By the 1950s, middle- and upper-class black doctors, lawyers, and executives and their families had formed their own society, with its own yearly rituals. When President Obama first vacationed on Martha's Vineyard, in 2009, it was seen as a nod to both the Kennedy-Clinton presence and the unique heritage of Oak Bluffs.

Victorian gingerbread cottages in Oak Bluffs John Greim/LightRocket via Getty Images

As the Smithsonian's new National Museum of African American History honors the town in a permanent exhibition, "Power of Place," opening in September, a few of the community's longtime residents share their stories.

HENRY LOUIS GATES JR. (director of the Hutchins Center for African and African American Research at Harvard) When I was growing up, there were only three places where black people could summer on the beach. There was Martha's Vineyard, there was Sag Harbor, and there was a place on the Chesapeake [Highland Beach, Maryland].

Henry Louis Gates Jr. on a dock in Aquinnah Chadwick Tyler

You might integrate the schools, but integrating beaches or swimming pools took longer. They wanted to keep semiclad people away from each other, because all they do is sit around and fantasize about screwing each other, especially if it's against the rules.

So, ironically, when most of America was discovering the wonders of the Cape and the Vineyard through the Kennedys, there was a class of black people who knew that world very well. And very few people could have imagined that when John Kennedy gave his civil rights speech saying that James Meredith was going to be admitted through the force of the federal government to the University of Mississippi, there were black families that had owned property on the Cape and on the Vineyard for a very long time by that point. That would have been a shock, I think, to most people. I know it was a shock to me when I later found that out.

SKIP FINLEY (columnist, the Vineyard Gazette) Nobody had ever heard of Martha's Vineyard, so we would say that we went to the Cape, because people had heard of Cape Cod. In the late '60s, when Ted Kennedy had the mishap, that all changed. Then it was cool to say you were from Martha's Vineyard.

I don't like Long Island, and I don't like Nantucket. I like Martha's Vineyard. —Vernon Jordan

VALERIE JARRETT (senior adviser to the president) When I was 10 my parents and my mother's sister and her family all decided to go. My aunt's godmother, who was from Chicago, retired there, and she was the initial hook. I instantly fell in love with the island. My cousins and I were allowed to roam freely without checking in with our parents until nightfall or locking our doors. Having grown up in a big city, I relished the ability to just run around and explore unencumbered by adult supervision.

VERNON JORDAN (civil rights activist and senior managing director, Lazard Frères) In 1971 a friend of mine invited me and my family to come to stay with him in Oak Bluffs for three or four days. We were new to New York and had heard about Martha's Vineyard but had never been, and when I got the invitation I did not hesitate. I got to play tennis with [former U.S. senator] Ed Brooke. I got to have conversations with college presidents. It was the very essence of the black community gathered for vacation. And I've been going ever since.

Vernon Jordan at the Chilmark Farmhouse he has rented every August for 20 years

I don't like Long Island, and I don't like Nantucket. I like Martha's Vineyard.

STANLEY NELSON (documentary filmmaker, The Black Panthers: Vanguard of the Revolution and 2001's A Place of Our Own, which is about Oak Bluffs) I went for a week when I was six years old, and it rained pretty much the whole week. We had such a good time that there was a feeling like, if the sun ever came out— which it must—then this place must be really something.

CHARLAYNE HUNTER-GAULT (journalist and author, To the Mountaintop: My Journey Through the Civil Rights Movement) In 1970 we were invited by the vice president of the Ford Foundation, where my husband was working, to come and visit for a weekend in what was then called

Gay Head, which has now changed its name to Aquinnah. We had heard about Oak Bluffs—I had a high school classmate whose father was a doctor, and they used to go every summer from Georgia because it was one of the few places, even in the '70s, that was fully open and welcoming to black people. So I said, "Why don't we go down there?" We stayed there for a couple of days and went to the Inkwell, Oak Bluffs' main beach, and I was so excited about what I saw there: artists and doctors and writers, just a range of interesting black people with interesting vocations. I called Arthur Gelb, who was my editor at the New York Times, and I said, "Look, I haven't discovered this, because it's always been here, but we've never written about it, and I'd like to stay a few extra days to do a piece." So I stayed and I interviewed various people about their connection to Oak Bluffs, many of whom went back years.

The one disappointment I had was that I had heard about the Vineyard, the Vineyard, the Vineyard, and I had expected to see grapevines everywhere! There are a few around, but not on the level that I'd thought.

There wasn't just one way to be black. Everybody would be accepted... whatever subdivision of blackness you fell into. —Henry Louis Gates Jr.

SKIP FINLEY The Methodists came here to pray in 1835—they actually founded Oak Bluffs. In the 1920s, 1930s, it became a vacation spot for middle-class and upper-class black folks from Boston. Then, in the 1950s, the New Yorkers, the D.C. crowd, and Philadelphia folks started coming up, and it's just grown ever since then.

Many people associated with the Harlem Renaissance came and stayed at Shearer Cottage, a black-owned bed-and-breakfast. Dorothy West, a little bit differently, chose to live here year-round. She started writing my column—the Oak Bluffs town column [in the Vineyard Gazette].

HENRY LOUIS GATES JR. Dorothy West's job was to report on the goings and comings of Oak Bluffs. You could go back and look up the summer of 1981 in the Vineyard Gazette; somewhere she'll say that Dr. Henry Gates and Dr. James Comer had drinks at Jeanne Curtis Davis's house.

Dorothy was buried on Martha's Vineyard in 1998. Charles Ogletree spoke at her memorial service, I spoke, and Hillary Clinton spoke.

Stanley Nelson at the home purchased by his father in 1966.

STANLEY NELSON There was nothing like being 15, 16, 17 on Martha's Vineyard. There was a whole contingent of black people my age—there were around 40 of us. We would spend the summers there because a lot of women did not work, and a lot of the women who did work were teachers, so they were off during the summer. The daddies would be at home in New York or Boston or Philly or wherever, and they would all come up on Friday night on the "daddy boat." That has changed, because women work now, and Martha's Vineyard has become so expensive that there are very few people who can come up and spend the whole summer with their kids.

Myles Reynolds, visiting from Pennsylvania.

HENRY LOUIS GATES JR. It was very free. There wasn't just one way to be black. Everybody would be accepted, even if your mother was white or your father was white, whatever sort of sub- division of blackness you fell into. It was a nurturing environment, but a socializing environment, too. It was a way of cementing the bonds among the black elite, A; and B, making a child who preferred books to basketball feel comfortable. There's a basketball court in the middle of Oak Bluffs, and it's packed every day and every evening, but the people playing basketball are also people who are probably gonna go to Harvard, Yale, or Princeton and become doctors or lawyers, and that's different.

CHARLAYNE HUNTER-GAULT My husband and I would come from South Africa, where I was reporting for NPR, and later for CNN. We would come and stay for a month.

SKIP FINLEY My folks came in 1955. We stayed the whole summer, and I've been here every summer since. That's when they bought my family house, which was built in 1872. We are the fifth family to own it and the fourth family to live in it, which sounds strange, but back in 1955 my dad had to buy it from his lawyer because the owner wouldn't sell it to him. Racism wasn't out front; people regulated it through the codicils on their homes.

HENRY LOUIS GATES JR. Now you can go to the beach anywhere, and maybe then—theoretically—you could have gone to the beach anywhere, but nobody did. Everybody went to the Inkwell. You'd go to the beach, the Inkwell, and then you'd come home about four and get ready to go to the five-to-sevens, which were cocktail parties from five to seven. And you would dress up and do your hair.

You can date the generations on Martha's Vineyard by the people who have heard of and attended five-to-sevens and those who haven't, and I would venture that many of the more recent black homeowners and renters on the Vineyard have never even heard of five-to-sevens, because nobody does them anymore. But in 1981, when I first visited, they were alive and well, and the whole day was spent in anticipation. It was quite a ritual, and their disappearance shows the broader integration of the black community into other elite structures on the island and the diversification of the black community. The dinner party circuits are now integrated in a way that they weren't when I started going. Or, if they were, I just was a little squirt and nobody even heard of me—and I wasn't invited!

VERNON JORDAN I like to go to the golf course and play nine holes or 18 holes at Farm Neck, where I'm a member and I've been a member for a long time. It is the most integrated golf course, racially and economically, anywhere, that I know of. I spend the afternoon just sort of relaxing. Sometimes I play tennis. It's also a time to do a little reading. I'm big on political biographies, and then sometimes I read junk—page-turners.

Valerie Jarrett at Nancy's, where she dines at least once a day.

VALERIE JARRETT For years I've gone to Nancy's [restaurant] every day, sometimes twice a day. I sit outside at the picnic tables, joined by family and friends. If it's raining we sit at the grass hut raw bar. And when I go for dinner, I sit on the patio and watch the sunset.

The homes where I have stayed since, probably, 2003 have faced the ocean. So I always rise early and watch the sun rise. Each day feels like a fresh start. The biggest decision we make on vacation is which beach to visit. The easiest decision I ever make! Those are the tough calls.

SKIP FINLEY Some of us who grew up here go to all of the unnamed, unheard of, untalked about beaches. There are 54 named beaches on this island. Most people know of three, and there are several I don't tell anybody about because I want to go there in the middle of August and have there be nobody but me and my lady, you know?

VERNON JORDAN I never go to the beach. There are many things I need; a suntan is not one of them.

HENRY LOUIS GATES JR. In the '90s we started an annual panel discussion at the Old Whaling Church, in Edgartown. I'd get Cornel West and Michael Dyson and Melissa Harris-Perry and Lani Guinier and Charles Ogletree, put them on a stage, and let them argue with each other about affirmative action. Because after a while there's only so much sauvignon blanc you can drink! I mean, how much lobster can a person actually consume?

Racism wasn't out front; people regulated it through the codicils on their homes. — Skip Finley

VERNON JORDAN I don't do fundraisers. If it were up to me, I would ban all fundraisers. And the same is true of book parties. When I did my book in '01, book- stores wanted me to have a book party, but I would not have one on the Vineyard, because I figure everybody has the same attitude that I have.

Macy and Maya Thames, visiting from Maryland

SKIP FINLEY Last summer I said, "I wonder how many of us, the original crowd"—and by original crowd I mean people who got to stay all summer long—"are still around." And after a couple of bottles of wine we came up with over 100 names of people who had been fortunate enough to grow up like that.

VERNON JORDAN My wife Ann, who grew up in Tuskegee, has been going to the Vineyard since she was a teenager. My daughter Vickee met her husband on Martha's Vineyard. So it's a second home. It's the second community.

Last year my wife gave a great party for me, for my 80th birthday, and I did the same thing for her the year before. And so we had two great parties. I was toasted by both President Obama and President Clinton, and my daughter, my wife, and then I said something. We didn't spend a whole lot of time toasting. We spent the evening dancing.

VALERIE JARRETT With each passing year my circle of friends and family expanded. It has reunite each summer in August. I now have about 25 or 30 relatives there [including her cousin Ann Jordan, Vernon's wife] and at least 40 good friends who all go at the same time every year. We love being able to actually just relax and spend quality time together, with only the occasional work interruption. I took the Vineyard for granted when I was young. What's different now is that I appreciate it so much. I have friends whom I envy greatly who stay for the entire month of August, and next year that's what I intend to do.

Photographs by Chadwick Tyler

This story was originally published in the August 2016 issue of Town & Country.