SAN FRANCISCO — Gerald Harris was walking along Ocean Beach, the blustery coastline at the western edge of the city, when he passed Danny Glover, a star of Hollywood action movies and a San Francisco native. The men exchanged glances.

“We were the only two black people in the area,” Mr. Harris said.

San Francisco was once a national beacon of African-American culture, home to a thriving jazz scene that had so many clubs it was known as the Harlem of the West. But these days, blacks say they take notice when they see another African-American in affluent and middle-class neighborhoods.

The jazz clubs of the Fillmore neighborhood have been replaced with upscale shops. Marcus Books, a cultural anchor of the black community and one of the first bookshops in the nation to focus on African-American topics, closed in 2014. Other black landmarks that have long since disappeared are commemorated with remembrances embedded in the sidewalk like tombstones to a forgotten culture.

The decline has been steady and noticeable. One of seven residents was black in 1970. Today, it is nearly one of 20, with most of the city’s 46,000 blacks living in public housing.