DJERBA, Tunisia — On a recent Sabbath morning, just five men sat reciting prayers in the sanctuary of the Ghriba synagogue, the oldest synagogue in North Africa and the holiest place for the Orthodox Jews who live here on the resort island of Djerba.

It was the weekend of the annual Ghriba pilgrimage, when hundreds of Jewish visitors from Israel, France and the United States visit the synagogue to celebrate the feast of Lag b’Omer. Yet on the Sabbath, when only those within walking distance attended, the true state of affairs was revealed: The village beside the synagogue had dwindled to just five Jewish families, barely 40 people.