Most people may think of Greece, Italy, or Spain when they grab a jar of table olives at the grocery store, but they’ve been grown in California since 1870. Though maybe not for much longer. Olives have never been California’s largest agricultural product – at its height the industry only had a little over 35,000 acres of table olives – but as growers face low prices, high labor costs and dwindling water supplies, many are choosing to leave generations of olive-growing for a more lucrative crop.

Though California growers recognize that farming is a business, it doesn’t mean there’s no sentiment involved when it’s time to move on.

“My grandfather planted our first olives in about 1945-1950,” says David DePaoli, an olive grower whose family started farming in California about 100 years ago. When DePaoli took over from his father, he increased their original 20 acres of olives to 40, then 60 acres. In 2010, he started tearing those trees out. “The last 20 acres that I still have, my father planted anywhere from 1957-1972,” DePaoli says. “It’s a family tradition and it’s hard to remove those trees.” At 61 years old, DePaoli has literally grown up with the same orchard that he’s now contemplating tearing down. “But within the next two or three years those trees will probably have to come out, too.”

Though California growers recognize that farming is a business, it doesn’t mean there’s no sentiment involved when it’s time to move on. In the case of olives – a tree that can live well over 500 years and still bear fruit – farmers are losing history as well. “Olives have been around since the start of civilization,” DePaoli says. “It literally hurts to pull them out.” Unlike growers who live in areas suited to only one crop, California’s olive growers have many other lucrative options to choose from.

The most popular option is almonds, whose price has climbed from less than a dollar in 1999 to more than $3 a pound today. “Almond trees are really good money,” says Burkett. Not only do new trees start producing within three years (compared to eight for another nut like pistachio), the harvesting process is mechanized, cutting down on annual labor costs. “Labor is another Achilles heel for the table olive industry,” admits Adin Hester, president of the Olive Growers Council.

Unlike almonds, the more delicate table olives need to be hand-harvested. “In the table olive business, harvest can take over 50 percent of the farmer’s gross income,” he adds. And when there’s not a lot of money coming in to begin with, those fees are even more daunting.

Almonds may have some financial advantages over olives, but the drought is causing problems for any plant California growers want to keep alive. Olives do have the benefit of being highly drought-tolerant trees that can survive on light rainfall alone, but to bear enough fruit to harvest commercially, growers need to keep their thirst quenched. An acre of commercial olive trees needs three acre-feet of water to bear fruit (“acre-foot” referring to the amount of water needed to cover an acre of land with one foot of water), which is not nothing. One acre-foot is equivalent to about 325,000 gallons of water. However, an acre of commercial almonds need even more – closer to four acre-feet on average. In a state where the majority is in an “extreme” or “exceptional” drought, the difference between water needs in crops quickly gets expensive for farmers and the public alike.

Over the last few years, farmers have lost their government water allocations and have resorted to a mixture of drilling for groundwater and purchasing water rights from neighboring farms. “I’ve got well water but I’m afraid my wells are going to go down,” says Burkett, who acknowledges his water situation is better than many. “The question is, why would I stay in olives?” Neighbors are growing everything from pecans to oranges and almonds, and there’s nothing stopping him from joining in.

Burkett recently purchased his neighbor’s water rights after the other grower decided to get out of the farming business. “I took that water and put it on stuff that’s profitable,” he adds. For most growers, that means anything but olives.

Though switching to almonds means using more water, growers are getting high enough prices from the crop to afford it. That is, until there’s no water left for anyone anymore.

As the California drought enters its second year as an officially declared “drought emergency,” some growers are hopeful that rain will come again soon. “Long-term, it’s going to rain,” says Burkett, who hopes that farmers will start getting water again next year rather than five or 10 years from now. “But in the meantime, you do what you have to do.”

Correction: A previous version of this article stated that Rod Burkett was chairman of the California Olive Council. He is chairman of the Olive Growers Council of California. The story has been corrected to reflect the correct amount of water needed to grow olive and almond trees. We regret the errors.