"This year is the worst it's been, ever," said Craig Underwood, who farms everything from strawberries to lemons to peppers, carrots, and turnips in Ventura County.

Some crops aren't get picked this season due to a lack of workers.

"We just left them in the field," he said.

The Western Growers Association told CNBC its members are reporting a 20 percent drop in laborers this year. Stronger border controls are keeping workers from crossing into the U.S. illegally, and the current guest worker program is not providing enough bodies. (Related: Massive US Drought Leads to Worst Fears for Corn Crop.)

"We have 100 fewer people this year," said Sergio Diaz, who provides workers under contract for growers. "We're having difficulty finding people to do this work."

The lack of workers is forcing farmers to pay more. In one of Underwood's fields, pickers are harvesting peppers for $9.25 a hour, or $5 a bucket, whichever is more. Craig Underwood said his workforce is aging and starting to retire, and no one is coming in to replace them. (Related: Recession-Proof Industries.)

"Migratory flows between Mexico and the United States have come to a halt," Carlos Gonzalez Gutierrez, consul general of Mexico in Sacramento, told a California farm bureau labor committee, according to AgAlert.

Growers of California's wine grapes are concerned there won't be enough pickers for this fall's harvest. Berry growers — among the highest paying — saw fewer field hands show up in the spring.

"Fruit that you should be picking is not being picked," said grower John Eiskamp.

Most pickers in California are not here legally, a fact of life for decades.

When asked if any local residents have come out to apply to work in the fields, Craig Underwood replied, "None. Absolutely none." He is even having trouble finding truck drivers and other semi-skilled labor for jobs that pay $12-$18 an hour. (Related: America's Top States for Job Creation.)