INA GARTEN, AKA THE Barefoot Contessa, waited until she was in her 40s to start playing with dirt. "When I first got married, I had no interest in gardening whatsoever," she said. She was far too busy, first working as a nuclear-policy analyst in the Office of Management and Budget for the Carter administration, then building the prepared-foods business in East Hampton, N.Y., that would become one of the most beloved—and successful—home-entertaining brands of our time. In the late 1980s, she was ready to grow something else. She cleared the area surrounding the East Hampton rental property where she and her husband, economist Jeffrey Garten, were living, and created a shade garden. "I was moving trees into the woodland," she said. "At which point my husband said, 'It's time to buy a house.' "

The couple found a shingle-style home in East Hampton and carved out a quarter-acre "kitchen garden" (though it's mainly comprised of flowers, it abuts the Gartens' kitchen). Conceived by landscape designer Edwina von Gal, the formal plot is filled with neat rows of parterres whose boxwood perimeters contain bursts of dahlias and roses, as well as a vegetable square brimming with tomatoes, cucumbers and herbs.

Six years ago, the Gartens purchased the adjoining property and built what they call "the barn," a grand, airy structure where Ms. Garten films her television shows and writes her cookbooks (the eighth, "Foolproof," comes out in October). The adjoining walled garden, built to keep deer away from the plants, is more intimate and whimsical than its older sister. At the center, squares of dirt house leeks and fennel as well as a jubilant bed of Swiss chard. Lavender and apricot roses scratch up the garden's periphery.

When Ms. Garten, 64, is not masterminding her brand, she and her husband host weekly dinner parties and travel to Paris, where they have an apartment (no garden). A recent visit to her East Hampton compound found her in the barn, listening to Frank Sinatra and ready to give a garden tour—but not before sitting down to a pot of coffee and thick, buttery slices of coffee cake.

In the beginning, I thought was going to grow everything from turnips to radishes. And then I found that everything had one bite taken out of it, thanks to the chipmunks and the deer. Now I grow much more flowers than vegetables.