Costco is one of America's favorite places to shop: The chain recently earned second place among national grocery stores in the annual Retailer Preference Index from consumer data firm Dunnhumby. That's pretty impressive, considering you typically need to purchase a membership to shop at Costco, a basic version of which costs $60 a year. More than 94 million members pay so they can buy in bulk and access the chain's best products. Still, depending on your circumstances, not everything sold there is going to be a great deal for you. Because Costco often sells products in bulk, you need to be able to eat any perishable food before it goes bad, for example — so, if you live by yourself, springing for Costco's 1-pound container of fresh spinach may not make sense, even though it is a steal at $4.99. To get the most out of your Costco experience, Charlene Haugsven, founder of MyFrugalAdventures.com, tells CNBC Make It that shoppers should do these three things: stick to a list, only buy what your family will actually consume and try to stick with pantry staples. So what products should be on your list? Shopping experts weigh in on the nine foods that are the best bargains at Costco.

1. Baked goods

Costco cannot be beat for its in-house bakery treats such as pies, cakes and cheesecakes, Haugsven says, "People swear by the sheet cakes that they sell for birthdays and parties." Costco sells half a sheet cake that serves 48 people for $18.99, according to a local store CNBC Make It contacted. Haugsven says Costco's bakery deals go beyond just dessert. She also recommends its bagels. Costco sells Einstein Bros. bagels for $5.99 for a dozen. "You'd easily pay double that at a bagel shop," she says.

2. Cheese

Blocks of basic cheeses like cheddar and mozzarella are well-priced at Costco. "It's an amazing value," Haugsven says, assuming you can go through the quantity that Costco sells. Haugsven says she pays at least $3 less than she would for the same cheeses, including the Tillamook brand, at a regular grocery store. Costco sells 32-ounce blocks of mild cheddar cheese under its store brand, Kirkland Signature, for $4.59, according to local New York City prices analyzed by CNBC using Basket, a grocery price comparison app. Local grocery store chain Key Food, meanwhile, sells an 8-ounce package of Kraft mild chunk cheddar cheese for $4.99. If you like convenient, pre-shredded bags of cheese, you're in luck. Prices on these items also undercut those at the grocery store. (Pro tip from my mom, a Costco fan: You can store bags of cheddar and mozzarella cheese in the freezer.) A two-pack of Kirkland shredded mozzarella (2.5-pound bags) is $10.49, according to Basket. That's about 26 cents per ounce. In contrast, Basket finds Walmart sells a bag of Kraft shredded mozzarella for $2.38, or about 30 cents per ounce. Gourmet cheeses like brie are also well-priced, but Haugsven recommends avoiding these because of the package sizes: "You really need to ask yourself if you're going to go through that quantity."

3. Chicken

Whether it's cooked or fresh, Costco has great prices on chicken. Its $5 rotisserie chicken is so popular, the company is building a brand-new production center in Nebraska to keep up with demand. NPR reports it sells about 60 million birds a year. The value here comes in the chicken's size, not the price, Joanie Demer, co-founder of The Krazy Coupon Lady, tells CNBC Make It. A bird at Cosco is usually a pound larger than a typical grocery store. Fresh, organic chicken is also typically a good value at Costco since many retailers don't put this product on sale as frequently. Costco sells packages of organic boneless, skinless chicken tenders in pre-portioned packs (usually a total weight of about 6 pounds) for $4.99 per pound, according to Basket. Target sells organic chicken breast tenderloins for $1 more per pound, Basket reports, while Trader Joe's chicken is $7.49 per pound.

4. Frozen fruit

Especially if you're buying organic, frozen fruit can get pricey. Not at Costco. "The prices really can't be beat at Costco for the organic frozen fruit," Haugsven says. Plus, the chain offers a wide selection of fruits, from bags of mixed berries to pineapple — perfect if you're a morning smoothie person. A 4-pound bag of Kirkland organic frozen strawberries is $4.95, according to Basket. Aldi and Trader Joe's organic frozen strawberries retail for $2.69 for a 12-ounce bag.

5. Olive oil

If you use olive oil on a regular basis, check out Costco's organic, extra virgin olive oil. "It's really good quality at a really good price," John Karolefski, grocery store analyst and editor of Grocery Stories, tells CNBC Make It. Independent consumer testing organization ConsumerLabs analyzed 10 popular olive oils to see which provides the best quality and value. It named Kirkland's $16.99 organic extra-virgin olive oil one of the top three, beating out more expensive brands like Colavita and Newman's Own. The Kirkland bottle delivers a full 2 liters, so make sure it's something you use frequently before purchasing. A similarly sized bottle (68-fluid ounces) of Colavita extra-virgin olive oil sells for $17.98 at Walmart, according to Basket.

6. Maple syrup

A "big cult item" is Costco's Kirkland organic pure maple syrup, Haugsven says. Costco sells a 33.8-ounce bottle for $12.79. This is an especially good value because coupons for, or sales on, pure and/or organic maple syrup are rare, according to the Krazy Coupon Lady. Basket reports Trader Joe's sells a similar organic pure maple syrup in a 12-fluid-ounce bottle for $7.99. That ends up being 29 cents more per ounce.

7. Snack foods

Snack foods are generally cheaper at Costco when the store offers coupons for them, Haugsven says: "They won't tell you the prices, they'll just say $3 off Cheez-Its, and then you go into the store and you'll usually get a great deal on those items." This is especially helpful if you need snacks for school lunches. Currently, Costco says, it's offering $2.70 off a 45-count box of Cheez-It 1.5-ounce bags and $2.50 off a 65-count box of Kirkland soft and chewy granola bars. That brings the price of the Cheez-Its down to $9.79 and the granola bars down to $7.29. In contrast, Target sells a 20-count box of snack-size Cheez-Its for $6.99 and a 10-pack box of Quaker Chewy chocolate chip granola bars for $2.49, according to Basket.

8. Vanilla

One of the best deals at Costco is on vanilla extract, Demer says. Costco sells its Kirkland pure vanilla extract for $26.99 for a 16-fluid ounce bottle and "it's the good stuff," she says. Food and cooking site Kitchn also names it its pick for best budget vanilla. The Kirkland brand is about half the price offered by other stores, too. Acme, ShopRite and Stop & Shop all sell an 8-fluid-ounce bottle of McCormick pure vanilla extract for $22.49. Buying a large quantity can be fine, Demer says: "Keep it in a dark, closed pantry and it will last forever."

9. Wine