On a recent trip to my parents' place in New Hampshire, I cooked up some chicken leg quarters on the Traeger Timberline 850 grill that I'd brought along to review. I set it to the exact temperature the manufacturer suggested, lit the grill with the press of a button, and, using an app, remotely monitored both the temperature inside of the grill and the internal temperature of the legs and thighs as they cooked.

When the chicken reached 165 degrees Fahrenheit, I opened the grill, expecting smoke-tinged chicken perfection. Instead, I called mom for help.

The internal temperature indeed read a perfect 165, but the skin was rubbery and there was no way I was going to get the heat jacked up to searing temperature without severely delaying dinner and potentially overcooking the food. Instead, my mom browned the skin in a skillet on the stove, saving dinner. The chicken turned out so fantastic that I wondered if the manufacturer's product and recipe testers instinctively counted on us to ask the lady who brought us into the world to come to the rescue.

The Traeger Timberline 850 is what's known as a pellet cooker, and this episode encapsulated some of the best strengths and weaknesses of the machine. Pellet cookers tend to have a recognizable grill-like interior, and a hopper full of wood pellets on the side, which are transported to a "burn pot" under the grill grates via a mechanized auger. (Or, as my wife put it: "You push the wood pellets in and it burns them.") The grill plugs into an outlet, allowing that auger to work and offering push-button ignition. The Traeger has a thermocouple probe at the top right of the interior, helping it to monitor the temperature and holding it steady like an oven, while imparting a lovely smoky flavor. For long, low, and slow cooks—ideal for foods like brisket—pellet cookers are stunningly effective. Searing, however, is their Achilles heel, which made me wish that Traeger would replace the stubby grill table on the left side of the Timberline with an electric griddle that would make searing a snap.

The Timberline also features Wi-Fi connectivity, allowing three-way communication between the app, the grill, and the cloud. There's some fantastic potential here, but this is Traeger's first foray into connected cooking and the lack of experience shows. Combine that with some mechanical difficulties and the grill simultaneously charges into the future and shoots itself in the foot. Fairly inexcusable, considering that it costs $1,700.

Smoke It If You've Got It

The potential here is enormous. The ability to simply set the temperature and walk away offers a huge amount of control, a difference exacerbated when you switch back to a "regular" gas grill or Weber kettle. Set a brisket to 180 and monitor the internal temperature thanks to the built-in probe thermometer and you begin to realize you could, with long, low-temp cooks, crib recipes from a sous vide cookbook and get fantastic, smoky results. Pair that with an app that could understand what's happening while you cook, and you've got a game changer.

Yet when I made beef brisket (Traeger's "Beginner's Brisket"), the one recipe that all pellet cookers should simply crush, limitations and defects were exposed. Most notable was the discrepancy between the set temperature of 180 degrees Fahrenheit and the actual temp the grill cooked at for the first two and a half hours—between 196 and 221—followed by a brief dip to 161 and a return to 190. Roughly, that meant it cooked about 20 degrees higher than the temperature it was set to for the first four hours of the cook. When I swung the temp up to 250 for the later stages of cooking, it did an amazing job of staying within a couple degrees of the target. (These impressive results were repeated a few days later when I made ribs at 225 degrees.)

Toward the end of the brisket's cook, I encountered a famous problem known as "the stall," where the internal temperature of the brisket climbed steadily toward a target temperature of 203 degrees, but plateaued at around 190 for a few maddening hours. If you realize it's happening, you can get around by jacking up the grill temperature, or tightly wrapping the brisket with foil (a.k.a. "the Texas crutch"). If the grill or app knows the temperature curve has flattened at the typical stall temperature, why not have the app alert me and tell me some options instead of pushing dinnertime into bedtime?

Nevertheless, it was a fantastic brisket, with a dark exterior "bark" and the pink circle just underneath called a smoke ring, a sign of quality so pronounced that looked like it was painted on with lipstick.

That temperature discrepancy in the first phase of the cook made me wonder about the accuracy of the thermometer, so I stuck the Traeger's probe thermometer under my tongue and learned that either the thermometer was off by almost 10 degrees or I was experiencing a freak attack of hypothermia.

For comparison, I put my mom's years-old Pyrex meat thermometer under my tongue and the dial pointed toward 98.5.

I ordered a ThermoWorks Smoke, a grill thermometer with both a probe thermometer to track internal temperatures of what you're cooking and an "air probe" that clips to the grill and monitors the cooking temperature; essentially, replicating the same setup built into the Traeger. Naturally, I put the ThermoWorks cooking probe under my tongue and learned that I was running at 97.2 degrees, but for more accuracy, I created a 32-degree ice bath. The ThermoWorks probes and my mom's Pyrex all clocked in within a degree, but the Traeger probe read 26.

Yikes.

The ThermoWorks Smoke has big magnets on the back and it clamped onto the side of the grill with gusto, as if it was eager to bore through the Traeger's hull and suck out its brains. I positioned the ThermoWorks "air probe" next to the Traeger's and found that the smoker ran six to 10 degrees higher than the temperature it was set at.

This is bad but forgivable for the set temperature of the grill. On the Traeger's probe thermometer, though, it's harder to ignore. Those degrees of difference means that what reads as rare comes out as medium-rare, medium is medium-well, and well-done is a briquette. What's really unforgivable? You can't calibrate the Traeger's thermometers. (Asked about this, a spokesman said, "We would replace them if there was ever a need.")

That means any programmed cooks I made had built-in flaws that forced me to buy a $100 grill thermometer that I could calibrate. With this setup in the long term, I'd have to forgo much of the allure and promise of the connected capabilities of the grill.

Bring the Heat

Unfortunately, there's more. When I made ribs, I turned up the temperature toward the end of the cook to get some caramelization and searing. When I returned a few minutes later, the temperature hadn't gone up but there was a clinking sound I mistook for the fire gaining momentum. Instead, I'd learn that the fire had gone out and the auger kept pushing pellets into the bottom of the grill.

I fished the ribs on my dad's grill.

The app was also a disappointment, mostly because there was little to no value added by using it. Creating custom cooks inside the app seems like a natural, but for reasons I don't understand, you can only make them using the dial on the side of the grill; have fun typing in "Aunt Janice's world-famous pulled pork" with that setup. Similarly, why not send an alarm to the app when a target temperature is reached, and drop the grill temperature to prevent overcooking? How about teaching it to deal with "the stall"?

Traeger could also do with some more modern recipe options, and make sure those recipes are tested. I ran into trouble where programmed recipes marched on from one step in the recipe to the next without completing the first. Another programmed cook called for a probe in the app, but not on the machine. The smoked salmon recipe marinates the fish for 2 to 4 hours in a "beg." Plus, the grill temps in Traeger's recipes tend to be higher than more-trusted modern cookbooks like Meathead Goldwyn's Meathead or Stephen Raichlen's Smoke.

I did make some great food with the Traeger, and the overall potential has me eager to see how this particular style of smart grill develops in the next few years.

That said, while many of the Timberline's problems are fixable with software updates, some of them, like the lack of a searing option or the inability to calibrate the thermometers, leave this machine hobbled from the get go. And at $1,700, that's a lot to stomach.

Food writer Joe Ray (@joe_diner) is a Lowell Thomas Travel Journalist of The Year, a restaurant critic, and author of "Sea and Smoke" with chef Blaine Wetzel.