After starting the year on a high note energy stocks have faltered this month, and two market watchers say there could be more downside for the sector.

The XLE energy ETF, which tracks some of the biggest energy players, shed nearly 4 percent last week, its worst weekly performance of the year.

The ETF is still green for the year — with shares up roughly 11 percent — but Piper Jaffray's Craig Johnson says this is more of a relief rally than anything else. In other words, he doesn't think the move higher is based on underlying strength.

"I think we're going to see energy sold, and that money is going to rotate into other areas where there's more momentum in the market," he said Friday on CNBC's "Trading Nation." He added that he would be "trimming positions in the XLE at this point in time."

Last quarter, the sector shed 24 percent — its worst quarter in a decade . Johnson attributes the move higher at the start of the year as simply a temporary break in selling.

Johnson also points out that from a technical perspective the sector is not flashing a buy signal. It's below its 200-day moving average, and while it has exhibited this kind of downward momentum before, this time its relative strength versus the S&P 500 has failed to move higher.

"What's interesting is we had the huge sell-off in Q4," he said. "Energy's been a terrific performing sector year to date, but we're not really seeing the relative outperformance the way we'd seen back in 2015, 2016. So I look at this as really a relief rally."

Like Johnson, BK Asset Management's Boris Schlossberg is staying away from the sector. He argues that at this point energy's fate is tied to a potential trade deal with China.

"I think the XLE story is really the China oil story at this point," he said. "An unwind of the China trade story" would be "very bad for oil, very bad for XLE."

While he warns not to "touch" it until and unless there's "positive news" regarding trade relations, he believes that if an agreement were reached, the XLE could shoot higher — at least temporarily.

"When we do have news on that front that is positive, I do think there will be a short-covering rally and then the XLE will be a very good buy," he said.