TORONTO -- Following a 103-99 road loss to the Toronto Raptors on Wednesday, the Cleveland Cavaliers held a players-only meeting during which LeBron James and James Jones got on the team for its inconsistent play through the Cavs' 11-4 start to the season, multiple sources told ESPN.com.

The loss to Toronto marked Cleveland's fourth defeat at the hands of an Eastern Conference opponent this season -- all of them coming on the road -- and left the Cavs' vocal leaders questioning the team's mental toughness, something that has been a bit of a recurring theme so far this year.

Cleveland was outscored 31-16 from 3:46 remaining in the third quarter until 2:01 remaining in the fourth quarter, as Toronto ran away with the game.

"It's all mindset," James said after the game, still visibly frustrated. "It comes from within. I've always had it; my upbringing had me like that. It's either you got it or you don't."

LeBron James rejects fatigue and injuries as contributors to what he feels is inconsistent play during the Cavs' 11-4 start. Vaughn Ridley/Getty Images

Making matters worse, Cleveland's defense allowed three and-1 buckets to occur in the fourth quarter (and six in the second half), failing to stop the Raptors' attempts from going in even while giving a foul.

"If we're going to foul, we got to make it count," Tristan Thompson said. "I'm not saying hurt anyone, but you can't give up and-1s. That's not acceptable. That's on the bigs. That's on me, K-Love [Kevin Love], Andy [Anderson Varejao], that's on us when we play soft."

Cleveland was equally poor on the offensive end in the fourth, shooting just 5-for-18 as a team in the quarter until J.R. Smith made two meaningless 3-pointers in the final 35.2 seconds with the game already out of reach.

Cavs coach David Blatt absolved his team of any fault after the game, citing the fact that the Cavs were missing four key rotation players in Kyrie Irving (left knee), Iman Shumpert (right wrist), Timofey Mozgov (right shoulder) and Matthew Dellavedova (left calf).

"I thought that we got tired for obvious reasons. Very short-handed. Thought our guys played hard and I thought we ran out of gas," Blatt said. "I thought fatigue played a big part of that, I really did."

James, however, rejected his coach's softer stance.

When asked whether fatigue was a factor, James said, "No. It's not an excuse." When another reporter asked whether injuries were to blame, James repeated, "It's not an excuse."

The win brings Toronto just 1 ½ games behind Cleveland for the top seed in the Eastern Conference. Currently 11 of the 15 teams in the East are .500 or better, while only six teams in the NBA's supposed power conference out West can make that claim.

"There's always been a lot of shade thrown at our conference the last few years, so it's great to see that our side is definitely picking it up and playing at a high level," James said.

With the conference improving around them, James and Jones, who already have delivered halftime speeches this season to turn around stagnant team efforts, are trying to instill a sense of urgency in the group.

"It's indicative of how we've been playing all year," Jones told ESPN.com. "We haven't been consistent. We haven't been playing to the level of physicality and with the sense of urgency that we need to, that we set out to maintain."

The loss snapped a three-game winning streak for the Cavs. Prior to the streak, many of the same comments about a lack of physicality and toughness were being made by the team following a two-game losing streak to Milwaukee and Detroit.

"For us, the season is about getting better," Jones said. "It's way too early in the season for us to even think that we've done anything or we've reached a level where we can't continue to improve. So, we're solid right now, but we have to get better. We're not a team that's chasing mediocrity. Being 'solid' isn't good enough."

While James might have disagreed with Blatt's statement giving the players a pass, he later called the Cavs "well coached." Jones pointed out that what has been troubling the Cavs comes down to the players.

"It's a focus, it's a mentality that we have that we put on display sporadically, but it's something that we have to do consistently," Jones said. "No coach, no scheme, no nothing can do that. It's us as players making that decision to do it every single time out."

The Cavs players, at James' urging, recently eschewed running out for player introductions, instead using that time for their pregame huddle and then immediately taking the court before their opponents could get to the center circle. James explained it was simply the Cavs "getting ready for the main thing," and the motivation was explained by other players to be a "businesslike approach" in hopes of getting off to faster starts. On Wednesday, all the routine did was lead to an awkward scene with the Cavs' starters standing amid a sea of Raptors dancers before tipoff, and it did little, if anything to prepare them mentally.

"It's got to be from tipoff. It's got to be out of the gate. Especially for myself," said Thompson, who started in place of the injured Mozgov. "When we're playing soft, I take that onus on myself. That's what I'm here to do. That's why I'm here. To play physical, make it tough for guys, rebound, set hard screens and be a force in the paint. So, this is totally on me. I've got to be more physical in the paint and protect the paint."

"We have to hold each other more accountable, we have to play better," James said. "And we will. But I don't think we improved tonight."

The loss capped an emotional day for James, as he penned "R.I.P. Ebony" on his sneakers and shared a photo on Instagram to honor Ebony Nettles-Bey, a teenager he befriended while he played for the Miami Heat who lost her battle with stage 4 soft-tissue sarcoma.