Says one insider who works on Duck Dynasty, "People who work on the show feel like it's a big pissing match and there is no way that anyone can win."

The current plan is to include Robertson in the upcoming fifth season, set to premiere Jan 15 on A&E. Season four had wrapped production before the controversy began. "Phil might be diminished but there's no way to cut him out altogether," says a source.

But can the family really walk away? Will a fifth season even happen?

The network owns the series and all of the intellectual property behind it, for at least one more season. But as one source points out, "The family could do appear on another network once their exclusivity is up with A&E. Under a normal contract, that usually means anywhere from six months to a year after the final episode has aired. However, if the family breaks their agreement with the network, the network could hold them for longer."