The tensions unsettling the Saudi royal family became clear in September, when Joseph Westphal, the U.S. ambassador to Riyadh, flew to Jiddah to meet Crown Prince Mohammed bin Nayef, nominally the heir to the throne. But when he arrived, he was told that the deputy crown prince, a brash 30-year-old named Mohammed bin Salman, wanted to see him urgently.

The ambassador was redirected. The United States and the crown prince swallowed the embarrassment.

Palace intrigue is a staple of monarchies, but it is impossible to overstate how out of character such a generational power play was for the desert kingdom. Robert Lacey, in his classic 1981 book, “The Kingdom,” described the tradition of deference that has held the Saudi royal family together through feast and famine: “Deference to elders is one of the Al-Saud’s inviolable ground rules, the best corset they know to discipline the outward thrust of so many assembled appetites.”

Not anymore: Starting in January 2015 with the accession of King Salman, Saudi Arabia has been shaken by the bold reform campaign of his son, known at home and abroad by his initials, MBS. By outmaneuvering and sometimes defying his elders, the young deputy crown prince has turned the politics of this conservative, sometimes sclerotic monarchy upside down.

MBS is the kind of prince that Machiavelli might conjure. He’s a big, fast-talking young man who dominates a room with the raw, instinctive energy of a natural leader. But his hardball tactics have offended some Saudis — especially his rebuffs of Mohammed bin Nayef, his elder at 56 and his nominal superior. In addition to detouring the U.S. ambassador, MBS is said to have engineered the firing of the crown prince’s closest aide in September.

New heirs to the Saudi throne Prince Salman became king on Jan. 22 and has moved the succession to the next generation.

(Cristina Rivero/The Washington Post/Sources: Washington Institute for Near East Policy, staff reporting)

Saudi internal politics are famously difficult for outsiders to understand, but it’s important now to make the effort. The Middle East is at a dangerous inflection point, with sectarian war and terrorism ripping countries at their seams. A stronger Saudi Arabia could make a big difference; a weaker one could add to the chaos. The following account is based on dozens of conversations over the past year with Saudi princes and ministers who support MBS, Saudi princes and advisers who are skeptical about MBS, and top U.S., European and Arab diplomats and intelligence experts. I’ve interviewed multiple sources for each strand of the narrative.

If “Game of Thrones” were set in the Arabian desert, it might have a plot like what has developed in Saudi Arabia over the past 18 months. Anonymous letters have circulated; whispering campaigns have swirled around the deputy crown prince and his rivals. President Obama has advised his aides to avoid any appearance of taking sides. But the president’s White House meeting on June 17 with MBS, which treated him almost like a head of state, may have cast an implicit vote of support for the reformer’s agenda.

It’s hard not to root for a young leader who seeks to transform a country whose conservatism and religious fundamentalism have been obstacles to change in the Muslim world for generations. A half-dozen prominent Saudi-watchers who have met MBS told me they think he has the potential to rebuild Saudi Arabia into a more dynamic country that’s much more able to protect its security and that of its neighbors. But many worry that he’s also capable of driving his country off a cliff with his headstrong, sometimes reckless behavior.

The stakes for the United States are enormous. For more than 50 years, the oil kingdom has been a key strategic ally, but also a continuing cause of concern. Saudi vulnerability to internal and external attack pushed huge U.S. defense spending in the Persian Gulf and fueled two Gulf wars. Saudi oil was a blessing, but sometimes, as in the 1973 oil embargo, a weapon against the United States. The kingdom was a partner against terrorism, but its “Salafist” brand of Islam inspired many of the extremists. Americans haven’t forgotten that Osama bin Laden was a Saudi and that 15 of the 19 men who carried out the 9/11 attacks were as well.

A reformer who could set Saudi Arabia on a more modern and stable path could be a game-changer for the United States and the Arabs. By appealing to disaffected Arab young people, a Saudi reformer could encourage a renaissance in a Sunni Arab world that has been shattered by civil war, terrorism and sectarian hatred. That’s the promise of MBS. The danger is that his impulsive “Prince Hal” behavior could cause Saudi Arabia to implode and make all these problems worse.

For a kingdom that has survived by hedging its bets and resisting change, MBS proposes a series of sweeping reforms. Saudi Aramco and other big, state-owned enterprises would be privatized; cinemas, museums and a “media city” would be created for a young population starving for entertainment; the power of the religious police would be curtailed; and, at some point, women would be allowed to drive.

A simple way to explain MBS’s goal is that he would like to make an inward-looking, hyper-cautious Saudi Arabia look more like the neighboring United Arab Emirates, with its dizzying skyscrapers and freewheeling market economy. MBS seems to recognize that this economic transformation won’t be possible without easing Saudi Arabia’s strict Islamic traditions. His reform blueprint, “Vision 2030,” offers a tantalizing but unspecific commitment: “Our vision is a tolerant country with Islam as its constitution and moderation as its method.”

Saudi Crown Prince Mohammed bin Nayef (L) and deputy crown prince. Prince Mohammed bin Salman. (AFP Photo/HO/SPA) (Ho/AFP/Getty Images)

But can the impulsive young prince pull it off? As deputy crown prince, he’s technically the country’s No. 3 official. He’s able to act in the name of his father, King Salman, who’s 80 and reportedly suffers from mild dementia. But the designated heir is Crown Prince Mohammed bin Nayef, who is also interior minister and controls the Saudi internal security force (and is Salman’s nephew). Some senior princes who are uneasy about MBS’s tactics are said to have quietly rallied behind the crown prince.

Many Saudi-watchers fear that the country is nearing an open power struggle between MBS and Mohammed bin Nayef. That puts the United States in an awkward spot, as it wants to maintain good relations with both. MBS wins points as a reformer. Mohammed bin Nayef has been the United States’ most reliable counterterrorism partner for more than a decade, has survived an al-Qaeda assassination attempt and is seen by many U.S. officials as a friend and key ally. U.S. officials don’t want to choose between the two.

*

I first visited Saudi Arabia in January 1981. The kingdom was still traumatized by a near-revolution that had taken place 14 months before, when followers of a radical cleric named Juhayman al-Otaybi seized the Grand Mosque in Mecca for two weeks. The Saudis had put on a show of strength with a conference for Muslims in Taif, near Mecca, but their anxiety was palpable. So was the rot in the Saudi system. I wrote a series of articles for the Wall Street Journal later that year about the pervasive corruption inside the kingdom.

The inner reserve that kept Saudi Arabia together, despite its internal and external problems, was symbolized by Prince Saud al-Faisal, who served 40 years as foreign minister. I interviewed him many times over those decades and found him to be a man who would bend, but not break, when facing change.

Saud made a prophetic comment in our last conversation, in November 2011. We were talking about the unruly revolution known as the Arab Spring sweeping the region. Most Saudi officials were terrified of this revolt, but Saud was calm, even hopeful. “It is a great transformation in the Arab world,” he said. “You can never avoid what the people want, no matter what government you have. . . . We are developing. Maybe not as quick as a revolution, but we are developing in a way that’s stable.”

The courtly Saud couldn’t have imagined the whirlwind now swirling through the royal family. It was symbolic, perhaps, that a memorial gathering in Riyadh some weeks ago to honor Saud’s legacy had to accommodate the deputy crown prince. MBS reportedly had scheduled a meeting of his economic reform council for the same time as the opening of the memorial. Saud’s close family is said to have rescheduled the opening so that there wouldn’t be a conflict.

The Saudi system was built to contain internal dissent. But it’s now facing the greatest test in the kingdom’s history.

How did this Saudi political battle begin? Analysts say the family tensions were building during the reign of King Abdullah, who took the throne in 2005. Before Abdullah died on Jan. 23, 2015, some of his close advisers hoped the succession might pass to his son Mutaib, who commands the Saudi National Guard. But Salman, who was then crown prince, moved quickly (with his son’s help) to claim the crown and consolidate power.

Less than a week after taking control, Salman issued decrees that altered the balance of power in the kingdom. Two of Abdullah’s sons, Prince Turki and Prince Mishaal, were removed as governors of Riyadh and Mecca, respectively. Prince Bandar bin Sultan, the wily former ambassador to Washington who was sometimes dubbed “Bandar Bush” because of his closeness to the two Bush presidents, was ousted as national security adviser.

Salman’s lightning decrees installed Mohammed bin Nayef as deputy crown prince behind then-Crown Prince Muqrin. Perhaps more important, his son was named defense minister and chairman of a new Council for Economic and Development Affairs. MBS had his hands on both the military and economic levers of power.

“All this was planned like clockwork,” says one Saudi who watched the maneuvers from Riyadh. “It was a bloodless coup.” Within a week, two very un-Saudi things had happened: The next generation had been put in the line of succession. And the new king and his son had signaled that they were breaking with the traditional consensual politics of the al-Saud family.

*

MBS is something unusual in modern Saudi Arabia — a throwback to the old, rough-and-tumble politics of the desert.

Most senior princes have been educated in the West and speak fluent English. Saud al-Faisal, the longtime foreign minister who died last year, went to Princeton, and his brother Turki, the former intelligence chief, is a Georgetown graduate. MBS’s older half-brothers, Sultan and Faisal, studied at the University of Denver and Oxford, respectively. These Saudis might wear Bedouin robes, but their hearts often seem to be in the West. Perhaps that’s one reason senior princes have tried so hard to maintain support from the religious elders — to distract the public from how Western the leadership really is.

But MBS was a homegrown scamp. His father, who was known in his youth as the family fixer of the al-Saud clan, is said to see himself in this son of his third wife. Young MBS was “rapacious” in his youth, according to one close observer; Saudis tell stories about his aggressive financial deals and his passion for huge mega-yachts, where he could entertain friends. He didn’t go abroad for college, and though he understands English, he prefers to speak Arabic.

MBS had a mentor in Sheik Mohammed bin Zayed, the military leader of the UAE; he saw the young prince an energetic change agent in a country that needed dynamism. Emirati skepticism toward the Saudi old guard was captured in an October 2009 cable from Richard Olson, then U.S. ambassador to the UAE, that was disclosed by WikiLeaks: “The leadership in Abu Dhabi never misses an opportunity to let visiting senior USG officials know that they regard the Kingdom as run by cantankerous old men surrounded by advisers who believe the earth is flat.”

MBS promised something different. Sheik Mohammed’s mentoring role may also have reflected long-standing tensions with Mohammed bin Nayef and his father, who had also been interior minister. The UAE leader told a visiting American official in January 2003 that “Interior Minister Nayef’s bumbling manner suggested that ‘Darwin was right,’ ” according to another WikiLeaks cable. Nayef’s family took that as likening him to a monkey.

UAE officials encouraged MBS’s ideas about reform — and suggested some prominent consultants, such as McKinsey & Co. and Boston Consulting Group, that had provided advice for the UAE. The consultants were soon at work drafting what became ambitious proposals for reform of every Saudi ministry.

As defense minister, MBS quickly broke from Saudi Arabia’s traditional wariness of military conflict and reliance on U.S. power. Convinced that Iranian meddling in Yemen threatened the kingdom, MBS went on the offensive — and in March 2015, Saudi planes began bombing the Iran-backed Houthi rebels. The UAE helped plan the assault and sent troops and planes.

From the beginning, U.S. officials were skeptical about the Yemen war. So was Mohammed bin Nayef, who is said to have worried that the campaign might strengthen al-Qaeda and the Islamic State. But the campaign rolled forward. The defense minister also talked about sending Saudi troops to Syria, and scenario planning for such an intervention began. But the idea was quickly dropped.

MBS took a decisive step up the ladder in April 2015. A royal decree dumped Muqrin as crown prince; Mohammed bin Nayef moved up to the No. 2 spot, and MBS was installed as No. 3. This change in the official succession plan upset some members of the royal family. Although Muqrin was widely seen as an unsuitable potential king, Saudis worried about the precedent, and the possibility the succession plan might be rejiggered again to install MBS.

The changes were ratified by the Allegiance Council, a sort of House of Saud family board of directors. According to a senior Saudi official who backs MBS, the young prince got more votes from the council than did either Muqrin or Mohammed bin Nayef when their posts were ratified. But some U.S. analysts worry about attempts to steer the voting.

Whatever the vote totals, recent events suggest that the consensual family decision-making that the Allegiance Council was meant to foster is now largely broken. That’s not necessarily a bad thing. Royal-family consensus often produced paralysis in the kingdom. But restless, unhappy senior princes could make trouble for MBS.

MBS has been willing to experiment, even in his dealings with Russia. When MBS visited Washington in May 2015, he told a senior U.S. official that he was suspicious of President Vladimir Putin’s machinations. But he had a chummy meeting with Putin when he visited St. Petersburg the next month. And Russia brokered a visit to Riyadh in late July by the Syrian intelligence chief, Ali Mamlouk, where MBS is said to have explored formulas for ending the Syrian civil war. Nothing resulted from the meeting.

The squeeze on Mohammed bin Nayef’s authority increased through last year, in various ways. The crown prince didn’t have his own “diwan,” the traditional gathering place where senior princes dispense favors and patronage. Instead, he shared one with the king, which meant that it was effectively controlled by MBS.

A decisive blow came in early September when Salman, at his son’s urging, fired Saad al-Jabri, who for years had been Mohammed bin Nayef’s closest adviser. A U.S. source explains what happened: Jabri was coming to the United States on a personal visit, and he decided to see his old friend John Brennan, the CIA director. He didn’t report this meeting to Salman, and when the king learned what had happened, Jabri was removed. One Arab source claims there was also documentary evidence that Jabri had secretly supported the Muslim Brotherhood, but U.S. officials dismiss that allegation as implausible.

The crown prince’s position was publicly undermined by the firing of his confidant, and friends describe him as withdrawn and somewhat passive last fall. He was also suffering from what officials vaguely describe as “health problems.” The crown prince spent six weeks in December and early January resting and recuperating in Algeria.

But Mohammed bin Nayef returned from his Algeria trip with what friends say was renewed commitment. He had worried for months that in the Yemen war, a Saudi proxy force called the al-Islah Party was, in effect, fighting alongside al-Qaeda and the Islamic State. In March, he pressed that argument harder, and it prevailed; the Saudis are said to have adjusted their strategy in Yemen to target the al-Qaeda and Islamic State extremists more effectively.

MBS’s adventure in Yemen seems to be winding down. In April, the Saudis opened negotiations with the Houthis in Kuwait. The United States has sponsored additional back-channel contacts. U.S., Saudi and European officials say that a negotiated truce is likely — not the glorious victory that MBS might have wanted, but a settlement that will stanch the bleeding in Yemen. The MBS camp boasts that the war showed that Iran is a “paper tiger.”

Can the fragile balance last, with the crown prince and deputy coexisting under an elderly, ailing king? If Salman should leave the scene, would the crown prince succeed him, as the current succession plan provides? Or would MBS try to jump the queue, with acquiescence from a pliant Allegiance Council?

Saudis don’t know the answer. In January, the rumor mill was buzzing with talk that King Salman might abdicate soon, in favor of his son. U.S. officials scrambled to assess the situation. Salman is said to have signaled that no changes in the succession plan were imminent, easing U.S. worries. Now, the rumors have started again. Last week, one Saudi in Riyadh summed things up this way: “There is high tension, and nerves are on edge.”

*

To appreciate the reformers’ challenge, it helps to have a clear mental picture of Saudi Arabia. This isn’t a tiny emirate like Kuwait, Bahrain or the UAE. It’s a vast sandy expanse, three times the size of Texas. Saudi citizens number more than 20 million, just a few million less than Australia. Nearly half the population is younger than 25. The CIA World Factbook estimates that unemployment among males is at more than 11 percent, while overall unemployment may be as high as 25 percent. It’s a big country, with complicated problems.

Visitors to the kingdom are often struck by the weird combination of modern and pre-modern. Riyadh is a huge city, snarled by traffic, with oversize villas for wealthy Saudis and their families. But it’s bereft of the glitz of a modern metropolis; you can’t find cinemas, nightclubs or art museums. Much of the work in Saudi businesses seems to be done by expatriates from South Asia. Among Saudis, especially the men, there’s a sense of torpor — too much time and too little to do, except eat. Saudi women are dynamic, and increasingly well educated, thanks to the late King Abdullah. But they’re largely invisible in public life.

This is the obstructed society that MBS and his advisers propose to liberate. Their reform ideas begin with the belief that unless the Saudi economy diversifies outside of the oil sector, it can’t possibly create enough jobs to satisfy its restless youth. The economy won’t boom in a country that’s constricted by reactionary clerics, so MBS and his advisers have decided they must take on the religious leadership, too.

The reform agenda is startling. The “Vision 2030” document is a slick brochure that illustrates a Saudi future that will be “vibrant,” “thriving” and “ambitious.” Details come in the 111-page “National Transformation Program” issued this month. It lists specific goals and initiatives to be achieved by 2020 in each of the kingdom’s 24 non-defense ministries and agencies.

For Saudi bureaucrats who are used to a featherbedded existence and who often assign the hard work to expats, recent months have been frantic. One minister told me that MBS drives his ministers and advisers — pushing them to work through the night to complete their agendas.

The Transformation Plan is chock full of the metrics beloved by the management consultants hired by MBS. There are 178 strategic objectives; progress will be measured by 371 “key performance indicators.” The ministries will embark on 543 new initiatives.

The metrics are highly specific: Non-oil revenue will more than triple by 2020. Water and electricity subsidies will be slashed by more than $50 billion. The percentage of Saudis with digital health records will rise from zero to 70 percent. The number of cultural events in the kingdom will rise to 400 annually from 190. The numbers describe a different country.

The biggest piece of MBS’s reform program may be his plan for partial privatization of Saudi Aramco. Supervising this transformation is Energy Minister Khalid al-Falih, a U.S.-educated technocrat who spent 30 years with Aramco and appreciates how the company created the modern Saudi Arabia. The oil giant built the first modern roads, schools, airports, television stations and even magazines. In modernizing Aramco, the kingdom is doing the equivalent of heart bypass surgery.

The world had never seen a privatization of this size: The Saudis reckon that Saudi Aramco’s valuation is between $2 trillion and $3 trillion. MBS and his advisers want to float less than 5 percent of the company to private investors, but even this tiny share could be worth more than $100 billion — which would make it far larger than any previous initial public offering.

Privatization would shake up Saudi Aramco and the welfare-state mentality spawned by the oil boom. “Our business sector has become very lazy,” says a top Saudi source. The deputy crown prince “wants the welfare society’s destruction. He wants Saudi citizens to become more self-reliant. He wants people to be less dependent on the government.”

But what about corruption? In Russia, privatizations created a class of oligarchs who have strangled the economy. Saudi officials say they will be vigilant in preventing a similar corrupting process in the kingdom. But skeptics question whether Saudi Arabia has the legal institutions and transparency to maintain rule of law. U.S. officials say that MBS himself has engaged in some questionable, non-transparent business practices.

MBS has been barnstorming in Washington, California and New York this month to promote the program. He told Silicon Valley entrepreneurs on June 22 that in pushing for change, Saudi Arabia has the benefit of authoritarian rule. “There is an advantage to quickness of decision-making, the kind of fast change that an absolute monarch can do in one step that would take a traditional democracy 10 steps,” he said, according to notes taken by one participant.

*

The biggest question mark is whether MBS can alter the alliance between the House of Saud and the conservative religious establishment. This pact created the Saudi state but has also enfeebled it. He has already made some moves, including a royal decree in April that blocked the power of the religious police, known as the mutaween, to arrest people.

MBS’s advisers promise more such reforms, including more entertainment sites in the kingdom, museums that display Western art, more mixing of sexes in public places and, “very soon,” the opportunity for women to drive. But MBS also seems wary. He doesn’t want to give religious extremists an easy target by moving too quickly. He tells advisers that resistance from the religious leadership can be overcome, “but it takes courage.”

Saudi Arabia will never be the pleasure dome of Dubai, let alone Paris or New York. The simple geographical fact that the kingdom hosts the two holy mosques in Mecca and Medina puts a limit on how much it can emulate the UAE. But MBS seems willing to test the limits.

A prominent Arab official sums up the Saudi challenge this way: “Saudi Arabia has been accused of being too old, too slow and too backward. Finally, we have someone who’s moving forward and changing the country. We need to give him some room to operate.”

The Obama administration, while careful not to take sides in the palace intrigue, seems to agree that the MBS reform agenda offers a chance for the breakthrough that Saudi Arabia needs. But U.S. officials hope that the impulsive and sometimes arrogant young prince doesn’t run so fast that he falls over — and takes the kingdom’s political stability down with him.

Read more from David Ignatius’s archive, follow him on Twitter or subscribe to his updates on Facebook.