When Ed Beltram took early retirement in 2001 from what was then Lucent Technologies, he seemed to be in good financial shape.

The native Sooner and Oklahoma University grad had worked in communications for Lucent and its predecessor, Western Electric, since 1970. He had accrued three decades’ worth of pension benefits and a 401(k) defined-contribution plan that held a nice chunk of company stock.

Lucent was one of the high flyers of the 1990s. From a share price of around $7.50 after its 1996 spin-off from the old AT&T, Lucent soared to a peak of $84, giving it a market value of $258 billion.

But Beltram held on until the stock dropped to $3 a share. Then he sold. “It has taken me until just this year to get back to the level I was at in my 401(k) in the late 1990s, early 2000s,” he said ruefully.

Unlike many other retirees, however, Beltram also had a traditional pension to rely on while his 401(k) recovered.

So when Alcatel-Lucent (the firm was sold to the French telecommunications giant in 2006) recently offered tens of thousands of retirees a one-time, lump-sum payout instead of their regular pension checks, Beltram said no.

“I recognize that I don’t have the market experience that would allow me to make the kind of investments that would provide me the revenue stream my previous pension did,” he told me.

Beltram, vice president of communications for retiree-advocacy group the National Retiree Legislative Network (NRLN), learned his lesson the hard way. Millions of others have had similar setbacks, or worse.

Many in Beltram’s age group (he turns 70 this year) spent their entire careers working for big companies and expected to live out their days on Social Security and their former employers’ steady, but often modest, pension checks.

The generations from the late baby boomers to Generation X to millennials live in a different world, where only a few big companies still offer traditional pensions. For the vast majority of retirees, 401(k)s and other defined-contribution plans are, besides Social Security, the only game in town.

“This is the first generation that does not have a defined-benefit plan to fall back on,” said Alicia Munnell, director of the Center for Retirement Research at Boston College. “We’ve gone from defined-benefit plans where the employer made all the decisions and took all the risks to where employees make all the decisions and take all the risks.”

Tectonic shifts in the global economy, technology, Wall Street and corporate governance have led employers to steadily abandon the risks and responsibility of providing retirement security to their employees, leaving many retirees at the mercy of a patchwork retirement system in a broken market.

In this part of MarketWatch’s “Fixing the Market” series, we’ll show how:

Starting in the 1980s, changing accounting standards, growing pressures to reduce risk and demands by Wall Street for higher shareholder returns pushed companies to reduce, change and shed pension liabilities, shifting the risks to employees.

Retirees who had to manage their own financial assets repeatedly made bad decisions — taking lump-sum payouts instead of annuity-income streams, investing too much in their own companies’ stock, taking early withdrawals or contributing too little to their plans. Many squeezed American workers also simply don’t have the extra money to save for retirement.

Nonetheless, some promising alternatives are emerging because of a major overhaul in federal retirement law and initiatives by companies and investment firms to provide pension-like alternatives to retirees.

The first pension plan

Private employers have provided retirement plans since American Express started one in 1875. In the early-20th century, booming industries like railroads and steel manufacturing offered pensions to get employees to work in those massive enterprises. In the 1930s, the landmark Social Security Act enshrined the principle of retirement security in federal law.

During World War II, wages and prices were frozen and employers attracted workers by offering pensions (and health-care coverage). That continued when the GIs returned, and unions, then at their apex, pushed for more pension benefits along with higher wages.

The next watershed event was in the 1980s, when 401(k) plans were created as a way for companies to offer deferred compensation to executives. But they really took off when Congress and the Internal Revenue Service ruled they must cover all employees. As of 2014, there were 52 million active participants in 401(k) plans, which had $4.6 trillion in assets, according to the Investment Company Institute (ICI).

The 401(k) was the right product at the right time. The Silent Generation of the 1950s, who expected to stay at one company their whole careers, was being supplanted by restless baby boomers, who liked the new retirement accounts’ portability and the freedom to choose their own investments.

But it was particularly good for corporate managers and the benefits consultants who advised them. The bull market of the 1980s coincided with radical changes in markets and corporate governance.

Leading thinkers from Nobel laureate Milton Friedman to Michael Jensen of Harvard University advised CEOs that maximizing profit should be a corporation’s predominant goal and executives’ compensation should reflect stock performance. That theoretically aligned managers’ incentives more with shareholders, but it also tied their pay — and thinking — more closely to short-term moves in their company’s stock.

Pension liabilities thus became a huge burden. Big moves in interest rates or a bear market in stocks could suddenly make those plans look underfunded — and raise the amount companies had to contribute.

The Financial Accounting Standards Board (FASB) ruled that public companies had to disclose those obligations as liabilities on their balance sheets. Sudden increases in pension liabilities could hurt earnings and the stock price, hitting shareholders and executives in the pocketbook.

“When you can’t control the market and market rates, and you have to put the volatility on your balance sheet, that can affect your quarterly earnings,” said David Certner, legislative counsel for AARP. “Most employers don’t want to take on the expense or the volatility associated with these plans.”

Hence the popularity of 401(k)s, which don’t carry traditional pensions’ risks and liabilities. The less executives had to worry about those long-term obligations, the more they could deal with rapid technological change, globalization, and “activist” investors and Wall Street demanding immediate stock market returns.

“Mergers and acquisitions and globalization totally changed the thought processes of corporate pension fund managers,” said Bill Kadereit, president of NRLN.

‘Retirement Heist’

The move from traditional pensions to 401(k)s really gathered steam in the 1990s, when, as Ellen Schultz of The Wall Street Journal noted in her 2011 book, “Retirement Heist,” many giant companies “had large aging workforces … entering their peak earning years, and … their pensions were about to spike.”

“It didn’t take benefits consultants long to realize that every dollar a company had promised a retiree … was the equivalent of a dollar that could potentially be added to a company’s income,” she wrote. “Suddenly the $1 trillion that companies owed three generations of employees and retirees for pensions and retiree health benefits became potential earnings enhancements.”

So it’s no wonder only 18% of private-sector employees participated in defined-benefit plans in 2011, roughly half the 35% that had them in 1990-1991, according to the Bureau of Labor Statistics. (Some 78% of public-sector workers still have traditional pension plans.)

The dot-com bust and bear market of 2000-2001 became “the catalyst for individual companies changing their pension plans” at an even more rapid pace, said Boston College’s Munnell, who is also co-author of the book “Falling Short: The Coming Retirement Crisis and What to Do About It.”

The Pension Rights Center, a Washington, D.C.-based advocacy and lobbying group, lists more than 180 companies, from IBM and Verizon to the Boston Red Sox and the Los Angeles Angels of Anaheim, that have frozen, terminated or significantly changed their pension plans since 2005.

In 2012 alone, nearly 200 companies offered lump-sum payments to retirees and to people not yet receiving benefits, according to human-resources-consulting firm Towers Watson. A lump-sum payout is effectively a “buyout” of the company’s future pension obligations — often at a discount.

The industry buzz word for those actions is “de-risking.” Although companies might shell out millions immediately, they rid themselves of future risk and liabilities.

But the risk doesn’t just disappear; it’s transferred from shareholders and executives to employees and retirees, who often are poorly equipped to handle it.

“If the retirees take their pensions as a one-time lump sum, the [companies’] obligation for their pensions falls to zero,” wrote Schultz. “Lump sums also shift longevity risk to retirees, as well as investment risk, interest-rate risk and inflation risk.”

In July, the IRS prohibited employers from offering lump-sum payouts to retirees and survivors already receiving pension payments.

Money management mistakes

Retirees’ difficulties go far beyond what to do about lump-sum payments. Decades of research show that individual investors generally do a terrible job of managing their own money.

“There’s limited ability to make complicated investment decisions,” said Moshe Milevsky, a retirement expert at York University in Toronto.

Brad Barber of UC Davis and Terrance Odean of UC Berkeley have found that only 1% of all active traders beat the market; in fact, the more they trade, the worse they do.

And, Barber and Odean wrote, “the evidence indicates that the average individual investor underperforms the market — both before and after costs.” They estimate the gap may be as high as 4 percentage points a year.

That’s in line with the estimates of Boston-based financial-services research firm Dalbar that investors’ equity fund returns lagged the S&P 500 index by 4.2 percentage points annually over 20-year periods.

“Attempts to correct irrational investor behavior through education have proved to be futile. The belief that investors will make prudent decisions after education and disclosure has been totally discredited,” Dalbar wrote almost despairingly in a report published last spring.

Why? According to Barber and Odean, individuals are generally overconfident about their investing ability; trade too frequently; don’t hold truly diversified portfolios; chase performance and “hot” stocks; and sell their winners while holding their losers.

But people make even more basic mistakes:

One in four participants in 401(k) plans didn’t contribute enough to get a matching contribution from their employers.

One in four raided their 401(k) plans for cash before retirement, despite often having to pay a 10% early withdrawal penalty and taxes at ordinary-income rates.

One in five workers eligible to invest in 401(k) plans don’t contribute at all.

A big reason for that may be that many Americans simply don’t have the money to invest for retirement. And some 57 million American workers don’t have a retirement plan at work at all.

No wonder the median account balance of Americans saving for retirement is less than $60,000, according to the Federal Reserve, and nearly half of all working households have no retirement savings whatsoever.

Individual investors’ proven difficulties in managing their own money are why most unbiased finance experts recommend a pension-like vehicle for retirees.

“I’m a big fan of an annuity that looks like, smells like, tastes like a pension,” York University’s Milevsky said.

Nearly two-thirds of retirees rely on the most popular annuity of all, Social Security, for most of their retirement income. Though it’s wildly popular, Social Security faces serious financial challenges and may need to be restructured.

Recent reforms

But some new solutions may give investors a better shot at retirement security, although they could reduce their freedom of choice and take away some government protection.

The Pension Protection Act of 2006 included several significant reforms, such as:

. Employees in company 401(k) plans would be enrolled automatically at a default savings rate. . That money could be invested directly in target-date retirement funds, which give employees an age-appropriate, diversified investment portfolio. . Workers who don’t want to participate — roughly 30% of those who are offered 401(k) plans — would have to affirmatively opt out of the plans.

“We had this big transition from defined-benefit to defined-contribution plans. These 401(k) plans weren’t working very well,” said Certner of AARP. “We ended up coming up with auto enrollment as a way of making them more defined-benefit-like.”

Defaulting employees into target-date funds prevents them from putting all their money into no-return money market funds — or hot stocks — while also giving them a diversified portfolio that allows their retirement nest eggs to grow over time.

Target-date funds are funds of funds that gradually reduce stock exposure and increase bond holdings as people approach retirement age. Vanguard projects 80% of its new plan participants will be entirely in target-date funds by 2018.

During the 2008 market crash, 401(k) investors were four times less likely to trade out of target-date funds than 401(k) investors in other equity funds, Barron’s reported.

“These changes reflect a victory for ‘behavioral finance’ by embracing the power of inertia,” wrote Jack VanDerhei of the Employee Benefit Research Institute (EBRI), adding that the law “encourages sponsors to set up 401(k) plans in a fashion that helps workers to help themselves simply by doing nothing.”

Target-date fund expenses are falling, although they can still range widely from Vanguard’s 0.17% to above 1%. According to the ICI, investors in equity mutual funds in 401(k) plans paid 0.58% of assets in management fees in 2013, 25% below the average expense ratio in 2000.

But 401(k)s have such an opaque fee structure — estimates on total expenses range from 1% to 2% of assets — that 80% of Americans don’t know how much they’re paying.

In May, the U.S. Supreme Court ruled unanimously that employers have a “continuing duty” to monitor fees in their 401(k) plans or face litigation from plan participants.

But though target-date funds are pretty good vehicles for saving for retirement, retirees need steady income to cover their expenses.

Last October, the Treasury Department and the IRS said retirement-plan sponsors could “include deferred income annuities in target date funds used as a default investment.” Although it’s unclear how popular they will be, they would give plan participants the option to set up their own annuity-funded “pensions.”

Meanwhile, “because of paralysis at the federal level, a lot of states have tried to expand coverage,” said Karen Friedman, policy director of the Pension Rights Center.

For example, a recent California law requires businesses with more than five workers to enroll employees automatically and deduct roughly 3% of their wages in an IRA-like retirement plan run professionally by the California Public Employees’ Retirement System (CalPERS) or a similar group.

Pension-like annuities

There are some promising private-sector solutions as well.

In 2012, General Motors paid $3.5 billion to $4.5 billion to buy annuities from Prudential for more than 40,000 former salaried employees who chose not to take a lump-sum payment. GM thereby shed $26 billion in pension liabilities, shifting the risk to Prudential. Verizon made a similar move the same year. Only a handful of companies have followed their lead, possibly because of the upfront costs.

Some retiree advocates worry that buying annuities from an insurance company instead of providing a pension would leave retirees at the mercy of the insurer’s financial stability. (Prudential’s insurance and annuities companies were recently rated A+ by A.M. Best.)

Participants also wouldn’t get some legal protections under the Employee Retirement Income Security Act of 1974 (ERISA) or Pension Benefits Guaranty Corp., and instead would have to seek redress from state insurance regulators.

Those reservations aside, the insurance-company annuity acts just like an old-fashioned company pension, providing a steady, lifetime income stream.

Some experts think 401(k)s are actually an improvement over traditional pension plans. “The effect of the move to defined contribution is that there are far more Americans who are getting benefits out of this system than ever did before,” said Dallas Salisbury, president of EBRI.

“In the good old days, all of that money went to a small number of employees who worked a long time for that company. Now that wad of money is being spread more widely,” he told me.

Yes, the “good old days” are gone forever. But little by little, the broken U.S. retirement system is being repaired, and a new one is emerging, in fits and starts, from the remnants of the old.

Howard R. Gold is a MarketWatch columnist and founder and editor of GoldenEgg Investing, which offers free market commentary and simple, low-cost, low-risk retirement investing plans. Follow him on Twitter @howardrgold.