In order to own our existing home without a mortgage, we could try to sell our other investments. But that would create a much higher net worth allocation towards real estate than desired. (Our goal is to keep real estate expenses to no more than 35% of our income.) Furthermore, selling our investments would trigger a tax liability on capital gains.

According to a report from the California Association of Realtors , you now need a minimum household income of $309,400 to afford a median-priced home in San Francisco, which is around $1.6 million . So even with our annual passive income of $250,000, we're still short nearly $60,000 per year.

Fast forward to the end of 2019, and my wife and I are no longer comfortable living the early retirement lifestyle, especially in a big, expensive city like San Francisco.

We traveled to more than 20 countries. I spent time doing things I enjoyed, like coaching a high school tennis team and writing on Financial Samurai , the personal finance website I started in 2009.

For seven years, I lived a charmed life in early retirement with my wife, who also retired from her finance job three years after I did. Together, we earned roughly $250,000 in passive income streams per year — mostly from dividend-paying stocks, interest from savings, municipal bonds, and rental income.

In 2012, I decided to quit my six-figure job in investment banking and retire at 34 . I had amassed a net worth of about $3 million that generated roughly $80,000 in investment income per year.

When I left my job (and even got a nice severance package that paid out all my deferred stock and cash compensation), I thought I could retire and be set for life. I was wrong, and I'm not afraid to admit it.

If my wife and I don't take action soon, we'll no longer be able to live comfortably off of our passive income streams in San Francisco.

Here are the main reasons I now need to retire from early retirement:

1. We had a child.

My wife and never planned on becoming parents, but that changed in 2017, when we were blessed with a baby boy.

Retiring without children is like a walk in the park compared with retiring with children. In addition to costs for diapers, clothes, toys, baby food, and occasional babysitting, our biggest concern is paying for our son's education, especially now that he's in preschool.

In San Francisco, the public school system starting in kindergarten is based on a lottery system, so even if you pay property taxes, your child isn't guaranteed a spot in your neighborhood schools. Most parents are forced to pay big bucks to send their kids to private school, but even paying $30,000 or more doesn't guarantee admission to top-rated private schools.

Don't get me wrong: Fatherhood has been the most rewarding experience in my life — and we're going to do whatever it takes to support our son's needs and give him access to as many opportunities as possible.

2. I underestimated how low interest rates would go.

I'm a believer of "low interest rates for life," but I didn't think the 10-year bond yield would ever drop to below 1.5% in 2019. I thought we'd stay around 2.5%.

Now, instead of only needing $2 million in additional capital to generate $50,000 at 2.5%, I need $2.5 million in capital to generate the same $50,000 in passive income at 2%. At 1.5%, the required capital to generate $50,000 in passive income is over $3.3 million. Seeing such a large shift in the goalpost when you don't want to take more investment risk is disconcerting.

The only way I benefited from the recent interest rate decline was by refinancing my mortgage down to 2.6%. My cash flow is now about $13,000 greater per year. While the increase helps, it still doesn't offset the expected investment income decline.

3. Rising health insurance premiums.

In 2019, the monthly premium on our health insurance was $1,820. But last month, I received notice that our premium will rise to $1,940 in 2020. That's $23,280 per year in annual health insurance premiums, excluding any co-pays and co-insurance.

This will be an even bigger problem if we decide to have more children later on, because the 2020 amount would go up by another $440 (to $2,560) per month.

I've finally reached a point where paying so much in health insurance premiums feels like highway robbery, especially since we're a healthy family with no pre-existing conditions. While I understand that it's our duty to help subsidize others who are less healthy, the cost has become untenable.

4. The bliss of early retirement didn't last as long as I thought it would.

Retiring early improved my quality of life in ways I never would have imagined: I was able to fully commit to fatherhood, my relationship with my parents strengthened, my aging slowed, and I even became more self-sufficient.

After a few years, however, the bliss of taking a permanent vacation went away. I started getting the itch to do something more productive than playing tennis and sleeping in.

Also, when you have to start paying $2,000 per month for preschool, not using your six to eight hours of free time finding ways to cover that cost just doesn't feel right.