To anyone looking at the biggest of pictures, 1981's rates are clearly a one-off aberration. Even Greenspan would agree that we're not going back to 15 percent 10-year yields. At the same time, those 15 percent rates are clouding his worldview — suggesting an overly-wide spectrum of possibility. To him, a move up to 5 percent to 6 percent on the 10-year might be well-within the realm of possibility. To anyone paying closer attention, it's nearly impossible.

In fact, it's highly unlikely that any of us will see rates that high in our lifetimes.

To explain why, let's make sure we understand several of the key reasons rates can move higher. Right off the bat, we can throw out "credit quality." While the U.S. may have the occasional government shutdown or debt ceiling debate, we're far from causing foreign investors any real concerns about not getting their money back. This is a non-issue, even for bond bubble prognosticators.

How about the core of Greenspan's argument: supply and demand? The fear is that as central banks begin to unwind their balance sheets that there won't be enough demand to soak up all the new supply. There is a glaring error in this conclusion and it has to do with the assumption that the Fed would return to its pre-crisis balance sheet levels under $1 trillion.

Let's face it. More than a few bond gurus have egg on their face because they failed to appreciate just how low central bank demand could push rates — and for how long. It's only natural for them to freak out (relatively) about the Fed removing some of that demand. "Won't be fooled again," and all that! But they need to pay closer attention to what the Fed has actually been saying.

In her recent congressional testimony, current Fed Chair Janet Yellen was quick to dismiss the notion of the balance sheet falling below $1 trillion. In June, Fed Governor Jerome Powell said he doesn't see it falling below $2.5 trillion, and in May, J.P. Morgan economists saw it falling to $3.0 trillion by 2021.

You might think J.P. Morgan's estimate is too high. After all, that's only a 33 percent reduction, but consider this: As a percentage of GDP, the balance sheet has been DECLINING since 2014. By the time the Fed expects "normalization" (2022, according to Yellen herself), a $3 trillion balance sheet would actually be 50 percent smaller in terms of GDP, even if growth is only mediocre. That's not even really a removal of the proverbial punch bowl — more like putting out individual bottles of punch and a note that says "gone to bed, but feel free to keep partying. Please clean up when you're done."

Such a move would be consistent with the Fed's recent track record. They've been anything but fast and aggressive when it comes to removing accommodation (to a fault, many have argued). Still, let's put this analysis to the test by asking what would happen if the Fed suddenly broke from character and smashed the proverbial punch bowl with a sledgehammer.

If the Fed rapidly returned to balance sheet levels under $1 trillion, rates would move higher at first, sure, but the economic impact would be severe.

Investors have been waiting (and waiting) for some grand cue that it was finally time for stocks to undergo the big selling spree that has thus far failed to materialize. There could scarcely be a better cue than the Fed reaching for its sledgehammer.

Money would flood out of equity markets and investors would seek safe havens. The newly stellar interest rate returns in the bond market would be an ideal location. If anything, the net effect of the sledgehammer approach would be a run on bond markets. At the very least, the supply/demand argument for a bubble is questionable.

After ruling out the sledgehammer or credit quality issues as motivations for a rate spike, we're left with the old school, quintessential foundation of rates: growth and inflation.

Inflation is alive and well in the form of housing/rental prices and medical expenses. But push any more onto consumers' plates and it'll be hard to swallow. Cracks are already starting to emerge in some sectors, such as auto financing.

Today's society can't drive inflation the way it did when real wages, savings, net worth, property values, and the general notions of hope and prosperity were all on the up and up. Either you or your parents bought a house in the early 80s with rates near 20 percent and they were stoked about it! That world is gone forever — never to return. People who think we're capable of a bond bubble bursting have that idealistic image of bygone prosperity somewhere in the back of their mind, poisoning their objectivity with respect to the current reality.

I'd personally be shocked if we see 10-year yields over 4 percent in our lifetimes. Moreover, and more to the point of immediate risks, I'd be stunned if we break 3 percent by the end of the year. Why?

Perhaps more important than all of the above is the fact that traders aren't idiots. They know where we are and what the Fed would like to do. They know about the detailed plan for the balance-sheet reduction. They know the European Central bank would also like to remove accommodation. They know that all creates net upward pressure on rates. And guess what? We've seen net upward pressure on rates already for all of those reasons.

People who see a bond bubble breaking aren't giving bond markets credit for ALREADY pricing in that removal of accommodation — at least to some extent. Could there be more pain to follow once triggers are actually pulled? You bet! But is it going to be a big, dramatic, bubble-bursting sort of thing? Not likely.

We're talking about 10-year yields maybe breaking above the 2.5 percent to 2.6 ceiling from earlier this year before rallying back down to the 2.1 percent to 2.2 percent range at some point in the next few months. This would translate to mortgage rates moving up from 4 percent currently to roughly 4.5 percent at worst. It's just as likely we'd only see a rise to 4.25 percent.

Commentary by Matthew Graham, chief of operations at mortgagenewsdaily.com. His passion is teaching both consumers and mortgage professionals about bond markets and rate movement. Twitter: @MG_MBS.

For more insight from CNBC contributors, follow @CNBCopinion on Twitter.