W3C took care of that in 1997 explaining how frames should be implemented in "Implementing HTML Frames":

Any frame that attempts to assign as its SRC a URL used by any of its ancestors is treated as if it has no SRC URL at all (basically a blank frame).

Iframe recursion bug/attack history

As kingdago found out and mentioned in the comment above, one browser that missed to implement a safeguard for this was Mozilla in 1999. Quote from one of the developers:

This is a parity bug (and a source of possible embarrasment) since MSIE5 doesn't have a problem with these kinds of pages.

I decided to dig some more into this and it turns out that in 2004 this happened again. However, this time JavaScript was involved:

This is the code, what causes it: <iframe name="productcatalog" id="productcatalog" src="page2.htm"></iframe> directly followed by a script with this in it: frames.productcatalog.location.replace(frames.productcatalog.location + location.hash); ... Actual Results: The parent window gets recursively loaded into the iframe, resulting sometimes in a crash. Expected Results: Just show it like in Internet Explorer.

Then again in 2008 with Firefox 2 (this also involved JavaScript).

And again in 2009. The interesting part here is that this bug is still open and this attachment: https://bugzilla.mozilla.org/attachment.cgi?id=414035 (will you restrain your curiosity?) will still crash/freeze your Firefox (I just tested it and I almost crashed the whole Ubuntu). In Chrome it just loads indefinitely (probably because each tab lives in a separate process).

As for the other browsers: