$\begingroup$

In mathematics we introduce many different kinds of notation, and sometimes even a single object or construction can be represented by many different notations. To take two very different examples, the derivative of a function $y = f(x)$ can be written $f'(x)$, $D_x f$, or $\frac{dy}{dx}$; while composition of morphisms in a monoidal category can be represented in traditional linear style, linearly but in diagrammatic order, using pasting diagrams, using string diagrams, or using linear logic / type theory. Each notation has advantages and disadvantages, including clarity, conciseness, ease of use for calculation, and so on; but even more basic than these, a notation ought to be correct, in that every valid instance of it actually denotes something, and that the syntactic manipulations permitted on the notation similarly correspond to equalities or operations on the objects denoted.

Mathematicians who introduce and use a notation do not usually study the notation formally or prove that it is correct. But although this task is trivial to the point of vacuity for simple notations, for more complicated notations it becomes a substantial undertaking, and in many cases has never actually been completed. For instance, in Joyal-Street The geometry of tensor calculus it took some substantial work to prove the correctness of string diagrams for monoidal categories, while the analogous string diagrams used for many other variants of monoidal categories have, in many cases, never been proven correct in the same way. Similarly, the correctness of the "Calculus of Constructions" dependent type theory as a notation for a kind of "contextual category" took a lot of work for Streicher to prove in his book Semantics of type theory, and most other dependent type theories have not been analogously shown to be correct as notations for category theory.

My question is, among all these notations which have never been formally proven correct, has any of them actually turned out to be wrong and led to mathematical mistakes?

This may be an ambiguous question, so let me try to clarify a bit what I'm looking for and what I'm not looking for (and of course I reserve the right to clarify further in response to comments).

Firstly, I'm only interested in cases where the underlying mathematics was precisely defined and correct, from a modern perspective, with the mistake only lying in an incorrect notation or an incorrect use of that notation. So, for instance, mistakes made by early pioneers in calculus due to an imprecise notion of "infinitesimal" obeying (what we would now regard as) ill-defined rules don't count; there the issue was with the mathematics, not (just) the notation.

Secondly, I'm only interested in cases where the mistake was made and at least temporarily believed publically by professional (or serious amateur) mathematician(s). Blog posts and arxiv preprints count, but not private conversations on a blackboard, and not mistakes made by students.

An example of the sort of thing I'm looking for, but which (probably) doesn't satisfy this last criterion, is the following derivation of an incorrect "chain rule for the second derivative" using differentials. First here is a correct derivation of the correct chain rule for the first derivative, based on the derivative notation $\frac{dy}{dx} = f'(x)$:

$$\begin{align} z &= g(y)\\ y &= f(x)\\ dy &= f'(x) dx\\ dz &= g'(y) dy\\ &= g'(f(x)) f'(x) dx \end{align}$$

And here is the incorrect one, based on the second derivative notation $\frac{d^2y}{dx^2} = f''(x)$:

$$\begin{align} d^2y &= f''(x) dx^2\\ dy^2 &= (f'(x) dx)^2 = (f'(x))^2 dx^2\\ d^2z &= g''(y) dy^2\\ &= g''(f(x)) (f'(x))^2 dx^2 \end{align}$$

(The correct second derivative of $g\circ f$ is $g''(f(x)) (f'(x))^2 + g'(f(x)) f''(x)$.) The problem is that the second derivative notation $\frac{d^2y}{dx^2}$ cannot be taken seriously as a "fraction" in the same way that $\frac{dy}{dx}$ can, so the manipulations that it justifies are incorrect. However, I'm not aware of this mistake ever being made and believed in public by a serious mathematician who understood the precise meaning of derivatives, in a modern sense, but was only led astray by the notation.

Edit 10 Aug 2018: This question has attracted some interesting answers, but none of them is quite what I'm looking for (though Joel's comes the closest), so let me clarify further. By "a notation" I mean a systematic collection of valid syntax and rules for manipulating that syntax. It doesn't have to be completely formalized, but it should apply to many different examples in the same way, and be understood by multiple mathematicians -- e.g. one person writing $e$ to mean two different numbers in the same paper doesn't count. String diagrams and categorical type theory are the real sort of examples I have in mind; my non-example of differentials is borderline, but could in theory be elaborated into a system of syntaxes for "differential objects" that can be quotiented, differentiated, multiplied, etc. And by saying that a notation is incorrect, I mean that the "understood" way to interpret the syntax as mathematical objects is not actually well-defined in general, or that the rules for manipulating the syntax don't correspond to the way those objects actually behave. For instance, if it turned out that string diagrams for some kind of monoidal category were not actually invariant under deformations, that would be an example of an incorrect notation.

It might help if I explain a bit more about why I'm asking. I'm looking for arguments for or against the claim that it's important to formalize notations like this and prove that they are correct. If notations sometimes turn out to be wrong, then that's a good argument that we should make sure they're right! But oppositely, if in practice mathematicians have good enough intuitions when choosing notations that they never turn out to be wrong, then that's some kind of argument that it's not as important to formalize them.