Since the early days of Fisher, Haldane, and Wright, a number of newer models for the mutation-selection process have been formulated. On a practical level it does not seem that most population biologists have actually believed that fitness could increase universally, continuously, and without bound. Indeed, the literature has reported many empirical and theoretical studies that indicate that fitness increase can be very problematic, and that fitness decline is a very real possibility for any population. It seems most population biologists have viewed Fisher’s theorem as being simply out of date and of modest historical interest. Yet theorems should not just fade away—mathematically they should be upheld, refuted, or corrected. Our goal is to correct and re-apply Fisher’s Theorem, such that it is consistent with real biology.

Most of the more recent mutation–selection models have a general framework which employs a method for describing all possible genotypes (called the state space), wherein organisms reproduce at rates proportional to the fitness determined by the parent genotype(s), and resulting progeny can be of a different genotype than the parents caused by mutations. Each model has its own set of variables that are studied and its own set of rules governing change over time.

Every model is only an approximation of some isolated subset of reality, and each model is only useful insofar as it: (1) includes the variables and rules to be studied and: (2) the rules governing change in the model accurately approximate the most important factors affecting change in reality. Simple rules make a model more mathematically tractable, but at the cost of utility as a useful model of reality. The general goal is to have rules that are as simple as possible, and yet capture all the driving factors contributing to the phenomena to be studied. In using a model to make general statements about behavior in reality, it is essential to consider the built-in assumptions implicit in the structure of the rules in a model.

Deterministic versus non-deterministic models

Deterministic models are models in which all future behavior is determined by the current state of the system. In non-deterministic models (sometimes called agent-based methods), mutations and Mendelian genetic principles (which are non-deterministic on an individual organism level) are used to determine genetics of offspring from parents. Numerical simulations can simulate change in genetics over time, and there is almost no limit to the complexity of the simulation. As with all numerical simulations, general principles are not derived mathematically as much as are made as general observations from repeated experiments. Non-deterministic agent-based methods are at the extreme end of enabling the most accurate approximation of factors driving genetic change in real biological systems at the cost of little accessibly to proving mathematical principles or laws. Non-deterministic models can be used to explain what is likely (or unlikely) to happen given some underlying set of governing rules, but are unlikely to provide must-happen rules in the form of physical laws that Fisher sought in his fundamental theorem.

The ideal situation in using models to understand reality is to have phenomena that is observable in reality, observable in non-deterministic models, and that has underlying principles provable in deterministic models.

Infinite population models

In this section we discuss models for mutation–selection that come under the general heading of infinite population models. In these models, the population size is held at carrying capacity but the population can be divided into infinitely small subsets. These models also have an explicitly defined state space describing the possible genotypes and mutations occuring but only between genes that are already present. Selection occurs as genetic varieties reproduce and compete at different rates depending on fitness, providing a first-principles approach to selection. These models are important for understanding the selection–mutation process as a population adapts to its environment, but do not provide direct insight into how a population forms new genotypes that may be more fit or less fit than the original population; no genes are lost and no new genes are created.

There are two main explicit models for the state space of genotypes upon which selection–mutation acts in infinite population models: measuring the frequency of alleles present and segregating population into subpopulations, each of which corresponds to a different genotype. Modeling from the allele frequencies is considered a population genetics approach because the focus is on the genetics and is the approach that Fisher used. Modeling each individual subpopulation by genotype is called the quasispecies theory because the population of a species is modeled as a cloud of separate genotypes each of which can mutate to any of the others, and is generally attributed to the work of Eigen in (1971), with the term quasispecies first used in Eigen and Schuster (1977). While the two approaches begin with different foundations, they are more appropriately described as two sides of the same coin as opposed to being entirely different models.

While these two types of models use differing descriptions of the state space, they both use equivalent rules for change over time: organisms reproduce proportional to fitness with mutations. Before introducing equations for specific models from each approach (which vary according to additional assumptions made), we discuss the context for using either approach.

The purpose of the allele-frequency approach (and other models based on the genetic components) is to investigate the change in the underlying genetics over time, as described in Burger (1989):

“Traditionally, models with only two alleles per locus have been treated. At the end of the fifties the first general results for multi-allele models with selection but without mutation were proved. In particular, conditions for the existence of a unique and stable interior equilibrium were derived and Fisher’s fundamental theorem of natural selection was proved to be valid (e.g. Mulholland and Smith 1959; Scheuer and Mandel 1959; Kingman 1961). It tells that in a one-locus multi-allele diploid model mean fitness always increases. It is well known now that in models with two loci or more this is wrong in general”.

An excellent exposition of the population genetics approach is provided in Burger (1989), which includes criteria for cases where a Lyapunov function [a function on the state space that is increasing with respect to time, also called a maximization principle (Hofbauer 1985)] may or may not exist. It is standard results that any system with a Lyapunov function (which may or may not be mean fitness) and a compact state space will have all solutions forward asymptotic to equilibria (Hofbauer 1985). In cases where the population approaches an equilibrium, new mutations tend to decrease fitness but are balanced by the selection process, called the mutation–selection balance. The mutation–selection balance is an important concept for real populations even though there is no guarantee of achieving such an equilibrium.

The objective in the quasispecies theory was originally to investigate error-prone self-replication of biological macromolecules with a focus on the origin on life. This theory has been applied with success to RNA viruses which replicate at high mutation rates and have extremely polymorphic populations (Pariente et al. 2001; Crotty et al. 2001; Grande-Perez et al. 2002; Anderson et al. 2004). The quasispecies approach enables investigation of the distribution of a population across a fitness landscape. Of primary importance is the measurement of when selection acting within a population will adapt effectively around the higher fitness varieties in the landscape vs. when mutations will cause the population to spread out across the landscape.

For a given fitness landscape, the mutation rate separating adaptation from spreading over low-fitness genotypes is called the error threshold. While the Quasispecies Equation always has an equilibrium and the form of the equation holds the total population fixed at carrying capacity, the idea that a high mutation rate causes the population to spread out over lower fitness genotypes has led to effective antiviral therapies in which the increase in mutation rate causes extinction of the population (Pariente et al. 2001; Crotty et al. 2001; Grande-Perez et al. 2002; Anderson et al. 2004). The infinite population models a priori prevent extinction (because of approximating assumptions in the models not present in the biological populations), and models that can exhibit such an extinction due to mutations will be provided in Sect. 2.3.

Although these two approaches use different models for the state space, they are more complementary than at opposition to each other. Because they both model change over time for organisms reproducing proportional to fitness with mutations, the behavior observed in each should at least be compatible. In Wilke (2005), the author explains “I review the pertinent literature, and demonstrate for a number of cases that the quasispecies concept is equivalent to the concept of mutation–selection balance developed in population genetics, and that there is no disagreement between the population genetics of haploid, asexually-replicating organisms and quasispecies theory”. The main difference between the approaches is how the measure that behavior and its effect on underlying variables (specifics of genetic variation in the population in the one case, and the distribution of the population over fitness landscape in the other). The population genetics approach allows the study of the effects of mutation–selection on allele frequencies and the quasispecies enables the study of the effects on the distribution of a population across a fitness landscape.

We now present equations for deterministic models for the mutation–selection process that assume an infinite population. The mutation process in the model is explicitly incorporated by a matrix of values that provide the mutation rate from one genotype (or allele) to a different one. As such, only mutations between pre-existing genotypes (alleles) are considered. Selection occurs via different reproduction rates for the genotypes (alleles) with the total population help at carrying capacity.

The infinite population assumption is implicit in the model; it is present by modelling each genotype frequency (or allele frequency) by a real number between 0 and 1. This makes the population “infinite” because a subset of the population corresponding to a genotype (or allele) can be a nonzero arbitrarily small fraction of the total population. Also implicit in the equations, connected to the infinite population assumption, is that every subpopulation is nonzero for all time making any form of extinction a priori impossible. In addition, these models only permit mutations among genetic varieties that are already present, limiting their utility for modeling ongoing change in a genes, either creating ongoing improvements as Fisher envisioned or building up deleterious mutations in fitness decline.

We start with the single locus case with multiple alleles \(\{A_1,\ldots ,A_n\}\). Denote the frequency of allele \(A_i\) by \(p_i\) and \(u_{ij}\) the mutation rate for allele \(A_j\) to \(A_i\). Denote the fitness of an organism with allele \(A_i\) by \(m_i\), and then the mutation–selection model from Crow and Kimura (1970) is

$$\begin{aligned} \frac{\text {d}P_i}{\text {d}t} = P_i(m_i-\bar{m}) + \sum _j u_{ij} P_j - P_i, \end{aligned}$$ (2.1)

where \(\bar{m}=\sum _i m_i P_i\). In the terminology of Burger (1989), this is the “classical haploid one-locus multi-allele model with mutation and selection”. It is explicitly solvable with a unique forward-time stable equilibrium solution (Burger 1989), which is the mutation–selection balance.

For the one-locus diploid model with alleles \(\{A_i,\ldots ,A_n\}\) and associated frequencies \(\{p_1,\ldots ,p_n\}\), we define the fitness of an organism with allele \(A_iA_j\) to be \(m_{ij}=m_{ji}\). The marginal fitness of allele \(A_i\) is then \(m_i=\sum _{j=1}^n m_{ij}p_j\). The mean fitness is \(\bar{m}=\sum _{i,j=1}^n m_{ij}p_ip_j\). We define \(u_{ij}\) to be the mutation rate for allele \(A_j\) to \(A_i\) for \(i

e j\) satisfying \(u_{ij}\ge 0\) and \(\sum _{j=1}^n u_{ij} =1\) for all j. Then the one locus diploid model given in Crow and Kimura (1970) is:

$$\begin{aligned} \frac{\text {d}P_i}{\text {d}t} = P_i(m_i-\bar{m}) + \sum _j (u_{ij} P_j - u_{ji} P_i). \end{aligned}$$ (2.2)

The dynamics for the diploid model are more complicated (Burger 1989), and can include stable periodic orbits (Hofbauer 1985).

To describe the quasispecies model, we want to define the state space in terms of a (finite) set of different genetic sequences (genotypes). Suppose now that \(P_i\) is the concentration of the \(i\mathrm{th}\) genetic sequence, \(m_i\) is the associated fitness, and that \(Q_{ij}\) is the mutation probability from j to i. Then the quasispecies model as presented in Wilke (2005) is

$$\begin{aligned} \frac{\text {d}P_i}{\text {d}t} = P_i(m_i Q_{ii}-\bar{m}) + \sum _{j

e i} m_j Q_{ij} P_j. \end{aligned}$$ (2.3)

From this form of the equation it is not difficult to see that this is equivalent to Eq. 2.1, with different interpretation of the constants. A more concise form of the quasispecies equation can be obtained by letting

$$\begin{aligned} \frac{\text {d}P_i}{\text {d}t} = \sum _j m_j Q_{ij} P_j - \bar{m}P_i. \end{aligned}$$ (2.4)

This is the form used in Nowak (2006), Chapter 3. This form has the advantage that all the reproduction–mutation creating genotype \(P_i\) is in the first term, \(\sum _j m_j Q_{ij} P_j\), and the second term \({-}\,\bar{m} P_i\) can be seen as the normalization term that keeps the total population constant. As with the haploid model 2.1, the quasispecies model has a unique for stable equilibrium solution.

The quasispecies equation allows examination of the distribution of the population across the fitness landscape. The distribution can be dense around the most fit genotypes (strongly adapted), or it can be spread out to include significant lower fitness genotypes (no adaptation). Selection tends to push the distribution to the higher fitness while mutations work in the opposite direction distribution the population evenly, the error threshold is the highest mutation rate at which adaptation occurs. Adaptation is only possible if the mutation rate per base is less than the inverse of the genome length using appropriate units (Nowak 2006).

The basic idea of Fisher’s Theorem, that mean fitness of a population is always increasing, is valid for the haploid one-locus and quasispecies models, but not for the diploid model. The basic idea of Fisher’s Corollary, that mutations add ongoing new varieties resulting in unbounded growth is untrue in all cases as they are asymptotic to a limit set.

More generally, given any model in this class of infinite population models in which mean fitness is always increasing, the mean fitness acts as a Lyapunov and all solutions are forward asymptotic to an equilibria (See Burger (1989) for specific criteria). Moreover, for any continuous mutation–selection model with non-decreasing continuous mean fitness function and a compact state space, there is a maximum possible mean fitness and all solutions will be asymptotic to a limit set [The state space will be compact if for example there is a finite upper bound on genome length. See Basener (2013b)]. In many cases, the existence of an increasing mean fitness mathematically precludes unbounded increase in mean fitness anticipated in Fisher’s Corollary.

The model we provide in Sect. 3 is not restricted in the genetic varieties that can be created via mutations, and thus can exhibit either ongoing increase or ongoing decrease in fitness depending on mutational effects. An error threshold type boundary between increasing and decreasing fitness is provided in Theorem 2.

Finite population models

Finite population mutation–selection models are models with rules assuming a finite number of organisms. Since there is a limited number of genetic varieties to select among, selection becomes less effective. As a result, finite population models are able to produce realistic phenomena that is a priori impossible in infinite population models, most importantly the build-up of deleterious mutations over time.

An early significant step in finite population modeling was the simple thought experiment that in a small asexually reproducing population, no parent can have offspring more fit than the parent (beneficial mutations are insignificant compared to deleterious ones, and back mutations are rare). It is possible that through random chance the most fit class of organisms might not produce offspring as fit as the parents, and the genetic makeup of this class would be lost. The next most fit class could suffer the same fate, and so on until the population loses fitness needed for population survival. This was pointed out by Muller (1964), and termed “Mullerś Ratchet” by Felsenstein (1974), with each click of the ratchet being a loss of a most fit class.

The predominance of deleterious mutations over beneficial ones is well established. James Crow in (1997) stated, “Since most mutations, if they have any effect at all, are harmful, the overall impact of the mutation process must be deleterious”. Keightley and Lynch (2003) given an excellent overview of mutation accumulation experiments and conclude that “...the vast majority of mutations are deleterious. This is one of the most well-established principles of evolutionary genetics, supported by both molecular and quantitative-genetic data. This provides an explanation for many key genetic properties of natural and laboratory populations”. In (1995), Lande concluded that \(90\%\) of new mutations are deleterious and, the rest are “quasineutral” (Also see Franklin and Frankham (1998)). Gerrish and Lenski estimate the ratio of deleterious to beneficial mutations at a million to one (Gerrish and Lenski 1998b), while other estimates indicate that the number of beneficial mutations is too low to be measured statistically (Ohta 1977; Kimura 1979; Elena et al. 1998; Gerrish and Lenski 1998a). Studies across different species estimate that apart from selection, the decrease in fitness from mutations is 0.2–2\(\%\) per generation, with human fitness decline estimated at \(1\%\) (See Lynch 2016; Lynch et al. 1999). Estimates suggest that the average human newborn has approximately 100 de novo mutations (Lynch 2016). Research using finite population models has been driven by the need to understand the impact of the buildup of deleterious mutations (called mutational load) in small populations of endangered species (See Lande 1995; Franklin and Frankham 1998). Of special interest is the mutational load in the human species given the relaxed selection due to social and medical advances (Kondrashov 1995; Crow 1997; Lynch 2016).

Lynch and Gabriel (1990), proposed a discrete-time (non-overlapping generations) model assuming that the effect of every deleterious mutation is equal to a value s, called the mutational effect or selection coefficient. The cumulative effects of mutations is assumed to be multiplicative, and so the fitness of a an organisms with n mutations is \(W=(1-s)^n\) (assuming an initial well-adapted baseline fitness of 1). Reproduction proceeds assuming a carrying capacity of K; after reproduction if there are more than K individuals then only the most fit K are kept. In the model it is assumed that the initial population is adapted to its environment with little mutation variance, and is at carrying capacity, for example a newly arisen from an ancestral sexual species. Analysis of the model includes both numerical simulations and analytic computations.

Lynch and Gabriel show that the resulting dynamics has three quantitative phases. First, the mutations are rare and selection is effective, and the mutation number grows slowly to approach the expected drift–selection–mutation equilibrium value. During the second phase, mutations continue to accumulate at a nearly constant rate until the mean viability is reduced to 1 / R (less than one surviving progeny per adult, or equivalently a negative growth rate). From this point on, the population decreases in size which accelerates the decrease in fitness, resulting in “mutational meltdown”.

The Lynch–Gabriel mutation accumulation model enables estimation of important relationships—for example the time-to-extinction for varying values of the mutational effect s and the carrying capacity K, (Lynch et al. 1993, 1995a, b), and the time to extinction is approximately equal to the log of the carrying capacity Lynch et al. (1993). The models shows that “an intermediate selection coefficient (s) minimizes time to extinction” (Lynch et al. 1993). Mutations with an effect in this intermediate range are small enough to not be selected out but large enough to still impact overall fitness. The model was extended to similar conclusions for sexually reproducing organisms (Lynch et al. 1995b), with a longer time to extinction. It has been used as an important guide in managing endangered species (Franklin and Frankham 1998).

While the Lynch–Gabriel mutational accumulation model assumes all mutations have the same impact on fitness, in reality the possible effects of deleterious mutations is a distribution ranging from effectively neutral to lethal (Lynch et al. 1999):

A lethal mutation is one that in the homozygous state reduces individual fitness to zero, whereas deleterious mutations have milder effects and neutral mutations have no effect on fitness. Because the distribution of mutational effects is continuous, it is difficult to objectively subdivide the class of deleterious mutations any further.

Of critical importance are deleterious mutations that are small enough in effect to accumulate, which Kondrashov calls “very slightly deleterious mutations” (VSDMs) (Kondrashov 1995). He states, “The study of VSDMs constitutes one of the pillars of population genetics” and attempts to quantify the most dangerous range of VSDMs as follows: “deleterious mutations with an effect less than 1 / G (where G is the length of the genome) have little effect no fitness even in large numbers, and that deleterious mutations with an effect greater than \(1/4N_e\) (where \(N_e\) is the effective population size) will be eliminated via selection”. He then observes, “In many vertebrates \(N_e\approx 10^4\), while \(G\approx 10^9\), so this dangerous range includes more than four orders of magnitude” (Kondrashov 1995). Other authors (e.g. Butcher 1995) have described this dangerous range in terms of Mullerś ratchet; deleterious mutations with a larger effect give a larger turn of the ratchet at each click but have a slower rate of clicks (because they are more susceptible to selection), while mutations with smaller effects give a smaller rotation at each click but have a higher click-rate. The mutations with the greatest long term impact on fitness are in the middle range with the greatest net rotation rate of the ratchet. These are the mutations, like those in the range of values observed by Lynch et al that minimize time to extinction (Lynch et al. 1993), which can accumulate over time and have significant net impact over time on fitness. In (1997), Crow describes the effect of these mutations as follows:

...diverse experiments in various species, especially Drosophila, show that the typical mutation is very mild. It usually has no overt effect, but shows up as a small decrease in viability or fertility, usually detected only statistically. ... that the effect may be minor does not mean that it is unimportant. A dominant mutation producing a very large effect, perhaps lethal, affects only a small number of individuals before it is eliminated from the population by death or failure to reproduce. If it has a mild effect, it persists longer and affects a correspondingly greater number. So, because they are more numerous, mild mutations in the long run can have as great an effect on fitness as drastic ones.

Computations based on this dangerous range of mutation effects suggest that populations in general should be accumulating very slightly deleterious mutations sufficient to affect mean fitness. Kondrashov conjectured in (1995) that mutational epistasis and soft selection may stop the accumulation of VSDMs. Mutational epistasis is the concept that multiple mutations will have a larger cumulative effect then the sum (or product, depending on the model) of the individual effects. If mutations all have the same mutational effect s and it is in the dangerous range, then as mutations accumulate epistasis implies that the effects of additional mutations will have increasingly large effects, eventually so large as to be no longer in the dangerous region, able to be selected out of the population.

The mutation–accumulation model of Lynch et al. (Butcher et al. 1993; Lynch et al. 1995a, b) was extended from considering mutations with a single fixed effect s to a continuous range of possible effects by Butcher (1995). Butcherś model is exactly the Gabriel-Lynch model with two modifications. First, instead of all mutations having the same effect, a probability distribution for the possible mutational effects is used. Second, Butcher has an epistasis term in his fitness to account for the mutational epistasis. Butcher shows “epistasis will not halt the ratchet provided that rather than a single deleterious mutation effect, there is a distribution of deleterious mutation effects with sufficient density near zero”. The phenomena is conceptually understandable even without the model details; as mutations accumulate, additional mutations that would have previously been in the dangerous range can now be selected out, but other mutations whose effects previously had been too minor to be dangerous will now be in the dangerous range. Butcher concludes that “This contradicts previous work that indicated that epistasis will halt the ratchet”. This is further supported by Baumgardner et al. (2013), which shows that in simulations with biologically realistic parameters, synergistic epistasis does not halt genetic degeneration—but actually accelerates it. Likewise, Brewer et al. (2013), shows that related mechanisms (such as the mutation-count mechanism), fail to halt genetic degeneration.

The observation that modeling of the mutation selection process, using realistic parameters measured from nature, suggests that mutations should accumulate to the significant detriment of even large populations is a paradox highlighted by Kondrashov (1995):

I interpret the results in terms of the whole genome and show, in agreement with Tachida (1990), that VSDMs can cause too high a mutation load even when \(N_e\approx 10^6-10^7\). After this, the data on the relevant parameters in nature is reviewed, showing that the conditions under which the load may be paradoxically high are quite realistic.

Kondrashov’s paradox suggests the critical need to better understand the mutation–selection process. Models show that deleterious mutations accumulate quickly in high mutation rate environments and small populations, and these models confirm observations from real organisms. These models further suggest that deleterious mutations will accumulate even in large populations with realistic parameters. The most complete models—those that like Butcher’s model use a distribution of mutational effects—suggest that the deleterious mutation accumulation is robust. There is a critical need for more complete models of mutation–selection and the surrounding biological parameters.

Conceptual selection models

In this section we describe some conceptual selection models. These models are not first-principle models, being more conceptual than derived from competing / reproducing organisms.

Truncation selection is an artificial selection process wherein every generation the members of a population are sorted and ranked based upon a specific criteria (such as total fitness), then a threshold of performance is defined, and all individuals below this threshold are unconditionally eliminated. This process is used in breeding plants and animals. While this is an effective method for amplifying desirable traits, there is no reason to believe that nature “ranks and truncates” (see Crow and Kimura 1979) populations. Moreover, while truncation selection is useful in breeding, it is only useful for amplifying a trait up to a limit; beyond that limit loss of genetic variance leads to diminishing returns and genetic pathologies.

Truncation selection can sometimes happen in nature; but only in isolated and largely artificial circumstances. For example, when a bacterial colony is exposed to an antibiotic, all cells dies except for the resistent ones. However, almost universally, total fitness is affected by many traits, many genetic factors, many selection factors, and many environmental effects. Therefore, truncation is not generally relevant in natural populations.