1- Model of the evolution of a population in presence of phenotypic variation

We consider a population with a continuous trait (e.g. reproductive effort, physiological parameters). We assume that there is no genetic variation for this trait in the population: all genotypes take the value g, but the phenotypic expression can vary randomly from φ min to φ max . This phenotypic variance can arise from several different processes (e.g. developmental instability, environmental variability, maternal effects, epistasis) as long as it is a random process such that any individual of genotype g can experience any phenotype between φ min and φ max . The simple evolutionary processes involved in the population model were chosen to isolate the effect of phenotypic variability on trait evolution and to increase the generality of model predictions.

The function a(φ,t) denotes the frequency of individuals in the population having the phenotype value φ at time t. We assume that instantaneous fitness depends on the phenotype, so these individuals have a reproductive rate b(φ)≥0 and a mortality rate of m(φ)>0. Therefore, at each instant t and per unit of time, m(φ)a(φ,t) individuals of phenotype φ die, while b(φ)a(φ,t) descendants are produced by individuals of phenotype φ.

For any individual with genotype g, the phenotype φ of its offspring is randomly distributed around the value g following the distribution function G(φ,g,σ). We consider that G(φ,g,σ) is a Gaussian function (default hypothesis for a quantitative characters, see [29]) centred around g with a variance σ2.

Therefore, offspring of phenotype φ are produced by parents of genotype g in the proportion G(φ,g,σ), regardless of the parents' own phenotype, i.e. there are no cross-generational effects. Therefore, at each time a total quantity of new individuals are produced in the population, among which only a fraction G(φ,g,σ) will have the phenotype φ.

The variation of the distribution of individuals of parameter φ over time is then given by the differential equation: (1)

We show in Supplementary Text S1 that such a population does not go extinct as long as (2)

The term in the integral represents the per capita growth rate w(l) = b(l)/m(l) (phenotype fitness) of individuals with phenotype l multiplied by their frequency in the population. Thus, the function F(g) represents the sum of phenotype fitness of all phenotypes weighted by their respective frequency, i.e. the weighted mean of phenotype fitness in the population. The growth rate function F(g) is therefore the genotype fitness of genotype g. This result holds for populations with limited resources (Supplementary Text S1).

Then, we consider two populations with limited growth, one with a genotype g 1 and the other with a genotype g 2 ≠g 1 . They are represented by their respective distribution a 1 (φ,t) and a 2 (φ,t). They are interacting with each other due to mutually shared resources that are limited. In order to have true competition, we assume that each population does not go extinct if it is alone, which is equivalent to F(g 1 )>1 and F(g 2 )>1.

The equations that describe the evolution of these populations and their distributions along time are: (4)There are four equilibrium points: coexistence of both populations; extinction of both populations; only one population survives while the other goes extinct (two combinations). Coexistence is possible only if g 1 = g 2 , which is excluded by hypothesis. Moreover since F(g 1 ) and F(g 2 ) are assumed to be strictly greater than 1 for each population, it can be shown that the extinction of both populations is not possible. Thus, under these hypotheses one population must invade the other. Then, a successful invasion of population g 1 into population g 2 (i.e. equilibrium a 1 ≠0 and a 2 = 0 stable) is possible if and only if: (5)

Hence, the evolutionary stable strategy (ESS) corresponds to the genotype g* that maximises the growth rate function F(g). The population of parameter g*, also called the super-mutant population, will invade any population of parameter g≠g*, while it cannot be invaded by other populations with a parameter g≠g*. Thus, the genotype g* is an ESS and should be observed in population at the equilibrium state, although the phenotype varies in the population.

To determine g*, it is necessary to calculate the maximum of the growth rate function F(g). Hence, the most efficient genotype is the one that maximises the success of the whole population by cumulating the relative success of each phenotypic trait weighted by their frequency.

When the fitness function is symmetric or if there is no phenotypic variance at all, the genotypic value g* associated with the maximum of the function F is equal to the value that maximizes the phenotype fitness w. However, when the fitness function is asymmetric and the phenotypic variance is non-zero, these two values do not match anymore, as predicted by Jensen's inequality. In this case, the optimum genotype g* value is systematically shifted from the maximum of the phenotype fitness in the direction of the least slope (Supplementary Text S2).