Neuroevolution

Neuroevolution is a machine learning technique that generates increasingly better neural networks for the problem its applied to by using a population-based optimization method that continuously increases the quality of each neural network in that population. Each individual in that population is not stored as complex neural network, but as a genome, which is a simplified genetic representation that can be mapped to a neural network. The neuroevolution process itself first initializes a set of these genomes and then applies them to the problem environment after which each genome is assigned a fitness score based on how well the neural network solves the applied problem. For example, this fitness score could be a function of the accuracy achieved in an image recognition task, how close a robotic arm moved to the intended trajectory or how well an agent performed in a virtual environment such as a video game.

Once the initial population has been created the optimization loop starts and the population is continuously mutated, recombined, evaluated and naturally selected. If these steps are performed iteratively with the whole population being in only one step at a time are we talking about generational neuroevolution. If the design of the neuroevolution algorithm allows for asynchronicity and the optimization loop is performed on a per genome basis is this usually termed competitive coevolution. In both cases does this endless loop of introducing innovation, evaluating it and then categorizing it constitute an optimization process that will eventually yield a neural network that solves the applied problem very well but might have been impossible to come up with by hand. Figure 1 below illustrates this typical optimization process of a generational neuroevolution algorithm — visualizing all steps of the aforementioned training loop.

Fig 1: Illustration of the typical generational neuroevolution process

Neuroevolution is an agnostic process that, while requiring parameters for its own evolutionary process, does not stipulate any specific topology or hyperparameters for the generated neural networks. Instead ANNs are developed through feedback on what effectively works well on the problem environment. This extent of choices (topologies, weights, hyperparameters, etc) that may be used for the formation and complete specification of the generated neural networks is generally referred to as the search space. While the agnostic nature of the neuroevolution process allows for a very broad search space can it be sensible to limit granularity of the search space in order to traverse it faster. The faculty that maps genomes to their according neural networks is responsible for the granularity of the search space by limiting the complexity of the genome’s encoding. This faculty is therefore called the genetic encoding.

In order to allow for a broad search space with an appropriate granularity is it crucial to design the genetic encoding and the neuroevolution algorithm that operates on it, according to the requirements of the problem environment. Hence in order to design a good neuroevolution scheme do we first need to establish the fundamental understanding of a genetic encoding, as will be done in the next section. Following that we’ll take a closer look at each step of the aforementioned evolutionary optimization loop that is displayed in figure 1 in the order in which they become relevant.

The Genetic Encoding

In order to efficiently mutate and recombine ANNs is an efficient representation of the ANN necessary. Efficient representation of neural networks allows the frequently employed genetic operations to not have to analyze highly complex data structures but for them to analyze compact genetic codes that can be processed fast. As a consequence can genetic operations quickly determine sensible mutations or if two genomes are eligible for recombination. Neuroevolution algorithms therefore operate exclusively on the genetic encoding instead of the complex data structures popular popular machine learning frameworks operate on. Though of course does the genetic encoding allow a mapping between the two representations.

These efficient genetic representations of a genome are termed genotypes, while the accordingly mapped neural networks are termed phenotypes. The terminology was adopted from genetic evolution, since neuroevolution is strictly speaking genetic evolution with the constraint that all phenotypes are neural networks.

Fig 2: Illustration of an example direct genetic encoding

Genetic encodings can generally be divided into two subcategories: direct encodings and indirect encodings. Though there is also the third category of developmental encodings are we omitting this encoding as it is of little relevance in recent years. Direct encodings represent each aspect of the neural network they encode explicitly in its genetic representation. Figure 2 above is an example of a direct encoding as it is employed by the popular NEAT algorithm [3][4][5]. It encodes each connection and its according weight directly in the genotype but limits the search space by excluding the possibility of bias and activation functions in the resulting ANN. Such direct encodings can represent arbitrary feed-forward and recurrent topologies, allowing for possibly perfectly suited and at the same time minimal topologies. Allowing such a high degree of flexibility in the topology however results in an enormous search space with fine granularity, requiring a well designed neuroevolution algorithm in order to traverse that search space quickly.

Fig 3: Illustration of an example indirect genetic encoding

Indirect encodings on the other hand stipulate a custom encoding that is often not intuitively translatable into an ANN, but requires a separate translation faculty that is also stipulated by the indirect encoding in order to map a genotype to a neural network. Figure 3 above showcases an example of an indirect layer encoding and the faculty that maps the stipulated indirect encoding into an ANN. If the indirect encoding is well designed does this allow for a meaningful and fast traversal through the search space by even simple mutations (e.g. a bit flip in the example above) as well as fast recombinations of otherwise complex neural networks.

However, whereas ANNs can be quickly created from direct encodings, is the translation faculty of indirect encodings slowing the processing speed and potentially too coarse-grained. Benefits and drawbacks of both encodings and especially how well the concrete implementation of either encoding fits the problem domain at hand have to be considered before deciding upon either one.

Yet both genetic encodings demonstrate well how the genetic encoding determines the size of the search space, e.g. by not allowing activation functions or certain layer types, as well as determine its granularity, e.g. by only allowing to adjust hidden layer sizes by a multiple of 32.

The Method of Reproduction and Solution Exploration

Closely related to the genetic encoding employed by the neuroevolution scheme is the method through which the defined search-space is actually traversed. This method is the process of reproduction and usually breaks down to creating new genomes through either mutating or recombining genomes that are deemed fit to be parents for the next generation. Mutation lets offspring genomes explore the viability of novel ANN architectures, weight distributions and hyperparameters. Recombining genomes, which is essentially the process of merging two promising genomes and their distinct features, spreads beneficial features throughout the rest of the population.

Mutation is closely related to the genetic encoding in the sense that it can only mutate the parameters of a neural network to the extent in that they are represented in the genetic encoding. Defining mutation for a neuroevolution algorithm is therefore defining three aspects within the possible range that the genetic encoding allows for. First, what part of the genetic encoding will be mutated? It could be the topology, weights, hyperparameters or in above’s example of an indirect encoding a random selection of bits. Furthermore, to what extent will the chosen part of the genome be mutated? A neuroevolution algorithm could for example employ extensive mutations for low-fitness genomes and minuscule mutations for high-performing genomes — effectively expressing the approach to drastically change the design of the ANN if it’s not working or working out the details of an ANN if it is working well. The last thing to be defined for mutation is if it is somehow directional or random. The mutation of weights is often performed randomly in environments that lack clear ground truth, though is kept at bay by, for example, obtaining the new random value from a normal distribution centered around the current value. Directed mutation however is also possible if the neuroevolution algorithm is able to map a lack of performance in the problem environment to a deficit in the design or parameters of the neural network.

Fig 4: Illustration of the Competing Conventions problem (Source: [5])

Recombination on the other hand does not mutate genomes, but still creates novelty by combining two parent-genomes and their distinctive features into an innovative offspring-genome. If the recombination method is well designed and can merge the beneficial features of both parent-genomes in a lossless way does this allow for the aforementioned spread of beneficial features throughout the whole population — raising the fitness of all present genomes. The core issue of a well designed recombination method is therefore its property of being lossless, which is the property of merging (also called crossing over) the genetic encoding of two genomes without losing any of the encoded ANN features. Prior to the neuroevolution algorithm NEAT did neuroevolution schemes that were utilizing direct encoding face the problem of lossy crossovers that came up during mutation and a later recombination of two independent genomes. Figure 4 above illustrates this problem that was termed the competing-conventions problem. The NEAT algorithm proposed a method called historical markings, which provided each mutation with a unique identifier that finally enabled a lossless recombination of genomes, making it still a benchmark neuroevolution algorithm to this day.

The Method of Evaluation

Though the step of evaluating genomes on the problem environment seems to be the simplest one in the overarching optimization loop is it still important to shortly address it — if only to point out potential caveats and possibilities. Fundamentally is the method of evaluation simply the process of mapping a genome to a neural network as stipulated by its genetic encoding, applying it to the problem environment and then calculating a fitness value depending on how well the ANN performed. Some neuroevolution algorithms also include the additional step of weight training a neural network before it is applied in the actual fitness scoring run. While this is method is very sensible is it only possible if the problem environment exhibits a clear ground truth.

The only reasonably modifiable aspect in this whole process of evaluation is the way in which fitness is determined, though this is completely dependent on the nature of the applied problem and has to be decided on a case to case basis. Common methods for determining the fitness value are setting it equal to its accuracy in image recognition environments or equal to the points achieved in a video game environment.

Novelty Search is also an important concept to consider when determining the fitness calculation, as this concept rewards genomes with novel approaches to the problem with higher fitness values. An agent in a video game environment would for example receive a fitness boost if he entered an unknown area though has overall achieved less points. This facilitates innovation in the gene pool and therefore an overall broader and more promising evolutionary process. Novelty Search was introduced in 2010 in the paper ‘Evolving Plastic Neural Networks with Novelty Search’ [7].

Creating a well designed method of evaluation is other than that mostly an issue of its concrete implementation. Logging possible ground truths or adequately parallelizing the evaluation environments — since the nature of having multiple neural networks that are completely independent renders it a trivially parallelizable problem. Though this is not a research issue, is it an equally important aspect to consider when implementing the neuroevolution scheme.

Natural & Parental Selection

While the method of reproduction defines the way in which the search space is explored, is the choice of which genomes will serve as the parents for offspring and which genomes will be removed from the gene-pool another very important aspect of a neuroevolution scheme. The method through which genomes are chosen to be parents for offspring is also heavily dependent on the kind of neuroevolution algorithm that is used.

In generational neuroevolution choosing the genomes suitable as parents for the next generation usually comes down to choosing a stipulated fraction of the best performing genomes in the current generation. This fraction is often supplied as a one of the many parameters that can still configure a predefined neuroevolution algorithm. The fraction of genomes chosen to be parents does not have to be complementary with the fraction of genomes chosen to be removed from the gene pool, as genomes can be promising enough to warrant potential for innovative mutation, though may be removed themselves in the following generation. An exemplary configuration for a neuroevolution algorithm would be the choice to utilize the top 20% of genomes as parents, though remove the 90% of the worst performing genomes for the following generation. Depending on if the generational neuroevolution algorithm employs speciation, does the choice of genomes that are selected to be parents or selected to be removed change significantly, usually enabling a more guided evolutionary process.

Speciation is a tool of generational neuroevolution that clusters the individuals of the population according to apparent features like topological similarities in the phenotype or more covert features like reaching certain states in the applied problem environment. The resulting clusters are then considered species and ideally — depending on how well the method of speciation is chosen — represent genuinely distinct approaches to the problem at hand. This allows the reproductive process to guide the evolution by allocating more offspring to species with a higher average fitness and therefore favor the development of distinct and promising offspring. Well designed speciation however also protects innovation in the gene pool by still allocating some offspring to species with novel distinct features that have not yet been given the time to optimize (e.g. a distinctive topological addition in the ANN, whose weights are not yet optimized). Designing an appropriate method of speciation that segregates individuals of the population in distinct species based on genuinely different approaches to the problem can therefore be a very powerful tool in the larger neuroevolutionary method that both guides the evolution into a beneficial direction as well as protects innovative new paths.

On the other hand, in a competitive coevolution scheme are genomes usually randomly paired and compared, with the fitter or more innovative one being chosen as parent and the other one being removed from the gene-pool. Due to the asynchronous nature of competitive coevolution is it impossible to categorize and classify all genomes in the population at once, significantly restricting the possibility to guide the evolution with methods like speciation or such. Though while the methods of natural and parental selection are simpler in competitive coevolution, is this made up by the effectively perfect parallelizability and scalability.

Closing Remarks on Neuroevolution

Neuroevolution is a very broad approach to optimizing artificial neural networks to better perform on the problem environment. It is a versatile tool that is not limited to one aspect of machine learning but can be applied to improve methods of Computer Vision, Natural Language Processing or agent-based environments — as its versatility and search space are only bound by the expressiveness of its genetic encoding. The aspects to consider and combine when creating a neuroevolution algorithm are plentiful, though foremost need to be determined through an exact analysis of the problem and its specific characteristics, such that they may be exploit the problem environment most effectively.