Before we go further, we need to understand the meaning of the word “multimedia”. Mayer defines it as the presentation of the source of information using both words and pictures. By words he means that information is presented in a verbal form, printed or spoken text. By pictures, he means that information is presented in graphic form, such as static graphics (illustrations, graphs, photos and so on) or dynamic graphics (like animation or video).

The core of the Cognitive Theory of Multimedia Learning is the inclusion of several principles. These principles speak about how we can best match multimedia displays to the cognitive architecture of the human mind. The first step to understand why this tool can be so powerful is to understand how the mind processes information. The brain takes information and processes it in different channels, based on the way this information is presented. The first channel deals with the visual material, and the second deals with the auditory material. When a player receives a visual input, such as photos, videos, charts or printed words, then the information is directed to the visual channel and processed there. By analogy — if he or she receives audio information, such as spoken words or sounds, it is processed by the brain in the audio channel. New information is registered in a sensory memory, then the player starts working with information to learn them, and this happens in a working memory. Two separate channels allow players to process more information because different material presentations are processed in different ways. Finally, basing on his or her experience and knowledge, the player combines the visual model with the auditory model and then the knowledge lands in the long-term memory.

Cognitive Load Theory describes the amount of mental effort involved in performing a task at a given moment. According to this theory, the brain has a finite number of resources, that it can allocate to loads. This theory differentiates between three types of cognitive load: intrinsic cognitive load, germane cognitive load, and extraneous cognitive load. If the additive sum of these loads exceeds the maximum, a man enters a state called the cognitive overload. Because the cognitive load accumulates, it is important that the load from every sense is always as small as possible.

How does this connect with game design? Well, this applies practically to every element of the video game: from the tutorial, through the UI, to teaching players how to play the game well. Dr. Mayer identified several principles of multimedia learning that we should consider when designing every little piece of our games. There are generally several principles, but the most important for games are:

1. Multimedia Principle — People learn better from words and pictures than from words alone. This means that a photo with accompanying text, such as an infographic, will be easier to remember than the image itself or the text itself. In games, this means that the combined effect is necessary to effectively teach players to play the game. Both mechanics and tutorials should be combined to teach a coherent message. Remember to show relationships and transformations, explain how things work.

2. Modality Principle — People learn better from graphics and narrations than from animation and on-screen text. When both text and graphics are shown on the screen, the players have to jump back and forth visually from the written text to the graphic to understand the content. It’s better to remove the written text and instead use audio narration to describe graphics. This principle is probably the easiest to implement in game design.

3. Coherence Principle — People learn better when extraneous words, pictures and sounds are excluded rather than included. This rule is a bit difficult for game designers. If we use level design and mechanics implementation to make sure that players only meet with the things that are relevant to the current learning goal, we will succeed in teaching them. If you add information that does not relate to specific learning objectives, there is always the chance that players will get distracted or think that this information is important to remember, which distracts them from what they actually should know.

4. Segmenting Principle — People learn better when a multimedia lesson is presented in user-paced segments rather than as a continuous unit. The rule is simple: break down complicated things into smaller ones and segments into pieces where possible, allow learners to control their own pace.

5. Pre-training Principle — People learn better from a multimedia lesson when they know the names and characteristics of the main concepts. This principle means that people need information before they are presented with more complex topics. In games, we generally apply it quite well in the form of tutorial levels. Remember to provide examples and use varied cases.

6. Personalization Principle — People learn better from multimedia lessons when words are in conversational style rather than formal style. In designing games, this means that we have to make sure that our instructions do not try to break the fourth wall — unless it is purposeful, of course — and that we do not impose anything on the players either in our text or sound instructions. Too stilted language creates additional cognitive load and hinders learning. Remember to use a conversational writing style, write in first or second person (“I”, “you”, “we”, “our”), use polite speech (“please”, “let’s”), speak naturally in recorded audio.

7. Redundancy Principle — People learn better from graphics and narration than from graphics, narration and on-screen text. Brain processes audio and video on different channels and the load caused by it is additive, so it is important that we cognitively not overload our players, wasting cognitive space on things that are unnecessary. People get additional cognitive load when information is presented to them on screen with accompanying sound and text at the same time. They try to compare and reconcile the text with the narrative, and this increases the cognitive load. This means that if you intend to design a video game and give instructions to the players on the screen, talk about them or write them, but not both.

8. Spatial Contiguity Principle — People learn better when corresponding words and pictures are presented near rather than far from each other on the page or screen. In designing a game, this means keeping relevant information close to the place on which it depends. Remember to place words in close proximity to the graphics they refer to.

9. Temporal Contiguity Principle — People learn better when corresponding words and pictures are presented simultaneously rather than successively. It means keeping audio about what things do temporally close to the object in question.