Fri 01 June 2018

Spectre and Meltdown took much of our community by surprise. I personally found these attacks fascinating because they didn't rely on a bug in any particular hardware implementation, but leveraged undefined behavior. Specifically, Spectre and Meltdown can exfiltrate potentially secret memory data by detecting the effects of speculative instructions that are later squashed.

Very cool!

Out of order processors are very complex. It would make it easier to understand exactly what causes speculation attacks like Spectre and Meltdown if we had a way to visualize the attacks. Luckily, gem5 already has a way to view the details of it's out of order CPU's pipeline.

The image above was created using the O3 pipeline viewer that is included with gem5. In this post, I'll explain how to use the O3 pipeline viewer and how to generate images like the above. There is also a new project which makes it easier to navigate large pipeline traces and it quite useful for comparing different pipeline designs: Konata created by Ryota Shioya. Ryota gave a presentation on Konata at a recent Learning gem5 tutorial. You can find the pdf of his presentation here. Konata is a cool tool that's written in javascript and Ryota describes it as "Google maps for an out of order pipeline".

Running Spectre The first step to visualizing what is going on in the pipeline during a Spectre attack is getting proof of concept exploit code. I used the code that was posted to a github gist by Erik August soon after the attack was announced. You can get that code here: https://gist.github.com/ErikAugust/724d4a969fb2c6ae1bbd7b2a9e3d4bb6. First, you need to compile the proof of concept code on your native machine (note: I'll be using x86 for all of my examples). gcc spectre.c -o spectre -static I used gcc 7.2 (the default on Ubuntu 17.10) for my tests, and you may want to do the same. Below I discuss the effects different compilers have on the Specre attack. For instance, if you use clang instead you may not be able to reproduce the Spectre attack in gem5. docker run --rm -v $PWD : $PWD -w $PWD powerjg/gem5-build-gcc48 gcc spectre.c -o spectre -static -std = c99 My native machine is still vulnerable to Spectre so when I run the binary generated above, I get the following output. Reading 40 bytes: Reading at malicious_x = 0xffffffffffdd76c8... Success: 0x54=’T’ score=2 Reading at malicious_x = 0xffffffffffdd76c9... Success: 0x68=’h’ score=2 Reading at malicious_x = 0xffffffffffdd76ca... Success: 0x65=’e’ score=2 Reading at malicious_x = 0xffffffffffdd76cb... Success: 0x20=’ ’ score=2 Reading at malicious_x = 0xffffffffffdd76cc... Success: 0x4D=’M’ score=2 Reading at malicious_x = 0xffffffffffdd76cd... Success: 0x61=’a’ score=2 Reading at malicious_x = 0xffffffffffdd76ce... Success: 0x67=’g’ score=2 Reading at malicious_x = 0xffffffffffdd76cf... Success: 0x69=’i’ score=2 Reading at malicious_x = 0xffffffffffdd76d0... Success: 0x63=’c’ score=2 Reading at malicious_x = 0xffffffffffdd76d1... Success: 0x20=’ ’ score=2 Reading at malicious_x = 0xffffffffffdd76d2... Success: 0x57=’W’ score=2 Reading at malicious_x = 0xffffffffffdd76d3... Success: 0x6F=’o’ score=2 Reading at malicious_x = 0xffffffffffdd76d4... Success: 0x72=’r’ score=2 Reading at malicious_x = 0xffffffffffdd76d5... Success: 0x64=’d’ score=2 Reading at malicious_x = 0xffffffffffdd76d6... Success: 0x73=’s’ score=2 Reading at malicious_x = 0xffffffffffdd76d7... Success: 0x20=’ ’ score=2 Reading at malicious_x = 0xffffffffffdd76d8... Success: 0x61=’a’ score=2 Reading at malicious_x = 0xffffffffffdd76d9... Success: 0x72=’r’ score=2 Reading at malicious_x = 0xffffffffffdd76da... Success: 0x65=’e’ score=2 Reading at malicious_x = 0xffffffffffdd76db... Success: 0x20=’ ’ score=2 Reading at malicious_x = 0xffffffffffdd76dc... Success: 0x53=’S’ score=2 Reading at malicious_x = 0xffffffffffdd76dd... Success: 0x71=’q’ score=2 Reading at malicious_x = 0xffffffffffdd76de... Success: 0x75=’u’ score=2 Reading at malicious_x = 0xffffffffffdd76df... Success: 0x65=’e’ score=2 Reading at malicious_x = 0xffffffffffdd76e0... Success: 0x61=’a’ score=2 Reading at malicious_x = 0xffffffffffdd76e1... Success: 0x6D=’m’ score=2 Reading at malicious_x = 0xffffffffffdd76e2... Success: 0x69=’i’ score=2 Reading at malicious_x = 0xffffffffffdd76e3... Success: 0x73=’s’ score=2 Reading at malicious_x = 0xffffffffffdd76e4... Success: 0x68=’h’ score=2 Reading at malicious_x = 0xffffffffffdd76e5... Success: 0x20=’ ’ score=2 Reading at malicious_x = 0xffffffffffdd76e6... Success: 0x50=’P’ score=9 (second best: 0x06 score=2) Reading at malicious_x = 0xffffffffffdd76e7... Success: 0x73=’s’ score=2 Reading at malicious_x = 0xffffffffffdd76e8... Success: 0x73=’s’ score=2 Reading at malicious_x = 0xffffffffffdd76e9... Success: 0x69=’i’ score=2 Reading at malicious_x = 0xffffffffffdd76ea... Success: 0x66=’f’ score=2 Reading at malicious_x = 0xffffffffffdd76eb... Success: 0x72=’r’ score=2 Reading at malicious_x = 0xffffffffffdd76ec... Success: 0x61=’a’ score=2 Reading at malicious_x = 0xffffffffffdd76ed... Success: 0x67=’g’ score=2 Reading at malicious_x = 0xffffffffffdd76ee... Success: 0x65=’e’ score=2 Reading at malicious_x = 0xffffffffffdd76ef... Success: 0x2E=’.’ score=2 Running Spectre in gem5 To find out if gem5's out of order CPU implementation is vulnerable to Spectre, we need to run the code in gem5. The simplest and fastest way to do this is by running in gem5's syscall-emulation (SE) mode. In SE mode we won't be modeling an OS or any user-mode to kernel-mode interaction, but this okay for Spectre since this proof of concept code is all in user-mode. If we were investigating Metldown, we would have to use full-system (FS) mode since Meltdown specifically allows user-mode processes to read data that should only be accessible in kernel mode. So, when running something in gem5, the first step is to create a Python runscript since this is the "interface" to gem5. For this example, what we need is a system with one CPU, an L1 cache, and memory. For simplicity, I'm going to modify one of the existing script, specifically the two_level.py script from the Learning gem5 book. In the file gem5/configs/learning_gem5/part1/two_level.py , I simply changed the CPU from TimingSimpleCPU() to DerivO3CPU(branchPred=LTAGE()) . I also set the O3CPU to use the LTAGE branch predictor instead of the default tournament branch predictor. It's important to use the LTAGE branch predictor as better branch predictors actually make Spectre easier to exploit as discussed further below. Now, we simply need to build gem5 and run it. scons -j8 build/X86/gem5.opt build/X86/gem5.opt configs/learning_gem5/part1/two_level.py spectre And, the output that I get is the following, just like above when I ran the spectre natively. Note: If you use GCC 4.8 it executes much faster than 7.2 (the default on Ubuntu 17.10). gem5 Simulator System. http://gem5.org gem5 is copyrighted software; use the --copyright option for details. gem5 compiled May 10 2018 09:40:08 gem5 started May 24 2018 11:21:16 gem5 executing on palisade, pid 27173 command line: build/X86/gem5.opt configs/learning_gem5/part1/two_level.py spectre Global frequency set at 1000000000000 ticks per second warn: DRAM device capacity (8192 Mbytes) does not match the address range assigned (512 Mbytes) 0: system.remote_gdb: listening for remote gdb on port 7000 Beginning simulation! info: Entering event queue @ 0. Starting simulation... warn: readlink() called on '/proc/self/exe' may yield unexpected results in various settings. Returning '/home/jlp/Code/gem5/spectre-vis/spectre' info: Increasing stack size by one page. warn: ignoring syscall access(...) Reading 40 bytes: tput cols Reading at malicious_x = 0xffffffffffdd76c8... Success: 0x54=’T’ score=2 Reading at malicious_x = 0xffffffffffdd76c9... Success: 0x68=’h’ score=2 Reading at malicious_x = 0xffffffffffdd76ca... Success: 0x65=’e’ score=2 Reading at malicious_x = 0xffffffffffdd76cb... Success: 0x20=’ ’ score=2 Reading at malicious_x = 0xffffffffffdd76cc... Success: 0x4D=’M’ score=2 Reading at malicious_x = 0xffffffffffdd76cd... Success: 0x61=’a’ score=2 Reading at malicious_x = 0xffffffffffdd76ce... Success: 0x67=’g’ score=2 Reading at malicious_x = 0xffffffffffdd76cf... Success: 0x69=’i’ score=2 Reading at malicious_x = 0xffffffffffdd76d0... Success: 0x63=’c’ score=2 Reading at malicious_x = 0xffffffffffdd76d1... Success: 0x20=’ ’ score=2 Reading at malicious_x = 0xffffffffffdd76d2... Success: 0x57=’W’ score=2 Reading at malicious_x = 0xffffffffffdd76d3... Success: 0x6F=’o’ score=2 Reading at malicious_x = 0xffffffffffdd76d4... Success: 0x72=’r’ score=2 Reading at malicious_x = 0xffffffffffdd76d5... Success: 0x64=’d’ score=2 Reading at malicious_x = 0xffffffffffdd76d6... Success: 0x73=’s’ score=2 Reading at malicious_x = 0xffffffffffdd76d7... Success: 0x20=’ ’ score=2 Reading at malicious_x = 0xffffffffffdd76d8... Success: 0x61=’a’ score=2 Reading at malicious_x = 0xffffffffffdd76d9... Success: 0x72=’r’ score=2 Reading at malicious_x = 0xffffffffffdd76da... Success: 0x65=’e’ score=2 Reading at malicious_x = 0xffffffffffdd76db... Success: 0x20=’ ’ score=2 Reading at malicious_x = 0xffffffffffdd76dc... Success: 0x53=’S’ score=2 Reading at malicious_x = 0xffffffffffdd76dd... Success: 0x71=’q’ score=2 Reading at malicious_x = 0xffffffffffdd76de... Success: 0x75=’u’ score=2 Reading at malicious_x = 0xffffffffffdd76df... Success: 0x65=’e’ score=2 Reading at malicious_x = 0xffffffffffdd76e0... Success: 0x61=’a’ score=2 Reading at malicious_x = 0xffffffffffdd76e1... Success: 0x6D=’m’ score=2 Reading at malicious_x = 0xffffffffffdd76e2... Success: 0x69=’i’ score=2 Reading at malicious_x = 0xffffffffffdd76e3... Success: 0x73=’s’ score=2 Reading at malicious_x = 0xffffffffffdd76e4... Success: 0x68=’h’ score=2 Reading at malicious_x = 0xffffffffffdd76e5... Success: 0x20=’ ’ score=2 Reading at malicious_x = 0xffffffffffdd76e6... Success: 0x4F=’O’ score=2 Reading at malicious_x = 0xffffffffffdd76e7... Success: 0x73=’s’ score=2 Reading at malicious_x = 0xffffffffffdd76e8... Success: 0x73=’s’ score=2 Reading at malicious_x = 0xffffffffffdd76e9... Success: 0x69=’i’ score=2 Reading at malicious_x = 0xffffffffffdd76ea... Success: 0x66=’f’ score=2 Reading at malicious_x = 0xffffffffffdd76eb... Success: 0x72=’r’ score=2 Reading at malicious_x = 0xffffffffffdd76ec... Success: 0x61=’a’ score=2 Reading at malicious_x = 0xffffffffffdd76ed... Success: 0x67=’g’ score=2 Reading at malicious_x = 0xffffffffffdd76ee... Success: 0x65=’e’ score=2 Reading at malicious_x = 0xffffffffffdd76ef... Success: 0x2E=’.’ score=2 Exiting @ tick 113568969000 because exiting with last active thread context

Visualizing the out of order pipeline To generate pipeline visualizations, we first need to generate a trace file of all of the instructions executed by the out of order CPU. To create this trace, we can use the O3PipeView debug flag. Now, the trace for the O3 CPU can be very large up to many GBs. When creating this trace, you need to be careful to create the smallest trace possible. Also, it's important to dump the trace to a file and not to stdout , which is the default when using debug flags. You can redirect the trace to a file by using the --debug-file option to gem5. To create the trace file, I used the following methodology: Start running spectre in gem5, then hit ctrl-c after the first couple of letters. At this point, I wrote down the tick which gem5 exited (13062347000). Run gem5 with the debug flag O3PipeView enabled. Watch the output and kill gem5 with ctrl-c after two more letters appeared than in step 1. To generate the trace, I ran the following command. Note: you may have a different value for when to start the debugging trace. Also note: when producing the trace gem5 will run much slower. build/X86/gem5.opt --debug-flags = O3PipeView --debug-file = pipeview.txt --debug-start = 13062347000 configs/learning_gem5/part1/two_level.py spectre My tracefile ( pipeview.txt ) was 600 MB for catching just two letters in the output. Now, we can process this file to generate the visualization with a script: util/o3-pipeview.py . This script requires the path to the file that contains the output generated with the O3PipeView debug flag. Above, we put the output into the file pipeview.txt , and this file was created in the default output directory of gem5 ( m5out/ ). util/o3-pipeview.py --store_completions m5out/pipeview.txt --color -w 150 In the above command, I wanted to see when the stores completed ( --store_completions ) and specified to use color ( --color ) in the output and use a width of 150 characters ( -w 150 ). Processing a large file like this one of 600 MB may take a few minutes. The output will be in a file called o3-pipeview.out in the current working directory. You can view this file with less -r o3-pipeview.out . You may want to use the -S option with less if your terminal is less than 150 characters wide (or whatever width value you used). Below is a screenshot of the top of my trace. Understanding the O3 pipeline viewer The above image details how to interpret the output from the pipeline viewer. Each . or = represents one cycle of time, which moves from left to right. The "tick" column shows the tick of the leftmost . or = . = is used to mark the instructions that were later squashed. The address of the instruction (and the micro-op number) as well as the disassembly is also shown. The sequence number can be ignored as it is always monotonically increasing and is the total order of every dynamic instruction. Finally, each stage of the O3 pipeline is shown with a different letter and color.