Formula One racing teams have the need for speed, and even more importantly, acceleration. It is what wins races, after all. But the funny thing about the Formula One sport is that the speed of the parallel clusters that run the software that is used to design racing cars is capped. And that means wringing performance out of those systems is absolutely vital if a team hopes to not be lapped by its rivals.

The high performance computing cap on the systems that run the computational fluid dynamics, or CFD, applications that are used to design the race cars is absolutely intentional, Otmar Szafnauer, chief operating officer of the Sahara Force India racing team, explains to The Next Platform. But Szafnauer is hopeful that this performance cap for clusters will be lifted sometime in the future for both the good of the sport and for the good of advancing the state of the art in CFD applications.

A Formula One racing team is a big business, one that mixes a sport team with a car manufacturer. Only in this case, the car manufacturer only produces four cars a year and it does so under very stringent rules handed down by the Federation Internationale de L’Automobile (FIA), an organisation that has aimed to level the playing field and helped spur engineering innovation in racing since the first Formula One World Championship race was held in 1950. The average team has somewhere around 400 employees or so and a budget of around $250 million a year; some teams are larger, some are smaller. But it takes a lot of people to design the cars and manufacture its components, which are constantly changing not only because the FIA periodically amends the rules governing the cars to keep driving innovation, but also because the engineers are constantly trying out new approaches for the car design to try to get an edge on the competition.

No edge can be sustained for long, of course, so Formula One racing embraces a kind of continuous development cycle that software developers among hyperscalers and within enterprises have only recently discovered for the applications they create. At Sahara Force India, a crucial share of the team’s more than 350 employees work solely in aerodynamics performance, says Szafnauer, just to give you an indication of how vital CFD software and wind tunnels are to a Formula One racing team.

“The four cars we produce each year are like prototype cars at big auto makers,” Szafnauer explains. “There is no tooling involved, everything is made from scratch or machined from solid. A big portion of our car is carbon fiber, and because of that it is very labor intensive to make, so the costs are high. Most of the car is carbon fiber now, and it is layered up, like making a cake. Cutting the carbon is automated, but we have people laying it up, piece by piece, by hand. People are expensive, especially skilled ones.”

Most teams do not make their own engines, which are carved from aluminum blocks and supplied by Renault, Honda, Mercedes-Benz, and Ferrari. Sahara Force India, which was founded in 2007 in its most recent incarnation but which dates back to 1991 in its history (Formula One teams change hands and names like other sports franchises), buys its engines and gearboxes from Mercedes.

A Formula One race team runs approximately 20 races a year, and it is pretty much standard to have four cars for the year. Under FIA rules, you can bring three cars to a race, but you can only use two of them. Most teams have a fourth car for which they can “push it and pull it,” as Szafnauer put it, to test out new ideas. Each car has something on the order of 80,000 parts, which is a lot more than many people realize, and a staggering 20 to 25 percent of the parts on each car change during the course of the racing year.

“There are some things that you don’t change, such as the chassis,” Szafnauer says. “Sometimes we change suspensions, it is rare that we change drive lines. The engine does get upgraded, as we did this year, and sometimes the gearbox gets upgraded and aerodynamically, it changes quite a bit.”

In 2014, there was a huge change handed down by the FIA, moving the cars from a 2.4 litre V8 engine to a 1.6 litre V6 motor. The idea was to push racing technology, therefore making it transferable to the wider automotive industry, while at the same time keeping the sport exciting. In the 2013 season, teams were allowed to have eight 2.4 litre V8 engines, rated at 730 horsepower. The cars weighed 642 kg empty and 792 kg when fueled up with approximately 150 kg of fuel. In the 2014 season, the 1.6 litre V6 engine was rated at 760 horsepower and the empty car was allowed to be heavier because the fuel was limited to 100 kg. Also, the FIA allowed only five engines in 2014 and reduced it to four engines in 2015. The organization wants to promote not only engine efficiency, but longevity, and also to shift to a smaller number of cylinders in the engine that better reflects the top-end of engines used in most road cars. Perhaps equally importantly, the FIA will be adding more and more restrictions over the seven-year period during which the regulations for the 1.6 litre V6 engine will be used, much as it has done for the 2.4 litre V8 engine that was a mainstay of the sport between 2006 and 2013.

The top speeds last season for cars using the V6 engines were around 225 miles per hour, which is fast but still a little slower than the 233 miles per hour speed of the 3 litre V10 engines used earlier than 2006, when the V8 engines were mandated. (Still, that is not bad for a V6 engine, of course.)

Another big change with the 2014 rules was a ramping up of the Kinetic Energy Recovery System, which recovers energy from the braking systems in the race cars so it can be deployed as power to drive more speed. The KERS in prior cars was capped at 80 horsepower, but this has been doubled up to 160 horsepower with the addition of thermal energy recovery in the new systems known as ERS.

As the new FIA racing rules get ratified, often a year in advance of a new season, the engineers at each team get to work. Testing of the new cars begins in February, the racing starts in March and runs through November, and the engineering never stops. During the 2015 season, Sahara Force India had two engine upgrades, crankshaft changes, and a fuel formula change that added more energy. “If you are allowed to do it and you don’t, then you go backwards because everybody is working hard to develop the engine when the rules change.”

In essence, the game has changed with Formula One racing from one of pure speed to one where the most efficient car – in terms of fuel consumption and energy reuse as well as lower total cost – will be the one that has the highest chance of winning races. Not so ironically, this is precisely the same flip that has happened in high performance computing over the past several years, where efficient and cost-effective performance is more important than raw performance.

Pushing The Limits

The engineers that design Formula One race cars have four main tools at their disposal. Wind tunnels, clusters that simulate the aerodynamics of the car, other systems that simulate the tires, and full car simulators that take the information from the car models and turn them into a simulator that allows drivers to see how changes to the car might turn out on the course. (It is better to test the ideas in a simulator before committing them to manufacturing.) Ultimately, of course, it is the drivers – in this case Nico Hulkenberg and Sergio Perez – who win the races.

Wind tunnels cost on the order of $50 million to $60 million, and a decade ago, some of the Formula One racing teams were building a second tunnel because running a wind tunnel 24×7, 365 days a year did not deliver enough capacity to do all of the model testing that was required to make the many, many modifications to the cars over the course of a year. In more recent years, the FIA has cut back the wind tunnel time to 25 weeks a year, and this has had the desired effects of keeping the cost of a racing team down while at the same time driving innovation in CFD software. However, the current FIA rules restrict the capacity of CFD modeling systems to 25 teraflops – an amount of capacity that seems huge looking back to 2008, but which could be done with a bunch of server nodes with a handful of accelerators each these days.

There is a continuous flurry of engineering activity at Sahara Force India’s facility in Silverstone, a village in the so-called Motorsport Valley that stretches from Surrey to East Anglia in England where the vast majority of Formula One teams are located. During the weeks when the wind tunnels are shut down according to FIA rules, the team is making parts, and when the wind tunnel fires up, it runs continuously as parts are tested on 60 percent scale models of the vehicles. It takes three to five minutes to test a part on the scale model, and after all of the data is collected, the team decides which full-sized parts to manufacture to test on the real cars.

Designing those parts is done on engineering workstations and a Linux cluster that back-ends them to run simulations. Like many car manufacturers and most Formula One teams, Sahara Force India uses the CATIA and ENOVIA applications from Dassault Systemes as its CFD tools.

“For us, the most important thing is surfacing, since most of the performance enhancements come from aerodynamics,” explains Szafnauer. “For us to be able to surface properly is more important than some mechanical design or manufacturing features, and CATIA is pretty good at both.”

You may recall that many of AMD’s top-end Opteron server chips were named after Formula One races, and the company took a shining to this niche market as it expanded its reach in the HPC market in the 2000s. The cluster that Sahara Force India has, for example, has specialized Interlagos chips that have been tuned specifically to get the most performance out of CFD applications within a 25 teraflops performance ceiling. This cluster has 28 pre-processing and post-processing nodes, and another 432 solver nodes to run the core fluid dynamics applications across more than 7,000 cores in the system. The cluster has 400 TB of storage, and the nodes and the storage are all linked together by 10 Gb/sec Ethernet switches. (Sahara Force India can’t talk specifically about this iron in terms of brands.)

The company’s prior CFD cluster, which was rated at 5 teraflops, is still used to do tire simulations. The models from both of the team clusters are pumped into a new, state-of-the-art car simulator for drivers to play with.

Szafnauer tells the story of how all of this comes together in Silverstone, and there is an incredible amount of specialization: “CFD is used to give us a clue as to what direction we want to take in doing these experiments. We have three teams in the car design: a front team, a rear team, and a mid team. The front team looks at the front wing and the vortices that it sets up, the mid team would take those vortices and translate it to the rear team which would manipulate it to get the best aerodynamics. Sometimes, you want vortices to hit a certain part of the car at high energy levels to create down force, and other times you want to shed force so you have less drag. It all works together. In those three teams, we have aerodynamicists, model makers, and computational fluid dynamicists. The latter start the experiments and the rest of the team take over from there, designing the parts, testing them in the tunnel, and analyzing them.”

All of this has to be done on a relatively modest CFD performance budget, as dictated by the rules governing Formula One, and one that has not been increased along the Moore’s Law curve that other HPC customers get to ride up. And that is why getting every last clock dedicated to designing parts and predicting their performance is key. When Sahara Force India got its latest 25 teraflops cluster two years ago, it partnered with Univa to put its Grid Engine workload scheduler on the system, which allows for engineers to submit jobs for modeling and push utilization on the CFD cluster up into the 97 percent range at a sustained level.

“If we parallel flow some jobs across the cluster, Univa manages all that, too,” says Szafnauer. “The fact that the computation is capped at 25 teraflops means you need to use it as efficiently as possible. If you can be more efficient, you gain a strategic or performance advantage, and that is where Univa helps us.”

With wind tunnels being so expensive to build and maintain and the parts manufacturing for models also being costly, teams increasingly rely on CFD simulations for their development work. Szafnauer estimates that it costs on the order of $10 million to make test parts for the cars every year, and this, in our estimation, is far less than might be spent buying compute capacity to run more complex CFD workloads with richer simulations.

Better CFD software is something that Szafnauer thinks the Formula One sport should help drive, much as it has so many other innovations in engine and powertrain design and body manufacturing for the automotive industry at large.

“I think if we are one day restricted from using wind tunnels, the innovation in CFD will increase by leaps and bounds,” he says. “Race car dynamics are much more sophisticated than with airplanes, which are pretty basic when it comes to aerodynamics. With race cars, there are interactions between many components around which you have to manage the aerodynamics–front wheels, wing end plates, and so on. When we drive around, there are edge effects and floor effects, such as the effect of tire squirt on aerodynamics. If you are driving in water, as the tire hits the water, it squirts the water out so the tire can grip the road. The same thing happens with air, but you don’t see it. If you don’t manage that tire squirt, it can cause aerodynamic effects on other structures around the car, and it has a huge impact. You don’t care about this kind of minutiae on an airplane because the surfaces and the shapes are pretty pure. On a race car, you really have to understand them.”

The one thing that will not happen, of course, is a free-for-all where the Formula One teams are allowed to use as much computing power to simulate their vehicles as they can afford. FIA will not grant that. But there could be a day, in the not too distant future, where the systems at Formula One racing teams are as close to the cutting edge as their actual cars are, and for very sound price/performance reasons that benefit the sport and any other field that depends on CFD applications.