7+ Million HTTP requests per second from a single server

It’s coming up to 2 years since I last posted about the performance of ASP.NET Core; during its preview, pre version 1.0. As preview 2 of has ASP.NET Core 3.0 has just been to released; it’s time to follow up, and find out how its evolved.

ASP.NET Core 2.2 (Current)

Looking at the latest run from the TechEmpower Benchmarks continuous results ASP.NET 2.2 is the 3rd fastest webserver (0.046% off the top spot); able to respond to 7 Million HTTP request per second:

I recently saw another post; about a different product, where throughput that was measured monthly; so if we use that time period, that’s 18.1 trillion HTTP requests per month.

It’s also an extraordinary amount of bandwidth; enough to continuously saturate a 10GBps link. These results are with the webserver and load tester running inside Docker containers, on two different physical Linux machines *; connected with a 10GbE network.

All this throughput from a single server! ASP.NET Core is fast on Linux (and on Windows).

*“Citrine” Environment: 14 Core, 28 HT, 32 GB RAM

Compared with Other Servers

How does it compare to other well known servers?

In these “platform” comparisons that’s:

x1.78 faster than ngnix

faster than ngnix x2.93 faster than Java’s Servlet ( x7.76 faster than Servlet on Tomcat)

faster than Java’s Servlet ( faster than Servlet on Tomcat) x7.36 faster than Golang’s “net/http” package

faster than Golang’s “net/http” package x8.06 faster than node.js running as a cluster of 28 processes (as node.js is single threaded)

Internet Facing Server

ASP.NET Core’s Kestrel Webserver; which is used in these benchmarks, is an edge server so can be used as an internet facing webserver, as explained in the documentation:

It doesn’t need a second webserver acting as a reverse-proxy server in front and can go full speed. It also works with reverse-proxies if that fits more with your infrastructure.

Data Access Performance

I often hear the defeatist argument that performance like this doesn’t matter because “my database is slow”; well times are a changing…

Postgres

Test type 5: Database updates

Exercises the ORM’s persistence of objects and the database driver’s performance at running UPDATE statements or similar. The spirit of this test is to exercise a variable number of read-then-write style database operations.

HTTP api request => 20 database queries => 20 database updates

ASP.NET Core occupies the 8th, 9th and 10th positions out of 331 entries, for raw, middleware and mvc (which are “platform”, “micro-framework”, and “full-framework” respectively)

The “platform” level runs 550,120 SQL statements per second (13,753 * (20 SELECT + 20 UPDATE ))

Looking that over longer time periods that’s:

550,120 per second

per second 33,007,200 per minute

per minute 1,980,432,000 per hour

per hour 47,530,368,000 per day

per day 1,425,911,712,955 per month

Test type 4: Fortunes

This is the most full featured test, most closely mimicking a wider range of activities a web application executes in combination to produce a web page:

Exercises the ORM, database connectivity, dynamic-size collections, sorting, server-side templates, XSS countermeasures, and character encoding.

ASP.NET Core performs well here also with 298,477 requests served per second; coming 7th out of 350 entries.

298,477 per second

per second 17,908,620 per minute

per minute 1,074,517,200 per hour

per hour 25,788,412,800 per day

per day 773,652,384,000 per month

.NET runs on the Common Language Runtime, which is language neutral, so its also not all about C#; with VB.NET in 14th position.

MySql

.NET can work with many different databases; so looking at MySQL

Test type 3: Multiple database queries

Multiple rows are fetched to more dramatically punish the database driver and connection pool. At the highest queries-per-request tested (20), this test demonstrates all frameworks’ convergence toward zero requests-per-second as database activity increases.

ASP.NET comes 6th out of 178 MySql entries; performing 419,460 queries per second (20,973 * 20)

419,460 per second

per second 25,167,600 per minute

per minute 1,510,056,000 per hour

per hour 36,241,344,000 per day

per day 1,087,240,320,000 per month

Why is Performance Important?

At Illyriad Games we are building a new scale of gaming for Age of Ascent, an Ultra-MMO with real-time twitch combat at unprecedented scale. High performance at a business level means we can do more with less – which directly affects our bottom line.

So the choice and performance of a framework is very important to us.

In the words of TechEmpower as to the motivations for setting up their benchmarks:

Application performance can be directly mapped to hosting dollars, and for companies both large and small, hosting costs can be a pain point. Weak performance can also cause premature and costly scale pain by requiring earlier optimization efforts and increased architectural complexity. Finally, slow applications yield poor user experience and may suffer penalties levied by search engines. What if building an application on one framework meant that at the very best your hardware is suitable for one tenth as much load as it would be had you chosen a different framework?

This has become all the more important with cloud-based hosting; where rather than paying a fixed cost for a server, you pay for only what you need and the usage of that at per minute granularity.

What are .NET Core and ASP.NET Core?

They are fully open-source frameworks under the .NET Foundation; commonly used with C#, VB.NET and F# languages which are all also open source and designed on GitHub.

They run on the open-source “Common Language Runtime” which has the aim:

The goal of the CLR is to make programming easy - from the Book of the Runtime

.NET Core and ASP.NET Core were born in the open-sourcing of .NET and the reimagining of ASP.NET; which also took them from Windows only to cross-platform; additionally supporting macOS and flavors of Linux, BSD and adding ARM to the supported chipsets.

The .NET Community has embraced the open-sourcing of .NET; and its progress has never been so vibrant. A .NET renaissance has begun!

(Graph below from: Matt Warren’s: Open Source .NET – 4 years later)

While grounded in .NET’s original aim of being highly productive to empower developers to achieve more; along the journey they gained a laser like focus on performance so your servers too can achieve more!

Ready for Production?

Yes, we use .NET Core and ASP.NET Core, Bing.com uses them and ASP.NET Core’s webserver Kestrel is already the 12th most used Webserver on the Internet even though it was only first released less than 2.5 years ago.

Jump in, the water’s warm!

ASP.NET Core 3.0 (Preview 2)

Every version of .NET Core has been making huge improvements in performance both to the framework’s core libraries (examples in 2.0 and in 2.1) and introducing new types and concepts to allow higher performance:

Continuing in these improvements, there are many changes in .NET Core 3.0 Preview 2 but perhaps the biggest to us is the .NET Hardware Intrinsics API

Platform Dependent Intrinsics are coming. Thanks for everybody’s work on this project. https://t.co/oK84LFJeU8 — Fei Peng (@fiigii) January 29, 2019

Which allows the Intel SSE, SSE2, AVX2 and Arm/Arm64 SIMD hardware instrinsics to be used directly from managed code (e.g. dotnet/coreclr#22127, dotnet/coreclr#2107, dotnet/coreclr#22187, dotnet/coreclr#22118); and many aspects of the runtime are moving from the C++ portion of the runtime to C# where the JIT can easily target exactly the CPU it’s running on and achieve higher performance (e.g. dotnet/coreclr#21729).

Reduced Allocations

One of the big focuses of the performance work in the frameworks has been to reduce allocations; as the less allocations, the less work the Garbage Collector has to do.

The allocations and performance of the .NET’s async statemachine has been dramatically improved:

It's impressive how much the allocations for the 'async/await' state machine were reduced in .NET Core 2.2 (kudos to @STeplyakov for the great tool https://t.co/cACIXg5u6A) pic.twitter.com/mJzoFNBO03 — Matt Warren (@matthewwarren) November 30, 2018

As have the allocations in ASP.NET Core:

Zero Allocations

In fact combined, with the merging of these two Pull Requests (PRs) aspnet/AspNetCore#4601 and dotnet/coreclr#21159 the steady state allocations for Plaintext on at the platform level has been entirely eliminated and is now zero allocation.

What does Zero allocations look like?

Running a 6 minute, 64 connection test on localhost:

Results in 122,946,688 HTTP requests and responses; and the allocations?

Less than 1MB allocated in total for processing more than 122 Million requests and generating their responses!

Continuous Performance Monitoring

I mentioned earlier “a laser like focus on performance”; but what does that really mean?

All the ASP.NET Core repositories have micro-benchmarks using the great tool BenchmarkDotNet to verify in isolation changes and how they will affect performance.

However, more importantly there is an entire suite of full system benchmarks of over 100 scenarios, which are then run on both Linux and Windows, physical servers and cloud for a total of 400+ full system performance tests, running continuously as part of their Continuous Integration (CI) several times a day to catch any regressions early.

Public Performance Dashboard

As its fully open source and open to contributors; the full 10 pages of all performance KPIs and time series graphs is completely available and public: aspnet/Benchmarks Results

Looking at the “KPIs - Baselines” page:

We already saw how ASP.NET Core was fast; but looking at the dashboard it shows ASP.NET Core 3.0 is already 30% faster than ASP.NET Core 2.2 for some scenarios!

You can also see that 3.0 preview is tracked and compared against all prior versions as are all the current patch and servicing releases; so there are no surprise impacts from any patching.

It’s not only public; but the dashboard is on continuous display in the Microsoft offices:

Finally got our replacement TV so the #aspnetcore SmurfLab dashboard is back up again! pic.twitter.com/64VbCLoXXr — Damian Edwards 😷🌈#BlackLivesMatter (@DamianEdwards) January 15, 2019

More Than Raw Throughput

As well as throughput; latency; time from cold start to first response, and memory usage is closely tracked over time; for every scenario.

Memory usage for ASP.NET Core 3.0 is greatly reduced:

Cold start time to first response have improved in .NET 3.0 with features like ReadyToRun (AoT) and Tiered Compilation; which when combined give both improved start-up and improved steady-state performance:

Good for serverless scenarios like Azure functions and AWS Lambda; as well as the first spin up of a new container, VM or server when scaling out; or deployment update.

Runtime Performance

The performance is also tracked below ASP.NET Core at lower levels of in the stack. The .NET Core runtime is continuously monitored also; on Windows, Linux, macOS, ARM; and the repository dotnet/performance contains benchmarks used for testing the performance of .NET Frameworks.

Regressions are quickly spotted and reported on so they can be addressed:

ASP.NET Core: Very Fast and Getting Even Faster

.NET and ASP.NET have always be highly productive; and the future with ASP.NET Core and .NET Core is as productive, but also fast; very fast!