Two year ago I’ve developed a data ingestion system and I’m planning to migrate it from Windows to Linux and Docker.

This system is composed of two types of services that are communicating over HTTP:

a front-end service (Web API) that receives various payloads in JSON format, validates them, does data transformation and aggregation then sends the data to the back-end service

a back-end service (Web API) that receives data from the front-end services and based on the payload type it persists it to various storages (PG, ES, Redis, OpenStack Swift)

At peak hours the current system has an ingestion rate of 1000 payloads per second and it’s using about 4GB RAM. Before porting it to a different technology, I have to make sure that the new tech can handle this kind of load and I also want a smaller memory footprint. Since I enjoy writing APIs with C# and Go, I decided to code a basic data flow in both technologies and run some tests on a staging server that has similar specs with the production ones.

Data ingestion prototype

The prototype is composed of two web apps that are running in Docker containers.

Front-end:

exposes a HTTP POST endpoint on which it receives a JSON payload from a caller

on receive it deserializes the payload into an object

serializes the object back to JSON

instantiates an HTTP client and makes an HTTP POST request to the back-end service

waits for the back-end service to process the payload

disposes the HTTP client and returns a HTTP 200 Code to the caller

Back-end:

exposes an HTTP POST endpoint on which it receives a JSON payload from the front-end service

on receive it deserializes the payload into an object

returns a HTTP 200 Code to the front-end service

Both services are instrumented with Prometheus. Prometheus collects the following metrics: rate of HTTP requests per second, HTTP requests latency, CPU, RAM, NET and IO usage of each container.

The front-end data ingestion handler in ASP.NET Core looks like this:

[HttpPost] public IActionResult Event ([ FromBody ] Payload payload ) { if (! string . IsNullOrEmpty ( _settings . ProxyFor )) { using ( var client = new HttpClient ()) { client . BaseAddress = new Uri ( _settings . ProxyFor ); var content = new StringContent ( JsonConvert . SerializeObject ( payload ), Encoding . UTF8 , "application/json" ); var result = client . PostAsync ( "/ingest/data" , content ). Result ; result . RequestMessage . Dispose (); result . Dispose (); } } return new EmptyResult (); }

And the back-end handler:

[HttpPost] public IActionResult Data ([ FromBody ] Payload payload ) { dynamic data = JsonConvert . DeserializeObject < dynamic >( payload . Data ); return new EmptyResult (); }

The front-end data ingestion handler in Go:

func eventIngestHandler ( w http . ResponseWriter , r * http . Request ) { decoder := json . NewDecoder ( r . Body ) var p Payload err := decoder . Decode ( & p ) if err != nil { http . Error ( w , http . StatusText ( 400 ), 400 ) } app , _ := r . Context () . Value ( "app" ) . ( AppContext ) if app . Role == "proxy" && app . Endpoints != "" { endpoints := strings . Split ( app . Endpoints , "," ) for _ , endpoint := range endpoints { err := redirectPayload ( p , endpoint + "/ingest/data" ) if err != nil { http . Error ( w , http . StatusText ( 502 ), 502 ) } } } } func redirectPayload ( p Payload , url string ) error { b := new ( bytes . Buffer ) json . NewEncoder ( b ) . Encode ( p ) r , err := http . Post ( url , "application/json; charset=utf-8" , b ) if err != nil { return err } defer r . Body . Close () return nil }

And the back-end handler in Go:

func dataIngestHandler ( w http . ResponseWriter , r * http . Request ) { decoder := json . NewDecoder ( r . Body ) var p Payload err := decoder . Decode ( & p ) if err != nil { http . Error ( w , http . StatusText ( 400 ), 400 ) } }

Load testing specs

The ASP.NET Core apps are using the microsoft/dotnet latest (1.0.1) Docker image and the Go apps are using golang:1.7.1-alpine image.

Server specs:

Ubuntu 16.04.1 LTS x64

Docker 1.12.2

Intel Xeon X5650 12M Cache, 2.66 GHz, 12 threads

12 GB RAM

Intel 82575EB Gigabit Ethernet Controller

Load test machine specs:

Ubuntu 16.04.1 LTS x64

ApacheBench v2.3

Intel Core i7-4790 8M Cache, 3.60 Ghz, 8 threads

ApacheBench command:

ab -k -l -p payload.json -T application/json -c 50 -n 10000 http://<server-ip>:<app-port>/ingest/event

JSON payload:

{ "data" : "{'job_id':'c4bb6d130003','container_id':'ab7b85dcac72','status':'Success: process exited with code 0.'}" }

Load testing results

ApacheBench reported that the ASP.NET Core front-end service has processed 10K request in 31 seconds at a rate of 319 request per second:

Concurrency Level: 50 Time taken for tests: 31.348 seconds Complete requests: 10000 Failed requests: 0 Keep-Alive requests: 10000 Total transferred: 1160000 bytes Total body sent: 2950000 HTML transferred: 0 bytes Requests per second: 319.00 [#/sec] (mean) Time per request: 156.741 [ms] (mean) Time per request: 3.135 [ms] (mean, across all concurrent requests) Transfer rate: 36.14 [Kbytes/sec] received 91.90 kb/s sent 128.04 kb/s total Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.5 0 34 Processing: 8 156 175.9 98 1144 Waiting: 8 156 175.9 98 1144 Total: 8 156 175.9 98 1144

ApacheBench reported that the Go front-end service has processed 10K request in 3 seconds at a rate of 3213 request per second:

Concurrency Level: 50 Time taken for tests: 3.112 seconds Complete requests: 10000 Failed requests: 0 Keep-Alive requests: 10000 Total transferred: 1400000 bytes Total body sent: 2950000 HTML transferred: 0 bytes Requests per second: 3213.09 [#/sec] (mean) Time per request: 15.561 [ms] (mean) Time per request: 0.311 [ms] (mean, across all concurrent requests) Transfer rate: 439.29 [Kbytes/sec] received 925.65 kb/s sent 1364.94 kb/s total Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.7 0 33 Processing: 4 15 15.9 13 380 Waiting: 4 15 15.9 13 380 Total: 4 15 16.3 13 385

Prometheus reported the following resource usage:

Service CPU RAM RAM (after test) ----------------------------------------------------- ASP.NET front-end 12.65% 224.04MB 104.74MB ASP.NET back-end 4.23% 134.15MB 54.98MB Go front-end 10.95% 9.13MB 4.40MB Go back-end 3.86% 6.72MB 3.36MB

Improving ASP.NET Core HTTP client code

Instead of creating a HTTP client on each call I changed the code and used a static client and switched to async:

private static HttpClient client = new HttpClient (); [HttpPost] public async Task < IActionResult > Event ([ FromBody ] Payload payload ) { if (! string . IsNullOrEmpty ( _settings . ProxyFor )) { var content = new StringContent ( JsonConvert . SerializeObject ( payload ), Encoding . UTF8 , "application/json" ); var result = await client . PostAsync ( _settings . ProxyFor + "/ingest/data" , content ); result . RequestMessage . Dispose (); result . Dispose (); } return new EmptyResult (); }

The results improved a lot. ApacheBench reported that the ASP.NET Core front-end service has processed 10K request in 10 seconds at a rate of 936 request per second:

Concurrency Level: 50 Time taken for tests: 10.674 seconds Complete requests: 10000 Failed requests: 0 Keep-Alive requests: 10000 Total transferred: 1160000 bytes Total body sent: 2950000 HTML transferred: 0 bytes Requests per second: 936.87 [#/sec] (mean) Time per request: 53.369 [ms] (mean) Time per request: 1.067 [ms] (mean, across all concurrent requests) Transfer rate: 106.13 [Kbytes/sec] received 269.90 kb/s sent 376.03 kb/s total Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.4 0 10 Processing: 5 53 11.3 51 235 Waiting: 5 53 11.3 51 235 Total: 5 53 11.4 51 239

Scale up with Docker Swarm

My goal is to make the system take 1K req/sec, so I decided to run the containers on Docker Swarm and scale the front-end service to x3.

ApacheBench reported that the ASP.NET Core front-end service x3 has processed 10K request in 6 seconds at a rate of 1615 request per second:

Concurrency Level: 100 Time taken for tests: 6.190 seconds Complete requests: 10000 Failed requests: 0 Keep-Alive requests: 10000 Total transferred: 1160000 bytes Total body sent: 2950000 HTML transferred: 0 bytes Requests per second: 1615.48 [#/sec] (mean) Time per request: 61.901 [ms] (mean) Time per request: 0.619 [ms] (mean, across all concurrent requests) Transfer rate: 183.00 [Kbytes/sec] received 465.40 kb/s sent 648.40 kb/s total Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.5 0 16 Processing: 5 60 28.2 53 415 Waiting: 5 59 28.2 53 415 Total: 5 60 28.4 53 419

As expected, Docker Swarm balanced the load between the front-end instances and I could reach over 1K req/sec. The back-end service didn’t need to be scaled at this load. I did a load test on the back-end service (single instance) and it got up to 4K req/sec.

For the Go app I’ve raised the stakes to 100K requests at a concurrency level of 100. ApacheBench reported that the Go front-end service x3 has processed 100K request in 11 seconds at a rate of 9017 request per second:

Concurrency Level: 100 Time taken for tests: 11.089 seconds Complete requests: 100000 Failed requests: 0 Keep-Alive requests: 100000 Total transferred: 14000000 bytes Total body sent: 29400000 HTML transferred: 0 bytes Requests per second: 9017.95 [#/sec] (mean) Time per request: 11.089 [ms] (mean) Time per request: 0.111 [ms] (mean, across all concurrent requests) Transfer rate: 1232.92 [Kbytes/sec] received 2589.14 kb/s sent 3822.06 kb/s total Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 0.2 0 22 Processing: 4 11 7.7 10 430 Waiting: 4 11 7.7 10 430 Total: 4 11 7.8 10 435

Observations

By default the ASP.NET Core garbage collector System.GC.Server mode is enabled, this works well on Windows but on Linux made the front-end service go up to 2GB of memory. After it reached 2GB, the HTTP client started to crash on every request. If you are deploying ASP.NET Core with Docker then you should set "System.GC.Server": false in project.json.

I’ve run this test for 6 hours every 10 minutes, while the Go performance was consistent on every run, the ASP.NET Core fluctuated a lot. I’ve seen ASP.NET Core jump to 20% CPU and drop to 200 req/s some times.

When I’ve run the tests with 100K requests at a concurrency level of 100, both HTTP clients crashed after a while. In Golang I could fix this by disabling KeepAlive and setting the MaxIdleConnsPerHost to 100. I suppose there is a way to set the same things on ASP.NET Core too.

Conclusions

Benchmark results:

REQ/sec TIME REQ Concurrency Memory ------------------------------------------------------------- ASP.NET Core 936 10sec 10K 50 224MB Go 3213 3sec 10K 50 9MB ASP.NET Core 1324 75sec 100K 300 235MB Go 6051 16sec 100K 300 12MB

For my use case, the load tests showed that the Go HTTP stack is way faster then ASP.NET Core. Scaling the front-end service to x3 made the ASP.NET Core reach my 1K req/s goal, but the memory usage is very high compared to Go.

I know that ASP.NET Core is for all intents and purposes a brand new platform. However, running the load test on the back-end service resulted in 4K req/s which leads me to believe that, while Kestrel is very fast, there may be a bottleneck when sending HTTP requests. The .NET Core team has made some huge improvements on the Kestrel server performance this year. In terms of serving data, Kestrel has hit 1M req/sec, so I expect the data ingestion rate to improve in the future.

If you wish to run the tests yourself, the ASP.NET Core repo is on GitHub here and the Go repo here.