How to test properties like this?

Every API endpoint requires authentication.

Every API endpoint responds well to supported types of authentication.

Every API endpoint accepts application/json and nothing else.

and nothing else. Every API endpoint returns certain error message when application database is not available.

No API endpoint is susceptible to fuzzing.

There are two usual approaches to problems of this type:

Test only one fixed route and make sure we are setting up the required properties in the application at an appropriate level of abstraction, so that departure of other routes from the standard we pinpointed at that one place is not possible without outright malice. Examples: Sensible developer won’t configure authentication per-route, but per-application. Organization full of sensible developers might just say: we believe that there won’t be a route with [Authorize] attribute forgotten, because we are not foolish and don’t use per-route [Authorize] attributes.

attribute forgotten, because we are not foolish and don’t use per-route attributes. Sensible developer will write code that reacts to a failed database dependency with a single, well-defined exception that won’t ever be swallowed and that would be translated to error message at some kind of interceptor called for every HTTP request. (“Filters” in ASP.NET Core.) Dedicate the task to specialized QA software.

The first approach can work well in practice, although most of professionals would not be comfortable with basically admitting that “we don’t test this: we depend on architectural choices ensuring the guarantees for us”. We all know that architecture is being circumvented all the time.

The second approach leads to unhealthy separation of dev and QA folk and, to be frank, is usually based on informal, human-driven-process-based definition of “all endpoints”.

xUnit to the rescue

Why not to just use white-box testing framework we are comfortable with for the task? Let’s spin up our API server in something like docker-compose during integration testing run and hit it with xUnit project requesting all the endpoints during its run. Because we have Turing-complete system on the side of our testing facility, we will certainly be able to cook up some mechanism how to do this in an elegant fashion.

Before we start, let’s confront quickly some issues we might have with this design. I would argue that certain uneasiness we feel when thinking over writing HTTP-endpoints hitting tests in C# is caused by pain-points that are no longer relevant:

“It’s not clean to not test in a unit-test fashion.” This is remnants of unit-testing evangelization from 00s. We are grown-up programmers now and we know when to use what kind of test. It would be foolish to call testing of live services “unit testing” – but as long we name what we are doing properly, we are fine.

testing of live services “unit testing” – but as long we name what we are doing properly, we are fine. “Testing living services is not reliable enough for programmers to care.” With growing adoption of docker in CI tooling, we are in position to fully control environment we test against.

“Statically-typed languages are not agile enough to be good for this use case.” Come on. Everything is better with static typing.

After we overcome the initial doubts, we have to decide how to tell the framework we want to run the test for “all routes”. This is, finally, what the technical part of the text is about.

xUnit has a concept called fact – that’s just a plain old unit test. It also has a theory, and that’s a proposition that should hold over a specified set of values. This specification is usually explicit like this:

[Theory] [Inline(2, 5)] [Inline(4, 9)] [Inline(15, 42)] void SumOfPositiveNumbersIsPositive(int a, int b) { Assert.True(a + b > 0); }

We would like to write a test like this:

[Theory] [?SomeKindOfSourceOfTruthAboutOurApi?] public async Task EveryOperationNeedsAuthentication(HttpMethod method, string path) { var response = await new HttpClient().SendAsync(new HttpRequestMessage(method, $"https://server.under.test/{path}")); Assert.Equal(HttpStatusCode.Unauthorized, response.StatusCode); }

A great thing about xUnit is that the extensibility point for extending “data sources” is very simple: it’s sufficient to inherit DataAttribute, like this:

public class SomeKindOfSourceOfTruthAboutOurApiAttribute : DataAttribute { public override IEnumerable<object[]> GetData(MethodInfo testMethod) { // TODO } }

We are expected to return, more or less, a list of lists of objects. object[] , the inner list, stands for arguments xUnit is going to pass into every single test invocation. IEnumerable<...> , the outer list, gathers these arguments for multiple API endpoints.

Where to get all the metadata about our API to feed into xUnit though?

OpenAPI to the rescue

OpenAPI is a nice standard for description of REST API interfaces. With a proper tooling, you can translate your controllers into language-independent JSON file describing what endpoints others can call. Then you can use the machine-readable description for various other transformations like automated client, docs, or mock generation.

We are going to take the JSON from build process, put it besides out integration testing library and then call OpenAPI.NET lib all over it. The actual implementation of the attribute we needed can be as simple as this:

public class AllOperationsAttribute : DataAttribute { private readonly string _specLocation; public AllOperationsAttribute(string specLocation) { _specLocation = specLocation; } public override IEnumerable<object[]> GetData(MethodInfo testMethod) { var openApiDoc = new OpenApiStringReader().Read(File.ReadAllText(_specLocation), out var _); foreach (var path in openApiDoc.Paths) { foreach (var operation in path.Value.Operations) { yield return new object[] { OperationToMethod(operation.Key), path.Key }; } } } private HttpMethod OperationToMethod(OperationType operation) { switch (operation) { case OperationType.Get: return HttpMethod.Get; case OperationType.Post: return HttpMethod.Post; case OperationType.Put: return HttpMethod.Put; case OperationType.Patch: return new HttpMethod("PATCH"); case OperationType.Options: return HttpMethod.Options; case OperationType.Head: return HttpMethod.Head; case OperationType.Delete: return HttpMethod.Delete; case OperationType.Trace: return HttpMethod.Trace; } throw new ArgumentException($"I can't translate this kind of OpenAPI operation: {operation}"); } }

The result of the AllOperationsAttribute applied like this:

[Theory] [AllOperations("petstore.yaml")] public async Task EveryOperationNeedsAuthentication(HttpMethod method, string path) { var response = await new HttpClient().SendAsync(new HttpRequestMessage(method, $"https://server.under.test/{path}")); Assert.Equal(HttpStatusCode.Unauthorized, response.StatusCode); }

… is then:

Plenty of work left

As presented, the approach is sufficient for testing authentication, but probably only that. Client need to prepare more precise requests to test more nuanced properties of server.

Luckily, this can be arranged as OpenAPI enables you to include information about how to call endpoints in a great detail. If this information is not yet present in your OpenAPI spec, you unfortunately need to do work – and add it. After you do that though, it would help not only your tests: it’s going to make life easier for consumers of your API as the documentation you give them is going to be richer.

Moreover, if you use OpenAPI spec in your testing process extensively, you make not only sure that your server is valid, but also that your spec is matching. This way, clients generated by your consumers from your OpenAPI spec are going to be more dependable as you won’t be able to release version that breaks the spec generation substantially.

All the examples can be found at GitHub.