Photo by imgix on Unsplash

Within modern (web) development we have seen a strong rise of more and better ways to connect our applications with external data sources. We have seen how REST took the world, GraphQL became a major buzzword and OData getting adopted by big names like SAP and Navision. And while the protocols to query data have grown massively, we haven’t yet seen a strong rise on the JavaScript side of things. Sure, there are all sorts of libraries out there that wrap some stringbuilder around a protocol but most of them don’t go much further then the good old-school SOAP client generators from the previous decade.

In this article I will show a way to query external data sources in a way that is close to LINQ and the IQueryables of C#. We will use expression trees to describe our queries like they are a real language and make generic connectors for a variability of data providers. All of this will lead to a client where we write queries like we write normal predicates and lambdas in TypeScript.

Abstract Syntax Tree

The backbone of our client will be the abstract syntax tree of the query. In this article I will limit myself to a filter operation as shown below to not bury the reader in code (everybody who has done anything with expression trees before should be able to confirm that those things can be huge).

The predicate I will be using as an example for the rest of this article

The AST exists of an expression that contains other expressions inside it. For our example we can define the expression type as a discriminated union of all the different expressions in the tree. Its definition is shown below. Here you can see that we have 3 expressions that all operate on a left and a right expression. Those are called the binary expressions. We also have an expression type for accessing properties of the parameter of the predicate, the index expression. And as last we have an expression that contains a value.

The AST is just an expression, for our example the &&

For those expressions we can create constructor functions so we can create our expression trees a bit easier and avoid christmas trees of nested object declarations:

With those constructors we could create our example expression as shown below. As you can see this gets hard to read very quickly when the tree grows in size and is nowhere close to how expressions are written in regular programming languages. To achieve our goal of writing code against an external system like we just writing a normal everyday line of code we will need to do some more work.

From the next chapter on I will focus on creating the public interface against which we will write our expressions. Here we will see how we can create a real expression builder.

Accessing Properties

If we take a closer look to our example expression we see that it starts with selecting a property from the item we are currently testing our predicate against: b => b.Id … . If this was plain JavaScript b would be the actual object in the collection that is passed into a filter function. But our entities live on the server, so we cannot fetch them all and test them one-by-one (well, we could but it would destroy the entire purpose of building this). So we need something to act like it is a Blog , but actually builds the node of the AST for accessing properties. What we want to achieve is that b => b.Id returns is actually the same as createIndexExpression(‘Id’) . Then we can use that node to continue building our expression until we have the predicate we want.

For this we can use the Proxy feature in JavaScript that allows us to capture all interaction with an object via one method. For this we need to create a handler with a get method that will be called when we access a property for the thing we are pretending to be. With this handler we can create a proxy of the desired type:

if we run fakeBlog.Id we will get an index expression with ‘Id’ for the id field. With this in place we have made sure that we can start creating expression trees by accessing one of the properties of the item. The next step is to wrap this expression in a builder so we can recreate b => b.Id.Equals … .

The Expression Builder

The task of the builder is to allows us to compose the current expression with a new one. All the binary expressions will put the current AST on the left side of the new AST and create a new expression from their input for the right side. Note that there is only one builder for all the different nodes. This is because we don’t have the type information on runtime needed to create different instances based on the current type. For the equals ( == ) operator the expression builder could look as follows:

If the Proxy we made before would wrap the expression inside this builder one could write b => b.Id.equals(1) and it would return an actual expression tree with == as the root node! This concept can be implemented for all the operations we want support and leads to the following:

The handler definition will be updated to use the builder as follows:

Now we can write expressions in a convenient way, like we are used to from our day-to-day programming languages. The example expression of the start of the article can now be written like this:

❤❤❤

Type Safe Expression Trees

In our current implementation we allow the creation of impossible expressions like b => b.Id.includes(‘foo’) . This expression does not represent a valid predicate and it just total nonsense. The builder would be a lot more convenient to use if we wouldn’t allow the construction of such expressions. First we create interfaces for every type of value our expressions can evaluate to: boolean , string and number . (e.g. == results in a boolean if we run it so that is what == evaluates to). After that we rewrite the Fake type to correctly map the properties of a model to the builder that represents their type:

Note that the return types of the operators are also of the type specific interfaces to make sure we can not use equals to compare a StringExprBuilder and BoolExprBuilder . The updated Fake type looks like this:

The IQueryable

The public facing interface of our library will be the IQueryable<T> . It wraps a data source inside this interface and exposes methods to build a query. In our case this will be limited to where and toArray . It will be asynchronous because the data sources are potentially external and will be queried via HTTP. The IQueryable is also lazy, meaning that it will store the query internally until toArray gets called and it is forced to execute the query.

The implementations that we can create of this interface fall into two major categories: external and in-memory. The implementations of external data sources are the ones for protocols like OData and GraphQL. The in-memory implementations will be those for arrays and other data collections inside JavaScript. I will show you two examples: one for OData and one for arrays.

OData IQueryable

For OData we need to compile the AST to a filter operation for the query. This is done with a recursive function that translates every node of the expression tree to a string:

With this we can implement the IQueryable interface for an OData model. Inside the queryable we have an array with all the expression that were passed to the where function. When we call toArray we combine them with the and operator to one query string.

Using this to run the predicate I introduced at the start of this article looks like this:

In-memory IQueryable

For the in-memory versions of the IQueryable we will need a little runtime to run the expressions inside TypeScript. This is also a recursive function, but this time we return the result of the expressions instead of a compiled string:

With this we can wrap an array inside the IQueryable interface. Here we will also keep the lazy nature of querying that we introduced with OData.

And using it looks like this like shown below. Note that with our powerful abstraction we use both arrays and an OData endpoint as like they are the same thing. Based on those two examples one could easily imagine how this would work for other data providers.

Conclusion

In this article I have shown some of the concepts and features inside TypeScript that can be use to take advantage of recent developments within query languages. In this article I have limited myself to where , but the same principles apply for working with selecting properties, expanding relations etc. Of course there exists many more data provider that this could be used for that I didn’t mention.

Resources

The complete source code of the project can be found on Github. To those that are interested in expanding the discussed concepts for other query operators I can recommend this article: