February 20, 2015

Pagination with Phoenix & Ecto

I’ve been working on a web application built in Elixir. I’m using Phoenix as the web framework and Ecto to talk to my database. As the amount of data in the application grew, I needed to paginate some of the views. I wasn’t able to find an existing pagination solution for these tools so I ended up building my own. This post will discuss what I built.

Goal

Once we’re done, we should be able to paginate any Ecto query using parameters provided by a Phoenix controller action. I’m going to assume you’re familiar with building composable Ecto queries. If you aren’t, read this post.

Let’s assume we have a post model.

defmodule Post do use Ecto . Model schema "posts" do field :name , :string field :content , :string field :published , :boolean end def published ( query ) do query |> where ([ p ], p . published == true ) end def order_by_name ( query ) do from p in query , order_by: [ asc: p . name ] end end

Let’s also assume we have an Ecto repository.

defmodule Repo do use Ecto . Repo , ... end

Our API for the pagination should accept an Ecto query and a map of options representing the params from a Phoenix controller. The options may provide the page key (defaulting to 1) and the page_size key (defaulting to 10). When we’re done, we should be able to write the following code.

defmodule PostController do use Phoenix . Controller def index ( conn , params ) do paginator = Post |> Post . published |> Post . order_by_name |> Paginator . new ( params ) render conn , :index , posts: paginator . entries , page_number: paginator . page_number , page_size: paginator . page_size , total_pages: paginator . total_pages end end

Building the Paginator

Here’s a skeleton module to kick things off.

defmodule Paginator do defstruct [ :entries , :page_number , :page_size , :total_pages ] def new ( query , params ) do % Paginator { entries: [], page_number: 0 , page_size: 0 , total_pages: 0 } end end

Ok, so we’ve got some dummy data and a struct that we’re going to return. Let’s start by calculating the current page of entries. We’ll do this by adding a limit and offset to our Ecto query based on the page size and page number.

defmodule Paginator do defstruct [ :entries , :page_number , :page_size , :total_pages ] def new ( query , params ) do page_number = params |> Dict . get ( "page" , 1 ) |> to_int page_size = params |> Dict . get ( "page_size" , 10 ) |> to_int % Paginator { entries: entries ( query , page_number , page_size ), page_number: page_number , page_size: page_size , total_pages: 0 } end defp entries ( query , page_number , page_size ) do offset = page_size * ( page_number - 1 ) query |> limit ([ _ ], ^ page_size ) |> offset ([ _ ], ^ offset ) |> Repo . all end defp to_int ( i ) when is_integer ( i ), do : i defp to_int ( s ) when is_binary ( s ) do case Integer . parse ( s ) do { i , _ } -> i :error -> :error end end end

To build the entries, we determing the offset by multiplying the page size by the current page number. We then limit the number of results returned to the page size. Finally, we fetch the results using our Ecto Repo. We also added a helper function to_int so that we can deal with string values provided by the params.

Now, all that’s left is calculating the total_pages . Let’s do it.

defmodule Paginator do defstruct [ :entries , :page_number , :page_size , :total_pages ] def new ( query , params ) do page_number = params |> Dict . get ( "page" , 1 ) |> to_int page_size = params |> Dict . get ( "page_size" , 10 ) |> to_int % Paginator { entries: entries ( query , page_number , page_size ), page_number: page_number , page_size: page_size , total_pages: total_pages ( query , page_size ) } end defp ceiling ( float ) do t = trunc ( float ) case float - t do neg when neg < 0 -> t pos when pos > 0 -> t + 1 _ -> t end end defp entries ( query , page_number , page_size ) do offset = page_size * ( page_number - 1 ) query |> limit ([ _ ], ^ page_size ) |> offset ([ _ ], ^ offset ) |> Repo . all end defp to_int ( i ) when is_integer ( i ), do : i defp to_int ( s ) when is_binary ( s ) do case Integer . parse ( s ) do { i , _ } -> i :error -> :error end end defp total_pages ( query , page_size ) do count = query |> exclude ( :order_by ) |> exclude ( :preload ) |> exclude ( :select ) |> select ([ e ], count ( e . id )) |> Repo . one ceiling ( count / page_size ) end end

We need to do a few things to calcuate our total pages. First, we exclude any non-countable components of the query (including order_by , preload , select , etc). We then add a select statement to grab the count. Finally, we divide the count by the page_size and calling ceiling on the result.

And that’s it! We can now paginate our Ecto queries.

Wrap Up

I hope this deep-dive was helpful in understanding pagination in Ecto and Phoenix applications. If you’re interested in using this in your application, I’ve released a more robust hex package for Ecto pagination called Scrivener.