Infinite scroll is a UX staple today. It’s a seamless way to get users to potentially see everything that you’ve got to offer without showing them where your content ends. It makes content discovery much easier for users, who no longer have to wait for an entire page to load every time they want to see more content!

But what about indexing bots?

Indexing bots can’t emulate user interactions like scroll (or can they? 🤔 Let’s assume they can’t for the purpose of this article 😄), which means that indexing bots get to see only a subset of the content that you have to offer … and this is extremely detrimental, from an SEO perspective, since your site is valued by the content that the bot sees.

An Example

Let’s say we are building a news feed that, on first load, will render five news articles. And as a user scrolls, five more articles are fetched asynchronously and rendered.

Now, a user can discover more articles as he/she scrolls, but we need to figure out a way to present an indexing bot with all of our articles, or at least be able to tell the bot where it can find the next set of articles.

Let’s fix this!

Step 1: Chunk the feed

Enable the feed to load additional pages of articles on the first load. One way to do this is by adding a parameter to the page url.

For example, let’s say our news feed loads on super-awesome-website.com. Accessing the url will render the first five articles.

But, if we access super-awesome-website.com?page=2, it should render with the next five articles, i.e., page 2.

Doing this actually kills two birds with one stone. This helps us fix our SEO issue, while also enhancing our site’s accessibility on non-javascript browsers!

Yay 🎉

Step 2: Let the indexing bot know

Now that we can individually load pages of articles by parameterising our url, we need to let the indexing bot know how to reach these pages. And this is done in the head section of our page.

By providing a <link rel="next"> and a <link rel="prev"> , we can show the bot how to fetch the next page of articles. (And previous page of articles, just in case the bot starts indexing from somewhere in the middle).

So, in our example, if we are on super-awesome-website.com?page=2, we need to add the following in our head section

<head>

<link rel="prev" href="super-awesome-website.com?page=1" />

<link rel="next" href="super-awesome-website.com?page=3" />

</head>

<!--

On the first and last page of the articles, you can omit the

<link rel="prev"> and <link rel="next"> tags respectively.

-->

And that’s about it actually!

Now any indexing bot visiting your site will know where to find the rest of the content that would have otherwise been hidden from it. 🤖