While it’s true that we already have an infinite scroll article at Unagi, some releases have happened since it was written regarding the amazing Pagy gem. The implementation is almost identical to our previous article, but the performance improvements are remarkable.

What ‘s changed?

With Pagy’s 9.13 version comes Keyset pagination, which is thought specifically to handle infinite scrolling with big amounts of records. Unlike traditional pagination, keyset pagination is optimized for large datasets and avoids performance issues caused by offset-based pagination. In order to get a better grip on this, let’s take a look at how offset pagination works, representing it with a query:

SELECT * FROM articles ORDER BY created_at DESC LIMIT 10 OFFSET 20;

This is what the helper pagy does: skip the first N records (OFFSET N) and retrieve the next batch based on LIMIT.

The issue with this is that, with large amounts of records, the database must scan and discard OFFSET rows before fetching the requested ones.

On the other side, a query executed by pagy_keyset looks like this:

SELECT * FROM articles WHERE created_at < '2024–03–07 12:00:00' 
ORDER BY created_at DESC LIMIT 10;

Instead of skipping rows, it filters based on the last record from the previous page. This is much more efficient, but it does require your database to have an unique, sequentially sortable column (like id or created_at).

Finally, it’s worth mentioning that keyset pagination has several front end limitations: pagy_keyset doesn’t enable jumping from an arbitrary page to another, only from one page to the next, which makes it suitable only for pages with ‘load more’ links or infinite scrolling.

Implementation

Having gone through the major performance differences between offset and keyset pagination, we’ll check out these new features with a practical example:

Imagine we have a basic app with Posts, which are listed in the index for the logged user to see. We’ll leave the Pagy setup to the gem’s tutorial, but let’s say you load and then display the posts like this:

class PostsController < ApplicationController
  def index
    @posts = Post.order(created_at: :desc)
  end

# index.html.erb
<ul>
  <%= render @posts %>
</ul>

As your database grows larger, you’ll probably notice that the website loads slower. This would be a great opportunity to think of ways of improving your website’s performance, and an easy way to do this is by adding pagination.

We already mentioned the advantages of pagy_keyset, and all you need to do here is load your records with it:

class PostsController < ApplicationController
include Pagy::Backend

  def index
    @pagy, @posts = pagy_keyset(Post.order(created_at: :desc))
  end

Pagy is going to take care about dividing the Posts in sections, and we’ll use Turbo to add each one of those sections to the page dynamically, without a full page refresh, making the infinite scroll seamless. In order to do that, we’ll wrap the list within a turbo frame, and you’ll also see a helper method after the posts list. We’ll get to what it does in a minute.

# index.html.erb
<%= turbo_frame_tag :posts do %>
  <ul>
    <%= render @posts %>
  </ul> 
<% end %>

<%= render_load_more(@pagy) %>

As its name implies, the helper will load the missing records. We’ll define it in ApplicationHelper (or whichever helper you want), and you’ll see that all it does is rendering a partial, passing the @pagy parameter which we defined in PostsController:

module ApplicationHelper
include Pagy::Frontend

  def render_load_more(pagy)
    render 'shared/load_more', pagy: pagy
  end

The magic starts here, in the partial. We’ll first check if there’s any more records to load by checking pagy&.next. If that’s the case, we’ll add them to the bottom of the posts list, using Hotwire.

<%# locals: (pagy: nil) -%>

<% if pagy&.next %>
  <%= turbo_frame_tag 'load_more', loading: :lazy, src: { page: pagy.next, format: :turbo_stream } do %>
    Loading...
  <% end %>
<% end %>

As you can see, we have another turbo frame (load_more) that will lazy load whatever is fetched in the source parameter, and display it in the view (remember this turbo frame is after the posts, going back to the <%= render_load_more(@pagy) %> sentence). However, if you’re familiar with Turbo frames, you might notice something odd: the source is a hash instead of an url.

What does this mean? The Rails’ url helper will interpret it as passing parameters to the current url, so we’ll be setting the page param as the next group of results, and then specifying Turbo Stream as format:

src: { page: pagy.next, format: :turbo_stream }

Remember that, by passing this hash as parameters, we’re sending them to the current url (the Posts index) so the server will respond with index.turbo_stream.erb, a view which we’ll have to add.

And what do we want to do in this request’s response? Just to add the next group of results to the bottom of the page, so that the user gets the famous effect of infinite scroll. Then, we’ll also need to update the load_more Turbo Frame to fetch the next batch of Posts if needed:

# index.turbo_stream.erb
<%= turbo_stream.append 'posts' do %>
   <%= render partial: 'post', collection: @posts %>
<% end %>

<%= turbo_stream.replace 'load_more' do %>
   <%= render_load_more(@pagy) %>
<% end %>

Replace and append are just a few of the Turbo Stream actions, and you can see their detail information and other actions here.

That’s all there is! What’s great about this approach is that, apart from being the most efficient way of achieving an infinite scroll, it’s very reusable, simply by loading your records with the pagy_keyset method and then adding the render_load_more helper in the view where you want infinite scroll.

If you found this interesting, at Unagi we’re always keeping an eye at new technology related releases and writing about them, so feel free to check out our other articles!