Efficient pagination on a table with many records

30 Oct 2017 in MySQL

On this post, the author shows how to efficient paginate on a table with million records.

He shows the simplest (and wrong solution) of selecting all data via

SELECT fields FROM table

with all the filtering done in application logic.

Then he shows the classic

SELECT fields FROM table LIMIT X, Y

which fails (proportional to time) when getting the results near the last pages.

Finally he presents the efficient solution using the id field to filter the results with a query of

SELECT fields FROM table WHERE id > X LIMIT Y

which avoids the need to scan the table to get the right offset. Instead this is optimized on the WHERE portion of the query.

As you may guessed, the problem is to get the id offset right. On web applications that present data in a sequential manner this doesn't seem to be a problem.