Cursor pagination returns a long list of records in pages, and each response carries a cursor: an opaque token that marks where the page ended. The client sends the cursor back to get the next page, so the server resumes from a position rather than counting rows from the start.
Why not an offset
Offset pagination asks for rows 10,000 to 10,099. It has two weaknesses. The database must step past 10,000 rows to answer, so late pages are slower than early ones. And if a row is inserted or deleted between two calls, every later row moves by one position, so the next page repeats a row or skips one.
A cursor avoids both because it encodes a position in a stable order, such as after this time and this record id. In SQL terms it is a keyset query, shown here on an illustrative table:
select *
from events
where (observed_at, id) > (:last_observed_at, :last_id)
order by observed_at, id
limit 100;Rows inserted behind the position do not move it, and the cost of a page does not grow with its depth.
Using a cursor
- Call the endpoint with your filters and no cursor.
- Store the rows from the response.
- Save the cursor from the response.
- Call again with that cursor, and repeat until a response comes back with no rows.
The cursor saved after the last page is the bookmark for incremental sync. Four rules keep the loop safe:
- Treat the cursor as opaque: do not build or edit one.
- Keep the filters the same while you follow a cursor.
- Save a cursor only after its page is stored, so a crash repeats a page and never loses one.
- Expect no random access, because you cannot jump to page 50. To parallelise a download, split it by a filter such as a time range, not by page.
In Fokals data
The Fokals API uses cursor pagination. Feeds run oldest first from a time you set, so the last cursor is the bookmark for incremental sync: the next run starts from it. Each page is a request, and requests count against the rate limits set per key by minute and by day, which pace a long backfill. For the parameters, see the API documentation and the delivery page. The use case on keeping a warehouse in step builds the full loop.
Related terms
- Incremental sync: keeping a copy current by fetching only what is new.
- Rate limit: the cap on requests that paging counts against.
- Bulk export: the route for a first load, instead of paging history.
Frequently asked questions
What is the difference between cursor and offset pagination?
Offset pagination sends a number of rows to skip and a page size, so the position is a count that shifts when rows are added or removed. Cursor pagination returns a token that marks where the last page ended, so the position is tied to a record in a stable order. Cursors stay correct as data changes and cost the same on every page, but they allow no jump to an arbitrary page.
Why does a paginated API return duplicate or missing records?
With offset pagination, a row inserted or deleted between two calls shifts every later row by one position, so the next page repeats a row or skips one. Cursor pagination avoids this because the cursor marks a position in a stable order. Duplicates can still arrive if you retry a page, so key your table on the record identifier and write with an upsert.
How do I resume a paginated download after a failure?
Restart from the cursor of the last page you stored. Save a cursor only after that page's rows are safely written, so a failure can repeat a page but never lose one. If an API expires old cursors, restart from the time of your last stored record instead, and make the load repeatable so a repeated page changes nothing.
The queries and code on this page are examples to adapt. Test them in your own environment before you rely on them.