Paginating through lists using offset pagination

The Zendesk Support, Talk, and Help Center APIs have several endpoints that return lists of items, such as tickets, users, or articles. For performance reasons, the API does not return large record sets all at once. It breaks up the results into smaller subsets and returns them in pages. The number of items per page varies by endpoint. For example, the tickets and users endpoints return 100 items per page while the articles endpoint returns 30 items per page.

This article explains how to paginate through the lists using offset pagination, previously known as pagination. While offset pagination remains supported for all resources, Zendesk recommends migrating to cursor pagination where possible. Cursor pagination provides greatly improved performance when retrieving extremely large record sets.

See Comparing cursor pagination and offset pagination and Paginating through lists using cursor pagination for details on migrating to cursor pagination.

Disclaimer: Zendesk provides this article for instructional purposes only. Zendesk does not support or guarantee the code. Zendesk also can't provide support for third-party technologies such as Python and Perl.

Using offset pagination

When using offset pagination, the JSON object returned contains a next_page property with the URL of the next page of results, if any. Example:

"tickets": [ ... ],
"next_page": "https://example.zendesk.com/api/v2/tickets.json?page=2",
...

If there's no next page, the value is null.

"tickets": [ ... ],
"next_page": null,
...

Your code should check the next_page property when it calls any list endpoint. If null, it can move on. If not null, it should make another request using the specified URL.

Sorting results

Certain resources allow sorting the record set by certain attributes. For example, tickets may be sorted by their creation date by specifying the sort_by=created_at request parameter:

url = 'https://example.zendesk.com/api/v2/tickets.json?sort_by=created_at'
...
url = 'https://example.zendesk.com/api/v2/tickets.json?page=2&sort_by=created_at'

Some endpoints have an additional sort_order parameter that should be set to asc. See Articles.

See the API documentation for the specific resource.

Pseudocode example

The following pseudocode is a common pattern for paginating through items:

url = 'https://example.zendesk.com/api/v2/users.json'
while url is not null:
    response = request(url)
    // do something with the response
    url = response('next_page')

Python example

This example requires the third-party Requests library.

url = 'https://example.zendesk.com/api/v2/help_center/sections/200646/articles.json'
while url:
    response = session.get(url)
    data = response.json()
    for article in data['articles']:
        print(article['title'])
    url = data['next_page']

After getting and processing the first page of results, the script checks the value of the next_page property. If the value is null, which Python considers false, the loop stops.

Limitations

Paginated data may be inaccurate because of the real-time nature of the data. One or more items may be added or removed from your database instance between requests and during the course of iterating over all the items. In the stateless, page-based pagination approach used by the Zendesk REST API and others like it, each next page request causes the server to query the database and return the specified subset. The full record set is not retrieved and stored statically in memory for follow-up requests.

The server uses the total record count divided by the maximum number of records per page to determine the subset of records to return with each next page request. If the total record count changes between requests, the subset of records selected for the subsequent requests may change too. If records are added, some records may be selected again. If records are removed, some records may be skipped. To better understand this phenomenon, see Paginating Real-Time Data with Cursor Based Pagination on sitepoint.com.

One way to reduce pagination inaccuracies -- though not eliminate them altogether -- is to sort the results from oldest to newest so that new records added during pagination affect only the last pages, if at all. All things being equal, the early pages always have the same 100 sorted records even if the total record count changes.