iterateAllDataSourceRows

iterateAllDataSourceRows<T>(fetchWindow): AsyncGenerator<T, void, undefined>

Defined in: src/helpers/pagination.helpers.ts:256

Iterates over every row of a data source query. It works around the API’s 10,000-result-per-query cap.

Data source and view queries cap at 10,000 results. When a query hits this cap, the response’s has_more field is still false. Following next_cursor and has_more alone silently truncates the result set. When a window hits the cap (request_status.type === 'incomplete'), this function starts a new query filtered to created_time >= the last row seen. It de-duplicates rows by id across the window boundary. The query must sort by created_time in ascending order.

Type Parameters

T

T extends object

Parameters

fetchWindow

WindowedFetchFunction<T>

Function that fetches one page. Pass the current cursor. After a window hits the cap, also pass a created_time lower bound for the next window.

Returns

AsyncGenerator<T, void, undefined>

Yields

Individual rows across all windows

Example

const rows = collectAllDataSourceRows((cursor, createdTimeCursor) =>
  notion.dataSources.query(dataSourceId, {
    start_cursor: cursor,
    sorts: [{ timestamp: 'created_time', direction: 'ascending' }],
    filter: createdTimeCursor
      ? filter.and(baseFilter, filter.createdTime().onOrAfter(createdTimeCursor))
      : baseFilter,
  }),
);