Async iterator over a paginated API

An API returns its results a page at a time. You're given fetchPage(cursor), which returns a promise of a page { items, nextCursor }. items is an array; nextCursor is the cursor for the following page, or null when this is the last page. The first page is fetchPage(null).

Write paginate(fetchPage). It returns an async iterable of every item of every page, in order, so you can loop over all the items with for await...of. Pages with no items are passed over, and it stops after the page whose nextCursor is null.

It fetches each page only when it's needed: nothing until the first item is asked for, and the next page only once every item of the current one has been handed out. When the caller stops early, it fetches no more pages.

If fetchPage rejects, the iteration throws that error, after the items it has already handed out.

const pages = {
  first: { items: ["a", "b"], nextCursor: "p2" },
  p2: { items: [], nextCursor: "p3" },
  p3: { items: ["c"], nextCursor: null },
};
const fetchPage = async (cursor) => pages[cursor ?? "first"];
 
for await (const item of paginate(fetchPage)) console.log(item);
// logs "a", "b", "c"
 
const items = paginate(fetchPage)[Symbol.asyncIterator]();
await items.next(); // { value: "a", done: false } (fetches the first page)
await items.next(); // { value: "b", done: false } (no fetch)
await items.next(); // { value: "c", done: false } (fetches p2, then p3)
await items.next(); // { value: undefined, done: true }

What the tests check

1. 1yields every item of every page, in order

   const pages = new Map([
     [null, { items: [1, 2], nextCursor: "b" }],
     ["b", { items: [3], nextCursor: "c" }],
     ["c", { items: [4, 5], nextCursor: null }],
   ]);
   const fetchPage = async (cursor) => pages.get(cursor);
   const seen = [];
   for await (const item of paginate(fetchPage)) seen.push(item);
   expect(seen).toEqual([1, 2, 3, 4, 5]);
   

2. 2starts with fetchPage(null) and passes each nextCursor on

   const pages = new Map([
     [null, { items: [1, 2], nextCursor: "b" }],
     ["b", { items: [3], nextCursor: "c" }],
     ["c", { items: [4, 5], nextCursor: null }],
   ]);
   const fetchPage = spy(async (cursor) => pages.get(cursor));
   const seen = [];
   for await (const item of paginate(fetchPage)) seen.push(item);
   expect(fetchPage.calls.map((call) => call.args)).toEqual([[null], ["b"], ["c"]]);
   

3. 3fetches nothing until the first item is asked for

   const fetchPage = spy(async () => ({ items: [1, 2], nextCursor: null }));
   const items = paginate(fetchPage)[Symbol.asyncIterator]();
   await flush();
   expect(fetchPage).not.toHaveBeenCalled();
   expect(await items.next()).toEqual({ value: 1, done: false });
   expect(fetchPage).toHaveBeenCalledTimes(1);
   

4. 4fetches the next page only when the current one is used up

   const pages = new Map([
     [null, { items: [1, 2], nextCursor: "b" }],
     ["b", { items: [3], nextCursor: null }],
   ]);
   const fetchPage = spy(async (cursor) => pages.get(cursor));
   const items = paginate(fetchPage)[Symbol.asyncIterator]();
   await items.next();
   await items.next();
   await flush();
   expect(fetchPage).toHaveBeenCalledTimes(1);
   expect(await items.next()).toEqual({ value: 3, done: false });
   expect(fetchPage).toHaveBeenCalledTimes(2);
   const { value, done } = await items.next();
   expect({ value, done }).toEqual({ value: undefined, done: true });
   

5. 5fetches no more pages when the caller stops early

   const pages = new Map([
     [null, { items: [1, 2], nextCursor: "b" }],
     ["b", { items: [3], nextCursor: "c" }],
     ["c", { items: [4, 5], nextCursor: null }],
   ]);
   const fetchPage = spy(async (cursor) => pages.get(cursor));
   const seen = [];
   for await (const item of paginate(fetchPage)) {
     seen.push(item);
     if (item === 3) break;
   }
   await flush();
   expect(seen).toEqual([1, 2, 3]);
   expect(fetchPage).toHaveBeenCalledTimes(2);
   

6. 6passes over pages with no items

   const pages = new Map([
     [null, { items: [], nextCursor: "b" }],
     ["b", { items: [], nextCursor: "c" }],
     ["c", { items: ["x"], nextCursor: null }],
   ]);
   const seen = [];
   for await (const item of paginate(async (cursor) => pages.get(cursor))) seen.push(item);
   expect(seen).toEqual(["x"]);
    
   const empty = [];
   for await (const item of paginate(async () => ({ items: [], nextCursor: null }))) empty.push(item);
   expect(empty).toEqual([]);
   

7. 7throws the fetcher's error after the items already handed out

   const error = new Error("network down");
   const fetchPage = async (cursor) => {
     if (cursor === null) return { items: [1, 2], nextCursor: "b" };
     throw error;
   };
   const seen = [];
   let caught;
   try {
     for await (const item of paginate(fetchPage)) seen.push(item);
   } catch (e) {
     caught = e;
   }
   expect(seen).toEqual([1, 2]);
   expect(caught).toBe(error);
   

Learn more

• Async iteration & async generatorsLesson
• async & awaitLesson