We're in an awkward situation needing to consume an API endpoint that is not built correctly. It returns pages of (unsortable) records without revealing beforehand how many pages there are in total.
I have Tanstack now working to keep modifying a queryFn passed to useQueries so it fetches pages in parallel until reaching the end, and I have each page keyed separately (key is like ['page', {pageNum}]. This an expensive fetch of data that changes slowly, and any changes will most likely come from the current browser session. These fetches occasionally fail so relying on Tanstack's retries is very useful.
I'm thinking of having another top-level key to cache each individual record by its id, something like ['record', {id}], and when the app sees a record change, it will update this cache entry. To build this part of the cache, would it make sense to have another call to useQuery where its queryFn finds all pages in the 'page' cache and returns the data broken out by individual record?
App components would rely on the 'record' cache, the 'page' cache just being the most efficient way to prime the former (and we may give it a long TTL such that pages will occasionally be refetched, at which point the 'record' cache should also refresh).
