I am currently working on retrieving data from a third party API that necessitates manual management of paging by keeping track of the number of records retrieved versus the total number of records available.
In attempting to handle this, I experimented with utilizing the expand and reduce operators in RxJs. Although, I have encountered an issue of infinite looping which I haven't faced before when dealing with APIs that offer a 'nextPage' link.
It seems that the root cause stems from the nextPosition variable not being appropriately updated when the output of expand is executed repeatedly within the expansion loop. At this point, I am unsure whether this method can resolve the problem at hand.
My primary question revolves around the feasibility of incorporating expand and reduce alongside a recursive function, and if so, what modifications should be made to rectify the following code snippet?
private async retrievePagedRecords<T>(companyURL: string, query: string, startPosition: number, totalCount: number, transformFn: (qbData: any) => T[]): Promise<T[]> {
const headers = this.getHttpConfig();
let pageQuery = `${query} STARTPOSITION ${startPosition} MAXRESULTS ${this.pagingSize}`;
const nextPosition = startPosition + this.pagingSize;
const records = await lastValueFrom(this.http.get(`${companyURL}/query?query=${pageQuery}`, headers)
.pipe(
map(x => x.data as any),
map(x => {
//Trivial transformation to property names etc.
return transformFn(x);
}),
expand(x => nextPosition > totalCount ? [] : this.retrievePagedRecords<T>(companyURL, query, nextPosition, totalCount, transformFn)),
reduce((acc: T[], x: T[]) => acc.concat(x ?? []), []),
catchError(error => {
return of([]);
})
));
return records;
}