Implementation of a recursive stream in fp-ts for paginated API with lazy evaluation

Question

Implementation of a recursive stream in fp-ts for paginated API with lazy evaluation

My objective involves making requests to an API for transactions and saving them to a database. The API response is paginated, so I need to read each page and save the transactions in batches.

After one request/response cycle, I aim to process the data and store it in the database before moving on to fetch the next batch of transactions.

In an effort to stay testable, I am using the actual HTTP fetch call as

(url:string, init:RequestInit) => TE.TaskEither<Error, Response>

within fp-ts. How can I achieve this in a more idiomatic way?

I have experimented with RxJS and fp-ts, but encountered complexity despite eventually getting it to work.

I've tried implementing both recursive functions and generators, yet I'm struggling to achieve lazy evaluation.

Are there any existing examples in fp-ts that demonstrate a lazily evaluated stream where each element depends on the previous one?

typescript recursion stream lazy-evaluation fp-ts

Answer 1

Answer №1

If you're looking to read a list of files where the next filename is stored in the current file, consider using StateReaderTaskEither.

import * as TE from 'fp-ts/TaskEither'
import * as RA from 'fp-ts/ReadonlyArray'
import { pipe } from 'fp-ts/function'

const files = {
  file1: 'file2',
  file2: 'file3',
  file3: 'file5',
  file4: 'file6',
  file5: 'file4',
}
const getFile = (filename: string, encoding: string) =>
  Promise.resolve(files[filename])

const fileSRTE: SRTE.StateReaderTaskEither<
  string,
  string,
  unknown,
  string
> = (filename: string) => (encoding: string) =>
  pipe(
    TE.tryCatch(
      () => getFile(filename, encoding),
      (err) => err
    ),
    TE.map((file) => [`file: ${file}`, file])
  )

const fileStream: TE.TaskEither<
  unknown,
  readonly string[]
> = pipe(
  RA.replicate(6, fileSRTE),
  SRTE.sequenceArray,
  SRTE.evaluate('file1')
)('utf-8')

fileStream().then(console.log)
//{ _tag: 'Right',right: [ 'file: file2','file: file3','file: file5','file: file4','file: file6','file: undefined' ] }

To delve deeper into State and Reader concepts, check out the fp-ts documentation.

Answer 2

If you're looking to read a list of files where the next filename is stored in the current file, consider using StateReaderTaskEither.

import * as TE from 'fp-ts/TaskEither'
import * as RA from 'fp-ts/ReadonlyArray'
import { pipe } from 'fp-ts/function'

const files = {
  file1: 'file2',
  file2: 'file3',
  file3: 'file5',
  file4: 'file6',
  file5: 'file4',
}
const getFile = (filename: string, encoding: string) =>
  Promise.resolve(files[filename])

const fileSRTE: SRTE.StateReaderTaskEither<
  string,
  string,
  unknown,
  string
> = (filename: string) => (encoding: string) =>
  pipe(
    TE.tryCatch(
      () => getFile(filename, encoding),
      (err) => err
    ),
    TE.map((file) => [`file: ${file}`, file])
  )

const fileStream: TE.TaskEither<
  unknown,
  readonly string[]
> = pipe(
  RA.replicate(6, fileSRTE),
  SRTE.sequenceArray,
  SRTE.evaluate('file1')
)('utf-8')

fileStream().then(console.log)
//{ _tag: 'Right',right: [ 'file: file2','file: file3','file: file5','file: file4','file: file6','file: undefined' ] }

To delve deeper into State and Reader concepts, check out the fp-ts documentation.

Implementation of a recursive stream in fp-ts for paginated API with lazy evaluation

Answer №1

Similar questions

leveraging parcel for importing typescript dependencies

If a component does not have a specified return type annotation, it will default to an 'any' return type

Jest is unable to handle ESM local imports during resolution

What is the best way to connect a text box to a nested object in the state of a React component?

Dragula drag and drop in a single direction with Angular 2 copy functionality

The aesthetic of the material tree design is not being reflected as expected

Leveraging TypeScript to sort and extract specific elements from two arrays

Developing microfrontends using Angular involves loading and navigating a compiled package from npm

A custom Typescript type for immutable values within an object

Exploring the world of chained JavaScript Promises for automatic pagination of an API

Tips on streamlining two similar TypeScript interfaces with distinct key names

Issues with Angular Reactive Forms Validation behaving strangely

When defining properties/data in Vue mixins, the properties/data of the mixin are not accessible

Saving a picture to your Ionic device

The sequence of initializing test hooks in inconsistent playwright tests

the process of extracting data from a request body in Angular 2

Issue TS1259: The module "".../node_modules/@types/bn.js/index"" can only be imported as the default using the 'esModuleInterop' flag

Error: `__WEBPACK_IMPORTED_MODULE_1_signature_pad__` does not function as a constructor

Methods for defining a variable in React with Typescript without encountering the "Index signature is missing in type" issue

TypeScript purity - "The variable exports is not defined"