What strategies are typically suggested for managing typescript monorepos in a development setting?

Question

What strategies are typically suggested for managing typescript monorepos in a development setting?

Over the past week, I've been transforming a large monolithic repository (npm/typescript) into a monorepo (yarn/lerna/typescript). Initially, the transition was smooth as I rearranged files into their respective folders and updated imports.

The real challenge arose when I began exploring different methods of running the monorepo in "development" mode with hot reload/watch functionality. I'm eager to avoid transpiling every single package upon each change since they all currently have dependencies on one another. After using lerna to bootstrap the project and install modules, I noticed that it links folders to node_modules, ensuring automatic updates in all repositories with each change. Despite trying various approaches, I could use some guidance on how to proceed or what industry standards dictate.

Issue:

The monorepo consists of 3 packages/projects containing service classes and 1 package serving as a CLI which utilizes those packages. Previously, executing the CLI command in the old monolithic repo involved:

ts-node --transpiler sucrase/ts-node-plugin --project tsconfig.json --require tsconfig-paths/register bin/cli.ts

This command internally transpiled all necessary ts files and executed within a reasonable time frame of 500ms-1s. Maintaining this acceptable transpile time is ideal. Additionally, utilizing tsconfig paths means having a dedicated tsconfig file per package, ideally retaining them while resorting to relative paths if necessary.

Potential Solutions:

Utilize tsc-watch for each directory, set the main entry in package.json to dist/index.js, and finalize the setup. However, the downside to this approach lies in the compilation/transpile time required by tsc – around 500ms-1s for source files, at least 1.5-2s for .d.ts file generation per package, and an additional 500ms for tspaths replacement (per project). Although testing babel-watch showed faster JS compilation, it lacked type generation.
Repeat the aforementioned command with the same file, seemingly functional yet encountering challenges with resolving tsconfig paths of nested packages. For instance, the CLI package resolves its paths using a relative tsconfig, then finds the local package Transpiler (linked folder to node_modules) with its own set of paths requiring specific definition. While sacrificing ts-paths for rapid dev transpile times may be viable, setting package.json main to src/index.ts complicates building procedures as it needs adjusting during production builds, similar to step 1's process.

I drew inspiration and referenced the AWS JS SDK monorepo to navigate through these challenges.

Project Structure:

packages
    - cli *(depends on transpiler)*
    - transpiler *(depends on common and statements)*
    - statements *(depends on common)*
    - common *(depends on nothing)*
package.json
tsconfig.json

typescript monorepo lerna ts-node

Answer 1

Answer №1

Upon further investigation following my initial comment on your query, I stumbled upon a valuable article Boost your productivity with TypeScript project references, which piqued my interest and led me to give it a shot.

In delving deeper into my monorepo project, known as Slickgrid-Universal, consisting of 17 packages within the monorepo structure, I utilize Lerna-Lite (an adaptation that I also maintain). Prior to each testing phase, I would clear all dist/ directories and compare the execution times of both methods:

The execution of lerna run command took approximately 1 minute and 30 seconds
Using tsc -b with TypeScript References only required 18 seconds

Discovering how drastically faster tsc -b (or --build) with TypeScript References is truly amazed me - clocking in at 5 times quicker, including the creation of declaration files (d.ts). Furthermore, enabling watch mode (incremental by default, hence eliminating the need for --incremental) enhances its efficiency.

tsc --build ./tsconfig.packages.json --watch

I proceeded to test another aspect; determining the duration for type propagation between packages when adding new properties to an existing interface. In my scenario, the common package (the largest containing numerous interfaces) showcased an approximate 10-second time lag before reflecting the changes in the frontend package displayed in the editor. While satisfactory, the waiting period was quite similar to utilizing lerna watch alongside tsc --incremental in individual packages. The real win undoubtedly lies in the significantly reduced build time from the original 1m30sec down to just 18 sec for the entire monorepo, along with Type declarations.

Can we achieve even greater speed?

An alternate method caught my attention, involving type checking alone (similar to ViteJS and ESBuild's approach) without generating declaration files through tsc --noEmit. However, adapting this strategy may not be compatible within a monorepo construct due to the necessity for Types when importing across packages. Therefore, opting for the aforementioned option 2 - tsc -b seems most promising moving forward (I've already implemented this change for Lerna-Lite post drafting this response).

Note: I speculate whether tsc -b aligns effectively with other tools like TurboRepo or Nx. Although untested in my setup, there may be limitations or peculiarities concerning this integration, possibly requiring separate tsc commands per package. Nevertheless, the potential expedited process through caching, rebuilding solely the modified package, retains the appeal, yet tsc -b stands out for its accelerated performance owing to incremental updates.

EDIT

In optimizing my code and initiating this PR, certain key observations surfaced:

An inherent drawback while employing TypeScript References and Composite lies in allowing interaction solely with one configuration, thereby hindering multiple distinctive builds with varied settings (e.g., hybrid build CJS/ESM), at least from my understanding.
Hence, I retained my original practice of using lerna watch coupled with tsc --incremental (non-composite) within each package due to several reasons:
1. Monitoring SASS files necessitates maintaining a unified lerna watch catering to all file extensions (.ts, .scss).
2. The comparable runtime of executing lerna watch alongside tsc --incremental specifically within the altered package, mirrored that of triggering tsc -b --watch from the root level. Thus, given the absence of development acceleration, the established singular lerna watch appears more efficient rather than managing assorted watches (1x tsc watch notwithstanding numerous SASS watches leading to increased thread counts).
3. Notably, lerna watch consumes lesser resources (memory/CPU usage) than tsc --watch, especially evident on Windows systems. Though a single tsc watch surpasses multifold instances of tsc watches for each package, the combination of monitoring SASS files under a sole lerna watch proves more resource-efficient.
Albeit these observations might be inconsequential in your venture, adhering to tsc -b represents the optimal course whenever feasible.

Consequently, despite these insights, I persist in running tsc -b initially at the root before commencing lerna watch to ensure swift initiation with requisite Types existence at the onset of development. Subsequent stages rely on lerna watch administering tsc --incremental on the revised package. Though not the swiftest route, it certainly outweighs prior experiences. A prospective endeavor includes exploring the amalgamation of lerna watch with tsc --noEmit exclusively for type checking (a future experimentation during ample downtime). Additionally, for my Cypress E2E tests, I execute tsb -b from the root sans watch requirements, leveraging the expeditious tsc -b, thus saving nearly 1.5 minutes on the CI job alone.

Furthermore, I chanced upon another informative piece Using TypeScript Project References with ts-loader and Webpack, offering valuable insights even though I employ ViteJS, underscoring the educational value it holds.

Answer 2

Upon further investigation following my initial comment on your query, I stumbled upon a valuable article Boost your productivity with TypeScript project references, which piqued my interest and led me to give it a shot.

In delving deeper into my monorepo project, known as Slickgrid-Universal, consisting of 17 packages within the monorepo structure, I utilize Lerna-Lite (an adaptation that I also maintain). Prior to each testing phase, I would clear all dist/ directories and compare the execution times of both methods:

The execution of lerna run command took approximately 1 minute and 30 seconds
Using tsc -b with TypeScript References only required 18 seconds

Discovering how drastically faster tsc -b (or --build) with TypeScript References is truly amazed me - clocking in at 5 times quicker, including the creation of declaration files (d.ts). Furthermore, enabling watch mode (incremental by default, hence eliminating the need for --incremental) enhances its efficiency.

tsc --build ./tsconfig.packages.json --watch

I proceeded to test another aspect; determining the duration for type propagation between packages when adding new properties to an existing interface. In my scenario, the common package (the largest containing numerous interfaces) showcased an approximate 10-second time lag before reflecting the changes in the frontend package displayed in the editor. While satisfactory, the waiting period was quite similar to utilizing lerna watch alongside tsc --incremental in individual packages. The real win undoubtedly lies in the significantly reduced build time from the original 1m30sec down to just 18 sec for the entire monorepo, along with Type declarations.

Can we achieve even greater speed?

An alternate method caught my attention, involving type checking alone (similar to ViteJS and ESBuild's approach) without generating declaration files through tsc --noEmit. However, adapting this strategy may not be compatible within a monorepo construct due to the necessity for Types when importing across packages. Therefore, opting for the aforementioned option 2 - tsc -b seems most promising moving forward (I've already implemented this change for Lerna-Lite post drafting this response).

Note: I speculate whether tsc -b aligns effectively with other tools like TurboRepo or Nx. Although untested in my setup, there may be limitations or peculiarities concerning this integration, possibly requiring separate tsc commands per package. Nevertheless, the potential expedited process through caching, rebuilding solely the modified package, retains the appeal, yet tsc -b stands out for its accelerated performance owing to incremental updates.

EDIT

In optimizing my code and initiating this PR, certain key observations surfaced:

An inherent drawback while employing TypeScript References and Composite lies in allowing interaction solely with one configuration, thereby hindering multiple distinctive builds with varied settings (e.g., hybrid build CJS/ESM), at least from my understanding.
Hence, I retained my original practice of using lerna watch coupled with tsc --incremental (non-composite) within each package due to several reasons:
1. Monitoring SASS files necessitates maintaining a unified lerna watch catering to all file extensions (.ts, .scss).
2. The comparable runtime of executing lerna watch alongside tsc --incremental specifically within the altered package, mirrored that of triggering tsc -b --watch from the root level. Thus, given the absence of development acceleration, the established singular lerna watch appears more efficient rather than managing assorted watches (1x tsc watch notwithstanding numerous SASS watches leading to increased thread counts).
3. Notably, lerna watch consumes lesser resources (memory/CPU usage) than tsc --watch, especially evident on Windows systems. Though a single tsc watch surpasses multifold instances of tsc watches for each package, the combination of monitoring SASS files under a sole lerna watch proves more resource-efficient.
Albeit these observations might be inconsequential in your venture, adhering to tsc -b represents the optimal course whenever feasible.

Consequently, despite these insights, I persist in running tsc -b initially at the root before commencing lerna watch to ensure swift initiation with requisite Types existence at the onset of development. Subsequent stages rely on lerna watch administering tsc --incremental on the revised package. Though not the swiftest route, it certainly outweighs prior experiences. A prospective endeavor includes exploring the amalgamation of lerna watch with tsc --noEmit exclusively for type checking (a future experimentation during ample downtime). Additionally, for my Cypress E2E tests, I execute tsb -b from the root sans watch requirements, leveraging the expeditious tsc -b, thus saving nearly 1.5 minutes on the CI job alone.

Furthermore, I chanced upon another informative piece Using TypeScript Project References with ts-loader and Webpack, offering valuable insights even though I employ ViteJS, underscoring the educational value it holds.

What strategies are typically suggested for managing typescript monorepos in a development setting?

Answer №1

Similar questions

When attempting to send a fetch request in the most recent rendition of NextJS, it ends up with an error of 'failed fetch'

Explain to me the process of passing functions in TypeScript

Angular 2: Dynamically Adjusting View Components Based on URL Path

Making decisions about data types during compilation?

"Enhance your forms with RadixUI's beautiful designs for

Passing a service into a promise in Angular 2 using TypeScript

How can I effectively integrate TypeScript with Jest to mock ES6 class static properties?

How to retrieve a variable from an object within an array using AngularJS code

Issue with PrimeReact dropdown component not recognizing an array in TypeScript

Having trouble with linting on Typescript 3.7 within the Angular 9 tslint environment

Replace a portion of text with a RxJS countdown timer

Interactive loadChild components

Angular HTTP Interceptor delays processing of http requests until a new refresh token is obtained

What is the procedure for incorporating a cookie jar into axios using typescript?

What is the process for extracting the paths of component files from an Angular ngModule file?

Type Error TS2322: Cannot assign type 'Object[]' to type '[Object]'

Unable to pass property to child component

What is the rationale behind placing the CSS outside of the React function components, near the imports?

TypeScript enables the use of optional arguments through method overloading

Do not overlook any new error messages related to abstract classes