Saving large amounts of data in bulk to PostgreSQL using TypeORM

Question

Saving large amounts of data in bulk to PostgreSQL using TypeORM

I am looking to perform a bulk insert/update using TypeORM

The Test entity is defined below:

export class Test {
   @PrimaryColumn('integer')
   id: number;
 
   @Column('varchar', { length: 255 })
   testName: string;
 }

I have the following objects that I want to insert in batch:

const test1 = new Test();
script.id = 1;
script.testName = 't1';

const test2 = new Test();
test2.id = 2;
test2.testName = 't2';

const test3 = new Test();
test3.id = 3;
test3.testName = 't3';

const tests = [test1, test2, test3];

However, when I attempt to save them together like this:

const testRepository = dataSource.getRepository(Test);
await testRepository.save(tests);

It doesn't work as expected. Strangely enough, testRepository.save(test1); works fine.

Can you guide me on how to perform a batch insert, especially when dealing with thousands of rows?

typescript typeorm node.js-typeorm typeorm-datamapper

Answer 1

Answer №1

When utilizing TypeORM, the insert method is handy for executing bulk inserts.

To perform a bulk insert, you can follow this basic syntax:

const dataRepository = dataSource.getRepository(Data);

await dataRepository.insert(dataEntries);

Some key points to keep in mind:

The insert method accepts an array of entity objects
If your database requires explicit ids (like postgres), make sure the primary keys/ids are populated for each entity
In databases supporting auto-increment ids (such as sqlite, mysql), omitting ids is acceptable

For instance:

const dataEntries = [
  { id: 1, entryName: 'entry1'},
  { id: 2, entryName: 'entry2'}, 
  { id: 3, entryName: 'entry3'} 
];

await dataRepository.insert(dataEntries);

This code snippet will insert all 3 data entries in one go.

Additional suggestions include:

You have the flexibility to mix existing and new entities within the array
Split huge batches into chunks to avoid hitting query size limitations
Employing transactions can ensure atomicity during inserts

Answer 2

When utilizing TypeORM, the insert method is handy for executing bulk inserts.

To perform a bulk insert, you can follow this basic syntax:

const dataRepository = dataSource.getRepository(Data);

await dataRepository.insert(dataEntries);

Some key points to keep in mind:

The insert method accepts an array of entity objects
If your database requires explicit ids (like postgres), make sure the primary keys/ids are populated for each entity
In databases supporting auto-increment ids (such as sqlite, mysql), omitting ids is acceptable

For instance:

const dataEntries = [
  { id: 1, entryName: 'entry1'},
  { id: 2, entryName: 'entry2'}, 
  { id: 3, entryName: 'entry3'} 
];

await dataRepository.insert(dataEntries);

This code snippet will insert all 3 data entries in one go.

Additional suggestions include:

You have the flexibility to mix existing and new entities within the array
Split huge batches into chunks to avoid hitting query size limitations
Employing transactions can ensure atomicity during inserts

Saving large amounts of data in bulk to PostgreSQL using TypeORM

Answer №1

Similar questions

Verify the length of an array within an object using JavaScript

What is the best way to change `props.children` into a JSX element?

Obtain the appropriate selection in the dropdown based on the model in Angular

API endpoint generating a Vue component as a rendered output

Is it recommended to keep Angular properties private and only access them through methods?

Encountering issues with Proxy functionality in the latest versions of Angular 13 and Spring Boot

What is the resolution if I need to utilize a property that is untyped?

add headers using a straightforward syntax

What is preventing TypeScript from automatically inferring the type of an argument in a generic function that utilizes `keyof`?

Angular CodeMirror Line-Break function not displaying line numbers

Specialized type for extra restriction on enum matching

Troubleshooting Node.js import module errors

Steps to make ng-packagr detect a Typescript type definition

Issue encountered with express-jwt and express-graphql: TypeScript error TS2339 - The 'user' property is not found on the 'Request' type

"Encountering issues with DefinePlugin when using the combination of Ionic, Angular, Webpack,

Ways to address observables in Angular in a manner similar to deferred objects

Using the as operator in TypeScript for type casting a string

A guide on iterating through a JSON object fetched using Http in Angular 2/Typescript

Leveraging Global Variables and Functions with Webpack and TypeScript

Why does it seem like Typescript Promise with JQuery is executing twice?