Saving large amounts of data in bulk to PostgreSQL using TypeORM

Question

Saving large amounts of data in bulk to PostgreSQL using TypeORM

I am looking to perform a bulk insert/update using TypeORM

The Test entity is defined below:

export class Test {
   @PrimaryColumn('integer')
   id: number;
 
   @Column('varchar', { length: 255 })
   testName: string;
 }

I have the following objects that I want to insert in batch:

const test1 = new Test();
script.id = 1;
script.testName = 't1';

const test2 = new Test();
test2.id = 2;
test2.testName = 't2';

const test3 = new Test();
test3.id = 3;
test3.testName = 't3';

const tests = [test1, test2, test3];

However, when I attempt to save them together like this:

const testRepository = dataSource.getRepository(Test);
await testRepository.save(tests);

It doesn't work as expected. Strangely enough, testRepository.save(test1); works fine.

Can you guide me on how to perform a batch insert, especially when dealing with thousands of rows?

typescript typeorm node.js-typeorm typeorm-datamapper

Answer 1

Answer №1

When utilizing TypeORM, the insert method is handy for executing bulk inserts.

To perform a bulk insert, you can follow this basic syntax:

const dataRepository = dataSource.getRepository(Data);

await dataRepository.insert(dataEntries);

Some key points to keep in mind:

The insert method accepts an array of entity objects
If your database requires explicit ids (like postgres), make sure the primary keys/ids are populated for each entity
In databases supporting auto-increment ids (such as sqlite, mysql), omitting ids is acceptable

For instance:

const dataEntries = [
  { id: 1, entryName: 'entry1'},
  { id: 2, entryName: 'entry2'}, 
  { id: 3, entryName: 'entry3'} 
];

await dataRepository.insert(dataEntries);

This code snippet will insert all 3 data entries in one go.

Additional suggestions include:

You have the flexibility to mix existing and new entities within the array
Split huge batches into chunks to avoid hitting query size limitations
Employing transactions can ensure atomicity during inserts

Answer 2

When utilizing TypeORM, the insert method is handy for executing bulk inserts.

To perform a bulk insert, you can follow this basic syntax:

const dataRepository = dataSource.getRepository(Data);

await dataRepository.insert(dataEntries);

Some key points to keep in mind:

The insert method accepts an array of entity objects
If your database requires explicit ids (like postgres), make sure the primary keys/ids are populated for each entity
In databases supporting auto-increment ids (such as sqlite, mysql), omitting ids is acceptable

For instance:

const dataEntries = [
  { id: 1, entryName: 'entry1'},
  { id: 2, entryName: 'entry2'}, 
  { id: 3, entryName: 'entry3'} 
];

await dataRepository.insert(dataEntries);

This code snippet will insert all 3 data entries in one go.

Additional suggestions include:

You have the flexibility to mix existing and new entities within the array
Split huge batches into chunks to avoid hitting query size limitations
Employing transactions can ensure atomicity during inserts

Saving large amounts of data in bulk to PostgreSQL using TypeORM

Answer №1

Similar questions

What is the best way to integrate environment-specific configuration options into an AngularJS and Typescript project?

How can I use Angular to dynamically open a video that corresponds to a clicked image?

Combine the selected values of two dropdowns and display the result in an input field using Angular

Setting a default value for a data type within Typescript

In TypeScript, there is a curious phenomenon where private properties seem to be mimicking the

Employ an asynchronous immediately-invoked function expression within the callback

Determine if a condition is met in Firebase Observable using scan() and return the

ngPrime table column selection and data extraction

How can 'this' be converted from D3 JavaScript to TypeScript?

Utilizing TypeScript to invoke a method via an index signature

Is it true that TypeScript prohibits the presence of circular references under the condition of having generic parameters

Transmit information between components through a form

Invalid sequencing of Nest.js async onModuleInit causing issues

A solution for resolving the RuntimeException in genuitec's TypeScript Editor for Eclipse Oxygen

Is Angular2 detecting changes based on value equivalence or reference equality?

Adding additional properties to Material UI shadows in Typescript is a simple process that can enhance the visual

Guide on updating a value in a Firestore document using Firebase

The chosen index in the Material Stepper feature is experiencing a malfunction

Is there a way to incorporate a loading spinner into a MaterialUI DataTable without having to specify a fixed height for the parent component?

Getting a string output from a Typescript promise