Is there a way to efficiently insert a large amount of data without overwhelming the JS heap memory? I have a model for Email and Category as shown below:
@Entity("email")
export class Email extends BaseEntity {
@PrimaryGeneratedColumn()
public id: number;
@ManyToOne((type) => Category, (cat) => cat.category, {nullable: false, cascade: ['insert']})
public category: Category;
@Column({type: "text", name: "email"})
public email: string;
}
and Category :
@Entity("category")
export class Category extends BaseEntity {
@PrimaryGeneratedColumn()
public id: number;
@Column({type: "text", name: "category"})
public category: string;
@OneToMany((type) => Email, (email) => email.category, {nullable: true})
public emails: Email[];
}
The issue arises when attempting to save an email record with
{email: '<a href="/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="cfada3aeadaea38fada3aea3adaea7e1aca0a2">[email protected]</a>', category: 'default'} public async bulkCreate(emails: Email[]): Promise<any> {
try {
const emailRepo = await getRepository(Email);
const categoryRepo = await getRepository(Category);
await Promise.all(emails.map(async (mail) => {
const cat = await categoryRepo.findOne({where: {category: mail.category}});
if (cat) {
// @ts-ignore
mail.category = cat.id;
} else {
const newCat = await categoryRepo.save(Object.assign(new Category(), mail));
// @ts-ignore
mail.category = newCat.id;
}
await emailRepo.save(mail);
}));
} catch (e) {
console.log(e);
throw new Error(e);
}
}
This approach worked for a limited number of emails, but memory usage spikes significantly when processing around 1,000 records, leading to system crashes.
What steps can be taken to address this issue so that more than 1,000 emails can be added at once?