Seeking advice on optimizing a typescript module used by a VSCode extension. This module accepts a directory and parses the content within the files, which can be time-consuming for directories with a large number of files.
To avoid copying the entire class files, a mock pseudocode containing relevant parts will be used.
class Parser {
constructor(_dir: string) {
this.dir = _dir;
}
parse() {
let tree: any = getFileTree(this.dir);
try {
let parsedObjects: MyDTO[] = await this.iterate(tree.children);
} catch (err) {
console.error(err);
}
}
async iterate(children: any[]): Promise<MyDTO[]> {
let objs: MyDTO[] = [];
for (let i = 0; i < children.length; i++) {
let child: any = children[i];
if (child.type === Constants.FILE) {
let dto: FileDTO = await this.heavyFileProcessingMethod(file); // this takes time
objs.push(dto);
} else {
let dtos: MyDTO[] = await this.iterateChildItems(child.children);
let dto: FolderDTO = new FolderDTO();
dto.files = dtos.filter(item => item instanceof FileDTO);
dto.folders = dtos.filter(item => item instanceof FolderDTO);
objs.push(FolderDTO);
}
}
return objs;
}
async heavyFileProcessingMethod(file: string): Promise<FileDTO> {
let content: string = readFile(file);
return new FileDTO(await this.parseFileContent(content));
}
async parseFileContent(content): Promise<any[]> {
let ast: any = await convertToAST(content);
let blocks = parseToBlocks(ast);
return await this.processBlocks(blocks);
}
async processBlocks(blocks: any[]): Promise<any[]> {
for (let i = 0; i < blocks.length; i++) {
let block: Block = blocks[i];
if (block.condition === true) {
await processBlock(block);
}
}
return blocks;
}
}
For optimization purposes, looking for a multithreading/Java-esque solution in TypeScript/NodeJS. In Java context, this.heavyFileProcessingMethod would be an instance of Callable object pushed into a List<Callable> executed parallelly by an ExecutorService, returning List<Future<Object>>.
Plan is to process all files in parallel while waiting for all files to finish processing before returning from the method.
Considering using worker threads in NodeJS for this purpose. How can it be implemented in TypeScript? Will it work in this scenario?
If refactoring the Parser class is necessary to accommodate changes, that's not an issue.
EDIT: Implementing Promise.all
async iterate(children: any[]): Promise<MyDTO>[] {
let promises: Promies<MyDTO>[] = [];
for(let i = 0; i <children.length; i++) {
let child: any = children[i];
if (child.type === Constants.FILE) {
let promise: Promise<FileDTO> = this.heavyFileProcessingMethod(file); // this takes time
promises.push(promise);
} else {
let dtos: Promise<MyDTO>[] = this.iterateChildItems(child.children);
let promise: Promise<FolderDTO> = this.getFolderPromise(dtos);
promises.push(promise);
}
}
return promises;
}
async getFolderPromise(promises: Promise<MyDTO>[]): Promise<FolderDTO> {
return Promise.all(promises).then(dtos => {
let dto: FolderDTO = new FolderDTO();
dto.files = dtos.filter(item => item instanceof FileDTO);
dto.folders = dtos.filter(item => item instanceof FolderDTO);
return dto;
})
}