Experiencing 429 Too Many Requests error on Angular 7 while attempting to perform multiple file uploads

Question

Experiencing 429 Too Many Requests error on Angular 7 while attempting to perform multiple file uploads

Whenever I attempt to upload a large number of files simultaneously, I encounter an issue.

The API interface only allows for the submission of one file at a time, requiring me to call the service for each individual file. Currently, my code looks like this:

onFilePaymentSelect(event): void {
    if (event.target.files.length > 0) {
      this.paymentFiles = event.target.files[0];
    }
    let i = 0;
    let save = 0;
    const numFiles = event.target.files.length;
    let processed = 0;
    if (event.target.files.length > 0) {
      while (event.target.files[i]) {
      const formData = new FormData();
      formData.append('file', event.target.files[i]);
      this.payrollsService.sendFilesPaymentName(formData).subscribe(
      (response) => {
        let added = null;
        processed++;
        if (response.status_message === 'File saved') {
          added = true;
          save++;
        } else {
          added = false;
        }
        this.payList.push({ filename, message, added });
      });
    i++;
  }
}

Essentially, I am using a loop to sequentially send each file to the API, but when handling a large volume of files, I receive a "429 too many request" error. Is there any way to optimize this process?

javascript angular typescript angular7 http-status-code-429

Answer 1

Answer №1

If you work with observables, it becomes easier to understand the task at hand compared to using imperative programming.

Typically, a browser can handle 6 requests in parallel and queue the rest. However, we don't want the browser to manage the queue for us. This is especially relevant in a node environment where such management may not be available.

What we aim for is efficient file uploads by queuing them and processing 5 requests in parallel continuously, while keeping 1 free for other app requests.

To demonstrate this concept, let's create some mock data:

function randomInteger(min, max) {
  return Math.floor(Math.random() * (max - min + 1)) + min;
}

const mockPayrollsService = {
  sendFilesPaymentName: (file: File) => {
    return of(file).pipe(
      // simulating network latency between 500ms to 1.5s
      delay(randomInteger(500, 1500))
    );
  }
};

// an array containing 50 mocked files
const files: File[] = Array.from({ length: 50 })
  .fill(null)
  .map(() => new File([], ""));

The above code should be self-explanatory. We are generating mocks to simulate the behavior without direct access to your application.

Now, onto the main part:

const NUMBER_OF_PARALLEL_CALLS = 5;

const onFilePaymentSelect = (files: File[]) => {
  const uploadQueue$ = from(files).pipe(
    map(file => mockPayrollsService.sendFilesPaymentName(file)),
    mergeAll(NUMBER_OF_PARALLEL_CALLS)
  );

  uploadQueue$
    .pipe(
      scan(nbUploadedFiles => nbUploadedFiles + 1, 0),
      tap(nbUploadedFiles =>
        console.log(`${nbUploadedFiles}/${files.length} file(s) uploaded`)
      ),
      tap({ complete: () => console.log("All files have been uploaded") })
    )
    .subscribe();
};

onFilePaymentSelect(files);

We use from to process files individually as observables
Through map, we prepare a request for each file, but since we don't subscribe yet, the request remains pending
With mergeMap, we control the concurrency level to run a maximum of 5 calls simultaneously
Using scan for visual tracking (counting successful uploads)

Check out the live demo here: https://stackblitz.com/edit/rxjs-zuwy33?file=index.ts

Observe the console to see how we handle the uploads sequentially rather than all at once.

Answer 2

If you work with observables, it becomes easier to understand the task at hand compared to using imperative programming.

Typically, a browser can handle 6 requests in parallel and queue the rest. However, we don't want the browser to manage the queue for us. This is especially relevant in a node environment where such management may not be available.

What we aim for is efficient file uploads by queuing them and processing 5 requests in parallel continuously, while keeping 1 free for other app requests.

To demonstrate this concept, let's create some mock data:

function randomInteger(min, max) {
  return Math.floor(Math.random() * (max - min + 1)) + min;
}

const mockPayrollsService = {
  sendFilesPaymentName: (file: File) => {
    return of(file).pipe(
      // simulating network latency between 500ms to 1.5s
      delay(randomInteger(500, 1500))
    );
  }
};

// an array containing 50 mocked files
const files: File[] = Array.from({ length: 50 })
  .fill(null)
  .map(() => new File([], ""));

The above code should be self-explanatory. We are generating mocks to simulate the behavior without direct access to your application.

Now, onto the main part:

const NUMBER_OF_PARALLEL_CALLS = 5;

const onFilePaymentSelect = (files: File[]) => {
  const uploadQueue$ = from(files).pipe(
    map(file => mockPayrollsService.sendFilesPaymentName(file)),
    mergeAll(NUMBER_OF_PARALLEL_CALLS)
  );

  uploadQueue$
    .pipe(
      scan(nbUploadedFiles => nbUploadedFiles + 1, 0),
      tap(nbUploadedFiles =>
        console.log(`${nbUploadedFiles}/${files.length} file(s) uploaded`)
      ),
      tap({ complete: () => console.log("All files have been uploaded") })
    )
    .subscribe();
};

onFilePaymentSelect(files);

We use from to process files individually as observables
Through map, we prepare a request for each file, but since we don't subscribe yet, the request remains pending
With mergeMap, we control the concurrency level to run a maximum of 5 calls simultaneously
Using scan for visual tracking (counting successful uploads)

Check out the live demo here: https://stackblitz.com/edit/rxjs-zuwy33?file=index.ts

Observe the console to see how we handle the uploads sequentially rather than all at once.

Experiencing 429 Too Many Requests error on Angular 7 while attempting to perform multiple file uploads

Answer №1

Similar questions

Disabling the range selection in the mat-date-range-picker when the date is disabled

Retrieve Content from a Different Web Page Using Ajax

Tips for displaying JSON data by formatting it into separate div elements for the result object

Error message stating: "The 'MktoForms2' property is not recognized within the scope of 'Window & typeof globalThis'."

Attempting to compile TypeScript by referencing ng2-bootstrap using Gulp within Visual Studio

Issues with JQuery list selectors

Tips for ensuring that Angular2 waits for a promise to resolve before rendering a component

What are the reasons behind memory leaks and decreased rendering speed when I exit and then reopen a React component (specifically Material-Table)?

JQuery Submission with Multiple Forms

Is there a way to simulate AWS Service Comprehend using Sinon and aws-sdk-mock?

Encountering issues with loading Angular formControl

After an ajax call in ASP .NET MVC3 using Razor, jQuery may not function properly

Error message stating 'Module not found' is displaying in the browser console

Having issues with npm python-shell integration within electron framework

Guide: Initiating an action in NuxtJs

Angular automatically substitutes a string with the location of an image

An alternative to passing AngularJS expressions to PHP or Ajax

Why is it beneficial to define a function as an immediately invoked function expression that returns another function?

Ways to eliminate a specific array from another array using JavaScript

Guide on setting the number of records to display on each page using ngx pagination in Angular 4