What is the best way to calculate checksum and convert it to a 64-bit value using Javascript for handling extremely large files to avoid RAM overflow?

Question

What is the best way to calculate checksum and convert it to a 64-bit value using Javascript for handling extremely large files to avoid RAM overflow?

Question:

What is the best method for generating a unique and consistent checksum across all browsers? Additionally, how can a SHA256/MD5 checksum string be converted to 64-bit?
How can files be read without requiring excessive amounts of RAM when generating a checksum? For example, how would one handle a 1 GB file without compromising RAM usage?

For more information on reading files without loading them into memory, you can visit this helpful thread on Stack Overflow.

If you're looking for a promising project to assist with this process, consider checking out this particular project.

I am aiming to generate the checksum progressively in chunks of X MBs to avoid overwhelming RAM usage. Below is the code I have been working with:

let SIZE_CHECKSUM = 10 * Math.pow(1024, 2); // 10 MB; Can also be adjusted to 1 MB
async function GetChecksum (file: File):
Promise<string>
{
  let hashAlgorithm: CryptoJS.lib.IHasher<Object> = CryptoJS.algo.SHA256.create();
  let totalChunks: number = Math.ceil(file.size / SIZE_CHECKSUM);
  for (let chunkCount = 0, start = 0, end = 0; chunkCount < totalChunks; ++chunkCount)
  {
    end = Math.min(start + SIZE_CHECKSUM, file.size);
    let resultChunk: string = await (new Response(file.slice(start, end)).text());
    hashAlgorithm.update(resultChunk);
    start = chunkCount * SIZE_CHECKSUM;
  }
  let long: bigInt.BigInteger = bigInt.fromArray(hashAlgorithm.finalize().words, 16, false);
  if(long.compareTo(bigInt.zero) < 0)
    long = long.add(bigInt.one.shiftLeft(64));
  return long.toString();
}

This approach has shown varying results in different web browsers.

javascript algorithm typescript checksum sha256

Answer 1

Answer №1

There appears to be a logical error in the code at the line below:

start = chunkCount * SIZE_CHECKSUM;  // <--- issue

The variable start is initialized as 0 and then reset to 0 again in the first iteration, which is incorrect.
To obtain a 32-byte SHA256 checksum using the library referred to in the question "emn178/js-sha256," you can follow these steps.

Although the mentioned library does not come with a Typescript interface, you can easily define one as shown below:

// Sha256.d.ts  (ensure JS file is named "Sha256.js")
declare class Sha256 {
  update (data: ArrayBuffer): Sha256;
  hex (): string;
}

declare var sha256: any;
declare interface sha256 {
  create (): Sha256;
}

Utilize it in this manner:

import "./external/Sha256"

async function GetChecksum (file: File):
Promise<string>
{
  let algorithm = sha256.create(); 
  for(let chunkCount = 0, totalChunks = Math.ceil(file.size / SIZE_CHECKSUM); 
      chunkCount < totalChunks;
      ++chunkCount)
  {
    let start = chunkCount * SIZE_CHECKSUM, end = Math.min(start + SIZE_CHECKSUM, file.size); 
    algorithm.update(await (new Response(file.slice(start, end)).arrayBuffer()));
  }
  return algorithm.hex();
}

This code consistently generates identical checksums across all my browsers regardless of chunk size.

Answer 2

There appears to be a logical error in the code at the line below:

start = chunkCount * SIZE_CHECKSUM;  // <--- issue

The variable start is initialized as 0 and then reset to 0 again in the first iteration, which is incorrect.
To obtain a 32-byte SHA256 checksum using the library referred to in the question "emn178/js-sha256," you can follow these steps.

Although the mentioned library does not come with a Typescript interface, you can easily define one as shown below:

// Sha256.d.ts  (ensure JS file is named "Sha256.js")
declare class Sha256 {
  update (data: ArrayBuffer): Sha256;
  hex (): string;
}

declare var sha256: any;
declare interface sha256 {
  create (): Sha256;
}

Utilize it in this manner:

import "./external/Sha256"

async function GetChecksum (file: File):
Promise<string>
{
  let algorithm = sha256.create(); 
  for(let chunkCount = 0, totalChunks = Math.ceil(file.size / SIZE_CHECKSUM); 
      chunkCount < totalChunks;
      ++chunkCount)
  {
    let start = chunkCount * SIZE_CHECKSUM, end = Math.min(start + SIZE_CHECKSUM, file.size); 
    algorithm.update(await (new Response(file.slice(start, end)).arrayBuffer()));
  }
  return algorithm.hex();
}

This code consistently generates identical checksums across all my browsers regardless of chunk size.

What is the best way to calculate checksum and convert it to a 64-bit value using Javascript for handling extremely large files to avoid RAM overflow?

Answer №1

Similar questions

Switch between two AppBars simultaneously while scrolling in Material UI

What is the process for accessing and storing an uploaded image in a Next.js application?

disallow rowOnClick function in the datatable

how to forward visitors from one URL to another in a Next.js application

Which Angular Lifecycle event should I utilize to trigger a specific action when either two buttons are pressed or when one of the buttons undergoes a change?

Retrieve an object using a variable

Utilizing the Redux Connect HOC's wrapped component type in React.RefObject without the need for re-importing

Using Angular $resource to store an object with arrays

Please click twice in order to log in to Angular 16

Factory not properly updating AngularJS controller

Can you provide some straightforward instances of single-axis zooming using d3?

Tips for resolving a jspdf naming collision situation

Implementing long polling for private messaging in PHP's chat functionality

What is the reason behind why create-react-app generates the App.js file as a functional component?

Creating a personalized message for a failed "Get" request using the HTTPS native module in combination with

A handy feature of HTML5 is the dataset attribute, which

Is there a way to make the hoverzoom effect only apply to a specific object instead of all objects?

Using Javascript arrays to assign values to a Json object in CanvasXpress

The JavaScript jump function quickly moves back to the top of the webpage

I find it impossible to avoid using the withRouter and withAlert functionalities in Reactjs