Recently, I've been experimenting with JS's XMLHttpRequest Class for handling file uploads. Initially, I attempted to upload files using the following code:
const file = thisFunctionReturnsAFileObject();
const request = new XMLHttpRequest();
request.open('POST', '/upload-file');
const rawFileData = await file.arrayBuffer();
request.send(rawFileData);
Fortunately, the above code functioned correctly and successfully sent the raw binary data of the file to the server.
However, I encountered a major issue with memory consumption. Due to JavaScript's lack of memory efficiency, the entire file was stored in memory, causing problems. On my machine with 16GB of RAM, I couldn't upload files larger than ~100MB as it led to excessive memory allocation, ultimately crashing the Chrome tab with a SIGILL code.
Considering the above challenge, I decided to explore the use of ReadableStreams as a solution. Despite good browser compatibility (https://caniuse.com/#search=ReadableStream) and the indication from my TypeScript compiler that request.send(...) supported ReadableStreams (which later turned out to be false), I attempted the following code:
const file = thisFunctionReturnsAFileObject();
const request = new XMLHttpRequest();
request.open('POST', '/upload-file');
const fileStream = file.stream();
request.send(fileStream);
Unfortunately, my TypeScript compiler provided unreliable information, resulting in the server receiving "[object ReadableStream]" (quite frustrating).
Although I haven't extensively explored this method, I am open to suggestions and assistance in this matter.
I believe splitting the request into chunks could be a more efficient solution. By sending chunks individually, we can free up memory as soon as each chunk is sent, rather than waiting for the entire request to be received.
Despite thorough research, I have yet to discover a method to implement this approach. In pseudocode, something like the following would be ideal:
const file = thisFunctionReturnsAFileObject();
const request = new XMLHttpRequest();
request.open('POST', '/upload-file');
const fileStream = file.stream();
const fileStreamReader = fileStream.getReader();
const sendNextChunk = async () => {
const chunk = await fileStreamReader.read();
if (!chunk.done) {
request.writeToBody(chunk.value);
} else {
request.end();
break;
}
}
sendNextChunk();
The expected outcome of the above code is to send the request in chunks and finalize the request when all chunks have been transmitted.
Despite my efforts, one particular resource that I found didn't yield the desired results:
Method for streaming data from browser to server via HTTP
Reasons it didn't work included:
- I require a solution that operates within a single request
- The use of RTCDataChannel is not an option, it must be a standard HTTP request (are there alternative methods besides XMLHttpRequest?)
- It needs to be compatible with modern Chrome/Firefox/Edge browsers (no need for IE support)
Edit: I prefer not to utilize multipart-form (FormData Class) and instead aim to transmit actual binary data extracted from the filestream in chunks.