Troubleshooting vague errors with uploading large files in Golang's net/http protocol

Question

Troubleshooting vague errors with uploading large files in Golang's net/http protocol

I've encountered a challenging error while uploading large files to a server built with Golang's default net/http package. The upload process is defined as follows:

    uploadForm.onsubmit = () => {
        const formData = new FormData(uploadForm);
        const isPublic : boolean = (<HTMLInputElement>document.getElementById('public_switch')).checked;
        formData.append('file', (<HTMLInputElement>document.getElementById('file')).files[0]);
        formData.append('compression', (<HTMLInputElement>document.getElementById('compression')).value);
        formData.append('public', String(isPublic));
        const xhr = new XMLHttpRequest();
        xhr.open("POST", "/upload/");
        xhr.send(formData);
        xhr.onreadystatechange = function() {
            console.log(xhr.responseText);
        }
    }

The server, written in Golang, is initiated like so:

var server = &http.Server{
    Addr:         ":" + Configuration.Port,
    ReadTimeout:  300 * time.Second,
    WriteTimeout: 300 * time.Second,
    ReadHeaderTimeout: 300 * time.Second,
    MaxHeaderBytes: 500000000}

http.HandleFunc("/upload/", uploadFile)

server.ListenAndServe()

Upon receiving the file, it goes through the following processing code:

func uploadFile(w http.ResponseWriter, r *http.Request) {
    //Parsing the upload arguments into the values we shall be working with
    r.ParseMultipartForm(5000000000000000)

    file, _, err := r.FormFile("file")
    //etc

However, when reaching 'r.FormFile("file")' an error message stating "multipart: NextPart: EOF" is displayed.

Could there be specific settings for file limits or timeouts that I am missing in either the Go code or JavaScript? The file being uploaded is approximately 1.7GB and well within the supported HTTP limits.

Any suggestions on how I can improve debugging this issue without diving deep into FormFile or capturing the request itself? The code functions flawlessly with smaller files (only a few MB).

javascript file http typescript go

Answer 1

Answer №1

Let me share the solution to my own question for future reference.

The issue lies with the shortcomings of JavaScript on the web APIs for form and files handling.

Here is the JavaScript solution I devised:

const uploadFile = (form_id: string) => {
    const uploadForm: HTMLFormElement = <HTMLFormElement>document.getElementById(form_id);
    document.getElementById("submit_form").addEventListener("click", function(e) {
        e.preventDefault()

        let reader: FileReader = new FileReader();
        reader.readAsArrayBuffer((<HTMLInputElement>document.getElementById('file')).files[0]);

        reader.onload = function(evt) {

            const formData = new FormData(uploadForm);
            const isPublic: boolean = (<HTMLInputElement>document.getElementById('public_switch')).checked;
            formData.append('file', (<any>evt.target).result);
            formData.append('compression', (<HTMLInputElement>document.getElementById('compression')).value);
            formData.append('public', String(isPublic));
            const xhr = new XMLHttpRequest();
            xhr.open("POST", "/upload/");
            xhr.send(formData);
            xhr.onreadystatechange = function() {
                console.log(xhr.responseText + '  \n status is: ' + xhr.statusText);
            }

        };
    });

}

Although simple, it took a while to find due to cluttered online examples in the JavaScript community.

To handle large files, you should use a FileReader object as follows:

let reader = new FileReader();
reader.readAsArrayBuffer(dom_element_with_file_input.files[0]);

Then, you can utilize it like any other async web API:

reader.onload = function(e) {
    const your_file = e.target.result
}

'your_file' contains the file content for further use. In my case, I appended it to form data like this:

reader.onload = function(e) {
    const your_file = e.target.result
    const formData = new FormData(dom_element_of_type_form);
    formData.append('my_file', your_file);
    formData.append('other_stuff', 'a string with metadata');
}

I'm sharing this example since clear demonstrations of the FileReader API are scarce. I found some guidance at Mozilla documentation, but opted for 'readAsArrayBuffer' for speed enhancements.

Lastly, note that the reader API can be slow and resource-intensive, potentially leading to browser crashes with very large files. If you have insights or solutions to prevent such crashes, please share them with me.

Answer 2

Let me share the solution to my own question for future reference.

The issue lies with the shortcomings of JavaScript on the web APIs for form and files handling.

Here is the JavaScript solution I devised:

const uploadFile = (form_id: string) => {
    const uploadForm: HTMLFormElement = <HTMLFormElement>document.getElementById(form_id);
    document.getElementById("submit_form").addEventListener("click", function(e) {
        e.preventDefault()

        let reader: FileReader = new FileReader();
        reader.readAsArrayBuffer((<HTMLInputElement>document.getElementById('file')).files[0]);

        reader.onload = function(evt) {

            const formData = new FormData(uploadForm);
            const isPublic: boolean = (<HTMLInputElement>document.getElementById('public_switch')).checked;
            formData.append('file', (<any>evt.target).result);
            formData.append('compression', (<HTMLInputElement>document.getElementById('compression')).value);
            formData.append('public', String(isPublic));
            const xhr = new XMLHttpRequest();
            xhr.open("POST", "/upload/");
            xhr.send(formData);
            xhr.onreadystatechange = function() {
                console.log(xhr.responseText + '  \n status is: ' + xhr.statusText);
            }

        };
    });

}

Although simple, it took a while to find due to cluttered online examples in the JavaScript community.

To handle large files, you should use a FileReader object as follows:

let reader = new FileReader();
reader.readAsArrayBuffer(dom_element_with_file_input.files[0]);

Then, you can utilize it like any other async web API:

reader.onload = function(e) {
    const your_file = e.target.result
}

'your_file' contains the file content for further use. In my case, I appended it to form data like this:

reader.onload = function(e) {
    const your_file = e.target.result
    const formData = new FormData(dom_element_of_type_form);
    formData.append('my_file', your_file);
    formData.append('other_stuff', 'a string with metadata');
}

I'm sharing this example since clear demonstrations of the FileReader API are scarce. I found some guidance at Mozilla documentation, but opted for 'readAsArrayBuffer' for speed enhancements.

Lastly, note that the reader API can be slow and resource-intensive, potentially leading to browser crashes with very large files. If you have insights or solutions to prevent such crashes, please share them with me.

Troubleshooting vague errors with uploading large files in Golang's net/http protocol

Answer №1

Similar questions

Step-by-step guide to developing an Angular 2+ component and publishing it on npm

Passing props from parent component to map function in ReactJS results in undefined props

Modify associated dropdown menus

What is the process to designate a specific value to a key in an array?

Filter the object by its unique identifier and locate the entry with the highest score

Transform JSON time data from Coordinated Universal Time (UTC) to Indian Standard

Are reflection problems a concern when using type-graphql mutations?

JavaScript checking the current page's URL is crucial for maintaining accurate and dynamic

Errors are being displayed in the console when attempting to use Vuex getters in a component. This is happening because the Vuex state properties are still null at the time the getters

Is it possible to use v-if in conjunction with a style tag to specify a different source file? Alternatively, is there a more efficient method I

Creating a Dynamic Soundtrack: How to Embed an Audio

What is the reason this JavaScript code functions properly within a <script> tag but not when placed in a separate .js

Maximizing the potential of Next JS 13 Server Components

Alternate the color over time using CSS

Can a self-referential type truly exist?

Is there a way to efficiently return an array of object and nested object keys with their full paths through recursion

Issue with dynamic-rooms causing networked Aframe project to malfunction

The icon for the weather on openweathermap is currently not displaying

Implementing event dispatch on Push notifications received with Workbox

How can I deactivate the main color of the FormLabel when the focus is on the radio button group?