Developer: JavaScript, you can process these ten G files for me, divide them into pieces, calculate a hash value for each shard, and the server gets the hash value and you can know that the shard has been uploaded (breakpoint continuous transmission), and the entire file is also calculated, maybe the entire file has been uploaded (repeated files do not need to be uploaded multiple times).
JavaScript: You can do this job for me, right? I will just let you get stuck in the browser.
const CHUNK_SIZE = 5 * 1024 * 1024; // Sizeasync function getFile(file) {
const result = [];
const chunkLength = ( / CHUNK_SIZE);
for (let i = 0; i < chunkLength; i++) {
const chunk = await getChunk(file, CHUNK_SIZE, i);
(chunk);
}
}
function getChunk(file, size, index) {
return new Promise((resolve, reject) => {
const start = index * size;
const end = start + size;
const chunkFile = (start, end);
const fr = new FileReader();
= function(e) {
const arrBuffer = ;
const hash = (arrBuffer);
resolve({
start,
end,
chunkFile,
index,
hash
})
}
(chunkFile);
})
}
JavaScript: Are you happy? Don’t move anymore. Can you still show off if you don’t listen to me? Developer: No, sir, are you really here?
Why does JavaScript want to give up?
In the browser's event loop, we know that different threads will handle different tasks, such as http threads, io threads, etc.
If we want to perform complex calculations in the browser, if all operations are in the main thread, the main thread will block, resulting in untimely response of the page and causing lag.
Is there any way to separate the main thread from the compute thread?
The answer is of course webworker
The webworker allows us to open a separate thread to handle some complex computing tasks. After the calculation is completed, we notify the main thread in the form of a callback. The main thread only needs to process the logic after obtaining the calculation result.
Then let's learn how to use it
1. Create a worker instance
const worker = new Worker("./");
// If you need to specify the worker's js, you can use ESM, you can add type parametersconst worker = new Worker("./", { type: "module" });
2. Tell the worker to start working
This worker is the code in the workerjs when I was just created.
("Get Started Work");
3. Listen to the worker's message
= (e) => {
// The internal worker has been executed or has been executed to a certain node};
4. Close the worker
();
5. How to communicate with the main thread within the worker
= (e) => {
// Received a message from an external worker // Complex logic ("Computation Completed");
};
Let the worker perform complex calculations instead of the main thread
const CHUNK_SIZE = 5 * 1024 * 1024; // Sizeconst worker = new Worker('./', {
type: 'module'
});
= function(e) {
const file = [0];
([file, CHUNK_SIZE]);
}
//
= async (e) => {
const [file, CHUNK_SIZE] = ;
const result = [];
const chunkLength = ( / CHUNK_SIZE);
for (let i = 0; i < chunkLength; i++) {
const chunk = await getChunk(file, CHUNK_SIZE, i);
(chunk);
}
// Processing is completed (result);
}
function getChunk(file, size, index) {
return new Promise((resolve, reject) => {
const start = index * size;
const end = start + size;
const chunkFile = (start, end);
const fr = new FileReader();
= function(e) {
const arrBuffer = ;
const hash = (arrBuffer);
resolve({
start,
end,
chunkFile,
index,
hash
})
}
(chunkFile);
})
}
As for threads, there are two or three when you open one. . .
// Turn on four workers directlyconst MAX_WORKER_NUM = 4;
const workers = new Array(MAX_WORKER_NUM).fill(0).map(() => new Worker('./', { type: 'module' }));
const wholeFileWorker = new Worker('./', { type: 'module' });
let finishedCount = 0;
= function(e) {
const file = [0];
// Calculate how many fragments there are in total const chunkLength = ( / CHUNK_SIZE);
// How many shards should each worker complete // If there are 99 shards, the first worker will handle 1-25, the second one is 26-50, the third one is 51-75, and the fourth one is 76-99 // We are programmers, so every index needs -1 const workerSize = (chunkLength / MAX_WORKER_NUM);
for(let i = 0; i < MAX_WORKER_NUM; i++) {
const worker = workers[i];
// Help the worker calculate the starting and ending positions of the sharding task const startIndex = i * workerSize;
const endIndex = (start + workerSize, chunkLength);
([file, CHUNK_SIZE, startIndex, endIndex]);
= (e) => {
finishedCount++;
();
// After calculating part of the hash, you can start uploading. Each return result has an index, which can tell the backend which shard is passed. The information is sufficient
}
}
// First calculate the hash of the shard first. If the hash calculated by the shard, you can start uploading directly. ([file]);
= (e) => {
// Finally process the hash of the entire file // The overall effect is that after the user selects the file, he can quickly start uploading the progress bar. // If the part has been uploaded before and the shards that have started uploading have been uploaded before, you can quickly skip these shards and upload the remaining shards directly // If the entire file was uploaded before, the progress bar will jump directly to 100% from very few places }
}
//
import "./"
= async (e) => {
const [file, CHUNK_SIZE, startIndex, endIndex] = ;
const result = [];
for (let i = startIndex; i < endIndex; i++) {
const chunk = await getChunk(file, CHUNK_SIZE, i);
(chunk);
}
// Processing is completed (result);
}
function getChunk(file, size, index) {
return new Promise((resolve, reject) => {
const start = index * size;
const end = start + size;
const chunkFile = (start, end);
const fr = new FileReader();
= function(e) {
const arrBuffer = ;
const hash = (arrBuffer);
resolve({
start,
end,
chunkFile,
index,
hash
})
}
(chunkFile);
})
}
//
import "./"
= (e) => {
const [file] = ;
const hash = (file);
(hash);
}
This is the article about JavaScript using multithreading to implement uploading a large file. For more related content on uploading large files in JavaScript, please search for my previous articles or continue browsing the related articles below. I hope everyone will support me in the future!