SoFunction
Updated on 2025-04-10

Specific implementation method for downloading large file fragments at front-end (read this article is enough)

summary

This article starts from the front-end aspect to realize the function of downloading large files in the browser. The transmission interruption caused by network abnormalities, web page closure, etc. is not considered. Shard download adopts serial method (parallel download requires computing hash for slices, comparing hash, missing retransmission, merging chunks in sequence, etc., which is very troublesome. It has the pursuit of transmission speed, and parallel shard download can be done if bandwidth allows)

Overall architecture process

1. Download using shards: Split large files into multiple small pieces for downloading, which can reduce the risk of memory usage and network transmission interruption. This can avoid performance problems caused by downloading the entire large file at once.

2. Breakpoint continuous transmission: Implement the breakpoint continuous transmission function, that is, after the download is interrupted during the download, you can continue to download from the downloaded part without re-downloading the entire file.

3. Progress bar display: Display the download progress on the page, allowing users to clearly see the progress of file download. If you download all the parameters in one go, you can directly get the parameters from the process (very fine). If it is a shard download, it will also calculate the total size of the downloaded sum, but the downloaded will increase in pieces (not very fine).

4. Cancel download and pause download function: Provides buttons to cancel download and pause download, allowing users to abort or pause the download process as needed.

5. Merge files: After the download is completed, merge all shard files into one complete file.

Specific implementation

1. Local storage of shard download (localForage)

Preface:

The browser's security policy prohibits web pages (JS) from directly accessing and operating file systems on the user's computer.

During the shard download process, each downloaded file chunk needs to be cached or stored on the client, which facilitates the implementation of the breakpoint continuous transmission function, and also facilitates the subsequent merge of these file chunks into complete files. These file blocks can be temporarily stored in memory or stored in the client's local storage (such as IndexedDB, LocalStorage, etc.).

Use encapsulation and directly upload code

import axios from 'axios';
import ElementUI from 'element-ui'
import Vue from 'vue'
import localForage from 'localforage'
import streamSaver from 'streamsaver';
import store from "@/store/index"
/**
  * localforage – is an efficient and powerful offline storage solution.
  * It encapsulates IndexedDB, WebSQL, or localStorage and provides a simplified API like localStorage.
  * By default, the use of IndexDB, WebSQL, and localStorage will be preferred for background storage.
  * That is, if the browser does not support IndexDB, it will try to use WebSQL. If IndexDB is not supported or WebSQL is not supported, it will use WebSQL.
  * localStorage for storage.
  *
  */
/**
  * @description Creates a database instance, creates and returns a new instance of localForage.  Each instance object has an independent database without affecting other instances
  * @param {Object} dataBase database name
  * @return {Object} storeName instance (repository instance)
  */
function createInstance(dataBase){
  return ({
    name: dataBase
  });
}

/**
  * @description Save data
  * @param {Object} name key name
  * @param {Object} value data
  * @param {Object} storeName repository
  */
async function saveData(name, value, storeName){
  await (name, value).then(() => {
    // When the value is stored, other operations can be performed    ("save success");
  }).catch(err => {
    // When an error occurs, the code here runs    (err)
  })
}

/**
  * @description Get saved data
  * @param {Object} name key name
  * @param {Object} storeName repository
  */
async function getData(name, storeName, callback){
  await (name).then((val) => {
    // After obtaining the value, you can perform other operations    callback(val);
  }).catch(err => {
    // When an error occurs, the code here runs    callback(false);
  })
}

/**
  * @description Remove saved data
  * @param {Object} name key name
  * @param {Object} storeName repository
  */
async function removeData(name, storeName){
  await (name).then(() => {
    ('Key is cleared!');
  }).catch(err => {
    // When an error occurs, the code here runs    (err);
  })
}

/**
  * @description Deleting a data warehouse will delete the specified "database" (and all data warehouses).
  * @param {Object} dataBase database name
  */
async function dropDataBase(dataBase){
  await ({
    name: dataBase
  }).then(() => {
     ('Dropped ' + dataBase + ' database');
  }).catch(err => {
    // When an error occurs, the code here runs    (err);
  })
}

/**
  * @description Delete the data warehouse
  * @param {Object} dataBase database name
  * @param {Object} storeName Repository name (instance)
  */
async function dropDataBaseNew(dataBase, storeName){
  await ({
    name: dataBase,
    storeName: storeName
  }).then(() => {
     ('Dropped',storeName)
  }).catch(err => {
    // When an error occurs, the code here runs    (err);
  })
}



2. Local data acquisition (get download list)

/**
  * @description Get the download list
  * @param {String} page
  * @param {String} user
  */
export async function getDownloadList(page, user, callback){
  // const key = user + "_" + page + "_";
  // const key = user + "_";
  const key = "_"; // Because the user will expire, do not save the user name. Just use the page as the key.  // Basic database  const baseDataBase = createInstance(baseDataBaseName);
  await ().then(async function(keys) {
      // Array containing all key names      let fileList = [];
      for(let i = 0; i < ; i++){
        if(keys[i].indexOf(key)>-1){
          // Get data          await getData(keys[i], baseDataBase, async (res) =>{
            (
              {
                'fileName': res, // file name                'dataInstance': keys[i] // The file corresponds to the database instance name              }
            );
          })
        }
      }
      // Get progress      for(let i = 0; i < ; i++){
        const dataBase = createInstance(fileList[i].dataInstance);
        await getData(progressKey, dataBase,  async (progress) => {
          if(progress){
            fileList[i].fileSize = ;
            fileList[i].progress =  ?  : 0;
            fileList[i].status =  ?  : "stopped";
            fileList[i].url = ;
          }
        });
      }
      callback(fileList);
  }).catch(function(err) {
      // When an error occurs, the code here runs      callback(err);
  });
}

3. Operation JS

1. Download progress monitoring

/**
  * File download progress monitoring
  */
const onDownloadProgres = (progress) =>{
  // The loaded in the progress object indicates the number of downloads, total indicates the total number, and the percentage is calculated here  let downProgress = (100 *  / ); 
  ('setProgress', {
      dataInstance: uniSign,
      fileName: ,
      progress: downProgress,
      status: downProgress == 100 ? 'success' : 'downloading',
      cancel: cancel
  })
}

2. File download status control

// Database progress data primary keyconst progressKey = "progress";
/**
  * @description Status Control
  * @param {String} fileName filename
  * @param {String} type Download method continue:Continue again:Re-download cancel:Cancel
  * @param {String} dataBaseName Database name Unique to each file
  *
  */
async function controlFile(fileName, type, dataBaseName){
  if(type == 'continue'){
    
  } else if(type == 'again'){
    // Delete file data    await dropDataBase(dataBaseName);
    // Basic database    const baseDataBase = createInstance(baseDataBaseName);
    // Basic database delete database instance    removeData(dataBaseName, baseDataBase);
  } else if(type == 'cancel'){
    // Delete file data    await dropDataBase(dataBaseName);
    // Basic database    const baseDataBase = createInstance(baseDataBaseName);
    // Basic database delete database instance    await removeData(dataBaseName, baseDataBase);
    ('delProgress', {
        dataInstance: dataBaseName
    })
  } else if(type == 'stop'){
    ('setProgress', {
        dataInstance: dataBaseName,
        status: 'stopped',
    })
  }
}

3. Download the fragments

/**
  * @description Sharding Download
  * @param {String} fileName filename
  * @param {String} url Download address
  * @param {String} dataBaseName Database name Unique to each file
  * @param {Object} progress Download progress type: continue:Continue again:Re-download cancel:Cancel
  *
  */
export async function downloadByBlock(fileName, url, dataBaseName, progress) {
  //Adjust the download status  if(){
    await controlFile(fileName, , dataBaseName);
  }
  
  // Determine whether the maximum number of downloads exceeds  let downloadNum = ;
  if(downloadNum){
    if(! &&  == downloadMaxNum){
      ("The maximum download volume has been exceeded, please wait for other files to be downloaded before trying!");
      return;
    }
  }
  
  // Basic database  const baseDataBase = createInstance(baseDataBaseName);
  // A download task to determine whether the file already exists in the database  let isError = false;
  await getData(dataBaseName, baseDataBase, async (res) =>{
    if(res && !){
      ("The file is already in the download task list, no need to download it again!");
      isError = true;
    }
  });
  if(isError){
    return;
  }
  
  // Database for storing data  const dataBase = createInstance(dataBaseName);
  
  // Get file size  const response = await (url);
  // File size  const fileSize = +['content-length'];
  // Size  const chunkSize = +['buffer-size'];
  // Start node  let start = 0;
  // End node  let end = chunkSize - 1;
  if (fileSize < chunkSize) {
    end = fileSize - 1;
  }
  // All shards  let ranges = [];
  // Calculate how many times it takes to download  const numChunks = (fileSize / chunkSize);
  // Write back file size   = fileSize;
   = url;
  
  // Save database instance  await saveData(dataBaseName, fileName, baseDataBase);
  if(!){
    // Save progress data    await saveData(progressKey, progress, dataBase);
  }
  // Save the download status to localstorage If the page is refreshed, you can re-read the status and determine whether you need to continue downloading.  (dataBaseName,"downloading");
  
  // Group to parameter  for (let i = 0; i < numChunks; i++) {
    // Create a request controller    const controller = new AbortController();
    
    const range = `bytes=${start}-${end}`;
    // If it is a continuous transmission, first determine whether it has been downloaded.    if( == 'continue'){
      // Modify the status first      ('setProgress', {
          dataInstance: dataBaseName,
          status: 'downloading'
      })
      let isContinue = false;
      await getData(range, dataBase, async function(res){ if(res) isContinue = true});
      if(isContinue){
        (range);
        // Reset the start node        start = end + 1;
        // Reset the end node        end = (end + chunkSize, fileSize - 1);
        continue;
      }
    }

    let cancel;
    const config = {
      headers: {
        Range: range
      },
      responseType: 'arraybuffer',
      // Bind the semaphore for canceling request      signal: , 
      // Listen to file download progress      onDownloadProgress: function (pro){
        if( == 'stop' ||  == 'cancel'){
           // Terminate the request          ();
        }
        
        // Loaded size        // The loaded in the progress object indicates the number of downloads, total indicates the total number, and the percentage is calculated here        let downProgress = ( / ); 
        downProgress = ( (i / numChunks) * 100 + downProgress  / numChunks * 100);
         = downProgress;
        // Set to Exception If the download is normal, this record will be deleted. If the merge fails, an exception will be displayed.         = downProgress == 100 ? 'error' : 'stopped';
        ('setProgress', {
            dataInstance: dataBaseName,
            fileName: fileName,
            progress: downProgress,
            status: 'downloading',
            cancel: cancel,
            url: url,
            dataBaseName: dataBaseName
        })
      }
    };
    try {
      // Start the download of shards      const response = await (url, config);
      // Save sharded data      await saveData(range, , dataBase);
        
      (range);
      // Reset the start node      start = end + 1;
      // Reset the end node      end = (end + chunkSize, fileSize - 1);
        
    } catch (error) {
      ("Error of catch when terminated request---", error);
      // Determine whether it is cancelled upload      if ( == "canceled"){
          // Follow-up processing          if( == 'stop'){
            // pause            await controlFile(fileName, , dataBaseName);
            (dataBaseName,"stop");
             // Terminate the request            ("stopped");
            return;
          } else if( == 'cancel'){
            // Cancel            await controlFile(fileName, , dataBaseName);
            (dataBaseName);
            // Terminate the request            ("canceled");
            return;
          }
      };
    }
  }

  ("Start join");
  // Streaming operation  const fileStream = (fileName, {size: fileSize});
  const writer = ();
  // Merge data  // Loop out the data and merge it  for (let i=0; i < ; i++) {
    var range = ranges[i];
    // Get segmented data from the database    await getData(range, dataBase, async function(res){
      if(res){
        // Read stream        const reader = new Blob([res]).stream().getReader();
        while (true) {
          const { done, value } = await ();
          if (done) break;
          // Write to stream          (value)
        }
        // End writing        if(i==-1){
          ();
          // Clear the database          dropDataBase(dataBaseName);
          // Basic database delete database instance          removeData(dataBaseName, baseDataBase);
          // Clear the store          ('delProgress', {
            dataInstance: dataBaseName
          })
        } 
      }
    }); 
  }
}

4. VUE page

<script>
import { downloadByBlock } from "JS above";
export default {
  name: "suspension",
  data() {
    return {
      // Download list      downloadDialog: false,
      fileStatus: {}
    };
  },
  watch: {
    "$": function() {
      this.$nextTick(function() {
        let progressList = this.$;
        (item => {
          // Get the previous download status Restore operation          const status = ();
          if (status == "downloading" &&  != status) {
            (item);
          }
        });
      });
    }
  },
  methods: {
    /**
      * @description Try again
      * @param {Object} row
      */
    retryDownload(row) {
      (row);
    },
    /**
      * @description Start downloading
      * @param {Object} row
      */
    startDownload(row) {
      [] = {
        type: "continue",
        progress: 
      };
      downloadByBlock(
        ,
        ,
        ,
        []
      );
    },
    /**
      * @description Pause download
      * @param {Object} row
      */
    stopDownload(row) {
      this.$set([], "type", "stop");
    },
    /**
      * @description Delete download
      * @param {Object} row
      */
    delDownload(row) {
      if ([] &&  != "stopped") {
        this.$set([], "type", "cancel");
      } else {
        [] = { type: "cancel" };
        downloadByBlock(
          ,
          ,
          ,
          []
        );
      }
    },
    /**
      * @description Shadow download file
      */
    downloadByBlock(fileName, url, dataBaseName, type) {
      [dataBaseName] = { type: type };
      downloadByBlock(
        fileName,
        url,
        dataBaseName,
        [dataBaseName]
      );
      ();
    },
    /**
      * @description Download List
      */
    btnDownload() {
      // Use routing information to determine which page you are currently on      const page = this.$;
       = true;
    },



    /**
      * @description Cancel pop-up window
      */
    downloadBtnCancel() {
       = false;
    }
  }
};
</script>

summary

This is just a basic example. In practical applications, you may need to consider the following issues:

Concurrent download: If multiple users download the same large file at the same time, it may put a lot of pressure on the server. Concurrent downloads can be used to improve performance.

Security: Ensure that only authorized users can download files and that sensitive information is not leaked.

Performance optimization: Use caching, compression and other technologies to improve download speed and user experience.

Server resource management: Downloading large files may occupy the server's network bandwidth and memory resources and requires appropriate management.

In short, downloading and merging large files shards is a complex task that requires comprehensive consideration of performance, security and user experience. In actual projects, you may use more advanced tools and techniques to deal with these issues.

Summarize

This is the article about the specific implementation method for downloading large front-end file fragments. For more related contents for downloading large front-end file fragments, please search for my previous articles or continue browsing the related articles below. I hope everyone will support me in the future!