This article comes from the author'sFlutter actual combat》, readers can also click to view the online electronic version.
Basic knowledge
The Http protocol defines the response header field for chunked transmission, but whether it supports it depends on the Server implementation. We can specify the "range" field of the request header to verify whether the server supports chunked transmission. For example, we can use the curl command to verify:
bogon:~ duwen$ curl -H "Range: bytes=0-10" /HBuilder.9.0.2.macosx_64.dmg -v # Request header> GET /HBuilder.9.0.2.macosx_64.dmg HTTP/1.1 > Host: > User-Agent: curl/7.54.0 > Accept: */* > Range: bytes=0-10 #Response header< HTTP/1.1 206 Partial Content < Content-Type: application/octet-stream < Content-Length: 11 < Connection: keep-alive < Date: Thu, 21 Feb 2019 06:25:15 GMT < Content-Range: bytes 0-10/233295878
The purpose of adding "Range: bytes=0-10" to the request header is to tell the server that we only want to get the content of the file 0-10 (including 10, total 11 bytes) for this request. If the server supports chunked transmission, the response status code is 206, indicating "partial content", and at the same time, the response header changes will contain the "Content-Range" field. If it is not supported, it will not be included. Let's take a look at the content of "Content-Range" above:
Content-Range: bytes 0-10/233295878
0-10 represents the block returned this time, 233295878 represents the total length of the file, and the units are all byte, which means that the file is about 233M or more.
accomplish
To sum up, we can design a simple multi-threaded file chunking downloader, and the implementation idea is:
- First check whether chunked transmission is supported. If not, download it directly; if supported, download it, download the remaining content in chunks.
- When downloading each chunk, save it to each temporary file, and merge the temporary file after all chunks are downloaded.
- Delete temporary files.
Here is the overall process:
// Check whether the server supports chunked transmission through the first chunking requestResponse response = await downloadChunk(url, 0, firstChunkSize, 0);
if ( == 206) { //If supported //Analyze the total length of the file and then calculate the remaining length total = (
().split("/").last);
int reserved = total -
(());
//Total number of blocks of the file (including the first block) int chunk = (reserved / firstChunkSize).ceil() + 1;
if (chunk > 1) {
int chunkSize = firstChunkSize;
if (chunk > maxChunk + 1) {
chunk = maxChunk + 1;
chunkSize = (reserved / maxChunk).ceil();
}
var futures = <Future>[];
for (int i = 0; i < maxChunk; ++i) {
int start = firstChunkSize + i * chunkSize;
//Download remaining files in chunks (downloadChunk(url, start, start + chunkSize, i + 1));
}
//Waiting for all blocks to be downloaded await (futures);
}
//Merge file files await mergeTempFiles(chunk);
}
Below we use the download API of the famous Http library dio under Flutter to implement downloadChunk:
//start represents the starting position of the current block, end represents the end position//no means what block is currentlyFuture<Response> downloadChunk(url, start, end, no) async {
(0); //Progress records the length of each piece of received data --end;
return (
url,
savePath + "temp$no", //Temporary files are named according to the block sequence number to facilitate final merging onReceiveProgress: createCallback(no), // Create a progress callback, and then implement it options: Options(
headers: {"range": "bytes=$start-$end"}, //Specify the requested content interval ),
);
}
Next, implement mergeTempFiles:
Future mergeTempFiles(chunk) async {
File f = File(savePath + "temp0");
IOSink ioSink= (mode: );
//Merge temporary files for (int i = 1; i < chunk; ++i) {
File _f = File(savePath + "temp$i");
await (_f.openRead());
await _f.delete(); //Delete temporary files }
await ();
await (savePath); //Rename the merged file to the real name}
Let's take a look at the complete implementation:
/// Downloading by spiting as file in chunks
Future downloadWithChunks(
url,
savePath, {
ProgressCallback onReceiveProgress,
}) async {
const firstChunkSize = 102;
const maxChunk = 3;
int total = 0;
var dio = Dio();
var progress = <int>[];
createCallback(no) {
return (int received, _) {
progress[no] = received;
if (onReceiveProgress != null && total != 0) {
onReceiveProgress(((a, b) => a + b), total);
}
};
}
Future<Response> downloadChunk(url, start, end, no) async {
(0);
--end;
return (
url,
savePath + "temp$no",
onReceiveProgress: createCallback(no),
options: Options(
headers: {"range": "bytes=$start-$end"},
),
);
}
Future mergeTempFiles(chunk) async {
File f = File(savePath + "temp0");
IOSink ioSink= (mode: );
for (int i = 1; i < chunk; ++i) {
File _f = File(savePath + "temp$i");
await (_f.openRead());
await _f.delete();
}
await ();
await (savePath);
}
Response response = await downloadChunk(url, 0, firstChunkSize, 0);
if ( == 206) {
total = (
().split("/").last);
int reserved = total -
(());
int chunk = (reserved / firstChunkSize).ceil() + 1;
if (chunk > 1) {
int chunkSize = firstChunkSize;
if (chunk > maxChunk + 1) {
chunk = maxChunk + 1;
chunkSize = (reserved / maxChunk).ceil();
}
var futures = <Future>[];
for (int i = 0; i < maxChunk; ++i) {
int start = firstChunkSize + i * chunkSize;
(downloadChunk(url, start, start + chunkSize, i + 1));
}
await (futures);
}
await mergeTempFiles(chunk);
}
}
Now you can download in chunks:
main() async {
var url = "/HBuilder.9.0.2.macosx_64.dmg";
var savePath = "./example/HBuilder.9.0.2.macosx_64.dmg";
await downloadWithChunks(url, savePath, onReceiveProgress: (received, total) {
if (total != -1) {
print("${(received / total * 100).floor()}%");
}
});
}
think
Can chunked downloads really improve download speed?
In fact, the main bottleneck of download speed depends on the network speed and the server's exit speed. If it is the same data source, the meaning of block download is not very important, because the server is the same, and the exit speed is determined mainly depends on the network speed. The above example is formally homologous to block download. Readers can compare the download speeds of blocks and non-blocks by themselves. If there are multiple download sources and the export bandwidth of each download source is limited, the download in blocks may be faster. The reason why it is said to be "possible" is that this is not certain. For example, there are three sources, and the exit bandwidth of the three sources is 1G/s, and the peak value of the network connected to our equipment is only 800M/s, then the bottleneck lies in our network. Even if the bandwidth of our device is greater than any source, the download speed is still not necessarily faster than single source single-line download. Just imagine, suppose there are two sources A and B, and the speed A source is 3 times that of source B. If you use block download and each of the two sources is half downloaded, readers can calculate the required download time, and then calculate the time required to download only from source A to see which one is faster.
The final speed of block download is affected by many factors such as the network bandwidth of the device, the source exit speed, the size of each block, and the number of blocks. It is difficult to ensure the optimal speed in the actual process. In actual development, readers can test and compare first before deciding whether to use it.
Is there any practical use for chunked downloads?
There is another relatively used scenario for block downloading, which is to continue transmission of breakpoints. The file can be divided into several blocks, and then a download status file is maintained to record the status of each block. In this way, even after the network is interrupted, the state before the interrupt can be restored. Readers can try the specific implementation by themselves. There are still some details that need special attention, such as the size of the block is appropriate? How to deal with half of the block downloaded? Do you want to maintain a task queue?
The above is all the content of this article. I hope it will be helpful to everyone's study and I hope everyone will support me more.