Recently, the company's projects involved the issue of uploading large files. The problem that arises when uploading large files using ordinary forms is that they cannot continue to exist at breakpoints. Once the upload is interrupted in the middle, they have to start over. This is obviously not what we want, so after some query, I learned the method of dividing and uploading large files. And simple to usephpMake a server-side processing program to implement a functional demo for future review. I am also a fledgling front-end novelist. I will record the implementation summary of various functions. If there are any errors in the code, please correct them.
1. Simple file upload
Normal form upload
Form uploading is a feature we often use, and it is very simple to use. We only need to declare the form content type asenctype="multipart/form-data", indicating the binary data of the file uploaded by the form.
Click the upload button to send the form to the server and useAccept the corresponding form data and save it$_GET/$_POSTIn super global variables, we only need to usemove_uploaded_fileMethod: storing the received file data, realizing the file upload function.
$myfile = $_FILES['myfile'];
//Upload path $path = "upload/" . $myfile['name'];
if(move_uploaded_file($myfile['tmp_name'], $path)){
echo "Uploaded successfully";
} else{
echo "Upload failed";
};ajax mock form upload file
When we have a need to submit a form asynchronously or make certain modifications to the uploaded file (for example: crop size), ordinary form uploading cannot meet our needs because we cannot modify the file value of the form, and at this time, ajax needs to appear. Here we use jQuery to use ajax more convenient and fast.
We need to make the following modifications:
HTML
We do not need to configure form, we only need to configure the corresponding ID to obtain the DOM element object.
JQuery
Notice,jQueryThe ajax method will configure some request information by default, so we need to reconfigure the placementjQueryThe default behavior causes problems with the data format or request header information.
HerecontentTypeandprocessDatais a must.
$('#submitForm').on('click', function(e){
// Block default form submission ();
// Create a form // Enctype="multipart/form-data" is configured by default var formData = new FormData();
('myfile',$('#myFile')[0].files[0])
// Submit the form $.ajax({
type: "POST",
url: '',
data: formData,
// Prevent jquery from assigning default properties, use FormData to configure enctype="multipart/form-data" by default contentType: false,
// Prevent jquery from automatically serializing data processData: false,
success: function(data){
('Request is normal',data);
}
})
})2. Large file segmentation and upload
Simple upload pain points
Simple upload. When submitting files to the server using a form, if the network is not good or the file is interrupted in the middle, the file upload will fail. Just imagine if the file to be uploaded is very large, and when you upload it to 99%, it suddenly interrupts and you have to upload it again, how crashing it would be. At that time, you may have a computer crash.
Implementation ideas
The method to upload large files is to upload the binary files of the uploaded files to the server one by one through division. After the upload is completed, the server then splices the files.
In order to identify which file it is, we must have aFile identifier, used to identify which file the received file data belongs to, and to avoid duplicate uploads, implement the function of instant transmission, etc.
Don’t forget that because it is an asynchronous operation and the data segment sizes of the operation are different, it will lead to the inability to confirm the splicing during integration, so we need an index to identify the location of the data segment.
Through preliminary sorting, we need the following parameters
File unique identifier
Split data segment
Sequential index value of segmented data segments
After thinking, we can establish two processing programs to process the accepting chunk data segment and the merged chunk data segment respectively.
file_getchunk.phpFunction: We will split the chunk data, organize and save it, and we will implement it in file form here.
file_integration.phpFunction: Receive integration notification, splice data segments, and generate files.
Configuration
Since the default PHP cooperation limits the size of POST and uploads, we need to modify it for testing.default configuration in.
post_max_size = 50M upload_max_filesize = 50M
HTML
<script src="/jquery-1.11."></script> <form > <input type="file" name="myfile" /> <input type="submit" value="Upload" /> </form>
JQuery
Get file object, file identifier, split file, and send cut blob data segments through ajax.
$('#submitForm').on('click', function(e){
// Block default form submission ();
var myfile = $('#myFile')[0].files[0];
// Define file identifier var fileId = getFileIdentifier(myfile);
// Data slicing var chunks = fileSlice(myfile);
// Send split data segments sendChunk(fileId, chunks);
})Generate file unique identifiergetFileIdentifier()
Here you can use md5 to generate a file unique md5 (the same file md5 is the same) as an identifier. Here I have only briefly processed the file identification.
function getFileIdentifier(file){
// Get the file identifier return + ;
}Segmentation methodfileSlice()
First, use the blob file inherited by slice to generate a blob string.
function fileSlice(file, chunkSize = 1024*1024*0.2){
// 1. Initialize the data var totalSize = ;
var start = 0;
var end = start + chunkSize;
var chunks = [];
// 2. Use the slice method provided by bolb to slice while(start < totalSize){
var chunk = (start, end);
(chunk);
start = end;
end += chunkSize;
}
// 3. Return to the slice group chunk[] return chunks;
}Send chunk methodsendChunk()
Use ajax to send the divided chunk in turn, and provide the corresponding data, and requestfile_getchunk.phpPerform processing. Here the task list is used to ensure that all file separators have been uploaded.
function sendChunk(id, chunks){
// Submit one by one // Used to ensure that ajax is sent for completion var task = [];
(function(chunk, index){
var formData = new FormData();
('fileId', id);
('myFileChunk', chunk);
('chunkIndex', index);
$.ajax({
type: "POST",
url: 'file_getchunk.php',
data: formData,
contentType: false,
processData: false,
success: function(done){
// Remove completed tasks ();
(done,'Completed');
if ( === 0) {
// Send it and integrate the file ('Notice Integration');
makeFileIntegration(id, );
}
}
})
('file Working');
})
}Notification integration methodmakeFileIntegration()
Received the integration notice and requestfile_integration.phpConsolidate files.
function makeFileIntegration(id, size){
// Notification transmission has been completed $.post(
"file_integration.php",
{
id: id,
size: size
},
function(data){
(data);
}
);
}PHP- file_getchunk.php
When PHP listens to the request, obtain the corresponding data, generate the folder, and followchunkIndexStore data segments.
if(!is_dir('upload')){
mkdir('upload', 0777);
}
$chunk = $_FILES['myFileChunk'];
// File unique identifier $fileId = $_POST['fileId'];
// Temporary folder name $length = strlen($fileId) - (strlen($fileId) - strpos($fileId, '.'));
$filedir = substr($fileId, 0, $length);
$chunkIndex = $_POST['chunkIndex'];
$filepath = 'upload/' . $filedir;
$filename = $filepath . '/' . $chunkIndex;
if(!is_dir($filepath)){
mkdir($filepath, 0777);
}
move_uploaded_file($chunk['tmp_name'], $filename);
echo $chunkIndex;PHP-file_integration.php
Listen to the integration request, splice all files under the folder in sequence, and generate the final restored files.
$fileId = $_POST['id'];
// Temporary folder name $length = strlen($fileId) - (strlen($fileId) - strpos($fileId, '.'));
$filedir = substr($fileId, 0, $length);
$size = $_POST['size'];
$file = './upload/' . $fileId;
// Create the final file if(!file_exists($file)){
// The final file does not exist, create the file $myfile = fopen($file, 'w+');
fclose($myfile);
}
// Open the final file by adding it $myfile = fopen($file, 'a');
for ($i = 0; $i < $size; $i++) {
// Single file path $filePart = 'upload/' . $filedir . '/' . $i;
if(file_exists($filePart)){
$chunk = file_get_contents($filePart);
// Write to chunk fwrite($myfile, $chunk);
} else{
echo "Part$i file is missing, please upload it again";
break;
}
}
fclose($myfile);
echo "Integration Completed";
3. Go further
The large file segmentation and upload function has been basically implemented, but we can still have many optimizations.
1. The breakpoint continues.
The files we need can be divided and uploaded normally, and the server can also receive slices normally, completing the merge of data segment slices. At this time, we can further realize the breakpoint continuation.
The breakpoint continues, and the implementation method is very simple. We only need to obtain the data segment slicing information completed by uploading to determine which data segment we should continue to transmit data.
To obtain information about the data segment slicing, we can use the front-end to save or the server to obtain it. Here we use the server interface detection to return the missing data position to achieve breakpoint continuation.
Ideas sorting
Before uploading, we need to request the server to query the location of the interruption and use the location information to filter the uploaded data segment slices.
Then the logic we want to add is:
offsetInterrupt location information
Query the interrupt location interface:file_get_breakpoint.php
accomplish
getFileBreakpoint()Get file breakpoint function
Here we must ensure the execution order of ajax in order to correctly obtain the offset offset, and there are many implementation ideas. Only use herejqueryThe provided changes the ajax request to synchronize and process it.
Note: When synchronizing the request, the return value of the success function cannot be directly returned. It must be saved in a variable and return outside the ajax request to take effect.
// Get file breakpoint function getFileBreakpoint(id, size){
var offset = '';
$.ajax({
type:"post",
url:"file_get_breakpoint.php",
data: {
id: id,
size: size
},
async: false,
success:function(res){
offset = parseInt(res);
}
})
return offset;
}existsendChunk()Get before sending dataoffset
// Before uploading, request file_integration.php interface to obtain the location where the data segment starts transmission var offset = getFileBreakpoint(id, );
When traversing chunks send data segments, add filtering logic
(function(chunk, index){
// ============================================ // Start transfer from offset if (index < offset) {
return;
}
// ============================================ var formData = new FormData();
('fileId', id);
('myFileChunk', chunk);
('chunkIndex', index);
$.ajax({
type: "POST",
url: 'file_getchunk.php',
data: formData,
contentType: false,
processData: false,
success: function(done){
();
(done,'Completed');
if ( === 0) {
('Notice Integration');
makeFileIntegration(id, );
}
}
})
(index+' is Working');
})Get the interrupt location interfacefile_get_breakpoint.php
The logic used here to obtain the interrupt location is very simple (not optimal). You only need to detect whether the folder exists and then check whether the data segment is missing in turn. Returns the missing segment when missingindex, already exists to return chunks lengthsize, return when it does not exist0
// 1. Detect whether the data file exists (file identification, total number of data segments) $fileId = $_POST['id'];
$size = $_POST['size'];
// Temporary folder name $length = strlen($fileId) - (strlen($fileId) - strpos($fileId, '.'));
$filedir = substr($fileId, 0, $length);
// 2. Detect the location of missing data segments in order // Check whether the folder exists if (is_dir("upload/$filedir")) {
$offset = $size;
// Detection data segment missing subscript for ($i = 0; $i < $size; $i++) {
$filepath = "upload/$filedir/$i";
if(!file_exists($filepath)){
// Missing i part $offset = $i;
break;
}
}
// Output offset echo $offset;
}
else {
// Whether the merged files exist if(file_exists("upload/$fileId")){
echo $size;
} else{
// The file has not been uploaded yet echo 0;
}
}2. File transfer in seconds
According to my understanding, the concept of file transmission in seconds is that after uploading the file request, the server side detects whether the same file exists in the database. If the same file exists, you can tell the user that the upload is completed.
After obtaining the offset, add a judgment to achieve
var offset = getFileBreakpoint(id, );
// Add judgment if( === offset) {
('The file has been uploaded');
return;
}Of course, this is just a very simple process. We can also use MD5 as the file identifier to use this identifier on the server side to see if the same file exists.
3.MD5 detects file integrity.
The file is encrypted through md5 and transmitted to the server side. After the server side merges, the file is encrypted again. Comparing whether the two strings of md5 strings are the same, you can know whether the file transfer is complete.
3. After the upload is completed, store the data segment folder and delete it.
The last step we have is to remove the temporary file. After the integration is completed, we only need tofile_integration.phpIn the interface, after the integration is completed, remove the folder and all files below.
function deldir($path){
//If it is a directory, continue if(is_dir($path)){
//Scan all folders and files in a folder and return the array $p = scandir($path);
foreach($p as $val){
//Exclude the . and .. in the directory if($val !="." && $val !=".."){
//If it is a directory, recursively subdirectory and continue operation if(is_dir($path.$val)){
//Operation in subdirectories deletes folders and files deldir($path.$val.'/');
//Delete empty folder after directory clear @rmdir($path.$val.'/');
}else{
//If it is a file, it is deleted directly unlink($path.$val);
}
}
}
// Delete the folder rmdir($path);
}
}
//Delete temporary folder deldir("upload/$filedir/");4. Summary
Following the above steps, you can follow the knowledge of simple uploading, large file segmentation and uploading, breakpoint continuation, etc. At least the next time you encounter uploading files, you will have some confidence in your heart. Since I am a novice in front-end, the code I wrote is relatively simple, but the functions are implemented, and there are many areas that can be optimized. If the code is incorrect, I hope to correct it.
The above is the detailed content of the entire process of implementing the large file upload function (slice upload). For more information about implementing the large file upload function of PHP+JS, please pay attention to my other related articles!