Implement MinIO breakpoint continuous transmission function using Python and FastAPI

Preface

In distributed storage and big data applications, breakpoint retransmission is an important feature, which allows large file uploads to be restored from interrupt points after interruption rather than re-uploading the entire file. This article will introduce how to use Python to encapsulate MinIO's breakpoint continuous transmission method, and use FastAPI to create an API interface, and finally use Axios to call the interface.

Step 1: Install the necessary Python libraries

First, we need to installminioandfastapilibrary.

pip install minio fastapi uvicorn

Step 2: Encapsulate MinIO breakpoint continuous transmission method

We will create a Python function that handles the file's breakpoint continuation.

from minio import Minio
from  import S3Error

def minio_client():
    return Minio(
        "",
        access_key="your-access-key",
        secret_key="your-secret-key",
        secure=True
    )

def upload_file_with_resume(client, bucket_name, object_name, file_path, part_size=10*1024*1024):
    upload_id = client.initiate_multipart_upload(bucket_name, object_name)
    try:
        with open(file_path, "rb") as file_data:
            part_number = 1
            while True:
                data = file_data.read(part_size)
                if not data:
                    break
                client.put_object(bucket_name, f"{object_name}.{part_number}", data, len(data), part_number=part_number, upload_id=upload_id)
                part_number += 1
        client.complete_multipart_upload(bucket_name, object_name, upload_id)
    except S3Error as exc:
        client.abort_multipart_upload(bucket_name, object_name, upload_id)
        raise exc

Code explanation:

• minio_clientThe function creates and returns a MinIO client instance.
• upload_file_with_resumeThe function accepts file path and bucket information and uses the MinIO client to upload it in chunks.
• If an error occurs during uploading, the upload will be terminated and an exception will be thrown.

Step 3: Create API interface using FastAPI

Next, we will use FastAPI to create an API interface that receives files and calls our breakpoint continuation function.

from fastapi import FastAPI, File, UploadFile
from  import JSONResponse

app = FastAPI()

@("/upload/")
async def upload_file(file: UploadFile = File(...)):
    try:
        client = minio_client()
        upload_file_with_resume(client, "my-bucketname", , ._file.name)
        return JSONResponse(status_code=200, content={"message": "File uploaded successfully"})
    except Exception as e:
        return JSONResponse(status_code=500, content={"error": str(e)})

Code explanation:

• FastAPI application creates a/upload/Routing, accepting POST requests.
• file: UploadFile = File(...)The parameter indicates that we expect to receive a file.
• upload_file_with_resumeThe function is called to handle uploads.
• If the upload is successful, a success message will be returned; if the upload fails, an error message will be returned.

Step 4: Call the FastAPI interface using Axios

On the client, we will use Axios to call the interface created by FastAPI.

async function uploadFileToMinIO(file) {
  const formData = new FormData();
  ('file', file);

  try {
    const response = await ('http://localhost:8000/upload/', formData, {
      headers: {
        'Content-Type': 'multipart/form-data'
      }
    });
    ();
  } catch (error) {
    ('Error uploading file:', error);
  }
}

// Call function to upload fileconst fileInput = ('fileInput');
('change', async (event) => {
  const file = [0];
  await uploadFileToMinIO(file);
});

Code explanation:

• We created auploadFileToMinIOfunction, which uses Axios to send POST requests to FastAPI server.
• FormDataObjects are used to build a request body containing file data.
• If the upload is successful, print the response data; if it fails, print the error message.

Things to note

• Security: Ensure that HTTPS is used in production and that the access key and secret key are properly configured.
• Error handling: Enhance error handling logic to gracefully handle various exception situations.
• Performance optimization: Adjust the chunk size according to actual needs to optimize upload performance.

Summarize

This article introduces how to use Python and FastAPI to implement the breakpoint continuous transmission function of MinIO, and uses Axios to call the API interface. By encapsulating MinIO's chunked upload logic, we can effectively handle large file uploads and recover from the interrupt point after the upload process is interrupted. FastAPI provides a concise API interface, while Axios makes it easy to initiate requests from the client. This approach provides strong support for processing large-scale data, making MinIO an ideal choice for data-intensive applications.

The above is the detailed content of using Python and FastAPI to implement the MinIO breakpoint continuous transmission function. For more information about Python and FastAPI MinIO breakpoint continuous transmission, please follow my other related articles!