Java大文件分片上傳超詳細教程(minio版)
更新時間：2024年08月02日 09:41:28 作者：兒時可乖了
Minio是一個開源的分布式對象存儲系統(tǒng),它允許用戶在存儲服務上存儲和檢索數(shù)據(jù),下面這篇文章主要給大家介紹了關于Java大文件分片上傳(minio版)的相關資料,文中通過代碼介紹的非常詳細,需要的朋友可以參考下
第一步：pom引入minio
<!--minio大文件上傳-->
<dependency>
<groupId>io.minio</groupId>
<artifactId>minio</artifactId>
<version>8.0.3</version>
</dependency>
第二步：配置minio
# minio 配置（大文件分片上傳使用）
minio:
endpoint: ${MINIO_ENDPOINT:http://minio服務IP:端口號}
accesskey: ${MINIO_ASSESSKEY:密鑰}
secretkey: ${MINIO_SECRETKEY:密匙}
bucket: ${MINIO_BUCKET:桶名稱}
第三步：自定義minio客戶端與工具類
1. 自定義客戶端
package com.gcddd.visual.uav.largeFileUpload.util;
import com.google.common.collect.Multimap;
import io.minio.CreateMultipartUploadResponse;
import io.minio.ListPartsResponse;
import io.minio.MinioClient;
import io.minio.ObjectWriteResponse;
import io.minio.errors.*;
import io.minio.messages.Part;
import java.io.IOException;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;
/**
* 自定義minio
* @date 2023-09-15
*/
public class CustomMinioClient extends MinioClient {
public CustomMinioClient(MinioClient client) {
super(client);
}
public String initMultiPartUpload(String bucket, String region, String object, Multimap<String, String> headers, Multimap<String, String> extraQueryParams) throws IOException, InvalidKeyException, NoSuchAlgorithmException, InsufficientDataException, ServerException, InternalException, XmlParserException, InvalidResponseException, ErrorResponseException {
CreateMultipartUploadResponse response = this.createMultipartUpload(bucket, region, object, headers, extraQueryParams);
return response.result().uploadId();
}
public ObjectWriteResponse mergeMultipartUpload(String bucketName, String region, String objectName, String uploadId, Part[] parts, Multimap<String, String> extraHeaders, Multimap<String, String> extraQueryParams) throws IOException, InvalidKeyException, NoSuchAlgorithmException, InsufficientDataException, ServerException, InternalException, XmlParserException, InvalidResponseException, ErrorResponseException {
return this.completeMultipartUpload(bucketName, region, objectName, uploadId, parts, extraHeaders, extraQueryParams);
}
public ListPartsResponse listMultipart(String bucketName, String region, String objectName, Integer maxParts, Integer partNumberMarker, String uploadId, Multimap<String, String> extraHeaders, Multimap<String, String> extraQueryParams) throws NoSuchAlgorithmException, InsufficientDataException, IOException, InvalidKeyException, ServerException, XmlParserException, ErrorResponseException, InternalException, InvalidResponseException {
return this.listParts(bucketName, region, objectName, maxParts, partNumberMarker, uploadId, extraHeaders, extraQueryParams);
}
}
2.封裝minio配置
package com.gcddd.visual.uav.largeFileUpload.configurer;
import lombok.Getter;
import lombok.Setter;
import org.springframework.boot.context.properties.ConfigurationProperties;
/**
* @Description: MinioProperties
* minio配置類
*/
@ConfigurationProperties(prefix = "minio")
@Getter
@Setter
public class MinioProperties {
private String endpoint;
private String accesskey;
private String secretkey;
private String bucket;
}
3.上傳工具類
package com.gcddd.visual.uav.largeFileUpload.configurer;
import cn.hutool.core.util.StrUtil;
import com.gcddd.visual.uav.largeFileUpload.util.CustomMinioClient;
import com.google.common.collect.HashMultimap;
import io.minio.*;
import io.minio.http.Method;
import io.minio.messages.Part;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Configuration;
import org.springframework.stereotype.Component;
import javax.annotation.PostConstruct;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.TimeUnit;
/**
* @Description: MinIoUtils
* 工具類
*/
@Component
@Configuration
@EnableConfigurationProperties({MinioProperties.class})
public class MinIoUtils {
private final MinioProperties minioProperties;
private CustomMinioClient customMinioClient;
public MinIoUtils(MinioProperties minioProperties) {
this.minioProperties = minioProperties;
}
@PostConstruct
public void init() {
MinioClient minioClient = MinioClient.builder()
.endpoint(minioProperties.getEndpoint())
.credentials(minioProperties.getAccesskey(), minioProperties.getSecretkey())
.build();
customMinioClient = new CustomMinioClient(minioClient);
}
/**
* 單文件簽名上傳
*
* @param objectName 文件全路徑名稱
* @return /
*/
public String getUploadObjectUrl(String objectName) {
try {
return customMinioClient.getPresignedObjectUrl(
GetPresignedObjectUrlArgs.builder()
.method(Method.PUT)
.bucket(minioProperties.getBucket())
.object(objectName)
.expiry(1, TimeUnit.DAYS)
//.extraHeaders(headers)
.build()
);
} catch (Exception e) {
e.printStackTrace();
return null;
}
}
/**
* 初始化分片上傳
*
* @param objectName 文件全路徑名稱
* @param partCount 分片數(shù)量
* @param contentType 類型，如果類型使用默認流會導致無法預覽
* @return /
*/
public Map<String, Object> initMultiPartUpload(String objectName, int partCount, String contentType) {
Map<String, Object> result = new HashMap<>();
try {
if (StrUtil.isBlank(contentType)) {
contentType = "application/octet-stream";
}
HashMultimap<String, String> headers = HashMultimap.create();
headers.put("Content-Type", contentType);
String uploadId = customMinioClient.initMultiPartUpload(minioProperties.getBucket(), null, objectName, headers, null);
result.put("uploadId", uploadId);
List<String> partList = new ArrayList<>();
Map<String, String> reqParams = new HashMap<>();
//reqParams.put("response-content-type", "application/json");
reqParams.put("uploadId", uploadId);
for (int i = 1; i <= partCount; i++) {
reqParams.put("partNumber", String.valueOf(i));
String uploadUrl = customMinioClient.getPresignedObjectUrl(
GetPresignedObjectUrlArgs.builder()
.method(Method.PUT)
.bucket(minioProperties.getBucket())
.object(objectName)
.expiry(1, TimeUnit.DAYS)
.extraQueryParams(reqParams)
.build());
partList.add(uploadUrl);
}
result.put("uploadUrls", partList);
} catch (Exception e) {
e.printStackTrace();
return null;
}
return result;
}
/**
* 分片上傳完后合并
*
* @param objectName 文件全路徑名稱
* @param uploadId 返回的uploadId
* @return /
*/
public boolean mergeMultipartUpload(String objectName, String uploadId) {
try {
//TODO::目前僅做了最大1000分片
Part[] parts = new Part[1000];
ListPartsResponse partResult = customMinioClient.listMultipart(minioProperties.getBucket(), null, objectName, 1000, 0, uploadId, null, null);
int partNumber = 1;
for (Part part : partResult.result().partList()) {
parts[partNumber - 1] = new Part(partNumber, part.etag());
partNumber++;
}
customMinioClient.mergeMultipartUpload(minioProperties.getBucket(), null, objectName, uploadId, parts, null, null);
} catch (Exception e) {
e.printStackTrace();
return false;
}
return true;
}
}
4.service
package com.gcddd.visual.uav.largeFileUpload.service;
import java.util.Map;
/**
* @date 2023-09-13
*/
public interface UploadService {
/**
* 分片上傳初始化
*
* @param path 路徑
* @param filename 文件名
* @param partCount 分片數(shù)量
* @param contentType /
* @return /
*/
Map<String, Object> initMultiPartUpload(String path, String filename, Integer partCount, String contentType);
/**
* 完成分片上傳
*
* @param objectName 文件名
* @param uploadId 標識
* @return /
*/
boolean mergeMultipartUpload(String objectName, String uploadId);
}
5.service實現(xiàn)
package com.gcddd.visual.uav.largeFileUpload.service.impl;
import com.gcddd.visual.uav.largeFileUpload.configurer.MinIoUtils;
import com.gcddd.visual.uav.largeFileUpload.service.UploadService;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
import lombok.RequiredArgsConstructor;
import org.springframework.stereotype.Service;
import java.util.Map;
/**
* @date 2023-09-13
*/
@Service
@RequiredArgsConstructor
public class UploadServiceImpl implements UploadService {
private final MinIoUtils minIoUtils;
@Override
public Map<String, Object> initMultiPartUpload(String path, String filename, Integer partCount, String contentType) {
path = path.replaceAll("/+", "/");
if (path.indexOf("/") == 0) {
path = path.substring(1);
}
String filePath = path + "/" + filename;
Map<String, Object> result;
// 單文件，直接上傳
if (partCount == 1) {
String uploadObjectUrl = minIoUtils.getUploadObjectUrl(filePath);
result = ImmutableMap.of("uploadUrls", ImmutableList.of(uploadObjectUrl));
} else {//多文件，分片上傳
result = minIoUtils.initMultiPartUpload(filePath, partCount, contentType);
}
return result;
}
@Override
public boolean mergeMultipartUpload(String objectName, String uploadId) {
return minIoUtils.mergeMultipartUpload(objectName, uploadId);
}
}
6.接口控制器
import cn.hutool.core.lang.Assert;
import cn.hutool.json.JSONObject;
import com.gcddd.visual.uav.largeFileUpload.service.UploadService;
import com.gcddd.visual.uav.util.DateUtils;
import com.google.common.collect.ImmutableMap;
import io.swagger.v3.oas.annotations.security.SecurityRequirement;
import io.swagger.v3.oas.annotations.tags.Tag;
import lombok.RequiredArgsConstructor;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.*;
import java.util.Map;
/**
* 大文件上傳控制器
*/
@RestController
@RequiredArgsConstructor
@RequestMapping("/largeFileUpload" )
@Tag(description = "largeFileUpload" , name = "大文件分片上傳" )
@SecurityRequirement(name = HttpHeaders.AUTHORIZATION)
public class uploadController {
private final UploadService uploadService;
// 1.用戶調用初始化接口，后端調用minio初始化，得到uploadId，生成每個分片的minio上傳url
//初始化 示例： $ curl --location --request POST 'http://127.0.0.1:9998/admin/largeFileUpload/multipart/init' \
// --header 'Content-Type: application/json' \
// --data-raw '{
// "filename": "result.png",
// "partCount": 2
// }'
// 返回示例：
//{
// "uploadId": "eb163e3d-bbf5-48ae-aa56-85cf2a3c0fa3",
// "uploadUrls": [
// "http://ip:端口號/uav/20230913/result.png?uploadId=eb163e3d-bbf5-48ae-aa56-85cf2a3c0fa3&partNumber=1&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=lengleng%2F20230913%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230913T031427Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=9e7bd28d6f4d42d0ed447750cadcd8782ccc2648a66ecb25a1e7639593ad7dd6",
// "http://ip:端口號/uav/20230913/result.png?uploadId=eb163e3d-bbf5-48ae-aa56-85cf2a3c0fa3&partNumber=2&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=lengleng%2F20230913%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230913T031427Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=60d31989b9dc3bdacf6bbc9c13cb5c982f79b1e3a0bbcb948e0ec90fede2f9f3"
// ]
//}
// 2.用戶調用對應分片的上傳地址，多次上傳會覆蓋
// 示例：
// curl --location --request PUT 'http://ip:端口號/uav/20230913/result.png?uploadId=eb163e3d-bbf5-48ae-aa56-85cf2a3c0fa3&partNumber=1&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=lengleng%2F20230913%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230913T031427Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=9e7bd28d6f4d42d0ed447750cadcd8782ccc2648a66ecb25a1e7639593ad7dd6' \
// --header 'Content-Type: application/octet-stream' \
// --data-binary '@/D:/jpgone'
// curl --location --request PUT 'http://ip:端口號/uav/20230913/result.png?uploadId=eb163e3d-bbf5-48ae-aa56-85cf2a3c0fa3&partNumber=2&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=lengleng%2F20230913%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230913T031427Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=60d31989b9dc3bdacf6bbc9c13cb5c982f79b1e3a0bbcb948e0ec90fede2f9f3' \
// --header 'Content-Type: application/octet-stream' \
// --data-binary '@/D:/jpgtwo'
// 3.調用完成接口，后端查詢所有上傳的分片并合并，uploadId為第一步返回的uploadId
// 示例：
// curl --location --request PUT '127.0.0.1:8006/multipart/complete' \
// --header 'Content-Type: application/json' \
// --data-raw '{
// "objectName":"20230913/result.png",
// "uploadId":"eb163e3d-bbf5-48ae-aa56-85cf2a3c0fa3"
// }'
/**
* 分片初始化
*
* @param requestParam 請求參數(shù)
* @return /
*/
@PostMapping("/multipart/init")
public ResponseEntity<Object> initMultiPartUpload(@RequestBody JSONObject requestParam) {
// 路徑（存入minio的路徑）
String pathText = DateUtils.getTodays("yyyyMMdd");//當前日期
String path = requestParam.getStr("path", pathText);
// 文件名
String filename = requestParam.getStr("filename", "test.obj");
// content-type
String contentType = requestParam.getStr("contentType", "application/octet-stream");
// md5-可進行秒傳判斷
String md5 = requestParam.getStr("md5", "");
// 分片數(shù)量
Integer partCount = requestParam.getInt("partCount", 100);
//TODO::業(yè)務判斷+秒傳判斷
Map<String, Object> result = uploadService.initMultiPartUpload(path, filename, partCount, contentType);
return new ResponseEntity<>(result, HttpStatus.OK);
}
/**
* 合并分片文件，完成上傳
*
* @param requestParam 用戶參數(shù)
* @return /
*/
@PutMapping("/multipart/complete")
public ResponseEntity<Object> completeMultiPartUpload(
@RequestBody JSONObject requestParam
) {
// 文件名完整路徑
String objectName = requestParam.getStr("objectName");
//初始化返回的uploadId
String uploadId = requestParam.getStr("uploadId");
Assert.notNull(objectName, "objectName must not be null");
Assert.notNull(uploadId, "uploadId must not be null");
boolean result = uploadService.mergeMultipartUpload(objectName, uploadId);
return new ResponseEntity<>(ImmutableMap.of("success", result), HttpStatus.OK);
}
}
第四步：文件切片
把result.png圖片以500M大小切片，重命名為result.png.后綴
split -b 500m result.png result.png.
第五步：示例截圖
這里測試為把result.png切為2片，返回了uploadId 和兩個minio文件上傳地址
分片上傳的地址上傳文件，請求方式為put
合并文件，完成上傳
檢查minio是否成功
總結
到此這篇關于Java大文件分片上傳(minio版)的文章就介紹到這了,更多相關Java大文件分片上傳內(nèi)容請搜索腳本之家以前的文章或繼續(xù)瀏覽下面的相關文章希望大家以后多多支持腳本之家！
相關文章
跟著思路走，之后從簡單題入手，反復去看，做過之后可能會忘記，之后再做一次，記不住就反復做，反復尋求思路和規(guī)律，慢慢積累就會發(fā)現(xiàn)質的變化2022-03-03
SpringBoot整合RedisTemplate實現(xiàn)緩存信息監(jiān)控的步驟
這篇文章主要介紹了SpringBoot整合RedisTemplate實現(xiàn)緩存信息監(jiān)控,一步一步的實現(xiàn)?Springboot?整合?Redis?來存儲數(shù)據(jù)，讀取數(shù)據(jù)，需要的朋友可以參考下2022-01-01
本文介紹了如何使用Shiro和JWT實現(xiàn)登錄認證,并通過自定義Realm和過濾器來完成身份驗證和授權,通過測試,展示了如何攜帶JWT令牌進行接口訪問,并驗證了其有效期2026-03-03
從Myeclipse 導入到eclipse中無法識別為 web項目 問題的解決步驟
這篇文章主要介紹了從Myeclipse 導入到eclipse中無法識別為 web項目 問題的解決步驟,需要的朋友可以參考下2018-05-05
創(chuàng)建Maven項目的時候，如果你選擇的Packaging為war，那么就會報錯，本文主要介紹了Maven中pom.xml文件報錯的原因解決，文中通過示例代碼介紹的非常詳細，對大家的學習或者工作具有一定的參考學習價值，需要的朋友們下面隨著小編來一起學習學習吧2023-07-07
本文主要介紹了講講java final不可變性，文中通過示例代碼介紹的非常詳細，具有一定的參考價值，感興趣的小伙伴們可以參考一下2021-08-08
文章介紹了服務分級存儲模型,通過Nacos將同一機房內(nèi)的實例劃分為一個集群,微服務互相訪問時應優(yōu)先訪問同集群實例,以提高訪問速度,通過配置application.yml文件,可以實現(xiàn)集群的創(chuàng)建和管理2026-02-02
idea?2024使用Maven創(chuàng)建Java?Web項目詳細圖文教程
這篇文章主要給大家介紹了關于idea?2024使用Maven創(chuàng)建Java?Web項目的相關資料,介紹了如何使用Maven創(chuàng)建一個Spring?MVC項目,并配置Tomcat服務器以運行一個簡單的Helloworld?JSP頁面,需要的朋友可以參考下2024-12-12
springboot @RequiredArgsConstructor的概念與使用方式
這篇文章主要介紹了springboot @RequiredArgsConstructor的概念與使用方式,具有很好的參考價值,希望對大家有所幫助,如有錯誤或未考慮完全的地方,望不吝賜教2024-09-09
下面小編就為大家?guī)硪黄狫ava基礎的幾道練習題(分享)。小編覺得挺不錯的，現(xiàn)在就分享給大家，也給大家做個參考。一起跟隨小編過來看看吧，希望可以幫到你2021-07-07