mirror of
https://github.com/milvus-io/milvus.git
synced 2026-07-21 10:15:43 +00:00
73eec01a0c
issue: #49977 ## Summary AWS SDK C++ 1.11.692 (bumped from 1.11.352 in #47810) changed two defaults that together break PutObject against non-AWS S3-compatible storage: 1. `PutObjectRequest::GetChecksumAlgorithmName()` returns `"crc64nvme"` by default 2. `ClientConfiguration::checksumConfig::requestChecksumCalculation` defaults to `WHEN_SUPPORTED` `ChecksumInterceptor` then attaches a request hash to every PutObject, and the V4 signer switches the request to `Transfer-Encoding: aws-chunked` + `x-amz-content-sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER`. Aliyun OSS, Tencent COS, Huawei OBS, and GCP all reject this streaming form. This makes any `MinioChunkManager::PutObjectBuffer` call fail in an infinite `JobStateRetry` loop — json_stats / bson_inverted shared_key_index builds and any non-DiskANN-style index upload (IVF_*, HNSW, FLAT) are affected. ## Fix Restrict `requestChecksumCalculation` to `WHEN_REQUIRED` on non-AWS providers so the SDK only adds checksums when the operation model demands them (PutObject does not). Same approach as milvus-io/milvus-storage#500. Patched in both client construction paths: - `MinioChunkManager::MinioChunkManager` (storage_type=minio, address-based dispatch) — gated on `storageType != S3` - `ChunkManager.cpp` subclasses (storage_type=remote, cloud_provider-based dispatch) — applies in `GcpChunkManager`, `AliyunChunkManager`, `TencentCloudChunkManager`, `HuaweiCloudChunkManager`. `AwsChunkManager` keeps the default `WHEN_SUPPORTED`. CARDINAL_TIERED / DiskANN indexes already work because they upload via Arrow's `S3FileSystem` provided by milvus-storage, which was fixed in #500. ## Test plan - [x] Added `TEST(MinioChecksumConfig, OverridesAreWhenRequired)` in `test_storage.cpp` — asserts the override flips both directions from `WHEN_SUPPORTED` to `WHEN_REQUIRED` - [ ] Build / CI green - [ ] Manual smoke test on Aliyun OSS endpoint — index build (IVF_SQ8) and json_stats no longer fail with `x-oss-ec=0017-00000804`. CI cannot regression-detect this since MinIO doesn't reject `aws-chunked`; a real Aliyun OSS endpoint is required. ## Workaround (no code change) Setting `AWS_REQUEST_CHECKSUM_CALCULATION=when_required` and `AWS_RESPONSE_CHECKSUM_VALIDATION=when_required` on milvus pods also fixes this — the AWS SDK reads them at `ClientConfiguration` default-construct time, so it works across all bundled SDK copies without rebuilding. Useful for hotfix on running clusters until this PR ships. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: cai.zhang <cai.zhang@zilliz.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>