mirror of
https://github.com/milvus-io/milvus.git
synced 2026-07-21 10:15:43 +00:00
73eec01a0c
issue: #49977 ## Summary AWS SDK C++ 1.11.692 (bumped from 1.11.352 in #47810) changed two defaults that together break PutObject against non-AWS S3-compatible storage: 1. `PutObjectRequest::GetChecksumAlgorithmName()` returns `"crc64nvme"` by default 2. `ClientConfiguration::checksumConfig::requestChecksumCalculation` defaults to `WHEN_SUPPORTED` `ChecksumInterceptor` then attaches a request hash to every PutObject, and the V4 signer switches the request to `Transfer-Encoding: aws-chunked` + `x-amz-content-sha256: STREAMING-UNSIGNED-PAYLOAD-TRAILER`. Aliyun OSS, Tencent COS, Huawei OBS, and GCP all reject this streaming form. This makes any `MinioChunkManager::PutObjectBuffer` call fail in an infinite `JobStateRetry` loop — json_stats / bson_inverted shared_key_index builds and any non-DiskANN-style index upload (IVF_*, HNSW, FLAT) are affected. ## Fix Restrict `requestChecksumCalculation` to `WHEN_REQUIRED` on non-AWS providers so the SDK only adds checksums when the operation model demands them (PutObject does not). Same approach as milvus-io/milvus-storage#500. Patched in both client construction paths: - `MinioChunkManager::MinioChunkManager` (storage_type=minio, address-based dispatch) — gated on `storageType != S3` - `ChunkManager.cpp` subclasses (storage_type=remote, cloud_provider-based dispatch) — applies in `GcpChunkManager`, `AliyunChunkManager`, `TencentCloudChunkManager`, `HuaweiCloudChunkManager`. `AwsChunkManager` keeps the default `WHEN_SUPPORTED`. CARDINAL_TIERED / DiskANN indexes already work because they upload via Arrow's `S3FileSystem` provided by milvus-storage, which was fixed in #500. ## Test plan - [x] Added `TEST(MinioChecksumConfig, OverridesAreWhenRequired)` in `test_storage.cpp` — asserts the override flips both directions from `WHEN_SUPPORTED` to `WHEN_REQUIRED` - [ ] Build / CI green - [ ] Manual smoke test on Aliyun OSS endpoint — index build (IVF_SQ8) and json_stats no longer fail with `x-oss-ec=0017-00000804`. CI cannot regression-detect this since MinIO doesn't reject `aws-chunked`; a real Aliyun OSS endpoint is required. ## Workaround (no code change) Setting `AWS_REQUEST_CHECKSUM_CALCULATION=when_required` and `AWS_RESPONSE_CHECKSUM_VALIDATION=when_required` on milvus pods also fixes this — the AWS SDK reads them at `ClientConfiguration` default-construct time, so it works across all bundled SDK copies without rebuilding. Useful for hotfix on running clusters until this PR ships. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: cai.zhang <cai.zhang@zilliz.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
44 lines
1.7 KiB
Bash
Executable File
44 lines
1.7 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
set -euo pipefail
|
|
|
|
# Licensed to the LF AI & Data foundation under one
|
|
# or more contributor license agreements. See the NOTICE file
|
|
# distributed with this work for additional information
|
|
# regarding copyright ownership. The ASF licenses this file
|
|
# to you under the Apache License, Version 2.0 (the
|
|
# "License"); you may not use this file except in compliance
|
|
# with the License. You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
|
|
SOURCE="${BASH_SOURCE[0]}"
|
|
while [ -h "$SOURCE" ]; do # resolve $SOURCE until the file is no longer a symlink
|
|
DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )"
|
|
SOURCE="$(readlink "$SOURCE")"
|
|
[[ $SOURCE != /* ]] && SOURCE="$DIR/$SOURCE" # if $SOURCE was a relative symlink, we need to resolve it relative to the path where the symlink file was located
|
|
done
|
|
ROOT_DIR="$( cd -P "$( dirname "$SOURCE" )/.." && pwd )"
|
|
|
|
CPP_SRC_DIR="${ROOT_DIR}/internal/core"
|
|
|
|
pushd "$CPP_SRC_DIR" >/dev/null
|
|
./run_clang_format.sh .
|
|
popd >/dev/null
|
|
|
|
# Exclude tantivy-binding.h: it is auto-generated by cbindgen and has its own formatting style
|
|
git_status=$(git -C "$ROOT_DIR" status --short -- "*.h" "*.hpp" "*.cc" "*.cpp" "*.c")
|
|
check_result=$(echo "$git_status" | grep -v "tantivy-binding.h" || true)
|
|
echo "check_result: $check_result"
|
|
if test -z "$check_result"; then
|
|
exit 0
|
|
else
|
|
echo "The cpp files are not formatted, please use internal/core/run_clang_format.sh"
|
|
exit 1
|
|
fi
|