issue: https://github.com/milvus-io/milvus/issues/47425
# Plan: upgrade Milvus to Knowhere with Conan `milvus-common` and Conan
OpenBLAS
## Context
Milvus currently fetches both `knowhere` and `milvus-common` from source
with CMake `FetchContent`. The requested Knowhere commit
`0e53e57cd1f67f74decbdf9af42e74d0e3850155` expects `milvus-common` and
Linux OpenBLAS from Conan 2.x instead:
-
`milvus-common/1.0.0-9ca5ea6@milvus/dev#274d428d85f1d3d996e1092f0c9c7144`
- `find_package(milvus-common REQUIRED)` and target
`milvus-common::milvus-common`
- Linux `openblas/0.3.30` with CMake target `OpenBLAS::OpenBLAS`
- macOS Apple BLAS/LAPACK via `BLA_VENDOR Apple`, not Conan OpenBLAS
Milvus also currently installs system OpenBLAS in builder Dockerfiles,
runtime Dockerfiles, dependency scripts, and packaging scripts. The
intended outcome is one build-time OpenBLAS source on Linux: Conan.
Runtime packaging should use the Conan-built shared libraries instead of
relying on distro `libopenblas-dev`/`openblas-devel` where possible.
## Implementation
1. **Update Conan dependency pins in `internal/core/conanfile.py`.**
- Add direct requirement
`milvus-common/1.0.0-9ca5ea6@milvus/dev#274d428d85f1d3d996e1092f0c9c7144`.
- Align dependencies required by Knowhere `0e53e57c`:
- `folly/2026.04.20.00@milvus/dev#06852bea5b6449f0c4eb0df002b5779c`
- `fmt/11.2.0#eb98daa559c7c59d591f4720dde4cd5c`
- `lz4/1.10.0#982d9b673900f665a1da109e09c17cab`
- `fast_float/8.0.0@milvus/dev#c7802833c74c5a86ffed70e4af1a795e`
- Linux-only `openblas/0.3.30`
- Add `openblas/*:dynamic_arch = True` to default options to match
Knowhere.
2. **Move Milvus core build settings to C++20.**
- Change `internal/core/CMakeLists.txt` from `CMAKE_CXX_STANDARD 17` to
`20`.
- Keep `OPENTELEMETRY_STL_VERSION=2017` unchanged; it controls
OpenTelemetry ABI mode, not the language standard.
- Change all `compiler.cppstd=17` Conan install settings in
`scripts/3rdparty_build.sh` to `compiler.cppstd=20`.
- Review `internal/core/src/storage/gcp-native-storage/CMakeLists.txt`;
if it inherits incompatible flags or includes C++20-dependent headers,
change its local `CMAKE_CXX_STANDARD 17` to `20` too.
3. **Expose Conan packages to Milvus CMake.**
- Add `find_package(milvus-common REQUIRED)` in
`internal/core/CMakeLists.txt` near the existing Conan
`find_package(...)` block.
- Add Linux-only `find_package(OpenBLAS CONFIG REQUIRED)` in
`internal/core/CMakeLists.txt` so missing Conan OpenBLAS fails before
Knowhere configures.
- Remove `-DOpenBLAS_SOURCE=AUTO` from `internal/core/build.sh`; the
requested Knowhere commit no longer consumes that flag and directly
calls `find_package(OpenBLAS CONFIG REQUIRED)` on non-Apple platforms.
4. **Stop executing source FetchContent for `milvus-common`.**
- Remove `add_subdirectory(milvus-common)` from
`internal/core/thirdparty/CMakeLists.txt`.
- Keep `add_subdirectory(knowhere)`.
- Do not keep a compatibility alias to the old source target; all
consumers should use the Conan target.
- Leave `internal/core/thirdparty/milvus-common/` files in place for the
first pass unless cleanup is explicitly desired; they will no longer be
active once the subdirectory is removed.
5. **Pin Knowhere to the requested commit.**
- Update `internal/core/thirdparty/knowhere/CMakeLists.txt`:
- `KNOWHERE_VERSION` -> `0e53e57cd1f67f74decbdf9af42e74d0e3850155`.
- Use the full SHA for deterministic FetchContent checkout.
6. **Migrate CMake links/includes from source target to Conan target.**
- In `internal/core/src/CMakeLists.txt`:
- Remove `${MILVUS_COMMON_INCLUDE_DIR}` from include directories.
- Add `milvus-common::milvus-common` to `CONAN_TARGETS` so
`milvus_conan_deps` propagates include directories to object libraries.
- Remove plain `milvus-common` from `LINK_TARGETS`.
- In `internal/core/unittest/CMakeLists.txt`:
- Remove `${MILVUS_COMMON_INCLUDE_DIR}` from include directories.
- Remove plain `milvus-common` from `all_tests` and `test_json_uint64`
link lists; both already link `milvus_conan_deps`, which will carry
`milvus-common::milvus-common`.
7. **Adjust build-time system OpenBLAS installs.**
- Remove Linux build-time OpenBLAS package installs that can shadow or
duplicate Conan OpenBLAS:
- `scripts/install_deps.sh`: remove Ubuntu `libopenblas-dev`; remove
Rocky/Amazon/CentOS `openblas-devel` where only needed for C++ core
build.
- `build/docker/builder/cpu/ubuntu20.04/Dockerfile`,
`ubuntu22.04/Dockerfile`, `ubuntu24.04/Dockerfile`: remove
`libopenblas-dev`.
- `build/docker/builder/gpu/ubuntu20.04/Dockerfile`,
`gpu/ubuntu22.04/Dockerfile`: remove `libopenblas-dev`.
- `build/docker/builder/cpu/amazonlinux2023/Dockerfile`,
`rockylinux9/Dockerfile`: remove `openblas-devel` and any OpenBLAS
header symlink workaround that only supported system OpenBLAS.
- `build/deb/build_deb.sh`: remove build-time `libopenblas-dev` install
if no longer needed after Conan generation.
- Keep Fortran/toolchain packages needed to build Conan OpenBLAS from
source when binary packages are missing.
- Do not change `scripts/install_deps_embd.sh` or
`scripts/install_deps_msys.sh` in the first pass unless the target build
path uses them; they are separate embedded/MSYS dependency paths and may
still require manual/system OpenBLAS.
- On macOS, remove `openblas` from `.github/workflows/mac.yaml` only
after confirming the new Knowhere path uses Apple BLAS/LAPACK and Milvus
has no other macOS OpenBLAS consumer.
8. **Adjust runtime/package OpenBLAS handling carefully.**
- The runtime should still include `libopenblas.so*` if the Milvus
binary or Conan-built Knowhere/Faiss links it dynamically.
- Prefer copying the Conan-provided OpenBLAS shared library from
`cmake_build/lib` or the Conan package output instead of installing
distro development packages in runtime images.
- Update packaging sites that currently copy or depend on system
OpenBLAS:
- `build/deb/build_deb.sh`: change the copied source from
`/usr/lib/x86_64-linux-gnu/libopenblas.so.0` to the Conan/build output
path if present.
- `build/rpm/milvus.spec`: stop assuming
`/usr/lib/libopenblas-r0.3.9.so`; use the packaged Conan/build output
library.
- Runtime Dockerfiles under `build/docker/milvus/**`: remove
`libopenblas-dev`/`openblas-devel` only after confirming the image gets
the Conan OpenBLAS `.so` with Milvus libs.
- `build/docker/milvus/gpu/ubuntu20.04/Dockerfile.base`: review the
explicit `milvusdb/openblas` stage; replace it with the Conan-produced
library if that image path is still active.
- Keep runtime `libgomp1`/`libgomp` and `libaio` if required by
OpenMP/libaio-linked dependencies.
9. **Clean stale build state before verification.**
- Use a fresh `cmake_build` or remove the old FetchContent
checkout/cache before rebuilding; stale `MILVUS_COMMON_INCLUDE_DIR`, old
`knowhere-src`, or system OpenBLAS paths can hide integration errors.
Signed-off-by: yhmo <yihua.mo@zilliz.com>
issue: #49977
## Summary
AWS SDK C++ 1.11.692 (bumped from 1.11.352 in #47810) changed two
defaults that together break PutObject against non-AWS S3-compatible
storage:
1. `PutObjectRequest::GetChecksumAlgorithmName()` returns `"crc64nvme"`
by default
2. `ClientConfiguration::checksumConfig::requestChecksumCalculation`
defaults to `WHEN_SUPPORTED`
`ChecksumInterceptor` then attaches a request hash to every PutObject,
and the V4 signer switches the request to `Transfer-Encoding:
aws-chunked` + `x-amz-content-sha256:
STREAMING-UNSIGNED-PAYLOAD-TRAILER`. Aliyun OSS, Tencent COS, Huawei
OBS, and GCP all reject this streaming form.
This makes any `MinioChunkManager::PutObjectBuffer` call fail in an
infinite `JobStateRetry` loop — json_stats / bson_inverted
shared_key_index builds and any non-DiskANN-style index upload (IVF_*,
HNSW, FLAT) are affected.
## Fix
Restrict `requestChecksumCalculation` to `WHEN_REQUIRED` on non-AWS
providers so the SDK only adds checksums when the operation model
demands them (PutObject does not). Same approach as
milvus-io/milvus-storage#500.
Patched in both client construction paths:
- `MinioChunkManager::MinioChunkManager` (storage_type=minio,
address-based dispatch) — gated on `storageType != S3`
- `ChunkManager.cpp` subclasses (storage_type=remote,
cloud_provider-based dispatch) — applies in `GcpChunkManager`,
`AliyunChunkManager`, `TencentCloudChunkManager`,
`HuaweiCloudChunkManager`. `AwsChunkManager` keeps the default
`WHEN_SUPPORTED`.
CARDINAL_TIERED / DiskANN indexes already work because they upload via
Arrow's `S3FileSystem` provided by milvus-storage, which was fixed in
#500.
## Test plan
- [x] Added `TEST(MinioChecksumConfig, OverridesAreWhenRequired)` in
`test_storage.cpp` — asserts the override flips both directions from
`WHEN_SUPPORTED` to `WHEN_REQUIRED`
- [ ] Build / CI green
- [ ] Manual smoke test on Aliyun OSS endpoint — index build (IVF_SQ8)
and json_stats no longer fail with `x-oss-ec=0017-00000804`. CI cannot
regression-detect this since MinIO doesn't reject `aws-chunked`; a real
Aliyun OSS endpoint is required.
## Workaround (no code change)
Setting `AWS_REQUEST_CHECKSUM_CALCULATION=when_required` and
`AWS_RESPONSE_CHECKSUM_VALIDATION=when_required` on milvus pods also
fixes this — the AWS SDK reads them at `ClientConfiguration`
default-construct time, so it works across all bundled SDK copies
without rebuilding. Useful for hotfix on running clusters until this PR
ships.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Related issue: #48145
scripts/install_deps.sh fails to install Conan on Ubuntu 24.04 in two
scenarios:
1. System Python (PEP 668): `pip install --user conan` fails with
`error: externally-managed-environment`
2. Active virtual environment: `pip install --user conan` fails with
`ERROR: Can not perform a '--user' install. User site-packages are not
visible in this virtualenv.`
Fall back to an isolated venv (~/.venv/conan) with a ~/.local/bin/conan
symlink when --user install is unavailable. Behavior on Ubuntu
20.04/22.04, macOS, and Rocky Linux is unchanged.
Signed-off-by: Vikhlyantsev Konstantin <vihlancevk@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
issue: [#45691](https://github.com/milvus-io/milvus/issues/45691)
## Summary
Bump rust toolchain `1.89 → 1.92` only in installation paths
(Dockerfiles + install scripts).
## Why
milvus-storage HEAD upgrades `iceberg` crate `0.8 → 0.9`, which declares
`edition = "2024"` and `rust-version = "1.92"`. The new MSRV requires
the builder image to ship rust 1.92 before milvus can take that
milvus-storage commit.
This PR is the **first step**:
1. Bump rust toolchain in installation scripts and Dockerfiles.
2. Once merged, builder image is republished with rust 1.92.
3. Follow-up PR will bump `.env` `DATE_VERSION` to the new tag, switch
`cargo +1.89` → `cargo +1.92` in
`internal/core/thirdparty/{tantivy,opendal}/CMakeLists.txt`, and bump
milvus-storage to the iceberg-0.9 commit.
This split avoids the chicken-egg problem: changing the cargo pin or
milvus-storage version in this PR would fail CI, since the currently
pinned builder image only ships rust 1.89.
## Files changed
| File | Change |
|---|---|
| `scripts/install_deps.sh` | `RUST_VERSION="1.89"` → `1.92` |
| `scripts/3rdparty_build.sh` | rust toolchain `1.89` → `1.92` (mac +
linux paths) |
| `build/docker/builder/{cpu,gpu}/*/Dockerfile` (7 files) | rustup
`--default-toolchain=1.89` → `1.92` |
| `DEVELOPMENT.md` | doc table rust version |
## Test plan
- [ ] CI passes on existing builder image (no behaviour change for
current build path)
- [ ] After merge, builder image is republished with rust 1.92
- [ ] Follow-up PR can then bump `.env` and milvus-storage
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Default llvm toolchain version in Ubuntu 20.04 is 10, while Ubuntu 22.04
does not have `clang-tidy-10` or `clang-format-10` by default.
issue: #33142
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
Signed-off-by: Yinzuo Jiang <jiangyinzuo@foxmail.com>
issue #32476
tested on x86_64 and aarch64. I'm not sure what needs to be done on some
exotic architectures.
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
Install openblas using apt or yum in scripts/install_deps.sh, update
documentations and fix some typos related to build and installation.
issue: #33056, #33066
Signed-off-by: Yinzuo Jiang <jiangyinzuo@foxmail.com>
cmake version needs to be greater or equal than 3.26.4, sync
`install_deps.sh` scripts version to 3.26.5
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
