issue: #50615
## What this PR does
Makes the C++ core build cleanly on **macOS (Apple Silicon) with
Homebrew LLVM/clang 19**.
> [!IMPORTANT]
> **Scope / impact is NOT macOS-only.** Although this PR is framed as
macOS/clang-19 enablement, two of the changes are **top-level
`requires(...)` in `internal/core/conanfile.py`, so they apply to every
platform, including Linux production builds:**
> - **`azure-sdk-for-cpp` `1.11.3@milvus/dev` → `1.16.0@milvus/dev`** (5
minor versions, `force=True`). This is the SDK Arrow uses for the
**Azure Blob remote-storage path**, so this is effectively a major Azure
SDK bump on the **production Azure object-storage path for Linux
deployments** too.
> - **`arrow` recipe `17.0.0@milvus/dev-2.6#c743ea7a…` →
`17.0.0@milvus/dev#f411aa73…`** — same Arrow version, but **rebuilt
against azure 1.16.0** with a patched `FindAzure.cmake`. The Arrow
**`package_id` (recipe revision) DID change**; it is not the same binary
as before.
> - **`libbson/1.30.6@milvus/dev`** added (also global, but new — no
pre-existing behavior to regress).
>
> **Testing blind spot:** CI green proves compile/link/UT/e2e pass, but
the UT and integration suites **do not exercise real Azure Blob
read/write**, so the Azure object-storage path is **not covered** by
this PR's CI. A real Azure Blob remote-storage regression is recommended
before merge (see Verification).
### Why the dependency bumps are required for clang 19
- **azure 1.16.0**: azure 1.11.3 bundles an older `nlohmann/json` that
uses `std::char_traits<unsigned char>`, which was **removed in libc++
19** → 1.11.3 fails to compile on clang 19. 1.16.0 fixes this.
- **arrow rebuilt against azure 1.16.0**: when Arrow is built from
source on clang 19 it must link the same azure 1.16.0; the recipe also
patches `FindAzure.cmake` to the monolithic `Azure` config so
`find_package(Azure)` resolves. Hence a new Arrow recipe revision.
### Toolchain / build enablement
- **`scripts/setenv.sh`**: probe LLVM 19/20/21 first (libc++ 17/18
`chrono operator<<` clashes with Arrow's vendored `date`; libc++ 19
fixes it).
- **`scripts/3rdparty_build.sh`**: arm64 `-march=armv8-a+crypto+crc` so
folly's F14 (SimdAndCrc on Apple Silicon) matches core/knowhere — fixes
`F14LinkCheck` undefined-symbol link errors.
- **`InvertedIndexTantivy.cpp`**: drop spurious `->template` on
non-template members (clang 19
`-Wmissing-template-arg-list-after-template-kw`).
### Drop bsoncxx / libmongoc → libbson only
The BSON layer used the **bsoncxx** C++ driver, whose CMake
unconditionally builds mongo-cxx-driver → **libmongoc** → bundled
**utf8proc**, which fails to configure under newer CMake/clang on macOS.
Milvus only needs BSON (de)serialization.
- New Conan package **`libbson/1.30.6@milvus/dev`** (libbson-only,
`ENABLE_MONGOC=OFF`; recipe published to milvus conanfiles &
production).
- New **`src/common/bson_shim.h`** — a thin `milvus::bson` C++ layer
over libbson's C API.
- Migrated `bson_view.h`, `bson_builder.{h,cpp}`, json_stats consumers
and unit tests off bsoncxx.
- CMake: `find_package(bson-1.0)` + link `mongo::bson_shared`; removed
FetchContent `thirdparty/bsoncxx`.
All three global Conan requires (arrow / libbson / azure) pin an
explicit recipe `#revision` for reproducibility.
## Verification
- Full clean rebuild on macOS arm64 / clang 19: `bin/milvus` builds;
`libmilvus_core` links **only `libbson-1.0.dylib`** (no
libmongoc/mongocxx/utf8proc).
- libbson / arrow / azure resolve from production Conan at the pinned
revisions.
- All 21 BSON unit tests (BsonView / BsonBuilder / DomNode) pass.
- Full ci-v2 suite green on Linux (build / build-ut-cov / ut-cpp / ut-go
/ integration / e2e / go-sdk).
- **TODO before merge:** real Azure Blob remote-storage read/write
regression to cover the azure-1.16.0 / arrow-rebuild blast radius (not
exercised by UT/integration).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What this PR does
Removes two unused C++ third-party dependencies from `internal/core` to
reduce dependency count, comprehension cost, and link/build surface.
### rapidjson
Linked into `milvus_core` but has **zero `#include` usage** in the C++
sources (only a stale comment in `common/Json.h`). knowhere v3.0.3 and
milvus-storage do not depend on it. Removes the conan requirement,
`find_package(RapidJSON)`, and the link target.
### OpenDAL backend
Experimental `ChunkManager` backend gated behind `USE_OPENDAL`
(**default OFF**) — released binaries never built it. Removes
`OpenDALChunkManager`, the vendored OpenDAL Rust dependency, and all
`USE_OPENDAL` build plumbing (Makefile / core_build.sh /
3rdparty_build.sh / CMake). The Go-side `"opendal"` storageType value
remains a backward-compatible alias to the standard remote chunk
manager.
### Kept
`simde` is **not** removed — knowhere v3.0.3 requires it
(`find_package(simde REQUIRED)`, used in
`src/index/sparse/codec/varintdecode.c`).
issue: #50551🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
issue: https://github.com/milvus-io/milvus/issues/47425
Milvus already asks Conan to build the C++ third-party stack with C++20
for the host profile, but Conan keeps the build profile separate. That
means packages used during the build can still fall back to the default
C++ standard, which caused protobuf and abseil to be compiled as gnu17
in some environments and fail during the dependency build.
This change passes C++20 to the Conan build profile as well, so both the
host context and the build context use the same language level. It
applies the setting to both Linux Conan install branches in the
third-party build script, including the gcc4 ABI path and the normal
libstdc++11 path.
Signed-off-by: jiaqizho <jiaqi.zhou@zilliz.com>
Co-authored-by: EC2 Default User <ec2-user@ip-10-15-6-84.us-west-2.compute.internal>
issue: https://github.com/milvus-io/milvus/issues/47425
# Plan: upgrade Milvus to Knowhere with Conan `milvus-common` and Conan
OpenBLAS
## Context
Milvus currently fetches both `knowhere` and `milvus-common` from source
with CMake `FetchContent`. The requested Knowhere commit
`0e53e57cd1f67f74decbdf9af42e74d0e3850155` expects `milvus-common` and
Linux OpenBLAS from Conan 2.x instead:
-
`milvus-common/1.0.0-9ca5ea6@milvus/dev#274d428d85f1d3d996e1092f0c9c7144`
- `find_package(milvus-common REQUIRED)` and target
`milvus-common::milvus-common`
- Linux `openblas/0.3.30` with CMake target `OpenBLAS::OpenBLAS`
- macOS Apple BLAS/LAPACK via `BLA_VENDOR Apple`, not Conan OpenBLAS
Milvus also currently installs system OpenBLAS in builder Dockerfiles,
runtime Dockerfiles, dependency scripts, and packaging scripts. The
intended outcome is one build-time OpenBLAS source on Linux: Conan.
Runtime packaging should use the Conan-built shared libraries instead of
relying on distro `libopenblas-dev`/`openblas-devel` where possible.
## Implementation
1. **Update Conan dependency pins in `internal/core/conanfile.py`.**
- Add direct requirement
`milvus-common/1.0.0-9ca5ea6@milvus/dev#274d428d85f1d3d996e1092f0c9c7144`.
- Align dependencies required by Knowhere `0e53e57c`:
- `folly/2026.04.20.00@milvus/dev#06852bea5b6449f0c4eb0df002b5779c`
- `fmt/11.2.0#eb98daa559c7c59d591f4720dde4cd5c`
- `lz4/1.10.0#982d9b673900f665a1da109e09c17cab`
- `fast_float/8.0.0@milvus/dev#c7802833c74c5a86ffed70e4af1a795e`
- Linux-only `openblas/0.3.30`
- Add `openblas/*:dynamic_arch = True` to default options to match
Knowhere.
2. **Move Milvus core build settings to C++20.**
- Change `internal/core/CMakeLists.txt` from `CMAKE_CXX_STANDARD 17` to
`20`.
- Keep `OPENTELEMETRY_STL_VERSION=2017` unchanged; it controls
OpenTelemetry ABI mode, not the language standard.
- Change all `compiler.cppstd=17` Conan install settings in
`scripts/3rdparty_build.sh` to `compiler.cppstd=20`.
- Review `internal/core/src/storage/gcp-native-storage/CMakeLists.txt`;
if it inherits incompatible flags or includes C++20-dependent headers,
change its local `CMAKE_CXX_STANDARD 17` to `20` too.
3. **Expose Conan packages to Milvus CMake.**
- Add `find_package(milvus-common REQUIRED)` in
`internal/core/CMakeLists.txt` near the existing Conan
`find_package(...)` block.
- Add Linux-only `find_package(OpenBLAS CONFIG REQUIRED)` in
`internal/core/CMakeLists.txt` so missing Conan OpenBLAS fails before
Knowhere configures.
- Remove `-DOpenBLAS_SOURCE=AUTO` from `internal/core/build.sh`; the
requested Knowhere commit no longer consumes that flag and directly
calls `find_package(OpenBLAS CONFIG REQUIRED)` on non-Apple platforms.
4. **Stop executing source FetchContent for `milvus-common`.**
- Remove `add_subdirectory(milvus-common)` from
`internal/core/thirdparty/CMakeLists.txt`.
- Keep `add_subdirectory(knowhere)`.
- Do not keep a compatibility alias to the old source target; all
consumers should use the Conan target.
- Leave `internal/core/thirdparty/milvus-common/` files in place for the
first pass unless cleanup is explicitly desired; they will no longer be
active once the subdirectory is removed.
5. **Pin Knowhere to the requested commit.**
- Update `internal/core/thirdparty/knowhere/CMakeLists.txt`:
- `KNOWHERE_VERSION` -> `0e53e57cd1f67f74decbdf9af42e74d0e3850155`.
- Use the full SHA for deterministic FetchContent checkout.
6. **Migrate CMake links/includes from source target to Conan target.**
- In `internal/core/src/CMakeLists.txt`:
- Remove `${MILVUS_COMMON_INCLUDE_DIR}` from include directories.
- Add `milvus-common::milvus-common` to `CONAN_TARGETS` so
`milvus_conan_deps` propagates include directories to object libraries.
- Remove plain `milvus-common` from `LINK_TARGETS`.
- In `internal/core/unittest/CMakeLists.txt`:
- Remove `${MILVUS_COMMON_INCLUDE_DIR}` from include directories.
- Remove plain `milvus-common` from `all_tests` and `test_json_uint64`
link lists; both already link `milvus_conan_deps`, which will carry
`milvus-common::milvus-common`.
7. **Adjust build-time system OpenBLAS installs.**
- Remove Linux build-time OpenBLAS package installs that can shadow or
duplicate Conan OpenBLAS:
- `scripts/install_deps.sh`: remove Ubuntu `libopenblas-dev`; remove
Rocky/Amazon/CentOS `openblas-devel` where only needed for C++ core
build.
- `build/docker/builder/cpu/ubuntu20.04/Dockerfile`,
`ubuntu22.04/Dockerfile`, `ubuntu24.04/Dockerfile`: remove
`libopenblas-dev`.
- `build/docker/builder/gpu/ubuntu20.04/Dockerfile`,
`gpu/ubuntu22.04/Dockerfile`: remove `libopenblas-dev`.
- `build/docker/builder/cpu/amazonlinux2023/Dockerfile`,
`rockylinux9/Dockerfile`: remove `openblas-devel` and any OpenBLAS
header symlink workaround that only supported system OpenBLAS.
- `build/deb/build_deb.sh`: remove build-time `libopenblas-dev` install
if no longer needed after Conan generation.
- Keep Fortran/toolchain packages needed to build Conan OpenBLAS from
source when binary packages are missing.
- Do not change `scripts/install_deps_embd.sh` or
`scripts/install_deps_msys.sh` in the first pass unless the target build
path uses them; they are separate embedded/MSYS dependency paths and may
still require manual/system OpenBLAS.
- On macOS, remove `openblas` from `.github/workflows/mac.yaml` only
after confirming the new Knowhere path uses Apple BLAS/LAPACK and Milvus
has no other macOS OpenBLAS consumer.
8. **Adjust runtime/package OpenBLAS handling carefully.**
- The runtime should still include `libopenblas.so*` if the Milvus
binary or Conan-built Knowhere/Faiss links it dynamically.
- Prefer copying the Conan-provided OpenBLAS shared library from
`cmake_build/lib` or the Conan package output instead of installing
distro development packages in runtime images.
- Update packaging sites that currently copy or depend on system
OpenBLAS:
- `build/deb/build_deb.sh`: change the copied source from
`/usr/lib/x86_64-linux-gnu/libopenblas.so.0` to the Conan/build output
path if present.
- `build/rpm/milvus.spec`: stop assuming
`/usr/lib/libopenblas-r0.3.9.so`; use the packaged Conan/build output
library.
- Runtime Dockerfiles under `build/docker/milvus/**`: remove
`libopenblas-dev`/`openblas-devel` only after confirming the image gets
the Conan OpenBLAS `.so` with Milvus libs.
- `build/docker/milvus/gpu/ubuntu20.04/Dockerfile.base`: review the
explicit `milvusdb/openblas` stage; replace it with the Conan-produced
library if that image path is still active.
- Keep runtime `libgomp1`/`libgomp` and `libaio` if required by
OpenMP/libaio-linked dependencies.
9. **Clean stale build state before verification.**
- Use a fresh `cmake_build` or remove the old FetchContent
checkout/cache before rebuilding; stale `MILVUS_COMMON_INCLUDE_DIR`, old
`knowhere-src`, or system OpenBLAS paths can hide integration errors.
Signed-off-by: yhmo <yihua.mo@zilliz.com>
issue: [#45691](https://github.com/milvus-io/milvus/issues/45691)
## Summary
Bump rust toolchain `1.89 → 1.92` only in installation paths
(Dockerfiles + install scripts).
## Why
milvus-storage HEAD upgrades `iceberg` crate `0.8 → 0.9`, which declares
`edition = "2024"` and `rust-version = "1.92"`. The new MSRV requires
the builder image to ship rust 1.92 before milvus can take that
milvus-storage commit.
This PR is the **first step**:
1. Bump rust toolchain in installation scripts and Dockerfiles.
2. Once merged, builder image is republished with rust 1.92.
3. Follow-up PR will bump `.env` `DATE_VERSION` to the new tag, switch
`cargo +1.89` → `cargo +1.92` in
`internal/core/thirdparty/{tantivy,opendal}/CMakeLists.txt`, and bump
milvus-storage to the iceberg-0.9 commit.
This split avoids the chicken-egg problem: changing the cargo pin or
milvus-storage version in this PR would fail CI, since the currently
pinned builder image only ships rust 1.89.
## Files changed
| File | Change |
|---|---|
| `scripts/install_deps.sh` | `RUST_VERSION="1.89"` → `1.92` |
| `scripts/3rdparty_build.sh` | rust toolchain `1.89` → `1.92` (mac +
linux paths) |
| `build/docker/builder/{cpu,gpu}/*/Dockerfile` (7 files) | rustup
`--default-toolchain=1.89` → `1.92` |
| `DEVELOPMENT.md` | doc table rust version |
## Test plan
- [ ] CI passes on existing builder image (no behaviour change for
current build path)
- [ ] After merge, builder image is republished with rust 1.92
- [ ] Follow-up PR can then bump `.env` and milvus-storage
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Fixes#49106.
Follow-up to #48165. Adds an opt-in `CONAN_CMD` environment variable so
developers working on both master (Conan 2.x) and release-2.5/2.6
(Conan 1.x) on the same machine can select a Conan binary per-build
without switching their default `conan`. See commit message for details.
## Test plan
- [x] `make` with default conan (2.x) — unchanged behavior
- [x] `CONAN_CMD=conan-1 make` — fails with clear version mismatch error
- [x] `CONAN_CMD=/nonexistent make` — fails with clear error
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
Switch CI pipelines from Release (`-O3`) to RelWithDebInfo (`-O2 -g
-DNDEBUG`), reducing peak compiler memory by ~15%. RelWithDebInfo is a
standard CMake build type that uses `-O2` natively and includes debug
symbols for better crash diagnostics in CI. Production image builds
remain at Release (`-O3`).
## Changes
- Update Jenkins (PR, PR-Arm, PR-for-go-sdk, UT-CPP) to use
`mode=RelWithDebInfo`
- Update GitHub Actions (`main.yaml`) to use `mode=RelWithDebInfo`
- Allow all standard CMake build types in `3rdparty_build.sh` validation
- Add gcc>=13 implicit include workaround in `CMakeLists.txt`
## Test plan
- [ ] CI builds with `mode=RelWithDebInfo` use `-O2` instead of `-O3`
- [ ] Default builds still use Release with `-O3`
- [ ] ASAN CI jobs pass with reduced memory usage
issue: #48400
Signed-off-by: xiaofan-luan <xiaofan.xl@zilliz.com>
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Summary
- Bump milvus-storage from f372333 to 0a9cb7ebe8
- Upgrade google-cloud-cpp from 2.5.0@milvus/2.4 to 2.28.0 (official
Conan Center package)
- Adapt storage layer code to new google-cloud-cpp 2.28.0 API changes
issue: #47863
## Changes
1. **conanfile.py**: Switch to upstream `google-cloud-cpp/2.28.0` from
custom `@milvus/2.4` package
2. **ChunkManager.cpp / MinioChunkManager.cpp**: Add `HttpClientFactory`
parameter to `ComputeEngineCredentials` constructor
3. **MinioChunkManager.h**: Add required headers, update
`AuthorizationHeader` to free function, qualify `StatusCodeToString`
4. **ffi_reader_c.cpp**: Rename `import_column_groups` →
`column_groups_import` per new milvus-storage API
5. **milvus-storage/CMakeLists.txt**: Bump version to `0a9cb7ebe8`
## Test plan
- [ ] CI build passes (C++ compilation with new dependencies)
- [ ] Unit tests pass
- [ ] Integration tests pass with GCP storage backend
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
See #42091
In this PR,
- Added a new `-t` option to specify the build type (Debug/Release).
- Updated the Makefile to pass the build type to the 3rdparty build
script.
- Included usage instructions for the script to improve user guidance.
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
issue: #40243
This helps identify the running OS version during the build process,
ensuring better logging.
Signed-off-by: Shubhendra Kushwaha <shubhendrakushwaha94@gmail.com>
