mirror of https://github.com/milvus-io/milvus.git synced 2026-07-21 10:15:43 +00:00

Files

T

grootandGitHub 965e4b1564 build(deps): Upgrade knowhere and manage milvus-common via conan (#49912 )

issue: https://github.com/milvus-io/milvus/issues/47425

# Plan: upgrade Milvus to Knowhere with Conan `milvus-common` and Conan
OpenBLAS

## Context

Milvus currently fetches both `knowhere` and `milvus-common` from source
with CMake `FetchContent`. The requested Knowhere commit
`0e53e57cd1f67f74decbdf9af42e74d0e3850155` expects `milvus-common` and
Linux OpenBLAS from Conan 2.x instead:

-
`milvus-common/1.0.0-9ca5ea6@milvus/dev#274d428d85f1d3d996e1092f0c9c7144`
- `find_package(milvus-common REQUIRED)` and target
`milvus-common::milvus-common`
- Linux `openblas/0.3.30` with CMake target `OpenBLAS::OpenBLAS`
- macOS Apple BLAS/LAPACK via `BLA_VENDOR Apple`, not Conan OpenBLAS

Milvus also currently installs system OpenBLAS in builder Dockerfiles,
runtime Dockerfiles, dependency scripts, and packaging scripts. The
intended outcome is one build-time OpenBLAS source on Linux: Conan.
Runtime packaging should use the Conan-built shared libraries instead of
relying on distro `libopenblas-dev`/`openblas-devel` where possible.

## Implementation

1. **Update Conan dependency pins in `internal/core/conanfile.py`.**
- Add direct requirement
`milvus-common/1.0.0-9ca5ea6@milvus/dev#274d428d85f1d3d996e1092f0c9c7144`.
   - Align dependencies required by Knowhere `0e53e57c`:
     - `folly/2026.04.20.00@milvus/dev#06852bea5b6449f0c4eb0df002b5779c`
     - `fmt/11.2.0#eb98daa559c7c59d591f4720dde4cd5c`
     - `lz4/1.10.0#982d9b673900f665a1da109e09c17cab`
     - `fast_float/8.0.0@milvus/dev#c7802833c74c5a86ffed70e4af1a795e`
     - Linux-only `openblas/0.3.30`
- Add `openblas/*:dynamic_arch = True` to default options to match
Knowhere.

2. **Move Milvus core build settings to C++20.**
- Change `internal/core/CMakeLists.txt` from `CMAKE_CXX_STANDARD 17` to
`20`.
- Keep `OPENTELEMETRY_STL_VERSION=2017` unchanged; it controls
OpenTelemetry ABI mode, not the language standard.
- Change all `compiler.cppstd=17` Conan install settings in
`scripts/3rdparty_build.sh` to `compiler.cppstd=20`.
- Review `internal/core/src/storage/gcp-native-storage/CMakeLists.txt`;
if it inherits incompatible flags or includes C++20-dependent headers,
change its local `CMAKE_CXX_STANDARD 17` to `20` too.

3. **Expose Conan packages to Milvus CMake.**
- Add `find_package(milvus-common REQUIRED)` in
`internal/core/CMakeLists.txt` near the existing Conan
`find_package(...)` block.
- Add Linux-only `find_package(OpenBLAS CONFIG REQUIRED)` in
`internal/core/CMakeLists.txt` so missing Conan OpenBLAS fails before
Knowhere configures.
- Remove `-DOpenBLAS_SOURCE=AUTO` from `internal/core/build.sh`; the
requested Knowhere commit no longer consumes that flag and directly
calls `find_package(OpenBLAS CONFIG REQUIRED)` on non-Apple platforms.

4. **Stop executing source FetchContent for `milvus-common`.**
- Remove `add_subdirectory(milvus-common)` from
`internal/core/thirdparty/CMakeLists.txt`.
   - Keep `add_subdirectory(knowhere)`.
- Do not keep a compatibility alias to the old source target; all
consumers should use the Conan target.
- Leave `internal/core/thirdparty/milvus-common/` files in place for the
first pass unless cleanup is explicitly desired; they will no longer be
active once the subdirectory is removed.

5. **Pin Knowhere to the requested commit.**
   - Update `internal/core/thirdparty/knowhere/CMakeLists.txt`:
     - `KNOWHERE_VERSION` -> `0e53e57cd1f67f74decbdf9af42e74d0e3850155`.
   - Use the full SHA for deterministic FetchContent checkout.

6. **Migrate CMake links/includes from source target to Conan target.**
   - In `internal/core/src/CMakeLists.txt`:
     - Remove `${MILVUS_COMMON_INCLUDE_DIR}` from include directories.
- Add `milvus-common::milvus-common` to `CONAN_TARGETS` so
`milvus_conan_deps` propagates include directories to object libraries.
     - Remove plain `milvus-common` from `LINK_TARGETS`.
   - In `internal/core/unittest/CMakeLists.txt`:
     - Remove `${MILVUS_COMMON_INCLUDE_DIR}` from include directories.
- Remove plain `milvus-common` from `all_tests` and `test_json_uint64`
link lists; both already link `milvus_conan_deps`, which will carry
`milvus-common::milvus-common`.

7. **Adjust build-time system OpenBLAS installs.**
- Remove Linux build-time OpenBLAS package installs that can shadow or
duplicate Conan OpenBLAS:
- `scripts/install_deps.sh`: remove Ubuntu `libopenblas-dev`; remove
Rocky/Amazon/CentOS `openblas-devel` where only needed for C++ core
build.
- `build/docker/builder/cpu/ubuntu20.04/Dockerfile`,
`ubuntu22.04/Dockerfile`, `ubuntu24.04/Dockerfile`: remove
`libopenblas-dev`.
- `build/docker/builder/gpu/ubuntu20.04/Dockerfile`,
`gpu/ubuntu22.04/Dockerfile`: remove `libopenblas-dev`.
- `build/docker/builder/cpu/amazonlinux2023/Dockerfile`,
`rockylinux9/Dockerfile`: remove `openblas-devel` and any OpenBLAS
header symlink workaround that only supported system OpenBLAS.
- `build/deb/build_deb.sh`: remove build-time `libopenblas-dev` install
if no longer needed after Conan generation.
- Keep Fortran/toolchain packages needed to build Conan OpenBLAS from
source when binary packages are missing.
- Do not change `scripts/install_deps_embd.sh` or
`scripts/install_deps_msys.sh` in the first pass unless the target build
path uses them; they are separate embedded/MSYS dependency paths and may
still require manual/system OpenBLAS.
- On macOS, remove `openblas` from `.github/workflows/mac.yaml` only
after confirming the new Knowhere path uses Apple BLAS/LAPACK and Milvus
has no other macOS OpenBLAS consumer.

8. **Adjust runtime/package OpenBLAS handling carefully.**
- The runtime should still include `libopenblas.so*` if the Milvus
binary or Conan-built Knowhere/Faiss links it dynamically.
- Prefer copying the Conan-provided OpenBLAS shared library from
`cmake_build/lib` or the Conan package output instead of installing
distro development packages in runtime images.
- Update packaging sites that currently copy or depend on system
OpenBLAS:
- `build/deb/build_deb.sh`: change the copied source from
`/usr/lib/x86_64-linux-gnu/libopenblas.so.0` to the Conan/build output
path if present.
- `build/rpm/milvus.spec`: stop assuming
`/usr/lib/libopenblas-r0.3.9.so`; use the packaged Conan/build output
library.
- Runtime Dockerfiles under `build/docker/milvus/**`: remove
`libopenblas-dev`/`openblas-devel` only after confirming the image gets
the Conan OpenBLAS `.so` with Milvus libs.
- `build/docker/milvus/gpu/ubuntu20.04/Dockerfile.base`: review the
explicit `milvusdb/openblas` stage; replace it with the Conan-produced
library if that image path is still active.
- Keep runtime `libgomp1`/`libgomp` and `libaio` if required by
OpenMP/libaio-linked dependencies.

9. **Clean stale build state before verification.**
- Use a fresh `cmake_build` or remove the old FetchContent
checkout/cache before rebuilding; stale `MILVUS_COMMON_INCLUDE_DIR`, old
`knowhere-src`, or system OpenBLAS paths can hide integration errors.

Signed-off-by: yhmo <yihua.mo@zilliz.com>

2026-06-03 14:48:19 +08:00

1.9 KiB

Raw Permalink Blame History

Compile and install milvus cluster

Environment

OS: Ubuntu 20.04
go：1.21
cmake: >=3.18
gcc: >= 11

Install dependencies

Install compile dependencies

$ sudo apt install -y g++ gcc make libssl-dev zlib1g-dev libboost-regex-dev \
    libboost-program-options-dev libboost-system-dev libboost-filesystem-dev \
    libboost-serialization-dev python3-dev libboost-python-dev libcurl4-openssl-dev gfortran libtbb-dev
$ export GO111MODULE=on
$ go get github.com/golang/protobuf/protoc-gen-go@v1.3.2

OpenBLAS is managed by Conan for C++ core builds. Install the compiler toolchain and Fortran runtime needed by Conan, then run the normal third-party build step.

Compile

Generate the go files from proto file

$ make check-proto-product

Check code specifications

$ make verifiers

Compile milvus

$ make milvus

Install docker-compose

refer: https://docs.docker.com/compose/install/

$ sudo curl -L "https://github.com/docker/compose/releases/download/1.27.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
$ sudo chmod +x /usr/local/bin/docker-compose
$ sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
$ docker-compose --version
$ docker compose --version

Start service

Start third-party service:

$ cd [milvus project path]/deployments/docker/cluster
$ docker-compose up -d
$ docker compose up -d

Start milvus cluster:

$ cd [milvus project path]
$ ./scripts/start_cluster.sh

Run unittest

Run all unittest including go and cpp cases:

$ make unittest

You also can run go unittest only:

$ make test-go

Run cpp unittest only:

$ make test-cpp

Run code coverage

Run code coverage including go and cpp:

$ make codecov

You also can run go code coverage only:

$ make codecov-go

Run cpp code coverage only:

$ make codecov-cpp

1.9 KiB Raw Permalink Blame History Unescape Escape