mirror of
https://github.com/milvus-io/milvus.git
synced 2026-07-21 18:25:44 +00:00
grootandGitHub
965e4b1564
issue: https://github.com/milvus-io/milvus/issues/47425 # Plan: upgrade Milvus to Knowhere with Conan `milvus-common` and Conan OpenBLAS ## Context Milvus currently fetches both `knowhere` and `milvus-common` from source with CMake `FetchContent`. The requested Knowhere commit `0e53e57cd1f67f74decbdf9af42e74d0e3850155` expects `milvus-common` and Linux OpenBLAS from Conan 2.x instead: - `milvus-common/1.0.0-9ca5ea6@milvus/dev#274d428d85f1d3d996e1092f0c9c7144` - `find_package(milvus-common REQUIRED)` and target `milvus-common::milvus-common` - Linux `openblas/0.3.30` with CMake target `OpenBLAS::OpenBLAS` - macOS Apple BLAS/LAPACK via `BLA_VENDOR Apple`, not Conan OpenBLAS Milvus also currently installs system OpenBLAS in builder Dockerfiles, runtime Dockerfiles, dependency scripts, and packaging scripts. The intended outcome is one build-time OpenBLAS source on Linux: Conan. Runtime packaging should use the Conan-built shared libraries instead of relying on distro `libopenblas-dev`/`openblas-devel` where possible. ## Implementation 1. **Update Conan dependency pins in `internal/core/conanfile.py`.** - Add direct requirement `milvus-common/1.0.0-9ca5ea6@milvus/dev#274d428d85f1d3d996e1092f0c9c7144`. - Align dependencies required by Knowhere `0e53e57c`: - `folly/2026.04.20.00@milvus/dev#06852bea5b6449f0c4eb0df002b5779c` - `fmt/11.2.0#eb98daa559c7c59d591f4720dde4cd5c` - `lz4/1.10.0#982d9b673900f665a1da109e09c17cab` - `fast_float/8.0.0@milvus/dev#c7802833c74c5a86ffed70e4af1a795e` - Linux-only `openblas/0.3.30` - Add `openblas/*:dynamic_arch = True` to default options to match Knowhere. 2. **Move Milvus core build settings to C++20.** - Change `internal/core/CMakeLists.txt` from `CMAKE_CXX_STANDARD 17` to `20`. - Keep `OPENTELEMETRY_STL_VERSION=2017` unchanged; it controls OpenTelemetry ABI mode, not the language standard. - Change all `compiler.cppstd=17` Conan install settings in `scripts/3rdparty_build.sh` to `compiler.cppstd=20`. - Review `internal/core/src/storage/gcp-native-storage/CMakeLists.txt`; if it inherits incompatible flags or includes C++20-dependent headers, change its local `CMAKE_CXX_STANDARD 17` to `20` too. 3. **Expose Conan packages to Milvus CMake.** - Add `find_package(milvus-common REQUIRED)` in `internal/core/CMakeLists.txt` near the existing Conan `find_package(...)` block. - Add Linux-only `find_package(OpenBLAS CONFIG REQUIRED)` in `internal/core/CMakeLists.txt` so missing Conan OpenBLAS fails before Knowhere configures. - Remove `-DOpenBLAS_SOURCE=AUTO` from `internal/core/build.sh`; the requested Knowhere commit no longer consumes that flag and directly calls `find_package(OpenBLAS CONFIG REQUIRED)` on non-Apple platforms. 4. **Stop executing source FetchContent for `milvus-common`.** - Remove `add_subdirectory(milvus-common)` from `internal/core/thirdparty/CMakeLists.txt`. - Keep `add_subdirectory(knowhere)`. - Do not keep a compatibility alias to the old source target; all consumers should use the Conan target. - Leave `internal/core/thirdparty/milvus-common/` files in place for the first pass unless cleanup is explicitly desired; they will no longer be active once the subdirectory is removed. 5. **Pin Knowhere to the requested commit.** - Update `internal/core/thirdparty/knowhere/CMakeLists.txt`: - `KNOWHERE_VERSION` -> `0e53e57cd1f67f74decbdf9af42e74d0e3850155`. - Use the full SHA for deterministic FetchContent checkout. 6. **Migrate CMake links/includes from source target to Conan target.** - In `internal/core/src/CMakeLists.txt`: - Remove `${MILVUS_COMMON_INCLUDE_DIR}` from include directories. - Add `milvus-common::milvus-common` to `CONAN_TARGETS` so `milvus_conan_deps` propagates include directories to object libraries. - Remove plain `milvus-common` from `LINK_TARGETS`. - In `internal/core/unittest/CMakeLists.txt`: - Remove `${MILVUS_COMMON_INCLUDE_DIR}` from include directories. - Remove plain `milvus-common` from `all_tests` and `test_json_uint64` link lists; both already link `milvus_conan_deps`, which will carry `milvus-common::milvus-common`. 7. **Adjust build-time system OpenBLAS installs.** - Remove Linux build-time OpenBLAS package installs that can shadow or duplicate Conan OpenBLAS: - `scripts/install_deps.sh`: remove Ubuntu `libopenblas-dev`; remove Rocky/Amazon/CentOS `openblas-devel` where only needed for C++ core build. - `build/docker/builder/cpu/ubuntu20.04/Dockerfile`, `ubuntu22.04/Dockerfile`, `ubuntu24.04/Dockerfile`: remove `libopenblas-dev`. - `build/docker/builder/gpu/ubuntu20.04/Dockerfile`, `gpu/ubuntu22.04/Dockerfile`: remove `libopenblas-dev`. - `build/docker/builder/cpu/amazonlinux2023/Dockerfile`, `rockylinux9/Dockerfile`: remove `openblas-devel` and any OpenBLAS header symlink workaround that only supported system OpenBLAS. - `build/deb/build_deb.sh`: remove build-time `libopenblas-dev` install if no longer needed after Conan generation. - Keep Fortran/toolchain packages needed to build Conan OpenBLAS from source when binary packages are missing. - Do not change `scripts/install_deps_embd.sh` or `scripts/install_deps_msys.sh` in the first pass unless the target build path uses them; they are separate embedded/MSYS dependency paths and may still require manual/system OpenBLAS. - On macOS, remove `openblas` from `.github/workflows/mac.yaml` only after confirming the new Knowhere path uses Apple BLAS/LAPACK and Milvus has no other macOS OpenBLAS consumer. 8. **Adjust runtime/package OpenBLAS handling carefully.** - The runtime should still include `libopenblas.so*` if the Milvus binary or Conan-built Knowhere/Faiss links it dynamically. - Prefer copying the Conan-provided OpenBLAS shared library from `cmake_build/lib` or the Conan package output instead of installing distro development packages in runtime images. - Update packaging sites that currently copy or depend on system OpenBLAS: - `build/deb/build_deb.sh`: change the copied source from `/usr/lib/x86_64-linux-gnu/libopenblas.so.0` to the Conan/build output path if present. - `build/rpm/milvus.spec`: stop assuming `/usr/lib/libopenblas-r0.3.9.so`; use the packaged Conan/build output library. - Runtime Dockerfiles under `build/docker/milvus/**`: remove `libopenblas-dev`/`openblas-devel` only after confirming the image gets the Conan OpenBLAS `.so` with Milvus libs. - `build/docker/milvus/gpu/ubuntu20.04/Dockerfile.base`: review the explicit `milvusdb/openblas` stage; replace it with the Conan-produced library if that image path is still active. - Keep runtime `libgomp1`/`libgomp` and `libaio` if required by OpenMP/libaio-linked dependencies. 9. **Clean stale build state before verification.** - Use a fresh `cmake_build` or remove the old FetchContent checkout/cache before rebuilding; stale `MILVUS_COMMON_INCLUDE_DIR`, old `knowhere-src`, or system OpenBLAS paths can hide integration errors. Signed-off-by: yhmo <yihua.mo@zilliz.com>
Compile and install milvus cluster
Environment
OS: Ubuntu 20.04
go：1.21
cmake: >=3.18
gcc: >= 11
Install dependencies
Install compile dependencies
$ sudo apt install -y g++ gcc make libssl-dev zlib1g-dev libboost-regex-dev \
libboost-program-options-dev libboost-system-dev libboost-filesystem-dev \
libboost-serialization-dev python3-dev libboost-python-dev libcurl4-openssl-dev gfortran libtbb-dev
$ export GO111MODULE=on
$ go get github.com/golang/protobuf/protoc-gen-go@v1.3.2
OpenBLAS is managed by Conan for C++ core builds. Install the compiler toolchain and Fortran runtime needed by Conan, then run the normal third-party build step.
Compile
Generate the go files from proto file
$ make check-proto-product
Check code specifications
$ make verifiers
Compile milvus
$ make milvus
Install docker-compose
refer: https://docs.docker.com/compose/install/
$ sudo curl -L "https://github.com/docker/compose/releases/download/1.27.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
$ sudo chmod +x /usr/local/bin/docker-compose
$ sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose
$ docker-compose --version
$ docker compose --version
Start service
Start third-party service:
$ cd [milvus project path]/deployments/docker/cluster
$ docker-compose up -d
$ docker compose up -d
Start milvus cluster:
$ cd [milvus project path]
$ ./scripts/start_cluster.sh
Run unittest
Run all unittest including go and cpp cases:
$ make unittest
You also can run go unittest only:
$ make test-go
Run cpp unittest only:
$ make test-cpp
Run code coverage
Run code coverage including go and cpp:
$ make codecov
You also can run go code coverage only:
$ make codecov-go
Run cpp code coverage only:
$ make codecov-cpp