mirror of https://github.com/milvus-io/milvus.git synced 2026-07-21 02:05:41 +00:00

Files

T

History

grootandGitHub 965e4b1564 build(deps): Upgrade knowhere and manage milvus-common via conan (#49912 )

issue: https://github.com/milvus-io/milvus/issues/47425

# Plan: upgrade Milvus to Knowhere with Conan `milvus-common` and Conan
OpenBLAS

## Context

Milvus currently fetches both `knowhere` and `milvus-common` from source
with CMake `FetchContent`. The requested Knowhere commit
`0e53e57cd1f67f74decbdf9af42e74d0e3850155` expects `milvus-common` and
Linux OpenBLAS from Conan 2.x instead:

-
`milvus-common/1.0.0-9ca5ea6@milvus/dev#274d428d85f1d3d996e1092f0c9c7144`
- `find_package(milvus-common REQUIRED)` and target
`milvus-common::milvus-common`
- Linux `openblas/0.3.30` with CMake target `OpenBLAS::OpenBLAS`
- macOS Apple BLAS/LAPACK via `BLA_VENDOR Apple`, not Conan OpenBLAS

Milvus also currently installs system OpenBLAS in builder Dockerfiles,
runtime Dockerfiles, dependency scripts, and packaging scripts. The
intended outcome is one build-time OpenBLAS source on Linux: Conan.
Runtime packaging should use the Conan-built shared libraries instead of
relying on distro `libopenblas-dev`/`openblas-devel` where possible.

## Implementation

1. **Update Conan dependency pins in `internal/core/conanfile.py`.**
- Add direct requirement
`milvus-common/1.0.0-9ca5ea6@milvus/dev#274d428d85f1d3d996e1092f0c9c7144`.
   - Align dependencies required by Knowhere `0e53e57c`:
     - `folly/2026.04.20.00@milvus/dev#06852bea5b6449f0c4eb0df002b5779c`
     - `fmt/11.2.0#eb98daa559c7c59d591f4720dde4cd5c`
     - `lz4/1.10.0#982d9b673900f665a1da109e09c17cab`
     - `fast_float/8.0.0@milvus/dev#c7802833c74c5a86ffed70e4af1a795e`
     - Linux-only `openblas/0.3.30`
- Add `openblas/*:dynamic_arch = True` to default options to match
Knowhere.

2. **Move Milvus core build settings to C++20.**
- Change `internal/core/CMakeLists.txt` from `CMAKE_CXX_STANDARD 17` to
`20`.
- Keep `OPENTELEMETRY_STL_VERSION=2017` unchanged; it controls
OpenTelemetry ABI mode, not the language standard.
- Change all `compiler.cppstd=17` Conan install settings in
`scripts/3rdparty_build.sh` to `compiler.cppstd=20`.
- Review `internal/core/src/storage/gcp-native-storage/CMakeLists.txt`;
if it inherits incompatible flags or includes C++20-dependent headers,
change its local `CMAKE_CXX_STANDARD 17` to `20` too.

3. **Expose Conan packages to Milvus CMake.**
- Add `find_package(milvus-common REQUIRED)` in
`internal/core/CMakeLists.txt` near the existing Conan
`find_package(...)` block.
- Add Linux-only `find_package(OpenBLAS CONFIG REQUIRED)` in
`internal/core/CMakeLists.txt` so missing Conan OpenBLAS fails before
Knowhere configures.
- Remove `-DOpenBLAS_SOURCE=AUTO` from `internal/core/build.sh`; the
requested Knowhere commit no longer consumes that flag and directly
calls `find_package(OpenBLAS CONFIG REQUIRED)` on non-Apple platforms.

4. **Stop executing source FetchContent for `milvus-common`.**
- Remove `add_subdirectory(milvus-common)` from
`internal/core/thirdparty/CMakeLists.txt`.
   - Keep `add_subdirectory(knowhere)`.
- Do not keep a compatibility alias to the old source target; all
consumers should use the Conan target.
- Leave `internal/core/thirdparty/milvus-common/` files in place for the
first pass unless cleanup is explicitly desired; they will no longer be
active once the subdirectory is removed.

5. **Pin Knowhere to the requested commit.**
   - Update `internal/core/thirdparty/knowhere/CMakeLists.txt`:
     - `KNOWHERE_VERSION` -> `0e53e57cd1f67f74decbdf9af42e74d0e3850155`.
   - Use the full SHA for deterministic FetchContent checkout.

6. **Migrate CMake links/includes from source target to Conan target.**
   - In `internal/core/src/CMakeLists.txt`:
     - Remove `${MILVUS_COMMON_INCLUDE_DIR}` from include directories.
- Add `milvus-common::milvus-common` to `CONAN_TARGETS` so
`milvus_conan_deps` propagates include directories to object libraries.
     - Remove plain `milvus-common` from `LINK_TARGETS`.
   - In `internal/core/unittest/CMakeLists.txt`:
     - Remove `${MILVUS_COMMON_INCLUDE_DIR}` from include directories.
- Remove plain `milvus-common` from `all_tests` and `test_json_uint64`
link lists; both already link `milvus_conan_deps`, which will carry
`milvus-common::milvus-common`.

7. **Adjust build-time system OpenBLAS installs.**
- Remove Linux build-time OpenBLAS package installs that can shadow or
duplicate Conan OpenBLAS:
- `scripts/install_deps.sh`: remove Ubuntu `libopenblas-dev`; remove
Rocky/Amazon/CentOS `openblas-devel` where only needed for C++ core
build.
- `build/docker/builder/cpu/ubuntu20.04/Dockerfile`,
`ubuntu22.04/Dockerfile`, `ubuntu24.04/Dockerfile`: remove
`libopenblas-dev`.
- `build/docker/builder/gpu/ubuntu20.04/Dockerfile`,
`gpu/ubuntu22.04/Dockerfile`: remove `libopenblas-dev`.
- `build/docker/builder/cpu/amazonlinux2023/Dockerfile`,
`rockylinux9/Dockerfile`: remove `openblas-devel` and any OpenBLAS
header symlink workaround that only supported system OpenBLAS.
- `build/deb/build_deb.sh`: remove build-time `libopenblas-dev` install
if no longer needed after Conan generation.
- Keep Fortran/toolchain packages needed to build Conan OpenBLAS from
source when binary packages are missing.
- Do not change `scripts/install_deps_embd.sh` or
`scripts/install_deps_msys.sh` in the first pass unless the target build
path uses them; they are separate embedded/MSYS dependency paths and may
still require manual/system OpenBLAS.
- On macOS, remove `openblas` from `.github/workflows/mac.yaml` only
after confirming the new Knowhere path uses Apple BLAS/LAPACK and Milvus
has no other macOS OpenBLAS consumer.

8. **Adjust runtime/package OpenBLAS handling carefully.**
- The runtime should still include `libopenblas.so*` if the Milvus
binary or Conan-built Knowhere/Faiss links it dynamically.
- Prefer copying the Conan-provided OpenBLAS shared library from
`cmake_build/lib` or the Conan package output instead of installing
distro development packages in runtime images.
- Update packaging sites that currently copy or depend on system
OpenBLAS:
- `build/deb/build_deb.sh`: change the copied source from
`/usr/lib/x86_64-linux-gnu/libopenblas.so.0` to the Conan/build output
path if present.
- `build/rpm/milvus.spec`: stop assuming
`/usr/lib/libopenblas-r0.3.9.so`; use the packaged Conan/build output
library.
- Runtime Dockerfiles under `build/docker/milvus/**`: remove
`libopenblas-dev`/`openblas-devel` only after confirming the image gets
the Conan OpenBLAS `.so` with Milvus libs.
- `build/docker/milvus/gpu/ubuntu20.04/Dockerfile.base`: review the
explicit `milvusdb/openblas` stage; replace it with the Conan-produced
library if that image path is still active.
- Keep runtime `libgomp1`/`libgomp` and `libaio` if required by
OpenMP/libaio-linked dependencies.

9. **Clean stale build state before verification.**
- Use a fresh `cmake_build` or remove the old FetchContent
checkout/cache before rebuilding; stale `MILVUS_COMMON_INCLUDE_DIR`, old
`knowhere-src`, or system OpenBLAS paths can hide integration errors.

Signed-off-by: yhmo <yihua.mo@zilliz.com>

2026-06-03 14:48:19 +08:00

debian

fix: Add tzdata dependency to enable IANA Time Zone ID recognition (#45475 )

2025-11-11 14:43:37 +08:00

scripts

doc: [skip e2e] fix the build deb (#34751 )

2024-07-17 19:39:42 +08:00

build_deb.sh

build(deps): Upgrade knowhere and manage milvus-common via conan (#49912 )

2026-06-03 14:48:19 +08:00

README.md

doc: [skip e2e] fix the build deb (#34751 )

2024-07-17 19:39:42 +08:00

README.md

Building Deb package with Docker

Building Milvus Deb package is easy if you take advantage of the containerized build environment. This document will guide you through this build process.

Docker, using one of the following configurations:

Linux with local Docker Install Docker according to the instructions for your OS.

Get the opensource milvus code

git clone https://github.com/milvus-io/milvus.git
cp -r milvus/build/deb .
cd deb

Start the milvus container and build the deb package

# Replace the VERSION with your own
sudo docker run -v .:/deb -ti --entrypoint /bin/bash milvusdb/milvus:v$VERSION
# in the container
cd /deb
bash build_deb.sh $VERSION $VERSION $MAINTAINER $DEBEMAIL

Install the deb package on ubuntu system

sudo apt-get update
sudo dpkg -i milvus_$VERSION-1_amd64.deb # This package is in the milvus-deb directory
sudo apt-get -f install

Check the status of Milvus

sudo systemctl status milvus

Connect to Milvus

Please refer to Hello Milvus, then run the example code.

Uninstall Milvus

sudo dpkg -P milvus

(Optional) By default, Milvus is started in embed mode. If you rely on external etcd and minio, you can modify the following configuration and then restart Milvus.

sudo vim /etc/milvus/configs/milvus.yaml

etcd:
  endpoints: etcd-ip:2379
  ...
  use:
    embed: false
minio:
  address: minio-ip
  ...
common:
  storageType: remote