issue: #50615
## What this PR does
Makes the C++ core build cleanly on **macOS (Apple Silicon) with
Homebrew LLVM/clang 19**.
> [!IMPORTANT]
> **Scope / impact is NOT macOS-only.** Although this PR is framed as
macOS/clang-19 enablement, two of the changes are **top-level
`requires(...)` in `internal/core/conanfile.py`, so they apply to every
platform, including Linux production builds:**
> - **`azure-sdk-for-cpp` `1.11.3@milvus/dev` → `1.16.0@milvus/dev`** (5
minor versions, `force=True`). This is the SDK Arrow uses for the
**Azure Blob remote-storage path**, so this is effectively a major Azure
SDK bump on the **production Azure object-storage path for Linux
deployments** too.
> - **`arrow` recipe `17.0.0@milvus/dev-2.6#c743ea7a…` →
`17.0.0@milvus/dev#f411aa73…`** — same Arrow version, but **rebuilt
against azure 1.16.0** with a patched `FindAzure.cmake`. The Arrow
**`package_id` (recipe revision) DID change**; it is not the same binary
as before.
> - **`libbson/1.30.6@milvus/dev`** added (also global, but new — no
pre-existing behavior to regress).
>
> **Testing blind spot:** CI green proves compile/link/UT/e2e pass, but
the UT and integration suites **do not exercise real Azure Blob
read/write**, so the Azure object-storage path is **not covered** by
this PR's CI. A real Azure Blob remote-storage regression is recommended
before merge (see Verification).
### Why the dependency bumps are required for clang 19
- **azure 1.16.0**: azure 1.11.3 bundles an older `nlohmann/json` that
uses `std::char_traits<unsigned char>`, which was **removed in libc++
19** → 1.11.3 fails to compile on clang 19. 1.16.0 fixes this.
- **arrow rebuilt against azure 1.16.0**: when Arrow is built from
source on clang 19 it must link the same azure 1.16.0; the recipe also
patches `FindAzure.cmake` to the monolithic `Azure` config so
`find_package(Azure)` resolves. Hence a new Arrow recipe revision.
### Toolchain / build enablement
- **`scripts/setenv.sh`**: probe LLVM 19/20/21 first (libc++ 17/18
`chrono operator<<` clashes with Arrow's vendored `date`; libc++ 19
fixes it).
- **`scripts/3rdparty_build.sh`**: arm64 `-march=armv8-a+crypto+crc` so
folly's F14 (SimdAndCrc on Apple Silicon) matches core/knowhere — fixes
`F14LinkCheck` undefined-symbol link errors.
- **`InvertedIndexTantivy.cpp`**: drop spurious `->template` on
non-template members (clang 19
`-Wmissing-template-arg-list-after-template-kw`).
### Drop bsoncxx / libmongoc → libbson only
The BSON layer used the **bsoncxx** C++ driver, whose CMake
unconditionally builds mongo-cxx-driver → **libmongoc** → bundled
**utf8proc**, which fails to configure under newer CMake/clang on macOS.
Milvus only needs BSON (de)serialization.
- New Conan package **`libbson/1.30.6@milvus/dev`** (libbson-only,
`ENABLE_MONGOC=OFF`; recipe published to milvus conanfiles &
production).
- New **`src/common/bson_shim.h`** — a thin `milvus::bson` C++ layer
over libbson's C API.
- Migrated `bson_view.h`, `bson_builder.{h,cpp}`, json_stats consumers
and unit tests off bsoncxx.
- CMake: `find_package(bson-1.0)` + link `mongo::bson_shared`; removed
FetchContent `thirdparty/bsoncxx`.
All three global Conan requires (arrow / libbson / azure) pin an
explicit recipe `#revision` for reproducibility.
## Verification
- Full clean rebuild on macOS arm64 / clang 19: `bin/milvus` builds;
`libmilvus_core` links **only `libbson-1.0.dylib`** (no
libmongoc/mongocxx/utf8proc).
- libbson / arrow / azure resolve from production Conan at the pinned
revisions.
- All 21 BSON unit tests (BsonView / BsonBuilder / DomNode) pass.
- Full ci-v2 suite green on Linux (build / build-ut-cov / ut-cpp / ut-go
/ integration / e2e / go-sdk).
- **TODO before merge:** real Azure Blob remote-storage read/write
regression to cover the azure-1.16.0 / arrow-rebuild blast radius (not
exercised by UT/integration).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## What this PR does
Removes two unused C++ third-party dependencies from `internal/core` to
reduce dependency count, comprehension cost, and link/build surface.
### rapidjson
Linked into `milvus_core` but has **zero `#include` usage** in the C++
sources (only a stale comment in `common/Json.h`). knowhere v3.0.3 and
milvus-storage do not depend on it. Removes the conan requirement,
`find_package(RapidJSON)`, and the link target.
### OpenDAL backend
Experimental `ChunkManager` backend gated behind `USE_OPENDAL`
(**default OFF**) — released binaries never built it. Removes
`OpenDALChunkManager`, the vendored OpenDAL Rust dependency, and all
`USE_OPENDAL` build plumbing (Makefile / core_build.sh /
3rdparty_build.sh / CMake). The Go-side `"opendal"` storageType value
remains a backward-compatible alias to the standard remote chunk
manager.
### Kept
`simde` is **not** removed — knowhere v3.0.3 requires it
(`find_package(simde REQUIRED)`, used in
`src/index/sparse/codec/varintdecode.c`).
issue: #50551🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: xiaofanluan <xf@hjjaq.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
issue: #47420
## What this PR does
Project-wide migration of raw `fmt.Errorf` / `errors.New` in function
bodies onto
the `merr` framework, plus the Sys-vs-Input error classification and the
machinery it drives (retriability, fine-grained metrics, segcore
unification),
plus the convention docs and a linter that keeps it from regressing.
Scope: storage, proxy, coordinators (root/data/query), query node, data
node,
`pkg/util` & `internal/util`, expression parser, message queue,
streaming, and
misc packages. Bare raw-error usages went from ~3000 to a ~340 allowlist
(package-level sentinels / build-tag / test sites).
---
## How to review this PR
It is large but the vast majority is mechanical. Changes fall into three
tiers;
spend review budget on Part 2 and Part 3.
### Part 1 — Mechanical standardization (low risk, verify by rule)
Each converted call follows one of a small fixed set of rules. To
review, check
that each site obeys the matching rule rather than reading every line:
| Pattern | Rule |
|---|---|
| `fmt.Errorf("...")` originating a new error | →
`merr.WrapErrXxxMsg("...")` with a code matching the failure's meaning |
| Adding context to an existing typed error | → `merr.Wrap(err, "...")`
/ `merr.Wrapf(...)` — **preserves** the inner code (never `WrapErr*Err`,
which overwrites it) |
| Errors inside the streaming subsystem | → `status.New*` factories
(StreamingError), **not** merr — this is the component-internal dialect
(see `docs/dev/error_handling_guide.md`) |
| Low-level / control-flow signal caught by `errors.Is` | → kept as a
package-level `errors.New` sentinel (lowercase, same-package) |
Conventions are documented in `docs/dev/error_handling_guide.md`
(how-to) and
`docs/dev/error_sentinel_convention.md` (rules + audit). A
`gocritic`/`ruleguard`
rule (`rawmerrerror`, in `rules.go`) enforces "no raw `return
errors.New/fmt.Errorf`"
under `make verifiers`.
### Part 2 — Behavior changes (review these closely)
These are the sites where the wire contract or runtime behavior changes,
not just
the source text. Listed by category; representative locations given,
full set in
the diff.
**A. gRPC wire-code shifts: `UnexpectedError(1)/Code 65535` → typed
code.**
Where a handler previously returned a raw error (collapsed to
`Code=65535` on the
wire), it now returns a typed merr, so the client sees a real code. The
most
common shift is to `IllegalArgument(5)/Code 1100` (ParameterInvalid).
Touch
points include datanode task handlers (CreateTask/Query/Drop), proxy
Upsert,
querynode GetMetrics, datacoord CreateIndex, httpserver query-response
builder,
and typeutil schema validation. One code refinement: an index-param
validation
moved `1100` → `1101` (ParameterMissing). **Client/SDK assertions and
any code
that switched on `Code=65535` for these paths must be re-checked** (the
go_client
e2e assertions were already aligned in this PR).
**B. Prometheus `status` label contract change (externally visible).**
The proxy metric's coarse `fail` / `rejected` values are split into
`fail_input` / `fail_system` and `rejected_user` / `rejected_system` (in
`requestutil.ParseMetricLabel`; auth/privilege rejections count as
`rejected_user`), so dashboards can attribute a failure to caller vs
operator.
**Dashboards/alerts querying `status="fail"` must migrate to
`status=~"fail_.*"`, and `status="rejected"` to
`status=~"rejected_.*"`.** The
in-repo Grafana dashboard is already migrated; external dashboards built
on the
old values silently go empty after upgrade. This is the one change that
requires an ops-side migration.
**C. Retriability semantics.**
- C1: `merr.Status(err)` now forces `Retriable=false` when the error is
an
`InputError` — a malformed request can never succeed on blind retry, so
clients
never get the self-contradictory "your input is wrong but you may
retry".
- C2: `retry.Do` short-circuits an `InputError` (non-retriable) — **but
only when
the caller did not pass a `RetryErr` predicate**. The check is an
`if c.isRetryErr != nil { ... } else if InputError { ... }` *mutually
exclusive*
branch (`pkg/util/retry/retry.go`): an explicit `RetryErr` takes
precedence and
bypasses the InputError abort. `retry.Handle` deliberately does **not**
apply
the InputError abort (its callers signal abort via `shouldRetry=false`).
Four
flusher startup callsites that must retry through transient "not ready"
errors
were given explicit `RetryErr` escape hatches.
**D. segcore (C++→Go) error classification.**
A single shared Go-side table (`pkg/util/merr/segcore.go`) maps each
segcore code
to a merr sentinel + InputError/signal category, replacing scattered
hand-written
`if errorCode == ...` switches in the cgo wrappers. **Wire `Code` values
change
for every segcore pass-through error, not just the remapped ones.**
Named
sentinels remap (C++ `2003` → merr `2001`, `2033` → `2002`,
Folly/Knowhere codes
likewise); **all remaining pass-through codes (`2004`–`2043`, previously
surfaced to clients as raw C++ enum values) now serialize as `2000`**
(`ErrSegcore`), with the original C++ code preserved in the `Reason`
text
(`segcoreCode=...`); unknown/future codes collapse to `2000` as well
(pinned by
the `wire_code_projection` test). Transient segcore classes (object
storage /
file IO / OOM / mmap / FieldNotLoaded — 11 codes) now report
`Retriable=true`.
**Any client switching on raw segcore codes in the `2004`–`2043` range
must be
re-checked**; the in-Reason code remains available for diagnostics.
Signal
codes (PretendFinished / FollyCancel) are recognized centrally.
`errors.Is`-based
control flow on these (e.g. scheduler skip/retry) is preserved.
**E. InputError classification (25 sentinels + dynamic marks).**
25 sentinels in `errors.go` carry `WithErrorType(InputError)` (the
Collection /
ResourceGroup / Database families, `ErrIndexDuplicate`,
`ErrParameterInvalid`,
`ErrPrivilegeNotAuthenticated`, `ErrImportFailed`, `ErrQueryPlan`, ...),
plus dynamic
marks for the 8 segcore input codes (ExprInvalid, DimNotMatch,
MetricTypeInvalid, FieldIDInvalid, ...) and
`WrapErrAsInputError`. The widest blast radius is `ErrParameterInvalid`
(1100):
~2335 `WrapErrParameterInvalid*` callsites now classify as input /
non-retriable. Because of C1/C2 this changes retriability for
any path that returns these. **The audit to confirm no transient path
was
mis-marked is the single most important review item** (see Part 3). One
reverse
correction: storage field-stats parsing moved from `ErrParameterInvalid`
(input)
to `ErrDataIntegrity` — a corrupted stored stat is data corruption, not
user
input.
### Part 3 — Known risks & traps (called out proactively)
1. **`merr.Wrap` vs `WrapErr*Err` (code-masking).** `WrapErr*Err` builds
a
`wrappedMilvusError{sentinel: ErrServiceInternal}` whose `code()`
returns the
*outer* sentinel — it overwrites the inner typed code and hides the
`errors.Is`
chain. This is intentional (use it to *deliberately* downgrade), but it
was a
recurring conversion defect; the rule "add context with `merr.Wrap`,
downgrade
with `WrapErr*Err`" is enforced by convention and reviewed across the
diff.
2. **InputError × `retry.Do` blast radius.** Marking a sentinel
`InputError` makes
any `retry.Do(...)` without a `RetryErr` predicate stop retrying it.
Reviewers
should sanity-check that no transient use of the 19 newly-marked
sentinels
(especially `ErrParameterInvalid`) sits inside a retry loop that needed
to keep
spinning. The known flusher cases were handled (see C2).
3. **The ~340 raw-error allowlist.** What remains as bare `errors.New`
is, by
design: package-level sentinels (caught by `errors.Is`), `//go:build
test`
sites, and out-of-band trees (`cmd/`, `tests/`, codegen, walimpls). The
linter
only bans the *direct-return* form; assignment-then-return escapes and
the full
no-exceptions ban are deferred to an AST-based linter (Tier 2,
documented).
4. **segcore C++ second step deferred.** This PR unifies classification
on the Go
side; splitting the dual-semantic C++ codes at the source is a
follow-up.
---
## Validation
- `make verifiers`: Go side clean (gofmt + static-check across modules,
including
the new `rawmerrerror` rule with a 0-hit baseline repo-wide).
- `make test-go`: passing; the one real regression introduced (a
datanode
`invalid_task_type` assertion shifting `1` → `5` from a ParameterInvalid
conversion) was fixed in-tree.
- go_client e2e CreateIndex assertions aligned to the new merr messages.
---------
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
issue: https://github.com/milvus-io/milvus/issues/47425
Milvus already asks Conan to build the C++ third-party stack with C++20
for the host profile, but Conan keeps the build profile separate. That
means packages used during the build can still fall back to the default
C++ standard, which caused protobuf and abseil to be compiled as gnu17
in some environments and fail during the dependency build.
This change passes C++20 to the Conan build profile as well, so both the
host context and the build context use the same language level. It
applies the setting to both Linux Conan install branches in the
third-party build script, including the gcc4 ABI path and the normal
libstdc++11 path.
Signed-off-by: jiaqizho <jiaqi.zhou@zilliz.com>
Co-authored-by: EC2 Default User <ec2-user@ip-10-15-6-84.us-west-2.compute.internal>
issue: https://github.com/milvus-io/milvus/issues/47425
# Plan: upgrade Milvus to Knowhere with Conan `milvus-common` and Conan
OpenBLAS
## Context
Milvus currently fetches both `knowhere` and `milvus-common` from source
with CMake `FetchContent`. The requested Knowhere commit
`0e53e57cd1f67f74decbdf9af42e74d0e3850155` expects `milvus-common` and
Linux OpenBLAS from Conan 2.x instead:
-
`milvus-common/1.0.0-9ca5ea6@milvus/dev#274d428d85f1d3d996e1092f0c9c7144`
- `find_package(milvus-common REQUIRED)` and target
`milvus-common::milvus-common`
- Linux `openblas/0.3.30` with CMake target `OpenBLAS::OpenBLAS`
- macOS Apple BLAS/LAPACK via `BLA_VENDOR Apple`, not Conan OpenBLAS
Milvus also currently installs system OpenBLAS in builder Dockerfiles,
runtime Dockerfiles, dependency scripts, and packaging scripts. The
intended outcome is one build-time OpenBLAS source on Linux: Conan.
Runtime packaging should use the Conan-built shared libraries instead of
relying on distro `libopenblas-dev`/`openblas-devel` where possible.
## Implementation
1. **Update Conan dependency pins in `internal/core/conanfile.py`.**
- Add direct requirement
`milvus-common/1.0.0-9ca5ea6@milvus/dev#274d428d85f1d3d996e1092f0c9c7144`.
- Align dependencies required by Knowhere `0e53e57c`:
- `folly/2026.04.20.00@milvus/dev#06852bea5b6449f0c4eb0df002b5779c`
- `fmt/11.2.0#eb98daa559c7c59d591f4720dde4cd5c`
- `lz4/1.10.0#982d9b673900f665a1da109e09c17cab`
- `fast_float/8.0.0@milvus/dev#c7802833c74c5a86ffed70e4af1a795e`
- Linux-only `openblas/0.3.30`
- Add `openblas/*:dynamic_arch = True` to default options to match
Knowhere.
2. **Move Milvus core build settings to C++20.**
- Change `internal/core/CMakeLists.txt` from `CMAKE_CXX_STANDARD 17` to
`20`.
- Keep `OPENTELEMETRY_STL_VERSION=2017` unchanged; it controls
OpenTelemetry ABI mode, not the language standard.
- Change all `compiler.cppstd=17` Conan install settings in
`scripts/3rdparty_build.sh` to `compiler.cppstd=20`.
- Review `internal/core/src/storage/gcp-native-storage/CMakeLists.txt`;
if it inherits incompatible flags or includes C++20-dependent headers,
change its local `CMAKE_CXX_STANDARD 17` to `20` too.
3. **Expose Conan packages to Milvus CMake.**
- Add `find_package(milvus-common REQUIRED)` in
`internal/core/CMakeLists.txt` near the existing Conan
`find_package(...)` block.
- Add Linux-only `find_package(OpenBLAS CONFIG REQUIRED)` in
`internal/core/CMakeLists.txt` so missing Conan OpenBLAS fails before
Knowhere configures.
- Remove `-DOpenBLAS_SOURCE=AUTO` from `internal/core/build.sh`; the
requested Knowhere commit no longer consumes that flag and directly
calls `find_package(OpenBLAS CONFIG REQUIRED)` on non-Apple platforms.
4. **Stop executing source FetchContent for `milvus-common`.**
- Remove `add_subdirectory(milvus-common)` from
`internal/core/thirdparty/CMakeLists.txt`.
- Keep `add_subdirectory(knowhere)`.
- Do not keep a compatibility alias to the old source target; all
consumers should use the Conan target.
- Leave `internal/core/thirdparty/milvus-common/` files in place for the
first pass unless cleanup is explicitly desired; they will no longer be
active once the subdirectory is removed.
5. **Pin Knowhere to the requested commit.**
- Update `internal/core/thirdparty/knowhere/CMakeLists.txt`:
- `KNOWHERE_VERSION` -> `0e53e57cd1f67f74decbdf9af42e74d0e3850155`.
- Use the full SHA for deterministic FetchContent checkout.
6. **Migrate CMake links/includes from source target to Conan target.**
- In `internal/core/src/CMakeLists.txt`:
- Remove `${MILVUS_COMMON_INCLUDE_DIR}` from include directories.
- Add `milvus-common::milvus-common` to `CONAN_TARGETS` so
`milvus_conan_deps` propagates include directories to object libraries.
- Remove plain `milvus-common` from `LINK_TARGETS`.
- In `internal/core/unittest/CMakeLists.txt`:
- Remove `${MILVUS_COMMON_INCLUDE_DIR}` from include directories.
- Remove plain `milvus-common` from `all_tests` and `test_json_uint64`
link lists; both already link `milvus_conan_deps`, which will carry
`milvus-common::milvus-common`.
7. **Adjust build-time system OpenBLAS installs.**
- Remove Linux build-time OpenBLAS package installs that can shadow or
duplicate Conan OpenBLAS:
- `scripts/install_deps.sh`: remove Ubuntu `libopenblas-dev`; remove
Rocky/Amazon/CentOS `openblas-devel` where only needed for C++ core
build.
- `build/docker/builder/cpu/ubuntu20.04/Dockerfile`,
`ubuntu22.04/Dockerfile`, `ubuntu24.04/Dockerfile`: remove
`libopenblas-dev`.
- `build/docker/builder/gpu/ubuntu20.04/Dockerfile`,
`gpu/ubuntu22.04/Dockerfile`: remove `libopenblas-dev`.
- `build/docker/builder/cpu/amazonlinux2023/Dockerfile`,
`rockylinux9/Dockerfile`: remove `openblas-devel` and any OpenBLAS
header symlink workaround that only supported system OpenBLAS.
- `build/deb/build_deb.sh`: remove build-time `libopenblas-dev` install
if no longer needed after Conan generation.
- Keep Fortran/toolchain packages needed to build Conan OpenBLAS from
source when binary packages are missing.
- Do not change `scripts/install_deps_embd.sh` or
`scripts/install_deps_msys.sh` in the first pass unless the target build
path uses them; they are separate embedded/MSYS dependency paths and may
still require manual/system OpenBLAS.
- On macOS, remove `openblas` from `.github/workflows/mac.yaml` only
after confirming the new Knowhere path uses Apple BLAS/LAPACK and Milvus
has no other macOS OpenBLAS consumer.
8. **Adjust runtime/package OpenBLAS handling carefully.**
- The runtime should still include `libopenblas.so*` if the Milvus
binary or Conan-built Knowhere/Faiss links it dynamically.
- Prefer copying the Conan-provided OpenBLAS shared library from
`cmake_build/lib` or the Conan package output instead of installing
distro development packages in runtime images.
- Update packaging sites that currently copy or depend on system
OpenBLAS:
- `build/deb/build_deb.sh`: change the copied source from
`/usr/lib/x86_64-linux-gnu/libopenblas.so.0` to the Conan/build output
path if present.
- `build/rpm/milvus.spec`: stop assuming
`/usr/lib/libopenblas-r0.3.9.so`; use the packaged Conan/build output
library.
- Runtime Dockerfiles under `build/docker/milvus/**`: remove
`libopenblas-dev`/`openblas-devel` only after confirming the image gets
the Conan OpenBLAS `.so` with Milvus libs.
- `build/docker/milvus/gpu/ubuntu20.04/Dockerfile.base`: review the
explicit `milvusdb/openblas` stage; replace it with the Conan-produced
library if that image path is still active.
- Keep runtime `libgomp1`/`libgomp` and `libaio` if required by
OpenMP/libaio-linked dependencies.
9. **Clean stale build state before verification.**
- Use a fresh `cmake_build` or remove the old FetchContent
checkout/cache before rebuilding; stale `MILVUS_COMMON_INCLUDE_DIR`, old
`knowhere-src`, or system OpenBLAS paths can hide integration errors.
Signed-off-by: yhmo <yihua.mo@zilliz.com>
## What this PR does
Adds `commit_timestamp` (uint64) to `SegmentInfo` and `SegmentLoadInfo`
so that import/CDC segments use their logical commit time — not raw row
timestamps — for all temporal decisions.
**Problem:** When bulk-insert imports rows, the row timestamps generated
during the import process may be slightly older than the actual commit
time. Every time-based check in the system (MVCC snapshot visibility, GC
eligibility, collection-level TTL compaction triggering, delete-buffer
anchoring) sees these outdated timestamps instead of the actual commit
time, producing correctness bugs.
**Solution:**
Two-layer approach:
1. **C++ segcore (load-time overwrite):** During `LoadFieldData`, the
in-memory timestamp column is overwritten to `commit_ts_` when
`commit_ts_ != 0`. This makes existing MVCC/delete logic work correctly
with zero hot-path changes.
2. **Compaction normalization:** During compaction, all compaction paths
(mix/sort/clustering) rewrite row timestamps to `commit_ts` in output
binlogs, then set `CommitTimestamp = 0` on the output segment. After
first compaction, the segment becomes a normal segment with no special
handling needed. `commit_ts` is a temporary state that only exists
between import and first compaction.
**Key design decisions:**
- **TTL field (per-row):** Not affected by commit_ts. TTL field values
represent user-specified expiration intent and are honored as-is.
- **Collection-level TTL:** Uses `max(row_ts, commit_ts)` as the
effective row age to prevent premature expiration.
- **Delete/Upsert before commit_ts:** A delete/upsert with `ts <
commit_ts` does NOT take effect, because the row did not exist at that
time. The original `search_pk(pk, delete_ts)` logic handles this
correctly since `row_ts` is overwritten to `commit_ts`.
- **CommitTimestamp assignment:** Currently TODO in `import_checker.go`,
pending companion PR for 2PC commit flow.
## Changes
- `pkg/proto/data_coord.proto`, `query_coord.proto`, `segcore.proto`:
add `commit_timestamp` field
- `internal/datacoord/segment_info.go`:
`segmentEffectiveTs`/`segmentEffectiveDmlTs`/`effectiveTimestamp`
helpers
- `internal/datacoord/meta.go`: compaction completion mutations set
`CommitTimestamp=0` and normalize position timestamps via
`normalizePositionTimestamp` helper
- `internal/datacoord/compaction_trigger.go`: use `effectiveTimestamp`
for TTL compaction trigger
- `internal/datacoord/handler.go`, `garbage_collector.go`,
`compaction_task_l0.go`: use effective timestamp helpers
- `internal/querycoordv2`: propagate `CommitTimestamp` in
`PackSegmentLoadInfo`
- `internal/querynodev2/delegator`: use `segmentEffectiveTs` for
delete-buffer pin/list anchoring
- `internal/core/src/segcore/ChunkedSegmentSealedImpl`: overwrite
timestamp column at load time when `commit_ts_ != 0`
- `internal/datanode/compactor/timestamp_overwrite.go`: Record/Reader
wrappers for compaction timestamp normalization
-
`internal/datanode/compactor/{mix,merge_sort,sort,clustering}_compactor.go`:
apply timestamp overwrite during compaction
## Tests
- Unit tests: DataCoord helpers, `UpdateCommitTimestamp` operator,
`GenSnapshot` filter, GC eligibility, TTL compaction trigger
- Unit tests: QueryNode `segmentEffectiveTs`, delete-buffer pin at
commit_ts
- Unit tests: `timestamp_overwrite.go` Record/Reader wrappers
- C++ tests: MVCC gate, TTL gate, pre-commit delete not applied, normal
segment unchanged
- Integration tests: MVCC visibility, delete/upsert on import segment,
delete/upsert before commit_ts (should not apply), compaction normalizes
commit_ts to 0
## issue
issue: #48471
design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260324-commit-timestamp.md
---------
Signed-off-by: Yihao Dai <yihao.dai@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
issue: #49977
## Summary
AWS SDK C++ 1.11.692 (bumped from 1.11.352 in #47810) changed two
defaults that together break PutObject against non-AWS S3-compatible
storage:
1. `PutObjectRequest::GetChecksumAlgorithmName()` returns `"crc64nvme"`
by default
2. `ClientConfiguration::checksumConfig::requestChecksumCalculation`
defaults to `WHEN_SUPPORTED`
`ChecksumInterceptor` then attaches a request hash to every PutObject,
and the V4 signer switches the request to `Transfer-Encoding:
aws-chunked` + `x-amz-content-sha256:
STREAMING-UNSIGNED-PAYLOAD-TRAILER`. Aliyun OSS, Tencent COS, Huawei
OBS, and GCP all reject this streaming form.
This makes any `MinioChunkManager::PutObjectBuffer` call fail in an
infinite `JobStateRetry` loop — json_stats / bson_inverted
shared_key_index builds and any non-DiskANN-style index upload (IVF_*,
HNSW, FLAT) are affected.
## Fix
Restrict `requestChecksumCalculation` to `WHEN_REQUIRED` on non-AWS
providers so the SDK only adds checksums when the operation model
demands them (PutObject does not). Same approach as
milvus-io/milvus-storage#500.
Patched in both client construction paths:
- `MinioChunkManager::MinioChunkManager` (storage_type=minio,
address-based dispatch) — gated on `storageType != S3`
- `ChunkManager.cpp` subclasses (storage_type=remote,
cloud_provider-based dispatch) — applies in `GcpChunkManager`,
`AliyunChunkManager`, `TencentCloudChunkManager`,
`HuaweiCloudChunkManager`. `AwsChunkManager` keeps the default
`WHEN_SUPPORTED`.
CARDINAL_TIERED / DiskANN indexes already work because they upload via
Arrow's `S3FileSystem` provided by milvus-storage, which was fixed in
#500.
## Test plan
- [x] Added `TEST(MinioChecksumConfig, OverridesAreWhenRequired)` in
`test_storage.cpp` — asserts the override flips both directions from
`WHEN_SUPPORTED` to `WHEN_REQUIRED`
- [ ] Build / CI green
- [ ] Manual smoke test on Aliyun OSS endpoint — index build (IVF_SQ8)
and json_stats no longer fail with `x-oss-ec=0017-00000804`. CI cannot
regression-detect this since MinIO doesn't reject `aws-chunked`; a real
Aliyun OSS endpoint is required.
## Workaround (no code change)
Setting `AWS_REQUEST_CHECKSUM_CALCULATION=when_required` and
`AWS_RESPONSE_CHECKSUM_VALIDATION=when_required` on milvus pods also
fixes this — the AWS SDK reads them at `ClientConfiguration`
default-construct time, so it works across all bundled SDK copies
without rebuilding. Useful for hotfix on running clusters until this PR
ships.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
PR #49325 introduced collection-rooted index storage and also added a
precomputed `index_store_path` string flowing through the build path.
This PR scopes that back down: the collection-rooted layout stays, but
the precomputed `index_store_path` string is removed from the
write/build path.
Build/write path:
- DataCoord sends `IndexStorePathVersion` with the build job.
- DataNode forwards the version to C++ indexbuilder.
- C++ `FileManager` assembles the index object prefix locally from
`(path_version, collID, partID, segID, buildID, indexVersion)`.
Read/load path:
- Go keeps passing full `IndexFilePaths` through DataCoord -> QueryCoord
-> QueryNode -> cgo/segcore.
- Load-side protos stay on full paths; no `index_file_keys` /
`index_store_path_version` split is added there.
- C++ load keeps PR #49325's direct full-path open behavior for
`index_files/...` and `index_v1/...` paths.
The v1 root is intentionally named `index_v1` before release. v0 remains
`index_files`.
issue: #49086
## Compatibility note
This PR is wire-incompatible with the short-lived #49325-era internal
schema that carried `index_store_path`.
That is intentional and acceptable because no release has shipped
between the merge of #49325 and this rework. In other words, there are
no released binaries or released persisted wire data that rely on the
#49325-era `index_store_path` field semantics. This PR treats the proto
changes as a pre-release schema correction rather than a compatibility
migration.
Because of that:
- `BuildIndexInfo.index_store_path = 18` is replaced by
`index_store_path_version = 18`.
- `CreateJobRequest.index_store_path = 21` is removed.
- `CreateJobRequest.index_store_path_version = 36` is kept.
- We do not reserve the removed field numbers/names in this PR; if this
schema had crossed a release boundary, we would reserve them and use
fresh field numbers instead.
## What changes
**Protos**
- `worker.proto`: drop `CreateJobRequest.index_store_path`; keep
`CreateJobRequest.index_store_path_version`.
- `index_cgo_msg.proto`: replace `BuildIndexInfo.index_store_path` with
`BuildIndexInfo.index_store_path_version`.
- Read/load-side protos keep their full-path shape: no `IndexFilePaths
-> IndexFileKeys` rename and no load-side `index_store_path_version`
propagation.
**C++**
- `storage::IndexMeta`: replace `std::string index_store_path` with
`IndexStorePathVersion index_store_path_version`.
- `FileManager::GetRemoteIndexObjectPrefix()`: assemble v0/v1 prefixes
from path version and IDs.
- `FileManager::OpenInputStream()`: keep direct full-path open support
for read/load paths.
- `indexbuilder/index_c.cpp`: pass `index_store_path_version` into
`IndexMeta`.
- `DiskFileManagerTest`: cover v0 and v1 prefix assembly.
**Go**
- DataCoord `task_index.go`: stop building/sending `IndexStorePath`;
continue sending `IndexStorePathVersion`.
- DataNode `index/task_index.go`: forward `IndexStorePathVersion` to C++
build info.
- DataNode `index_services.go`: drop logs for the removed
`index_store_path` field.
- DataCoord FinishTask validation and
`SegmentIndex.IndexStorePathVersion` metadata remain unchanged.
- GC/copy/import/snapshot/read path keep full-path behavior; only
expected v1 root strings are updated from `index_files_v1` to
`index_v1`.
## Notes for reviewers
- The read/load path intentionally remains full-path based. QueryNode
should not convert full paths to basenames or parse path version for
load.
- `IndexFilePathInfo.IndexStorePathVersion` remains as DataCoord
metadata, but QueryCoord does not forward it for loading.
- `metautil.IndexPathBuilder` remains needed for Go-side object storage
operations such as GC, snapshot, copy-segment, and DataCoord
`GetIndexInfos` path construction.
- Mixed-version concerns are limited to unreleased #49325-era binaries,
which is why this PR can remove/retype `index_store_path` without
compatibility scaffolding.
## Test plan
- [x] C++ build: `ninja milvus_core` clean.
- [x]
`DiskAnnFileManagerTest.GetRemoteIndexObjectPrefix_{V0BuildRooted,V1CollectionRooted}`
covers v0/v1 prefix assembly.
- [x] Updated Go/C++ tests for the `index_v1` root rename.
- [ ] CI on this PR: code-check, UT, E2E.
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
this PR allows to create Sort/Bitmap/Hybrid index for json path index,
and also removed the `JsonInvertedIndex` class, to use
`InvertedIndexTantivy` directly for Inverted index type.
C++ changes:
- Add ConvertJsonToTypedFieldData<T>() to extract typed values from JSON
field data with separate tracking of non_exist_offsets for EXISTS
semantics
- Add JsonScalarIndexWrapper<T, BaseIndex> template for Sort/Bitmap with
dual file-manager pattern (original JSON schema for reading, cast-type
schema for base index dispatching)
- Add JsonHybridScalarIndex<T> with validity-aware cardinality counting
- Add IndexBase::Exists() virtual method; override in Sort/Bitmap/Hybrid
wrappers using non_exist_offsets (serialized via
WriteEntries/LoadEntries)
- Simplify ExistsExpr to use index->Exists() uniformly
- Extend IndexFactory::CreateJsonIndex() to route STL_SORT/BITMAP/HYBRID
Go changes:
- Update STL_SORT/Bitmap/Hybrid checkers to accept JSON with cast_type
and json_path validation
- Change AUTOINDEX default for JSON from INVERTED to HYBRID
- Bump ScalarIndexEngineVersion to 4
- Add version gate in DataCoord CreateIndex and snapshot RestoreIndexes
- Update test fixtures to cover new (index_type, cast_type) combinations
<img width="2380" height="708" alt="image"
src="https://github.com/user-attachments/assets/8b3923a0-2cb3-4af7-b73a-b76d1d1ec2d0"
/>
issue: https://github.com/milvus-io/milvus/issues/48954
design-doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260410-json_path_index_multi_type.md
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
Related issue: #48145
scripts/install_deps.sh fails to install Conan on Ubuntu 24.04 in two
scenarios:
1. System Python (PEP 668): `pip install --user conan` fails with
`error: externally-managed-environment`
2. Active virtual environment: `pip install --user conan` fails with
`ERROR: Can not perform a '--user' install. User site-packages are not
visible in this virtualenv.`
Fall back to an isolated venv (~/.venv/conan) with a ~/.local/bin/conan
symlink when --user install is unavailable. Behavior on Ubuntu
20.04/22.04, macOS, and Rocky Linux is unchanged.
Signed-off-by: Vikhlyantsev Konstantin <vihlancevk@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Related to #49398
Bump Go module references from pkg/v2 and milvus-proto/go-api/v2 to
pkg/v3 and milvus-proto/go-api/v3 so the client tracks the Milvus 3.x
release line.
This prepares the repository for the upcoming 3.x.y release by aligning
imports, module dependencies, and proto API references with the new
major-version module paths.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
## Summary
Add `scripts/milvus-debug.sh` to automate GDB coredump analysis using
the `-debug` image that accompanies each Milvus release starting from
v2.6.14.
## Background
Starting with v2.6.14, each Milvus release publishes two image variants:
| Tag | Content | Size | Use |
|-----|---------|------|-----|
| `milvusdb/milvus:<tag>` | **stripped** (production) | ~1/3 |
Production deployment |
| `milvusdb/milvus:<tag>-debug` | unstripped (full symbols) | full | GDB
coredump analysis |
The stripped production image is **byte-identical** to the debug image
in its code sections (`strip --strip-debug` removes only `.debug_*`
sections and symbol tables). So GDB can use the debug image's symbols to
resolve addresses in a coredump produced by the stripped image.
## What the script does
1. Pulls the debug image
2. Mounts the coredump file read-only into a container
3. Installs `gdb` on demand (not present in the runtime image by
default)
4. Launches GDB with `solib-search-path=/milvus/lib` so symbols from all
shared libraries (`libknowhere.so`, `libtantivy-binding.so`, etc.)
resolve correctly
5. Auto-prints the initial backtrace
## Typical usage after a production crash
```bash
# 1. Copy the coredump out of the crashed pod
kubectl cp <ns>/<pod>:/tmp/cores/core.<pid> ./core.<pid>
# 2. Run this script (GDB starts automatically with backtrace)
./scripts/milvus-debug.sh ./core.<pid> \
milvusdb/milvus:v2.6.14 milvusdb/milvus:v2.6.14-debug
```
## Test plan
- [x] `bash -n scripts/milvus-debug.sh` (syntax check passes)
- [x] Mirrors the documented procedure in the
strip-image-coredump-debug-guide Feishu doc
- [ ] Manual validation on a real coredump (waiting for v2.6.14
production crash; can be exercised with any coredump for now)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Zhikun Yao <zhikun.yao@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
- Preserve generated proto output mtimes when regenerated content is
unchanged.
- Back up generated files under `cmake_build/.proto-backup.*` during
`generate_proto.sh`.
- Avoid unnecessary C++ incremental rebuilds caused by unchanged `.pb.h`
/ `.pb.cc` timestamp churn.
issue: #49406Closes#49406
## Test plan
- [x] `bash -n scripts/generate_proto.sh`
- [x] `git diff --check master...HEAD`
- [ ] `make milvus-gpu` twice and confirm unchanged proto outputs no
longer trigger proto-dependent C++ rebuilds
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Signed-off-by: marcelo-cjl <marcelo.chen@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
issue: [#45691](https://github.com/milvus-io/milvus/issues/45691)
## Summary
Bump rust toolchain `1.89 → 1.92` only in installation paths
(Dockerfiles + install scripts).
## Why
milvus-storage HEAD upgrades `iceberg` crate `0.8 → 0.9`, which declares
`edition = "2024"` and `rust-version = "1.92"`. The new MSRV requires
the builder image to ship rust 1.92 before milvus can take that
milvus-storage commit.
This PR is the **first step**:
1. Bump rust toolchain in installation scripts and Dockerfiles.
2. Once merged, builder image is republished with rust 1.92.
3. Follow-up PR will bump `.env` `DATE_VERSION` to the new tag, switch
`cargo +1.89` → `cargo +1.92` in
`internal/core/thirdparty/{tantivy,opendal}/CMakeLists.txt`, and bump
milvus-storage to the iceberg-0.9 commit.
This split avoids the chicken-egg problem: changing the cargo pin or
milvus-storage version in this PR would fail CI, since the currently
pinned builder image only ships rust 1.89.
## Files changed
| File | Change |
|---|---|
| `scripts/install_deps.sh` | `RUST_VERSION="1.89"` → `1.92` |
| `scripts/3rdparty_build.sh` | rust toolchain `1.89` → `1.92` (mac +
linux paths) |
| `build/docker/builder/{cpu,gpu}/*/Dockerfile` (7 files) | rustup
`--default-toolchain=1.89` → `1.92` |
| `DEVELOPMENT.md` | doc table rust version |
## Test plan
- [ ] CI passes on existing builder image (no behaviour change for
current build path)
- [ ] After merge, builder image is republished with rust 1.92
- [ ] Follow-up PR can then bump `.env` and milvus-storage
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Fixes#49106.
Follow-up to #48165. Adds an opt-in `CONAN_CMD` environment variable so
developers working on both master (Conan 2.x) and release-2.5/2.6
(Conan 1.x) on the same machine can select a Conan binary per-build
without switching their default `conan`. See commit message for details.
## Test plan
- [x] `make` with default conan (2.x) — unchanged behavior
- [x] `CONAN_CMD=conan-1 make` — fails with clear version mismatch error
- [x] `CONAN_CMD=/nonexistent make` — fails with clear error
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
## Summary
- Skip `AzureChunkManagerTest` via TCP probe when Azurite emulator is
not running locally
- Remove Linux-specific `read()` single-call size limit assertion from
`UtilIndex.ReadFromFD`
- Fix `IterativeFilter.SealedIndex` platform-dependent RNG: explicitly
construct int16 column with guaranteed coverage then shuffle
- Skip faiss mmap tests on macOS with `#ifdef __APPLE__` —
`MmappedFileMappingOwner` is not implemented in faiss upstream for Apple
platforms (`mapped_io.cpp` only covers Linux/FreeBSD and Windows)
After these fixes: **6745 passed, 25 skipped (all with documented
reasons), 0 failed** on macOS ARM64.
issue: #48740
## Test plan
- [ ] Run `internal/core/output/unittest/all_tests` on macOS ARM64 —
expect 0 FAILED
- [ ] Run `internal/core/output/unittest/all_tests` on Linux — expect no
regressions (Azure/mmap skips are guarded by `__APPLE__` or runtime
checks)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: james <james@mini.local>
Co-authored-by: james <james@mini.local>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
issue: #48574
## Summary
- Upgrade Go from 1.24.12 to 1.25.8 across all go.mod files and
Dockerfiles
- Fixes CVE-2025-68121 (CRITICAL), CVE-2026-27142 (HIGH), CVE-2026-25679
(HIGH) in Go stdlib
- All three CVEs affect the Go standard library and are resolved by
upgrading to Go 1.25.8
## Changes
- Updated `go` directive in go.mod files (root, pkg/, client/,
tests/go_client/, examples/)
- Updated Go download URLs in Dockerfiles (build/docker/builder/)
- Updated `toolchain` directives where present
issue: https://github.com/milvus-io/milvus/issues/TBD
## Test plan
- [x] `make milvus` builds successfully with Go 1.25.8
- [ ] CI passes
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Li Liu <li.liu@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Layer 2 (OpenSSL FIPS) changes:
- Add programmatic FIPS activation via OSSL_LIB_CTX_load_config in
boring_enabled.go (gated by //go:build boringcrypto)
- Add openssl-fips.cnf with fips + default providers and
default_properties = fips=yes
- Use absolute .include path for fipsmodule.cnf — OpenSSL resolves
relative .include from the process working directory, not the config
file's directory, causing silent FIPS provider load failure
- Add RAND_bytes probe after config load to verify the FIPS provider is
truly functional (EVP_default_properties_is_fips_enabled only checks the
property string, not whether the provider loaded)
- Dockerfiles: add openssl fipsinstall + OPENSSL_MODULES env var
- Log OpenSSL FIPS status from C++ via
EVP_default_properties_is_fips_enabled
Layer 1 (Go BoringCrypto) changes:
- Add GOEXPERIMENT=boringcrypto build flag (conditional on
MILVUS_FIPS_ENABLED=ON)
- Add boringEnabled() build-tagged functions for startup logging
s2n-tls upgrade:
- Override s2n 1.4.1 (from aws-c-io) to 1.6.0 in conanfile.py. s2n 1.4.1
only detects FIPS via the legacy OPENSSL_FIPS define (not set by OpenSSL
3.x). s2n 1.6.0 adds EVP_default_properties_is_fips_enabled() detection
so s2n enters FIPS mode and uses RAND_bytes() through the FIPS provider.
See also: #48202, #48301
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
Switch CI pipelines from Release (`-O3`) to RelWithDebInfo (`-O2 -g
-DNDEBUG`), reducing peak compiler memory by ~15%. RelWithDebInfo is a
standard CMake build type that uses `-O2` natively and includes debug
symbols for better crash diagnostics in CI. Production image builds
remain at Release (`-O3`).
## Changes
- Update Jenkins (PR, PR-Arm, PR-for-go-sdk, UT-CPP) to use
`mode=RelWithDebInfo`
- Update GitHub Actions (`main.yaml`) to use `mode=RelWithDebInfo`
- Allow all standard CMake build types in `3rdparty_build.sh` validation
- Add gcc>=13 implicit include workaround in `CMakeLists.txt`
## Test plan
- [ ] CI builds with `mode=RelWithDebInfo` use `-O2` instead of `-O3`
- [ ] Default builds still use Release with `-O3`
- [ ] ASAN CI jobs pass with reduced memory usage
issue: #48400
Signed-off-by: xiaofan-luan <xiaofan.xl@zilliz.com>
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Summary
- macOS `ld` treats each `-r` flag as a single directory path, unlike
Linux `ld` which accepts colon-separated paths
- Since d69bdd288c added `cmake_build/lib` to `DYLD_LIBRARY_PATH`,
`RPATH=$DYLD_LIBRARY_PATH` produces a colon-separated value like
`path1:path2`
- The Makefile's `-r $RPATH` embeds this as a single invalid path in the
binary, causing `Library not loaded: @rpath/libmilvus-storage.dylib` at
runtime
- Fix: emit two separate `-r` flags so the Makefile expands to `-r path1
-r path2`
## Test plan
- [x] Verified `RPATH` variable output: `path1 -r path2` (correct)
- [x] Verified binary loads correctly after fix (`otool -l` shows two
separate `LC_RPATH` entries)
- [x] Verified Milvus standalone starts and serves requests successfully
issue: #47809
Signed-off-by: sijie-ni-0214 <sijie.ni@zilliz.com>
## Summary
- Bump milvus-storage from f372333 to 0a9cb7ebe8
- Upgrade google-cloud-cpp from 2.5.0@milvus/2.4 to 2.28.0 (official
Conan Center package)
- Adapt storage layer code to new google-cloud-cpp 2.28.0 API changes
issue: #47863
## Changes
1. **conanfile.py**: Switch to upstream `google-cloud-cpp/2.28.0` from
custom `@milvus/2.4` package
2. **ChunkManager.cpp / MinioChunkManager.cpp**: Add `HttpClientFactory`
parameter to `ComputeEngineCredentials` constructor
3. **MinioChunkManager.h**: Add required headers, update
`AuthorizationHeader` to free function, qualify `StatusCodeToString`
4. **ffi_reader_c.cpp**: Rename `import_column_groups` →
`column_groups_import` per new milvus-storage API
5. **milvus-storage/CMakeLists.txt**: Bump version to `0a9cb7ebe8`
## Test plan
- [ ] CI build passes (C++ compilation with new dependencies)
- [ ] Unit tests pass
- [ ] Integration tests pass with GCP storage backend
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
On Linux, setenv.sh accidentally sets LD_LIBRARY_PATH and RPATH to look
up shared objects in the current working directory first if nothing
explicitly set an LD_LIBRARY_PATH already, because an empty path on
either variable is interpreted as the current working directory. This
results in races during parallel builds (e.g. tools invoked by Conan
finding corrupt shared libraries, because said libraries are currently
compiled and written out by another process in the build). Since the
LD_LIBRARY_PATH is also used as the RPATH, the compiled Milvus binary
also looks up shared objects in the current working directory,
potentially picking up unintended versions of a shared object.
Fix this and similar "empty path" problems for other environment
variables on Linux, macOS, and Windows by placing the path separator
only if there are pre-existing paths.
issue: #47828
Signed-off-by: Markus Boehme <markus.boehme@chainguard.dev>
Add parallel execution capability for C++ unit tests using GTest
sharding, enabling faster CI feedback while maintaining test isolation.
Changes:
- Add run_cpp_ut.sh: New script for parallel UT execution without
coverage
- Update run_cpp_codecov.sh: Add optional parallel support (disabled by
default)
- Add GetTestTempBasePath/GetTestTempPath helpers for shard-specific
temp paths
- Initialize ArrowFileSystemSingleton in test global setup
- Fix flaky ThreadLocalGEOSContextTest with synchronization barrier
- Update test files to use dynamic temp paths instead of hardcoded /tmp/
The coverage script (run_cpp_codecov.sh) defaults to sequential mode for
accurate coverage data. The new run_cpp_ut.sh script defaults to
parallel mode (3 shards) for fast CI feedback.
Signed-off-by: Zhikun Yao <zhikun.yao@zilliz.com>
issue: #44358
Implement complete snapshot management system including creation,
deletion, listing, description, and restoration capabilities across all
system components.
Key features:
- Create snapshots for entire collections
- Drop snapshots by name with proper cleanup
- List snapshots with collection filtering
- Describe snapshot details and metadata
Components added/modified:
- Client SDK with full snapshot API support and options
- DataCoord snapshot service with metadata management
- Proxy layer with task-based snapshot operations
- Protocol buffer definitions for snapshot RPCs
- Comprehensive unit tests with mockey framework
- Integration tests for end-to-end validation
Technical implementation:
- Snapshot metadata storage in etcd with proper indexing
- File-based snapshot data persistence in object storage
- Garbage collection integration for snapshot cleanup
- Error handling and validation across all operations
- Thread-safe operations with proper locking mechanisms
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
- Core invariant/assumption: snapshots are immutable point‑in‑time
captures identified by (collection, snapshot name/ID); etcd snapshot
metadata is authoritative for lifecycle (PENDING → COMMITTED → DELETING)
and per‑segment manifests live in object storage (Avro / StorageV2). GC
and restore logic must see snapshotRefIndex loaded
(snapshotMeta.IsRefIndexLoaded) before reclaiming or relying on
segment/index files.
- New capability added: full end‑to‑end snapshot subsystem — client SDK
APIs (Create/Drop/List/Describe/Restore + restore job queries),
DataCoord SnapshotWriter/Reader (Avro + StorageV2 manifests),
snapshotMeta in meta, SnapshotManager orchestration
(create/drop/describe/list/restore), copy‑segment restore
tasks/inspector/checker, proxy & RPC surface, GC integration, and
docs/tests — enabling point‑in‑time collection snapshots persisted to
object storage and restorations orchestrated across components.
- Logic removed/simplified and why: duplicated recursive
compaction/delta‑log traversal and ad‑hoc lookup code were consolidated
behind two focused APIs/owners (Handler.GetDeltaLogFromCompactTo for
delta traversal and SnapshotManager/SnapshotReader for snapshot I/O).
MixCoord/coordinator broker paths were converted to thin RPC proxies.
This eliminates multiple implementations of the same traversal/lookup,
reducing divergence and simplifying responsibility boundaries.
- Why this does NOT introduce data loss or regressions: snapshot
create/drop use explicit two‑phase semantics (PENDING → COMMIT/DELETING)
with SnapshotWriter writing manifests and metadata before commit; GC
uses snapshotRefIndex guards and
IsRefIndexLoaded/GetSnapshotBySegment/GetSnapshotByIndex checks to avoid
removing referenced files; restore flow pre‑allocates job IDs, validates
resources (partitions/indexes), performs rollback on failure
(rollbackRestoreSnapshot), and converts/updates segment/index metadata
only after successful copy tasks. Extensive unit and integration tests
exercise pending/deleting/GC/restore/error paths to ensure idempotence
and protection against premature deletion.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Bump milvus version to v2.6.8
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
**Core Invariant**: This PR assumes the Milvus v2.6.8 Docker image is a
stable, compatible release that can transparently replace v2.6.7 in
standalone embed configurations without breaking backward compatibility.
**What Changed**: Updated the Milvus Docker image tag from `v2.6.7` to
`v2.6.8` in two standalone embedding configuration scripts:
- `scripts/standalone_embed.bat` (line 83)
- `scripts/standalone_embed.sh` (line 62)
**Why This Is Safe**: These scripts only specify the container image
version and pass through pre-existing configuration files
(`embedEtcd.yaml`, `user.yaml`) to the container. No local logic, data
schemas, or API contracts are modified—the container startup behavior
remains identical, just pulling a newer upstream image tag. Version
increments within the same major.minor series (v2.6.x) follow semantic
versioning conventions ensuring no breaking changes.
**Impact**: Users pulling or running these standalone embed scripts will
automatically use the newer v2.6.8 Milvus release, receiving bug fixes
and enhancements from the patch version bump while maintaining
compatible behavior with existing configurations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
issue: #46500
- simplify the run_go_codecov.sh to make sure the set -e to protect any
sub command failure.
- remove all embed etcd in test to make full test can be run at local.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## PR Summary: Simplify Go Unit Tests by Removing Embedded etcd and
Async Startup Scaffolding
**Core Invariant:**
This PR assumes that unit tests can be simplified by running without
embedded etcd servers (delegating to environment-based or external etcd
instances via `kvfactory.GetEtcdAndPath()` or `ETCD_ENDPOINTS`) and by
removing goroutine-based async startup scaffolding in favor of
synchronous component initialization. Tests remain functionally
equivalent while becoming simpler to run and debug locally.
**What is Removed or Simplified:**
1. **Embedded etcd test infrastructure deleted**: Removes
`EmbedEtcdUtil` type and its public methods (SetupEtcd,
TearDownEmbedEtcd) from `pkg/util/testutils/embed_etcd.go`, removes the
`StartTestEmbedEtcdServer()` helper from `pkg/util/etcd/etcd_util.go`,
and removes etcd embedding from test suites (e.g., `TaskSuite`,
`EtcdSourceSuite`, `mixcoord/client_test.go`). Tests now either skip
etcd-dependent tests (via `MILVUS_UT_WITHOUT_KAFKA=1` environment flag
in `kafka_test.go`) or source etcd from external configuration (via
`kvfactory.GetEtcdAndPath()` in `task_test.go`, or `ETCD_ENDPOINTS`
environment variable in `etcd_source_test.go`). This eliminates the
overhead of spinning up temporary etcd servers for unit tests.
2. **Async startup scaffolding replaced with synchronous
initialization**: In `internal/proxy/proxy_test.go` and
`proxy_rpc_test.go`, the `startGrpc()` method signature removes the
`sync.WaitGroup` parameter; components are now created, prepared, and
run synchronously in-place rather than in goroutines (e.g., `go
testServer.startGrpc(ctx, &p)` becomes `testServer.startGrpc(ctx, &p)`
running synchronously). Readiness checks (e.g., `waitForGrpcReady()`)
remain in place to ensure startup safety without concurrency constructs.
This simplifies control flow and reduces debugging complexity.
3. **Shell script orchestration unified with proper error handling**: In
`scripts/run_go_codecov.sh` and `scripts/run_intergration_test.sh`,
per-package inline test invocations are consolidated into a single
`test_cmd()` function with unified `TEST_CMD_WITH_ARGS` array containing
race, coverage, verbose, and other flags. The problematic `set -ex` is
replaced with `set -e` alone (removing debug output noise while
preserving strict error semantics), ensuring the scripts fail fast on
any command failure.
**Why No Regression:**
- Test assertions and code paths remain unchanged; only deployment
source of etcd (embedded → external) and startup orchestration (async →
sync) change.
- Readiness verification (e.g., `waitForGrpcReady()`) is retained,
ensuring components are initialized before test execution.
- Test flags (race detection, coverage, verbosity) are uniformly applied
across all packages via unified `TEST_CMD_WITH_ARGS`, preserving test
coverage and quality.
- `set -e` alone is sufficient for strict failure detection without the
`-x` flag's verbose output.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: chyezh <chyezh@outlook.com>
### **Description**
- Add `-buildvcs=false` flag to Go test commands in coverage script
- Increase default session TTL from 10s to 15s
- Update SessionTTL parameter default value from 30 to 15
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: chyezh <chyezh@outlook.com>
Co-authored-by: czs007 <zhenshan.cao@zilliz.com>
