#issue: #50182
Storage version compaction used to only care whether a segment had
reached the target storage version. That is not enough once StorageV3
segments can still have different column-group formats, because a
segment may already be V3 but still not match the format DataNode is
configured to write. This change lets DataCoord use the same compaction
path to refresh those segments by format as well.
The implementation adds a separate storage format compaction switch and
enables the policy when either version or format compaction is turned
on. For V3 segments, DataCoord now checks every field binlog format
against the configured target format, schedules only the segments that
actually need a rewrite, and keeps the existing health, L0, importing,
compacting, and snapshot-protection filters in place. External
collections are skipped so table-backed segments are not accidentally
pulled into this internal rewrite flow.
Segment metrics now include a segment format label in addition to state,
level, sorted status, and storage version. The metric update path also
tracks format transitions when binlogs are added or rewritten, including
legacy, unknown, and mixed-format cases, so DataCoord can report how
storage formats are distributed while segments move through refresh and
compaction.
---------
Signed-off-by: jiaqizho <jiaqi.zhou@zilliz.com>
issue: https://github.com/milvus-io/milvus/issues/47425
Milvus already asks Conan to build the C++ third-party stack with C++20
for the host profile, but Conan keeps the build profile separate. That
means packages used during the build can still fall back to the default
C++ standard, which caused protobuf and abseil to be compiled as gnu17
in some environments and fail during the dependency build.
This change passes C++20 to the Conan build profile as well, so both the
host context and the build context use the same language level. It
applies the setting to both Linux Conan install branches in the
third-party build script, including the gcc4 ABI path and the normal
libstdc++11 path.
Signed-off-by: jiaqizho <jiaqi.zhou@zilliz.com>
Co-authored-by: EC2 Default User <ec2-user@ip-10-15-6-84.us-west-2.compute.internal>
issue: #50293
The sparse different-nq test was flaky because the collection data and
query data were generated from different sparse dimension ranges. The
collection rows used the test default dimension, while the query helper
defaulted to a much larger dimension. With drop_ratio_search=0.2, some
queries could lose all useful overlapping terms and return zero hits,
even though the test only wanted to verify nq=1 and nq=100 batch search
behavior.
ex.
```text
1. sparse data in collection is generated with dim=128
The inserted sparse vectors only use dimensions in [0, 127].
The generator also forces every vector to contain dimensions 0 and 1.
sparse_vector: {0: 0.80, 1: 0.60, 20: 0.40, 77: 0.90, ...}
2. query was generated with dim=1000
The query sparse vector may use dimensions in [0, 999].
It also contains dimensions 0 and 1, but most random dimensions may be outside [0, 127].
sparse_vector: {0: 0.01, 1: 0.02, 250: 0.70, 600: 0.50, 900: 0.80, ...}
3. after drop_ratio_search=0.2, low-weight query terms may be dropped
sparse_vector after pruning: {250: 0.70, 600: 0.50, 900: 0.80, ...}
4. the remaining query dimensions do not exist in the collection sparse data
collection dimensions: [0, 127]
remaining query dimensions: 250, 600, 900
There is no sparse term overlap, so the sparse index has no candidates for this query.
5. search can return 0 hits
The test expects topK=10 for every query result group, so this random no-overlap query makes the case fail.
```
This fixes the test by generating query sparse vectors with
ct.default_dim, the same dimension range used by the inserted sparse
data. That keeps the test focused on the intended behavior: sparse
search should handle different nq values and return the expected result
shape, without depending on accidental cross-dimension overlap surviving
sparse term pruning.
Signed-off-by: jiaqizho <jiaqi.zhou@zilliz.com>
issue: #49931
Azure needs a slightly different endpoint shape than S3-compatible
storage when the packed storage path builds properties for Loon. The old
flow treated every provider the same and always converted the address
into a full HTTP(S) URI, which works for S3 and MinIO but does not match
the Azure address format expected by milvus-storage.
This change makes the provider-specific normalization explicit at the
FFI boundary. Non-Azure storage keeps the existing useSSL based scheme
handling, while Azure addresses are passed as a bare authority by
stripping any http:// or https:// prefix from user config. This keeps
Azure compatible with milvus-storage without changing the behavior for
other object storage providers.
issue: #50182
Storage V3 can now keep writing a segment after
`dataNode.storage.format` changes. The main problem was that an
already-started growing segment may have a committed manifest written
with one format, while the current global config has moved to another
format. In that case, the next append must follow the format already
recorded in the manifest instead of blindly using the new global config.
This keeps writer.format as the current global setting, but makes the
actual split policy format local to the writer path. QueryNode growing
flush resolves writer.split.single.format from the acknowledged manifest
when one exists, and falls back to the global config only for a
brand-new segment. DataNode Storage V3 flush carries column group
formats through SegmentInfo and passes them as
writer.split.schema_based.formats, so schema-based writes can append
with the formats already known for each column group.
The metadata path now records the column group format in FieldBinlog and
preserves it through recovery, merge, duplicate filtering, fake external
segment binlogs, and V3 compaction output. This lets metacache rebuild
the current split with enough information to choose writer formats
without reading the manifest on every flush.
The storage writer wrappers now accept explicit writer format properties
for both regular packed writers and TEXT-aware segment writers. TEXT LOB
files still use their own LOB path and storage behavior; the format
selection here applies to the segment column groups that store normal
data and LOB references, not to changing the LOB payload file layout.
Signed-off-by: jiaqizho <jiaqi.zhou@zilliz.com>
issue: #48359
This change makes Milvus automatically use CRC32C for S3 PutObject
requests when OpenSSL FIPS mode is active. In FIPS mode, checksum
choices are more constrained, so relying on the configured MinIO
checksum behavior can lead to requests using an algorithm that is not
acceptable in that runtime.
The implementation moves OpenSSL FIPS enablement into the shared
`pkg/util/fips` package and makes it return whether FIPS was actually
enabled. That lets startup code keep enabling OpenSSL FIPS as before,
while paramtable can also reuse the same state during config
initialization.
When FIPS is successfully enabled and `minio.ssl.useCRC32C` is still
false, Milvus now logs a warning and temporarily overrides the runtime
value to true. This keeps the user-facing config unchanged, but makes
the effective runtime behavior compatible with FIPS requirements.
Signed-off-by: jiaqizho <jiaqi.zhou@zilliz.com>
issue: #50147
This change pulls the Vortex writer-format switch into its own focused
update. The goal is to make the insert data storage format configurable
from Milvus config, so we can turn Vortex on for CI validation without
mixing it with the unrelated AssertInfo cleanup from the previous PR.
The implementation adds dataNode.storage.format and wires it through
paramtable as DataNodeCfg.StorageFormat, with vortex as the current
default for this branch. When storagev2 builds milvus-storage
properties, it now passes the selected value through writer.format,
which matches the updated milvus-storage property contract.
This also updates the C++ writer path to use PROPERTY_WRITER_FORMAT
instead of the old PROPERTY_FORMAT key. That keeps the growing-segment
flush path aligned with the new storage property naming while still
forcing the TEXT flush writer to parquet where that path expects parquet
output.
---------
Signed-off-by: jiaqizho <jiaqi.zhou@zilliz.com>
issue: #49789
Milvus and milvus-storage were both defining the same property keys,
which made it easy for the two sides to drift apart. This changes Milvus
to read those keys directly from the const char* values exported by
milvus-storage, so the Go side stays aligned with the storage FFI
contract.
These values need to be Go vars instead of consts because C.GoString
reads from cgo-exported C symbols at init time. Go consts must be
compile-time constants, and a C const char* symbol is not a Go
compile-time string.
Signed-off-by: jiaqizho <jiaqi.zhou@zilliz.com>
issue: #49437
Make minio.ssl.tlsCACert default to an empty value instead of the
placeholder /path/to/public.crt. This way Milvus will not set
SSL_CERT_FILE by default, and the process env only gets touched when the
user actually configures a custom CA file. The generated milvus.yaml and
the related paramtable test are updated to match the new default.
Signed-off-by: jiaqizho <jiaqi.zhou@zilliz.com>
issue: https://github.com/milvus-io/milvus/issues/48360https://github.com/milvus-io/milvus/issues/48359
Add CRC32C checksum validation for MinIO/S3 PutObject requests and
enhance TLS configuration for object storage connections.
- Add minio.ssl.useCRC32C config option to enable CRC32C checksum on
PutObject requests for data integrity validation
- Pass TLS minimum version to GoogleHttpClientFactory and add
TlsHttpClientFactory for non-GCP storage when TLS is configured
- Add use_crc32c_checksum field to StorageConfig proto and propagate
through CGO layers
- Bump milvus-storage dependency
Signed-off-by: jiaqizho <jiaqi.zhou@zilliz.com>
issue: https://github.com/milvus-io/milvus/issues/48088
The nullable vector reading code was duplicated across 5 functions
(BinaryVector, Float16Vector, BFloat16Vector, FloatVector, Int8Vector),
all doing the same per-element Value(i) loop to extract valid entries.
This pulls that logic into a shared readNullableVectorData helper and
switches to bulk-copying via ValueBytes(). This works because our
PayloadWriter uses BinaryBuilder.AppendNull() which produces zero-length
null entries in the Arrow Binary values buffer, so ValueBytes()
naturally gives us only the valid data packed together. A length
assertion is added just in case this invariant ever breaks.
Benchmarked with 100k rows at 50% null rate, seeing about 10-15% speedup
across the board (only benchmarked the affected read functions).
Signed-off-by: jiaqizho <jiaqi.zhou@zilliz.com>
issue: https://github.com/milvus-io/milvus/issues/44999
The previous PR (#48000) added the configurable TLS minimum version on
the Go side but didn't wire it through to the C++ core. This commit
fills that gap by adding the tls_min_version field to CStorageConfig and
passing it down to all the places that build storage configs — index
builder, clustering analyzer, packed reader/writer, chunk manager, arrow
filesystem, and the loon FFI layer. Also bumps milvus-storage to a
version that supports the new property.
Signed-off-by: jiaqizho <jiaqi.zhou@zilliz.com>
Related to https://github.com/milvus-io/milvus/issues/44999
Currently Milvus doesn't allow users to control the TLS version used
when connecting to object storage (MinIO/S3/Azure/GCP). Some
environments require enforcing TLS 1.3 for compliance, but there's no
way to set that today.
This adds a new config option `minio.ssl.tlsMinVersion` that lets users
specify the minimum TLS version ("1.0", "1.1", "1.2", "1.3", or
"default"). It works across all supported storage backends including
MinIO/S3, Azure Blob, and GCP native. The setting is plumbed through
paramtable, proto StorageConfig, and all the places that create storage
clients (compaction, datacoord, datanode, storagev2, etc.).
For the GCP native backend, this also adds proper UseIAM/ADC support
that was previously missing, since the TLS transport injection needed to
handle both credential modes correctly.
Also fixed the GCP MinIO-compatible path to reuse any custom transport
(e.g. with TLS config) as the backend for the OAuth2 token wrapping,
instead of always creating a new default transport.
Unit tests cover the TLS version parsing, HTTP client construction, and
version enforcement (proving a TLS 1.3 client correctly rejects a TLS
1.2-only server). Integration tests are included but gated behind
environment variables.
Signed-off-by: jiaqizho <jiaqi.zhou@zilliz.com>
