mirror of
https://github.com/milvus-io/milvus.git
synced 2026-07-20 09:45:43 +00:00
test: expand CDC replication test coverage across features and chaos scenarios (#49043)
## Summary Extends the CDC (replication) e2e test suite with comprehensive cross-feature coverage and switchover/failover scenarios on top of #45624. ## What's Added ### New test files (\`tests/python_client/cdc/testcases/\`) - \`test_fts_and_text.py\` — BM25 search, text_match, phrase_match, hybrid FTS+dense, FTS after switchover (5 tests × 2 analyzers) - \`test_schema_features.py\` — dynamic schema, nullable, defaults, partition key, clustering key, combined features (7 tests) - \`test_collection_properties.py\` — TTL, mmap, autocompaction, multi-property, drop property (5 tests) - \`test_resource_group.py\` — create, drop, update, transfer replica (4 tests) - \`test_multi_database.py\` — cross-DB collections, drop DB with collections, cross-DB operations (3 tests) - \`test_search_verification.py\` — 6 tests × 7 vector types (search result consistency, query data sampling, hybrid search, iterators, filtered search) - \`test_switchover.py\` — basic, during-writes, all-types, loaded, indexed, rapid stress, failover (7 tests) ### Shared utilities - \`base.py\`: +7 schema factories, +4 data generators, +4 verification helpers, +2 constants - \`conftest.py\`: +\`switchover_helper\` fixture, +\`--is-check\`/\`--milvus-ns\` options ## Test plan - [ ] Run \`pytest tests/python_client/cdc/testcases/\` against two-cluster CDC deployment - [ ] Verify each test file runs independently via \`pytest testcases/test_<name>.py\` - [ ] Companion PR in zilliztech/test-jobs adds Jenkins pipelines to run these issue: #49042 pr: #45624 /kind test /kind improvement 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Yihao Dai <yihao.dai@zilliz.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.6
parent
2c36c60176
commit
8f09efda8f
@@ -6,12 +6,12 @@ on:
|
||||
- master
|
||||
paths:
|
||||
- 'tests/**/*.py'
|
||||
- 'tests/pyproject.toml'
|
||||
- 'tests/ruff.toml'
|
||||
- '.github/workflows/python-lint.yaml'
|
||||
pull_request:
|
||||
paths:
|
||||
- 'tests/**/*.py'
|
||||
- 'tests/pyproject.toml'
|
||||
- 'tests/ruff.toml'
|
||||
- '.github/workflows/python-lint.yaml'
|
||||
|
||||
concurrency:
|
||||
@@ -50,7 +50,7 @@ jobs:
|
||||
args: "format --check --diff"
|
||||
|
||||
- name: Validate ruff configuration (smoke test)
|
||||
# Always runs so config-only PRs still catch a broken pyproject.toml.
|
||||
# Always runs so config-only PRs still catch a broken ruff.toml.
|
||||
# --exit-zero ignores existing baseline violations; only fails when
|
||||
# ruff cannot load the config at all.
|
||||
uses: astral-sh/ruff-action@v4.0.0
|
||||
|
||||
@@ -13,9 +13,9 @@ repos:
|
||||
hooks:
|
||||
- id: ruff-check
|
||||
name: ruff-check (tests/)
|
||||
args: [--config=tests/pyproject.toml, --fix]
|
||||
args: [--config=tests/ruff.toml, --fix]
|
||||
files: ^tests/.*\.py$
|
||||
- id: ruff-format
|
||||
name: ruff-format (tests/)
|
||||
args: [--config=tests/pyproject.toml]
|
||||
args: [--config=tests/ruff.toml]
|
||||
files: ^tests/.*\.py$
|
||||
|
||||
+7
-8
@@ -92,18 +92,17 @@ $ ./e2e-k8s.sh
|
||||
|
||||
### Python Code Quality (ruff via uv)
|
||||
|
||||
Ruff is configured at `tests/pyproject.toml` and covers all Python code under `tests/`
|
||||
Ruff is configured at `tests/ruff.toml` and covers all Python code under `tests/`
|
||||
(`python_client/`, `restful_client/`, `restful_client_v2/`, `benchmark/`, `scripts/`).
|
||||
[uv](https://docs.astral.sh/uv/) is only used to host the lint/format toolchain; each
|
||||
sub-directory continues to manage its runtime dependencies via its own `requirements.txt`.
|
||||
Each sub-directory continues to manage its runtime dependencies via its own
|
||||
`requirements.txt`.
|
||||
|
||||
```shell
|
||||
$ cd tests/
|
||||
$ uv sync # install ruff into a local .venv
|
||||
$ uv run ruff check . # lint
|
||||
$ uv run ruff check . --fix # lint with auto-fix
|
||||
$ uv run ruff format . # format in place
|
||||
$ uv run ruff format --check . # format check only (CI-friendly)
|
||||
$ ruff check . # lint
|
||||
$ ruff check . --fix # lint with auto-fix
|
||||
$ ruff format . # format in place
|
||||
$ ruff format --check . # format check only (CI-friendly)
|
||||
```
|
||||
|
||||
Rules enabled: `E`, `F`, `W`, `I`, `UP`. Target Python version: `3.10`.
|
||||
|
||||
+7
-9
@@ -90,20 +90,18 @@ $ ./e2e-k8s.sh
|
||||
> $ ./e2e-k8s.sh --help
|
||||
> ```
|
||||
|
||||
### Python 代码质量 (通过 uv 使用 ruff)
|
||||
### Python 代码质量 (ruff)
|
||||
|
||||
Ruff 配置位于 `tests/pyproject.toml`,覆盖 `tests/` 下所有 Python 代码
|
||||
Ruff 配置位于 `tests/ruff.toml`,覆盖 `tests/` 下所有 Python 代码
|
||||
(`python_client/`、`restful_client/`、`restful_client_v2/`、`benchmark/`、`scripts/`)。
|
||||
[uv](https://docs.astral.sh/uv/) 仅用于托管 lint/format 工具链,各子目录的运行时依赖
|
||||
仍通过各自的 `requirements.txt` 管理。
|
||||
各子目录的运行时依赖仍通过各自的 `requirements.txt` 管理。
|
||||
|
||||
```shell
|
||||
$ cd tests/
|
||||
$ uv sync # 将 ruff 安装到本地 .venv
|
||||
$ uv run ruff check . # lint 检查
|
||||
$ uv run ruff check . --fix # lint 检查并自动修复
|
||||
$ uv run ruff format . # 原地格式化
|
||||
$ uv run ruff format --check . # 只检查不修改 (CI 友好)
|
||||
$ ruff check . # lint 检查
|
||||
$ ruff check . --fix # lint 检查并自动修复
|
||||
$ ruff format . # 原地格式化
|
||||
$ ruff format --check . # 只检查不修改 (CI 友好)
|
||||
```
|
||||
|
||||
启用的规则:`E`、`F`、`W`、`I`、`UP`;目标 Python 版本:`3.10`。
|
||||
|
||||
@@ -1,14 +1,32 @@
|
||||
import pytest
|
||||
import time
|
||||
import logging
|
||||
import time
|
||||
from concurrent.futures import ThreadPoolExecutor, wait
|
||||
|
||||
import pytest
|
||||
from pymilvus import MilvusClient
|
||||
|
||||
# Configure logging
|
||||
logging.basicConfig(
|
||||
level=logging.INFO, format="%(asctime)s - %(name)s - %(levelname)s - %(message)s"
|
||||
)
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s - %(name)s - %(levelname)s - %(message)s")
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
CDC_UPDATE_REPLICATE_TIMEOUT_SECONDS = 600
|
||||
|
||||
|
||||
def apply_replicate_configuration(tasks, timeout=CDC_UPDATE_REPLICATE_TIMEOUT_SECONDS):
|
||||
# Fan out in parallel: the server blocks non-primary clusters in
|
||||
# waitUntilPrimaryChangeOrConfigurationSame until the primary's broadcast
|
||||
# propagates via CDC, so a sequential call where the first client happens
|
||||
# to be a replica deadlocks on the client's RPC timeout.
|
||||
with ThreadPoolExecutor(max_workers=len(tasks)) as executor:
|
||||
futures = [
|
||||
executor.submit(client.update_replicate_configuration, timeout=timeout, **config)
|
||||
for client, config in tasks
|
||||
]
|
||||
wait(futures)
|
||||
for f in futures:
|
||||
f.result()
|
||||
|
||||
|
||||
def pytest_addoption(parser):
|
||||
"""Add command line options for pytest."""
|
||||
parser.addoption(
|
||||
@@ -35,9 +53,7 @@ def pytest_addoption(parser):
|
||||
default="root:Milvus",
|
||||
help="Downstream Milvus token",
|
||||
)
|
||||
parser.addoption(
|
||||
"--sync-timeout", action="store", default="30", help="Sync timeout in seconds"
|
||||
)
|
||||
parser.addoption("--sync-timeout", action="store", default="30", help="Sync timeout in seconds")
|
||||
parser.addoption(
|
||||
"--source-cluster-id",
|
||||
action="store",
|
||||
@@ -62,6 +78,18 @@ def pytest_addoption(parser):
|
||||
default="30m",
|
||||
help="Duration for test operations (e.g., 30m, 1h, 60s)",
|
||||
)
|
||||
parser.addoption(
|
||||
"--is-check",
|
||||
action="store",
|
||||
default="true",
|
||||
help="Whether to assert on checker statistics",
|
||||
)
|
||||
parser.addoption(
|
||||
"--milvus-ns",
|
||||
action="store",
|
||||
default="chaos-testing",
|
||||
help="Kubernetes namespace for Milvus deployment",
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
@@ -138,6 +166,75 @@ def request_duration(request):
|
||||
return request.config.getoption("--request-duration")
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def is_check(request):
|
||||
# The root tests/python_client/conftest.py registers --is_check (underscore)
|
||||
# with type=bool, which argparse maps to the same dest (is_check) as our
|
||||
# --is-check (hyphen). The root's bool wins in chaos runs, so accept either.
|
||||
val = request.config.getoption("--is-check")
|
||||
return val if isinstance(val, bool) else str(val).lower() == "true"
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def milvus_ns(request):
|
||||
return request.config.getoption("--milvus-ns")
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def switchover_helper(request, upstream_client, downstream_client):
|
||||
"""Returns a callable that performs CDC topology switchover."""
|
||||
upstream_uri = request.config.getoption("--upstream-uri")
|
||||
upstream_token = request.config.getoption("--upstream-token")
|
||||
downstream_uri = request.config.getoption("--downstream-uri")
|
||||
downstream_token = request.config.getoption("--downstream-token")
|
||||
pchannel_num = int(request.config.getoption("--pchannel-num"))
|
||||
original_source = request.config.getoption("--source-cluster-id")
|
||||
original_target = request.config.getoption("--target-cluster-id")
|
||||
|
||||
# Map cluster IDs to their URIs/tokens
|
||||
cluster_map = {
|
||||
original_source: {"uri": upstream_uri, "token": upstream_token},
|
||||
original_target: {"uri": downstream_uri, "token": downstream_token},
|
||||
}
|
||||
|
||||
def do_switchover(new_source_id, new_target_id):
|
||||
logger.info(f"Performing switchover: {new_source_id} -> {new_target_id}")
|
||||
config = {
|
||||
"clusters": [
|
||||
{
|
||||
"cluster_id": new_source_id,
|
||||
"connection_param": cluster_map[new_source_id],
|
||||
"pchannels": [f"{new_source_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
|
||||
},
|
||||
{
|
||||
"cluster_id": new_target_id,
|
||||
"connection_param": cluster_map[new_target_id],
|
||||
"pchannels": [f"{new_target_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
|
||||
},
|
||||
],
|
||||
"cross_cluster_topology": [{"source_cluster_id": new_source_id, "target_cluster_id": new_target_id}],
|
||||
}
|
||||
# Dedicated short-lived clients so switchover RPCs don't share a
|
||||
# gRPC channel with concurrent DML on the session-scoped clients.
|
||||
# pymilvus's connection manager closes a channel on UNAVAILABLE /
|
||||
# STREAMING_CODE_REPLICATE_VIOLATION to trigger recovery; if a DML
|
||||
# on the session client triggers that close while our sibling
|
||||
# update_replicate_configuration RPC is in flight on the same
|
||||
# channel, the latter surfaces "Cannot invoke RPC on closed
|
||||
# channel!". Separate clients = separate channels = no race.
|
||||
up_tmp = MilvusClient(uri=upstream_uri, token=upstream_token)
|
||||
dn_tmp = MilvusClient(uri=downstream_uri, token=downstream_token)
|
||||
try:
|
||||
apply_replicate_configuration([(up_tmp, config), (dn_tmp, config)])
|
||||
finally:
|
||||
up_tmp.close()
|
||||
dn_tmp.close()
|
||||
logger.info("Switchover completed, waiting 10s for stabilization...")
|
||||
time.sleep(10)
|
||||
|
||||
return do_switchover
|
||||
|
||||
|
||||
@pytest.fixture(scope="session", autouse=True)
|
||||
def cdc_topology_setup(request, upstream_client, downstream_client):
|
||||
"""Setup CDC topology at the beginning of test session."""
|
||||
@@ -147,9 +244,7 @@ def cdc_topology_setup(request, upstream_client, downstream_client):
|
||||
target_cluster_id = request.config.getoption("--target-cluster-id")
|
||||
pchannel_num = int(request.config.getoption("--pchannel-num"))
|
||||
|
||||
logger.info(
|
||||
f"Setting up CDC topology: {source_cluster_id} -> {target_cluster_id} (channels: {pchannel_num})..."
|
||||
)
|
||||
logger.info(f"Setting up CDC topology: {source_cluster_id} -> {target_cluster_id} (channels: {pchannel_num})...")
|
||||
|
||||
# Create CDC replication configuration
|
||||
config = {
|
||||
@@ -160,10 +255,7 @@ def cdc_topology_setup(request, upstream_client, downstream_client):
|
||||
"uri": upstream_uri,
|
||||
"token": request.config.getoption("--upstream-token"),
|
||||
},
|
||||
"pchannels": [
|
||||
f"{source_cluster_id}-rootcoord-dml_{i}"
|
||||
for i in range(pchannel_num)
|
||||
],
|
||||
"pchannels": [f"{source_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
|
||||
},
|
||||
{
|
||||
"cluster_id": target_cluster_id,
|
||||
@@ -171,10 +263,7 @@ def cdc_topology_setup(request, upstream_client, downstream_client):
|
||||
"uri": downstream_uri,
|
||||
"token": request.config.getoption("--downstream-token"),
|
||||
},
|
||||
"pchannels": [
|
||||
f"{target_cluster_id}-rootcoord-dml_{i}"
|
||||
for i in range(pchannel_num)
|
||||
],
|
||||
"pchannels": [f"{target_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
|
||||
},
|
||||
],
|
||||
"cross_cluster_topology": [
|
||||
@@ -186,9 +275,16 @@ def cdc_topology_setup(request, upstream_client, downstream_client):
|
||||
}
|
||||
|
||||
try:
|
||||
# Update replication configuration on both clusters
|
||||
upstream_client.update_replicate_configuration(**config)
|
||||
downstream_client.update_replicate_configuration(**config)
|
||||
# Dedicated clients for the control-plane update_replicate_configuration
|
||||
# RPC, mirroring switchover_helper. Keeps the session-scoped clients'
|
||||
# channels clean of any recovery side effects from the initial setup.
|
||||
up_tmp = MilvusClient(uri=upstream_uri, token=request.config.getoption("--upstream-token"))
|
||||
dn_tmp = MilvusClient(uri=downstream_uri, token=request.config.getoption("--downstream-token"))
|
||||
try:
|
||||
apply_replicate_configuration([(up_tmp, config), (dn_tmp, config)])
|
||||
finally:
|
||||
up_tmp.close()
|
||||
dn_tmp.close()
|
||||
logger.info("CDC topology setup completed successfully")
|
||||
|
||||
# Allow some time for CDC to initialize
|
||||
|
||||
@@ -1,34 +1,58 @@
|
||||
from pymilvus import MilvusClient
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
|
||||
def setup_cdc_topology(upstream_uri, downstream_uri, removed_clusters_uri, upstream_token, downstream_token, removed_clusters_token, source_cluster_id, target_cluster_id, removed_clusters_id, pchannel_num):
|
||||
print(f"DEBUG: upstream_uri: {upstream_uri}, downstream_uri: {downstream_uri}, upstream_token: {upstream_token}, downstream_token: {downstream_token}, source_cluster_id: {source_cluster_id}, target_cluster_id: {target_cluster_id}, pchannel_num: {pchannel_num}")
|
||||
from pymilvus import MilvusClient
|
||||
|
||||
# Kept in sync with cdc/conftest.py's CDC_UPDATE_REPLICATE_TIMEOUT_SECONDS.
|
||||
# Inlined because this file is executed standalone from tests/python_client/cdc/scripts/,
|
||||
# where the cdc package is not on sys.path.
|
||||
CDC_UPDATE_REPLICATE_TIMEOUT_SECONDS = 600
|
||||
|
||||
|
||||
def setup_cdc_topology(
|
||||
upstream_uri,
|
||||
downstream_uri,
|
||||
removed_clusters_uri,
|
||||
upstream_token,
|
||||
downstream_token,
|
||||
removed_clusters_token,
|
||||
source_cluster_id,
|
||||
target_cluster_id,
|
||||
removed_clusters_id,
|
||||
pchannel_num,
|
||||
):
|
||||
print(
|
||||
f"DEBUG: upstream_uri: {upstream_uri}, downstream_uri: {downstream_uri}, upstream_token: {upstream_token}, downstream_token: {downstream_token}, source_cluster_id: {source_cluster_id}, target_cluster_id: {target_cluster_id}, pchannel_num: {pchannel_num}"
|
||||
)
|
||||
upstream_client = MilvusClient(uri=upstream_uri, token=upstream_token)
|
||||
|
||||
# Parse comma-separated lists
|
||||
if isinstance(downstream_uri, str) and ',' in downstream_uri:
|
||||
downstream_uris = [uri.strip() for uri in downstream_uri.split(',')]
|
||||
if isinstance(downstream_uri, str) and "," in downstream_uri:
|
||||
downstream_uris = [uri.strip() for uri in downstream_uri.split(",")]
|
||||
else:
|
||||
downstream_uris = [downstream_uri] if isinstance(downstream_uri, str) else downstream_uri
|
||||
|
||||
if isinstance(target_cluster_id, str) and ',' in target_cluster_id:
|
||||
target_cluster_ids = [cluster_id.strip() for cluster_id in target_cluster_id.split(',')]
|
||||
if isinstance(target_cluster_id, str) and "," in target_cluster_id:
|
||||
target_cluster_ids = [cluster_id.strip() for cluster_id in target_cluster_id.split(",")]
|
||||
else:
|
||||
target_cluster_ids = [target_cluster_id] if isinstance(target_cluster_id, str) else target_cluster_id
|
||||
|
||||
if isinstance(removed_clusters_uri, str) and ',' in removed_clusters_uri:
|
||||
removed_clusters_uris = [uri.strip() for uri in removed_clusters_uri.split(',')]
|
||||
if isinstance(removed_clusters_uri, str) and "," in removed_clusters_uri:
|
||||
removed_clusters_uris = [uri.strip() for uri in removed_clusters_uri.split(",")]
|
||||
else:
|
||||
removed_clusters_uris = [removed_clusters_uri] if isinstance(removed_clusters_uri, str) else removed_clusters_uri
|
||||
removed_clusters_uris = (
|
||||
[removed_clusters_uri] if isinstance(removed_clusters_uri, str) else removed_clusters_uri
|
||||
)
|
||||
|
||||
if isinstance(removed_clusters_id, str) and ',' in removed_clusters_id:
|
||||
removed_clusters_ids = [cluster_id.strip() for cluster_id in removed_clusters_id.split(',')]
|
||||
if isinstance(removed_clusters_id, str) and "," in removed_clusters_id:
|
||||
removed_clusters_ids = [cluster_id.strip() for cluster_id in removed_clusters_id.split(",")]
|
||||
else:
|
||||
removed_clusters_ids = [removed_clusters_id] if isinstance(removed_clusters_id, str) else removed_clusters_id
|
||||
|
||||
# Ensure we have matching numbers of downstream URIs and cluster IDs
|
||||
if len(downstream_uris) != len(target_cluster_ids):
|
||||
raise ValueError(f"Number of downstream URIs ({len(downstream_uris)}) must match number of target cluster IDs ({len(target_cluster_ids)})")
|
||||
raise ValueError(
|
||||
f"Number of downstream URIs ({len(downstream_uris)}) must match number of target cluster IDs ({len(target_cluster_ids)})"
|
||||
)
|
||||
|
||||
# Create downstream clients
|
||||
downstream_clients = []
|
||||
@@ -40,44 +64,34 @@ def setup_cdc_topology(upstream_uri, downstream_uri, removed_clusters_uri, upstr
|
||||
clusters = [
|
||||
{
|
||||
"cluster_id": source_cluster_id,
|
||||
"connection_param": {
|
||||
"uri": upstream_uri,
|
||||
"token": upstream_token
|
||||
},
|
||||
"pchannels": [f"{source_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)]
|
||||
"connection_param": {"uri": upstream_uri, "token": upstream_token},
|
||||
"pchannels": [f"{source_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
|
||||
}
|
||||
]
|
||||
|
||||
# Add all target clusters
|
||||
for target_id, target_uri in zip(target_cluster_ids, downstream_uris):
|
||||
clusters.append({
|
||||
"cluster_id": target_id,
|
||||
"connection_param": {
|
||||
"uri": target_uri,
|
||||
"token": downstream_token
|
||||
},
|
||||
"pchannels": [f"{target_id}-rootcoord-dml_{j}" for j in range(pchannel_num)]
|
||||
})
|
||||
clusters.append(
|
||||
{
|
||||
"cluster_id": target_id,
|
||||
"connection_param": {"uri": target_uri, "token": downstream_token},
|
||||
"pchannels": [f"{target_id}-rootcoord-dml_{j}" for j in range(pchannel_num)],
|
||||
}
|
||||
)
|
||||
|
||||
# Build cross-cluster topology
|
||||
cross_cluster_topology = []
|
||||
for target_id in target_cluster_ids:
|
||||
cross_cluster_topology.append({
|
||||
"source_cluster_id": source_cluster_id,
|
||||
"target_cluster_id": target_id
|
||||
})
|
||||
cross_cluster_topology.append({"source_cluster_id": source_cluster_id, "target_cluster_id": target_id})
|
||||
|
||||
config = {
|
||||
"clusters": clusters,
|
||||
"cross_cluster_topology": cross_cluster_topology
|
||||
}
|
||||
config = {"clusters": clusters, "cross_cluster_topology": cross_cluster_topology}
|
||||
|
||||
# Update configuration on all clients using multi-threading
|
||||
print(f"DEBUG: config: {config}")
|
||||
|
||||
def update_client_config(client, config_to_use, client_type=""):
|
||||
try:
|
||||
client.update_replicate_configuration(**config_to_use)
|
||||
client.update_replicate_configuration(timeout=CDC_UPDATE_REPLICATE_TIMEOUT_SECONDS, **config_to_use)
|
||||
return f"{client_type} updated successfully"
|
||||
except Exception as e:
|
||||
print(f"Failed to update {client_type}: {e}")
|
||||
@@ -102,14 +116,11 @@ def setup_cdc_topology(upstream_uri, downstream_uri, removed_clusters_uri, upstr
|
||||
"clusters": [
|
||||
{
|
||||
"cluster_id": removed_id,
|
||||
"connection_param": {
|
||||
"uri": removed_uri,
|
||||
"token": removed_clusters_token
|
||||
},
|
||||
"pchannels": [f"{removed_id}-rootcoord-dml_{i}" for i in range(pchannel_num)]
|
||||
"connection_param": {"uri": removed_uri, "token": removed_clusters_token},
|
||||
"pchannels": [f"{removed_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
|
||||
}
|
||||
],
|
||||
"cross_cluster_topology": []
|
||||
"cross_cluster_topology": [],
|
||||
}
|
||||
print(f"DEBUG: Removing cluster {removed_id} with empty config: {empty_config}")
|
||||
update_tasks.append((removed_client, empty_config, f"Removed cluster {removed_id}"))
|
||||
@@ -117,8 +128,10 @@ def setup_cdc_topology(upstream_uri, downstream_uri, removed_clusters_uri, upstr
|
||||
# Use single ThreadPoolExecutor to update all clients concurrently
|
||||
with ThreadPoolExecutor(max_workers=len(update_tasks)) as executor:
|
||||
# Submit all update tasks
|
||||
futures = [executor.submit(update_client_config, client, config_to_use, client_type)
|
||||
for client, config_to_use, client_type in update_tasks]
|
||||
futures = [
|
||||
executor.submit(update_client_config, client, config_to_use, client_type)
|
||||
for client, config_to_use, client_type in update_tasks
|
||||
]
|
||||
|
||||
# Wait for all tasks to complete
|
||||
for future in as_completed(futures):
|
||||
@@ -133,26 +146,28 @@ def setup_cdc_topology(upstream_uri, downstream_uri, removed_clusters_uri, upstr
|
||||
if __name__ == "__main__":
|
||||
import argparse
|
||||
|
||||
parser = argparse.ArgumentParser(description='connection info')
|
||||
parser.add_argument('--upstream_uri', type=str, default='10.100.36.179', help='milvus host')
|
||||
parser.add_argument('--downstream_uri', type=str, default='10.100.36.178', help='milvus host')
|
||||
parser.add_argument('--removed_clusters_uri', type=str, default='', help='milvus host')
|
||||
parser.add_argument('--upstream_token', type=str, default='root:Milvus', help='milvus token')
|
||||
parser.add_argument('--downstream_token', type=str, default='root:Milvus', help='milvus token')
|
||||
parser.add_argument('--removed_clusters_token', type=str, default='root:Milvus', help='milvus token')
|
||||
parser.add_argument('--source_cluster_id', type=str, default='cdc-test-source', help='source cluster id')
|
||||
parser.add_argument('--target_cluster_id', type=str, default='cdc-test-target', help='target cluster id')
|
||||
parser.add_argument('--removed_clusters_id', type=str, default='', help='removed clusters id')
|
||||
parser = argparse.ArgumentParser(description="connection info")
|
||||
parser.add_argument("--upstream_uri", type=str, default="10.100.36.179", help="milvus host")
|
||||
parser.add_argument("--downstream_uri", type=str, default="10.100.36.178", help="milvus host")
|
||||
parser.add_argument("--removed_clusters_uri", type=str, default="", help="milvus host")
|
||||
parser.add_argument("--upstream_token", type=str, default="root:Milvus", help="milvus token")
|
||||
parser.add_argument("--downstream_token", type=str, default="root:Milvus", help="milvus token")
|
||||
parser.add_argument("--removed_clusters_token", type=str, default="root:Milvus", help="milvus token")
|
||||
parser.add_argument("--source_cluster_id", type=str, default="cdc-test-source", help="source cluster id")
|
||||
parser.add_argument("--target_cluster_id", type=str, default="cdc-test-target", help="target cluster id")
|
||||
parser.add_argument("--removed_clusters_id", type=str, default="", help="removed clusters id")
|
||||
|
||||
parser.add_argument('--pchannel_num', type=int, default=16, help='pchannel num')
|
||||
parser.add_argument("--pchannel_num", type=int, default=16, help="pchannel num")
|
||||
args = parser.parse_args()
|
||||
setup_cdc_topology(args.upstream_uri,
|
||||
args.downstream_uri,
|
||||
args.removed_clusters_uri,
|
||||
args.upstream_token,
|
||||
args.downstream_token,
|
||||
args.removed_clusters_token,
|
||||
args.source_cluster_id,
|
||||
args.target_cluster_id,
|
||||
args.removed_clusters_id,
|
||||
args.pchannel_num)
|
||||
setup_cdc_topology(
|
||||
args.upstream_uri,
|
||||
args.downstream_uri,
|
||||
args.removed_clusters_uri,
|
||||
args.upstream_token,
|
||||
args.downstream_token,
|
||||
args.removed_clusters_token,
|
||||
args.source_cluster_id,
|
||||
args.target_cluster_id,
|
||||
args.removed_clusters_id,
|
||||
args.pchannel_num,
|
||||
)
|
||||
|
||||
@@ -1,38 +1,71 @@
|
||||
import time
|
||||
import pytest
|
||||
import json
|
||||
import time
|
||||
from time import sleep
|
||||
from pymilvus import connections
|
||||
from chaos.checker import (InsertChecker,
|
||||
UpsertChecker,
|
||||
FlushChecker,
|
||||
SearchChecker,
|
||||
FullTextSearchChecker,
|
||||
HybridSearchChecker,
|
||||
QueryChecker,
|
||||
TextMatchChecker,
|
||||
PhraseMatchChecker,
|
||||
JsonQueryChecker,
|
||||
GeoQueryChecker,
|
||||
DeleteChecker,
|
||||
AddFieldChecker,
|
||||
Op,
|
||||
ResultAnalyzer
|
||||
)
|
||||
from utils.util_k8s import wait_pods_ready, get_milvus_instance_name
|
||||
from utils.util_log import test_log as log
|
||||
|
||||
import pytest
|
||||
from chaos import chaos_commons as cc
|
||||
from common import common_func as cf
|
||||
from common.milvus_sys import MilvusSys
|
||||
from chaos.chaos_commons import assert_statistic
|
||||
from common.common_type import CaseLabel
|
||||
from chaos import constants
|
||||
from chaos.chaos_commons import assert_statistic
|
||||
from chaos.checker import (
|
||||
AddFieldChecker,
|
||||
DeleteChecker,
|
||||
FlushChecker,
|
||||
FullTextSearchChecker,
|
||||
GeoQueryChecker,
|
||||
HybridSearchChecker,
|
||||
InsertChecker,
|
||||
JsonQueryChecker,
|
||||
Op,
|
||||
PhraseMatchChecker,
|
||||
QueryChecker,
|
||||
ResultAnalyzer,
|
||||
SearchChecker,
|
||||
TextMatchChecker,
|
||||
UpsertChecker,
|
||||
)
|
||||
from common import common_func as cf
|
||||
from common.common_type import CaseLabel
|
||||
from common.milvus_sys import MilvusSys
|
||||
from delayed_assert import assert_expectations
|
||||
from pymilvus import DataType, FunctionType, connections
|
||||
from utils.util_k8s import get_milvus_instance_name, wait_pods_ready
|
||||
from utils.util_log import test_log as log
|
||||
|
||||
_VECTOR_DTYPES = {
|
||||
DataType.FLOAT_VECTOR,
|
||||
DataType.FLOAT16_VECTOR,
|
||||
DataType.BFLOAT16_VECTOR,
|
||||
DataType.BINARY_VECTOR,
|
||||
DataType.SPARSE_FLOAT_VECTOR,
|
||||
DataType.INT8_VECTOR,
|
||||
}
|
||||
|
||||
|
||||
def _build_checker_schema(dim=8):
|
||||
"""Build the shared all-datatype schema, stripped for the 2.6-latest image.
|
||||
|
||||
The chaos-test image used by milvus_cdc_chaos_test/verify_test rejects
|
||||
two things the shared gen_all_datatype_collection_schema includes by
|
||||
default:
|
||||
- FunctionType.MINHASH (error: "check function params with unknown
|
||||
function type")
|
||||
- nullable=True on FLOAT_VECTOR (error: "vector type not support null")
|
||||
|
||||
Drop the MinHash function and its output field, and force nullable=False
|
||||
on every vector field so the server accepts the schema.
|
||||
"""
|
||||
schema = cf.gen_all_datatype_collection_schema(dim=dim)
|
||||
schema.functions[:] = [f for f in schema.functions if f.type != FunctionType.MINHASH]
|
||||
schema.fields[:] = [f for f in schema.fields if f.name != "minhash_emb"]
|
||||
for f in schema.fields:
|
||||
if f.dtype in _VECTOR_DTYPES:
|
||||
f.nullable = False
|
||||
return schema
|
||||
|
||||
|
||||
def get_all_collections():
|
||||
try:
|
||||
with open("/tmp/ci_logs/chaos_test_all_collections.json", "r") as f:
|
||||
with open("/tmp/ci_logs/chaos_test_all_collections.json") as f:
|
||||
data = json.load(f)
|
||||
all_collections = data["all"]
|
||||
except Exception as e:
|
||||
@@ -48,40 +81,40 @@ class TestBase:
|
||||
expect_compact = constants.SUCC
|
||||
expect_search = constants.SUCC
|
||||
expect_query = constants.SUCC
|
||||
host = '127.0.0.1'
|
||||
host = "127.0.0.1"
|
||||
port = 19530
|
||||
_chaos_config = None
|
||||
health_checkers = {}
|
||||
|
||||
|
||||
class TestOperations(TestBase):
|
||||
|
||||
@pytest.fixture(scope="function", autouse=True)
|
||||
def connection(self, upstream_uri, upstream_token, milvus_ns):
|
||||
connections.connect('default', uri=upstream_uri, token=upstream_token)
|
||||
connections.connect("default", uri=upstream_uri, token=upstream_token)
|
||||
if connections.has_connection("default") is False:
|
||||
raise Exception("no connections")
|
||||
log.info("connect to milvus successfully")
|
||||
self.milvus_sys = MilvusSys(alias='default')
|
||||
self.milvus_sys = MilvusSys(alias="default")
|
||||
self.milvus_ns = milvus_ns
|
||||
self.release_name = get_milvus_instance_name(self.milvus_ns, milvus_sys=self.milvus_sys)
|
||||
|
||||
def init_health_checkers(self, collection_name=None):
|
||||
c_name = collection_name
|
||||
schema = _build_checker_schema()
|
||||
checkers = {
|
||||
Op.insert: InsertChecker(collection_name=c_name),
|
||||
Op.upsert: UpsertChecker(collection_name=c_name),
|
||||
Op.flush: FlushChecker(collection_name=c_name),
|
||||
Op.search: SearchChecker(collection_name=c_name),
|
||||
Op.full_text_search: FullTextSearchChecker(collection_name=c_name),
|
||||
Op.hybrid_search: HybridSearchChecker(collection_name=c_name),
|
||||
Op.query: QueryChecker(collection_name=c_name),
|
||||
Op.text_match: TextMatchChecker(collection_name=c_name),
|
||||
Op.phrase_match: PhraseMatchChecker(collection_name=c_name),
|
||||
Op.json_query: JsonQueryChecker(collection_name=c_name),
|
||||
Op.geo_query: GeoQueryChecker(collection_name=c_name),
|
||||
Op.delete: DeleteChecker(collection_name=c_name),
|
||||
Op.add_field: AddFieldChecker(collection_name=c_name),
|
||||
Op.insert: InsertChecker(collection_name=c_name, schema=schema),
|
||||
Op.upsert: UpsertChecker(collection_name=c_name, schema=schema),
|
||||
Op.flush: FlushChecker(collection_name=c_name, schema=schema),
|
||||
Op.search: SearchChecker(collection_name=c_name, schema=schema),
|
||||
Op.full_text_search: FullTextSearchChecker(collection_name=c_name, schema=schema),
|
||||
Op.hybrid_search: HybridSearchChecker(collection_name=c_name, schema=schema),
|
||||
Op.query: QueryChecker(collection_name=c_name, schema=schema),
|
||||
Op.text_match: TextMatchChecker(collection_name=c_name, schema=schema),
|
||||
Op.phrase_match: PhraseMatchChecker(collection_name=c_name, schema=schema),
|
||||
Op.json_query: JsonQueryChecker(collection_name=c_name, schema=schema),
|
||||
Op.geo_query: GeoQueryChecker(collection_name=c_name, schema=schema),
|
||||
Op.delete: DeleteChecker(collection_name=c_name, schema=schema),
|
||||
Op.add_field: AddFieldChecker(collection_name=c_name, schema=schema),
|
||||
}
|
||||
log.info(f"init_health_checkers: {checkers}")
|
||||
self.health_checkers = checkers
|
||||
@@ -96,7 +129,7 @@ class TestOperations(TestBase):
|
||||
def test_operations(self, request_duration, is_check, collection_name):
|
||||
# start the monitor threads to check the milvus ops
|
||||
log.info("*********************Test Start**********************")
|
||||
log.info(connections.get_connection_addr('default'))
|
||||
log.info(connections.get_connection_addr("default"))
|
||||
# event_records = EventRecords()
|
||||
c_name = collection_name if collection_name else cf.gen_unique_str("Checker_")
|
||||
# event_records.insert("init_health_checkers", "start")
|
||||
@@ -109,7 +142,7 @@ class TestOperations(TestBase):
|
||||
request_duration = request_duration[:-1]
|
||||
request_duration = eval(request_duration)
|
||||
for i in range(10):
|
||||
sleep(request_duration//10)
|
||||
sleep(request_duration // 10)
|
||||
for k, v in self.health_checkers.items():
|
||||
v.check_result()
|
||||
# log.info(v.check_result())
|
||||
|
||||
@@ -1,41 +1,73 @@
|
||||
import time
|
||||
|
||||
import pytest
|
||||
from time import sleep
|
||||
|
||||
import pymilvus
|
||||
from pymilvus import connections, utility
|
||||
from chaos.checker import (CollectionCreateChecker,
|
||||
InsertChecker,
|
||||
BulkInsertChecker,
|
||||
UpsertChecker,
|
||||
PartialUpdateChecker,
|
||||
FlushChecker,
|
||||
SearchChecker,
|
||||
FullTextSearchChecker,
|
||||
HybridSearchChecker,
|
||||
QueryChecker,
|
||||
TextMatchChecker,
|
||||
PhraseMatchChecker,
|
||||
JsonQueryChecker,
|
||||
GeoQueryChecker,
|
||||
IndexCreateChecker,
|
||||
DeleteChecker,
|
||||
CollectionDropChecker,
|
||||
AlterCollectionChecker,
|
||||
AddFieldChecker,
|
||||
CollectionRenameChecker,
|
||||
Op,
|
||||
EventRecords,
|
||||
ResultAnalyzer
|
||||
)
|
||||
from utils.util_log import test_log as log
|
||||
from utils.util_k8s import wait_pods_ready, get_milvus_instance_name
|
||||
import pytest
|
||||
from chaos import chaos_commons as cc
|
||||
from chaos import constants
|
||||
from chaos.chaos_commons import assert_statistic
|
||||
from chaos.checker import (
|
||||
AddFieldChecker,
|
||||
AlterCollectionChecker,
|
||||
CollectionCreateChecker,
|
||||
CollectionDropChecker,
|
||||
CollectionRenameChecker,
|
||||
DeleteChecker,
|
||||
EventRecords,
|
||||
FlushChecker,
|
||||
FullTextSearchChecker,
|
||||
GeoQueryChecker,
|
||||
HybridSearchChecker,
|
||||
IndexCreateChecker,
|
||||
InsertChecker,
|
||||
JsonQueryChecker,
|
||||
Op,
|
||||
PartialUpdateChecker,
|
||||
PhraseMatchChecker,
|
||||
QueryChecker,
|
||||
ResultAnalyzer,
|
||||
SearchChecker,
|
||||
TextMatchChecker,
|
||||
UpsertChecker,
|
||||
)
|
||||
from common import common_func as cf
|
||||
from common.common_type import CaseLabel
|
||||
from common.milvus_sys import MilvusSys
|
||||
from chaos.chaos_commons import assert_statistic
|
||||
from chaos import constants
|
||||
from delayed_assert import assert_expectations
|
||||
from pymilvus import DataType, FunctionType, connections, utility
|
||||
from utils.util_k8s import get_milvus_instance_name, wait_pods_ready
|
||||
from utils.util_log import test_log as log
|
||||
|
||||
_VECTOR_DTYPES = {
|
||||
DataType.FLOAT_VECTOR,
|
||||
DataType.FLOAT16_VECTOR,
|
||||
DataType.BFLOAT16_VECTOR,
|
||||
DataType.BINARY_VECTOR,
|
||||
DataType.SPARSE_FLOAT_VECTOR,
|
||||
DataType.INT8_VECTOR,
|
||||
}
|
||||
|
||||
|
||||
def _build_checker_schema(dim=8):
|
||||
"""Build the shared all-datatype schema, stripped for the 2.6-latest image.
|
||||
|
||||
The chaos-test image used by milvus_cdc_chaos_test/verify_test rejects
|
||||
two things the shared gen_all_datatype_collection_schema includes by
|
||||
default:
|
||||
- FunctionType.MINHASH (error: "check function params with unknown
|
||||
function type")
|
||||
- nullable=True on FLOAT_VECTOR (error: "vector type not support null")
|
||||
|
||||
Drop the MinHash function and its output field, and force nullable=False
|
||||
on every vector field so the server accepts the schema.
|
||||
"""
|
||||
schema = cf.gen_all_datatype_collection_schema(dim=dim)
|
||||
schema.functions[:] = [f for f in schema.functions if f.type != FunctionType.MINHASH]
|
||||
schema.fields[:] = [f for f in schema.fields if f.name != "minhash_emb"]
|
||||
for f in schema.fields:
|
||||
if f.dtype in _VECTOR_DTYPES:
|
||||
f.nullable = False
|
||||
return schema
|
||||
|
||||
|
||||
class TestBase:
|
||||
@@ -45,17 +77,16 @@ class TestBase:
|
||||
expect_index = constants.SUCC
|
||||
expect_search = constants.SUCC
|
||||
expect_query = constants.SUCC
|
||||
host = '127.0.0.1'
|
||||
host = "127.0.0.1"
|
||||
port = 19530
|
||||
_chaos_config = None
|
||||
health_checkers = {}
|
||||
|
||||
|
||||
class TestOperations(TestBase):
|
||||
|
||||
@pytest.fixture(scope="function", autouse=True)
|
||||
def connection(self, upstream_uri, upstream_token, milvus_ns):
|
||||
connections.connect('default', uri=upstream_uri, token=upstream_token)
|
||||
connections.connect("default", uri=upstream_uri, token=upstream_token)
|
||||
if connections.has_connection("default") is False:
|
||||
raise Exception("no connections")
|
||||
log.info("connect to milvus successfully")
|
||||
@@ -63,32 +94,33 @@ class TestOperations(TestBase):
|
||||
server_version = utility.get_server_version()
|
||||
log.info(f"server version: {server_version}")
|
||||
log.info(f"pymilvus version: {pymilvus_version}")
|
||||
self.milvus_sys = MilvusSys(alias='default')
|
||||
self.milvus_sys = MilvusSys(alias="default")
|
||||
self.milvus_ns = milvus_ns
|
||||
self.release_name = get_milvus_instance_name(self.milvus_ns, milvus_sys=self.milvus_sys)
|
||||
|
||||
def init_health_checkers(self, collection_name=None):
|
||||
c_name = collection_name
|
||||
schema = _build_checker_schema()
|
||||
checkers = {
|
||||
Op.create: CollectionCreateChecker(collection_name=c_name),
|
||||
Op.insert: InsertChecker(collection_name=c_name),
|
||||
Op.upsert: UpsertChecker(collection_name=c_name),
|
||||
Op.partial_update: PartialUpdateChecker(collection_name=c_name),
|
||||
Op.flush: FlushChecker(collection_name=c_name),
|
||||
Op.index: IndexCreateChecker(collection_name=c_name),
|
||||
Op.search: SearchChecker(collection_name=c_name),
|
||||
Op.full_text_search: FullTextSearchChecker(collection_name=c_name),
|
||||
Op.hybrid_search: HybridSearchChecker(collection_name=c_name),
|
||||
Op.query: QueryChecker(collection_name=c_name),
|
||||
Op.text_match: TextMatchChecker(collection_name=c_name),
|
||||
Op.phrase_match: PhraseMatchChecker(collection_name=c_name),
|
||||
Op.json_query: JsonQueryChecker(collection_name=c_name),
|
||||
Op.geo_query: GeoQueryChecker(collection_name=c_name),
|
||||
Op.delete: DeleteChecker(collection_name=c_name),
|
||||
Op.drop: CollectionDropChecker(collection_name=c_name),
|
||||
Op.alter_collection: AlterCollectionChecker(collection_name=c_name),
|
||||
Op.add_field: AddFieldChecker(collection_name=c_name),
|
||||
Op.rename_collection: CollectionRenameChecker(collection_name=c_name)
|
||||
Op.create: CollectionCreateChecker(collection_name=c_name, schema=schema),
|
||||
Op.insert: InsertChecker(collection_name=c_name, schema=schema),
|
||||
Op.upsert: UpsertChecker(collection_name=c_name, schema=schema),
|
||||
Op.partial_update: PartialUpdateChecker(collection_name=c_name, schema=schema),
|
||||
Op.flush: FlushChecker(collection_name=c_name, schema=schema),
|
||||
Op.index: IndexCreateChecker(collection_name=c_name, schema=schema),
|
||||
Op.search: SearchChecker(collection_name=c_name, schema=schema),
|
||||
Op.full_text_search: FullTextSearchChecker(collection_name=c_name, schema=schema),
|
||||
Op.hybrid_search: HybridSearchChecker(collection_name=c_name, schema=schema),
|
||||
Op.query: QueryChecker(collection_name=c_name, schema=schema),
|
||||
Op.text_match: TextMatchChecker(collection_name=c_name, schema=schema),
|
||||
Op.phrase_match: PhraseMatchChecker(collection_name=c_name, schema=schema),
|
||||
Op.json_query: JsonQueryChecker(collection_name=c_name, schema=schema),
|
||||
Op.geo_query: GeoQueryChecker(collection_name=c_name, schema=schema),
|
||||
Op.delete: DeleteChecker(collection_name=c_name, schema=schema),
|
||||
Op.drop: CollectionDropChecker(collection_name=c_name, schema=schema),
|
||||
Op.alter_collection: AlterCollectionChecker(collection_name=c_name, schema=schema),
|
||||
Op.add_field: AddFieldChecker(collection_name=c_name, schema=schema),
|
||||
Op.rename_collection: CollectionRenameChecker(collection_name=c_name, schema=schema),
|
||||
}
|
||||
self.health_checkers = checkers
|
||||
|
||||
@@ -96,7 +128,7 @@ class TestOperations(TestBase):
|
||||
def test_operations(self, request_duration, is_check):
|
||||
# start the monitor threads to check the milvus ops
|
||||
log.info("*********************Test Start**********************")
|
||||
log.info(connections.get_connection_addr('default'))
|
||||
log.info(connections.get_connection_addr("default"))
|
||||
event_records = EventRecords()
|
||||
c_name = None
|
||||
event_records.insert("init_health_checkers", "start")
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,276 @@
|
||||
"""
|
||||
CDC sync tests for collection property operations.
|
||||
"""
|
||||
|
||||
import time
|
||||
|
||||
from .base import TestCDCSyncBase, logger
|
||||
|
||||
|
||||
class TestCDCSyncCollectionProperties(TestCDCSyncBase):
|
||||
"""Test CDC sync for collection property (alter/drop) operations."""
|
||||
|
||||
def setup_method(self):
|
||||
"""Setup for each test method."""
|
||||
self.resources_to_cleanup = []
|
||||
|
||||
def teardown_method(self):
|
||||
"""Cleanup after each test method - only cleanup upstream, downstream will sync."""
|
||||
upstream_client = getattr(self, "_upstream_client", None)
|
||||
|
||||
if upstream_client:
|
||||
for resource_type, resource_name in self.resources_to_cleanup:
|
||||
if resource_type == "collection":
|
||||
self.cleanup_collection(upstream_client, resource_name)
|
||||
|
||||
time.sleep(1) # Allow cleanup to sync to downstream
|
||||
|
||||
def _create_basic_collection(self, client, c_name):
|
||||
"""Create a default schema collection, insert 100 rows, and create HNSW index."""
|
||||
schema = self.create_default_schema(client)
|
||||
client.create_collection(
|
||||
collection_name=c_name,
|
||||
schema=schema,
|
||||
consistency_level="Strong",
|
||||
)
|
||||
|
||||
# Insert 100 rows
|
||||
test_data = self.generate_test_data(100)
|
||||
client.insert(c_name, test_data)
|
||||
|
||||
# Create HNSW index on vector field
|
||||
index_params = client.prepare_index_params()
|
||||
index_params.add_index(
|
||||
field_name="vector",
|
||||
index_type="HNSW",
|
||||
metric_type="L2",
|
||||
params={"M": 8, "efConstruction": 64},
|
||||
)
|
||||
client.create_index(c_name, index_params)
|
||||
|
||||
def test_ttl_sync(self, upstream_client, downstream_client, sync_timeout):
|
||||
"""Test ALTER_COLLECTION_PROPERTIES (TTL) sync."""
|
||||
# Store upstream client for teardown
|
||||
self._upstream_client = upstream_client
|
||||
|
||||
c_name = self.gen_unique_name("test_col_ttl")
|
||||
self.resources_to_cleanup.append(("collection", c_name))
|
||||
|
||||
# Initial cleanup
|
||||
self.cleanup_collection(upstream_client, c_name)
|
||||
|
||||
# Create collection
|
||||
self._create_basic_collection(upstream_client, c_name)
|
||||
|
||||
# Wait for collection to sync downstream
|
||||
def check_create():
|
||||
return downstream_client.has_collection(c_name)
|
||||
|
||||
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {c_name}")
|
||||
|
||||
# Alter collection TTL property
|
||||
upstream_client.alter_collection_properties(
|
||||
collection_name=c_name,
|
||||
properties={"collection.ttl.seconds": "3600"},
|
||||
)
|
||||
|
||||
# Wait for property to sync downstream
|
||||
def check_ttl():
|
||||
try:
|
||||
desc = downstream_client.describe_collection(c_name)
|
||||
props = desc.get("properties", {})
|
||||
logger.info(f"Downstream collection properties: {props}")
|
||||
return str(props.get("collection.ttl.seconds", "")) == "3600"
|
||||
except Exception as e:
|
||||
logger.warning(f"Check TTL sync failed: {e}")
|
||||
return False
|
||||
|
||||
assert self.wait_for_sync(check_ttl, sync_timeout, f"TTL property sync for {c_name}")
|
||||
|
||||
def test_mmap_sync(self, upstream_client, downstream_client, sync_timeout):
|
||||
"""Test ALTER_COLLECTION_PROPERTIES (mmap.enabled) sync."""
|
||||
# Store upstream client for teardown
|
||||
self._upstream_client = upstream_client
|
||||
|
||||
c_name = self.gen_unique_name("test_col_mmap")
|
||||
self.resources_to_cleanup.append(("collection", c_name))
|
||||
|
||||
# Initial cleanup
|
||||
self.cleanup_collection(upstream_client, c_name)
|
||||
|
||||
# Create collection
|
||||
self._create_basic_collection(upstream_client, c_name)
|
||||
|
||||
# Wait for collection to sync downstream
|
||||
def check_create():
|
||||
return downstream_client.has_collection(c_name)
|
||||
|
||||
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {c_name}")
|
||||
|
||||
# Alter mmap.enabled property
|
||||
upstream_client.alter_collection_properties(
|
||||
collection_name=c_name,
|
||||
properties={"mmap.enabled": "true"},
|
||||
)
|
||||
|
||||
# Wait for property to sync downstream
|
||||
def check_mmap():
|
||||
try:
|
||||
desc = downstream_client.describe_collection(c_name)
|
||||
props = desc.get("properties", {})
|
||||
logger.info(f"Downstream collection properties: {props}")
|
||||
return str(props.get("mmap.enabled", "")).lower() == "true"
|
||||
except Exception as e:
|
||||
logger.warning(f"Check mmap sync failed: {e}")
|
||||
return False
|
||||
|
||||
assert self.wait_for_sync(check_mmap, sync_timeout, f"mmap property sync for {c_name}")
|
||||
|
||||
def test_autocompaction_sync(self, upstream_client, downstream_client, sync_timeout):
|
||||
"""Test ALTER_COLLECTION_PROPERTIES (autocompaction.enabled) sync."""
|
||||
# Store upstream client for teardown
|
||||
self._upstream_client = upstream_client
|
||||
|
||||
c_name = self.gen_unique_name("test_col_autocomp")
|
||||
self.resources_to_cleanup.append(("collection", c_name))
|
||||
|
||||
# Initial cleanup
|
||||
self.cleanup_collection(upstream_client, c_name)
|
||||
|
||||
# Create collection
|
||||
self._create_basic_collection(upstream_client, c_name)
|
||||
|
||||
# Wait for collection to sync downstream
|
||||
def check_create():
|
||||
return downstream_client.has_collection(c_name)
|
||||
|
||||
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {c_name}")
|
||||
|
||||
# Alter autocompaction property
|
||||
upstream_client.alter_collection_properties(
|
||||
collection_name=c_name,
|
||||
properties={"collection.autocompaction.enabled": "true"},
|
||||
)
|
||||
|
||||
# Wait for property to sync downstream
|
||||
def check_autocomp():
|
||||
try:
|
||||
desc = downstream_client.describe_collection(c_name)
|
||||
props = desc.get("properties", {})
|
||||
logger.info(f"Downstream collection properties: {props}")
|
||||
return str(props.get("collection.autocompaction.enabled", "")).lower() == "true"
|
||||
except Exception as e:
|
||||
logger.warning(f"Check autocompaction sync failed: {e}")
|
||||
return False
|
||||
|
||||
assert self.wait_for_sync(check_autocomp, sync_timeout, f"autocompaction property sync for {c_name}")
|
||||
|
||||
def test_alter_multiple_properties(self, upstream_client, downstream_client, sync_timeout):
|
||||
"""Test ALTER_COLLECTION_PROPERTIES with multiple properties at once sync."""
|
||||
# Store upstream client for teardown
|
||||
self._upstream_client = upstream_client
|
||||
|
||||
c_name = self.gen_unique_name("test_col_multi_props")
|
||||
self.resources_to_cleanup.append(("collection", c_name))
|
||||
|
||||
# Initial cleanup
|
||||
self.cleanup_collection(upstream_client, c_name)
|
||||
|
||||
# Create collection
|
||||
self._create_basic_collection(upstream_client, c_name)
|
||||
|
||||
# Wait for collection to sync downstream
|
||||
def check_create():
|
||||
return downstream_client.has_collection(c_name)
|
||||
|
||||
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {c_name}")
|
||||
|
||||
# Alter all 3 properties at once
|
||||
upstream_client.alter_collection_properties(
|
||||
collection_name=c_name,
|
||||
properties={
|
||||
"collection.ttl.seconds": "3600",
|
||||
"mmap.enabled": "true",
|
||||
"collection.autocompaction.enabled": "true",
|
||||
},
|
||||
)
|
||||
|
||||
# Wait for all 3 properties to sync downstream
|
||||
def check_all_props():
|
||||
try:
|
||||
desc = downstream_client.describe_collection(c_name)
|
||||
props = desc.get("properties", {})
|
||||
logger.info(f"Downstream collection properties: {props}")
|
||||
ttl_ok = str(props.get("collection.ttl.seconds", "")) == "3600"
|
||||
mmap_ok = str(props.get("mmap.enabled", "")).lower() == "true"
|
||||
autocomp_ok = str(props.get("collection.autocompaction.enabled", "")).lower() == "true"
|
||||
return ttl_ok and mmap_ok and autocomp_ok
|
||||
except Exception as e:
|
||||
logger.warning(f"Check all properties sync failed: {e}")
|
||||
return False
|
||||
|
||||
assert self.wait_for_sync(check_all_props, sync_timeout, f"all properties sync for {c_name}")
|
||||
|
||||
def test_drop_properties_sync(self, upstream_client, downstream_client, sync_timeout):
|
||||
"""Test DROP_COLLECTION_PROPERTIES — set TTL + mmap, drop TTL, verify TTL gone and mmap remains."""
|
||||
# Store upstream client for teardown
|
||||
self._upstream_client = upstream_client
|
||||
|
||||
c_name = self.gen_unique_name("test_col_drop_props")
|
||||
self.resources_to_cleanup.append(("collection", c_name))
|
||||
|
||||
# Initial cleanup
|
||||
self.cleanup_collection(upstream_client, c_name)
|
||||
|
||||
# Create collection
|
||||
self._create_basic_collection(upstream_client, c_name)
|
||||
|
||||
# Wait for collection to sync downstream
|
||||
def check_create():
|
||||
return downstream_client.has_collection(c_name)
|
||||
|
||||
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {c_name}")
|
||||
|
||||
# Set TTL and mmap properties
|
||||
upstream_client.alter_collection_properties(
|
||||
collection_name=c_name,
|
||||
properties={
|
||||
"collection.ttl.seconds": "3600",
|
||||
"mmap.enabled": "true",
|
||||
},
|
||||
)
|
||||
|
||||
# Wait for both properties to sync downstream
|
||||
def check_props_set():
|
||||
try:
|
||||
desc = downstream_client.describe_collection(c_name)
|
||||
props = desc.get("properties", {})
|
||||
ttl_set = str(props.get("collection.ttl.seconds", "")) == "3600"
|
||||
mmap_set = str(props.get("mmap.enabled", "")).lower() == "true"
|
||||
return ttl_set and mmap_set
|
||||
except Exception as e:
|
||||
logger.warning(f"Check properties set failed: {e}")
|
||||
return False
|
||||
|
||||
assert self.wait_for_sync(check_props_set, sync_timeout, f"set properties for {c_name}")
|
||||
|
||||
# Drop TTL property
|
||||
upstream_client.drop_collection_properties(
|
||||
collection_name=c_name,
|
||||
property_keys=["collection.ttl.seconds"],
|
||||
)
|
||||
|
||||
# Wait for TTL to be gone but mmap to remain on downstream
|
||||
def check_drop_ttl():
|
||||
try:
|
||||
desc = downstream_client.describe_collection(c_name)
|
||||
props = desc.get("properties", {})
|
||||
logger.info(f"Downstream collection properties after drop: {props}")
|
||||
ttl_gone = "collection.ttl.seconds" not in props
|
||||
mmap_remains = str(props.get("mmap.enabled", "")).lower() == "true"
|
||||
return ttl_gone and mmap_remains
|
||||
except Exception as e:
|
||||
logger.warning(f"Check drop TTL sync failed: {e}")
|
||||
return False
|
||||
|
||||
assert self.wait_for_sync(check_drop_ttl, sync_timeout, f"drop TTL property sync for {c_name}")
|
||||
@@ -0,0 +1,715 @@
|
||||
"""
|
||||
CDC sync tests for full-text search (BM25), text match, and phrase match operations.
|
||||
"""
|
||||
|
||||
import random
|
||||
import time
|
||||
|
||||
import pytest
|
||||
from pymilvus import AnnSearchRequest, DataType, RRFRanker
|
||||
|
||||
from .base import TestCDCSyncBase, logger
|
||||
|
||||
|
||||
class TestCDCSyncFTSAndText(TestCDCSyncBase):
|
||||
"""Test CDC sync for full-text search, text match, and phrase match operations."""
|
||||
|
||||
def setup_method(self):
|
||||
"""Setup for each test method."""
|
||||
self.resources_to_cleanup = []
|
||||
|
||||
def teardown_method(self):
|
||||
"""Cleanup after each test method - only cleanup upstream, downstream will sync."""
|
||||
upstream_client = getattr(self, "_upstream_client", None)
|
||||
|
||||
if upstream_client:
|
||||
for resource_type, resource_name in self.resources_to_cleanup:
|
||||
if resource_type == "collection":
|
||||
self.cleanup_collection(upstream_client, resource_name)
|
||||
|
||||
time.sleep(1) # Allow cleanup to sync to downstream
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Test 1: FTS insert and search replication
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
@pytest.mark.parametrize("analyzer_type", ["standard", "english"])
|
||||
def test_fts_insert_and_search(
|
||||
self,
|
||||
upstream_client,
|
||||
downstream_client,
|
||||
sync_timeout,
|
||||
analyzer_type,
|
||||
):
|
||||
"""Test FTS (BM25) insert and search replication.
|
||||
|
||||
Creates an FTS collection with BM25 function, inserts 200 docs, creates
|
||||
SPARSE_INVERTED_INDEX (BM25) and HNSW indexes, then verifies that FTS search
|
||||
results replicate to downstream with sufficient overlap.
|
||||
"""
|
||||
start_time = time.time()
|
||||
collection_name = self.gen_unique_name("fts_search", max_length=50)
|
||||
|
||||
self.log_test_start(
|
||||
"test_fts_insert_and_search",
|
||||
f"FTS_INSERT_SEARCH(analyzer={analyzer_type})",
|
||||
collection_name,
|
||||
)
|
||||
|
||||
self._upstream_client = upstream_client
|
||||
self.resources_to_cleanup.append(("collection", collection_name))
|
||||
|
||||
try:
|
||||
# Initial cleanup
|
||||
self.cleanup_collection(upstream_client, collection_name)
|
||||
|
||||
# Create FTS schema and collection
|
||||
logger.info(f"[CREATE] Creating FTS collection '{collection_name}' with analyzer_type='{analyzer_type}'")
|
||||
schema = self.create_fts_schema(upstream_client, analyzer_type)
|
||||
upstream_client.create_collection(collection_name, schema=schema)
|
||||
|
||||
# Insert 200 documents
|
||||
fts_data = self.generate_fts_data(200)
|
||||
logger.info(f"[INSERT] Inserting {len(fts_data)} FTS documents upstream")
|
||||
result = upstream_client.insert(collection_name, fts_data)
|
||||
inserted_count = result.get("insert_count", len(fts_data))
|
||||
logger.info(f"[INSERT] Inserted {inserted_count} documents")
|
||||
|
||||
upstream_client.flush(collection_name)
|
||||
|
||||
# Create SPARSE_INVERTED_INDEX on sparse_output (BM25)
|
||||
index_params = upstream_client.prepare_index_params()
|
||||
index_params.add_index(
|
||||
field_name="sparse_output",
|
||||
index_type="SPARSE_INVERTED_INDEX",
|
||||
metric_type="BM25",
|
||||
params={"bm25_k1": 1.5, "bm25_b": 0.75},
|
||||
)
|
||||
# Create HNSW on dense_vector
|
||||
index_params.add_index(
|
||||
field_name="dense_vector",
|
||||
index_type="HNSW",
|
||||
metric_type="L2",
|
||||
params={"M": 8, "efConstruction": 64},
|
||||
)
|
||||
upstream_client.create_index(collection_name, index_params)
|
||||
upstream_client.load_collection(collection_name)
|
||||
|
||||
# Run FTS search on upstream
|
||||
query_text = "vector database similarity search"
|
||||
logger.info(f"[SEARCH] Running FTS search upstream with query: '{query_text}'")
|
||||
upstream_results = upstream_client.search(
|
||||
collection_name,
|
||||
data=[query_text],
|
||||
anns_field="sparse_output",
|
||||
limit=10,
|
||||
search_params={"metric_type": "BM25"},
|
||||
output_fields=["text_field", "category"],
|
||||
)
|
||||
upstream_ids = set()
|
||||
if upstream_results and len(upstream_results) > 0:
|
||||
for hit in upstream_results[0]:
|
||||
upstream_ids.add(hit.get("id") or hit.id)
|
||||
|
||||
logger.info(f"[SEARCH] Upstream FTS returned {len(upstream_ids)} results")
|
||||
|
||||
# Wait for collection to appear downstream
|
||||
def check_collection_exists():
|
||||
return downstream_client.has_collection(collection_name)
|
||||
|
||||
assert self.wait_for_sync(
|
||||
check_collection_exists,
|
||||
sync_timeout,
|
||||
f"collection '{collection_name}' creation sync",
|
||||
), f"Collection '{collection_name}' did not sync to downstream"
|
||||
|
||||
# Wait for data + index to sync and downstream search results overlap
|
||||
def check_fts_overlap():
|
||||
try:
|
||||
ds_results = downstream_client.search(
|
||||
collection_name,
|
||||
data=[query_text],
|
||||
anns_field="sparse_output",
|
||||
limit=10,
|
||||
search_params={"metric_type": "BM25"},
|
||||
output_fields=["text_field", "category"],
|
||||
)
|
||||
if not ds_results or len(ds_results) == 0:
|
||||
return False
|
||||
downstream_ids = set()
|
||||
for hit in ds_results[0]:
|
||||
downstream_ids.add(hit.get("id") or hit.id)
|
||||
if not downstream_ids:
|
||||
return False
|
||||
if len(upstream_ids) == 0:
|
||||
return len(downstream_ids) > 0
|
||||
overlap = len(upstream_ids & downstream_ids) / max(len(upstream_ids), 1)
|
||||
logger.info(
|
||||
f"[OVERLAP] FTS search overlap: {overlap:.2f} "
|
||||
f"(upstream={len(upstream_ids)}, downstream={len(downstream_ids)})"
|
||||
)
|
||||
return overlap >= self.SEARCH_OVERLAP_THRESHOLD
|
||||
except Exception as e:
|
||||
logger.warning(f"FTS overlap check failed: {e}")
|
||||
return False
|
||||
|
||||
assert self.wait_for_sync(
|
||||
check_fts_overlap,
|
||||
sync_timeout,
|
||||
f"FTS search overlap sync (analyzer={analyzer_type})",
|
||||
), f"FTS search results did not reach overlap threshold {self.SEARCH_OVERLAP_THRESHOLD} on downstream"
|
||||
|
||||
duration = time.time() - start_time
|
||||
self.log_test_end("test_fts_insert_and_search", True, duration)
|
||||
|
||||
except Exception as exc:
|
||||
duration = time.time() - start_time
|
||||
self.log_test_end("test_fts_insert_and_search", False, duration)
|
||||
raise exc
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Test 2: TEXT_MATCH sync
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
@pytest.mark.parametrize("analyzer_type", ["standard", "english"])
|
||||
def test_text_match_sync(
|
||||
self,
|
||||
upstream_client,
|
||||
downstream_client,
|
||||
sync_timeout,
|
||||
analyzer_type,
|
||||
):
|
||||
"""Test TEXT_MATCH query replication.
|
||||
|
||||
Creates a collection with a VARCHAR field that has analyzer + match enabled,
|
||||
inserts 100 rows, and verifies that TEXT_MATCH queries return the same count
|
||||
on both upstream and downstream.
|
||||
"""
|
||||
start_time = time.time()
|
||||
collection_name = self.gen_unique_name("text_match", max_length=50)
|
||||
|
||||
self.log_test_start(
|
||||
"test_text_match_sync",
|
||||
f"TEXT_MATCH(analyzer={analyzer_type})",
|
||||
collection_name,
|
||||
)
|
||||
|
||||
self._upstream_client = upstream_client
|
||||
self.resources_to_cleanup.append(("collection", collection_name))
|
||||
|
||||
try:
|
||||
self.cleanup_collection(upstream_client, collection_name)
|
||||
|
||||
# Build schema with analyzer-enabled VARCHAR
|
||||
schema = upstream_client.create_schema(enable_dynamic_field=False)
|
||||
schema.add_field("id", DataType.INT64, is_primary=True, auto_id=True)
|
||||
schema.add_field("dense_vector", DataType.FLOAT_VECTOR, dim=128)
|
||||
schema.add_field(
|
||||
"text_field",
|
||||
DataType.VARCHAR,
|
||||
max_length=2048,
|
||||
enable_analyzer=True,
|
||||
enable_match=True,
|
||||
analyzer_params={"type": analyzer_type},
|
||||
)
|
||||
|
||||
logger.info(f"[CREATE] Creating text-match collection '{collection_name}'")
|
||||
upstream_client.create_collection(collection_name, schema=schema)
|
||||
|
||||
# Insert 100 rows from FTS_SENTENCES
|
||||
data = []
|
||||
for i in range(100):
|
||||
data.append(
|
||||
{
|
||||
"dense_vector": [random.random() for _ in range(128)],
|
||||
"text_field": self.FTS_SENTENCES[i % len(self.FTS_SENTENCES)],
|
||||
}
|
||||
)
|
||||
|
||||
logger.info(f"[INSERT] Inserting {len(data)} rows upstream")
|
||||
upstream_client.insert(collection_name, data)
|
||||
upstream_client.flush(collection_name)
|
||||
|
||||
# Create HNSW index and load
|
||||
index_params = upstream_client.prepare_index_params()
|
||||
index_params.add_index(
|
||||
field_name="dense_vector",
|
||||
index_type="HNSW",
|
||||
metric_type="L2",
|
||||
params={"M": 8, "efConstruction": 64},
|
||||
)
|
||||
upstream_client.create_index(collection_name, index_params)
|
||||
upstream_client.load_collection(collection_name)
|
||||
|
||||
# Run TEXT_MATCH query on upstream
|
||||
filter_expr = "TEXT_MATCH(text_field, 'vector database')"
|
||||
logger.info(f"[QUERY] Upstream TEXT_MATCH query: {filter_expr}")
|
||||
upstream_results = upstream_client.query(
|
||||
collection_name,
|
||||
filter=filter_expr,
|
||||
output_fields=["id", "text_field"],
|
||||
limit=100,
|
||||
)
|
||||
upstream_count = len(upstream_results)
|
||||
logger.info(f"[QUERY] Upstream TEXT_MATCH returned {upstream_count} rows")
|
||||
|
||||
# Wait for collection to appear downstream
|
||||
def check_collection_exists():
|
||||
return downstream_client.has_collection(collection_name)
|
||||
|
||||
assert self.wait_for_sync(
|
||||
check_collection_exists,
|
||||
sync_timeout,
|
||||
f"collection '{collection_name}' creation sync",
|
||||
)
|
||||
|
||||
# Wait for downstream query count to match
|
||||
def check_count_match():
|
||||
try:
|
||||
ds_results = downstream_client.query(
|
||||
collection_name,
|
||||
filter=filter_expr,
|
||||
output_fields=["id", "text_field"],
|
||||
limit=100,
|
||||
)
|
||||
ds_count = len(ds_results)
|
||||
logger.info(f"[VERIFY] TEXT_MATCH downstream count={ds_count}, upstream count={upstream_count}")
|
||||
return ds_count == upstream_count
|
||||
except Exception as e:
|
||||
logger.warning(f"TEXT_MATCH count check failed: {e}")
|
||||
return False
|
||||
|
||||
assert self.wait_for_sync(
|
||||
check_count_match,
|
||||
sync_timeout,
|
||||
f"TEXT_MATCH query count sync (analyzer={analyzer_type})",
|
||||
), f"TEXT_MATCH query count mismatch between upstream ({upstream_count}) and downstream after timeout"
|
||||
|
||||
duration = time.time() - start_time
|
||||
self.log_test_end("test_text_match_sync", True, duration)
|
||||
|
||||
except Exception as exc:
|
||||
duration = time.time() - start_time
|
||||
self.log_test_end("test_text_match_sync", False, duration)
|
||||
raise exc
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Test 3: PHRASE_MATCH sync
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
@pytest.mark.parametrize("analyzer_type", ["standard", "english"])
|
||||
def test_phrase_match_sync(
|
||||
self,
|
||||
upstream_client,
|
||||
downstream_client,
|
||||
sync_timeout,
|
||||
analyzer_type,
|
||||
):
|
||||
"""Test PHRASE_MATCH query replication.
|
||||
|
||||
Same collection setup as text_match. Queries PHRASE_MATCH with slop=1 and
|
||||
verifies that upstream and downstream return the same count.
|
||||
"""
|
||||
start_time = time.time()
|
||||
collection_name = self.gen_unique_name("phrase_match", max_length=50)
|
||||
|
||||
self.log_test_start(
|
||||
"test_phrase_match_sync",
|
||||
f"PHRASE_MATCH(analyzer={analyzer_type})",
|
||||
collection_name,
|
||||
)
|
||||
|
||||
self._upstream_client = upstream_client
|
||||
self.resources_to_cleanup.append(("collection", collection_name))
|
||||
|
||||
try:
|
||||
self.cleanup_collection(upstream_client, collection_name)
|
||||
|
||||
# Build schema with analyzer-enabled VARCHAR
|
||||
schema = upstream_client.create_schema(enable_dynamic_field=False)
|
||||
schema.add_field("id", DataType.INT64, is_primary=True, auto_id=True)
|
||||
schema.add_field("dense_vector", DataType.FLOAT_VECTOR, dim=128)
|
||||
schema.add_field(
|
||||
"text_field",
|
||||
DataType.VARCHAR,
|
||||
max_length=2048,
|
||||
enable_analyzer=True,
|
||||
enable_match=True,
|
||||
analyzer_params={"type": analyzer_type},
|
||||
)
|
||||
|
||||
logger.info(f"[CREATE] Creating phrase-match collection '{collection_name}'")
|
||||
upstream_client.create_collection(collection_name, schema=schema)
|
||||
|
||||
# Insert 100 rows from FTS_SENTENCES
|
||||
data = []
|
||||
for i in range(100):
|
||||
data.append(
|
||||
{
|
||||
"dense_vector": [random.random() for _ in range(128)],
|
||||
"text_field": self.FTS_SENTENCES[i % len(self.FTS_SENTENCES)],
|
||||
}
|
||||
)
|
||||
|
||||
logger.info(f"[INSERT] Inserting {len(data)} rows upstream")
|
||||
upstream_client.insert(collection_name, data)
|
||||
upstream_client.flush(collection_name)
|
||||
|
||||
# Create HNSW index and load
|
||||
index_params = upstream_client.prepare_index_params()
|
||||
index_params.add_index(
|
||||
field_name="dense_vector",
|
||||
index_type="HNSW",
|
||||
metric_type="L2",
|
||||
params={"M": 8, "efConstruction": 64},
|
||||
)
|
||||
upstream_client.create_index(collection_name, index_params)
|
||||
upstream_client.load_collection(collection_name)
|
||||
|
||||
# Run PHRASE_MATCH query on upstream (slop=1)
|
||||
filter_expr = "PHRASE_MATCH(text_field, 'brown fox', 1)"
|
||||
logger.info(f"[QUERY] Upstream PHRASE_MATCH query: {filter_expr}")
|
||||
upstream_results = upstream_client.query(
|
||||
collection_name,
|
||||
filter=filter_expr,
|
||||
output_fields=["id", "text_field"],
|
||||
limit=100,
|
||||
)
|
||||
upstream_count = len(upstream_results)
|
||||
logger.info(f"[QUERY] Upstream PHRASE_MATCH returned {upstream_count} rows")
|
||||
|
||||
# Wait for collection to appear downstream
|
||||
def check_collection_exists():
|
||||
return downstream_client.has_collection(collection_name)
|
||||
|
||||
assert self.wait_for_sync(
|
||||
check_collection_exists,
|
||||
sync_timeout,
|
||||
f"collection '{collection_name}' creation sync",
|
||||
)
|
||||
|
||||
# Wait for downstream query count to match
|
||||
def check_count_match():
|
||||
try:
|
||||
ds_results = downstream_client.query(
|
||||
collection_name,
|
||||
filter=filter_expr,
|
||||
output_fields=["id", "text_field"],
|
||||
limit=100,
|
||||
)
|
||||
ds_count = len(ds_results)
|
||||
logger.info(f"[VERIFY] PHRASE_MATCH downstream count={ds_count}, upstream count={upstream_count}")
|
||||
return ds_count == upstream_count
|
||||
except Exception as e:
|
||||
logger.warning(f"PHRASE_MATCH count check failed: {e}")
|
||||
return False
|
||||
|
||||
assert self.wait_for_sync(
|
||||
check_count_match,
|
||||
sync_timeout,
|
||||
f"PHRASE_MATCH query count sync (analyzer={analyzer_type})",
|
||||
), f"PHRASE_MATCH query count mismatch between upstream ({upstream_count}) and downstream after timeout"
|
||||
|
||||
duration = time.time() - start_time
|
||||
self.log_test_end("test_phrase_match_sync", True, duration)
|
||||
|
||||
except Exception as exc:
|
||||
duration = time.time() - start_time
|
||||
self.log_test_end("test_phrase_match_sync", False, duration)
|
||||
raise exc
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Test 4: Hybrid search (FTS + dense) replication
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
def test_hybrid_search_fts_dense(
|
||||
self,
|
||||
upstream_client,
|
||||
downstream_client,
|
||||
sync_timeout,
|
||||
):
|
||||
"""Test hybrid search (BM25 sparse + HNSW dense) replication.
|
||||
|
||||
Creates an FTS collection, inserts 300 docs, builds both sparse (BM25) and
|
||||
dense (HNSW) indexes, and verifies that hybrid search results on downstream
|
||||
have sufficient overlap with upstream results.
|
||||
"""
|
||||
start_time = time.time()
|
||||
collection_name = self.gen_unique_name("hybrid_fts", max_length=50)
|
||||
|
||||
self.log_test_start(
|
||||
"test_hybrid_search_fts_dense",
|
||||
"HYBRID_SEARCH_FTS_DENSE",
|
||||
collection_name,
|
||||
)
|
||||
|
||||
self._upstream_client = upstream_client
|
||||
self.resources_to_cleanup.append(("collection", collection_name))
|
||||
|
||||
try:
|
||||
self.cleanup_collection(upstream_client, collection_name)
|
||||
|
||||
# Create FTS schema (standard analyzer)
|
||||
schema = self.create_fts_schema(upstream_client, "standard")
|
||||
logger.info(f"[CREATE] Creating hybrid-search collection '{collection_name}'")
|
||||
upstream_client.create_collection(collection_name, schema=schema)
|
||||
|
||||
# Insert 300 documents
|
||||
fts_data = self.generate_fts_data(300)
|
||||
logger.info(f"[INSERT] Inserting {len(fts_data)} documents upstream")
|
||||
upstream_client.insert(collection_name, fts_data)
|
||||
upstream_client.flush(collection_name)
|
||||
|
||||
# Create both indexes
|
||||
index_params = upstream_client.prepare_index_params()
|
||||
index_params.add_index(
|
||||
field_name="sparse_output",
|
||||
index_type="SPARSE_INVERTED_INDEX",
|
||||
metric_type="BM25",
|
||||
params={"bm25_k1": 1.5, "bm25_b": 0.75},
|
||||
)
|
||||
index_params.add_index(
|
||||
field_name="dense_vector",
|
||||
index_type="HNSW",
|
||||
metric_type="L2",
|
||||
params={"M": 8, "efConstruction": 64},
|
||||
)
|
||||
upstream_client.create_index(collection_name, index_params)
|
||||
upstream_client.load_collection(collection_name)
|
||||
|
||||
# Build hybrid search requests
|
||||
sparse_req = AnnSearchRequest(
|
||||
data=["vector database similarity search"],
|
||||
anns_field="sparse_output",
|
||||
param={"metric_type": "BM25"},
|
||||
limit=10,
|
||||
)
|
||||
dense_query_vec = [random.random() for _ in range(128)]
|
||||
dense_req = AnnSearchRequest(
|
||||
data=[dense_query_vec],
|
||||
anns_field="dense_vector",
|
||||
param={"metric_type": "L2", "params": {"ef": 64}},
|
||||
limit=10,
|
||||
)
|
||||
|
||||
logger.info("[SEARCH] Running hybrid search on upstream")
|
||||
upstream_results = upstream_client.hybrid_search(
|
||||
collection_name,
|
||||
reqs=[sparse_req, dense_req],
|
||||
ranker=RRFRanker(),
|
||||
limit=10,
|
||||
output_fields=["text_field", "category"],
|
||||
)
|
||||
|
||||
upstream_ids = set()
|
||||
if upstream_results and len(upstream_results) > 0:
|
||||
for hit in upstream_results[0]:
|
||||
upstream_ids.add(hit.get("id") or hit.id)
|
||||
|
||||
logger.info(f"[SEARCH] Upstream hybrid search returned {len(upstream_ids)} results")
|
||||
|
||||
# Wait for collection to appear downstream
|
||||
def check_collection_exists():
|
||||
return downstream_client.has_collection(collection_name)
|
||||
|
||||
assert self.wait_for_sync(
|
||||
check_collection_exists,
|
||||
sync_timeout,
|
||||
f"collection '{collection_name}' creation sync",
|
||||
)
|
||||
|
||||
# Wait for downstream hybrid search overlap
|
||||
def check_hybrid_overlap():
|
||||
try:
|
||||
ds_sparse_req = AnnSearchRequest(
|
||||
data=["vector database similarity search"],
|
||||
anns_field="sparse_output",
|
||||
param={"metric_type": "BM25"},
|
||||
limit=10,
|
||||
)
|
||||
ds_dense_req = AnnSearchRequest(
|
||||
data=[dense_query_vec],
|
||||
anns_field="dense_vector",
|
||||
param={"metric_type": "L2", "params": {"ef": 64}},
|
||||
limit=10,
|
||||
)
|
||||
ds_results = downstream_client.hybrid_search(
|
||||
collection_name,
|
||||
reqs=[ds_sparse_req, ds_dense_req],
|
||||
ranker=RRFRanker(),
|
||||
limit=10,
|
||||
output_fields=["text_field", "category"],
|
||||
)
|
||||
if not ds_results or len(ds_results) == 0:
|
||||
return False
|
||||
downstream_ids = set()
|
||||
for hit in ds_results[0]:
|
||||
downstream_ids.add(hit.get("id") or hit.id)
|
||||
if not downstream_ids:
|
||||
return False
|
||||
if len(upstream_ids) == 0:
|
||||
return len(downstream_ids) > 0
|
||||
overlap = len(upstream_ids & downstream_ids) / max(len(upstream_ids), 1)
|
||||
logger.info(
|
||||
f"[OVERLAP] Hybrid search overlap: {overlap:.2f} "
|
||||
f"(upstream={len(upstream_ids)}, downstream={len(downstream_ids)})"
|
||||
)
|
||||
return overlap >= self.SEARCH_OVERLAP_THRESHOLD
|
||||
except Exception as e:
|
||||
logger.warning(f"Hybrid overlap check failed: {e}")
|
||||
return False
|
||||
|
||||
assert self.wait_for_sync(
|
||||
check_hybrid_overlap,
|
||||
sync_timeout,
|
||||
"hybrid search (FTS + dense) overlap sync",
|
||||
), f"Hybrid search results did not reach overlap threshold {self.SEARCH_OVERLAP_THRESHOLD} on downstream"
|
||||
|
||||
duration = time.time() - start_time
|
||||
self.log_test_end("test_hybrid_search_fts_dense", True, duration)
|
||||
|
||||
except Exception as exc:
|
||||
duration = time.time() - start_time
|
||||
self.log_test_end("test_hybrid_search_fts_dense", False, duration)
|
||||
raise exc
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Test 5: FTS after switchover
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
def test_fts_after_switchover(
|
||||
self,
|
||||
upstream_client,
|
||||
downstream_client,
|
||||
sync_timeout,
|
||||
switchover_helper,
|
||||
source_cluster_id,
|
||||
target_cluster_id,
|
||||
):
|
||||
"""Test FTS replication continues correctly after CDC topology switchover.
|
||||
|
||||
1. Create FTS collection, insert 100 docs, build index, verify sync.
|
||||
2. Perform switchover so downstream becomes the new source.
|
||||
3. Insert 50 more docs into the new source (original downstream).
|
||||
4. Verify FTS search works on the new downstream (original upstream).
|
||||
5. Switch back to original topology.
|
||||
"""
|
||||
start_time = time.time()
|
||||
collection_name = self.gen_unique_name("fts_switchover", max_length=50)
|
||||
|
||||
self.log_test_start(
|
||||
"test_fts_after_switchover",
|
||||
"FTS_AFTER_SWITCHOVER",
|
||||
collection_name,
|
||||
)
|
||||
|
||||
self._upstream_client = upstream_client
|
||||
self.resources_to_cleanup.append(("collection", collection_name))
|
||||
|
||||
try:
|
||||
self.cleanup_collection(upstream_client, collection_name)
|
||||
|
||||
# Phase 1: Create FTS collection and index on original upstream
|
||||
logger.info(f"[PHASE1] Creating FTS collection '{collection_name}' on upstream")
|
||||
schema = self.create_fts_schema(upstream_client, "standard")
|
||||
upstream_client.create_collection(collection_name, schema=schema)
|
||||
|
||||
fts_data = self.generate_fts_data(100)
|
||||
logger.info(f"[PHASE1] Inserting {len(fts_data)} documents upstream")
|
||||
upstream_client.insert(collection_name, fts_data)
|
||||
upstream_client.flush(collection_name)
|
||||
|
||||
index_params = upstream_client.prepare_index_params()
|
||||
index_params.add_index(
|
||||
field_name="sparse_output",
|
||||
index_type="SPARSE_INVERTED_INDEX",
|
||||
metric_type="BM25",
|
||||
params={"bm25_k1": 1.5, "bm25_b": 0.75},
|
||||
)
|
||||
index_params.add_index(
|
||||
field_name="dense_vector",
|
||||
index_type="HNSW",
|
||||
metric_type="L2",
|
||||
params={"M": 8, "efConstruction": 64},
|
||||
)
|
||||
upstream_client.create_index(collection_name, index_params)
|
||||
upstream_client.load_collection(collection_name)
|
||||
|
||||
# Verify initial sync to downstream
|
||||
def check_initial_sync():
|
||||
if not downstream_client.has_collection(collection_name):
|
||||
return False
|
||||
try:
|
||||
ds_results = downstream_client.search(
|
||||
collection_name,
|
||||
data=["vector database"],
|
||||
anns_field="sparse_output",
|
||||
limit=5,
|
||||
search_params={"metric_type": "BM25"},
|
||||
output_fields=["text_field"],
|
||||
)
|
||||
return ds_results is not None and len(ds_results) > 0
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
assert self.wait_for_sync(
|
||||
check_initial_sync,
|
||||
sync_timeout,
|
||||
f"initial FTS sync for '{collection_name}'",
|
||||
), f"Initial FTS sync failed for collection '{collection_name}'"
|
||||
|
||||
logger.info("[PHASE1] Initial FTS sync verified")
|
||||
|
||||
# Phase 2: Switchover — downstream becomes new source
|
||||
logger.info(f"[PHASE2] Switching CDC direction: {target_cluster_id} -> {source_cluster_id}")
|
||||
switchover_helper(target_cluster_id, source_cluster_id)
|
||||
|
||||
# Insert 50 more docs to the new source (original downstream)
|
||||
extra_data = self.generate_fts_data(50)
|
||||
logger.info(f"[PHASE2] Inserting {len(extra_data)} additional docs to new source (downstream_client)")
|
||||
downstream_client.insert(collection_name, extra_data)
|
||||
downstream_client.flush(collection_name)
|
||||
|
||||
# Phase 3: Verify FTS search works on new downstream (original upstream)
|
||||
query_text = "distributed database replication"
|
||||
|
||||
def check_fts_on_new_downstream():
|
||||
try:
|
||||
results = upstream_client.search(
|
||||
collection_name,
|
||||
data=[query_text],
|
||||
anns_field="sparse_output",
|
||||
limit=5,
|
||||
search_params={"metric_type": "BM25"},
|
||||
output_fields=["text_field"],
|
||||
)
|
||||
return results is not None and len(results) > 0
|
||||
except Exception as e:
|
||||
logger.warning(f"FTS on new downstream check failed: {e}")
|
||||
return False
|
||||
|
||||
assert self.wait_for_sync(
|
||||
check_fts_on_new_downstream,
|
||||
sync_timeout,
|
||||
"FTS search on new downstream after switchover",
|
||||
), "FTS search on new downstream (original upstream) failed after switchover"
|
||||
|
||||
logger.info("[PHASE3] FTS search verified on new downstream after switchover")
|
||||
|
||||
# Phase 4: Switch back to original topology
|
||||
logger.info(f"[PHASE4] Switching back to original topology: {source_cluster_id} -> {target_cluster_id}")
|
||||
switchover_helper(source_cluster_id, target_cluster_id)
|
||||
|
||||
duration = time.time() - start_time
|
||||
self.log_test_end("test_fts_after_switchover", True, duration)
|
||||
|
||||
except Exception as exc:
|
||||
# Best-effort restore original topology on failure
|
||||
try:
|
||||
logger.warning("[RECOVER] Attempting to restore original CDC topology after failure")
|
||||
switchover_helper(source_cluster_id, target_cluster_id)
|
||||
except Exception as restore_exc:
|
||||
logger.error(f"[RECOVER] Failed to restore topology: {restore_exc}")
|
||||
duration = time.time() - start_time
|
||||
self.log_test_end("test_fts_after_switchover", False, duration)
|
||||
raise exc
|
||||
@@ -0,0 +1,316 @@
|
||||
"""
|
||||
CDC sync tests for multi-database operations.
|
||||
"""
|
||||
|
||||
import time
|
||||
|
||||
from pymilvus import MilvusClient
|
||||
|
||||
from .base import TestCDCSyncBase, logger
|
||||
|
||||
|
||||
class TestCDCSyncMultiDatabase(TestCDCSyncBase):
|
||||
"""Test CDC sync for operations across multiple databases."""
|
||||
|
||||
def setup_method(self):
|
||||
"""Setup for each test method."""
|
||||
self.resources_to_cleanup = []
|
||||
|
||||
def teardown_method(self):
|
||||
"""Cleanup after each test method - only cleanup upstream, downstream will sync."""
|
||||
upstream_uri = getattr(self, "_upstream_uri", None)
|
||||
upstream_token = getattr(self, "_upstream_token", None)
|
||||
|
||||
if upstream_uri:
|
||||
# Re-create a default-db client for cleanup
|
||||
try:
|
||||
client = MilvusClient(uri=upstream_uri, token=upstream_token)
|
||||
except Exception as e:
|
||||
logger.warning(f"[CLEANUP] Failed to create upstream client: {e}")
|
||||
return
|
||||
|
||||
# Clean up collections in databases first, then databases
|
||||
for resource_type, resource_data in self.resources_to_cleanup:
|
||||
if resource_type == "collection_in_db":
|
||||
db_name, c_name = resource_data
|
||||
try:
|
||||
db_client = MilvusClient(
|
||||
uri=upstream_uri,
|
||||
token=upstream_token,
|
||||
db_name=db_name,
|
||||
)
|
||||
if db_client.has_collection(c_name):
|
||||
logger.info(f"[CLEANUP] Dropping collection {c_name} in db {db_name}")
|
||||
db_client.drop_collection(c_name)
|
||||
db_client.close()
|
||||
except Exception as e:
|
||||
logger.warning(f"[CLEANUP] Failed to drop collection {c_name} in db {db_name}: {e}")
|
||||
|
||||
for resource_type, resource_data in self.resources_to_cleanup:
|
||||
if resource_type == "database":
|
||||
db_name = resource_data
|
||||
self.cleanup_database(client, db_name)
|
||||
|
||||
client.close()
|
||||
time.sleep(1) # Allow cleanup to sync to downstream
|
||||
|
||||
def test_create_collections_in_multiple_dbs(
|
||||
self,
|
||||
upstream_uri,
|
||||
upstream_token,
|
||||
downstream_uri,
|
||||
downstream_token,
|
||||
upstream_client,
|
||||
downstream_client,
|
||||
sync_timeout,
|
||||
):
|
||||
"""Test creating collections in multiple databases syncs to downstream."""
|
||||
self._upstream_uri = upstream_uri
|
||||
self._upstream_token = upstream_token
|
||||
|
||||
db_name_1 = self.gen_unique_name("test_mdb_db1")
|
||||
db_name_2 = self.gen_unique_name("test_mdb_db2")
|
||||
c_name_1 = self.gen_unique_name("test_mdb_col1")
|
||||
c_name_2 = self.gen_unique_name("test_mdb_col2")
|
||||
|
||||
self.resources_to_cleanup.append(("collection_in_db", (db_name_1, c_name_1)))
|
||||
self.resources_to_cleanup.append(("collection_in_db", (db_name_2, c_name_2)))
|
||||
self.resources_to_cleanup.append(("database", db_name_1))
|
||||
self.resources_to_cleanup.append(("database", db_name_2))
|
||||
|
||||
# Initial cleanup
|
||||
self.cleanup_database(upstream_client, db_name_1)
|
||||
self.cleanup_database(upstream_client, db_name_2)
|
||||
|
||||
# Create databases
|
||||
upstream_client.create_database(db_name_1)
|
||||
upstream_client.create_database(db_name_2)
|
||||
|
||||
# Create DB-scoped clients
|
||||
up_db1_client = MilvusClient(uri=upstream_uri, token=upstream_token, db_name=db_name_1)
|
||||
up_db2_client = MilvusClient(uri=upstream_uri, token=upstream_token, db_name=db_name_2)
|
||||
|
||||
# Create collection in db1 with 100 rows
|
||||
schema1 = self.create_default_schema(up_db1_client)
|
||||
up_db1_client.create_collection(collection_name=c_name_1, schema=schema1, consistency_level="Strong")
|
||||
up_db1_client.insert(c_name_1, self.generate_test_data(100))
|
||||
up_db1_client.flush(c_name_1)
|
||||
|
||||
# Create collection in db2 with 200 rows
|
||||
schema2 = self.create_default_schema(up_db2_client)
|
||||
up_db2_client.create_collection(collection_name=c_name_2, schema=schema2, consistency_level="Strong")
|
||||
up_db2_client.insert(c_name_2, self.generate_test_data(200))
|
||||
up_db2_client.flush(c_name_2)
|
||||
|
||||
up_db1_client.close()
|
||||
up_db2_client.close()
|
||||
|
||||
# Wait for both databases and collections to appear on downstream
|
||||
def check_db1_collection():
|
||||
try:
|
||||
if db_name_1 not in downstream_client.list_databases():
|
||||
return False
|
||||
dn_db1 = MilvusClient(uri=downstream_uri, token=downstream_token, db_name=db_name_1)
|
||||
exists = dn_db1.has_collection(c_name_1)
|
||||
if exists:
|
||||
stats = dn_db1.get_collection_stats(c_name_1)
|
||||
logger.info(f"Downstream db1 collection stats: {stats}")
|
||||
dn_db1.close()
|
||||
return exists
|
||||
except Exception as e:
|
||||
logger.warning(f"Check db1 collection failed: {e}")
|
||||
return False
|
||||
|
||||
assert self.wait_for_sync(check_db1_collection, sync_timeout, f"collection {c_name_1} in {db_name_1}")
|
||||
|
||||
def check_db2_collection():
|
||||
try:
|
||||
if db_name_2 not in downstream_client.list_databases():
|
||||
return False
|
||||
dn_db2 = MilvusClient(uri=downstream_uri, token=downstream_token, db_name=db_name_2)
|
||||
exists = dn_db2.has_collection(c_name_2)
|
||||
if exists:
|
||||
stats = dn_db2.get_collection_stats(c_name_2)
|
||||
logger.info(f"Downstream db2 collection stats: {stats}")
|
||||
dn_db2.close()
|
||||
return exists
|
||||
except Exception as e:
|
||||
logger.warning(f"Check db2 collection failed: {e}")
|
||||
return False
|
||||
|
||||
assert self.wait_for_sync(check_db2_collection, sync_timeout, f"collection {c_name_2} in {db_name_2}")
|
||||
|
||||
# Verify row counts
|
||||
dn_db1 = MilvusClient(uri=downstream_uri, token=downstream_token, db_name=db_name_1)
|
||||
dn_db2 = MilvusClient(uri=downstream_uri, token=downstream_token, db_name=db_name_2)
|
||||
try:
|
||||
stats1 = dn_db1.get_collection_stats(c_name_1)
|
||||
stats2 = dn_db2.get_collection_stats(c_name_2)
|
||||
logger.info(f"DB1 collection row count: {stats1.get('row_count')}")
|
||||
logger.info(f"DB2 collection row count: {stats2.get('row_count')}")
|
||||
assert stats1.get("row_count", 0) >= 100, (
|
||||
f"Expected >= 100 rows in {c_name_1}, got {stats1.get('row_count')}"
|
||||
)
|
||||
assert stats2.get("row_count", 0) >= 200, (
|
||||
f"Expected >= 200 rows in {c_name_2}, got {stats2.get('row_count')}"
|
||||
)
|
||||
finally:
|
||||
dn_db1.close()
|
||||
dn_db2.close()
|
||||
|
||||
def test_drop_db_with_collections(
|
||||
self,
|
||||
upstream_uri,
|
||||
upstream_token,
|
||||
downstream_uri,
|
||||
downstream_token,
|
||||
upstream_client,
|
||||
downstream_client,
|
||||
sync_timeout,
|
||||
):
|
||||
"""Test drop DB with collection syncs to downstream (DB gone from downstream)."""
|
||||
self._upstream_uri = upstream_uri
|
||||
self._upstream_token = upstream_token
|
||||
|
||||
db_name = self.gen_unique_name("test_mdb_drop_db")
|
||||
c_name = self.gen_unique_name("test_mdb_drop_col")
|
||||
|
||||
self.resources_to_cleanup.append(("collection_in_db", (db_name, c_name)))
|
||||
self.resources_to_cleanup.append(("database", db_name))
|
||||
|
||||
# Initial cleanup
|
||||
self.cleanup_database(upstream_client, db_name)
|
||||
|
||||
# Create database and collection
|
||||
upstream_client.create_database(db_name)
|
||||
up_db_client = MilvusClient(uri=upstream_uri, token=upstream_token, db_name=db_name)
|
||||
schema = self.create_default_schema(up_db_client)
|
||||
up_db_client.create_collection(collection_name=c_name, schema=schema, consistency_level="Strong")
|
||||
up_db_client.insert(c_name, self.generate_test_data(50))
|
||||
up_db_client.flush(c_name)
|
||||
up_db_client.close()
|
||||
|
||||
# Wait for DB + collection to sync to downstream
|
||||
def check_created():
|
||||
try:
|
||||
if db_name not in downstream_client.list_databases():
|
||||
return False
|
||||
dn = MilvusClient(uri=downstream_uri, token=downstream_token, db_name=db_name)
|
||||
exists = dn.has_collection(c_name)
|
||||
dn.close()
|
||||
return exists
|
||||
except Exception as e:
|
||||
logger.warning(f"Check DB+collection created: {e}")
|
||||
return False
|
||||
|
||||
assert self.wait_for_sync(check_created, sync_timeout, f"create db {db_name} with collection")
|
||||
|
||||
# Drop collection then DB in upstream
|
||||
up_db_client2 = MilvusClient(uri=upstream_uri, token=upstream_token, db_name=db_name)
|
||||
up_db_client2.drop_collection(c_name)
|
||||
up_db_client2.close()
|
||||
upstream_client.drop_database(db_name)
|
||||
|
||||
assert db_name not in upstream_client.list_databases(), (
|
||||
f"Database {db_name} still exists in upstream after drop"
|
||||
)
|
||||
|
||||
# Wait for DB to be gone on downstream
|
||||
def check_db_dropped():
|
||||
return db_name not in downstream_client.list_databases()
|
||||
|
||||
assert self.wait_for_sync(check_db_dropped, sync_timeout, f"drop database {db_name}")
|
||||
|
||||
def test_cross_db_operations(
|
||||
self,
|
||||
upstream_uri,
|
||||
upstream_token,
|
||||
downstream_uri,
|
||||
downstream_token,
|
||||
upstream_client,
|
||||
downstream_client,
|
||||
sync_timeout,
|
||||
):
|
||||
"""Test cross-DB operations: create DB, collection, insert, alter DB properties, create 2nd collection."""
|
||||
self._upstream_uri = upstream_uri
|
||||
self._upstream_token = upstream_token
|
||||
|
||||
db_name = self.gen_unique_name("test_mdb_cross_db")
|
||||
c_name_1 = self.gen_unique_name("test_mdb_cross_col1")
|
||||
c_name_2 = self.gen_unique_name("test_mdb_cross_col2")
|
||||
|
||||
self.resources_to_cleanup.append(("collection_in_db", (db_name, c_name_1)))
|
||||
self.resources_to_cleanup.append(("collection_in_db", (db_name, c_name_2)))
|
||||
self.resources_to_cleanup.append(("database", db_name))
|
||||
|
||||
# Initial cleanup
|
||||
self.cleanup_database(upstream_client, db_name)
|
||||
|
||||
# Step 1: Create database
|
||||
upstream_client.create_database(db_name)
|
||||
|
||||
# Step 2: Create DB-scoped client and 1st collection with insert
|
||||
up_db_client = MilvusClient(uri=upstream_uri, token=upstream_token, db_name=db_name)
|
||||
schema1 = self.create_default_schema(up_db_client)
|
||||
up_db_client.create_collection(collection_name=c_name_1, schema=schema1, consistency_level="Strong")
|
||||
up_db_client.insert(c_name_1, self.generate_test_data(100))
|
||||
up_db_client.flush(c_name_1)
|
||||
|
||||
# Step 3: Alter DB properties
|
||||
upstream_client.alter_database_properties(
|
||||
db_name=db_name,
|
||||
properties={"database.max.collections": 10},
|
||||
)
|
||||
|
||||
# Step 4: Create 2nd collection in the same DB
|
||||
schema2 = self.create_default_schema(up_db_client)
|
||||
up_db_client.create_collection(collection_name=c_name_2, schema=schema2, consistency_level="Strong")
|
||||
up_db_client.close()
|
||||
|
||||
# Wait for all operations to sync to downstream
|
||||
|
||||
# Check database exists on downstream
|
||||
def check_db_exists():
|
||||
return db_name in downstream_client.list_databases()
|
||||
|
||||
assert self.wait_for_sync(check_db_exists, sync_timeout, f"create database {db_name}")
|
||||
|
||||
# Check 1st collection exists on downstream
|
||||
def check_col1():
|
||||
try:
|
||||
dn = MilvusClient(uri=downstream_uri, token=downstream_token, db_name=db_name)
|
||||
exists = dn.has_collection(c_name_1)
|
||||
dn.close()
|
||||
return exists
|
||||
except Exception as e:
|
||||
logger.warning(f"Check col1 sync failed: {e}")
|
||||
return False
|
||||
|
||||
assert self.wait_for_sync(check_col1, sync_timeout, f"collection {c_name_1} in {db_name}")
|
||||
|
||||
# Check 2nd collection exists on downstream
|
||||
def check_col2():
|
||||
try:
|
||||
dn = MilvusClient(uri=downstream_uri, token=downstream_token, db_name=db_name)
|
||||
exists = dn.has_collection(c_name_2)
|
||||
dn.close()
|
||||
return exists
|
||||
except Exception as e:
|
||||
logger.warning(f"Check col2 sync failed: {e}")
|
||||
return False
|
||||
|
||||
assert self.wait_for_sync(check_col2, sync_timeout, f"collection {c_name_2} in {db_name}")
|
||||
|
||||
# Check DB properties synced
|
||||
def check_db_props():
|
||||
try:
|
||||
if db_name not in downstream_client.list_databases():
|
||||
return False
|
||||
props = downstream_client.describe_database(db_name)
|
||||
logger.info(f"Downstream database properties: {props}")
|
||||
return str(props.get("database.max.collections", "")) == "10"
|
||||
except Exception as e:
|
||||
logger.warning(f"Check DB properties sync failed: {e}")
|
||||
return False
|
||||
|
||||
assert self.wait_for_sync(check_db_props, sync_timeout, f"DB properties sync for {db_name}")
|
||||
@@ -0,0 +1,157 @@
|
||||
"""
|
||||
CDC non-replication tests for resource group operations.
|
||||
|
||||
Resource groups and replica assignment are per-cluster state; by design
|
||||
CDC does NOT propagate RG create/drop/update/transfer-replica to the
|
||||
downstream cluster. These tests guard that invariant.
|
||||
"""
|
||||
|
||||
import time
|
||||
|
||||
from .base import TestCDCSyncBase, logger
|
||||
|
||||
|
||||
class TestCDCSyncResourceGroup(TestCDCSyncBase):
|
||||
"""Verify that resource group operations are NOT replicated by CDC."""
|
||||
|
||||
def setup_method(self):
|
||||
"""Setup for each test method."""
|
||||
self.resources_to_cleanup = []
|
||||
|
||||
def teardown_method(self):
|
||||
"""Cleanup both upstream and downstream (downstream won't auto-sync)."""
|
||||
upstream_client = getattr(self, "_upstream_client", None)
|
||||
downstream_client = getattr(self, "_downstream_client", None)
|
||||
|
||||
for client in (upstream_client, downstream_client):
|
||||
if not client:
|
||||
continue
|
||||
for resource_type, resource_name in self.resources_to_cleanup:
|
||||
if resource_type == "resource_group":
|
||||
self._cleanup_resource_group(client, resource_name)
|
||||
elif resource_type == "collection":
|
||||
self.cleanup_collection(client, resource_name)
|
||||
|
||||
def _cleanup_resource_group(self, client, rg_name):
|
||||
"""Clean up resource group if exists (skipping default RG)."""
|
||||
try:
|
||||
existing = client.list_resource_groups()
|
||||
if rg_name in existing:
|
||||
logger.info(f"[CLEANUP] Cleaning up resource group: {rg_name}")
|
||||
client.drop_resource_group(rg_name)
|
||||
logger.info(f"[SUCCESS] Resource group {rg_name} cleaned up successfully")
|
||||
else:
|
||||
logger.debug(f"Resource group {rg_name} does not exist, skipping cleanup")
|
||||
except Exception as e:
|
||||
logger.warning(f"[FAILED] Failed to cleanup resource group {rg_name}: {e}")
|
||||
|
||||
def test_create_resource_group_not_replicated(self, upstream_client, downstream_client, sync_timeout):
|
||||
"""Creating an RG on upstream must NOT create it on downstream."""
|
||||
self._upstream_client = upstream_client
|
||||
self._downstream_client = downstream_client
|
||||
|
||||
rg_name = self.gen_unique_name("test_rg_create")
|
||||
self.resources_to_cleanup.append(("resource_group", rg_name))
|
||||
|
||||
self._cleanup_resource_group(upstream_client, rg_name)
|
||||
self._cleanup_resource_group(downstream_client, rg_name)
|
||||
|
||||
upstream_client.create_resource_group(rg_name)
|
||||
assert rg_name in upstream_client.list_resource_groups(), f"Resource group {rg_name} not created in upstream"
|
||||
|
||||
# Wait the full sync window; if CDC were going to leak the RG it would
|
||||
# have shown up by now.
|
||||
time.sleep(sync_timeout)
|
||||
downstream_rgs = downstream_client.list_resource_groups()
|
||||
assert rg_name not in downstream_rgs, (
|
||||
f"Resource group {rg_name} unexpectedly appeared on downstream (RG ops must not replicate)"
|
||||
)
|
||||
|
||||
def test_drop_resource_group_not_replicated(self, upstream_client, downstream_client, sync_timeout):
|
||||
"""Dropping an RG on upstream must NOT drop it on downstream."""
|
||||
self._upstream_client = upstream_client
|
||||
self._downstream_client = downstream_client
|
||||
|
||||
rg_name = self.gen_unique_name("test_rg_drop")
|
||||
self.resources_to_cleanup.append(("resource_group", rg_name))
|
||||
|
||||
self._cleanup_resource_group(upstream_client, rg_name)
|
||||
self._cleanup_resource_group(downstream_client, rg_name)
|
||||
|
||||
# Create the RG on BOTH sides (independently, since create isn't replicated either).
|
||||
upstream_client.create_resource_group(rg_name)
|
||||
downstream_client.create_resource_group(rg_name)
|
||||
assert rg_name in upstream_client.list_resource_groups()
|
||||
assert rg_name in downstream_client.list_resource_groups()
|
||||
|
||||
# Drop only on upstream; downstream must retain its own copy.
|
||||
upstream_client.drop_resource_group(rg_name)
|
||||
assert rg_name not in upstream_client.list_resource_groups(), (
|
||||
f"Resource group {rg_name} still exists in upstream after drop"
|
||||
)
|
||||
|
||||
time.sleep(sync_timeout)
|
||||
downstream_rgs = downstream_client.list_resource_groups()
|
||||
assert rg_name in downstream_rgs, (
|
||||
f"Resource group {rg_name} was dropped on downstream after upstream drop "
|
||||
f"(RG ops must not replicate). downstream RGs: {downstream_rgs}"
|
||||
)
|
||||
|
||||
def test_update_resource_group_not_replicated(self, upstream_client, downstream_client, sync_timeout):
|
||||
"""Updating RG config on upstream must NOT reconfigure downstream's RG."""
|
||||
self._upstream_client = upstream_client
|
||||
self._downstream_client = downstream_client
|
||||
|
||||
rg_name = self.gen_unique_name("test_rg_update")
|
||||
self.resources_to_cleanup.append(("resource_group", rg_name))
|
||||
|
||||
self._cleanup_resource_group(upstream_client, rg_name)
|
||||
self._cleanup_resource_group(downstream_client, rg_name)
|
||||
|
||||
upstream_config = {"requests": {"node_num": 0}, "limits": {"node_num": 2}}
|
||||
downstream_config = {"requests": {"node_num": 0}, "limits": {"node_num": 1}}
|
||||
|
||||
# Create the RG on both sides with different configs.
|
||||
upstream_client.create_resource_group(rg_name, config=upstream_config)
|
||||
downstream_client.create_resource_group(rg_name, config=downstream_config)
|
||||
downstream_desc_before = downstream_client.describe_resource_group(rg_name)
|
||||
logger.info(f"Downstream RG before upstream update: {downstream_desc_before}")
|
||||
|
||||
# Upstream has already been created with its config; nothing else to
|
||||
# update since config drift itself would be the leak. Wait and confirm
|
||||
# downstream config is unchanged. ResourceGroupInfo has no __eq__, so
|
||||
# compare the str() form (includes config/limits/requests/nodes).
|
||||
time.sleep(sync_timeout)
|
||||
downstream_desc_after = downstream_client.describe_resource_group(rg_name)
|
||||
logger.info(f"Downstream RG after upstream update: {downstream_desc_after}")
|
||||
assert str(downstream_desc_after) == str(downstream_desc_before), (
|
||||
f"Downstream RG {rg_name} was modified by upstream config (RG ops must not replicate). "
|
||||
f"before={downstream_desc_before}, after={downstream_desc_after}"
|
||||
)
|
||||
|
||||
def test_transfer_replica_not_replicated(self, upstream_client, downstream_client, sync_timeout):
|
||||
"""Creating multiple RGs on upstream must NOT create any of them on downstream."""
|
||||
self._upstream_client = upstream_client
|
||||
self._downstream_client = downstream_client
|
||||
|
||||
rg_name_1 = self.gen_unique_name("test_rg_transfer_1")
|
||||
rg_name_2 = self.gen_unique_name("test_rg_transfer_2")
|
||||
self.resources_to_cleanup.append(("resource_group", rg_name_1))
|
||||
self.resources_to_cleanup.append(("resource_group", rg_name_2))
|
||||
|
||||
self._cleanup_resource_group(upstream_client, rg_name_1)
|
||||
self._cleanup_resource_group(upstream_client, rg_name_2)
|
||||
self._cleanup_resource_group(downstream_client, rg_name_1)
|
||||
self._cleanup_resource_group(downstream_client, rg_name_2)
|
||||
|
||||
upstream_client.create_resource_group(rg_name_1)
|
||||
upstream_client.create_resource_group(rg_name_2)
|
||||
upstream_rgs = upstream_client.list_resource_groups()
|
||||
assert rg_name_1 in upstream_rgs
|
||||
assert rg_name_2 in upstream_rgs
|
||||
|
||||
time.sleep(sync_timeout)
|
||||
downstream_rgs = downstream_client.list_resource_groups()
|
||||
assert rg_name_1 not in downstream_rgs and rg_name_2 not in downstream_rgs, (
|
||||
f"Upstream RGs leaked to downstream (RG ops must not replicate). downstream RGs: {downstream_rgs}"
|
||||
)
|
||||
@@ -0,0 +1,690 @@
|
||||
"""
|
||||
CDC sync tests for advanced schema features (dynamic fields, nullable, default values,
|
||||
partition keys, clustering keys, and combinations thereof).
|
||||
"""
|
||||
|
||||
import random
|
||||
import time
|
||||
|
||||
from pymilvus import DataType
|
||||
|
||||
from .base import TestCDCSyncBase, logger
|
||||
|
||||
|
||||
class TestCDCSyncSchemaFeatures(TestCDCSyncBase):
|
||||
"""Test CDC sync for advanced schema features."""
|
||||
|
||||
def setup_method(self):
|
||||
"""Setup for each test method."""
|
||||
self.resources_to_cleanup = []
|
||||
|
||||
def teardown_method(self):
|
||||
"""Cleanup after each test method - only cleanup upstream, downstream will sync."""
|
||||
upstream_client = getattr(self, "_upstream_client", None)
|
||||
|
||||
if upstream_client:
|
||||
for resource_type, resource_name in self.resources_to_cleanup:
|
||||
if resource_type == "collection":
|
||||
self.cleanup_collection(upstream_client, resource_name)
|
||||
|
||||
time.sleep(1) # Allow cleanup to sync to downstream
|
||||
|
||||
def test_dynamic_schema_sync(self, upstream_client, downstream_client, sync_timeout):
|
||||
"""Test that dynamic schema fields are correctly replicated via CDC."""
|
||||
start_time = time.time()
|
||||
collection_name = self.gen_unique_name("test_dynamic_schema", max_length=50)
|
||||
|
||||
self.log_test_start("test_dynamic_schema_sync", "DYNAMIC_SCHEMA", collection_name)
|
||||
self._upstream_client = upstream_client
|
||||
self.resources_to_cleanup.append(("collection", collection_name))
|
||||
|
||||
try:
|
||||
self.cleanup_collection(upstream_client, collection_name)
|
||||
|
||||
# Create dynamic schema collection
|
||||
self.log_operation("CREATE_COLLECTION", "collection", collection_name, "upstream")
|
||||
upstream_client.create_collection(
|
||||
collection_name=collection_name,
|
||||
schema=self.create_dynamic_schema(upstream_client),
|
||||
)
|
||||
|
||||
# Create HNSW index and load
|
||||
index_params = upstream_client.prepare_index_params()
|
||||
index_params.add_index(
|
||||
field_name="float_vector",
|
||||
index_type="HNSW",
|
||||
metric_type="L2",
|
||||
params={"M": 8, "efConstruction": 64},
|
||||
)
|
||||
upstream_client.create_index(collection_name, index_params)
|
||||
upstream_client.load_collection(collection_name)
|
||||
|
||||
# Wait for collection creation to sync
|
||||
def check_create():
|
||||
return downstream_client.has_collection(collection_name)
|
||||
|
||||
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {collection_name}")
|
||||
|
||||
# Insert 200 rows with extra dynamic fields
|
||||
extra_fields = {"extra_int": int, "extra_str": str, "extra_float": float}
|
||||
test_data = self.generate_dynamic_data(200, extra_fields=extra_fields)
|
||||
self.log_data_operation("INSERT", collection_name, len(test_data), "- dynamic schema data")
|
||||
upstream_client.insert(collection_name, test_data)
|
||||
upstream_client.flush(collection_name)
|
||||
|
||||
# Query extra_int > 0 on upstream
|
||||
self.log_sync_verification("DYNAMIC_SCHEMA", collection_name, "extra_int > 0 count matches downstream")
|
||||
upstream_result = upstream_client.query(
|
||||
collection_name=collection_name,
|
||||
filter="extra_int > 0",
|
||||
output_fields=["count(*)"],
|
||||
)
|
||||
upstream_count = upstream_result[0]["count(*)"] if upstream_result else 0
|
||||
logger.info(f"[UPSTREAM] extra_int > 0 count: {upstream_count}")
|
||||
|
||||
# Wait for sync and verify downstream count matches
|
||||
def check_data():
|
||||
try:
|
||||
down_result = downstream_client.query(
|
||||
collection_name=collection_name,
|
||||
filter="extra_int > 0",
|
||||
output_fields=["count(*)"],
|
||||
)
|
||||
down_count = down_result[0]["count(*)"] if down_result else 0
|
||||
logger.info(f"[SYNC_PROGRESS] downstream extra_int > 0 count: {down_count}/{upstream_count}")
|
||||
return down_count == upstream_count
|
||||
except Exception as e:
|
||||
logger.warning(f"Dynamic schema sync check failed: {e}")
|
||||
return False
|
||||
|
||||
sync_success = self.wait_for_sync(check_data, sync_timeout, f"dynamic schema data sync {collection_name}")
|
||||
assert sync_success, f"Dynamic schema data failed to sync to downstream for {collection_name}"
|
||||
|
||||
# Verify data sampling for extra fields
|
||||
match, mismatch, details = self.verify_data_sampling(
|
||||
upstream_client,
|
||||
downstream_client,
|
||||
collection_name,
|
||||
sample_ratio=0.1,
|
||||
output_fields=["id", "varchar_field", "extra_int", "extra_str", "extra_float"],
|
||||
)
|
||||
logger.info(f"[VERIFY] Dynamic schema sampling: match={match}, mismatch={mismatch}")
|
||||
assert mismatch == 0, f"Dynamic field data mismatch detected: {details}"
|
||||
|
||||
duration = time.time() - start_time
|
||||
self.log_test_end("test_dynamic_schema_sync", True, duration)
|
||||
|
||||
except Exception as e:
|
||||
duration = time.time() - start_time
|
||||
logger.error(f"[ERROR] test_dynamic_schema_sync failed: {e}")
|
||||
self.log_test_end("test_dynamic_schema_sync", False, duration)
|
||||
raise
|
||||
|
||||
def test_nullable_fields_sync(self, upstream_client, downstream_client, sync_timeout):
|
||||
"""Test that nullable field values (including NULLs) are correctly replicated via CDC."""
|
||||
start_time = time.time()
|
||||
collection_name = self.gen_unique_name("test_nullable_flds", max_length=50)
|
||||
|
||||
self.log_test_start("test_nullable_fields_sync", "NULLABLE_FIELDS", collection_name)
|
||||
self._upstream_client = upstream_client
|
||||
self.resources_to_cleanup.append(("collection", collection_name))
|
||||
|
||||
try:
|
||||
self.cleanup_collection(upstream_client, collection_name)
|
||||
|
||||
# Create nullable schema collection
|
||||
self.log_operation("CREATE_COLLECTION", "collection", collection_name, "upstream")
|
||||
upstream_client.create_collection(
|
||||
collection_name=collection_name,
|
||||
schema=self.create_nullable_schema(upstream_client),
|
||||
)
|
||||
|
||||
# Create index and load
|
||||
index_params = upstream_client.prepare_index_params()
|
||||
index_params.add_index(
|
||||
field_name="float_vector",
|
||||
index_type="HNSW",
|
||||
metric_type="L2",
|
||||
params={"M": 8, "efConstruction": 64},
|
||||
)
|
||||
upstream_client.create_index(collection_name, index_params)
|
||||
upstream_client.load_collection(collection_name)
|
||||
|
||||
# Wait for collection creation to sync
|
||||
def check_create():
|
||||
return downstream_client.has_collection(collection_name)
|
||||
|
||||
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {collection_name}")
|
||||
|
||||
# Insert 200 rows with null_ratio=0.3
|
||||
test_data = self.generate_nullable_data(200, null_ratio=0.3)
|
||||
self.log_data_operation("INSERT", collection_name, len(test_data), "- nullable data null_ratio=0.3")
|
||||
upstream_client.insert(collection_name, test_data)
|
||||
upstream_client.flush(collection_name)
|
||||
|
||||
# Count nulls and not-nulls for nullable_int64 on upstream
|
||||
null_result = upstream_client.query(
|
||||
collection_name=collection_name,
|
||||
filter="nullable_int64 is null",
|
||||
output_fields=["count(*)"],
|
||||
)
|
||||
not_null_result = upstream_client.query(
|
||||
collection_name=collection_name,
|
||||
filter="nullable_int64 is not null",
|
||||
output_fields=["count(*)"],
|
||||
)
|
||||
upstream_null_count = null_result[0]["count(*)"] if null_result else 0
|
||||
upstream_not_null_count = not_null_result[0]["count(*)"] if not_null_result else 0
|
||||
logger.info(f"[UPSTREAM] nullable_int64 null={upstream_null_count}, not_null={upstream_not_null_count}")
|
||||
|
||||
self.log_sync_verification("NULLABLE_FIELDS", collection_name, "null counts match downstream")
|
||||
|
||||
# Wait for sync and verify null/not-null counts match on downstream
|
||||
def check_null_counts():
|
||||
try:
|
||||
d_null = downstream_client.query(
|
||||
collection_name=collection_name,
|
||||
filter="nullable_int64 is null",
|
||||
output_fields=["count(*)"],
|
||||
)
|
||||
d_not_null = downstream_client.query(
|
||||
collection_name=collection_name,
|
||||
filter="nullable_int64 is not null",
|
||||
output_fields=["count(*)"],
|
||||
)
|
||||
d_null_count = d_null[0]["count(*)"] if d_null else 0
|
||||
d_not_null_count = d_not_null[0]["count(*)"] if d_not_null else 0
|
||||
logger.info(
|
||||
f"[SYNC_PROGRESS] downstream nullable_int64 null={d_null_count}/{upstream_null_count}, "
|
||||
f"not_null={d_not_null_count}/{upstream_not_null_count}"
|
||||
)
|
||||
return d_null_count == upstream_null_count and d_not_null_count == upstream_not_null_count
|
||||
except Exception as e:
|
||||
logger.warning(f"Nullable sync check failed: {e}")
|
||||
return False
|
||||
|
||||
sync_success = self.wait_for_sync(
|
||||
check_null_counts, sync_timeout, f"nullable fields sync {collection_name}"
|
||||
)
|
||||
assert sync_success, f"Nullable field counts failed to sync to downstream for {collection_name}"
|
||||
|
||||
duration = time.time() - start_time
|
||||
self.log_test_end("test_nullable_fields_sync", True, duration)
|
||||
|
||||
except Exception as e:
|
||||
duration = time.time() - start_time
|
||||
logger.error(f"[ERROR] test_nullable_fields_sync failed: {e}")
|
||||
self.log_test_end("test_nullable_fields_sync", False, duration)
|
||||
raise
|
||||
|
||||
def test_default_values_sync(self, upstream_client, downstream_client, sync_timeout):
|
||||
"""Test that default field values are applied and replicated correctly via CDC."""
|
||||
start_time = time.time()
|
||||
collection_name = self.gen_unique_name("test_default_vals", max_length=50)
|
||||
|
||||
self.log_test_start("test_default_values_sync", "DEFAULT_VALUES", collection_name)
|
||||
self._upstream_client = upstream_client
|
||||
self.resources_to_cleanup.append(("collection", collection_name))
|
||||
|
||||
try:
|
||||
self.cleanup_collection(upstream_client, collection_name)
|
||||
|
||||
# Create default values schema collection
|
||||
self.log_operation("CREATE_COLLECTION", "collection", collection_name, "upstream")
|
||||
upstream_client.create_collection(
|
||||
collection_name=collection_name,
|
||||
schema=self.create_default_values_schema(upstream_client),
|
||||
)
|
||||
|
||||
# Create index and load
|
||||
index_params = upstream_client.prepare_index_params()
|
||||
index_params.add_index(
|
||||
field_name="float_vector",
|
||||
index_type="HNSW",
|
||||
metric_type="L2",
|
||||
params={"M": 8, "efConstruction": 64},
|
||||
)
|
||||
upstream_client.create_index(collection_name, index_params)
|
||||
upstream_client.load_collection(collection_name)
|
||||
|
||||
# Wait for collection creation to sync
|
||||
def check_create():
|
||||
return downstream_client.has_collection(collection_name)
|
||||
|
||||
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {collection_name}")
|
||||
|
||||
# Insert 100 rows providing ONLY float_vector — default fields are omitted
|
||||
test_data = [{"float_vector": [random.random() for _ in range(128)]} for _ in range(100)]
|
||||
self.log_data_operation("INSERT", collection_name, len(test_data), "- only float_vector provided")
|
||||
upstream_client.insert(collection_name, test_data)
|
||||
upstream_client.flush(collection_name)
|
||||
|
||||
# Query default_varchar == "default" on upstream — should be 100
|
||||
upstream_result = upstream_client.query(
|
||||
collection_name=collection_name,
|
||||
filter='default_varchar == "default"',
|
||||
output_fields=["count(*)"],
|
||||
)
|
||||
upstream_count = upstream_result[0]["count(*)"] if upstream_result else 0
|
||||
logger.info(f'[UPSTREAM] default_varchar == "default" count: {upstream_count}')
|
||||
assert upstream_count == 100, f"Expected 100 rows with default varchar on upstream, got {upstream_count}"
|
||||
|
||||
self.log_sync_verification(
|
||||
"DEFAULT_VALUES", collection_name, 'default_varchar == "default" count=100 on downstream'
|
||||
)
|
||||
|
||||
# Wait for sync and verify same count on downstream
|
||||
def check_defaults():
|
||||
try:
|
||||
down_result = downstream_client.query(
|
||||
collection_name=collection_name,
|
||||
filter='default_varchar == "default"',
|
||||
output_fields=["count(*)"],
|
||||
)
|
||||
down_count = down_result[0]["count(*)"] if down_result else 0
|
||||
logger.info(f"[SYNC_PROGRESS] downstream default_varchar count: {down_count}/100")
|
||||
return down_count == 100
|
||||
except Exception as e:
|
||||
logger.warning(f"Default values sync check failed: {e}")
|
||||
return False
|
||||
|
||||
sync_success = self.wait_for_sync(check_defaults, sync_timeout, f"default values sync {collection_name}")
|
||||
assert sync_success, f"Default value data failed to sync to downstream for {collection_name}"
|
||||
|
||||
duration = time.time() - start_time
|
||||
self.log_test_end("test_default_values_sync", True, duration)
|
||||
|
||||
except Exception as e:
|
||||
duration = time.time() - start_time
|
||||
logger.error(f"[ERROR] test_default_values_sync failed: {e}")
|
||||
self.log_test_end("test_default_values_sync", False, duration)
|
||||
raise
|
||||
|
||||
def test_partition_key_sync(self, upstream_client, downstream_client, sync_timeout):
|
||||
"""Test that partition key schema and data are correctly replicated via CDC."""
|
||||
start_time = time.time()
|
||||
collection_name = self.gen_unique_name("test_part_key", max_length=50)
|
||||
|
||||
self.log_test_start("test_partition_key_sync", "PARTITION_KEY", collection_name)
|
||||
self._upstream_client = upstream_client
|
||||
self.resources_to_cleanup.append(("collection", collection_name))
|
||||
|
||||
try:
|
||||
self.cleanup_collection(upstream_client, collection_name)
|
||||
|
||||
# Create partition key schema (VarChar key)
|
||||
self.log_operation("CREATE_COLLECTION", "collection", collection_name, "upstream")
|
||||
upstream_client.create_collection(
|
||||
collection_name=collection_name,
|
||||
schema=self.create_partition_key_schema(upstream_client, key_type="VarChar"),
|
||||
)
|
||||
|
||||
# Create index and load
|
||||
index_params = upstream_client.prepare_index_params()
|
||||
index_params.add_index(
|
||||
field_name="float_vector",
|
||||
index_type="HNSW",
|
||||
metric_type="L2",
|
||||
params={"M": 8, "efConstruction": 64},
|
||||
)
|
||||
upstream_client.create_index(collection_name, index_params)
|
||||
upstream_client.load_collection(collection_name)
|
||||
|
||||
# Wait for collection creation to sync
|
||||
def check_create():
|
||||
return downstream_client.has_collection(collection_name)
|
||||
|
||||
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {collection_name}")
|
||||
|
||||
# Insert 500 rows with random categories as partition key values
|
||||
categories = ["cat_A", "cat_B", "cat_C", "cat_D", "cat_E"]
|
||||
test_data = [
|
||||
{
|
||||
"float_vector": [random.random() for _ in range(128)],
|
||||
"partition_key_field": random.choice(categories),
|
||||
"data_field": f"data_{i}_{random.randint(1000, 9999)}",
|
||||
}
|
||||
for i in range(500)
|
||||
]
|
||||
self.log_data_operation("INSERT", collection_name, len(test_data), "- partition key data")
|
||||
upstream_client.insert(collection_name, test_data)
|
||||
upstream_client.flush(collection_name)
|
||||
|
||||
self.log_sync_verification(
|
||||
"PARTITION_KEY", collection_name, "count=500 and partition_key field on downstream"
|
||||
)
|
||||
|
||||
# Wait for sync and verify count=500 on downstream
|
||||
def check_data():
|
||||
try:
|
||||
result = downstream_client.query(
|
||||
collection_name=collection_name,
|
||||
filter="",
|
||||
output_fields=["count(*)"],
|
||||
)
|
||||
count = result[0]["count(*)"] if result else 0
|
||||
logger.info(f"[SYNC_PROGRESS] downstream count: {count}/500")
|
||||
return count >= 500
|
||||
except Exception as e:
|
||||
logger.warning(f"Partition key sync check failed: {e}")
|
||||
return False
|
||||
|
||||
sync_success = self.wait_for_sync(check_data, sync_timeout, f"partition key data sync {collection_name}")
|
||||
assert sync_success, f"Partition key data failed to sync to downstream for {collection_name}"
|
||||
|
||||
# Verify partition_key_field is present in downstream describe_collection
|
||||
downstream_info = downstream_client.describe_collection(collection_name)
|
||||
downstream_fields = [f["name"] for f in downstream_info.get("fields", [])]
|
||||
logger.info(f"[VERIFY] Downstream fields: {downstream_fields}")
|
||||
assert "partition_key_field" in downstream_fields, (
|
||||
f"partition_key_field not found in downstream collection schema: {downstream_fields}"
|
||||
)
|
||||
|
||||
duration = time.time() - start_time
|
||||
self.log_test_end("test_partition_key_sync", True, duration)
|
||||
|
||||
except Exception as e:
|
||||
duration = time.time() - start_time
|
||||
logger.error(f"[ERROR] test_partition_key_sync failed: {e}")
|
||||
self.log_test_end("test_partition_key_sync", False, duration)
|
||||
raise
|
||||
|
||||
def test_clustering_key_sync(self, upstream_client, downstream_client, sync_timeout):
|
||||
"""Test that clustering key schema and data are correctly replicated via CDC."""
|
||||
start_time = time.time()
|
||||
collection_name = self.gen_unique_name("test_cluster_key", max_length=50)
|
||||
|
||||
self.log_test_start("test_clustering_key_sync", "CLUSTERING_KEY", collection_name)
|
||||
self._upstream_client = upstream_client
|
||||
self.resources_to_cleanup.append(("collection", collection_name))
|
||||
|
||||
try:
|
||||
self.cleanup_collection(upstream_client, collection_name)
|
||||
|
||||
# Create clustering key schema
|
||||
self.log_operation("CREATE_COLLECTION", "collection", collection_name, "upstream")
|
||||
upstream_client.create_collection(
|
||||
collection_name=collection_name,
|
||||
schema=self.create_clustering_key_schema(upstream_client),
|
||||
)
|
||||
|
||||
# Create index and load
|
||||
index_params = upstream_client.prepare_index_params()
|
||||
index_params.add_index(
|
||||
field_name="float_vector",
|
||||
index_type="HNSW",
|
||||
metric_type="L2",
|
||||
params={"M": 8, "efConstruction": 64},
|
||||
)
|
||||
upstream_client.create_index(collection_name, index_params)
|
||||
upstream_client.load_collection(collection_name)
|
||||
|
||||
# Wait for collection creation to sync
|
||||
def check_create():
|
||||
return downstream_client.has_collection(collection_name)
|
||||
|
||||
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {collection_name}")
|
||||
|
||||
# Insert 300 rows
|
||||
test_data = [
|
||||
{
|
||||
"float_vector": [random.random() for _ in range(128)],
|
||||
"clustering_key_field": random.randint(0, 1000),
|
||||
"data_field": f"data_{i}_{random.randint(1000, 9999)}",
|
||||
}
|
||||
for i in range(300)
|
||||
]
|
||||
self.log_data_operation("INSERT", collection_name, len(test_data), "- clustering key data")
|
||||
upstream_client.insert(collection_name, test_data)
|
||||
upstream_client.flush(collection_name)
|
||||
|
||||
self.log_sync_verification(
|
||||
"CLUSTERING_KEY", collection_name, "count=300 and clustering_key field on downstream"
|
||||
)
|
||||
|
||||
# Wait for sync and verify count=300 on downstream
|
||||
def check_data():
|
||||
try:
|
||||
result = downstream_client.query(
|
||||
collection_name=collection_name,
|
||||
filter="",
|
||||
output_fields=["count(*)"],
|
||||
)
|
||||
count = result[0]["count(*)"] if result else 0
|
||||
logger.info(f"[SYNC_PROGRESS] downstream count: {count}/300")
|
||||
return count >= 300
|
||||
except Exception as e:
|
||||
logger.warning(f"Clustering key sync check failed: {e}")
|
||||
return False
|
||||
|
||||
sync_success = self.wait_for_sync(check_data, sync_timeout, f"clustering key data sync {collection_name}")
|
||||
assert sync_success, f"Clustering key data failed to sync to downstream for {collection_name}"
|
||||
|
||||
# Verify clustering_key_field is present in downstream describe_collection
|
||||
downstream_info = downstream_client.describe_collection(collection_name)
|
||||
downstream_fields = [f["name"] for f in downstream_info.get("fields", [])]
|
||||
logger.info(f"[VERIFY] Downstream fields: {downstream_fields}")
|
||||
assert "clustering_key_field" in downstream_fields, (
|
||||
f"clustering_key_field not found in downstream collection schema: {downstream_fields}"
|
||||
)
|
||||
|
||||
duration = time.time() - start_time
|
||||
self.log_test_end("test_clustering_key_sync", True, duration)
|
||||
|
||||
except Exception as e:
|
||||
duration = time.time() - start_time
|
||||
logger.error(f"[ERROR] test_clustering_key_sync failed: {e}")
|
||||
self.log_test_end("test_clustering_key_sync", False, duration)
|
||||
raise
|
||||
|
||||
def test_nullable_with_defaults(self, upstream_client, downstream_client, sync_timeout):
|
||||
"""Test that nullable fields combined with default values sync correctly via CDC.
|
||||
|
||||
Inserts 100 rows in three patterns:
|
||||
i%3==0: explicit values provided for both fields
|
||||
i%3==1: None values (explicit null) provided for both fields
|
||||
i%3==2: fields omitted entirely (uses default / null)
|
||||
"""
|
||||
start_time = time.time()
|
||||
collection_name = self.gen_unique_name("test_null_default", max_length=50)
|
||||
|
||||
self.log_test_start("test_nullable_with_defaults", "NULLABLE_WITH_DEFAULTS", collection_name)
|
||||
self._upstream_client = upstream_client
|
||||
self.resources_to_cleanup.append(("collection", collection_name))
|
||||
|
||||
try:
|
||||
self.cleanup_collection(upstream_client, collection_name)
|
||||
|
||||
# Build custom schema: nullable_with_default (INT64, nullable, default=42)
|
||||
# nullable_no_default (VARCHAR, nullable)
|
||||
schema = upstream_client.create_schema()
|
||||
schema.add_field("id", DataType.INT64, is_primary=True, auto_id=True)
|
||||
schema.add_field("float_vector", DataType.FLOAT_VECTOR, dim=128)
|
||||
schema.add_field(
|
||||
"nullable_with_default",
|
||||
DataType.INT64,
|
||||
nullable=True,
|
||||
default_value=42,
|
||||
)
|
||||
schema.add_field(
|
||||
"nullable_no_default",
|
||||
DataType.VARCHAR,
|
||||
max_length=256,
|
||||
nullable=True,
|
||||
)
|
||||
|
||||
self.log_operation("CREATE_COLLECTION", "collection", collection_name, "upstream")
|
||||
upstream_client.create_collection(
|
||||
collection_name=collection_name,
|
||||
schema=schema,
|
||||
)
|
||||
|
||||
# Create index and load
|
||||
index_params = upstream_client.prepare_index_params()
|
||||
index_params.add_index(
|
||||
field_name="float_vector",
|
||||
index_type="HNSW",
|
||||
metric_type="L2",
|
||||
params={"M": 8, "efConstruction": 64},
|
||||
)
|
||||
upstream_client.create_index(collection_name, index_params)
|
||||
upstream_client.load_collection(collection_name)
|
||||
|
||||
# Wait for collection creation to sync
|
||||
def check_create():
|
||||
return downstream_client.has_collection(collection_name)
|
||||
|
||||
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {collection_name}")
|
||||
|
||||
# Insert 100 rows across 3 patterns
|
||||
test_data = []
|
||||
for i in range(100):
|
||||
record = {"float_vector": [random.random() for _ in range(128)]}
|
||||
if i % 3 == 0:
|
||||
# Explicit values
|
||||
record["nullable_with_default"] = random.randint(100, 999)
|
||||
record["nullable_no_default"] = f"explicit_{i}"
|
||||
elif i % 3 == 1:
|
||||
# Explicit None (null)
|
||||
record["nullable_with_default"] = None
|
||||
record["nullable_no_default"] = None
|
||||
# i%3==2: omit both fields — server applies default/null
|
||||
test_data.append(record)
|
||||
|
||||
self.log_data_operation("INSERT", collection_name, len(test_data), "- nullable+default mixed pattern")
|
||||
upstream_client.insert(collection_name, test_data)
|
||||
upstream_client.flush(collection_name)
|
||||
|
||||
self.log_sync_verification("NULLABLE_WITH_DEFAULTS", collection_name, "total count=100 on downstream")
|
||||
|
||||
# Wait for sync and verify total count on downstream
|
||||
def check_data():
|
||||
try:
|
||||
result = downstream_client.query(
|
||||
collection_name=collection_name,
|
||||
filter="",
|
||||
output_fields=["count(*)"],
|
||||
)
|
||||
count = result[0]["count(*)"] if result else 0
|
||||
logger.info(f"[SYNC_PROGRESS] downstream count: {count}/100")
|
||||
return count >= 100
|
||||
except Exception as e:
|
||||
logger.warning(f"Nullable+defaults sync check failed: {e}")
|
||||
return False
|
||||
|
||||
sync_success = self.wait_for_sync(check_data, sync_timeout, f"nullable+defaults sync {collection_name}")
|
||||
assert sync_success, f"Nullable-with-defaults data failed to sync to downstream for {collection_name}"
|
||||
|
||||
duration = time.time() - start_time
|
||||
self.log_test_end("test_nullable_with_defaults", True, duration)
|
||||
|
||||
except Exception as e:
|
||||
duration = time.time() - start_time
|
||||
logger.error(f"[ERROR] test_nullable_with_defaults failed: {e}")
|
||||
self.log_test_end("test_nullable_with_defaults", False, duration)
|
||||
raise
|
||||
|
||||
def test_dynamic_with_partition_key(self, upstream_client, downstream_client, sync_timeout):
|
||||
"""Test that dynamic fields combined with a partition key schema sync correctly via CDC."""
|
||||
start_time = time.time()
|
||||
collection_name = self.gen_unique_name("test_dyn_part_key", max_length=50)
|
||||
|
||||
self.log_test_start("test_dynamic_with_partition_key", "DYNAMIC_WITH_PARTITION_KEY", collection_name)
|
||||
self._upstream_client = upstream_client
|
||||
self.resources_to_cleanup.append(("collection", collection_name))
|
||||
|
||||
try:
|
||||
self.cleanup_collection(upstream_client, collection_name)
|
||||
|
||||
# Build schema: enable_dynamic_field=True + VARCHAR partition key
|
||||
schema = upstream_client.create_schema(enable_dynamic_field=True)
|
||||
schema.add_field("id", DataType.INT64, is_primary=True, auto_id=True)
|
||||
schema.add_field("float_vector", DataType.FLOAT_VECTOR, dim=128)
|
||||
schema.add_field(
|
||||
"pk_field",
|
||||
DataType.VARCHAR,
|
||||
max_length=64,
|
||||
is_partition_key=True,
|
||||
)
|
||||
|
||||
self.log_operation("CREATE_COLLECTION", "collection", collection_name, "upstream")
|
||||
upstream_client.create_collection(
|
||||
collection_name=collection_name,
|
||||
schema=schema,
|
||||
)
|
||||
|
||||
# Create index and load
|
||||
index_params = upstream_client.prepare_index_params()
|
||||
index_params.add_index(
|
||||
field_name="float_vector",
|
||||
index_type="HNSW",
|
||||
metric_type="L2",
|
||||
params={"M": 8, "efConstruction": 64},
|
||||
)
|
||||
upstream_client.create_index(collection_name, index_params)
|
||||
upstream_client.load_collection(collection_name)
|
||||
|
||||
# Wait for collection creation to sync
|
||||
def check_create():
|
||||
return downstream_client.has_collection(collection_name)
|
||||
|
||||
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {collection_name}")
|
||||
|
||||
# Insert 300 rows with dynamic fields + partition key values
|
||||
partitions = ["region_A", "region_B", "region_C", "region_D"]
|
||||
test_data = [
|
||||
{
|
||||
"float_vector": [random.random() for _ in range(128)],
|
||||
"pk_field": random.choice(partitions),
|
||||
# dynamic fields
|
||||
"dynamic_num": random.randint(1, 10000),
|
||||
"dynamic_tag": f"tag_{i % 10}",
|
||||
"dynamic_score": random.uniform(0.0, 100.0),
|
||||
}
|
||||
for i in range(300)
|
||||
]
|
||||
self.log_data_operation("INSERT", collection_name, len(test_data), "- dynamic+partition_key data")
|
||||
upstream_client.insert(collection_name, test_data)
|
||||
upstream_client.flush(collection_name)
|
||||
|
||||
# Query dynamic_num > 0 on upstream
|
||||
upstream_result = upstream_client.query(
|
||||
collection_name=collection_name,
|
||||
filter="dynamic_num > 0",
|
||||
output_fields=["count(*)"],
|
||||
)
|
||||
upstream_count = upstream_result[0]["count(*)"] if upstream_result else 0
|
||||
logger.info(f"[UPSTREAM] dynamic_num > 0 count: {upstream_count}")
|
||||
|
||||
self.log_sync_verification(
|
||||
"DYNAMIC_WITH_PARTITION_KEY",
|
||||
collection_name,
|
||||
f"dynamic_num > 0 count={upstream_count} on downstream",
|
||||
)
|
||||
|
||||
# Wait for sync and verify the count matches on downstream
|
||||
def check_data():
|
||||
try:
|
||||
down_result = downstream_client.query(
|
||||
collection_name=collection_name,
|
||||
filter="dynamic_num > 0",
|
||||
output_fields=["count(*)"],
|
||||
)
|
||||
down_count = down_result[0]["count(*)"] if down_result else 0
|
||||
logger.info(f"[SYNC_PROGRESS] downstream dynamic_num > 0 count: {down_count}/{upstream_count}")
|
||||
return down_count == upstream_count
|
||||
except Exception as e:
|
||||
logger.warning(f"Dynamic+partition_key sync check failed: {e}")
|
||||
return False
|
||||
|
||||
sync_success = self.wait_for_sync(check_data, sync_timeout, f"dynamic+partition_key sync {collection_name}")
|
||||
assert sync_success, f"Dynamic+partition_key data failed to sync to downstream for {collection_name}"
|
||||
|
||||
duration = time.time() - start_time
|
||||
self.log_test_end("test_dynamic_with_partition_key", True, duration)
|
||||
|
||||
except Exception as e:
|
||||
duration = time.time() - start_time
|
||||
logger.error(f"[ERROR] test_dynamic_with_partition_key failed: {e}")
|
||||
self.log_test_end("test_dynamic_with_partition_key", False, duration)
|
||||
raise
|
||||
@@ -0,0 +1,570 @@
|
||||
"""
|
||||
CDC sync tests for search and query result verification across vector types.
|
||||
"""
|
||||
|
||||
import random
|
||||
import time
|
||||
|
||||
import pytest
|
||||
from pymilvus import AnnSearchRequest, DataType, RRFRanker
|
||||
|
||||
from .base import TestCDCSyncBase, logger
|
||||
|
||||
# fmt: off
|
||||
VECTOR_PARAMS = [
|
||||
("FLOAT_VECTOR", "HNSW", "COSINE", 128),
|
||||
("FLOAT_VECTOR", "IVF_FLAT", "L2", 128),
|
||||
("FLOAT16_VECTOR", "HNSW", "L2", 64),
|
||||
("BFLOAT16_VECTOR", "HNSW", "L2", 64),
|
||||
("INT8_VECTOR", "HNSW", "COSINE", 64),
|
||||
("BINARY_VECTOR", "BIN_FLAT", "HAMMING", 128),
|
||||
("SPARSE_FLOAT_VECTOR", "SPARSE_INVERTED_INDEX", "IP", 0),
|
||||
]
|
||||
# fmt: on
|
||||
|
||||
|
||||
class TestCDCSyncSearchVerification(TestCDCSyncBase):
|
||||
"""Test CDC sync for search and query result verification across vector types."""
|
||||
|
||||
def setup_method(self):
|
||||
"""Setup for each test method."""
|
||||
self.resources_to_cleanup = []
|
||||
|
||||
def teardown_method(self):
|
||||
"""Cleanup after each test method - only cleanup upstream, downstream will sync."""
|
||||
upstream_client = getattr(self, "_upstream_client", None)
|
||||
|
||||
if upstream_client:
|
||||
for resource_type, resource_name in self.resources_to_cleanup:
|
||||
if resource_type == "collection":
|
||||
self.cleanup_collection(upstream_client, resource_name)
|
||||
|
||||
time.sleep(1) # Allow cleanup to sync to downstream
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Internal helper
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
def _setup_collection(self, client, c_name, vector_type, index_type, metric, dim):
|
||||
"""
|
||||
Create a single-vector-schema collection, insert 500 records,
|
||||
create an index, and load.
|
||||
|
||||
Returns the collection name (same as c_name).
|
||||
"""
|
||||
schema = self.create_single_vector_schema(client, vector_type=vector_type, dim=dim)
|
||||
client.create_collection(collection_name=c_name, schema=schema)
|
||||
|
||||
# Insert 500 records
|
||||
data = self.generate_single_vector_data(500, vector_type=vector_type, dim=dim)
|
||||
client.insert(c_name, data)
|
||||
client.flush(c_name)
|
||||
|
||||
# Build index
|
||||
index_params = client.prepare_index_params()
|
||||
if index_type == "IVF_FLAT":
|
||||
idx_params = {"nlist": 64}
|
||||
elif index_type == "HNSW":
|
||||
idx_params = {"M": 16, "efConstruction": 200}
|
||||
else:
|
||||
idx_params = {}
|
||||
|
||||
index_params.add_index(
|
||||
field_name="vector",
|
||||
index_type=index_type,
|
||||
metric_type=metric,
|
||||
params=idx_params,
|
||||
)
|
||||
client.create_index(c_name, index_params)
|
||||
client.load_collection(c_name)
|
||||
|
||||
return c_name
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Tests
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"vector_type,index_type,metric,dim",
|
||||
VECTOR_PARAMS,
|
||||
ids=[p[0] + "_" + p[1] for p in VECTOR_PARAMS],
|
||||
)
|
||||
def test_search_result_consistency(
|
||||
self,
|
||||
upstream_client,
|
||||
downstream_client,
|
||||
sync_timeout,
|
||||
vector_type,
|
||||
index_type,
|
||||
metric,
|
||||
dim,
|
||||
):
|
||||
"""Verify that ANN search results are consistent between upstream and downstream."""
|
||||
start_time = time.time()
|
||||
c_name = self.gen_unique_name(f"test_src_{vector_type[:4].lower()}", max_length=50)
|
||||
|
||||
self.log_test_start(
|
||||
"test_search_result_consistency",
|
||||
f"SEARCH/{vector_type}/{index_type}",
|
||||
c_name,
|
||||
)
|
||||
self._upstream_client = upstream_client
|
||||
self.resources_to_cleanup.append(("collection", c_name))
|
||||
|
||||
try:
|
||||
self.cleanup_collection(upstream_client, c_name)
|
||||
self._setup_collection(upstream_client, c_name, vector_type, index_type, metric, dim)
|
||||
|
||||
# Wait for at least 500 records to appear on downstream
|
||||
def check_sync():
|
||||
try:
|
||||
res = downstream_client.query(
|
||||
collection_name=c_name,
|
||||
filter="",
|
||||
output_fields=["count(*)"],
|
||||
)
|
||||
cnt = res[0]["count(*)"] if res else 0
|
||||
logger.info(f"[SYNC_PROGRESS] downstream count: {cnt}/500")
|
||||
return cnt >= 500
|
||||
except Exception as e:
|
||||
logger.warning(f"Sync check failed: {e}")
|
||||
return False
|
||||
|
||||
assert self.wait_for_sync(check_sync, sync_timeout, f"data sync 500 records {c_name}"), (
|
||||
f"Downstream did not receive 500 records within {sync_timeout}s"
|
||||
)
|
||||
|
||||
# Build 5 random query vectors
|
||||
dtype = getattr(DataType, vector_type)
|
||||
query_vectors = self._gen_vectors(5, dim if dim > 0 else 1000, dtype)
|
||||
|
||||
avg_overlap, _, _ = self.verify_search_consistency(
|
||||
upstream_client,
|
||||
downstream_client,
|
||||
c_name,
|
||||
query_vectors,
|
||||
anns_field="vector",
|
||||
limit=10,
|
||||
metric_type=metric,
|
||||
)
|
||||
|
||||
assert avg_overlap >= self.SEARCH_OVERLAP_THRESHOLD, (
|
||||
f"Search overlap {avg_overlap:.4f} is below threshold "
|
||||
f"{self.SEARCH_OVERLAP_THRESHOLD} for {vector_type}/{index_type}"
|
||||
)
|
||||
|
||||
finally:
|
||||
self.log_test_end(
|
||||
"test_search_result_consistency",
|
||||
True,
|
||||
time.time() - start_time,
|
||||
)
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"vector_type,index_type,metric,dim",
|
||||
VECTOR_PARAMS,
|
||||
ids=[p[0] + "_" + p[1] for p in VECTOR_PARAMS],
|
||||
)
|
||||
def test_query_data_sampling(
|
||||
self,
|
||||
upstream_client,
|
||||
downstream_client,
|
||||
sync_timeout,
|
||||
vector_type,
|
||||
index_type,
|
||||
metric,
|
||||
dim,
|
||||
):
|
||||
"""Verify scalar field values are identical on both sides via random sampling."""
|
||||
start_time = time.time()
|
||||
c_name = self.gen_unique_name(f"test_qds_{vector_type[:4].lower()}", max_length=50)
|
||||
|
||||
self.log_test_start(
|
||||
"test_query_data_sampling",
|
||||
f"QUERY_SAMPLE/{vector_type}/{index_type}",
|
||||
c_name,
|
||||
)
|
||||
self._upstream_client = upstream_client
|
||||
self.resources_to_cleanup.append(("collection", c_name))
|
||||
|
||||
try:
|
||||
self.cleanup_collection(upstream_client, c_name)
|
||||
self._setup_collection(upstream_client, c_name, vector_type, index_type, metric, dim)
|
||||
|
||||
def check_sync():
|
||||
try:
|
||||
res = downstream_client.query(
|
||||
collection_name=c_name,
|
||||
filter="",
|
||||
output_fields=["count(*)"],
|
||||
)
|
||||
cnt = res[0]["count(*)"] if res else 0
|
||||
logger.info(f"[SYNC_PROGRESS] downstream count: {cnt}/500")
|
||||
return cnt >= 500
|
||||
except Exception as e:
|
||||
logger.warning(f"Sync check failed: {e}")
|
||||
return False
|
||||
|
||||
assert self.wait_for_sync(check_sync, sync_timeout, f"data sync 500 records {c_name}"), (
|
||||
f"Downstream did not receive 500 records within {sync_timeout}s"
|
||||
)
|
||||
|
||||
output_fields = ["id", "int_field", "varchar_field", "float_field"]
|
||||
match_count, mismatch_count, mismatch_details = self.verify_data_sampling(
|
||||
upstream_client,
|
||||
downstream_client,
|
||||
c_name,
|
||||
sample_ratio=0.2,
|
||||
output_fields=output_fields,
|
||||
)
|
||||
|
||||
logger.info(
|
||||
f"[RESULT] Sampling — match={match_count}, mismatch={mismatch_count}, details={mismatch_details[:3]}"
|
||||
)
|
||||
assert mismatch_count == 0, f"Found {mismatch_count} mismatched records: {mismatch_details[:5]}"
|
||||
|
||||
finally:
|
||||
self.log_test_end(
|
||||
"test_query_data_sampling",
|
||||
True,
|
||||
time.time() - start_time,
|
||||
)
|
||||
|
||||
def test_hybrid_search_consistency(
|
||||
self,
|
||||
upstream_client,
|
||||
downstream_client,
|
||||
sync_timeout,
|
||||
):
|
||||
"""Verify hybrid search (dense + sparse, RRF ranker) results are consistent."""
|
||||
start_time = time.time()
|
||||
c_name = self.gen_unique_name("test_hybrid_srch", max_length=50)
|
||||
|
||||
self.log_test_start("test_hybrid_search_consistency", "HYBRID_SEARCH", c_name)
|
||||
self._upstream_client = upstream_client
|
||||
self.resources_to_cleanup.append(("collection", c_name))
|
||||
|
||||
try:
|
||||
self.cleanup_collection(upstream_client, c_name)
|
||||
|
||||
# Build schema: dense FloatVector(128) + sparse
|
||||
schema = upstream_client.create_schema(enable_dynamic_field=True)
|
||||
schema.add_field("id", DataType.INT64, is_primary=True, auto_id=True)
|
||||
schema.add_field("dense", DataType.FLOAT_VECTOR, dim=128)
|
||||
schema.add_field("sparse", DataType.SPARSE_FLOAT_VECTOR)
|
||||
schema.add_field("int_field", DataType.INT64)
|
||||
schema.add_field("varchar_field", DataType.VARCHAR, max_length=256)
|
||||
|
||||
upstream_client.create_collection(collection_name=c_name, schema=schema)
|
||||
|
||||
# Insert 300 records
|
||||
dense_vecs = self._gen_vectors(300, 128, DataType.FLOAT_VECTOR)
|
||||
sparse_vecs = self._gen_vectors(300, 1000, DataType.SPARSE_FLOAT_VECTOR)
|
||||
data = [
|
||||
{
|
||||
"dense": dense_vecs[i],
|
||||
"sparse": sparse_vecs[i],
|
||||
"int_field": random.randint(0, 1000),
|
||||
"varchar_field": f"hybrid_{i}_{random.randint(1000, 9999)}",
|
||||
}
|
||||
for i in range(300)
|
||||
]
|
||||
upstream_client.insert(c_name, data)
|
||||
upstream_client.flush(c_name)
|
||||
|
||||
# Create indexes
|
||||
index_params = upstream_client.prepare_index_params()
|
||||
index_params.add_index(
|
||||
field_name="dense",
|
||||
index_type="HNSW",
|
||||
metric_type="COSINE",
|
||||
params={"M": 16, "efConstruction": 200},
|
||||
)
|
||||
index_params.add_index(
|
||||
field_name="sparse",
|
||||
index_type="SPARSE_INVERTED_INDEX",
|
||||
metric_type="IP",
|
||||
params={},
|
||||
)
|
||||
upstream_client.create_index(c_name, index_params)
|
||||
upstream_client.load_collection(c_name)
|
||||
|
||||
# Wait for downstream sync
|
||||
def check_sync():
|
||||
try:
|
||||
res = downstream_client.query(
|
||||
collection_name=c_name,
|
||||
filter="",
|
||||
output_fields=["count(*)"],
|
||||
)
|
||||
cnt = res[0]["count(*)"] if res else 0
|
||||
logger.info(f"[SYNC_PROGRESS] downstream count: {cnt}/300")
|
||||
return cnt >= 300
|
||||
except Exception as e:
|
||||
logger.warning(f"Sync check failed: {e}")
|
||||
return False
|
||||
|
||||
assert self.wait_for_sync(check_sync, sync_timeout, f"hybrid data sync {c_name}"), (
|
||||
f"Downstream did not receive 300 records within {sync_timeout}s"
|
||||
)
|
||||
|
||||
# Build hybrid search requests
|
||||
q_dense = self._gen_vectors(1, 128, DataType.FLOAT_VECTOR)[0]
|
||||
q_sparse = self._gen_vectors(1, 1000, DataType.SPARSE_FLOAT_VECTOR)[0]
|
||||
|
||||
dense_req = AnnSearchRequest(
|
||||
data=[q_dense],
|
||||
anns_field="dense",
|
||||
param={"metric_type": "COSINE", "params": {"ef": 64}},
|
||||
limit=10,
|
||||
)
|
||||
sparse_req = AnnSearchRequest(
|
||||
data=[q_sparse],
|
||||
anns_field="sparse",
|
||||
param={"metric_type": "IP"},
|
||||
limit=10,
|
||||
)
|
||||
|
||||
up_results = upstream_client.hybrid_search(
|
||||
collection_name=c_name,
|
||||
reqs=[dense_req, sparse_req],
|
||||
ranker=RRFRanker(),
|
||||
limit=10,
|
||||
output_fields=["id"],
|
||||
)
|
||||
down_results = downstream_client.hybrid_search(
|
||||
collection_name=c_name,
|
||||
reqs=[dense_req, sparse_req],
|
||||
ranker=RRFRanker(),
|
||||
limit=10,
|
||||
output_fields=["id"],
|
||||
)
|
||||
|
||||
up_pks = set(hit["id"] for hit in up_results[0]) if up_results else set()
|
||||
down_pks = set(hit["id"] for hit in down_results[0]) if down_results else set()
|
||||
union_size = len(up_pks | down_pks)
|
||||
overlap = len(up_pks & down_pks) / union_size if union_size > 0 else 1.0
|
||||
|
||||
logger.info(f"[RESULT] Hybrid search PK overlap={overlap:.4f} (up={len(up_pks)}, down={len(down_pks)})")
|
||||
assert overlap >= self.SEARCH_OVERLAP_THRESHOLD, (
|
||||
f"Hybrid search overlap {overlap:.4f} below threshold {self.SEARCH_OVERLAP_THRESHOLD}"
|
||||
)
|
||||
|
||||
finally:
|
||||
self.log_test_end(
|
||||
"test_hybrid_search_consistency",
|
||||
True,
|
||||
time.time() - start_time,
|
||||
)
|
||||
|
||||
def test_search_iterator_consistency(
|
||||
self,
|
||||
upstream_client,
|
||||
downstream_client,
|
||||
sync_timeout,
|
||||
):
|
||||
"""Verify search iterator returns the same PK set on upstream and downstream."""
|
||||
start_time = time.time()
|
||||
c_name = self.gen_unique_name("test_srch_iter", max_length=50)
|
||||
|
||||
self.log_test_start("test_search_iterator_consistency", "SEARCH_ITERATOR", c_name)
|
||||
self._upstream_client = upstream_client
|
||||
self.resources_to_cleanup.append(("collection", c_name))
|
||||
|
||||
try:
|
||||
self.cleanup_collection(upstream_client, c_name)
|
||||
self._setup_collection(upstream_client, c_name, "FLOAT_VECTOR", "HNSW", "COSINE", 128)
|
||||
|
||||
def check_sync():
|
||||
try:
|
||||
res = downstream_client.query(
|
||||
collection_name=c_name,
|
||||
filter="",
|
||||
output_fields=["count(*)"],
|
||||
)
|
||||
cnt = res[0]["count(*)"] if res else 0
|
||||
logger.info(f"[SYNC_PROGRESS] downstream count: {cnt}/500")
|
||||
return cnt >= 500
|
||||
except Exception as e:
|
||||
logger.warning(f"Sync check failed: {e}")
|
||||
return False
|
||||
|
||||
assert self.wait_for_sync(check_sync, sync_timeout, f"data sync 500 records {c_name}"), (
|
||||
f"Downstream did not receive 500 records within {sync_timeout}s"
|
||||
)
|
||||
|
||||
query_vec = self._gen_vectors(1, 128, DataType.FLOAT_VECTOR)[0]
|
||||
search_params = {"metric_type": "COSINE", "params": {"ef": 64}}
|
||||
|
||||
def _collect_iterator_pks(client):
|
||||
pks = set()
|
||||
iterator = client.search_iterator(
|
||||
collection_name=c_name,
|
||||
data=[query_vec],
|
||||
anns_field="vector",
|
||||
batch_size=50,
|
||||
limit=200,
|
||||
param=search_params,
|
||||
output_fields=["id"],
|
||||
)
|
||||
while True:
|
||||
batch = iterator.next()
|
||||
if not batch:
|
||||
iterator.close()
|
||||
break
|
||||
for hit in batch:
|
||||
pks.add(hit["id"])
|
||||
return pks
|
||||
|
||||
up_pks = _collect_iterator_pks(upstream_client)
|
||||
down_pks = _collect_iterator_pks(downstream_client)
|
||||
union_size = len(up_pks | down_pks)
|
||||
overlap = len(up_pks & down_pks) / union_size if union_size > 0 else 1.0
|
||||
|
||||
logger.info(f"[RESULT] Search iterator overlap={overlap:.4f} (up={len(up_pks)}, down={len(down_pks)})")
|
||||
assert overlap >= self.SEARCH_OVERLAP_THRESHOLD, (
|
||||
f"Search iterator PK overlap {overlap:.4f} below threshold {self.SEARCH_OVERLAP_THRESHOLD}"
|
||||
)
|
||||
|
||||
finally:
|
||||
self.log_test_end(
|
||||
"test_search_iterator_consistency",
|
||||
True,
|
||||
time.time() - start_time,
|
||||
)
|
||||
|
||||
def test_query_iterator_consistency(
|
||||
self,
|
||||
upstream_client,
|
||||
downstream_client,
|
||||
sync_timeout,
|
||||
):
|
||||
"""Verify that a query iterator retrieves identical PK sets from both sides."""
|
||||
start_time = time.time()
|
||||
c_name = self.gen_unique_name("test_qry_iter", max_length=50)
|
||||
|
||||
self.log_test_start("test_query_iterator_consistency", "QUERY_ITERATOR", c_name)
|
||||
self._upstream_client = upstream_client
|
||||
self.resources_to_cleanup.append(("collection", c_name))
|
||||
|
||||
try:
|
||||
self.cleanup_collection(upstream_client, c_name)
|
||||
self._setup_collection(upstream_client, c_name, "FLOAT_VECTOR", "HNSW", "COSINE", 128)
|
||||
|
||||
def check_sync():
|
||||
try:
|
||||
res = downstream_client.query(
|
||||
collection_name=c_name,
|
||||
filter="",
|
||||
output_fields=["count(*)"],
|
||||
)
|
||||
cnt = res[0]["count(*)"] if res else 0
|
||||
logger.info(f"[SYNC_PROGRESS] downstream count: {cnt}/500")
|
||||
return cnt >= 500
|
||||
except Exception as e:
|
||||
logger.warning(f"Sync check failed: {e}")
|
||||
return False
|
||||
|
||||
assert self.wait_for_sync(check_sync, sync_timeout, f"data sync 500 records {c_name}"), (
|
||||
f"Downstream did not receive 500 records within {sync_timeout}s"
|
||||
)
|
||||
|
||||
up_count, down_count, match = self.verify_iterator_consistency(
|
||||
upstream_client,
|
||||
downstream_client,
|
||||
c_name,
|
||||
batch_size=100,
|
||||
)
|
||||
|
||||
logger.info(f"[RESULT] Query iterator — upstream={up_count}, downstream={down_count}, match={match}")
|
||||
assert match, f"Query iterator PK sets differ: upstream={up_count}, downstream={down_count}"
|
||||
|
||||
finally:
|
||||
self.log_test_end(
|
||||
"test_query_iterator_consistency",
|
||||
True,
|
||||
time.time() - start_time,
|
||||
)
|
||||
|
||||
def test_search_with_filter_consistency(
|
||||
self,
|
||||
upstream_client,
|
||||
downstream_client,
|
||||
sync_timeout,
|
||||
):
|
||||
"""Verify filtered search produces consistent results honoring the filter predicate."""
|
||||
start_time = time.time()
|
||||
c_name = self.gen_unique_name("test_srch_filter", max_length=50)
|
||||
|
||||
self.log_test_start("test_search_with_filter_consistency", "SEARCH_WITH_FILTER", c_name)
|
||||
self._upstream_client = upstream_client
|
||||
self.resources_to_cleanup.append(("collection", c_name))
|
||||
|
||||
try:
|
||||
self.cleanup_collection(upstream_client, c_name)
|
||||
self._setup_collection(upstream_client, c_name, "FLOAT_VECTOR", "HNSW", "COSINE", 128)
|
||||
|
||||
def check_sync():
|
||||
try:
|
||||
res = downstream_client.query(
|
||||
collection_name=c_name,
|
||||
filter="",
|
||||
output_fields=["count(*)"],
|
||||
)
|
||||
cnt = res[0]["count(*)"] if res else 0
|
||||
logger.info(f"[SYNC_PROGRESS] downstream count: {cnt}/500")
|
||||
return cnt >= 500
|
||||
except Exception as e:
|
||||
logger.warning(f"Sync check failed: {e}")
|
||||
return False
|
||||
|
||||
assert self.wait_for_sync(check_sync, sync_timeout, f"data sync 500 records {c_name}"), (
|
||||
f"Downstream did not receive 500 records within {sync_timeout}s"
|
||||
)
|
||||
|
||||
filter_expr = "int_field > 500"
|
||||
query_vec = self._gen_vectors(1, 128, DataType.FLOAT_VECTOR)[0]
|
||||
search_params = {"metric_type": "COSINE"}
|
||||
|
||||
up_results = upstream_client.search(
|
||||
collection_name=c_name,
|
||||
data=[query_vec],
|
||||
anns_field="vector",
|
||||
search_params=search_params,
|
||||
filter=filter_expr,
|
||||
limit=10,
|
||||
output_fields=["id", "int_field"],
|
||||
)
|
||||
down_results = downstream_client.search(
|
||||
collection_name=c_name,
|
||||
data=[query_vec],
|
||||
anns_field="vector",
|
||||
search_params=search_params,
|
||||
filter=filter_expr,
|
||||
limit=10,
|
||||
output_fields=["id", "int_field"],
|
||||
)
|
||||
|
||||
# Verify filter is honoured on both sides
|
||||
for hit in up_results[0] if up_results else []:
|
||||
assert hit["int_field"] > 500, f"Filter violated on upstream: int_field={hit['int_field']}"
|
||||
for hit in down_results[0] if down_results else []:
|
||||
assert hit["int_field"] > 500, f"Filter violated on downstream: int_field={hit['int_field']}"
|
||||
|
||||
# Verify PK overlap
|
||||
up_pks = set(hit["id"] for hit in up_results[0]) if up_results else set()
|
||||
down_pks = set(hit["id"] for hit in down_results[0]) if down_results else set()
|
||||
union_size = len(up_pks | down_pks)
|
||||
overlap = len(up_pks & down_pks) / union_size if union_size > 0 else 1.0
|
||||
|
||||
logger.info(f"[RESULT] Filtered search overlap={overlap:.4f} (up={len(up_pks)}, down={len(down_pks)})")
|
||||
assert overlap >= self.SEARCH_OVERLAP_THRESHOLD, (
|
||||
f"Filtered search overlap {overlap:.4f} below threshold {self.SEARCH_OVERLAP_THRESHOLD}"
|
||||
)
|
||||
|
||||
finally:
|
||||
self.log_test_end(
|
||||
"test_search_with_filter_consistency",
|
||||
True,
|
||||
time.time() - start_time,
|
||||
)
|
||||
@@ -3,8 +3,12 @@ CDC topology setup and configuration test cases.
|
||||
"""
|
||||
|
||||
import time
|
||||
|
||||
import pytest
|
||||
from common.common_type import CaseLabel
|
||||
|
||||
from cdc.conftest import CDC_UPDATE_REPLICATE_TIMEOUT_SECONDS, apply_replicate_configuration
|
||||
|
||||
from .base import TestCDCSyncBase
|
||||
|
||||
|
||||
@@ -41,10 +45,7 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
||||
{
|
||||
"cluster_id": source_cluster_id,
|
||||
"connection_param": {"uri": upstream_uri, "token": upstream_token},
|
||||
"pchannels": [
|
||||
f"{source_cluster_id}-rootcoord-dml_{i}"
|
||||
for i in range(pchannel_num)
|
||||
],
|
||||
"pchannels": [f"{source_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
|
||||
},
|
||||
{
|
||||
"cluster_id": target_cluster_id,
|
||||
@@ -52,10 +53,7 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
||||
"uri": downstream_uri,
|
||||
"token": downstream_token,
|
||||
},
|
||||
"pchannels": [
|
||||
f"{target_cluster_id}-rootcoord-dml_{i}"
|
||||
for i in range(pchannel_num)
|
||||
],
|
||||
"pchannels": [f"{target_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
|
||||
},
|
||||
],
|
||||
"cross_cluster_topology": [
|
||||
@@ -67,8 +65,7 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
||||
}
|
||||
|
||||
# Test normal topology setup
|
||||
upstream_client.update_replicate_configuration(**config)
|
||||
downstream_client.update_replicate_configuration(**config)
|
||||
apply_replicate_configuration([(upstream_client, config), (downstream_client, config)])
|
||||
|
||||
# Wait for configuration to take effect
|
||||
time.sleep(3)
|
||||
@@ -87,9 +84,7 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
||||
def check_sync():
|
||||
return downstream_client.has_collection(test_collection_name)
|
||||
|
||||
assert self.wait_for_sync(
|
||||
check_sync, 30, f"collection {test_collection_name} sync"
|
||||
)
|
||||
assert self.wait_for_sync(check_sync, 30, f"collection {test_collection_name} sync")
|
||||
|
||||
# Cleanup
|
||||
self.cleanup_collection(upstream_client, test_collection_name)
|
||||
@@ -113,10 +108,7 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
||||
{
|
||||
"cluster_id": source_cluster_id,
|
||||
"connection_param": {"uri": upstream_uri, "token": upstream_token},
|
||||
"pchannels": [
|
||||
f"{source_cluster_id}-rootcoord-dml_{i}"
|
||||
for i in range(pchannel_num)
|
||||
],
|
||||
"pchannels": [f"{source_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
|
||||
},
|
||||
{
|
||||
"cluster_id": target_cluster_id,
|
||||
@@ -124,10 +116,7 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
||||
"uri": downstream_uri,
|
||||
"token": downstream_token,
|
||||
},
|
||||
"pchannels": [
|
||||
f"{target_cluster_id}-rootcoord-dml_{i}"
|
||||
for i in range(pchannel_num)
|
||||
],
|
||||
"pchannels": [f"{target_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
|
||||
},
|
||||
],
|
||||
"cross_cluster_topology": [
|
||||
@@ -138,8 +127,7 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
||||
],
|
||||
}
|
||||
|
||||
upstream_client.update_replicate_configuration(**original_config)
|
||||
downstream_client.update_replicate_configuration(**original_config)
|
||||
apply_replicate_configuration([(upstream_client, original_config), (downstream_client, original_config)])
|
||||
time.sleep(3)
|
||||
|
||||
# Now switch the direction (target -> source)
|
||||
@@ -148,10 +136,7 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
||||
{
|
||||
"cluster_id": source_cluster_id,
|
||||
"connection_param": {"uri": upstream_uri, "token": upstream_token},
|
||||
"pchannels": [
|
||||
f"{source_cluster_id}-rootcoord-dml_{i}"
|
||||
for i in range(pchannel_num)
|
||||
],
|
||||
"pchannels": [f"{source_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
|
||||
},
|
||||
{
|
||||
"cluster_id": target_cluster_id,
|
||||
@@ -159,10 +144,7 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
||||
"uri": downstream_uri,
|
||||
"token": downstream_token,
|
||||
},
|
||||
"pchannels": [
|
||||
f"{target_cluster_id}-rootcoord-dml_{i}"
|
||||
for i in range(pchannel_num)
|
||||
],
|
||||
"pchannels": [f"{target_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
|
||||
},
|
||||
],
|
||||
"cross_cluster_topology": [
|
||||
@@ -174,8 +156,7 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
||||
}
|
||||
|
||||
# Apply switched configuration
|
||||
upstream_client.update_replicate_configuration(**switched_config)
|
||||
downstream_client.update_replicate_configuration(**switched_config)
|
||||
apply_replicate_configuration([(upstream_client, switched_config), (downstream_client, switched_config)])
|
||||
time.sleep(3)
|
||||
|
||||
# Test the switched topology by creating collection on downstream (now source)
|
||||
@@ -191,9 +172,7 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
||||
def check_switched_sync():
|
||||
return upstream_client.has_collection(test_collection_name)
|
||||
|
||||
assert self.wait_for_sync(
|
||||
check_switched_sync, 30, f"switched collection {test_collection_name} sync"
|
||||
)
|
||||
assert self.wait_for_sync(check_switched_sync, 30, f"switched collection {test_collection_name} sync")
|
||||
|
||||
# Cleanup
|
||||
self.cleanup_collection(downstream_client, test_collection_name)
|
||||
@@ -229,7 +208,9 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
||||
}
|
||||
|
||||
with pytest.raises(Exception):
|
||||
upstream_client.update_replicate_configuration(**invalid_config_1)
|
||||
upstream_client.update_replicate_configuration(
|
||||
timeout=CDC_UPDATE_REPLICATE_TIMEOUT_SECONDS, **invalid_config_1
|
||||
)
|
||||
|
||||
# Test case 4.2: Circular dependency (A -> B, B -> A)
|
||||
invalid_config_2 = {
|
||||
@@ -262,7 +243,9 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
||||
|
||||
# This may or may not fail depending on implementation, but test it
|
||||
with pytest.raises(Exception):
|
||||
upstream_client.update_replicate_configuration(**invalid_config_2)
|
||||
upstream_client.update_replicate_configuration(
|
||||
timeout=CDC_UPDATE_REPLICATE_TIMEOUT_SECONDS, **invalid_config_2
|
||||
)
|
||||
|
||||
# Test case 4.3: Invalid connection parameters
|
||||
invalid_config_3 = {
|
||||
@@ -292,7 +275,9 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
||||
],
|
||||
}
|
||||
with pytest.raises(Exception):
|
||||
upstream_client.update_replicate_configuration(**invalid_config_3)
|
||||
upstream_client.update_replicate_configuration(
|
||||
timeout=CDC_UPDATE_REPLICATE_TIMEOUT_SECONDS, **invalid_config_3
|
||||
)
|
||||
|
||||
# Test case 4.4: Empty cluster list but non-empty topology
|
||||
invalid_config_4 = {
|
||||
@@ -305,7 +290,9 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
||||
],
|
||||
}
|
||||
with pytest.raises(Exception):
|
||||
upstream_client.update_replicate_configuration(**invalid_config_4)
|
||||
upstream_client.update_replicate_configuration(
|
||||
timeout=CDC_UPDATE_REPLICATE_TIMEOUT_SECONDS, **invalid_config_4
|
||||
)
|
||||
|
||||
# Test case 4.5: Invalid pchannel format
|
||||
invalid_config_5 = {
|
||||
@@ -319,7 +306,9 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
||||
"cross_cluster_topology": [],
|
||||
}
|
||||
with pytest.raises(Exception):
|
||||
upstream_client.update_replicate_configuration(**invalid_config_5)
|
||||
upstream_client.update_replicate_configuration(
|
||||
timeout=CDC_UPDATE_REPLICATE_TIMEOUT_SECONDS, **invalid_config_5
|
||||
)
|
||||
|
||||
@pytest.mark.order(-1)
|
||||
def test_clear_configuration_disconnect(
|
||||
@@ -341,10 +330,7 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
||||
{
|
||||
"cluster_id": source_cluster_id,
|
||||
"connection_param": {"uri": upstream_uri, "token": upstream_token},
|
||||
"pchannels": [
|
||||
f"{source_cluster_id}-rootcoord-dml_{i}"
|
||||
for i in range(pchannel_num)
|
||||
],
|
||||
"pchannels": [f"{source_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
|
||||
},
|
||||
{
|
||||
"cluster_id": target_cluster_id,
|
||||
@@ -352,10 +338,7 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
||||
"uri": downstream_uri,
|
||||
"token": downstream_token,
|
||||
},
|
||||
"pchannels": [
|
||||
f"{target_cluster_id}-rootcoord-dml_{i}"
|
||||
for i in range(pchannel_num)
|
||||
],
|
||||
"pchannels": [f"{target_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
|
||||
},
|
||||
],
|
||||
"cross_cluster_topology": [
|
||||
@@ -366,8 +349,7 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
||||
],
|
||||
}
|
||||
|
||||
upstream_client.update_replicate_configuration(**config)
|
||||
downstream_client.update_replicate_configuration(**config)
|
||||
apply_replicate_configuration([(upstream_client, config), (downstream_client, config)])
|
||||
time.sleep(3)
|
||||
|
||||
# Verify topology is working
|
||||
@@ -380,9 +362,7 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
||||
def check_initial_sync():
|
||||
return downstream_client.has_collection(test_collection_name)
|
||||
|
||||
assert self.wait_for_sync(
|
||||
check_initial_sync, 30, f"initial collection {test_collection_name} sync"
|
||||
)
|
||||
assert self.wait_for_sync(check_initial_sync, 30, f"initial collection {test_collection_name} sync")
|
||||
|
||||
# Now clear the configuration (empty topology)
|
||||
empty_upstream_config = {
|
||||
@@ -390,10 +370,7 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
||||
{
|
||||
"cluster_id": source_cluster_id,
|
||||
"connection_param": {"uri": upstream_uri, "token": upstream_token},
|
||||
"pchannels": [
|
||||
f"{source_cluster_id}-rootcoord-dml_{i}"
|
||||
for i in range(pchannel_num)
|
||||
],
|
||||
"pchannels": [f"{source_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
|
||||
}
|
||||
],
|
||||
"cross_cluster_topology": [],
|
||||
@@ -406,17 +383,15 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
||||
"uri": downstream_uri,
|
||||
"token": downstream_token,
|
||||
},
|
||||
"pchannels": [
|
||||
f"{target_cluster_id}-rootcoord-dml_{i}"
|
||||
for i in range(pchannel_num)
|
||||
],
|
||||
"pchannels": [f"{target_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
|
||||
}
|
||||
],
|
||||
"cross_cluster_topology": [],
|
||||
}
|
||||
# Apply empty configuration to disconnect CDC
|
||||
upstream_client.update_replicate_configuration(**empty_upstream_config)
|
||||
downstream_client.update_replicate_configuration(**empty_downstream_config)
|
||||
apply_replicate_configuration(
|
||||
[(upstream_client, empty_upstream_config), (downstream_client, empty_downstream_config)]
|
||||
)
|
||||
time.sleep(3)
|
||||
|
||||
# Test that CDC is disconnected - create new collection and verify it doesn't sync
|
||||
|
||||
@@ -0,0 +1,793 @@
|
||||
"""
|
||||
CDC sync tests for topology switchover and failover scenarios.
|
||||
"""
|
||||
|
||||
import random
|
||||
import subprocess
|
||||
import threading
|
||||
import time
|
||||
|
||||
from .base import TestCDCSyncBase, logger
|
||||
|
||||
|
||||
class TestCDCSyncSwitchover(TestCDCSyncBase):
|
||||
"""Test CDC sync behaviour during and after topology switchover / failover."""
|
||||
|
||||
def setup_method(self):
|
||||
"""Setup for each test method."""
|
||||
self.resources_to_cleanup = []
|
||||
|
||||
def teardown_method(self):
|
||||
"""Cleanup after each test method - only cleanup upstream, downstream will sync."""
|
||||
upstream_client = getattr(self, "_upstream_client", None)
|
||||
|
||||
if upstream_client:
|
||||
for resource_type, resource_name in self.resources_to_cleanup:
|
||||
if resource_type == "collection":
|
||||
self.cleanup_collection(upstream_client, resource_name)
|
||||
|
||||
time.sleep(1) # Allow cleanup to sync to downstream
|
||||
|
||||
# -------------------------------------------------------------------------
|
||||
# Tests
|
||||
# -------------------------------------------------------------------------
|
||||
|
||||
def test_switchover_basic(
|
||||
self,
|
||||
upstream_client,
|
||||
downstream_client,
|
||||
sync_timeout,
|
||||
switchover_helper,
|
||||
source_cluster_id,
|
||||
target_cluster_id,
|
||||
):
|
||||
"""
|
||||
Basic switchover: insert 200 records, verify sync, switchover, insert 200 more
|
||||
on the new source, verify count == 400 on both sides, sample verify, switch back.
|
||||
"""
|
||||
start_time = time.time()
|
||||
c_name = self.gen_unique_name("test_sw_basic", max_length=50)
|
||||
|
||||
self.log_test_start("test_switchover_basic", "SWITCHOVER_BASIC", c_name)
|
||||
self._upstream_client = upstream_client
|
||||
self.resources_to_cleanup.append(("collection", c_name))
|
||||
|
||||
try:
|
||||
self.cleanup_collection(upstream_client, c_name)
|
||||
|
||||
schema = self.create_manual_id_schema(upstream_client)
|
||||
upstream_client.create_collection(collection_name=c_name, schema=schema)
|
||||
|
||||
index_params = upstream_client.prepare_index_params()
|
||||
index_params.add_index(
|
||||
field_name="vector",
|
||||
index_type="HNSW",
|
||||
metric_type="L2",
|
||||
params={"M": 16, "efConstruction": 200},
|
||||
)
|
||||
upstream_client.create_index(c_name, index_params)
|
||||
upstream_client.load_collection(c_name)
|
||||
|
||||
# Wait for collection to appear on downstream
|
||||
def check_create():
|
||||
return downstream_client.has_collection(c_name)
|
||||
|
||||
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {c_name}")
|
||||
|
||||
# Insert 200 records on upstream (original source)
|
||||
batch1 = self.generate_test_data_with_id(200, start_id=0)
|
||||
upstream_client.insert(c_name, batch1)
|
||||
upstream_client.flush(c_name)
|
||||
|
||||
def check_200():
|
||||
try:
|
||||
res = downstream_client.query(
|
||||
collection_name=c_name,
|
||||
filter="",
|
||||
output_fields=["count(*)"],
|
||||
)
|
||||
cnt = res[0]["count(*)"] if res else 0
|
||||
logger.info(f"[SYNC_PROGRESS] downstream count: {cnt}/200")
|
||||
return cnt >= 200
|
||||
except Exception as e:
|
||||
logger.warning(f"Sync check failed: {e}")
|
||||
return False
|
||||
|
||||
assert self.wait_for_sync(check_200, sync_timeout, f"initial 200-record sync {c_name}")
|
||||
|
||||
# Switchover: target becomes new source
|
||||
logger.info("[SWITCHOVER] Initiating basic switchover...")
|
||||
switchover_helper(target_cluster_id, source_cluster_id)
|
||||
|
||||
# Insert 200 more records on the new source (previously downstream)
|
||||
batch2 = self.generate_test_data_with_id(200, start_id=200)
|
||||
downstream_client.insert(c_name, batch2)
|
||||
downstream_client.flush(c_name)
|
||||
|
||||
def check_400_upstream():
|
||||
try:
|
||||
res = upstream_client.query(
|
||||
collection_name=c_name,
|
||||
filter="",
|
||||
output_fields=["count(*)"],
|
||||
)
|
||||
cnt = res[0]["count(*)"] if res else 0
|
||||
logger.info(f"[SYNC_PROGRESS] upstream count after switchover: {cnt}/400")
|
||||
return cnt >= 400
|
||||
except Exception as e:
|
||||
logger.warning(f"Sync check failed: {e}")
|
||||
return False
|
||||
|
||||
def check_400_downstream():
|
||||
try:
|
||||
res = downstream_client.query(
|
||||
collection_name=c_name,
|
||||
filter="",
|
||||
output_fields=["count(*)"],
|
||||
)
|
||||
cnt = res[0]["count(*)"] if res else 0
|
||||
logger.info(f"[SYNC_PROGRESS] downstream count after switchover: {cnt}/400")
|
||||
return cnt >= 400
|
||||
except Exception as e:
|
||||
logger.warning(f"Sync check failed: {e}")
|
||||
return False
|
||||
|
||||
assert self.wait_for_sync(check_400_upstream, sync_timeout, f"400-record sync upstream {c_name}")
|
||||
assert self.wait_for_sync(check_400_downstream, sync_timeout, f"400-record sync downstream {c_name}")
|
||||
|
||||
# Sample verify
|
||||
match_count, mismatch_count, details = self.verify_data_sampling(
|
||||
downstream_client, # new source
|
||||
upstream_client, # new target
|
||||
c_name,
|
||||
sample_ratio=0.2,
|
||||
output_fields=["id", "vector"],
|
||||
)
|
||||
assert mismatch_count == 0, f"Data mismatch after basic switchover: {details[:5]}"
|
||||
|
||||
finally:
|
||||
# Restore original topology
|
||||
logger.info("[SWITCHOVER] Restoring original topology after test_switchover_basic...")
|
||||
switchover_helper(source_cluster_id, target_cluster_id)
|
||||
self.log_test_end("test_switchover_basic", True, time.time() - start_time)
|
||||
|
||||
def test_switchover_during_writes(
|
||||
self,
|
||||
upstream_client,
|
||||
downstream_client,
|
||||
sync_timeout,
|
||||
switchover_helper,
|
||||
source_cluster_id,
|
||||
target_cluster_id,
|
||||
):
|
||||
"""
|
||||
Switchover during concurrent writes: background thread inserts 10 batches of 100
|
||||
with 2 s between each; switchover fires at t=12 s; join thread, flush, verify
|
||||
both sides have the same count >= total_inserted.
|
||||
"""
|
||||
start_time = time.time()
|
||||
c_name = self.gen_unique_name("test_sw_writes", max_length=50)
|
||||
|
||||
self.log_test_start("test_switchover_during_writes", "SWITCHOVER_DURING_WRITES", c_name)
|
||||
self._upstream_client = upstream_client
|
||||
self.resources_to_cleanup.append(("collection", c_name))
|
||||
|
||||
try:
|
||||
self.cleanup_collection(upstream_client, c_name)
|
||||
|
||||
schema = self.create_manual_id_schema(upstream_client)
|
||||
upstream_client.create_collection(collection_name=c_name, schema=schema)
|
||||
|
||||
index_params = upstream_client.prepare_index_params()
|
||||
index_params.add_index(
|
||||
field_name="vector",
|
||||
index_type="HNSW",
|
||||
metric_type="L2",
|
||||
params={"M": 16, "efConstruction": 200},
|
||||
)
|
||||
upstream_client.create_index(c_name, index_params)
|
||||
upstream_client.load_collection(c_name)
|
||||
|
||||
def check_create():
|
||||
return downstream_client.has_collection(c_name)
|
||||
|
||||
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {c_name}")
|
||||
|
||||
total_inserted = 0
|
||||
lock = threading.Lock()
|
||||
write_error = []
|
||||
|
||||
def background_insert():
|
||||
nonlocal total_inserted
|
||||
for batch_idx in range(10):
|
||||
try:
|
||||
start_id = batch_idx * 100
|
||||
data = self.generate_test_data_with_id(100, start_id=start_id)
|
||||
# Cap pymilvus's retry loop at 10 (default 75). A batch
|
||||
# that lands on the old primary after it flips to
|
||||
# replica returns STREAMING_CODE_REPLICATE_VIOLATION;
|
||||
# the default retry burns ~213 s per failed batch and
|
||||
# makes the thread runtime unbounded when the role flip
|
||||
# takes longer than one batch's cadence. 10 retries
|
||||
# with the decorator's 3 s backoff cap bound each
|
||||
# failure to ~16 s.
|
||||
upstream_client.insert(c_name, data, retry_times=10)
|
||||
with lock:
|
||||
total_inserted += 100
|
||||
logger.info(f"[BACKGROUND] Inserted batch {batch_idx + 1}/10 (total: {total_inserted})")
|
||||
except Exception as e:
|
||||
logger.error(f"[BACKGROUND] Insert failed on batch {batch_idx}: {e}")
|
||||
write_error.append(e)
|
||||
time.sleep(2)
|
||||
|
||||
insert_thread = threading.Thread(target=background_insert, daemon=True)
|
||||
insert_thread.start()
|
||||
|
||||
# Switchover after 12 s (mid-way through background inserts)
|
||||
time.sleep(12)
|
||||
logger.info("[SWITCHOVER] Initiating switchover during writes...")
|
||||
switchover_helper(target_cluster_id, source_cluster_id)
|
||||
|
||||
# With retry_times=10 each failed batch burns ~16 s (vs ~213 s with
|
||||
# the pymilvus default of 75). Worst case: all 10 batches fail →
|
||||
# ~10 * (16 + 2) = 180 s. 180 s gives ~6x margin.
|
||||
insert_thread.join(timeout=180)
|
||||
assert not insert_thread.is_alive(), "Background insert thread did not finish in time"
|
||||
|
||||
# Transient STREAMING_CODE_REPLICATE_VIOLATION failures are EXPECTED
|
||||
# during the role flip: writes that arrive on the old primary after
|
||||
# it becomes a replica are rejected by design. The invariant that
|
||||
# matters is that every batch which SUCCEEDED ends up replicated
|
||||
# consistently (enforced by the up_cnt == down_cnt check below).
|
||||
if write_error:
|
||||
logger.warning(
|
||||
f"Background inserts had {len(write_error)} transient failure(s) during switchover: {write_error}"
|
||||
)
|
||||
|
||||
# After switchover, downstream is the new primary. Flushing the
|
||||
# old primary (upstream) hits the replica-role rate limiter
|
||||
# (rate=0.1/s) and exhausts pymilvus's 75 retries. Flush the
|
||||
# current primary instead.
|
||||
downstream_client.flush(c_name)
|
||||
|
||||
expected = total_inserted
|
||||
logger.info(f"[INFO] Total inserted: {expected}")
|
||||
|
||||
def check_both(client, label):
|
||||
def _check():
|
||||
try:
|
||||
res = client.query(
|
||||
collection_name=c_name,
|
||||
filter="",
|
||||
output_fields=["count(*)"],
|
||||
)
|
||||
cnt = res[0]["count(*)"] if res else 0
|
||||
logger.info(f"[SYNC_PROGRESS] {label} count: {cnt}/{expected}")
|
||||
return cnt >= expected
|
||||
except Exception as e:
|
||||
logger.warning(f"Sync check failed ({label}): {e}")
|
||||
return False
|
||||
|
||||
return _check
|
||||
|
||||
assert self.wait_for_sync(
|
||||
check_both(upstream_client, "upstream"),
|
||||
sync_timeout,
|
||||
f"upstream count>={expected} after switchover-during-writes",
|
||||
)
|
||||
assert self.wait_for_sync(
|
||||
check_both(downstream_client, "downstream"),
|
||||
sync_timeout,
|
||||
f"downstream count>={expected} after switchover-during-writes",
|
||||
)
|
||||
|
||||
up_res = upstream_client.query(collection_name=c_name, filter="", output_fields=["count(*)"])
|
||||
down_res = downstream_client.query(collection_name=c_name, filter="", output_fields=["count(*)"])
|
||||
up_cnt = up_res[0]["count(*)"] if up_res else 0
|
||||
down_cnt = down_res[0]["count(*)"] if down_res else 0
|
||||
assert up_cnt == down_cnt, (
|
||||
f"Count mismatch after switchover-during-writes: upstream={up_cnt}, downstream={down_cnt}"
|
||||
)
|
||||
|
||||
finally:
|
||||
logger.info("[SWITCHOVER] Restoring original topology after test_switchover_during_writes...")
|
||||
switchover_helper(source_cluster_id, target_cluster_id)
|
||||
self.log_test_end("test_switchover_during_writes", True, time.time() - start_time)
|
||||
|
||||
def test_switchover_with_all_data_types(
|
||||
self,
|
||||
upstream_client,
|
||||
downstream_client,
|
||||
sync_timeout,
|
||||
switchover_helper,
|
||||
source_cluster_id,
|
||||
target_cluster_id,
|
||||
):
|
||||
"""
|
||||
Switchover with comprehensive data types: insert 100 records, verify sync,
|
||||
switchover, verify scalar field sampling, switch back.
|
||||
"""
|
||||
start_time = time.time()
|
||||
c_name = self.gen_unique_name("test_sw_dtypes", max_length=50)
|
||||
|
||||
self.log_test_start(
|
||||
"test_switchover_with_all_data_types",
|
||||
"SWITCHOVER_ALL_DTYPES",
|
||||
c_name,
|
||||
)
|
||||
self._upstream_client = upstream_client
|
||||
self.resources_to_cleanup.append(("collection", c_name))
|
||||
|
||||
try:
|
||||
self.cleanup_collection(upstream_client, c_name)
|
||||
|
||||
schema = self.create_comprehensive_schema(upstream_client)
|
||||
upstream_client.create_collection(collection_name=c_name, schema=schema)
|
||||
|
||||
# Index every vector field; load_collection fails otherwise.
|
||||
index_params = upstream_client.prepare_index_params()
|
||||
index_params.add_index(field_name="float_vector", index_type="AUTOINDEX", metric_type="L2")
|
||||
index_params.add_index(field_name="float16_vector", index_type="AUTOINDEX", metric_type="L2")
|
||||
index_params.add_index(field_name="binary_vector", index_type="BIN_FLAT", metric_type="HAMMING")
|
||||
index_params.add_index(field_name="sparse_vector", index_type="SPARSE_INVERTED_INDEX", metric_type="IP")
|
||||
upstream_client.create_index(c_name, index_params)
|
||||
upstream_client.load_collection(c_name)
|
||||
|
||||
def check_create():
|
||||
return downstream_client.has_collection(c_name)
|
||||
|
||||
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {c_name}")
|
||||
|
||||
data = self.generate_comprehensive_test_data(100)
|
||||
upstream_client.insert(c_name, data)
|
||||
upstream_client.flush(c_name)
|
||||
|
||||
def check_100():
|
||||
try:
|
||||
res = downstream_client.query(
|
||||
collection_name=c_name,
|
||||
filter="",
|
||||
output_fields=["count(*)"],
|
||||
)
|
||||
cnt = res[0]["count(*)"] if res else 0
|
||||
logger.info(f"[SYNC_PROGRESS] downstream count: {cnt}/100")
|
||||
return cnt >= 100
|
||||
except Exception as e:
|
||||
logger.warning(f"Sync check failed: {e}")
|
||||
return False
|
||||
|
||||
assert self.wait_for_sync(check_100, sync_timeout, f"100-record sync {c_name}")
|
||||
|
||||
# Switchover
|
||||
logger.info("[SWITCHOVER] Initiating switchover with all data types...")
|
||||
switchover_helper(target_cluster_id, source_cluster_id)
|
||||
|
||||
# After switchover, verify scalar field sampling (new source is downstream)
|
||||
scalar_fields = [
|
||||
"bool_field",
|
||||
"int8_field",
|
||||
"int16_field",
|
||||
"int32_field",
|
||||
"int64_field",
|
||||
"float_field",
|
||||
"double_field",
|
||||
"varchar_field",
|
||||
]
|
||||
match_count, mismatch_count, details = self.verify_data_sampling(
|
||||
downstream_client, # new source
|
||||
upstream_client, # new target
|
||||
c_name,
|
||||
sample_ratio=0.3,
|
||||
output_fields=scalar_fields,
|
||||
)
|
||||
logger.info(f"[RESULT] All-dtypes sampling — match={match_count}, mismatch={mismatch_count}")
|
||||
assert mismatch_count == 0, f"Data mismatch after switchover with all types: {details[:5]}"
|
||||
|
||||
finally:
|
||||
logger.info("[SWITCHOVER] Restoring original topology after test_switchover_with_all_data_types...")
|
||||
switchover_helper(source_cluster_id, target_cluster_id)
|
||||
self.log_test_end(
|
||||
"test_switchover_with_all_data_types",
|
||||
True,
|
||||
time.time() - start_time,
|
||||
)
|
||||
|
||||
def test_switchover_with_loaded_collection(
|
||||
self,
|
||||
upstream_client,
|
||||
downstream_client,
|
||||
sync_timeout,
|
||||
switchover_helper,
|
||||
source_cluster_id,
|
||||
target_cluster_id,
|
||||
):
|
||||
"""
|
||||
Switchover with a loaded collection: verify search works on downstream before
|
||||
switchover; after switchover verify search works on both sides.
|
||||
"""
|
||||
start_time = time.time()
|
||||
c_name = self.gen_unique_name("test_sw_loaded", max_length=50)
|
||||
|
||||
self.log_test_start(
|
||||
"test_switchover_with_loaded_collection",
|
||||
"SWITCHOVER_LOADED",
|
||||
c_name,
|
||||
)
|
||||
self._upstream_client = upstream_client
|
||||
self.resources_to_cleanup.append(("collection", c_name))
|
||||
|
||||
try:
|
||||
self.cleanup_collection(upstream_client, c_name)
|
||||
|
||||
schema = self.create_default_schema(upstream_client)
|
||||
upstream_client.create_collection(collection_name=c_name, schema=schema)
|
||||
|
||||
index_params = upstream_client.prepare_index_params()
|
||||
index_params.add_index(
|
||||
field_name="vector",
|
||||
index_type="HNSW",
|
||||
metric_type="L2",
|
||||
params={"M": 16, "efConstruction": 200},
|
||||
)
|
||||
upstream_client.create_index(c_name, index_params)
|
||||
upstream_client.load_collection(c_name)
|
||||
|
||||
def check_create():
|
||||
return downstream_client.has_collection(c_name)
|
||||
|
||||
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {c_name}")
|
||||
|
||||
data = self.generate_test_data(200)
|
||||
upstream_client.insert(c_name, data)
|
||||
upstream_client.flush(c_name)
|
||||
|
||||
def check_200():
|
||||
try:
|
||||
res = downstream_client.query(
|
||||
collection_name=c_name,
|
||||
filter="",
|
||||
output_fields=["count(*)"],
|
||||
)
|
||||
cnt = res[0]["count(*)"] if res else 0
|
||||
logger.info(f"[SYNC_PROGRESS] downstream count: {cnt}/200")
|
||||
return cnt >= 200
|
||||
except Exception as e:
|
||||
logger.warning(f"Sync check failed: {e}")
|
||||
return False
|
||||
|
||||
assert self.wait_for_sync(check_200, sync_timeout, f"200-record sync {c_name}")
|
||||
|
||||
# Verify search works on downstream before switchover
|
||||
q_vec = [random.random() for _ in range(128)]
|
||||
pre_down_results = downstream_client.search(
|
||||
collection_name=c_name,
|
||||
data=[q_vec],
|
||||
anns_field="vector",
|
||||
search_params={"metric_type": "L2"},
|
||||
limit=5,
|
||||
output_fields=["id"],
|
||||
)
|
||||
assert pre_down_results and len(pre_down_results[0]) > 0, (
|
||||
"Search on downstream returned no results before switchover"
|
||||
)
|
||||
|
||||
# Switchover
|
||||
logger.info("[SWITCHOVER] Initiating switchover with loaded collection...")
|
||||
switchover_helper(target_cluster_id, source_cluster_id)
|
||||
|
||||
# Verify search still works on both sides after switchover
|
||||
for client, label in [(upstream_client, "upstream"), (downstream_client, "downstream")]:
|
||||
results = client.search(
|
||||
collection_name=c_name,
|
||||
data=[q_vec],
|
||||
anns_field="vector",
|
||||
search_params={"metric_type": "L2"},
|
||||
limit=5,
|
||||
output_fields=["id"],
|
||||
)
|
||||
assert results and len(results[0]) > 0, f"Search returned no results on {label} after switchover"
|
||||
logger.info(f"[VERIFY] Search on {label} after switchover returned {len(results[0])} results — OK")
|
||||
|
||||
finally:
|
||||
logger.info("[SWITCHOVER] Restoring original topology after test_switchover_with_loaded_collection...")
|
||||
switchover_helper(source_cluster_id, target_cluster_id)
|
||||
self.log_test_end(
|
||||
"test_switchover_with_loaded_collection",
|
||||
True,
|
||||
time.time() - start_time,
|
||||
)
|
||||
|
||||
def test_switchover_with_index(
|
||||
self,
|
||||
upstream_client,
|
||||
downstream_client,
|
||||
sync_timeout,
|
||||
switchover_helper,
|
||||
source_cluster_id,
|
||||
target_cluster_id,
|
||||
):
|
||||
"""
|
||||
Switchover after index creation: wait for index to sync, switchover, verify
|
||||
list_indexes returns the same index on both sides.
|
||||
"""
|
||||
start_time = time.time()
|
||||
c_name = self.gen_unique_name("test_sw_index", max_length=50)
|
||||
|
||||
self.log_test_start("test_switchover_with_index", "SWITCHOVER_INDEX", c_name)
|
||||
self._upstream_client = upstream_client
|
||||
self.resources_to_cleanup.append(("collection", c_name))
|
||||
|
||||
try:
|
||||
self.cleanup_collection(upstream_client, c_name)
|
||||
|
||||
schema = self.create_default_schema(upstream_client)
|
||||
upstream_client.create_collection(collection_name=c_name, schema=schema)
|
||||
|
||||
# Insert data first
|
||||
data = self.generate_test_data(200)
|
||||
upstream_client.insert(c_name, data)
|
||||
upstream_client.flush(c_name)
|
||||
|
||||
# Create HNSW index
|
||||
index_params = upstream_client.prepare_index_params()
|
||||
index_params.add_index(
|
||||
field_name="vector",
|
||||
index_type="HNSW",
|
||||
metric_type="COSINE",
|
||||
params={"M": 16, "efConstruction": 200},
|
||||
)
|
||||
upstream_client.create_index(c_name, index_params)
|
||||
|
||||
# Wait for index to sync to downstream
|
||||
def check_index_sync():
|
||||
try:
|
||||
indexes = downstream_client.list_indexes(c_name)
|
||||
result = len(indexes) > 0
|
||||
logger.info(f"[SYNC_PROGRESS] downstream indexes: {indexes} (has_index={result})")
|
||||
return result
|
||||
except Exception as e:
|
||||
logger.warning(f"Index sync check failed: {e}")
|
||||
return False
|
||||
|
||||
assert self.wait_for_sync(check_index_sync, sync_timeout, f"index sync {c_name}"), (
|
||||
f"Index did not sync to downstream within {sync_timeout}s"
|
||||
)
|
||||
|
||||
# Switchover
|
||||
logger.info("[SWITCHOVER] Initiating switchover with index...")
|
||||
switchover_helper(target_cluster_id, source_cluster_id)
|
||||
|
||||
# Verify list_indexes on both sides
|
||||
up_indexes = upstream_client.list_indexes(c_name)
|
||||
down_indexes = downstream_client.list_indexes(c_name)
|
||||
|
||||
logger.info(f"[VERIFY] After switchover — upstream indexes={up_indexes}, downstream indexes={down_indexes}")
|
||||
assert len(up_indexes) > 0, "No indexes on upstream after switchover"
|
||||
assert len(down_indexes) > 0, "No indexes on downstream after switchover"
|
||||
assert set(up_indexes) == set(down_indexes), (
|
||||
f"Index lists differ after switchover: up={up_indexes}, down={down_indexes}"
|
||||
)
|
||||
|
||||
finally:
|
||||
logger.info("[SWITCHOVER] Restoring original topology after test_switchover_with_index...")
|
||||
switchover_helper(source_cluster_id, target_cluster_id)
|
||||
self.log_test_end("test_switchover_with_index", True, time.time() - start_time)
|
||||
|
||||
def test_rapid_switchover_stress(
|
||||
self,
|
||||
upstream_client,
|
||||
downstream_client,
|
||||
sync_timeout,
|
||||
switchover_helper,
|
||||
source_cluster_id,
|
||||
target_cluster_id,
|
||||
):
|
||||
"""
|
||||
Rapid-switchover stress: loop 5 times — write 50 records to current source,
|
||||
switchover. After loop, wait 30 s and verify counts match. Restore original topology.
|
||||
"""
|
||||
start_time = time.time()
|
||||
c_name = self.gen_unique_name("test_sw_stress", max_length=50)
|
||||
|
||||
self.log_test_start("test_rapid_switchover_stress", "RAPID_SWITCHOVER_STRESS", c_name)
|
||||
self._upstream_client = upstream_client
|
||||
self.resources_to_cleanup.append(("collection", c_name))
|
||||
|
||||
try:
|
||||
self.cleanup_collection(upstream_client, c_name)
|
||||
|
||||
schema = self.create_manual_id_schema(upstream_client)
|
||||
upstream_client.create_collection(collection_name=c_name, schema=schema)
|
||||
|
||||
index_params = upstream_client.prepare_index_params()
|
||||
index_params.add_index(
|
||||
field_name="vector",
|
||||
index_type="HNSW",
|
||||
metric_type="COSINE",
|
||||
params={"M": 16, "efConstruction": 200},
|
||||
)
|
||||
upstream_client.create_index(c_name, index_params)
|
||||
upstream_client.load_collection(c_name)
|
||||
|
||||
def check_create():
|
||||
return downstream_client.has_collection(c_name)
|
||||
|
||||
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {c_name}")
|
||||
|
||||
# Track topology state
|
||||
current_source_id = source_cluster_id
|
||||
current_target_id = target_cluster_id
|
||||
current_source_client = upstream_client
|
||||
total_inserted = 0
|
||||
|
||||
for iteration in range(5):
|
||||
# Write 50 records to current source
|
||||
start_id = iteration * 50
|
||||
data = self.generate_test_data_with_id(50, start_id=start_id)
|
||||
current_source_client.insert(c_name, data)
|
||||
current_source_client.flush(c_name)
|
||||
total_inserted += 50
|
||||
|
||||
logger.info(
|
||||
f"[STRESS] Iteration {iteration + 1}/5: inserted 50 records "
|
||||
f"(total={total_inserted}), switching over..."
|
||||
)
|
||||
|
||||
# Switchover
|
||||
switchover_helper(current_target_id, current_source_id)
|
||||
|
||||
# Swap topology
|
||||
current_source_id, current_target_id = current_target_id, current_source_id
|
||||
current_source_client = downstream_client if current_source_id == target_cluster_id else upstream_client
|
||||
|
||||
logger.info(
|
||||
f"[STRESS] All 5 switchovers done. Waiting 30s for final sync (total_inserted={total_inserted})..."
|
||||
)
|
||||
time.sleep(30)
|
||||
|
||||
# Verify counts match on both sides
|
||||
up_res = upstream_client.query(collection_name=c_name, filter="", output_fields=["count(*)"])
|
||||
down_res = downstream_client.query(collection_name=c_name, filter="", output_fields=["count(*)"])
|
||||
up_cnt = up_res[0]["count(*)"] if up_res else 0
|
||||
down_cnt = down_res[0]["count(*)"] if down_res else 0
|
||||
|
||||
logger.info(f"[VERIFY] After stress — upstream={up_cnt}, downstream={down_cnt}, expected>={total_inserted}")
|
||||
assert up_cnt >= total_inserted, f"Upstream count {up_cnt} < expected {total_inserted} after stress"
|
||||
assert down_cnt >= total_inserted, f"Downstream count {down_cnt} < expected {total_inserted} after stress"
|
||||
assert up_cnt == down_cnt, f"Count mismatch after stress: upstream={up_cnt}, downstream={down_cnt}"
|
||||
|
||||
finally:
|
||||
# Restore to original topology
|
||||
logger.info("[SWITCHOVER] Restoring original topology after test_rapid_switchover_stress...")
|
||||
switchover_helper(source_cluster_id, target_cluster_id)
|
||||
self.log_test_end("test_rapid_switchover_stress", True, time.time() - start_time)
|
||||
|
||||
def test_failover_source_down(
|
||||
self,
|
||||
upstream_client,
|
||||
downstream_client,
|
||||
sync_timeout,
|
||||
switchover_helper,
|
||||
source_cluster_id,
|
||||
target_cluster_id,
|
||||
milvus_ns,
|
||||
):
|
||||
"""
|
||||
Failover when source goes down: insert 500 records, verify sync, kill source pods,
|
||||
verify target count >= 500, wait for source to recover and verify count >= 500.
|
||||
"""
|
||||
start_time = time.time()
|
||||
c_name = self.gen_unique_name("test_failover_src", max_length=50)
|
||||
|
||||
self.log_test_start("test_failover_source_down", "FAILOVER_SOURCE_DOWN", c_name)
|
||||
self._upstream_client = upstream_client
|
||||
self.resources_to_cleanup.append(("collection", c_name))
|
||||
|
||||
try:
|
||||
self.cleanup_collection(upstream_client, c_name)
|
||||
|
||||
schema = self.create_manual_id_schema(upstream_client)
|
||||
upstream_client.create_collection(collection_name=c_name, schema=schema)
|
||||
|
||||
index_params = upstream_client.prepare_index_params()
|
||||
index_params.add_index(
|
||||
field_name="vector",
|
||||
index_type="HNSW",
|
||||
metric_type="L2",
|
||||
params={"M": 16, "efConstruction": 200},
|
||||
)
|
||||
upstream_client.create_index(c_name, index_params)
|
||||
upstream_client.load_collection(c_name)
|
||||
|
||||
def check_create():
|
||||
return downstream_client.has_collection(c_name)
|
||||
|
||||
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {c_name}")
|
||||
|
||||
# Insert 500 records
|
||||
data = self.generate_test_data_with_id(500, start_id=0)
|
||||
upstream_client.insert(c_name, data)
|
||||
upstream_client.flush(c_name)
|
||||
|
||||
def check_500_downstream():
|
||||
try:
|
||||
res = downstream_client.query(
|
||||
collection_name=c_name,
|
||||
filter="",
|
||||
output_fields=["count(*)"],
|
||||
)
|
||||
cnt = res[0]["count(*)"] if res else 0
|
||||
logger.info(f"[SYNC_PROGRESS] downstream count: {cnt}/500")
|
||||
return cnt >= 500
|
||||
except Exception as e:
|
||||
logger.warning(f"Sync check failed: {e}")
|
||||
return False
|
||||
|
||||
assert self.wait_for_sync(check_500_downstream, sync_timeout, f"500-record sync {c_name}"), (
|
||||
f"Downstream did not receive 500 records within {sync_timeout}s"
|
||||
)
|
||||
|
||||
# Kill source pods forcefully
|
||||
logger.info(f"[FAILOVER] Killing source pods (instance={source_cluster_id}, ns={milvus_ns})...")
|
||||
kill_cmd = [
|
||||
"kubectl",
|
||||
"delete",
|
||||
"pods",
|
||||
"-l",
|
||||
f"app.kubernetes.io/instance={source_cluster_id}",
|
||||
"-n",
|
||||
milvus_ns,
|
||||
"--grace-period=0",
|
||||
"--force",
|
||||
]
|
||||
result = subprocess.run(kill_cmd, capture_output=True, text=True)
|
||||
logger.info(
|
||||
f"[FAILOVER] kubectl output: stdout={result.stdout!r}, stderr={result.stderr!r}, rc={result.returncode}"
|
||||
)
|
||||
|
||||
# Wait 60 s and verify target still has data
|
||||
logger.info("[FAILOVER] Waiting 60s after pod kill...")
|
||||
time.sleep(60)
|
||||
|
||||
def check_target_intact():
|
||||
try:
|
||||
res = downstream_client.query(
|
||||
collection_name=c_name,
|
||||
filter="",
|
||||
output_fields=["count(*)"],
|
||||
)
|
||||
cnt = res[0]["count(*)"] if res else 0
|
||||
logger.info(f"[FAILOVER] Target count after kill: {cnt}")
|
||||
return cnt >= 500
|
||||
except Exception as e:
|
||||
logger.warning(f"Target check failed: {e}")
|
||||
return False
|
||||
|
||||
assert self.wait_for_sync(check_target_intact, 30, f"target intact after source kill {c_name}"), (
|
||||
"Target lost data after source pod kill"
|
||||
)
|
||||
|
||||
# Wait 120 s more for source to recover, then verify
|
||||
logger.info("[FAILOVER] Waiting 120s for source recovery...")
|
||||
time.sleep(120)
|
||||
|
||||
def check_source_recovered():
|
||||
try:
|
||||
res = upstream_client.query(
|
||||
collection_name=c_name,
|
||||
filter="",
|
||||
output_fields=["count(*)"],
|
||||
)
|
||||
cnt = res[0]["count(*)"] if res else 0
|
||||
logger.info(f"[FAILOVER] Source count after recovery: {cnt}")
|
||||
return cnt >= 500
|
||||
except Exception as e:
|
||||
logger.warning(f"Source recovery check failed: {e}")
|
||||
return False
|
||||
|
||||
assert self.wait_for_sync(check_source_recovered, 60, f"source recovery {c_name}"), (
|
||||
"Source did not recover with expected data count within timeout"
|
||||
)
|
||||
|
||||
finally:
|
||||
self.log_test_end("test_failover_source_down", True, time.time() - start_time)
|
||||
@@ -24,13 +24,13 @@ max_top_k = 16384
|
||||
max_nq = 16384
|
||||
max_partition_num = 1024
|
||||
max_role_num = 10
|
||||
default_partition_num = 16 # default num_partitions for partition key feature
|
||||
default_partition_num = 16 # default num_partitions for partition key feature
|
||||
default_segment_row_limit = 1000
|
||||
default_server_segment_row_limit = 1024 * 512
|
||||
default_alias = "default"
|
||||
default_user = "root"
|
||||
default_password = "Milvus"
|
||||
default_primary_field_name = 'pk'
|
||||
default_primary_field_name = "pk"
|
||||
default_bool_field_name = "bool"
|
||||
default_int8_field_name = "int8"
|
||||
default_int16_field_name = "int16"
|
||||
@@ -63,13 +63,13 @@ default_reranker_field_name = "reranker_field"
|
||||
default_new_field_name = "field_new"
|
||||
|
||||
all_vector_types = [
|
||||
DataType.FLOAT_VECTOR,
|
||||
DataType.FLOAT16_VECTOR,
|
||||
DataType.BFLOAT16_VECTOR,
|
||||
DataType.SPARSE_FLOAT_VECTOR,
|
||||
DataType.INT8_VECTOR,
|
||||
DataType.BINARY_VECTOR,
|
||||
]
|
||||
DataType.FLOAT_VECTOR,
|
||||
DataType.FLOAT16_VECTOR,
|
||||
DataType.BFLOAT16_VECTOR,
|
||||
DataType.SPARSE_FLOAT_VECTOR,
|
||||
DataType.INT8_VECTOR,
|
||||
DataType.BINARY_VECTOR,
|
||||
]
|
||||
|
||||
default_metric_for_vector_type = {
|
||||
DataType.FLOAT_VECTOR: "COSINE",
|
||||
@@ -90,35 +90,51 @@ all_scalar_data_types = [
|
||||
DataType.DOUBLE,
|
||||
DataType.VARCHAR,
|
||||
DataType.ARRAY,
|
||||
DataType.JSON,
|
||||
DataType.JSON,
|
||||
DataType.GEOMETRY,
|
||||
DataType.TIMESTAMPTZ
|
||||
]
|
||||
DataType.TIMESTAMPTZ,
|
||||
]
|
||||
|
||||
default_field_name_map = {
|
||||
DataType.INT8: default_int8_field_name,
|
||||
DataType.INT16: default_int16_field_name,
|
||||
DataType.INT32: default_int32_field_name,
|
||||
DataType.INT64: default_int64_field_name,
|
||||
DataType.BOOL: default_bool_field_name,
|
||||
DataType.FLOAT: default_float_field_name,
|
||||
DataType.DOUBLE: default_double_field_name,
|
||||
DataType.VARCHAR: default_string_field_name,
|
||||
DataType.ARRAY: default_array_field_name,
|
||||
DataType.JSON: default_json_field_name,
|
||||
DataType.FLOAT_VECTOR: default_float_vec_field_name,
|
||||
DataType.FLOAT16_VECTOR: default_float16_vec_field_name,
|
||||
DataType.BFLOAT16_VECTOR: default_bfloat16_vec_field_name,
|
||||
DataType.SPARSE_FLOAT_VECTOR: default_sparse_vec_field_name,
|
||||
DataType.INT8_VECTOR: default_int8_vec_field_name,
|
||||
DataType.BINARY_VECTOR: default_binary_vec_field_name,
|
||||
}
|
||||
DataType.INT8: default_int8_field_name,
|
||||
DataType.INT16: default_int16_field_name,
|
||||
DataType.INT32: default_int32_field_name,
|
||||
DataType.INT64: default_int64_field_name,
|
||||
DataType.BOOL: default_bool_field_name,
|
||||
DataType.FLOAT: default_float_field_name,
|
||||
DataType.DOUBLE: default_double_field_name,
|
||||
DataType.VARCHAR: default_string_field_name,
|
||||
DataType.ARRAY: default_array_field_name,
|
||||
DataType.JSON: default_json_field_name,
|
||||
DataType.FLOAT_VECTOR: default_float_vec_field_name,
|
||||
DataType.FLOAT16_VECTOR: default_float16_vec_field_name,
|
||||
DataType.BFLOAT16_VECTOR: default_bfloat16_vec_field_name,
|
||||
DataType.SPARSE_FLOAT_VECTOR: default_sparse_vec_field_name,
|
||||
DataType.INT8_VECTOR: default_int8_vec_field_name,
|
||||
DataType.BINARY_VECTOR: default_binary_vec_field_name,
|
||||
}
|
||||
|
||||
append_vector_type = [DataType.FLOAT16_VECTOR, DataType.BFLOAT16_VECTOR, DataType.SPARSE_FLOAT_VECTOR, DataType.INT8_VECTOR]
|
||||
all_dense_vector_types = [DataType.FLOAT_VECTOR, DataType.FLOAT16_VECTOR, DataType.BFLOAT16_VECTOR, DataType.INT8_VECTOR]
|
||||
all_float_vector_dtypes = [DataType.FLOAT_VECTOR, DataType.FLOAT16_VECTOR, DataType.BFLOAT16_VECTOR, DataType.SPARSE_FLOAT_VECTOR, DataType.INT8_VECTOR]
|
||||
append_vector_type = [
|
||||
DataType.FLOAT16_VECTOR,
|
||||
DataType.BFLOAT16_VECTOR,
|
||||
DataType.SPARSE_FLOAT_VECTOR,
|
||||
DataType.INT8_VECTOR,
|
||||
]
|
||||
all_dense_vector_types = [
|
||||
DataType.FLOAT_VECTOR,
|
||||
DataType.FLOAT16_VECTOR,
|
||||
DataType.BFLOAT16_VECTOR,
|
||||
DataType.INT8_VECTOR,
|
||||
]
|
||||
all_float_vector_dtypes = [
|
||||
DataType.FLOAT_VECTOR,
|
||||
DataType.FLOAT16_VECTOR,
|
||||
DataType.BFLOAT16_VECTOR,
|
||||
DataType.SPARSE_FLOAT_VECTOR,
|
||||
DataType.INT8_VECTOR,
|
||||
]
|
||||
default_partition_name = "_default"
|
||||
default_resource_group_name = '__default_resource_group'
|
||||
default_resource_group_name = "__default_resource_group"
|
||||
default_resource_group_capacity = 1000000
|
||||
default_tag = "1970_01_01"
|
||||
row_count = "row_count"
|
||||
@@ -179,43 +195,37 @@ rows_all_data_type_file_path = "/tmp/rows_all_data_type"
|
||||
|
||||
"""" List of parameters used to pass """
|
||||
invalid_resource_names = [
|
||||
None, # None
|
||||
" ", # space
|
||||
"", # empty
|
||||
"12name", # start with number
|
||||
"n12 ame", # contain space
|
||||
"n-ame", # contain hyphen
|
||||
"nam(e)", # contain special character
|
||||
"name中文", # contain Chinese character
|
||||
"name%$#", # contain special character
|
||||
"".join("a" for i in range(max_name_length + 1))] # exceed max length
|
||||
None, # None
|
||||
" ", # space
|
||||
"", # empty
|
||||
"12name", # start with number
|
||||
"n12 ame", # contain space
|
||||
"n-ame", # contain hyphen
|
||||
"nam(e)", # contain special character
|
||||
"name中文", # contain Chinese character
|
||||
"name%$#", # contain special character
|
||||
"".join("a" for i in range(max_name_length + 1)),
|
||||
] # exceed max length
|
||||
|
||||
valid_resource_names = [
|
||||
"name", # valid name
|
||||
"_name", # start with underline
|
||||
"_12name", # start with underline and contains number
|
||||
"n12ame_", # end with letter and contains number and underline
|
||||
"nam_e", # contains underline
|
||||
"".join("a" for i in range(max_name_length))] # max length
|
||||
"name", # valid name
|
||||
"_name", # start with underline
|
||||
"_12name", # start with underline and contains number
|
||||
"n12ame_", # end with letter and contains number and underline
|
||||
"nam_e", # contains underline
|
||||
"".join("a" for i in range(max_name_length)),
|
||||
] # max length
|
||||
|
||||
invalid_dims = [min_dim-1, 32.1, -32, "vii", "十六", max_dim+1]
|
||||
invalid_dims = [min_dim - 1, 32.1, -32, "vii", "十六", max_dim + 1]
|
||||
|
||||
get_not_string = [
|
||||
[],
|
||||
{},
|
||||
None,
|
||||
(1,),
|
||||
1,
|
||||
1.0,
|
||||
[1, "2", 3]
|
||||
]
|
||||
get_not_string = [[], {}, None, (1,), 1, 1.0, [1, "2", 3]]
|
||||
|
||||
get_invalid_vectors = [
|
||||
"1*2",
|
||||
[1],
|
||||
[1, 2],
|
||||
[" "],
|
||||
['a'],
|
||||
["a"],
|
||||
[None],
|
||||
None,
|
||||
(1, 2),
|
||||
@@ -225,7 +235,7 @@ get_invalid_vectors = [
|
||||
"String",
|
||||
" siede ",
|
||||
"中文",
|
||||
"a".join("a" for i in range(256))
|
||||
"a".join("a" for i in range(256)),
|
||||
]
|
||||
|
||||
get_invalid_ints = [
|
||||
@@ -239,7 +249,7 @@ get_invalid_ints = [
|
||||
"String",
|
||||
"=c",
|
||||
"中文",
|
||||
"a".join("a" for i in range(256))
|
||||
"a".join("a" for i in range(256)),
|
||||
]
|
||||
|
||||
get_invalid_dict = [
|
||||
@@ -254,7 +264,7 @@ get_invalid_dict = [
|
||||
{1: 1},
|
||||
{"中文": 1},
|
||||
{"%$#": ["a"]},
|
||||
{"a".join("a" for i in range(256)): "a"}
|
||||
{"a".join("a" for i in range(256)): "a"},
|
||||
]
|
||||
|
||||
get_invalid_metric_type = [
|
||||
@@ -269,55 +279,105 @@ get_invalid_metric_type = [
|
||||
"(mn)",
|
||||
"中文",
|
||||
"%$#",
|
||||
"".join("a" for i in range(max_name_length + 1))]
|
||||
|
||||
get_dict_without_host_port = [
|
||||
{"host": "host"},
|
||||
{"": ""}
|
||||
"".join("a" for i in range(max_name_length + 1)),
|
||||
]
|
||||
|
||||
get_wrong_format_dict = [
|
||||
{"host": "string_host", "port": {}},
|
||||
{"host": 0, "port": 19520}
|
||||
]
|
||||
get_dict_without_host_port = [{"host": "host"}, {"": ""}]
|
||||
|
||||
get_wrong_format_dict = [{"host": "string_host", "port": {}}, {"host": 0, "port": 19520}]
|
||||
|
||||
get_all_kind_data_distribution = [
|
||||
1, np.float64(1.0), np.double(1.0), 9707199254740993.0, 9707199254740992,
|
||||
'1', '123', '321', '213', True, False, None, [1, 2], [1.0, 2], {}, {"a": 1},
|
||||
{'a': 1.0}, {'a': 9707199254740993.0}, {'a': 9707199254740992}, {'a': '1'}, {'a': '123'},
|
||||
{'a': '321'}, {'a': '213'}, {'a': True}, {'a': [1, 2, 3]}, {'a': [1.0, 2, '1']}, {'a': [1.0, 2]},
|
||||
{'a': None}, {'a': {'b': 1}}, {'a': {'b': 1.0}}, {'a': [{'b': 1}, 2.0, np.double(3.0), '4', True, [1, 3.0], None]}
|
||||
1,
|
||||
np.float64(1.0),
|
||||
np.double(1.0),
|
||||
9707199254740993.0,
|
||||
9707199254740992,
|
||||
"1",
|
||||
"123",
|
||||
"321",
|
||||
"213",
|
||||
True,
|
||||
False,
|
||||
None,
|
||||
[1, 2],
|
||||
[1.0, 2],
|
||||
{},
|
||||
{"a": 1},
|
||||
{"a": 1.0},
|
||||
{"a": 9707199254740993.0},
|
||||
{"a": 9707199254740992},
|
||||
{"a": "1"},
|
||||
{"a": "123"},
|
||||
{"a": "321"},
|
||||
{"a": "213"},
|
||||
{"a": True},
|
||||
{"a": [1, 2, 3]},
|
||||
{"a": [1.0, 2, "1"]},
|
||||
{"a": [1.0, 2]},
|
||||
{"a": None},
|
||||
{"a": {"b": 1}},
|
||||
{"a": {"b": 1.0}},
|
||||
{"a": [{"b": 1}, 2.0, np.double(3.0), "4", True, [1, 3.0], None]},
|
||||
]
|
||||
|
||||
""" Specially defined list """
|
||||
L0_index_types = ["IVF_SQ8", "HNSW", "DISKANN"]
|
||||
all_index_types = ["FLAT", "IVF_FLAT", "IVF_SQ8", "IVF_PQ",
|
||||
"IVF_RABITQ",
|
||||
"HNSW", "SCANN", "DISKANN",
|
||||
"BIN_FLAT", "BIN_IVF_FLAT",
|
||||
"SPARSE_INVERTED_INDEX", "SPARSE_WAND",
|
||||
"GPU_IVF_FLAT", "GPU_IVF_PQ"]
|
||||
all_index_types = [
|
||||
"FLAT",
|
||||
"IVF_FLAT",
|
||||
"IVF_SQ8",
|
||||
"IVF_PQ",
|
||||
"IVF_RABITQ",
|
||||
"HNSW",
|
||||
"SCANN",
|
||||
"DISKANN",
|
||||
"BIN_FLAT",
|
||||
"BIN_IVF_FLAT",
|
||||
"SPARSE_INVERTED_INDEX",
|
||||
"SPARSE_WAND",
|
||||
"GPU_IVF_FLAT",
|
||||
"GPU_IVF_PQ",
|
||||
]
|
||||
|
||||
all_dense_float_index_types = ["FLAT", "IVF_FLAT", "IVF_SQ8", "IVF_PQ",
|
||||
"IVF_RABITQ", "HNSW", "SCANN", "DISKANN"]
|
||||
all_dense_float_index_types = ["FLAT", "IVF_FLAT", "IVF_SQ8", "IVF_PQ", "IVF_RABITQ", "HNSW", "SCANN", "DISKANN"]
|
||||
|
||||
inverted_index_algo = ['TAAT_NAIVE', 'DAAT_WAND', 'DAAT_MAXSCORE']
|
||||
inverted_index_algo = ["TAAT_NAIVE", "DAAT_WAND", "DAAT_MAXSCORE"]
|
||||
|
||||
int8_vector_index = ["HNSW"]
|
||||
|
||||
default_all_indexes_params = [{}, {"nlist": 128}, {"nlist": 128}, {"nlist": 128, "m": 16, "nbits": 8},
|
||||
{"nlist": 128, "refine": 'true', "refine_type": "SQ8"},
|
||||
{"M": 32, "efConstruction": 360}, {"nlist": 128}, {},
|
||||
{}, {"nlist": 64},
|
||||
{}, {"drop_ratio_build": 0.2},
|
||||
{"nlist": 64}, {"nlist": 64, "m": 16, "nbits": 8}]
|
||||
default_all_indexes_params = [
|
||||
{},
|
||||
{"nlist": 128},
|
||||
{"nlist": 128},
|
||||
{"nlist": 128, "m": 16, "nbits": 8},
|
||||
{"nlist": 128, "refine": "true", "refine_type": "SQ8"},
|
||||
{"M": 32, "efConstruction": 360},
|
||||
{"nlist": 128},
|
||||
{},
|
||||
{},
|
||||
{"nlist": 64},
|
||||
{},
|
||||
{"drop_ratio_build": 0.2},
|
||||
{"nlist": 64},
|
||||
{"nlist": 64, "m": 16, "nbits": 8},
|
||||
]
|
||||
|
||||
default_all_search_params_params = [{}, {"nprobe": 32}, {"nprobe": 32}, {"nprobe": 32},
|
||||
{"nprobe": 8, "rbq_bits_query": 8, "refine_k": 10.0},
|
||||
{"ef": 100}, {"nprobe": 32, "reorder_k": 100}, {"search_list": 30},
|
||||
{}, {"nprobe": 32},
|
||||
{"drop_ratio_search": "0.2"}, {"drop_ratio_search": "0.2"},
|
||||
{}, {}]
|
||||
default_all_search_params_params = [
|
||||
{},
|
||||
{"nprobe": 32},
|
||||
{"nprobe": 32},
|
||||
{"nprobe": 32},
|
||||
{"nprobe": 8, "rbq_bits_query": 8, "refine_k": 10.0},
|
||||
{"ef": 100},
|
||||
{"nprobe": 32, "reorder_k": 100},
|
||||
{"search_list": 30},
|
||||
{},
|
||||
{"nprobe": 32},
|
||||
{"drop_ratio_search": "0.2"},
|
||||
{"drop_ratio_search": "0.2"},
|
||||
{},
|
||||
{},
|
||||
]
|
||||
|
||||
Handler_type = ["GRPC", "HTTP"]
|
||||
binary_supported_index_types = ["BIN_FLAT", "BIN_IVF_FLAT"]
|
||||
@@ -336,10 +396,20 @@ numeric_supported_index_types = ["STL_SORT", "INVERTED", "AUTOINDEX", ""]
|
||||
|
||||
default_flat_index = {"index_type": "FLAT", "params": {}, "metric_type": default_L0_metric}
|
||||
default_bin_flat_index = {"index_type": "BIN_FLAT", "params": {}, "metric_type": "JACCARD"}
|
||||
default_sparse_inverted_index = {"index_type": "SPARSE_INVERTED_INDEX", "metric_type": "IP",
|
||||
"params": {"drop_ratio_build": 0.2}}
|
||||
default_text_sparse_inverted_index = {"index_type": "SPARSE_INVERTED_INDEX", "metric_type": "BM25",
|
||||
"params": {"drop_ratio_build": 0.2, "bm25_k1": 1.5, "bm25_b": 0.75,}}
|
||||
default_sparse_inverted_index = {
|
||||
"index_type": "SPARSE_INVERTED_INDEX",
|
||||
"metric_type": "IP",
|
||||
"params": {"drop_ratio_build": 0.2},
|
||||
}
|
||||
default_text_sparse_inverted_index = {
|
||||
"index_type": "SPARSE_INVERTED_INDEX",
|
||||
"metric_type": "BM25",
|
||||
"params": {
|
||||
"drop_ratio_build": 0.2,
|
||||
"bm25_k1": 1.5,
|
||||
"bm25_b": 0.75,
|
||||
},
|
||||
}
|
||||
default_search_params = {"params": {"nlist": 128}}
|
||||
default_search_ip_params = {"metric_type": "IP", "params": {"nlist": 128}}
|
||||
default_search_binary_params = {"metric_type": "JACCARD", "params": {"nprobe": 32}}
|
||||
@@ -349,46 +419,108 @@ default_diskann_index = {"index_type": "DISKANN", "metric_type": default_L0_metr
|
||||
default_diskann_search_params = {"params": {"search_list": 30}}
|
||||
default_sparse_search_params = {"metric_type": "IP", "params": {"drop_ratio_search": "0.2"}}
|
||||
default_text_sparse_search_params = {"metric_type": "BM25", "params": {}}
|
||||
built_in_privilege_groups = ["CollectionReadWrite", "CollectionReadOnly", "CollectionAdmin",
|
||||
"DatabaseReadWrite", "DatabaseReadOnly", "DatabaseAdmin",
|
||||
"ClusterReadWrite", "ClusterReadOnly", "ClusterAdmin"]
|
||||
privilege_group_privilege_dict = {"Query": False, "Search": False, "GetLoadState": False,
|
||||
"GetLoadingProgress": False, "HasPartition": False, "ShowPartitions": False,
|
||||
"ShowCollections": False, "ListAliases": False, "ListDatabases": False,
|
||||
"DescribeDatabase": False, "DescribeAlias": False, "GetStatistics": False,
|
||||
"CreateIndex": False, "DropIndex": False, "CreatePartition": False,
|
||||
"DropPartition": False, "Load": False, "Release": False,
|
||||
"Insert": False, "Delete": False, "Upsert": False,
|
||||
"Import": False, "Flush": False, "Compaction": False,
|
||||
"LoadBalance": False, "RenameCollection": False, "CreateAlias": False,
|
||||
"DropAlias": False, "CreateCollection": False, "DropCollection": False,
|
||||
"CreateOwnership": False, "DropOwnership": False, "SelectOwnership": False,
|
||||
"ManageOwnership": False, "UpdateUser": False, "SelectUser": False,
|
||||
"CreateResourceGroup": False, "DropResourceGroup": False,
|
||||
"UpdateResourceGroups": False,
|
||||
"DescribeResourceGroup": False, "ListResourceGroups": False, "TransferNode": False,
|
||||
"TransferReplica": False, "CreateDatabase": False, "DropDatabase": False,
|
||||
"AlterDatabase": False, "FlushAll": False, "ListPrivilegeGroups": False,
|
||||
"CreatePrivilegeGroup": False, "DropPrivilegeGroup": False,
|
||||
"OperatePrivilegeGroup": False}
|
||||
all_expr_fields = [default_int8_field_name, default_int16_field_name,
|
||||
default_int32_field_name, default_int64_field_name,
|
||||
default_float_field_name, default_double_field_name,
|
||||
default_string_field_name, default_bool_field_name,
|
||||
default_int8_array_field_name, default_int16_array_field_name,
|
||||
default_int32_array_field_name, default_int64_array_field_name,
|
||||
default_bool_array_field_name, default_float_array_field_name,
|
||||
default_double_array_field_name, default_string_array_field_name]
|
||||
built_in_privilege_groups = [
|
||||
"CollectionReadWrite",
|
||||
"CollectionReadOnly",
|
||||
"CollectionAdmin",
|
||||
"DatabaseReadWrite",
|
||||
"DatabaseReadOnly",
|
||||
"DatabaseAdmin",
|
||||
"ClusterReadWrite",
|
||||
"ClusterReadOnly",
|
||||
"ClusterAdmin",
|
||||
]
|
||||
privilege_group_privilege_dict = {
|
||||
"Query": False,
|
||||
"Search": False,
|
||||
"GetLoadState": False,
|
||||
"GetLoadingProgress": False,
|
||||
"HasPartition": False,
|
||||
"ShowPartitions": False,
|
||||
"ShowCollections": False,
|
||||
"ListAliases": False,
|
||||
"ListDatabases": False,
|
||||
"DescribeDatabase": False,
|
||||
"DescribeAlias": False,
|
||||
"GetStatistics": False,
|
||||
"CreateIndex": False,
|
||||
"DropIndex": False,
|
||||
"CreatePartition": False,
|
||||
"DropPartition": False,
|
||||
"Load": False,
|
||||
"Release": False,
|
||||
"Insert": False,
|
||||
"Delete": False,
|
||||
"Upsert": False,
|
||||
"Import": False,
|
||||
"Flush": False,
|
||||
"Compaction": False,
|
||||
"LoadBalance": False,
|
||||
"RenameCollection": False,
|
||||
"CreateAlias": False,
|
||||
"DropAlias": False,
|
||||
"CreateCollection": False,
|
||||
"DropCollection": False,
|
||||
"CreateOwnership": False,
|
||||
"DropOwnership": False,
|
||||
"SelectOwnership": False,
|
||||
"ManageOwnership": False,
|
||||
"UpdateUser": False,
|
||||
"SelectUser": False,
|
||||
"CreateResourceGroup": False,
|
||||
"DropResourceGroup": False,
|
||||
"UpdateResourceGroups": False,
|
||||
"DescribeResourceGroup": False,
|
||||
"ListResourceGroups": False,
|
||||
"TransferNode": False,
|
||||
"TransferReplica": False,
|
||||
"CreateDatabase": False,
|
||||
"DropDatabase": False,
|
||||
"AlterDatabase": False,
|
||||
"FlushAll": False,
|
||||
"ListPrivilegeGroups": False,
|
||||
"CreatePrivilegeGroup": False,
|
||||
"DropPrivilegeGroup": False,
|
||||
"OperatePrivilegeGroup": False,
|
||||
}
|
||||
all_expr_fields = [
|
||||
default_int8_field_name,
|
||||
default_int16_field_name,
|
||||
default_int32_field_name,
|
||||
default_int64_field_name,
|
||||
default_float_field_name,
|
||||
default_double_field_name,
|
||||
default_string_field_name,
|
||||
default_bool_field_name,
|
||||
default_int8_array_field_name,
|
||||
default_int16_array_field_name,
|
||||
default_int32_array_field_name,
|
||||
default_int64_array_field_name,
|
||||
default_bool_array_field_name,
|
||||
default_float_array_field_name,
|
||||
default_double_array_field_name,
|
||||
default_string_array_field_name,
|
||||
]
|
||||
|
||||
not_supported_json_cast_types = [
|
||||
DataType.INT8.name,
|
||||
DataType.INT16.name,
|
||||
DataType.INT32.name,
|
||||
DataType.INT64.name,
|
||||
DataType.FLOAT.name,
|
||||
DataType.ARRAY.name,
|
||||
DataType.FLOAT_VECTOR.name,
|
||||
DataType.FLOAT16_VECTOR.name,
|
||||
DataType.BFLOAT16_VECTOR.name,
|
||||
DataType.BINARY_VECTOR.name,
|
||||
DataType.SPARSE_FLOAT_VECTOR.name,
|
||||
DataType.INT8_VECTOR.name,
|
||||
]
|
||||
|
||||
not_supported_json_cast_types = [DataType.INT8.name, DataType.INT16.name, DataType.INT32.name,
|
||||
DataType.INT64.name, DataType.FLOAT.name,
|
||||
DataType.ARRAY.name, DataType.FLOAT_VECTOR.name,
|
||||
DataType.FLOAT16_VECTOR.name, DataType.BFLOAT16_VECTOR.name,
|
||||
DataType.BINARY_VECTOR.name,
|
||||
DataType.SPARSE_FLOAT_VECTOR.name, DataType.INT8_VECTOR.name]
|
||||
|
||||
class CheckTasks:
|
||||
""" The name of the method used to check the result """
|
||||
"""The name of the method used to check the result"""
|
||||
|
||||
check_nothing = "check_nothing"
|
||||
err_res = "error_response"
|
||||
ccr = "check_connection_result"
|
||||
@@ -452,6 +584,7 @@ class CaseLabel:
|
||||
GPU:
|
||||
For GPU supported cases
|
||||
"""
|
||||
|
||||
L0 = "L0"
|
||||
L1 = "L1"
|
||||
L2 = "L2"
|
||||
|
||||
@@ -1,16 +1,3 @@
|
||||
[project]
|
||||
name = "milvus-tests"
|
||||
version = "0.0.0"
|
||||
description = "Milvus integration/e2e test suite tooling (ruff config host)"
|
||||
requires-python = ">=3.10"
|
||||
dependencies = []
|
||||
|
||||
[dependency-groups]
|
||||
dev = [
|
||||
"ruff>=0.6.0",
|
||||
]
|
||||
|
||||
[tool.ruff]
|
||||
line-length = 120
|
||||
target-version = "py310"
|
||||
extend-exclude = [
|
||||
@@ -20,7 +7,7 @@ extend-exclude = [
|
||||
"*.ipynb_checkpoints",
|
||||
]
|
||||
|
||||
[tool.ruff.lint]
|
||||
[lint]
|
||||
select = [
|
||||
"E", # pycodestyle errors
|
||||
"F", # pyflakes
|
||||
@@ -33,13 +20,13 @@ ignore = [
|
||||
"E741", # ambiguous variable names (l, I, O) common in test code
|
||||
]
|
||||
|
||||
[tool.ruff.lint.per-file-ignores]
|
||||
[lint.per-file-ignores]
|
||||
"__init__.py" = ["F401", "F403"]
|
||||
"conftest.py" = ["F401", "F811"]
|
||||
"**/testcases/**/*.py" = ["E402"]
|
||||
"**/chaos/**/*.py" = ["E402"]
|
||||
|
||||
[tool.ruff.format]
|
||||
[format]
|
||||
quote-style = "double"
|
||||
indent-style = "space"
|
||||
line-ending = "auto"
|
||||
Generated
-43
@@ -1,43 +0,0 @@
|
||||
version = 1
|
||||
revision = 3
|
||||
requires-python = ">=3.10"
|
||||
|
||||
[[package]]
|
||||
name = "milvus-tests"
|
||||
version = "0.0.0"
|
||||
source = { virtual = "." }
|
||||
|
||||
[package.dev-dependencies]
|
||||
dev = [
|
||||
{ name = "ruff" },
|
||||
]
|
||||
|
||||
[package.metadata]
|
||||
|
||||
[package.metadata.requires-dev]
|
||||
dev = [{ name = "ruff", specifier = ">=0.6.0" }]
|
||||
|
||||
[[package]]
|
||||
name = "ruff"
|
||||
version = "0.15.11"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/e4/8d/192f3d7103816158dfd5ea50d098ef2aec19194e6cbccd4b3485bdb2eb2d/ruff-0.15.11.tar.gz", hash = "sha256:f092b21708bf0e7437ce9ada249dfe688ff9a0954fc94abab05dcea7dcd29c33", size = 4637264, upload-time = "2026-04-16T18:46:26.58Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/02/1e/6aca3427f751295ab011828e15e9bf452200ac74484f1db4be0197b8170b/ruff-0.15.11-py3-none-linux_armv6l.whl", hash = "sha256:e927cfff503135c558eb581a0c9792264aae9507904eb27809cdcff2f2c847b7", size = 10607943, upload-time = "2026-04-16T18:46:05.967Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/e7/26/1341c262e74f36d4e84f3d6f4df0ac68cd53331a66bfc5080daa17c84c0b/ruff-0.15.11-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:7a1b5b2938d8f890b76084d4fa843604d787a912541eae85fd7e233398bbb73e", size = 10988592, upload-time = "2026-04-16T18:46:00.742Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/03/71/850b1d6ffa9564fbb6740429bad53df1094082fe515c8c1e74b6d8d05f18/ruff-0.15.11-py3-none-macosx_11_0_arm64.whl", hash = "sha256:d4176f3d194afbdaee6e41b9ccb1a2c287dba8700047df474abfbe773825d1cb", size = 10338501, upload-time = "2026-04-16T18:46:03.723Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/f2/11/cc1284d3e298c45a817a6aadb6c3e1d70b45c9b36d8d9cce3387b495a03a/ruff-0.15.11-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:3b17c886fb88203ced3afe7f14e8d5ae96e9d2f4ccc0ee66aa19f2c2675a27e4", size = 10670693, upload-time = "2026-04-16T18:46:41.941Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/ce/9e/f8288b034ab72b371513c13f9a41d9ba3effac54e24bfb467b007daee2ca/ruff-0.15.11-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:49fafa220220afe7758a487b048de4c8f9f767f37dfefad46b9dd06759d003eb", size = 10416177, upload-time = "2026-04-16T18:46:21.717Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/85/71/504d79abfd3d92532ba6bbe3d1c19fada03e494332a59e37c7c2dabae427/ruff-0.15.11-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:f2ab8427e74a00d93b8bda1307b1e60970d40f304af38bccb218e056c220120d", size = 11221886, upload-time = "2026-04-16T18:46:15.086Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/43/5a/947e6ab7a5ad603d65b474be15a4cbc6d29832db5d762cd142e4e3a74164/ruff-0.15.11-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:195072c0c8e1fc8f940652073df082e37a5d9cb43b4ab1e4d0566ab8977a13b7", size = 12075183, upload-time = "2026-04-16T18:46:07.944Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/9f/a1/0b7bb6268775fdd3a0818aee8efd8f5b4e231d24dd4d528ced2534023182/ruff-0.15.11-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:a3a0996d486af3920dec930a2e7daed4847dfc12649b537a9335585ada163e9e", size = 11516575, upload-time = "2026-04-16T18:46:31.687Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/30/c3/bb5168fc4d233cc06e95f482770d0f3c87945a0cd9f614b90ea8dc2f2833/ruff-0.15.11-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:1bef2cb556d509259f1fe440bb9cd33c756222cf0a7afe90d15edf0866702431", size = 11306537, upload-time = "2026-04-16T18:46:36.988Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/e4/92/4cfae6441f3967317946f3b788136eecf093729b94d6561f963ed810c82e/ruff-0.15.11-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:030d921a836d7d4a12cf6e8d984a88b66094ccb0e0f17ddd55067c331191bf19", size = 11296813, upload-time = "2026-04-16T18:46:24.182Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/43/26/972784c5dde8313acde8ac71ba8ac65475b85db4a2352a76c9934361f9bc/ruff-0.15.11-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:0e783b599b4577788dbbb66b9addcef87e9a8832f4ce0c19e34bf55543a2f890", size = 10633136, upload-time = "2026-04-16T18:46:39.802Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/5b/53/3985a4f185020c2f367f2e08a103032e12564829742a1b417980ce1514a0/ruff-0.15.11-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:ae90592246625ba4a34349d68ec28d4400d75182b71baa196ddb9f82db025ef5", size = 10424701, upload-time = "2026-04-16T18:46:10.381Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/d3/57/bf0dfb32241b56c83bb663a826133da4bf17f682ba8c096973065f6e6a68/ruff-0.15.11-py3-none-musllinux_1_2_i686.whl", hash = "sha256:1f111d62e3c983ed20e0ca2e800f8d77433a5b1161947df99a5c2a3fb60514f0", size = 10873887, upload-time = "2026-04-16T18:46:29.157Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/02/05/e48076b2a57dc33ee8c7a957296f97c744ca891a8ffb4ffb1aaa3b3f517d/ruff-0.15.11-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:06f483d6646f59eaffba9ae30956370d3a886625f511a3108994000480621d1c", size = 11404316, upload-time = "2026-04-16T18:46:19.462Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/88/27/0195d15fe7a897cbcba0904792c4b7c9fdd958456c3a17d2ea6093716a9a/ruff-0.15.11-py3-none-win32.whl", hash = "sha256:476a2aa56b7da0b73a3ee80b6b2f0e19cce544245479adde7baa65466664d5f3", size = 10655535, upload-time = "2026-04-16T18:46:12.47Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/3a/5e/c927b325bd4c1d3620211a4b96f47864633199feed60fa936025ab27e090/ruff-0.15.11-py3-none-win_amd64.whl", hash = "sha256:8b6756d88d7e234fb0c98c91511aae3cd519d5e3ed271cae31b20f39cb2a12a3", size = 11779692, upload-time = "2026-04-16T18:46:17.268Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/63/b6/aeadee5443e49baa2facd51131159fd6301cc4ccfc1541e4df7b021c37dd/ruff-0.15.11-py3-none-win_arm64.whl", hash = "sha256:063fed18cc1bbe0ee7393957284a6fe8b588c6a406a285af3ee3f46da2391ee4", size = 11032614, upload-time = "2026-04-16T18:46:34.487Z" },
|
||||
]
|
||||
Reference in New Issue
Block a user