mirror of
https://github.com/milvus-io/milvus.git
synced 2026-07-21 02:05:41 +00:00
test: expand CDC replication test coverage across features and chaos scenarios (#49043)
## Summary Extends the CDC (replication) e2e test suite with comprehensive cross-feature coverage and switchover/failover scenarios on top of #45624. ## What's Added ### New test files (\`tests/python_client/cdc/testcases/\`) - \`test_fts_and_text.py\` — BM25 search, text_match, phrase_match, hybrid FTS+dense, FTS after switchover (5 tests × 2 analyzers) - \`test_schema_features.py\` — dynamic schema, nullable, defaults, partition key, clustering key, combined features (7 tests) - \`test_collection_properties.py\` — TTL, mmap, autocompaction, multi-property, drop property (5 tests) - \`test_resource_group.py\` — create, drop, update, transfer replica (4 tests) - \`test_multi_database.py\` — cross-DB collections, drop DB with collections, cross-DB operations (3 tests) - \`test_search_verification.py\` — 6 tests × 7 vector types (search result consistency, query data sampling, hybrid search, iterators, filtered search) - \`test_switchover.py\` — basic, during-writes, all-types, loaded, indexed, rapid stress, failover (7 tests) ### Shared utilities - \`base.py\`: +7 schema factories, +4 data generators, +4 verification helpers, +2 constants - \`conftest.py\`: +\`switchover_helper\` fixture, +\`--is-check\`/\`--milvus-ns\` options ## Test plan - [ ] Run \`pytest tests/python_client/cdc/testcases/\` against two-cluster CDC deployment - [ ] Verify each test file runs independently via \`pytest testcases/test_<name>.py\` - [ ] Companion PR in zilliztech/test-jobs adds Jenkins pipelines to run these issue: #49042 pr: #45624 /kind test /kind improvement 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Yihao Dai <yihao.dai@zilliz.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.6
parent
2c36c60176
commit
8f09efda8f
@@ -6,12 +6,12 @@ on:
|
|||||||
- master
|
- master
|
||||||
paths:
|
paths:
|
||||||
- 'tests/**/*.py'
|
- 'tests/**/*.py'
|
||||||
- 'tests/pyproject.toml'
|
- 'tests/ruff.toml'
|
||||||
- '.github/workflows/python-lint.yaml'
|
- '.github/workflows/python-lint.yaml'
|
||||||
pull_request:
|
pull_request:
|
||||||
paths:
|
paths:
|
||||||
- 'tests/**/*.py'
|
- 'tests/**/*.py'
|
||||||
- 'tests/pyproject.toml'
|
- 'tests/ruff.toml'
|
||||||
- '.github/workflows/python-lint.yaml'
|
- '.github/workflows/python-lint.yaml'
|
||||||
|
|
||||||
concurrency:
|
concurrency:
|
||||||
@@ -50,7 +50,7 @@ jobs:
|
|||||||
args: "format --check --diff"
|
args: "format --check --diff"
|
||||||
|
|
||||||
- name: Validate ruff configuration (smoke test)
|
- name: Validate ruff configuration (smoke test)
|
||||||
# Always runs so config-only PRs still catch a broken pyproject.toml.
|
# Always runs so config-only PRs still catch a broken ruff.toml.
|
||||||
# --exit-zero ignores existing baseline violations; only fails when
|
# --exit-zero ignores existing baseline violations; only fails when
|
||||||
# ruff cannot load the config at all.
|
# ruff cannot load the config at all.
|
||||||
uses: astral-sh/ruff-action@v4.0.0
|
uses: astral-sh/ruff-action@v4.0.0
|
||||||
|
|||||||
@@ -13,9 +13,9 @@ repos:
|
|||||||
hooks:
|
hooks:
|
||||||
- id: ruff-check
|
- id: ruff-check
|
||||||
name: ruff-check (tests/)
|
name: ruff-check (tests/)
|
||||||
args: [--config=tests/pyproject.toml, --fix]
|
args: [--config=tests/ruff.toml, --fix]
|
||||||
files: ^tests/.*\.py$
|
files: ^tests/.*\.py$
|
||||||
- id: ruff-format
|
- id: ruff-format
|
||||||
name: ruff-format (tests/)
|
name: ruff-format (tests/)
|
||||||
args: [--config=tests/pyproject.toml]
|
args: [--config=tests/ruff.toml]
|
||||||
files: ^tests/.*\.py$
|
files: ^tests/.*\.py$
|
||||||
|
|||||||
+7
-8
@@ -92,18 +92,17 @@ $ ./e2e-k8s.sh
|
|||||||
|
|
||||||
### Python Code Quality (ruff via uv)
|
### Python Code Quality (ruff via uv)
|
||||||
|
|
||||||
Ruff is configured at `tests/pyproject.toml` and covers all Python code under `tests/`
|
Ruff is configured at `tests/ruff.toml` and covers all Python code under `tests/`
|
||||||
(`python_client/`, `restful_client/`, `restful_client_v2/`, `benchmark/`, `scripts/`).
|
(`python_client/`, `restful_client/`, `restful_client_v2/`, `benchmark/`, `scripts/`).
|
||||||
[uv](https://docs.astral.sh/uv/) is only used to host the lint/format toolchain; each
|
Each sub-directory continues to manage its runtime dependencies via its own
|
||||||
sub-directory continues to manage its runtime dependencies via its own `requirements.txt`.
|
`requirements.txt`.
|
||||||
|
|
||||||
```shell
|
```shell
|
||||||
$ cd tests/
|
$ cd tests/
|
||||||
$ uv sync # install ruff into a local .venv
|
$ ruff check . # lint
|
||||||
$ uv run ruff check . # lint
|
$ ruff check . --fix # lint with auto-fix
|
||||||
$ uv run ruff check . --fix # lint with auto-fix
|
$ ruff format . # format in place
|
||||||
$ uv run ruff format . # format in place
|
$ ruff format --check . # format check only (CI-friendly)
|
||||||
$ uv run ruff format --check . # format check only (CI-friendly)
|
|
||||||
```
|
```
|
||||||
|
|
||||||
Rules enabled: `E`, `F`, `W`, `I`, `UP`. Target Python version: `3.10`.
|
Rules enabled: `E`, `F`, `W`, `I`, `UP`. Target Python version: `3.10`.
|
||||||
|
|||||||
+7
-9
@@ -90,20 +90,18 @@ $ ./e2e-k8s.sh
|
|||||||
> $ ./e2e-k8s.sh --help
|
> $ ./e2e-k8s.sh --help
|
||||||
> ```
|
> ```
|
||||||
|
|
||||||
### Python 代码质量 (通过 uv 使用 ruff)
|
### Python 代码质量 (ruff)
|
||||||
|
|
||||||
Ruff 配置位于 `tests/pyproject.toml`,覆盖 `tests/` 下所有 Python 代码
|
Ruff 配置位于 `tests/ruff.toml`,覆盖 `tests/` 下所有 Python 代码
|
||||||
(`python_client/`、`restful_client/`、`restful_client_v2/`、`benchmark/`、`scripts/`)。
|
(`python_client/`、`restful_client/`、`restful_client_v2/`、`benchmark/`、`scripts/`)。
|
||||||
[uv](https://docs.astral.sh/uv/) 仅用于托管 lint/format 工具链,各子目录的运行时依赖
|
各子目录的运行时依赖仍通过各自的 `requirements.txt` 管理。
|
||||||
仍通过各自的 `requirements.txt` 管理。
|
|
||||||
|
|
||||||
```shell
|
```shell
|
||||||
$ cd tests/
|
$ cd tests/
|
||||||
$ uv sync # 将 ruff 安装到本地 .venv
|
$ ruff check . # lint 检查
|
||||||
$ uv run ruff check . # lint 检查
|
$ ruff check . --fix # lint 检查并自动修复
|
||||||
$ uv run ruff check . --fix # lint 检查并自动修复
|
$ ruff format . # 原地格式化
|
||||||
$ uv run ruff format . # 原地格式化
|
$ ruff format --check . # 只检查不修改 (CI 友好)
|
||||||
$ uv run ruff format --check . # 只检查不修改 (CI 友好)
|
|
||||||
```
|
```
|
||||||
|
|
||||||
启用的规则:`E`、`F`、`W`、`I`、`UP`;目标 Python 版本:`3.10`。
|
启用的规则:`E`、`F`、`W`、`I`、`UP`;目标 Python 版本:`3.10`。
|
||||||
|
|||||||
@@ -1,14 +1,32 @@
|
|||||||
import pytest
|
|
||||||
import time
|
|
||||||
import logging
|
import logging
|
||||||
|
import time
|
||||||
|
from concurrent.futures import ThreadPoolExecutor, wait
|
||||||
|
|
||||||
|
import pytest
|
||||||
from pymilvus import MilvusClient
|
from pymilvus import MilvusClient
|
||||||
|
|
||||||
# Configure logging
|
# Configure logging
|
||||||
logging.basicConfig(
|
logging.basicConfig(level=logging.INFO, format="%(asctime)s - %(name)s - %(levelname)s - %(message)s")
|
||||||
level=logging.INFO, format="%(asctime)s - %(name)s - %(levelname)s - %(message)s"
|
|
||||||
)
|
|
||||||
logger = logging.getLogger(__name__)
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
CDC_UPDATE_REPLICATE_TIMEOUT_SECONDS = 600
|
||||||
|
|
||||||
|
|
||||||
|
def apply_replicate_configuration(tasks, timeout=CDC_UPDATE_REPLICATE_TIMEOUT_SECONDS):
|
||||||
|
# Fan out in parallel: the server blocks non-primary clusters in
|
||||||
|
# waitUntilPrimaryChangeOrConfigurationSame until the primary's broadcast
|
||||||
|
# propagates via CDC, so a sequential call where the first client happens
|
||||||
|
# to be a replica deadlocks on the client's RPC timeout.
|
||||||
|
with ThreadPoolExecutor(max_workers=len(tasks)) as executor:
|
||||||
|
futures = [
|
||||||
|
executor.submit(client.update_replicate_configuration, timeout=timeout, **config)
|
||||||
|
for client, config in tasks
|
||||||
|
]
|
||||||
|
wait(futures)
|
||||||
|
for f in futures:
|
||||||
|
f.result()
|
||||||
|
|
||||||
|
|
||||||
def pytest_addoption(parser):
|
def pytest_addoption(parser):
|
||||||
"""Add command line options for pytest."""
|
"""Add command line options for pytest."""
|
||||||
parser.addoption(
|
parser.addoption(
|
||||||
@@ -35,9 +53,7 @@ def pytest_addoption(parser):
|
|||||||
default="root:Milvus",
|
default="root:Milvus",
|
||||||
help="Downstream Milvus token",
|
help="Downstream Milvus token",
|
||||||
)
|
)
|
||||||
parser.addoption(
|
parser.addoption("--sync-timeout", action="store", default="30", help="Sync timeout in seconds")
|
||||||
"--sync-timeout", action="store", default="30", help="Sync timeout in seconds"
|
|
||||||
)
|
|
||||||
parser.addoption(
|
parser.addoption(
|
||||||
"--source-cluster-id",
|
"--source-cluster-id",
|
||||||
action="store",
|
action="store",
|
||||||
@@ -62,6 +78,18 @@ def pytest_addoption(parser):
|
|||||||
default="30m",
|
default="30m",
|
||||||
help="Duration for test operations (e.g., 30m, 1h, 60s)",
|
help="Duration for test operations (e.g., 30m, 1h, 60s)",
|
||||||
)
|
)
|
||||||
|
parser.addoption(
|
||||||
|
"--is-check",
|
||||||
|
action="store",
|
||||||
|
default="true",
|
||||||
|
help="Whether to assert on checker statistics",
|
||||||
|
)
|
||||||
|
parser.addoption(
|
||||||
|
"--milvus-ns",
|
||||||
|
action="store",
|
||||||
|
default="chaos-testing",
|
||||||
|
help="Kubernetes namespace for Milvus deployment",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture(scope="session")
|
@pytest.fixture(scope="session")
|
||||||
@@ -138,6 +166,75 @@ def request_duration(request):
|
|||||||
return request.config.getoption("--request-duration")
|
return request.config.getoption("--request-duration")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="session")
|
||||||
|
def is_check(request):
|
||||||
|
# The root tests/python_client/conftest.py registers --is_check (underscore)
|
||||||
|
# with type=bool, which argparse maps to the same dest (is_check) as our
|
||||||
|
# --is-check (hyphen). The root's bool wins in chaos runs, so accept either.
|
||||||
|
val = request.config.getoption("--is-check")
|
||||||
|
return val if isinstance(val, bool) else str(val).lower() == "true"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="session")
|
||||||
|
def milvus_ns(request):
|
||||||
|
return request.config.getoption("--milvus-ns")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="session")
|
||||||
|
def switchover_helper(request, upstream_client, downstream_client):
|
||||||
|
"""Returns a callable that performs CDC topology switchover."""
|
||||||
|
upstream_uri = request.config.getoption("--upstream-uri")
|
||||||
|
upstream_token = request.config.getoption("--upstream-token")
|
||||||
|
downstream_uri = request.config.getoption("--downstream-uri")
|
||||||
|
downstream_token = request.config.getoption("--downstream-token")
|
||||||
|
pchannel_num = int(request.config.getoption("--pchannel-num"))
|
||||||
|
original_source = request.config.getoption("--source-cluster-id")
|
||||||
|
original_target = request.config.getoption("--target-cluster-id")
|
||||||
|
|
||||||
|
# Map cluster IDs to their URIs/tokens
|
||||||
|
cluster_map = {
|
||||||
|
original_source: {"uri": upstream_uri, "token": upstream_token},
|
||||||
|
original_target: {"uri": downstream_uri, "token": downstream_token},
|
||||||
|
}
|
||||||
|
|
||||||
|
def do_switchover(new_source_id, new_target_id):
|
||||||
|
logger.info(f"Performing switchover: {new_source_id} -> {new_target_id}")
|
||||||
|
config = {
|
||||||
|
"clusters": [
|
||||||
|
{
|
||||||
|
"cluster_id": new_source_id,
|
||||||
|
"connection_param": cluster_map[new_source_id],
|
||||||
|
"pchannels": [f"{new_source_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cluster_id": new_target_id,
|
||||||
|
"connection_param": cluster_map[new_target_id],
|
||||||
|
"pchannels": [f"{new_target_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
"cross_cluster_topology": [{"source_cluster_id": new_source_id, "target_cluster_id": new_target_id}],
|
||||||
|
}
|
||||||
|
# Dedicated short-lived clients so switchover RPCs don't share a
|
||||||
|
# gRPC channel with concurrent DML on the session-scoped clients.
|
||||||
|
# pymilvus's connection manager closes a channel on UNAVAILABLE /
|
||||||
|
# STREAMING_CODE_REPLICATE_VIOLATION to trigger recovery; if a DML
|
||||||
|
# on the session client triggers that close while our sibling
|
||||||
|
# update_replicate_configuration RPC is in flight on the same
|
||||||
|
# channel, the latter surfaces "Cannot invoke RPC on closed
|
||||||
|
# channel!". Separate clients = separate channels = no race.
|
||||||
|
up_tmp = MilvusClient(uri=upstream_uri, token=upstream_token)
|
||||||
|
dn_tmp = MilvusClient(uri=downstream_uri, token=downstream_token)
|
||||||
|
try:
|
||||||
|
apply_replicate_configuration([(up_tmp, config), (dn_tmp, config)])
|
||||||
|
finally:
|
||||||
|
up_tmp.close()
|
||||||
|
dn_tmp.close()
|
||||||
|
logger.info("Switchover completed, waiting 10s for stabilization...")
|
||||||
|
time.sleep(10)
|
||||||
|
|
||||||
|
return do_switchover
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture(scope="session", autouse=True)
|
@pytest.fixture(scope="session", autouse=True)
|
||||||
def cdc_topology_setup(request, upstream_client, downstream_client):
|
def cdc_topology_setup(request, upstream_client, downstream_client):
|
||||||
"""Setup CDC topology at the beginning of test session."""
|
"""Setup CDC topology at the beginning of test session."""
|
||||||
@@ -147,9 +244,7 @@ def cdc_topology_setup(request, upstream_client, downstream_client):
|
|||||||
target_cluster_id = request.config.getoption("--target-cluster-id")
|
target_cluster_id = request.config.getoption("--target-cluster-id")
|
||||||
pchannel_num = int(request.config.getoption("--pchannel-num"))
|
pchannel_num = int(request.config.getoption("--pchannel-num"))
|
||||||
|
|
||||||
logger.info(
|
logger.info(f"Setting up CDC topology: {source_cluster_id} -> {target_cluster_id} (channels: {pchannel_num})...")
|
||||||
f"Setting up CDC topology: {source_cluster_id} -> {target_cluster_id} (channels: {pchannel_num})..."
|
|
||||||
)
|
|
||||||
|
|
||||||
# Create CDC replication configuration
|
# Create CDC replication configuration
|
||||||
config = {
|
config = {
|
||||||
@@ -160,10 +255,7 @@ def cdc_topology_setup(request, upstream_client, downstream_client):
|
|||||||
"uri": upstream_uri,
|
"uri": upstream_uri,
|
||||||
"token": request.config.getoption("--upstream-token"),
|
"token": request.config.getoption("--upstream-token"),
|
||||||
},
|
},
|
||||||
"pchannels": [
|
"pchannels": [f"{source_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
|
||||||
f"{source_cluster_id}-rootcoord-dml_{i}"
|
|
||||||
for i in range(pchannel_num)
|
|
||||||
],
|
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cluster_id": target_cluster_id,
|
"cluster_id": target_cluster_id,
|
||||||
@@ -171,10 +263,7 @@ def cdc_topology_setup(request, upstream_client, downstream_client):
|
|||||||
"uri": downstream_uri,
|
"uri": downstream_uri,
|
||||||
"token": request.config.getoption("--downstream-token"),
|
"token": request.config.getoption("--downstream-token"),
|
||||||
},
|
},
|
||||||
"pchannels": [
|
"pchannels": [f"{target_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
|
||||||
f"{target_cluster_id}-rootcoord-dml_{i}"
|
|
||||||
for i in range(pchannel_num)
|
|
||||||
],
|
|
||||||
},
|
},
|
||||||
],
|
],
|
||||||
"cross_cluster_topology": [
|
"cross_cluster_topology": [
|
||||||
@@ -186,9 +275,16 @@ def cdc_topology_setup(request, upstream_client, downstream_client):
|
|||||||
}
|
}
|
||||||
|
|
||||||
try:
|
try:
|
||||||
# Update replication configuration on both clusters
|
# Dedicated clients for the control-plane update_replicate_configuration
|
||||||
upstream_client.update_replicate_configuration(**config)
|
# RPC, mirroring switchover_helper. Keeps the session-scoped clients'
|
||||||
downstream_client.update_replicate_configuration(**config)
|
# channels clean of any recovery side effects from the initial setup.
|
||||||
|
up_tmp = MilvusClient(uri=upstream_uri, token=request.config.getoption("--upstream-token"))
|
||||||
|
dn_tmp = MilvusClient(uri=downstream_uri, token=request.config.getoption("--downstream-token"))
|
||||||
|
try:
|
||||||
|
apply_replicate_configuration([(up_tmp, config), (dn_tmp, config)])
|
||||||
|
finally:
|
||||||
|
up_tmp.close()
|
||||||
|
dn_tmp.close()
|
||||||
logger.info("CDC topology setup completed successfully")
|
logger.info("CDC topology setup completed successfully")
|
||||||
|
|
||||||
# Allow some time for CDC to initialize
|
# Allow some time for CDC to initialize
|
||||||
|
|||||||
@@ -1,34 +1,58 @@
|
|||||||
from pymilvus import MilvusClient
|
|
||||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||||
|
|
||||||
def setup_cdc_topology(upstream_uri, downstream_uri, removed_clusters_uri, upstream_token, downstream_token, removed_clusters_token, source_cluster_id, target_cluster_id, removed_clusters_id, pchannel_num):
|
from pymilvus import MilvusClient
|
||||||
print(f"DEBUG: upstream_uri: {upstream_uri}, downstream_uri: {downstream_uri}, upstream_token: {upstream_token}, downstream_token: {downstream_token}, source_cluster_id: {source_cluster_id}, target_cluster_id: {target_cluster_id}, pchannel_num: {pchannel_num}")
|
|
||||||
|
# Kept in sync with cdc/conftest.py's CDC_UPDATE_REPLICATE_TIMEOUT_SECONDS.
|
||||||
|
# Inlined because this file is executed standalone from tests/python_client/cdc/scripts/,
|
||||||
|
# where the cdc package is not on sys.path.
|
||||||
|
CDC_UPDATE_REPLICATE_TIMEOUT_SECONDS = 600
|
||||||
|
|
||||||
|
|
||||||
|
def setup_cdc_topology(
|
||||||
|
upstream_uri,
|
||||||
|
downstream_uri,
|
||||||
|
removed_clusters_uri,
|
||||||
|
upstream_token,
|
||||||
|
downstream_token,
|
||||||
|
removed_clusters_token,
|
||||||
|
source_cluster_id,
|
||||||
|
target_cluster_id,
|
||||||
|
removed_clusters_id,
|
||||||
|
pchannel_num,
|
||||||
|
):
|
||||||
|
print(
|
||||||
|
f"DEBUG: upstream_uri: {upstream_uri}, downstream_uri: {downstream_uri}, upstream_token: {upstream_token}, downstream_token: {downstream_token}, source_cluster_id: {source_cluster_id}, target_cluster_id: {target_cluster_id}, pchannel_num: {pchannel_num}"
|
||||||
|
)
|
||||||
upstream_client = MilvusClient(uri=upstream_uri, token=upstream_token)
|
upstream_client = MilvusClient(uri=upstream_uri, token=upstream_token)
|
||||||
|
|
||||||
# Parse comma-separated lists
|
# Parse comma-separated lists
|
||||||
if isinstance(downstream_uri, str) and ',' in downstream_uri:
|
if isinstance(downstream_uri, str) and "," in downstream_uri:
|
||||||
downstream_uris = [uri.strip() for uri in downstream_uri.split(',')]
|
downstream_uris = [uri.strip() for uri in downstream_uri.split(",")]
|
||||||
else:
|
else:
|
||||||
downstream_uris = [downstream_uri] if isinstance(downstream_uri, str) else downstream_uri
|
downstream_uris = [downstream_uri] if isinstance(downstream_uri, str) else downstream_uri
|
||||||
|
|
||||||
if isinstance(target_cluster_id, str) and ',' in target_cluster_id:
|
if isinstance(target_cluster_id, str) and "," in target_cluster_id:
|
||||||
target_cluster_ids = [cluster_id.strip() for cluster_id in target_cluster_id.split(',')]
|
target_cluster_ids = [cluster_id.strip() for cluster_id in target_cluster_id.split(",")]
|
||||||
else:
|
else:
|
||||||
target_cluster_ids = [target_cluster_id] if isinstance(target_cluster_id, str) else target_cluster_id
|
target_cluster_ids = [target_cluster_id] if isinstance(target_cluster_id, str) else target_cluster_id
|
||||||
|
|
||||||
if isinstance(removed_clusters_uri, str) and ',' in removed_clusters_uri:
|
if isinstance(removed_clusters_uri, str) and "," in removed_clusters_uri:
|
||||||
removed_clusters_uris = [uri.strip() for uri in removed_clusters_uri.split(',')]
|
removed_clusters_uris = [uri.strip() for uri in removed_clusters_uri.split(",")]
|
||||||
else:
|
else:
|
||||||
removed_clusters_uris = [removed_clusters_uri] if isinstance(removed_clusters_uri, str) else removed_clusters_uri
|
removed_clusters_uris = (
|
||||||
|
[removed_clusters_uri] if isinstance(removed_clusters_uri, str) else removed_clusters_uri
|
||||||
|
)
|
||||||
|
|
||||||
if isinstance(removed_clusters_id, str) and ',' in removed_clusters_id:
|
if isinstance(removed_clusters_id, str) and "," in removed_clusters_id:
|
||||||
removed_clusters_ids = [cluster_id.strip() for cluster_id in removed_clusters_id.split(',')]
|
removed_clusters_ids = [cluster_id.strip() for cluster_id in removed_clusters_id.split(",")]
|
||||||
else:
|
else:
|
||||||
removed_clusters_ids = [removed_clusters_id] if isinstance(removed_clusters_id, str) else removed_clusters_id
|
removed_clusters_ids = [removed_clusters_id] if isinstance(removed_clusters_id, str) else removed_clusters_id
|
||||||
|
|
||||||
# Ensure we have matching numbers of downstream URIs and cluster IDs
|
# Ensure we have matching numbers of downstream URIs and cluster IDs
|
||||||
if len(downstream_uris) != len(target_cluster_ids):
|
if len(downstream_uris) != len(target_cluster_ids):
|
||||||
raise ValueError(f"Number of downstream URIs ({len(downstream_uris)}) must match number of target cluster IDs ({len(target_cluster_ids)})")
|
raise ValueError(
|
||||||
|
f"Number of downstream URIs ({len(downstream_uris)}) must match number of target cluster IDs ({len(target_cluster_ids)})"
|
||||||
|
)
|
||||||
|
|
||||||
# Create downstream clients
|
# Create downstream clients
|
||||||
downstream_clients = []
|
downstream_clients = []
|
||||||
@@ -40,44 +64,34 @@ def setup_cdc_topology(upstream_uri, downstream_uri, removed_clusters_uri, upstr
|
|||||||
clusters = [
|
clusters = [
|
||||||
{
|
{
|
||||||
"cluster_id": source_cluster_id,
|
"cluster_id": source_cluster_id,
|
||||||
"connection_param": {
|
"connection_param": {"uri": upstream_uri, "token": upstream_token},
|
||||||
"uri": upstream_uri,
|
"pchannels": [f"{source_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
|
||||||
"token": upstream_token
|
|
||||||
},
|
|
||||||
"pchannels": [f"{source_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)]
|
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
|
|
||||||
# Add all target clusters
|
# Add all target clusters
|
||||||
for target_id, target_uri in zip(target_cluster_ids, downstream_uris):
|
for target_id, target_uri in zip(target_cluster_ids, downstream_uris):
|
||||||
clusters.append({
|
clusters.append(
|
||||||
"cluster_id": target_id,
|
{
|
||||||
"connection_param": {
|
"cluster_id": target_id,
|
||||||
"uri": target_uri,
|
"connection_param": {"uri": target_uri, "token": downstream_token},
|
||||||
"token": downstream_token
|
"pchannels": [f"{target_id}-rootcoord-dml_{j}" for j in range(pchannel_num)],
|
||||||
},
|
}
|
||||||
"pchannels": [f"{target_id}-rootcoord-dml_{j}" for j in range(pchannel_num)]
|
)
|
||||||
})
|
|
||||||
|
|
||||||
# Build cross-cluster topology
|
# Build cross-cluster topology
|
||||||
cross_cluster_topology = []
|
cross_cluster_topology = []
|
||||||
for target_id in target_cluster_ids:
|
for target_id in target_cluster_ids:
|
||||||
cross_cluster_topology.append({
|
cross_cluster_topology.append({"source_cluster_id": source_cluster_id, "target_cluster_id": target_id})
|
||||||
"source_cluster_id": source_cluster_id,
|
|
||||||
"target_cluster_id": target_id
|
|
||||||
})
|
|
||||||
|
|
||||||
config = {
|
config = {"clusters": clusters, "cross_cluster_topology": cross_cluster_topology}
|
||||||
"clusters": clusters,
|
|
||||||
"cross_cluster_topology": cross_cluster_topology
|
|
||||||
}
|
|
||||||
|
|
||||||
# Update configuration on all clients using multi-threading
|
# Update configuration on all clients using multi-threading
|
||||||
print(f"DEBUG: config: {config}")
|
print(f"DEBUG: config: {config}")
|
||||||
|
|
||||||
def update_client_config(client, config_to_use, client_type=""):
|
def update_client_config(client, config_to_use, client_type=""):
|
||||||
try:
|
try:
|
||||||
client.update_replicate_configuration(**config_to_use)
|
client.update_replicate_configuration(timeout=CDC_UPDATE_REPLICATE_TIMEOUT_SECONDS, **config_to_use)
|
||||||
return f"{client_type} updated successfully"
|
return f"{client_type} updated successfully"
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
print(f"Failed to update {client_type}: {e}")
|
print(f"Failed to update {client_type}: {e}")
|
||||||
@@ -102,14 +116,11 @@ def setup_cdc_topology(upstream_uri, downstream_uri, removed_clusters_uri, upstr
|
|||||||
"clusters": [
|
"clusters": [
|
||||||
{
|
{
|
||||||
"cluster_id": removed_id,
|
"cluster_id": removed_id,
|
||||||
"connection_param": {
|
"connection_param": {"uri": removed_uri, "token": removed_clusters_token},
|
||||||
"uri": removed_uri,
|
"pchannels": [f"{removed_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
|
||||||
"token": removed_clusters_token
|
|
||||||
},
|
|
||||||
"pchannels": [f"{removed_id}-rootcoord-dml_{i}" for i in range(pchannel_num)]
|
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"cross_cluster_topology": []
|
"cross_cluster_topology": [],
|
||||||
}
|
}
|
||||||
print(f"DEBUG: Removing cluster {removed_id} with empty config: {empty_config}")
|
print(f"DEBUG: Removing cluster {removed_id} with empty config: {empty_config}")
|
||||||
update_tasks.append((removed_client, empty_config, f"Removed cluster {removed_id}"))
|
update_tasks.append((removed_client, empty_config, f"Removed cluster {removed_id}"))
|
||||||
@@ -117,8 +128,10 @@ def setup_cdc_topology(upstream_uri, downstream_uri, removed_clusters_uri, upstr
|
|||||||
# Use single ThreadPoolExecutor to update all clients concurrently
|
# Use single ThreadPoolExecutor to update all clients concurrently
|
||||||
with ThreadPoolExecutor(max_workers=len(update_tasks)) as executor:
|
with ThreadPoolExecutor(max_workers=len(update_tasks)) as executor:
|
||||||
# Submit all update tasks
|
# Submit all update tasks
|
||||||
futures = [executor.submit(update_client_config, client, config_to_use, client_type)
|
futures = [
|
||||||
for client, config_to_use, client_type in update_tasks]
|
executor.submit(update_client_config, client, config_to_use, client_type)
|
||||||
|
for client, config_to_use, client_type in update_tasks
|
||||||
|
]
|
||||||
|
|
||||||
# Wait for all tasks to complete
|
# Wait for all tasks to complete
|
||||||
for future in as_completed(futures):
|
for future in as_completed(futures):
|
||||||
@@ -133,26 +146,28 @@ def setup_cdc_topology(upstream_uri, downstream_uri, removed_clusters_uri, upstr
|
|||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
import argparse
|
import argparse
|
||||||
|
|
||||||
parser = argparse.ArgumentParser(description='connection info')
|
parser = argparse.ArgumentParser(description="connection info")
|
||||||
parser.add_argument('--upstream_uri', type=str, default='10.100.36.179', help='milvus host')
|
parser.add_argument("--upstream_uri", type=str, default="10.100.36.179", help="milvus host")
|
||||||
parser.add_argument('--downstream_uri', type=str, default='10.100.36.178', help='milvus host')
|
parser.add_argument("--downstream_uri", type=str, default="10.100.36.178", help="milvus host")
|
||||||
parser.add_argument('--removed_clusters_uri', type=str, default='', help='milvus host')
|
parser.add_argument("--removed_clusters_uri", type=str, default="", help="milvus host")
|
||||||
parser.add_argument('--upstream_token', type=str, default='root:Milvus', help='milvus token')
|
parser.add_argument("--upstream_token", type=str, default="root:Milvus", help="milvus token")
|
||||||
parser.add_argument('--downstream_token', type=str, default='root:Milvus', help='milvus token')
|
parser.add_argument("--downstream_token", type=str, default="root:Milvus", help="milvus token")
|
||||||
parser.add_argument('--removed_clusters_token', type=str, default='root:Milvus', help='milvus token')
|
parser.add_argument("--removed_clusters_token", type=str, default="root:Milvus", help="milvus token")
|
||||||
parser.add_argument('--source_cluster_id', type=str, default='cdc-test-source', help='source cluster id')
|
parser.add_argument("--source_cluster_id", type=str, default="cdc-test-source", help="source cluster id")
|
||||||
parser.add_argument('--target_cluster_id', type=str, default='cdc-test-target', help='target cluster id')
|
parser.add_argument("--target_cluster_id", type=str, default="cdc-test-target", help="target cluster id")
|
||||||
parser.add_argument('--removed_clusters_id', type=str, default='', help='removed clusters id')
|
parser.add_argument("--removed_clusters_id", type=str, default="", help="removed clusters id")
|
||||||
|
|
||||||
parser.add_argument('--pchannel_num', type=int, default=16, help='pchannel num')
|
parser.add_argument("--pchannel_num", type=int, default=16, help="pchannel num")
|
||||||
args = parser.parse_args()
|
args = parser.parse_args()
|
||||||
setup_cdc_topology(args.upstream_uri,
|
setup_cdc_topology(
|
||||||
args.downstream_uri,
|
args.upstream_uri,
|
||||||
args.removed_clusters_uri,
|
args.downstream_uri,
|
||||||
args.upstream_token,
|
args.removed_clusters_uri,
|
||||||
args.downstream_token,
|
args.upstream_token,
|
||||||
args.removed_clusters_token,
|
args.downstream_token,
|
||||||
args.source_cluster_id,
|
args.removed_clusters_token,
|
||||||
args.target_cluster_id,
|
args.source_cluster_id,
|
||||||
args.removed_clusters_id,
|
args.target_cluster_id,
|
||||||
args.pchannel_num)
|
args.removed_clusters_id,
|
||||||
|
args.pchannel_num,
|
||||||
|
)
|
||||||
|
|||||||
@@ -1,38 +1,71 @@
|
|||||||
import time
|
|
||||||
import pytest
|
|
||||||
import json
|
import json
|
||||||
|
import time
|
||||||
from time import sleep
|
from time import sleep
|
||||||
from pymilvus import connections
|
|
||||||
from chaos.checker import (InsertChecker,
|
import pytest
|
||||||
UpsertChecker,
|
|
||||||
FlushChecker,
|
|
||||||
SearchChecker,
|
|
||||||
FullTextSearchChecker,
|
|
||||||
HybridSearchChecker,
|
|
||||||
QueryChecker,
|
|
||||||
TextMatchChecker,
|
|
||||||
PhraseMatchChecker,
|
|
||||||
JsonQueryChecker,
|
|
||||||
GeoQueryChecker,
|
|
||||||
DeleteChecker,
|
|
||||||
AddFieldChecker,
|
|
||||||
Op,
|
|
||||||
ResultAnalyzer
|
|
||||||
)
|
|
||||||
from utils.util_k8s import wait_pods_ready, get_milvus_instance_name
|
|
||||||
from utils.util_log import test_log as log
|
|
||||||
from chaos import chaos_commons as cc
|
from chaos import chaos_commons as cc
|
||||||
from common import common_func as cf
|
|
||||||
from common.milvus_sys import MilvusSys
|
|
||||||
from chaos.chaos_commons import assert_statistic
|
|
||||||
from common.common_type import CaseLabel
|
|
||||||
from chaos import constants
|
from chaos import constants
|
||||||
|
from chaos.chaos_commons import assert_statistic
|
||||||
|
from chaos.checker import (
|
||||||
|
AddFieldChecker,
|
||||||
|
DeleteChecker,
|
||||||
|
FlushChecker,
|
||||||
|
FullTextSearchChecker,
|
||||||
|
GeoQueryChecker,
|
||||||
|
HybridSearchChecker,
|
||||||
|
InsertChecker,
|
||||||
|
JsonQueryChecker,
|
||||||
|
Op,
|
||||||
|
PhraseMatchChecker,
|
||||||
|
QueryChecker,
|
||||||
|
ResultAnalyzer,
|
||||||
|
SearchChecker,
|
||||||
|
TextMatchChecker,
|
||||||
|
UpsertChecker,
|
||||||
|
)
|
||||||
|
from common import common_func as cf
|
||||||
|
from common.common_type import CaseLabel
|
||||||
|
from common.milvus_sys import MilvusSys
|
||||||
from delayed_assert import assert_expectations
|
from delayed_assert import assert_expectations
|
||||||
|
from pymilvus import DataType, FunctionType, connections
|
||||||
|
from utils.util_k8s import get_milvus_instance_name, wait_pods_ready
|
||||||
|
from utils.util_log import test_log as log
|
||||||
|
|
||||||
|
_VECTOR_DTYPES = {
|
||||||
|
DataType.FLOAT_VECTOR,
|
||||||
|
DataType.FLOAT16_VECTOR,
|
||||||
|
DataType.BFLOAT16_VECTOR,
|
||||||
|
DataType.BINARY_VECTOR,
|
||||||
|
DataType.SPARSE_FLOAT_VECTOR,
|
||||||
|
DataType.INT8_VECTOR,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _build_checker_schema(dim=8):
|
||||||
|
"""Build the shared all-datatype schema, stripped for the 2.6-latest image.
|
||||||
|
|
||||||
|
The chaos-test image used by milvus_cdc_chaos_test/verify_test rejects
|
||||||
|
two things the shared gen_all_datatype_collection_schema includes by
|
||||||
|
default:
|
||||||
|
- FunctionType.MINHASH (error: "check function params with unknown
|
||||||
|
function type")
|
||||||
|
- nullable=True on FLOAT_VECTOR (error: "vector type not support null")
|
||||||
|
|
||||||
|
Drop the MinHash function and its output field, and force nullable=False
|
||||||
|
on every vector field so the server accepts the schema.
|
||||||
|
"""
|
||||||
|
schema = cf.gen_all_datatype_collection_schema(dim=dim)
|
||||||
|
schema.functions[:] = [f for f in schema.functions if f.type != FunctionType.MINHASH]
|
||||||
|
schema.fields[:] = [f for f in schema.fields if f.name != "minhash_emb"]
|
||||||
|
for f in schema.fields:
|
||||||
|
if f.dtype in _VECTOR_DTYPES:
|
||||||
|
f.nullable = False
|
||||||
|
return schema
|
||||||
|
|
||||||
|
|
||||||
def get_all_collections():
|
def get_all_collections():
|
||||||
try:
|
try:
|
||||||
with open("/tmp/ci_logs/chaos_test_all_collections.json", "r") as f:
|
with open("/tmp/ci_logs/chaos_test_all_collections.json") as f:
|
||||||
data = json.load(f)
|
data = json.load(f)
|
||||||
all_collections = data["all"]
|
all_collections = data["all"]
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
@@ -48,40 +81,40 @@ class TestBase:
|
|||||||
expect_compact = constants.SUCC
|
expect_compact = constants.SUCC
|
||||||
expect_search = constants.SUCC
|
expect_search = constants.SUCC
|
||||||
expect_query = constants.SUCC
|
expect_query = constants.SUCC
|
||||||
host = '127.0.0.1'
|
host = "127.0.0.1"
|
||||||
port = 19530
|
port = 19530
|
||||||
_chaos_config = None
|
_chaos_config = None
|
||||||
health_checkers = {}
|
health_checkers = {}
|
||||||
|
|
||||||
|
|
||||||
class TestOperations(TestBase):
|
class TestOperations(TestBase):
|
||||||
|
|
||||||
@pytest.fixture(scope="function", autouse=True)
|
@pytest.fixture(scope="function", autouse=True)
|
||||||
def connection(self, upstream_uri, upstream_token, milvus_ns):
|
def connection(self, upstream_uri, upstream_token, milvus_ns):
|
||||||
connections.connect('default', uri=upstream_uri, token=upstream_token)
|
connections.connect("default", uri=upstream_uri, token=upstream_token)
|
||||||
if connections.has_connection("default") is False:
|
if connections.has_connection("default") is False:
|
||||||
raise Exception("no connections")
|
raise Exception("no connections")
|
||||||
log.info("connect to milvus successfully")
|
log.info("connect to milvus successfully")
|
||||||
self.milvus_sys = MilvusSys(alias='default')
|
self.milvus_sys = MilvusSys(alias="default")
|
||||||
self.milvus_ns = milvus_ns
|
self.milvus_ns = milvus_ns
|
||||||
self.release_name = get_milvus_instance_name(self.milvus_ns, milvus_sys=self.milvus_sys)
|
self.release_name = get_milvus_instance_name(self.milvus_ns, milvus_sys=self.milvus_sys)
|
||||||
|
|
||||||
def init_health_checkers(self, collection_name=None):
|
def init_health_checkers(self, collection_name=None):
|
||||||
c_name = collection_name
|
c_name = collection_name
|
||||||
|
schema = _build_checker_schema()
|
||||||
checkers = {
|
checkers = {
|
||||||
Op.insert: InsertChecker(collection_name=c_name),
|
Op.insert: InsertChecker(collection_name=c_name, schema=schema),
|
||||||
Op.upsert: UpsertChecker(collection_name=c_name),
|
Op.upsert: UpsertChecker(collection_name=c_name, schema=schema),
|
||||||
Op.flush: FlushChecker(collection_name=c_name),
|
Op.flush: FlushChecker(collection_name=c_name, schema=schema),
|
||||||
Op.search: SearchChecker(collection_name=c_name),
|
Op.search: SearchChecker(collection_name=c_name, schema=schema),
|
||||||
Op.full_text_search: FullTextSearchChecker(collection_name=c_name),
|
Op.full_text_search: FullTextSearchChecker(collection_name=c_name, schema=schema),
|
||||||
Op.hybrid_search: HybridSearchChecker(collection_name=c_name),
|
Op.hybrid_search: HybridSearchChecker(collection_name=c_name, schema=schema),
|
||||||
Op.query: QueryChecker(collection_name=c_name),
|
Op.query: QueryChecker(collection_name=c_name, schema=schema),
|
||||||
Op.text_match: TextMatchChecker(collection_name=c_name),
|
Op.text_match: TextMatchChecker(collection_name=c_name, schema=schema),
|
||||||
Op.phrase_match: PhraseMatchChecker(collection_name=c_name),
|
Op.phrase_match: PhraseMatchChecker(collection_name=c_name, schema=schema),
|
||||||
Op.json_query: JsonQueryChecker(collection_name=c_name),
|
Op.json_query: JsonQueryChecker(collection_name=c_name, schema=schema),
|
||||||
Op.geo_query: GeoQueryChecker(collection_name=c_name),
|
Op.geo_query: GeoQueryChecker(collection_name=c_name, schema=schema),
|
||||||
Op.delete: DeleteChecker(collection_name=c_name),
|
Op.delete: DeleteChecker(collection_name=c_name, schema=schema),
|
||||||
Op.add_field: AddFieldChecker(collection_name=c_name),
|
Op.add_field: AddFieldChecker(collection_name=c_name, schema=schema),
|
||||||
}
|
}
|
||||||
log.info(f"init_health_checkers: {checkers}")
|
log.info(f"init_health_checkers: {checkers}")
|
||||||
self.health_checkers = checkers
|
self.health_checkers = checkers
|
||||||
@@ -96,7 +129,7 @@ class TestOperations(TestBase):
|
|||||||
def test_operations(self, request_duration, is_check, collection_name):
|
def test_operations(self, request_duration, is_check, collection_name):
|
||||||
# start the monitor threads to check the milvus ops
|
# start the monitor threads to check the milvus ops
|
||||||
log.info("*********************Test Start**********************")
|
log.info("*********************Test Start**********************")
|
||||||
log.info(connections.get_connection_addr('default'))
|
log.info(connections.get_connection_addr("default"))
|
||||||
# event_records = EventRecords()
|
# event_records = EventRecords()
|
||||||
c_name = collection_name if collection_name else cf.gen_unique_str("Checker_")
|
c_name = collection_name if collection_name else cf.gen_unique_str("Checker_")
|
||||||
# event_records.insert("init_health_checkers", "start")
|
# event_records.insert("init_health_checkers", "start")
|
||||||
@@ -109,7 +142,7 @@ class TestOperations(TestBase):
|
|||||||
request_duration = request_duration[:-1]
|
request_duration = request_duration[:-1]
|
||||||
request_duration = eval(request_duration)
|
request_duration = eval(request_duration)
|
||||||
for i in range(10):
|
for i in range(10):
|
||||||
sleep(request_duration//10)
|
sleep(request_duration // 10)
|
||||||
for k, v in self.health_checkers.items():
|
for k, v in self.health_checkers.items():
|
||||||
v.check_result()
|
v.check_result()
|
||||||
# log.info(v.check_result())
|
# log.info(v.check_result())
|
||||||
|
|||||||
@@ -1,41 +1,73 @@
|
|||||||
import time
|
import time
|
||||||
|
|
||||||
import pytest
|
|
||||||
from time import sleep
|
from time import sleep
|
||||||
|
|
||||||
import pymilvus
|
import pymilvus
|
||||||
from pymilvus import connections, utility
|
import pytest
|
||||||
from chaos.checker import (CollectionCreateChecker,
|
|
||||||
InsertChecker,
|
|
||||||
BulkInsertChecker,
|
|
||||||
UpsertChecker,
|
|
||||||
PartialUpdateChecker,
|
|
||||||
FlushChecker,
|
|
||||||
SearchChecker,
|
|
||||||
FullTextSearchChecker,
|
|
||||||
HybridSearchChecker,
|
|
||||||
QueryChecker,
|
|
||||||
TextMatchChecker,
|
|
||||||
PhraseMatchChecker,
|
|
||||||
JsonQueryChecker,
|
|
||||||
GeoQueryChecker,
|
|
||||||
IndexCreateChecker,
|
|
||||||
DeleteChecker,
|
|
||||||
CollectionDropChecker,
|
|
||||||
AlterCollectionChecker,
|
|
||||||
AddFieldChecker,
|
|
||||||
CollectionRenameChecker,
|
|
||||||
Op,
|
|
||||||
EventRecords,
|
|
||||||
ResultAnalyzer
|
|
||||||
)
|
|
||||||
from utils.util_log import test_log as log
|
|
||||||
from utils.util_k8s import wait_pods_ready, get_milvus_instance_name
|
|
||||||
from chaos import chaos_commons as cc
|
from chaos import chaos_commons as cc
|
||||||
|
from chaos import constants
|
||||||
|
from chaos.chaos_commons import assert_statistic
|
||||||
|
from chaos.checker import (
|
||||||
|
AddFieldChecker,
|
||||||
|
AlterCollectionChecker,
|
||||||
|
CollectionCreateChecker,
|
||||||
|
CollectionDropChecker,
|
||||||
|
CollectionRenameChecker,
|
||||||
|
DeleteChecker,
|
||||||
|
EventRecords,
|
||||||
|
FlushChecker,
|
||||||
|
FullTextSearchChecker,
|
||||||
|
GeoQueryChecker,
|
||||||
|
HybridSearchChecker,
|
||||||
|
IndexCreateChecker,
|
||||||
|
InsertChecker,
|
||||||
|
JsonQueryChecker,
|
||||||
|
Op,
|
||||||
|
PartialUpdateChecker,
|
||||||
|
PhraseMatchChecker,
|
||||||
|
QueryChecker,
|
||||||
|
ResultAnalyzer,
|
||||||
|
SearchChecker,
|
||||||
|
TextMatchChecker,
|
||||||
|
UpsertChecker,
|
||||||
|
)
|
||||||
|
from common import common_func as cf
|
||||||
from common.common_type import CaseLabel
|
from common.common_type import CaseLabel
|
||||||
from common.milvus_sys import MilvusSys
|
from common.milvus_sys import MilvusSys
|
||||||
from chaos.chaos_commons import assert_statistic
|
|
||||||
from chaos import constants
|
|
||||||
from delayed_assert import assert_expectations
|
from delayed_assert import assert_expectations
|
||||||
|
from pymilvus import DataType, FunctionType, connections, utility
|
||||||
|
from utils.util_k8s import get_milvus_instance_name, wait_pods_ready
|
||||||
|
from utils.util_log import test_log as log
|
||||||
|
|
||||||
|
_VECTOR_DTYPES = {
|
||||||
|
DataType.FLOAT_VECTOR,
|
||||||
|
DataType.FLOAT16_VECTOR,
|
||||||
|
DataType.BFLOAT16_VECTOR,
|
||||||
|
DataType.BINARY_VECTOR,
|
||||||
|
DataType.SPARSE_FLOAT_VECTOR,
|
||||||
|
DataType.INT8_VECTOR,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _build_checker_schema(dim=8):
|
||||||
|
"""Build the shared all-datatype schema, stripped for the 2.6-latest image.
|
||||||
|
|
||||||
|
The chaos-test image used by milvus_cdc_chaos_test/verify_test rejects
|
||||||
|
two things the shared gen_all_datatype_collection_schema includes by
|
||||||
|
default:
|
||||||
|
- FunctionType.MINHASH (error: "check function params with unknown
|
||||||
|
function type")
|
||||||
|
- nullable=True on FLOAT_VECTOR (error: "vector type not support null")
|
||||||
|
|
||||||
|
Drop the MinHash function and its output field, and force nullable=False
|
||||||
|
on every vector field so the server accepts the schema.
|
||||||
|
"""
|
||||||
|
schema = cf.gen_all_datatype_collection_schema(dim=dim)
|
||||||
|
schema.functions[:] = [f for f in schema.functions if f.type != FunctionType.MINHASH]
|
||||||
|
schema.fields[:] = [f for f in schema.fields if f.name != "minhash_emb"]
|
||||||
|
for f in schema.fields:
|
||||||
|
if f.dtype in _VECTOR_DTYPES:
|
||||||
|
f.nullable = False
|
||||||
|
return schema
|
||||||
|
|
||||||
|
|
||||||
class TestBase:
|
class TestBase:
|
||||||
@@ -45,17 +77,16 @@ class TestBase:
|
|||||||
expect_index = constants.SUCC
|
expect_index = constants.SUCC
|
||||||
expect_search = constants.SUCC
|
expect_search = constants.SUCC
|
||||||
expect_query = constants.SUCC
|
expect_query = constants.SUCC
|
||||||
host = '127.0.0.1'
|
host = "127.0.0.1"
|
||||||
port = 19530
|
port = 19530
|
||||||
_chaos_config = None
|
_chaos_config = None
|
||||||
health_checkers = {}
|
health_checkers = {}
|
||||||
|
|
||||||
|
|
||||||
class TestOperations(TestBase):
|
class TestOperations(TestBase):
|
||||||
|
|
||||||
@pytest.fixture(scope="function", autouse=True)
|
@pytest.fixture(scope="function", autouse=True)
|
||||||
def connection(self, upstream_uri, upstream_token, milvus_ns):
|
def connection(self, upstream_uri, upstream_token, milvus_ns):
|
||||||
connections.connect('default', uri=upstream_uri, token=upstream_token)
|
connections.connect("default", uri=upstream_uri, token=upstream_token)
|
||||||
if connections.has_connection("default") is False:
|
if connections.has_connection("default") is False:
|
||||||
raise Exception("no connections")
|
raise Exception("no connections")
|
||||||
log.info("connect to milvus successfully")
|
log.info("connect to milvus successfully")
|
||||||
@@ -63,32 +94,33 @@ class TestOperations(TestBase):
|
|||||||
server_version = utility.get_server_version()
|
server_version = utility.get_server_version()
|
||||||
log.info(f"server version: {server_version}")
|
log.info(f"server version: {server_version}")
|
||||||
log.info(f"pymilvus version: {pymilvus_version}")
|
log.info(f"pymilvus version: {pymilvus_version}")
|
||||||
self.milvus_sys = MilvusSys(alias='default')
|
self.milvus_sys = MilvusSys(alias="default")
|
||||||
self.milvus_ns = milvus_ns
|
self.milvus_ns = milvus_ns
|
||||||
self.release_name = get_milvus_instance_name(self.milvus_ns, milvus_sys=self.milvus_sys)
|
self.release_name = get_milvus_instance_name(self.milvus_ns, milvus_sys=self.milvus_sys)
|
||||||
|
|
||||||
def init_health_checkers(self, collection_name=None):
|
def init_health_checkers(self, collection_name=None):
|
||||||
c_name = collection_name
|
c_name = collection_name
|
||||||
|
schema = _build_checker_schema()
|
||||||
checkers = {
|
checkers = {
|
||||||
Op.create: CollectionCreateChecker(collection_name=c_name),
|
Op.create: CollectionCreateChecker(collection_name=c_name, schema=schema),
|
||||||
Op.insert: InsertChecker(collection_name=c_name),
|
Op.insert: InsertChecker(collection_name=c_name, schema=schema),
|
||||||
Op.upsert: UpsertChecker(collection_name=c_name),
|
Op.upsert: UpsertChecker(collection_name=c_name, schema=schema),
|
||||||
Op.partial_update: PartialUpdateChecker(collection_name=c_name),
|
Op.partial_update: PartialUpdateChecker(collection_name=c_name, schema=schema),
|
||||||
Op.flush: FlushChecker(collection_name=c_name),
|
Op.flush: FlushChecker(collection_name=c_name, schema=schema),
|
||||||
Op.index: IndexCreateChecker(collection_name=c_name),
|
Op.index: IndexCreateChecker(collection_name=c_name, schema=schema),
|
||||||
Op.search: SearchChecker(collection_name=c_name),
|
Op.search: SearchChecker(collection_name=c_name, schema=schema),
|
||||||
Op.full_text_search: FullTextSearchChecker(collection_name=c_name),
|
Op.full_text_search: FullTextSearchChecker(collection_name=c_name, schema=schema),
|
||||||
Op.hybrid_search: HybridSearchChecker(collection_name=c_name),
|
Op.hybrid_search: HybridSearchChecker(collection_name=c_name, schema=schema),
|
||||||
Op.query: QueryChecker(collection_name=c_name),
|
Op.query: QueryChecker(collection_name=c_name, schema=schema),
|
||||||
Op.text_match: TextMatchChecker(collection_name=c_name),
|
Op.text_match: TextMatchChecker(collection_name=c_name, schema=schema),
|
||||||
Op.phrase_match: PhraseMatchChecker(collection_name=c_name),
|
Op.phrase_match: PhraseMatchChecker(collection_name=c_name, schema=schema),
|
||||||
Op.json_query: JsonQueryChecker(collection_name=c_name),
|
Op.json_query: JsonQueryChecker(collection_name=c_name, schema=schema),
|
||||||
Op.geo_query: GeoQueryChecker(collection_name=c_name),
|
Op.geo_query: GeoQueryChecker(collection_name=c_name, schema=schema),
|
||||||
Op.delete: DeleteChecker(collection_name=c_name),
|
Op.delete: DeleteChecker(collection_name=c_name, schema=schema),
|
||||||
Op.drop: CollectionDropChecker(collection_name=c_name),
|
Op.drop: CollectionDropChecker(collection_name=c_name, schema=schema),
|
||||||
Op.alter_collection: AlterCollectionChecker(collection_name=c_name),
|
Op.alter_collection: AlterCollectionChecker(collection_name=c_name, schema=schema),
|
||||||
Op.add_field: AddFieldChecker(collection_name=c_name),
|
Op.add_field: AddFieldChecker(collection_name=c_name, schema=schema),
|
||||||
Op.rename_collection: CollectionRenameChecker(collection_name=c_name)
|
Op.rename_collection: CollectionRenameChecker(collection_name=c_name, schema=schema),
|
||||||
}
|
}
|
||||||
self.health_checkers = checkers
|
self.health_checkers = checkers
|
||||||
|
|
||||||
@@ -96,7 +128,7 @@ class TestOperations(TestBase):
|
|||||||
def test_operations(self, request_duration, is_check):
|
def test_operations(self, request_duration, is_check):
|
||||||
# start the monitor threads to check the milvus ops
|
# start the monitor threads to check the milvus ops
|
||||||
log.info("*********************Test Start**********************")
|
log.info("*********************Test Start**********************")
|
||||||
log.info(connections.get_connection_addr('default'))
|
log.info(connections.get_connection_addr("default"))
|
||||||
event_records = EventRecords()
|
event_records = EventRecords()
|
||||||
c_name = None
|
c_name = None
|
||||||
event_records.insert("init_health_checkers", "start")
|
event_records.insert("init_health_checkers", "start")
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,276 @@
|
|||||||
|
"""
|
||||||
|
CDC sync tests for collection property operations.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import time
|
||||||
|
|
||||||
|
from .base import TestCDCSyncBase, logger
|
||||||
|
|
||||||
|
|
||||||
|
class TestCDCSyncCollectionProperties(TestCDCSyncBase):
|
||||||
|
"""Test CDC sync for collection property (alter/drop) operations."""
|
||||||
|
|
||||||
|
def setup_method(self):
|
||||||
|
"""Setup for each test method."""
|
||||||
|
self.resources_to_cleanup = []
|
||||||
|
|
||||||
|
def teardown_method(self):
|
||||||
|
"""Cleanup after each test method - only cleanup upstream, downstream will sync."""
|
||||||
|
upstream_client = getattr(self, "_upstream_client", None)
|
||||||
|
|
||||||
|
if upstream_client:
|
||||||
|
for resource_type, resource_name in self.resources_to_cleanup:
|
||||||
|
if resource_type == "collection":
|
||||||
|
self.cleanup_collection(upstream_client, resource_name)
|
||||||
|
|
||||||
|
time.sleep(1) # Allow cleanup to sync to downstream
|
||||||
|
|
||||||
|
def _create_basic_collection(self, client, c_name):
|
||||||
|
"""Create a default schema collection, insert 100 rows, and create HNSW index."""
|
||||||
|
schema = self.create_default_schema(client)
|
||||||
|
client.create_collection(
|
||||||
|
collection_name=c_name,
|
||||||
|
schema=schema,
|
||||||
|
consistency_level="Strong",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Insert 100 rows
|
||||||
|
test_data = self.generate_test_data(100)
|
||||||
|
client.insert(c_name, test_data)
|
||||||
|
|
||||||
|
# Create HNSW index on vector field
|
||||||
|
index_params = client.prepare_index_params()
|
||||||
|
index_params.add_index(
|
||||||
|
field_name="vector",
|
||||||
|
index_type="HNSW",
|
||||||
|
metric_type="L2",
|
||||||
|
params={"M": 8, "efConstruction": 64},
|
||||||
|
)
|
||||||
|
client.create_index(c_name, index_params)
|
||||||
|
|
||||||
|
def test_ttl_sync(self, upstream_client, downstream_client, sync_timeout):
|
||||||
|
"""Test ALTER_COLLECTION_PROPERTIES (TTL) sync."""
|
||||||
|
# Store upstream client for teardown
|
||||||
|
self._upstream_client = upstream_client
|
||||||
|
|
||||||
|
c_name = self.gen_unique_name("test_col_ttl")
|
||||||
|
self.resources_to_cleanup.append(("collection", c_name))
|
||||||
|
|
||||||
|
# Initial cleanup
|
||||||
|
self.cleanup_collection(upstream_client, c_name)
|
||||||
|
|
||||||
|
# Create collection
|
||||||
|
self._create_basic_collection(upstream_client, c_name)
|
||||||
|
|
||||||
|
# Wait for collection to sync downstream
|
||||||
|
def check_create():
|
||||||
|
return downstream_client.has_collection(c_name)
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {c_name}")
|
||||||
|
|
||||||
|
# Alter collection TTL property
|
||||||
|
upstream_client.alter_collection_properties(
|
||||||
|
collection_name=c_name,
|
||||||
|
properties={"collection.ttl.seconds": "3600"},
|
||||||
|
)
|
||||||
|
|
||||||
|
# Wait for property to sync downstream
|
||||||
|
def check_ttl():
|
||||||
|
try:
|
||||||
|
desc = downstream_client.describe_collection(c_name)
|
||||||
|
props = desc.get("properties", {})
|
||||||
|
logger.info(f"Downstream collection properties: {props}")
|
||||||
|
return str(props.get("collection.ttl.seconds", "")) == "3600"
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Check TTL sync failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_ttl, sync_timeout, f"TTL property sync for {c_name}")
|
||||||
|
|
||||||
|
def test_mmap_sync(self, upstream_client, downstream_client, sync_timeout):
|
||||||
|
"""Test ALTER_COLLECTION_PROPERTIES (mmap.enabled) sync."""
|
||||||
|
# Store upstream client for teardown
|
||||||
|
self._upstream_client = upstream_client
|
||||||
|
|
||||||
|
c_name = self.gen_unique_name("test_col_mmap")
|
||||||
|
self.resources_to_cleanup.append(("collection", c_name))
|
||||||
|
|
||||||
|
# Initial cleanup
|
||||||
|
self.cleanup_collection(upstream_client, c_name)
|
||||||
|
|
||||||
|
# Create collection
|
||||||
|
self._create_basic_collection(upstream_client, c_name)
|
||||||
|
|
||||||
|
# Wait for collection to sync downstream
|
||||||
|
def check_create():
|
||||||
|
return downstream_client.has_collection(c_name)
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {c_name}")
|
||||||
|
|
||||||
|
# Alter mmap.enabled property
|
||||||
|
upstream_client.alter_collection_properties(
|
||||||
|
collection_name=c_name,
|
||||||
|
properties={"mmap.enabled": "true"},
|
||||||
|
)
|
||||||
|
|
||||||
|
# Wait for property to sync downstream
|
||||||
|
def check_mmap():
|
||||||
|
try:
|
||||||
|
desc = downstream_client.describe_collection(c_name)
|
||||||
|
props = desc.get("properties", {})
|
||||||
|
logger.info(f"Downstream collection properties: {props}")
|
||||||
|
return str(props.get("mmap.enabled", "")).lower() == "true"
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Check mmap sync failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_mmap, sync_timeout, f"mmap property sync for {c_name}")
|
||||||
|
|
||||||
|
def test_autocompaction_sync(self, upstream_client, downstream_client, sync_timeout):
|
||||||
|
"""Test ALTER_COLLECTION_PROPERTIES (autocompaction.enabled) sync."""
|
||||||
|
# Store upstream client for teardown
|
||||||
|
self._upstream_client = upstream_client
|
||||||
|
|
||||||
|
c_name = self.gen_unique_name("test_col_autocomp")
|
||||||
|
self.resources_to_cleanup.append(("collection", c_name))
|
||||||
|
|
||||||
|
# Initial cleanup
|
||||||
|
self.cleanup_collection(upstream_client, c_name)
|
||||||
|
|
||||||
|
# Create collection
|
||||||
|
self._create_basic_collection(upstream_client, c_name)
|
||||||
|
|
||||||
|
# Wait for collection to sync downstream
|
||||||
|
def check_create():
|
||||||
|
return downstream_client.has_collection(c_name)
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {c_name}")
|
||||||
|
|
||||||
|
# Alter autocompaction property
|
||||||
|
upstream_client.alter_collection_properties(
|
||||||
|
collection_name=c_name,
|
||||||
|
properties={"collection.autocompaction.enabled": "true"},
|
||||||
|
)
|
||||||
|
|
||||||
|
# Wait for property to sync downstream
|
||||||
|
def check_autocomp():
|
||||||
|
try:
|
||||||
|
desc = downstream_client.describe_collection(c_name)
|
||||||
|
props = desc.get("properties", {})
|
||||||
|
logger.info(f"Downstream collection properties: {props}")
|
||||||
|
return str(props.get("collection.autocompaction.enabled", "")).lower() == "true"
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Check autocompaction sync failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_autocomp, sync_timeout, f"autocompaction property sync for {c_name}")
|
||||||
|
|
||||||
|
def test_alter_multiple_properties(self, upstream_client, downstream_client, sync_timeout):
|
||||||
|
"""Test ALTER_COLLECTION_PROPERTIES with multiple properties at once sync."""
|
||||||
|
# Store upstream client for teardown
|
||||||
|
self._upstream_client = upstream_client
|
||||||
|
|
||||||
|
c_name = self.gen_unique_name("test_col_multi_props")
|
||||||
|
self.resources_to_cleanup.append(("collection", c_name))
|
||||||
|
|
||||||
|
# Initial cleanup
|
||||||
|
self.cleanup_collection(upstream_client, c_name)
|
||||||
|
|
||||||
|
# Create collection
|
||||||
|
self._create_basic_collection(upstream_client, c_name)
|
||||||
|
|
||||||
|
# Wait for collection to sync downstream
|
||||||
|
def check_create():
|
||||||
|
return downstream_client.has_collection(c_name)
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {c_name}")
|
||||||
|
|
||||||
|
# Alter all 3 properties at once
|
||||||
|
upstream_client.alter_collection_properties(
|
||||||
|
collection_name=c_name,
|
||||||
|
properties={
|
||||||
|
"collection.ttl.seconds": "3600",
|
||||||
|
"mmap.enabled": "true",
|
||||||
|
"collection.autocompaction.enabled": "true",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
# Wait for all 3 properties to sync downstream
|
||||||
|
def check_all_props():
|
||||||
|
try:
|
||||||
|
desc = downstream_client.describe_collection(c_name)
|
||||||
|
props = desc.get("properties", {})
|
||||||
|
logger.info(f"Downstream collection properties: {props}")
|
||||||
|
ttl_ok = str(props.get("collection.ttl.seconds", "")) == "3600"
|
||||||
|
mmap_ok = str(props.get("mmap.enabled", "")).lower() == "true"
|
||||||
|
autocomp_ok = str(props.get("collection.autocompaction.enabled", "")).lower() == "true"
|
||||||
|
return ttl_ok and mmap_ok and autocomp_ok
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Check all properties sync failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_all_props, sync_timeout, f"all properties sync for {c_name}")
|
||||||
|
|
||||||
|
def test_drop_properties_sync(self, upstream_client, downstream_client, sync_timeout):
|
||||||
|
"""Test DROP_COLLECTION_PROPERTIES — set TTL + mmap, drop TTL, verify TTL gone and mmap remains."""
|
||||||
|
# Store upstream client for teardown
|
||||||
|
self._upstream_client = upstream_client
|
||||||
|
|
||||||
|
c_name = self.gen_unique_name("test_col_drop_props")
|
||||||
|
self.resources_to_cleanup.append(("collection", c_name))
|
||||||
|
|
||||||
|
# Initial cleanup
|
||||||
|
self.cleanup_collection(upstream_client, c_name)
|
||||||
|
|
||||||
|
# Create collection
|
||||||
|
self._create_basic_collection(upstream_client, c_name)
|
||||||
|
|
||||||
|
# Wait for collection to sync downstream
|
||||||
|
def check_create():
|
||||||
|
return downstream_client.has_collection(c_name)
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {c_name}")
|
||||||
|
|
||||||
|
# Set TTL and mmap properties
|
||||||
|
upstream_client.alter_collection_properties(
|
||||||
|
collection_name=c_name,
|
||||||
|
properties={
|
||||||
|
"collection.ttl.seconds": "3600",
|
||||||
|
"mmap.enabled": "true",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
# Wait for both properties to sync downstream
|
||||||
|
def check_props_set():
|
||||||
|
try:
|
||||||
|
desc = downstream_client.describe_collection(c_name)
|
||||||
|
props = desc.get("properties", {})
|
||||||
|
ttl_set = str(props.get("collection.ttl.seconds", "")) == "3600"
|
||||||
|
mmap_set = str(props.get("mmap.enabled", "")).lower() == "true"
|
||||||
|
return ttl_set and mmap_set
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Check properties set failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_props_set, sync_timeout, f"set properties for {c_name}")
|
||||||
|
|
||||||
|
# Drop TTL property
|
||||||
|
upstream_client.drop_collection_properties(
|
||||||
|
collection_name=c_name,
|
||||||
|
property_keys=["collection.ttl.seconds"],
|
||||||
|
)
|
||||||
|
|
||||||
|
# Wait for TTL to be gone but mmap to remain on downstream
|
||||||
|
def check_drop_ttl():
|
||||||
|
try:
|
||||||
|
desc = downstream_client.describe_collection(c_name)
|
||||||
|
props = desc.get("properties", {})
|
||||||
|
logger.info(f"Downstream collection properties after drop: {props}")
|
||||||
|
ttl_gone = "collection.ttl.seconds" not in props
|
||||||
|
mmap_remains = str(props.get("mmap.enabled", "")).lower() == "true"
|
||||||
|
return ttl_gone and mmap_remains
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Check drop TTL sync failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_drop_ttl, sync_timeout, f"drop TTL property sync for {c_name}")
|
||||||
@@ -0,0 +1,715 @@
|
|||||||
|
"""
|
||||||
|
CDC sync tests for full-text search (BM25), text match, and phrase match operations.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import random
|
||||||
|
import time
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from pymilvus import AnnSearchRequest, DataType, RRFRanker
|
||||||
|
|
||||||
|
from .base import TestCDCSyncBase, logger
|
||||||
|
|
||||||
|
|
||||||
|
class TestCDCSyncFTSAndText(TestCDCSyncBase):
|
||||||
|
"""Test CDC sync for full-text search, text match, and phrase match operations."""
|
||||||
|
|
||||||
|
def setup_method(self):
|
||||||
|
"""Setup for each test method."""
|
||||||
|
self.resources_to_cleanup = []
|
||||||
|
|
||||||
|
def teardown_method(self):
|
||||||
|
"""Cleanup after each test method - only cleanup upstream, downstream will sync."""
|
||||||
|
upstream_client = getattr(self, "_upstream_client", None)
|
||||||
|
|
||||||
|
if upstream_client:
|
||||||
|
for resource_type, resource_name in self.resources_to_cleanup:
|
||||||
|
if resource_type == "collection":
|
||||||
|
self.cleanup_collection(upstream_client, resource_name)
|
||||||
|
|
||||||
|
time.sleep(1) # Allow cleanup to sync to downstream
|
||||||
|
|
||||||
|
# -------------------------------------------------------------------------
|
||||||
|
# Test 1: FTS insert and search replication
|
||||||
|
# -------------------------------------------------------------------------
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("analyzer_type", ["standard", "english"])
|
||||||
|
def test_fts_insert_and_search(
|
||||||
|
self,
|
||||||
|
upstream_client,
|
||||||
|
downstream_client,
|
||||||
|
sync_timeout,
|
||||||
|
analyzer_type,
|
||||||
|
):
|
||||||
|
"""Test FTS (BM25) insert and search replication.
|
||||||
|
|
||||||
|
Creates an FTS collection with BM25 function, inserts 200 docs, creates
|
||||||
|
SPARSE_INVERTED_INDEX (BM25) and HNSW indexes, then verifies that FTS search
|
||||||
|
results replicate to downstream with sufficient overlap.
|
||||||
|
"""
|
||||||
|
start_time = time.time()
|
||||||
|
collection_name = self.gen_unique_name("fts_search", max_length=50)
|
||||||
|
|
||||||
|
self.log_test_start(
|
||||||
|
"test_fts_insert_and_search",
|
||||||
|
f"FTS_INSERT_SEARCH(analyzer={analyzer_type})",
|
||||||
|
collection_name,
|
||||||
|
)
|
||||||
|
|
||||||
|
self._upstream_client = upstream_client
|
||||||
|
self.resources_to_cleanup.append(("collection", collection_name))
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Initial cleanup
|
||||||
|
self.cleanup_collection(upstream_client, collection_name)
|
||||||
|
|
||||||
|
# Create FTS schema and collection
|
||||||
|
logger.info(f"[CREATE] Creating FTS collection '{collection_name}' with analyzer_type='{analyzer_type}'")
|
||||||
|
schema = self.create_fts_schema(upstream_client, analyzer_type)
|
||||||
|
upstream_client.create_collection(collection_name, schema=schema)
|
||||||
|
|
||||||
|
# Insert 200 documents
|
||||||
|
fts_data = self.generate_fts_data(200)
|
||||||
|
logger.info(f"[INSERT] Inserting {len(fts_data)} FTS documents upstream")
|
||||||
|
result = upstream_client.insert(collection_name, fts_data)
|
||||||
|
inserted_count = result.get("insert_count", len(fts_data))
|
||||||
|
logger.info(f"[INSERT] Inserted {inserted_count} documents")
|
||||||
|
|
||||||
|
upstream_client.flush(collection_name)
|
||||||
|
|
||||||
|
# Create SPARSE_INVERTED_INDEX on sparse_output (BM25)
|
||||||
|
index_params = upstream_client.prepare_index_params()
|
||||||
|
index_params.add_index(
|
||||||
|
field_name="sparse_output",
|
||||||
|
index_type="SPARSE_INVERTED_INDEX",
|
||||||
|
metric_type="BM25",
|
||||||
|
params={"bm25_k1": 1.5, "bm25_b": 0.75},
|
||||||
|
)
|
||||||
|
# Create HNSW on dense_vector
|
||||||
|
index_params.add_index(
|
||||||
|
field_name="dense_vector",
|
||||||
|
index_type="HNSW",
|
||||||
|
metric_type="L2",
|
||||||
|
params={"M": 8, "efConstruction": 64},
|
||||||
|
)
|
||||||
|
upstream_client.create_index(collection_name, index_params)
|
||||||
|
upstream_client.load_collection(collection_name)
|
||||||
|
|
||||||
|
# Run FTS search on upstream
|
||||||
|
query_text = "vector database similarity search"
|
||||||
|
logger.info(f"[SEARCH] Running FTS search upstream with query: '{query_text}'")
|
||||||
|
upstream_results = upstream_client.search(
|
||||||
|
collection_name,
|
||||||
|
data=[query_text],
|
||||||
|
anns_field="sparse_output",
|
||||||
|
limit=10,
|
||||||
|
search_params={"metric_type": "BM25"},
|
||||||
|
output_fields=["text_field", "category"],
|
||||||
|
)
|
||||||
|
upstream_ids = set()
|
||||||
|
if upstream_results and len(upstream_results) > 0:
|
||||||
|
for hit in upstream_results[0]:
|
||||||
|
upstream_ids.add(hit.get("id") or hit.id)
|
||||||
|
|
||||||
|
logger.info(f"[SEARCH] Upstream FTS returned {len(upstream_ids)} results")
|
||||||
|
|
||||||
|
# Wait for collection to appear downstream
|
||||||
|
def check_collection_exists():
|
||||||
|
return downstream_client.has_collection(collection_name)
|
||||||
|
|
||||||
|
assert self.wait_for_sync(
|
||||||
|
check_collection_exists,
|
||||||
|
sync_timeout,
|
||||||
|
f"collection '{collection_name}' creation sync",
|
||||||
|
), f"Collection '{collection_name}' did not sync to downstream"
|
||||||
|
|
||||||
|
# Wait for data + index to sync and downstream search results overlap
|
||||||
|
def check_fts_overlap():
|
||||||
|
try:
|
||||||
|
ds_results = downstream_client.search(
|
||||||
|
collection_name,
|
||||||
|
data=[query_text],
|
||||||
|
anns_field="sparse_output",
|
||||||
|
limit=10,
|
||||||
|
search_params={"metric_type": "BM25"},
|
||||||
|
output_fields=["text_field", "category"],
|
||||||
|
)
|
||||||
|
if not ds_results or len(ds_results) == 0:
|
||||||
|
return False
|
||||||
|
downstream_ids = set()
|
||||||
|
for hit in ds_results[0]:
|
||||||
|
downstream_ids.add(hit.get("id") or hit.id)
|
||||||
|
if not downstream_ids:
|
||||||
|
return False
|
||||||
|
if len(upstream_ids) == 0:
|
||||||
|
return len(downstream_ids) > 0
|
||||||
|
overlap = len(upstream_ids & downstream_ids) / max(len(upstream_ids), 1)
|
||||||
|
logger.info(
|
||||||
|
f"[OVERLAP] FTS search overlap: {overlap:.2f} "
|
||||||
|
f"(upstream={len(upstream_ids)}, downstream={len(downstream_ids)})"
|
||||||
|
)
|
||||||
|
return overlap >= self.SEARCH_OVERLAP_THRESHOLD
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"FTS overlap check failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
assert self.wait_for_sync(
|
||||||
|
check_fts_overlap,
|
||||||
|
sync_timeout,
|
||||||
|
f"FTS search overlap sync (analyzer={analyzer_type})",
|
||||||
|
), f"FTS search results did not reach overlap threshold {self.SEARCH_OVERLAP_THRESHOLD} on downstream"
|
||||||
|
|
||||||
|
duration = time.time() - start_time
|
||||||
|
self.log_test_end("test_fts_insert_and_search", True, duration)
|
||||||
|
|
||||||
|
except Exception as exc:
|
||||||
|
duration = time.time() - start_time
|
||||||
|
self.log_test_end("test_fts_insert_and_search", False, duration)
|
||||||
|
raise exc
|
||||||
|
|
||||||
|
# -------------------------------------------------------------------------
|
||||||
|
# Test 2: TEXT_MATCH sync
|
||||||
|
# -------------------------------------------------------------------------
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("analyzer_type", ["standard", "english"])
|
||||||
|
def test_text_match_sync(
|
||||||
|
self,
|
||||||
|
upstream_client,
|
||||||
|
downstream_client,
|
||||||
|
sync_timeout,
|
||||||
|
analyzer_type,
|
||||||
|
):
|
||||||
|
"""Test TEXT_MATCH query replication.
|
||||||
|
|
||||||
|
Creates a collection with a VARCHAR field that has analyzer + match enabled,
|
||||||
|
inserts 100 rows, and verifies that TEXT_MATCH queries return the same count
|
||||||
|
on both upstream and downstream.
|
||||||
|
"""
|
||||||
|
start_time = time.time()
|
||||||
|
collection_name = self.gen_unique_name("text_match", max_length=50)
|
||||||
|
|
||||||
|
self.log_test_start(
|
||||||
|
"test_text_match_sync",
|
||||||
|
f"TEXT_MATCH(analyzer={analyzer_type})",
|
||||||
|
collection_name,
|
||||||
|
)
|
||||||
|
|
||||||
|
self._upstream_client = upstream_client
|
||||||
|
self.resources_to_cleanup.append(("collection", collection_name))
|
||||||
|
|
||||||
|
try:
|
||||||
|
self.cleanup_collection(upstream_client, collection_name)
|
||||||
|
|
||||||
|
# Build schema with analyzer-enabled VARCHAR
|
||||||
|
schema = upstream_client.create_schema(enable_dynamic_field=False)
|
||||||
|
schema.add_field("id", DataType.INT64, is_primary=True, auto_id=True)
|
||||||
|
schema.add_field("dense_vector", DataType.FLOAT_VECTOR, dim=128)
|
||||||
|
schema.add_field(
|
||||||
|
"text_field",
|
||||||
|
DataType.VARCHAR,
|
||||||
|
max_length=2048,
|
||||||
|
enable_analyzer=True,
|
||||||
|
enable_match=True,
|
||||||
|
analyzer_params={"type": analyzer_type},
|
||||||
|
)
|
||||||
|
|
||||||
|
logger.info(f"[CREATE] Creating text-match collection '{collection_name}'")
|
||||||
|
upstream_client.create_collection(collection_name, schema=schema)
|
||||||
|
|
||||||
|
# Insert 100 rows from FTS_SENTENCES
|
||||||
|
data = []
|
||||||
|
for i in range(100):
|
||||||
|
data.append(
|
||||||
|
{
|
||||||
|
"dense_vector": [random.random() for _ in range(128)],
|
||||||
|
"text_field": self.FTS_SENTENCES[i % len(self.FTS_SENTENCES)],
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
logger.info(f"[INSERT] Inserting {len(data)} rows upstream")
|
||||||
|
upstream_client.insert(collection_name, data)
|
||||||
|
upstream_client.flush(collection_name)
|
||||||
|
|
||||||
|
# Create HNSW index and load
|
||||||
|
index_params = upstream_client.prepare_index_params()
|
||||||
|
index_params.add_index(
|
||||||
|
field_name="dense_vector",
|
||||||
|
index_type="HNSW",
|
||||||
|
metric_type="L2",
|
||||||
|
params={"M": 8, "efConstruction": 64},
|
||||||
|
)
|
||||||
|
upstream_client.create_index(collection_name, index_params)
|
||||||
|
upstream_client.load_collection(collection_name)
|
||||||
|
|
||||||
|
# Run TEXT_MATCH query on upstream
|
||||||
|
filter_expr = "TEXT_MATCH(text_field, 'vector database')"
|
||||||
|
logger.info(f"[QUERY] Upstream TEXT_MATCH query: {filter_expr}")
|
||||||
|
upstream_results = upstream_client.query(
|
||||||
|
collection_name,
|
||||||
|
filter=filter_expr,
|
||||||
|
output_fields=["id", "text_field"],
|
||||||
|
limit=100,
|
||||||
|
)
|
||||||
|
upstream_count = len(upstream_results)
|
||||||
|
logger.info(f"[QUERY] Upstream TEXT_MATCH returned {upstream_count} rows")
|
||||||
|
|
||||||
|
# Wait for collection to appear downstream
|
||||||
|
def check_collection_exists():
|
||||||
|
return downstream_client.has_collection(collection_name)
|
||||||
|
|
||||||
|
assert self.wait_for_sync(
|
||||||
|
check_collection_exists,
|
||||||
|
sync_timeout,
|
||||||
|
f"collection '{collection_name}' creation sync",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Wait for downstream query count to match
|
||||||
|
def check_count_match():
|
||||||
|
try:
|
||||||
|
ds_results = downstream_client.query(
|
||||||
|
collection_name,
|
||||||
|
filter=filter_expr,
|
||||||
|
output_fields=["id", "text_field"],
|
||||||
|
limit=100,
|
||||||
|
)
|
||||||
|
ds_count = len(ds_results)
|
||||||
|
logger.info(f"[VERIFY] TEXT_MATCH downstream count={ds_count}, upstream count={upstream_count}")
|
||||||
|
return ds_count == upstream_count
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"TEXT_MATCH count check failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
assert self.wait_for_sync(
|
||||||
|
check_count_match,
|
||||||
|
sync_timeout,
|
||||||
|
f"TEXT_MATCH query count sync (analyzer={analyzer_type})",
|
||||||
|
), f"TEXT_MATCH query count mismatch between upstream ({upstream_count}) and downstream after timeout"
|
||||||
|
|
||||||
|
duration = time.time() - start_time
|
||||||
|
self.log_test_end("test_text_match_sync", True, duration)
|
||||||
|
|
||||||
|
except Exception as exc:
|
||||||
|
duration = time.time() - start_time
|
||||||
|
self.log_test_end("test_text_match_sync", False, duration)
|
||||||
|
raise exc
|
||||||
|
|
||||||
|
# -------------------------------------------------------------------------
|
||||||
|
# Test 3: PHRASE_MATCH sync
|
||||||
|
# -------------------------------------------------------------------------
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("analyzer_type", ["standard", "english"])
|
||||||
|
def test_phrase_match_sync(
|
||||||
|
self,
|
||||||
|
upstream_client,
|
||||||
|
downstream_client,
|
||||||
|
sync_timeout,
|
||||||
|
analyzer_type,
|
||||||
|
):
|
||||||
|
"""Test PHRASE_MATCH query replication.
|
||||||
|
|
||||||
|
Same collection setup as text_match. Queries PHRASE_MATCH with slop=1 and
|
||||||
|
verifies that upstream and downstream return the same count.
|
||||||
|
"""
|
||||||
|
start_time = time.time()
|
||||||
|
collection_name = self.gen_unique_name("phrase_match", max_length=50)
|
||||||
|
|
||||||
|
self.log_test_start(
|
||||||
|
"test_phrase_match_sync",
|
||||||
|
f"PHRASE_MATCH(analyzer={analyzer_type})",
|
||||||
|
collection_name,
|
||||||
|
)
|
||||||
|
|
||||||
|
self._upstream_client = upstream_client
|
||||||
|
self.resources_to_cleanup.append(("collection", collection_name))
|
||||||
|
|
||||||
|
try:
|
||||||
|
self.cleanup_collection(upstream_client, collection_name)
|
||||||
|
|
||||||
|
# Build schema with analyzer-enabled VARCHAR
|
||||||
|
schema = upstream_client.create_schema(enable_dynamic_field=False)
|
||||||
|
schema.add_field("id", DataType.INT64, is_primary=True, auto_id=True)
|
||||||
|
schema.add_field("dense_vector", DataType.FLOAT_VECTOR, dim=128)
|
||||||
|
schema.add_field(
|
||||||
|
"text_field",
|
||||||
|
DataType.VARCHAR,
|
||||||
|
max_length=2048,
|
||||||
|
enable_analyzer=True,
|
||||||
|
enable_match=True,
|
||||||
|
analyzer_params={"type": analyzer_type},
|
||||||
|
)
|
||||||
|
|
||||||
|
logger.info(f"[CREATE] Creating phrase-match collection '{collection_name}'")
|
||||||
|
upstream_client.create_collection(collection_name, schema=schema)
|
||||||
|
|
||||||
|
# Insert 100 rows from FTS_SENTENCES
|
||||||
|
data = []
|
||||||
|
for i in range(100):
|
||||||
|
data.append(
|
||||||
|
{
|
||||||
|
"dense_vector": [random.random() for _ in range(128)],
|
||||||
|
"text_field": self.FTS_SENTENCES[i % len(self.FTS_SENTENCES)],
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
logger.info(f"[INSERT] Inserting {len(data)} rows upstream")
|
||||||
|
upstream_client.insert(collection_name, data)
|
||||||
|
upstream_client.flush(collection_name)
|
||||||
|
|
||||||
|
# Create HNSW index and load
|
||||||
|
index_params = upstream_client.prepare_index_params()
|
||||||
|
index_params.add_index(
|
||||||
|
field_name="dense_vector",
|
||||||
|
index_type="HNSW",
|
||||||
|
metric_type="L2",
|
||||||
|
params={"M": 8, "efConstruction": 64},
|
||||||
|
)
|
||||||
|
upstream_client.create_index(collection_name, index_params)
|
||||||
|
upstream_client.load_collection(collection_name)
|
||||||
|
|
||||||
|
# Run PHRASE_MATCH query on upstream (slop=1)
|
||||||
|
filter_expr = "PHRASE_MATCH(text_field, 'brown fox', 1)"
|
||||||
|
logger.info(f"[QUERY] Upstream PHRASE_MATCH query: {filter_expr}")
|
||||||
|
upstream_results = upstream_client.query(
|
||||||
|
collection_name,
|
||||||
|
filter=filter_expr,
|
||||||
|
output_fields=["id", "text_field"],
|
||||||
|
limit=100,
|
||||||
|
)
|
||||||
|
upstream_count = len(upstream_results)
|
||||||
|
logger.info(f"[QUERY] Upstream PHRASE_MATCH returned {upstream_count} rows")
|
||||||
|
|
||||||
|
# Wait for collection to appear downstream
|
||||||
|
def check_collection_exists():
|
||||||
|
return downstream_client.has_collection(collection_name)
|
||||||
|
|
||||||
|
assert self.wait_for_sync(
|
||||||
|
check_collection_exists,
|
||||||
|
sync_timeout,
|
||||||
|
f"collection '{collection_name}' creation sync",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Wait for downstream query count to match
|
||||||
|
def check_count_match():
|
||||||
|
try:
|
||||||
|
ds_results = downstream_client.query(
|
||||||
|
collection_name,
|
||||||
|
filter=filter_expr,
|
||||||
|
output_fields=["id", "text_field"],
|
||||||
|
limit=100,
|
||||||
|
)
|
||||||
|
ds_count = len(ds_results)
|
||||||
|
logger.info(f"[VERIFY] PHRASE_MATCH downstream count={ds_count}, upstream count={upstream_count}")
|
||||||
|
return ds_count == upstream_count
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"PHRASE_MATCH count check failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
assert self.wait_for_sync(
|
||||||
|
check_count_match,
|
||||||
|
sync_timeout,
|
||||||
|
f"PHRASE_MATCH query count sync (analyzer={analyzer_type})",
|
||||||
|
), f"PHRASE_MATCH query count mismatch between upstream ({upstream_count}) and downstream after timeout"
|
||||||
|
|
||||||
|
duration = time.time() - start_time
|
||||||
|
self.log_test_end("test_phrase_match_sync", True, duration)
|
||||||
|
|
||||||
|
except Exception as exc:
|
||||||
|
duration = time.time() - start_time
|
||||||
|
self.log_test_end("test_phrase_match_sync", False, duration)
|
||||||
|
raise exc
|
||||||
|
|
||||||
|
# -------------------------------------------------------------------------
|
||||||
|
# Test 4: Hybrid search (FTS + dense) replication
|
||||||
|
# -------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def test_hybrid_search_fts_dense(
|
||||||
|
self,
|
||||||
|
upstream_client,
|
||||||
|
downstream_client,
|
||||||
|
sync_timeout,
|
||||||
|
):
|
||||||
|
"""Test hybrid search (BM25 sparse + HNSW dense) replication.
|
||||||
|
|
||||||
|
Creates an FTS collection, inserts 300 docs, builds both sparse (BM25) and
|
||||||
|
dense (HNSW) indexes, and verifies that hybrid search results on downstream
|
||||||
|
have sufficient overlap with upstream results.
|
||||||
|
"""
|
||||||
|
start_time = time.time()
|
||||||
|
collection_name = self.gen_unique_name("hybrid_fts", max_length=50)
|
||||||
|
|
||||||
|
self.log_test_start(
|
||||||
|
"test_hybrid_search_fts_dense",
|
||||||
|
"HYBRID_SEARCH_FTS_DENSE",
|
||||||
|
collection_name,
|
||||||
|
)
|
||||||
|
|
||||||
|
self._upstream_client = upstream_client
|
||||||
|
self.resources_to_cleanup.append(("collection", collection_name))
|
||||||
|
|
||||||
|
try:
|
||||||
|
self.cleanup_collection(upstream_client, collection_name)
|
||||||
|
|
||||||
|
# Create FTS schema (standard analyzer)
|
||||||
|
schema = self.create_fts_schema(upstream_client, "standard")
|
||||||
|
logger.info(f"[CREATE] Creating hybrid-search collection '{collection_name}'")
|
||||||
|
upstream_client.create_collection(collection_name, schema=schema)
|
||||||
|
|
||||||
|
# Insert 300 documents
|
||||||
|
fts_data = self.generate_fts_data(300)
|
||||||
|
logger.info(f"[INSERT] Inserting {len(fts_data)} documents upstream")
|
||||||
|
upstream_client.insert(collection_name, fts_data)
|
||||||
|
upstream_client.flush(collection_name)
|
||||||
|
|
||||||
|
# Create both indexes
|
||||||
|
index_params = upstream_client.prepare_index_params()
|
||||||
|
index_params.add_index(
|
||||||
|
field_name="sparse_output",
|
||||||
|
index_type="SPARSE_INVERTED_INDEX",
|
||||||
|
metric_type="BM25",
|
||||||
|
params={"bm25_k1": 1.5, "bm25_b": 0.75},
|
||||||
|
)
|
||||||
|
index_params.add_index(
|
||||||
|
field_name="dense_vector",
|
||||||
|
index_type="HNSW",
|
||||||
|
metric_type="L2",
|
||||||
|
params={"M": 8, "efConstruction": 64},
|
||||||
|
)
|
||||||
|
upstream_client.create_index(collection_name, index_params)
|
||||||
|
upstream_client.load_collection(collection_name)
|
||||||
|
|
||||||
|
# Build hybrid search requests
|
||||||
|
sparse_req = AnnSearchRequest(
|
||||||
|
data=["vector database similarity search"],
|
||||||
|
anns_field="sparse_output",
|
||||||
|
param={"metric_type": "BM25"},
|
||||||
|
limit=10,
|
||||||
|
)
|
||||||
|
dense_query_vec = [random.random() for _ in range(128)]
|
||||||
|
dense_req = AnnSearchRequest(
|
||||||
|
data=[dense_query_vec],
|
||||||
|
anns_field="dense_vector",
|
||||||
|
param={"metric_type": "L2", "params": {"ef": 64}},
|
||||||
|
limit=10,
|
||||||
|
)
|
||||||
|
|
||||||
|
logger.info("[SEARCH] Running hybrid search on upstream")
|
||||||
|
upstream_results = upstream_client.hybrid_search(
|
||||||
|
collection_name,
|
||||||
|
reqs=[sparse_req, dense_req],
|
||||||
|
ranker=RRFRanker(),
|
||||||
|
limit=10,
|
||||||
|
output_fields=["text_field", "category"],
|
||||||
|
)
|
||||||
|
|
||||||
|
upstream_ids = set()
|
||||||
|
if upstream_results and len(upstream_results) > 0:
|
||||||
|
for hit in upstream_results[0]:
|
||||||
|
upstream_ids.add(hit.get("id") or hit.id)
|
||||||
|
|
||||||
|
logger.info(f"[SEARCH] Upstream hybrid search returned {len(upstream_ids)} results")
|
||||||
|
|
||||||
|
# Wait for collection to appear downstream
|
||||||
|
def check_collection_exists():
|
||||||
|
return downstream_client.has_collection(collection_name)
|
||||||
|
|
||||||
|
assert self.wait_for_sync(
|
||||||
|
check_collection_exists,
|
||||||
|
sync_timeout,
|
||||||
|
f"collection '{collection_name}' creation sync",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Wait for downstream hybrid search overlap
|
||||||
|
def check_hybrid_overlap():
|
||||||
|
try:
|
||||||
|
ds_sparse_req = AnnSearchRequest(
|
||||||
|
data=["vector database similarity search"],
|
||||||
|
anns_field="sparse_output",
|
||||||
|
param={"metric_type": "BM25"},
|
||||||
|
limit=10,
|
||||||
|
)
|
||||||
|
ds_dense_req = AnnSearchRequest(
|
||||||
|
data=[dense_query_vec],
|
||||||
|
anns_field="dense_vector",
|
||||||
|
param={"metric_type": "L2", "params": {"ef": 64}},
|
||||||
|
limit=10,
|
||||||
|
)
|
||||||
|
ds_results = downstream_client.hybrid_search(
|
||||||
|
collection_name,
|
||||||
|
reqs=[ds_sparse_req, ds_dense_req],
|
||||||
|
ranker=RRFRanker(),
|
||||||
|
limit=10,
|
||||||
|
output_fields=["text_field", "category"],
|
||||||
|
)
|
||||||
|
if not ds_results or len(ds_results) == 0:
|
||||||
|
return False
|
||||||
|
downstream_ids = set()
|
||||||
|
for hit in ds_results[0]:
|
||||||
|
downstream_ids.add(hit.get("id") or hit.id)
|
||||||
|
if not downstream_ids:
|
||||||
|
return False
|
||||||
|
if len(upstream_ids) == 0:
|
||||||
|
return len(downstream_ids) > 0
|
||||||
|
overlap = len(upstream_ids & downstream_ids) / max(len(upstream_ids), 1)
|
||||||
|
logger.info(
|
||||||
|
f"[OVERLAP] Hybrid search overlap: {overlap:.2f} "
|
||||||
|
f"(upstream={len(upstream_ids)}, downstream={len(downstream_ids)})"
|
||||||
|
)
|
||||||
|
return overlap >= self.SEARCH_OVERLAP_THRESHOLD
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Hybrid overlap check failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
assert self.wait_for_sync(
|
||||||
|
check_hybrid_overlap,
|
||||||
|
sync_timeout,
|
||||||
|
"hybrid search (FTS + dense) overlap sync",
|
||||||
|
), f"Hybrid search results did not reach overlap threshold {self.SEARCH_OVERLAP_THRESHOLD} on downstream"
|
||||||
|
|
||||||
|
duration = time.time() - start_time
|
||||||
|
self.log_test_end("test_hybrid_search_fts_dense", True, duration)
|
||||||
|
|
||||||
|
except Exception as exc:
|
||||||
|
duration = time.time() - start_time
|
||||||
|
self.log_test_end("test_hybrid_search_fts_dense", False, duration)
|
||||||
|
raise exc
|
||||||
|
|
||||||
|
# -------------------------------------------------------------------------
|
||||||
|
# Test 5: FTS after switchover
|
||||||
|
# -------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def test_fts_after_switchover(
|
||||||
|
self,
|
||||||
|
upstream_client,
|
||||||
|
downstream_client,
|
||||||
|
sync_timeout,
|
||||||
|
switchover_helper,
|
||||||
|
source_cluster_id,
|
||||||
|
target_cluster_id,
|
||||||
|
):
|
||||||
|
"""Test FTS replication continues correctly after CDC topology switchover.
|
||||||
|
|
||||||
|
1. Create FTS collection, insert 100 docs, build index, verify sync.
|
||||||
|
2. Perform switchover so downstream becomes the new source.
|
||||||
|
3. Insert 50 more docs into the new source (original downstream).
|
||||||
|
4. Verify FTS search works on the new downstream (original upstream).
|
||||||
|
5. Switch back to original topology.
|
||||||
|
"""
|
||||||
|
start_time = time.time()
|
||||||
|
collection_name = self.gen_unique_name("fts_switchover", max_length=50)
|
||||||
|
|
||||||
|
self.log_test_start(
|
||||||
|
"test_fts_after_switchover",
|
||||||
|
"FTS_AFTER_SWITCHOVER",
|
||||||
|
collection_name,
|
||||||
|
)
|
||||||
|
|
||||||
|
self._upstream_client = upstream_client
|
||||||
|
self.resources_to_cleanup.append(("collection", collection_name))
|
||||||
|
|
||||||
|
try:
|
||||||
|
self.cleanup_collection(upstream_client, collection_name)
|
||||||
|
|
||||||
|
# Phase 1: Create FTS collection and index on original upstream
|
||||||
|
logger.info(f"[PHASE1] Creating FTS collection '{collection_name}' on upstream")
|
||||||
|
schema = self.create_fts_schema(upstream_client, "standard")
|
||||||
|
upstream_client.create_collection(collection_name, schema=schema)
|
||||||
|
|
||||||
|
fts_data = self.generate_fts_data(100)
|
||||||
|
logger.info(f"[PHASE1] Inserting {len(fts_data)} documents upstream")
|
||||||
|
upstream_client.insert(collection_name, fts_data)
|
||||||
|
upstream_client.flush(collection_name)
|
||||||
|
|
||||||
|
index_params = upstream_client.prepare_index_params()
|
||||||
|
index_params.add_index(
|
||||||
|
field_name="sparse_output",
|
||||||
|
index_type="SPARSE_INVERTED_INDEX",
|
||||||
|
metric_type="BM25",
|
||||||
|
params={"bm25_k1": 1.5, "bm25_b": 0.75},
|
||||||
|
)
|
||||||
|
index_params.add_index(
|
||||||
|
field_name="dense_vector",
|
||||||
|
index_type="HNSW",
|
||||||
|
metric_type="L2",
|
||||||
|
params={"M": 8, "efConstruction": 64},
|
||||||
|
)
|
||||||
|
upstream_client.create_index(collection_name, index_params)
|
||||||
|
upstream_client.load_collection(collection_name)
|
||||||
|
|
||||||
|
# Verify initial sync to downstream
|
||||||
|
def check_initial_sync():
|
||||||
|
if not downstream_client.has_collection(collection_name):
|
||||||
|
return False
|
||||||
|
try:
|
||||||
|
ds_results = downstream_client.search(
|
||||||
|
collection_name,
|
||||||
|
data=["vector database"],
|
||||||
|
anns_field="sparse_output",
|
||||||
|
limit=5,
|
||||||
|
search_params={"metric_type": "BM25"},
|
||||||
|
output_fields=["text_field"],
|
||||||
|
)
|
||||||
|
return ds_results is not None and len(ds_results) > 0
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
assert self.wait_for_sync(
|
||||||
|
check_initial_sync,
|
||||||
|
sync_timeout,
|
||||||
|
f"initial FTS sync for '{collection_name}'",
|
||||||
|
), f"Initial FTS sync failed for collection '{collection_name}'"
|
||||||
|
|
||||||
|
logger.info("[PHASE1] Initial FTS sync verified")
|
||||||
|
|
||||||
|
# Phase 2: Switchover — downstream becomes new source
|
||||||
|
logger.info(f"[PHASE2] Switching CDC direction: {target_cluster_id} -> {source_cluster_id}")
|
||||||
|
switchover_helper(target_cluster_id, source_cluster_id)
|
||||||
|
|
||||||
|
# Insert 50 more docs to the new source (original downstream)
|
||||||
|
extra_data = self.generate_fts_data(50)
|
||||||
|
logger.info(f"[PHASE2] Inserting {len(extra_data)} additional docs to new source (downstream_client)")
|
||||||
|
downstream_client.insert(collection_name, extra_data)
|
||||||
|
downstream_client.flush(collection_name)
|
||||||
|
|
||||||
|
# Phase 3: Verify FTS search works on new downstream (original upstream)
|
||||||
|
query_text = "distributed database replication"
|
||||||
|
|
||||||
|
def check_fts_on_new_downstream():
|
||||||
|
try:
|
||||||
|
results = upstream_client.search(
|
||||||
|
collection_name,
|
||||||
|
data=[query_text],
|
||||||
|
anns_field="sparse_output",
|
||||||
|
limit=5,
|
||||||
|
search_params={"metric_type": "BM25"},
|
||||||
|
output_fields=["text_field"],
|
||||||
|
)
|
||||||
|
return results is not None and len(results) > 0
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"FTS on new downstream check failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
assert self.wait_for_sync(
|
||||||
|
check_fts_on_new_downstream,
|
||||||
|
sync_timeout,
|
||||||
|
"FTS search on new downstream after switchover",
|
||||||
|
), "FTS search on new downstream (original upstream) failed after switchover"
|
||||||
|
|
||||||
|
logger.info("[PHASE3] FTS search verified on new downstream after switchover")
|
||||||
|
|
||||||
|
# Phase 4: Switch back to original topology
|
||||||
|
logger.info(f"[PHASE4] Switching back to original topology: {source_cluster_id} -> {target_cluster_id}")
|
||||||
|
switchover_helper(source_cluster_id, target_cluster_id)
|
||||||
|
|
||||||
|
duration = time.time() - start_time
|
||||||
|
self.log_test_end("test_fts_after_switchover", True, duration)
|
||||||
|
|
||||||
|
except Exception as exc:
|
||||||
|
# Best-effort restore original topology on failure
|
||||||
|
try:
|
||||||
|
logger.warning("[RECOVER] Attempting to restore original CDC topology after failure")
|
||||||
|
switchover_helper(source_cluster_id, target_cluster_id)
|
||||||
|
except Exception as restore_exc:
|
||||||
|
logger.error(f"[RECOVER] Failed to restore topology: {restore_exc}")
|
||||||
|
duration = time.time() - start_time
|
||||||
|
self.log_test_end("test_fts_after_switchover", False, duration)
|
||||||
|
raise exc
|
||||||
@@ -0,0 +1,316 @@
|
|||||||
|
"""
|
||||||
|
CDC sync tests for multi-database operations.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import time
|
||||||
|
|
||||||
|
from pymilvus import MilvusClient
|
||||||
|
|
||||||
|
from .base import TestCDCSyncBase, logger
|
||||||
|
|
||||||
|
|
||||||
|
class TestCDCSyncMultiDatabase(TestCDCSyncBase):
|
||||||
|
"""Test CDC sync for operations across multiple databases."""
|
||||||
|
|
||||||
|
def setup_method(self):
|
||||||
|
"""Setup for each test method."""
|
||||||
|
self.resources_to_cleanup = []
|
||||||
|
|
||||||
|
def teardown_method(self):
|
||||||
|
"""Cleanup after each test method - only cleanup upstream, downstream will sync."""
|
||||||
|
upstream_uri = getattr(self, "_upstream_uri", None)
|
||||||
|
upstream_token = getattr(self, "_upstream_token", None)
|
||||||
|
|
||||||
|
if upstream_uri:
|
||||||
|
# Re-create a default-db client for cleanup
|
||||||
|
try:
|
||||||
|
client = MilvusClient(uri=upstream_uri, token=upstream_token)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"[CLEANUP] Failed to create upstream client: {e}")
|
||||||
|
return
|
||||||
|
|
||||||
|
# Clean up collections in databases first, then databases
|
||||||
|
for resource_type, resource_data in self.resources_to_cleanup:
|
||||||
|
if resource_type == "collection_in_db":
|
||||||
|
db_name, c_name = resource_data
|
||||||
|
try:
|
||||||
|
db_client = MilvusClient(
|
||||||
|
uri=upstream_uri,
|
||||||
|
token=upstream_token,
|
||||||
|
db_name=db_name,
|
||||||
|
)
|
||||||
|
if db_client.has_collection(c_name):
|
||||||
|
logger.info(f"[CLEANUP] Dropping collection {c_name} in db {db_name}")
|
||||||
|
db_client.drop_collection(c_name)
|
||||||
|
db_client.close()
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"[CLEANUP] Failed to drop collection {c_name} in db {db_name}: {e}")
|
||||||
|
|
||||||
|
for resource_type, resource_data in self.resources_to_cleanup:
|
||||||
|
if resource_type == "database":
|
||||||
|
db_name = resource_data
|
||||||
|
self.cleanup_database(client, db_name)
|
||||||
|
|
||||||
|
client.close()
|
||||||
|
time.sleep(1) # Allow cleanup to sync to downstream
|
||||||
|
|
||||||
|
def test_create_collections_in_multiple_dbs(
|
||||||
|
self,
|
||||||
|
upstream_uri,
|
||||||
|
upstream_token,
|
||||||
|
downstream_uri,
|
||||||
|
downstream_token,
|
||||||
|
upstream_client,
|
||||||
|
downstream_client,
|
||||||
|
sync_timeout,
|
||||||
|
):
|
||||||
|
"""Test creating collections in multiple databases syncs to downstream."""
|
||||||
|
self._upstream_uri = upstream_uri
|
||||||
|
self._upstream_token = upstream_token
|
||||||
|
|
||||||
|
db_name_1 = self.gen_unique_name("test_mdb_db1")
|
||||||
|
db_name_2 = self.gen_unique_name("test_mdb_db2")
|
||||||
|
c_name_1 = self.gen_unique_name("test_mdb_col1")
|
||||||
|
c_name_2 = self.gen_unique_name("test_mdb_col2")
|
||||||
|
|
||||||
|
self.resources_to_cleanup.append(("collection_in_db", (db_name_1, c_name_1)))
|
||||||
|
self.resources_to_cleanup.append(("collection_in_db", (db_name_2, c_name_2)))
|
||||||
|
self.resources_to_cleanup.append(("database", db_name_1))
|
||||||
|
self.resources_to_cleanup.append(("database", db_name_2))
|
||||||
|
|
||||||
|
# Initial cleanup
|
||||||
|
self.cleanup_database(upstream_client, db_name_1)
|
||||||
|
self.cleanup_database(upstream_client, db_name_2)
|
||||||
|
|
||||||
|
# Create databases
|
||||||
|
upstream_client.create_database(db_name_1)
|
||||||
|
upstream_client.create_database(db_name_2)
|
||||||
|
|
||||||
|
# Create DB-scoped clients
|
||||||
|
up_db1_client = MilvusClient(uri=upstream_uri, token=upstream_token, db_name=db_name_1)
|
||||||
|
up_db2_client = MilvusClient(uri=upstream_uri, token=upstream_token, db_name=db_name_2)
|
||||||
|
|
||||||
|
# Create collection in db1 with 100 rows
|
||||||
|
schema1 = self.create_default_schema(up_db1_client)
|
||||||
|
up_db1_client.create_collection(collection_name=c_name_1, schema=schema1, consistency_level="Strong")
|
||||||
|
up_db1_client.insert(c_name_1, self.generate_test_data(100))
|
||||||
|
up_db1_client.flush(c_name_1)
|
||||||
|
|
||||||
|
# Create collection in db2 with 200 rows
|
||||||
|
schema2 = self.create_default_schema(up_db2_client)
|
||||||
|
up_db2_client.create_collection(collection_name=c_name_2, schema=schema2, consistency_level="Strong")
|
||||||
|
up_db2_client.insert(c_name_2, self.generate_test_data(200))
|
||||||
|
up_db2_client.flush(c_name_2)
|
||||||
|
|
||||||
|
up_db1_client.close()
|
||||||
|
up_db2_client.close()
|
||||||
|
|
||||||
|
# Wait for both databases and collections to appear on downstream
|
||||||
|
def check_db1_collection():
|
||||||
|
try:
|
||||||
|
if db_name_1 not in downstream_client.list_databases():
|
||||||
|
return False
|
||||||
|
dn_db1 = MilvusClient(uri=downstream_uri, token=downstream_token, db_name=db_name_1)
|
||||||
|
exists = dn_db1.has_collection(c_name_1)
|
||||||
|
if exists:
|
||||||
|
stats = dn_db1.get_collection_stats(c_name_1)
|
||||||
|
logger.info(f"Downstream db1 collection stats: {stats}")
|
||||||
|
dn_db1.close()
|
||||||
|
return exists
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Check db1 collection failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_db1_collection, sync_timeout, f"collection {c_name_1} in {db_name_1}")
|
||||||
|
|
||||||
|
def check_db2_collection():
|
||||||
|
try:
|
||||||
|
if db_name_2 not in downstream_client.list_databases():
|
||||||
|
return False
|
||||||
|
dn_db2 = MilvusClient(uri=downstream_uri, token=downstream_token, db_name=db_name_2)
|
||||||
|
exists = dn_db2.has_collection(c_name_2)
|
||||||
|
if exists:
|
||||||
|
stats = dn_db2.get_collection_stats(c_name_2)
|
||||||
|
logger.info(f"Downstream db2 collection stats: {stats}")
|
||||||
|
dn_db2.close()
|
||||||
|
return exists
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Check db2 collection failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_db2_collection, sync_timeout, f"collection {c_name_2} in {db_name_2}")
|
||||||
|
|
||||||
|
# Verify row counts
|
||||||
|
dn_db1 = MilvusClient(uri=downstream_uri, token=downstream_token, db_name=db_name_1)
|
||||||
|
dn_db2 = MilvusClient(uri=downstream_uri, token=downstream_token, db_name=db_name_2)
|
||||||
|
try:
|
||||||
|
stats1 = dn_db1.get_collection_stats(c_name_1)
|
||||||
|
stats2 = dn_db2.get_collection_stats(c_name_2)
|
||||||
|
logger.info(f"DB1 collection row count: {stats1.get('row_count')}")
|
||||||
|
logger.info(f"DB2 collection row count: {stats2.get('row_count')}")
|
||||||
|
assert stats1.get("row_count", 0) >= 100, (
|
||||||
|
f"Expected >= 100 rows in {c_name_1}, got {stats1.get('row_count')}"
|
||||||
|
)
|
||||||
|
assert stats2.get("row_count", 0) >= 200, (
|
||||||
|
f"Expected >= 200 rows in {c_name_2}, got {stats2.get('row_count')}"
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
dn_db1.close()
|
||||||
|
dn_db2.close()
|
||||||
|
|
||||||
|
def test_drop_db_with_collections(
|
||||||
|
self,
|
||||||
|
upstream_uri,
|
||||||
|
upstream_token,
|
||||||
|
downstream_uri,
|
||||||
|
downstream_token,
|
||||||
|
upstream_client,
|
||||||
|
downstream_client,
|
||||||
|
sync_timeout,
|
||||||
|
):
|
||||||
|
"""Test drop DB with collection syncs to downstream (DB gone from downstream)."""
|
||||||
|
self._upstream_uri = upstream_uri
|
||||||
|
self._upstream_token = upstream_token
|
||||||
|
|
||||||
|
db_name = self.gen_unique_name("test_mdb_drop_db")
|
||||||
|
c_name = self.gen_unique_name("test_mdb_drop_col")
|
||||||
|
|
||||||
|
self.resources_to_cleanup.append(("collection_in_db", (db_name, c_name)))
|
||||||
|
self.resources_to_cleanup.append(("database", db_name))
|
||||||
|
|
||||||
|
# Initial cleanup
|
||||||
|
self.cleanup_database(upstream_client, db_name)
|
||||||
|
|
||||||
|
# Create database and collection
|
||||||
|
upstream_client.create_database(db_name)
|
||||||
|
up_db_client = MilvusClient(uri=upstream_uri, token=upstream_token, db_name=db_name)
|
||||||
|
schema = self.create_default_schema(up_db_client)
|
||||||
|
up_db_client.create_collection(collection_name=c_name, schema=schema, consistency_level="Strong")
|
||||||
|
up_db_client.insert(c_name, self.generate_test_data(50))
|
||||||
|
up_db_client.flush(c_name)
|
||||||
|
up_db_client.close()
|
||||||
|
|
||||||
|
# Wait for DB + collection to sync to downstream
|
||||||
|
def check_created():
|
||||||
|
try:
|
||||||
|
if db_name not in downstream_client.list_databases():
|
||||||
|
return False
|
||||||
|
dn = MilvusClient(uri=downstream_uri, token=downstream_token, db_name=db_name)
|
||||||
|
exists = dn.has_collection(c_name)
|
||||||
|
dn.close()
|
||||||
|
return exists
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Check DB+collection created: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_created, sync_timeout, f"create db {db_name} with collection")
|
||||||
|
|
||||||
|
# Drop collection then DB in upstream
|
||||||
|
up_db_client2 = MilvusClient(uri=upstream_uri, token=upstream_token, db_name=db_name)
|
||||||
|
up_db_client2.drop_collection(c_name)
|
||||||
|
up_db_client2.close()
|
||||||
|
upstream_client.drop_database(db_name)
|
||||||
|
|
||||||
|
assert db_name not in upstream_client.list_databases(), (
|
||||||
|
f"Database {db_name} still exists in upstream after drop"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Wait for DB to be gone on downstream
|
||||||
|
def check_db_dropped():
|
||||||
|
return db_name not in downstream_client.list_databases()
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_db_dropped, sync_timeout, f"drop database {db_name}")
|
||||||
|
|
||||||
|
def test_cross_db_operations(
|
||||||
|
self,
|
||||||
|
upstream_uri,
|
||||||
|
upstream_token,
|
||||||
|
downstream_uri,
|
||||||
|
downstream_token,
|
||||||
|
upstream_client,
|
||||||
|
downstream_client,
|
||||||
|
sync_timeout,
|
||||||
|
):
|
||||||
|
"""Test cross-DB operations: create DB, collection, insert, alter DB properties, create 2nd collection."""
|
||||||
|
self._upstream_uri = upstream_uri
|
||||||
|
self._upstream_token = upstream_token
|
||||||
|
|
||||||
|
db_name = self.gen_unique_name("test_mdb_cross_db")
|
||||||
|
c_name_1 = self.gen_unique_name("test_mdb_cross_col1")
|
||||||
|
c_name_2 = self.gen_unique_name("test_mdb_cross_col2")
|
||||||
|
|
||||||
|
self.resources_to_cleanup.append(("collection_in_db", (db_name, c_name_1)))
|
||||||
|
self.resources_to_cleanup.append(("collection_in_db", (db_name, c_name_2)))
|
||||||
|
self.resources_to_cleanup.append(("database", db_name))
|
||||||
|
|
||||||
|
# Initial cleanup
|
||||||
|
self.cleanup_database(upstream_client, db_name)
|
||||||
|
|
||||||
|
# Step 1: Create database
|
||||||
|
upstream_client.create_database(db_name)
|
||||||
|
|
||||||
|
# Step 2: Create DB-scoped client and 1st collection with insert
|
||||||
|
up_db_client = MilvusClient(uri=upstream_uri, token=upstream_token, db_name=db_name)
|
||||||
|
schema1 = self.create_default_schema(up_db_client)
|
||||||
|
up_db_client.create_collection(collection_name=c_name_1, schema=schema1, consistency_level="Strong")
|
||||||
|
up_db_client.insert(c_name_1, self.generate_test_data(100))
|
||||||
|
up_db_client.flush(c_name_1)
|
||||||
|
|
||||||
|
# Step 3: Alter DB properties
|
||||||
|
upstream_client.alter_database_properties(
|
||||||
|
db_name=db_name,
|
||||||
|
properties={"database.max.collections": 10},
|
||||||
|
)
|
||||||
|
|
||||||
|
# Step 4: Create 2nd collection in the same DB
|
||||||
|
schema2 = self.create_default_schema(up_db_client)
|
||||||
|
up_db_client.create_collection(collection_name=c_name_2, schema=schema2, consistency_level="Strong")
|
||||||
|
up_db_client.close()
|
||||||
|
|
||||||
|
# Wait for all operations to sync to downstream
|
||||||
|
|
||||||
|
# Check database exists on downstream
|
||||||
|
def check_db_exists():
|
||||||
|
return db_name in downstream_client.list_databases()
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_db_exists, sync_timeout, f"create database {db_name}")
|
||||||
|
|
||||||
|
# Check 1st collection exists on downstream
|
||||||
|
def check_col1():
|
||||||
|
try:
|
||||||
|
dn = MilvusClient(uri=downstream_uri, token=downstream_token, db_name=db_name)
|
||||||
|
exists = dn.has_collection(c_name_1)
|
||||||
|
dn.close()
|
||||||
|
return exists
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Check col1 sync failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_col1, sync_timeout, f"collection {c_name_1} in {db_name}")
|
||||||
|
|
||||||
|
# Check 2nd collection exists on downstream
|
||||||
|
def check_col2():
|
||||||
|
try:
|
||||||
|
dn = MilvusClient(uri=downstream_uri, token=downstream_token, db_name=db_name)
|
||||||
|
exists = dn.has_collection(c_name_2)
|
||||||
|
dn.close()
|
||||||
|
return exists
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Check col2 sync failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_col2, sync_timeout, f"collection {c_name_2} in {db_name}")
|
||||||
|
|
||||||
|
# Check DB properties synced
|
||||||
|
def check_db_props():
|
||||||
|
try:
|
||||||
|
if db_name not in downstream_client.list_databases():
|
||||||
|
return False
|
||||||
|
props = downstream_client.describe_database(db_name)
|
||||||
|
logger.info(f"Downstream database properties: {props}")
|
||||||
|
return str(props.get("database.max.collections", "")) == "10"
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Check DB properties sync failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_db_props, sync_timeout, f"DB properties sync for {db_name}")
|
||||||
@@ -0,0 +1,157 @@
|
|||||||
|
"""
|
||||||
|
CDC non-replication tests for resource group operations.
|
||||||
|
|
||||||
|
Resource groups and replica assignment are per-cluster state; by design
|
||||||
|
CDC does NOT propagate RG create/drop/update/transfer-replica to the
|
||||||
|
downstream cluster. These tests guard that invariant.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import time
|
||||||
|
|
||||||
|
from .base import TestCDCSyncBase, logger
|
||||||
|
|
||||||
|
|
||||||
|
class TestCDCSyncResourceGroup(TestCDCSyncBase):
|
||||||
|
"""Verify that resource group operations are NOT replicated by CDC."""
|
||||||
|
|
||||||
|
def setup_method(self):
|
||||||
|
"""Setup for each test method."""
|
||||||
|
self.resources_to_cleanup = []
|
||||||
|
|
||||||
|
def teardown_method(self):
|
||||||
|
"""Cleanup both upstream and downstream (downstream won't auto-sync)."""
|
||||||
|
upstream_client = getattr(self, "_upstream_client", None)
|
||||||
|
downstream_client = getattr(self, "_downstream_client", None)
|
||||||
|
|
||||||
|
for client in (upstream_client, downstream_client):
|
||||||
|
if not client:
|
||||||
|
continue
|
||||||
|
for resource_type, resource_name in self.resources_to_cleanup:
|
||||||
|
if resource_type == "resource_group":
|
||||||
|
self._cleanup_resource_group(client, resource_name)
|
||||||
|
elif resource_type == "collection":
|
||||||
|
self.cleanup_collection(client, resource_name)
|
||||||
|
|
||||||
|
def _cleanup_resource_group(self, client, rg_name):
|
||||||
|
"""Clean up resource group if exists (skipping default RG)."""
|
||||||
|
try:
|
||||||
|
existing = client.list_resource_groups()
|
||||||
|
if rg_name in existing:
|
||||||
|
logger.info(f"[CLEANUP] Cleaning up resource group: {rg_name}")
|
||||||
|
client.drop_resource_group(rg_name)
|
||||||
|
logger.info(f"[SUCCESS] Resource group {rg_name} cleaned up successfully")
|
||||||
|
else:
|
||||||
|
logger.debug(f"Resource group {rg_name} does not exist, skipping cleanup")
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"[FAILED] Failed to cleanup resource group {rg_name}: {e}")
|
||||||
|
|
||||||
|
def test_create_resource_group_not_replicated(self, upstream_client, downstream_client, sync_timeout):
|
||||||
|
"""Creating an RG on upstream must NOT create it on downstream."""
|
||||||
|
self._upstream_client = upstream_client
|
||||||
|
self._downstream_client = downstream_client
|
||||||
|
|
||||||
|
rg_name = self.gen_unique_name("test_rg_create")
|
||||||
|
self.resources_to_cleanup.append(("resource_group", rg_name))
|
||||||
|
|
||||||
|
self._cleanup_resource_group(upstream_client, rg_name)
|
||||||
|
self._cleanup_resource_group(downstream_client, rg_name)
|
||||||
|
|
||||||
|
upstream_client.create_resource_group(rg_name)
|
||||||
|
assert rg_name in upstream_client.list_resource_groups(), f"Resource group {rg_name} not created in upstream"
|
||||||
|
|
||||||
|
# Wait the full sync window; if CDC were going to leak the RG it would
|
||||||
|
# have shown up by now.
|
||||||
|
time.sleep(sync_timeout)
|
||||||
|
downstream_rgs = downstream_client.list_resource_groups()
|
||||||
|
assert rg_name not in downstream_rgs, (
|
||||||
|
f"Resource group {rg_name} unexpectedly appeared on downstream (RG ops must not replicate)"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_drop_resource_group_not_replicated(self, upstream_client, downstream_client, sync_timeout):
|
||||||
|
"""Dropping an RG on upstream must NOT drop it on downstream."""
|
||||||
|
self._upstream_client = upstream_client
|
||||||
|
self._downstream_client = downstream_client
|
||||||
|
|
||||||
|
rg_name = self.gen_unique_name("test_rg_drop")
|
||||||
|
self.resources_to_cleanup.append(("resource_group", rg_name))
|
||||||
|
|
||||||
|
self._cleanup_resource_group(upstream_client, rg_name)
|
||||||
|
self._cleanup_resource_group(downstream_client, rg_name)
|
||||||
|
|
||||||
|
# Create the RG on BOTH sides (independently, since create isn't replicated either).
|
||||||
|
upstream_client.create_resource_group(rg_name)
|
||||||
|
downstream_client.create_resource_group(rg_name)
|
||||||
|
assert rg_name in upstream_client.list_resource_groups()
|
||||||
|
assert rg_name in downstream_client.list_resource_groups()
|
||||||
|
|
||||||
|
# Drop only on upstream; downstream must retain its own copy.
|
||||||
|
upstream_client.drop_resource_group(rg_name)
|
||||||
|
assert rg_name not in upstream_client.list_resource_groups(), (
|
||||||
|
f"Resource group {rg_name} still exists in upstream after drop"
|
||||||
|
)
|
||||||
|
|
||||||
|
time.sleep(sync_timeout)
|
||||||
|
downstream_rgs = downstream_client.list_resource_groups()
|
||||||
|
assert rg_name in downstream_rgs, (
|
||||||
|
f"Resource group {rg_name} was dropped on downstream after upstream drop "
|
||||||
|
f"(RG ops must not replicate). downstream RGs: {downstream_rgs}"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_update_resource_group_not_replicated(self, upstream_client, downstream_client, sync_timeout):
|
||||||
|
"""Updating RG config on upstream must NOT reconfigure downstream's RG."""
|
||||||
|
self._upstream_client = upstream_client
|
||||||
|
self._downstream_client = downstream_client
|
||||||
|
|
||||||
|
rg_name = self.gen_unique_name("test_rg_update")
|
||||||
|
self.resources_to_cleanup.append(("resource_group", rg_name))
|
||||||
|
|
||||||
|
self._cleanup_resource_group(upstream_client, rg_name)
|
||||||
|
self._cleanup_resource_group(downstream_client, rg_name)
|
||||||
|
|
||||||
|
upstream_config = {"requests": {"node_num": 0}, "limits": {"node_num": 2}}
|
||||||
|
downstream_config = {"requests": {"node_num": 0}, "limits": {"node_num": 1}}
|
||||||
|
|
||||||
|
# Create the RG on both sides with different configs.
|
||||||
|
upstream_client.create_resource_group(rg_name, config=upstream_config)
|
||||||
|
downstream_client.create_resource_group(rg_name, config=downstream_config)
|
||||||
|
downstream_desc_before = downstream_client.describe_resource_group(rg_name)
|
||||||
|
logger.info(f"Downstream RG before upstream update: {downstream_desc_before}")
|
||||||
|
|
||||||
|
# Upstream has already been created with its config; nothing else to
|
||||||
|
# update since config drift itself would be the leak. Wait and confirm
|
||||||
|
# downstream config is unchanged. ResourceGroupInfo has no __eq__, so
|
||||||
|
# compare the str() form (includes config/limits/requests/nodes).
|
||||||
|
time.sleep(sync_timeout)
|
||||||
|
downstream_desc_after = downstream_client.describe_resource_group(rg_name)
|
||||||
|
logger.info(f"Downstream RG after upstream update: {downstream_desc_after}")
|
||||||
|
assert str(downstream_desc_after) == str(downstream_desc_before), (
|
||||||
|
f"Downstream RG {rg_name} was modified by upstream config (RG ops must not replicate). "
|
||||||
|
f"before={downstream_desc_before}, after={downstream_desc_after}"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_transfer_replica_not_replicated(self, upstream_client, downstream_client, sync_timeout):
|
||||||
|
"""Creating multiple RGs on upstream must NOT create any of them on downstream."""
|
||||||
|
self._upstream_client = upstream_client
|
||||||
|
self._downstream_client = downstream_client
|
||||||
|
|
||||||
|
rg_name_1 = self.gen_unique_name("test_rg_transfer_1")
|
||||||
|
rg_name_2 = self.gen_unique_name("test_rg_transfer_2")
|
||||||
|
self.resources_to_cleanup.append(("resource_group", rg_name_1))
|
||||||
|
self.resources_to_cleanup.append(("resource_group", rg_name_2))
|
||||||
|
|
||||||
|
self._cleanup_resource_group(upstream_client, rg_name_1)
|
||||||
|
self._cleanup_resource_group(upstream_client, rg_name_2)
|
||||||
|
self._cleanup_resource_group(downstream_client, rg_name_1)
|
||||||
|
self._cleanup_resource_group(downstream_client, rg_name_2)
|
||||||
|
|
||||||
|
upstream_client.create_resource_group(rg_name_1)
|
||||||
|
upstream_client.create_resource_group(rg_name_2)
|
||||||
|
upstream_rgs = upstream_client.list_resource_groups()
|
||||||
|
assert rg_name_1 in upstream_rgs
|
||||||
|
assert rg_name_2 in upstream_rgs
|
||||||
|
|
||||||
|
time.sleep(sync_timeout)
|
||||||
|
downstream_rgs = downstream_client.list_resource_groups()
|
||||||
|
assert rg_name_1 not in downstream_rgs and rg_name_2 not in downstream_rgs, (
|
||||||
|
f"Upstream RGs leaked to downstream (RG ops must not replicate). downstream RGs: {downstream_rgs}"
|
||||||
|
)
|
||||||
@@ -0,0 +1,690 @@
|
|||||||
|
"""
|
||||||
|
CDC sync tests for advanced schema features (dynamic fields, nullable, default values,
|
||||||
|
partition keys, clustering keys, and combinations thereof).
|
||||||
|
"""
|
||||||
|
|
||||||
|
import random
|
||||||
|
import time
|
||||||
|
|
||||||
|
from pymilvus import DataType
|
||||||
|
|
||||||
|
from .base import TestCDCSyncBase, logger
|
||||||
|
|
||||||
|
|
||||||
|
class TestCDCSyncSchemaFeatures(TestCDCSyncBase):
|
||||||
|
"""Test CDC sync for advanced schema features."""
|
||||||
|
|
||||||
|
def setup_method(self):
|
||||||
|
"""Setup for each test method."""
|
||||||
|
self.resources_to_cleanup = []
|
||||||
|
|
||||||
|
def teardown_method(self):
|
||||||
|
"""Cleanup after each test method - only cleanup upstream, downstream will sync."""
|
||||||
|
upstream_client = getattr(self, "_upstream_client", None)
|
||||||
|
|
||||||
|
if upstream_client:
|
||||||
|
for resource_type, resource_name in self.resources_to_cleanup:
|
||||||
|
if resource_type == "collection":
|
||||||
|
self.cleanup_collection(upstream_client, resource_name)
|
||||||
|
|
||||||
|
time.sleep(1) # Allow cleanup to sync to downstream
|
||||||
|
|
||||||
|
def test_dynamic_schema_sync(self, upstream_client, downstream_client, sync_timeout):
|
||||||
|
"""Test that dynamic schema fields are correctly replicated via CDC."""
|
||||||
|
start_time = time.time()
|
||||||
|
collection_name = self.gen_unique_name("test_dynamic_schema", max_length=50)
|
||||||
|
|
||||||
|
self.log_test_start("test_dynamic_schema_sync", "DYNAMIC_SCHEMA", collection_name)
|
||||||
|
self._upstream_client = upstream_client
|
||||||
|
self.resources_to_cleanup.append(("collection", collection_name))
|
||||||
|
|
||||||
|
try:
|
||||||
|
self.cleanup_collection(upstream_client, collection_name)
|
||||||
|
|
||||||
|
# Create dynamic schema collection
|
||||||
|
self.log_operation("CREATE_COLLECTION", "collection", collection_name, "upstream")
|
||||||
|
upstream_client.create_collection(
|
||||||
|
collection_name=collection_name,
|
||||||
|
schema=self.create_dynamic_schema(upstream_client),
|
||||||
|
)
|
||||||
|
|
||||||
|
# Create HNSW index and load
|
||||||
|
index_params = upstream_client.prepare_index_params()
|
||||||
|
index_params.add_index(
|
||||||
|
field_name="float_vector",
|
||||||
|
index_type="HNSW",
|
||||||
|
metric_type="L2",
|
||||||
|
params={"M": 8, "efConstruction": 64},
|
||||||
|
)
|
||||||
|
upstream_client.create_index(collection_name, index_params)
|
||||||
|
upstream_client.load_collection(collection_name)
|
||||||
|
|
||||||
|
# Wait for collection creation to sync
|
||||||
|
def check_create():
|
||||||
|
return downstream_client.has_collection(collection_name)
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {collection_name}")
|
||||||
|
|
||||||
|
# Insert 200 rows with extra dynamic fields
|
||||||
|
extra_fields = {"extra_int": int, "extra_str": str, "extra_float": float}
|
||||||
|
test_data = self.generate_dynamic_data(200, extra_fields=extra_fields)
|
||||||
|
self.log_data_operation("INSERT", collection_name, len(test_data), "- dynamic schema data")
|
||||||
|
upstream_client.insert(collection_name, test_data)
|
||||||
|
upstream_client.flush(collection_name)
|
||||||
|
|
||||||
|
# Query extra_int > 0 on upstream
|
||||||
|
self.log_sync_verification("DYNAMIC_SCHEMA", collection_name, "extra_int > 0 count matches downstream")
|
||||||
|
upstream_result = upstream_client.query(
|
||||||
|
collection_name=collection_name,
|
||||||
|
filter="extra_int > 0",
|
||||||
|
output_fields=["count(*)"],
|
||||||
|
)
|
||||||
|
upstream_count = upstream_result[0]["count(*)"] if upstream_result else 0
|
||||||
|
logger.info(f"[UPSTREAM] extra_int > 0 count: {upstream_count}")
|
||||||
|
|
||||||
|
# Wait for sync and verify downstream count matches
|
||||||
|
def check_data():
|
||||||
|
try:
|
||||||
|
down_result = downstream_client.query(
|
||||||
|
collection_name=collection_name,
|
||||||
|
filter="extra_int > 0",
|
||||||
|
output_fields=["count(*)"],
|
||||||
|
)
|
||||||
|
down_count = down_result[0]["count(*)"] if down_result else 0
|
||||||
|
logger.info(f"[SYNC_PROGRESS] downstream extra_int > 0 count: {down_count}/{upstream_count}")
|
||||||
|
return down_count == upstream_count
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Dynamic schema sync check failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
sync_success = self.wait_for_sync(check_data, sync_timeout, f"dynamic schema data sync {collection_name}")
|
||||||
|
assert sync_success, f"Dynamic schema data failed to sync to downstream for {collection_name}"
|
||||||
|
|
||||||
|
# Verify data sampling for extra fields
|
||||||
|
match, mismatch, details = self.verify_data_sampling(
|
||||||
|
upstream_client,
|
||||||
|
downstream_client,
|
||||||
|
collection_name,
|
||||||
|
sample_ratio=0.1,
|
||||||
|
output_fields=["id", "varchar_field", "extra_int", "extra_str", "extra_float"],
|
||||||
|
)
|
||||||
|
logger.info(f"[VERIFY] Dynamic schema sampling: match={match}, mismatch={mismatch}")
|
||||||
|
assert mismatch == 0, f"Dynamic field data mismatch detected: {details}"
|
||||||
|
|
||||||
|
duration = time.time() - start_time
|
||||||
|
self.log_test_end("test_dynamic_schema_sync", True, duration)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
duration = time.time() - start_time
|
||||||
|
logger.error(f"[ERROR] test_dynamic_schema_sync failed: {e}")
|
||||||
|
self.log_test_end("test_dynamic_schema_sync", False, duration)
|
||||||
|
raise
|
||||||
|
|
||||||
|
def test_nullable_fields_sync(self, upstream_client, downstream_client, sync_timeout):
|
||||||
|
"""Test that nullable field values (including NULLs) are correctly replicated via CDC."""
|
||||||
|
start_time = time.time()
|
||||||
|
collection_name = self.gen_unique_name("test_nullable_flds", max_length=50)
|
||||||
|
|
||||||
|
self.log_test_start("test_nullable_fields_sync", "NULLABLE_FIELDS", collection_name)
|
||||||
|
self._upstream_client = upstream_client
|
||||||
|
self.resources_to_cleanup.append(("collection", collection_name))
|
||||||
|
|
||||||
|
try:
|
||||||
|
self.cleanup_collection(upstream_client, collection_name)
|
||||||
|
|
||||||
|
# Create nullable schema collection
|
||||||
|
self.log_operation("CREATE_COLLECTION", "collection", collection_name, "upstream")
|
||||||
|
upstream_client.create_collection(
|
||||||
|
collection_name=collection_name,
|
||||||
|
schema=self.create_nullable_schema(upstream_client),
|
||||||
|
)
|
||||||
|
|
||||||
|
# Create index and load
|
||||||
|
index_params = upstream_client.prepare_index_params()
|
||||||
|
index_params.add_index(
|
||||||
|
field_name="float_vector",
|
||||||
|
index_type="HNSW",
|
||||||
|
metric_type="L2",
|
||||||
|
params={"M": 8, "efConstruction": 64},
|
||||||
|
)
|
||||||
|
upstream_client.create_index(collection_name, index_params)
|
||||||
|
upstream_client.load_collection(collection_name)
|
||||||
|
|
||||||
|
# Wait for collection creation to sync
|
||||||
|
def check_create():
|
||||||
|
return downstream_client.has_collection(collection_name)
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {collection_name}")
|
||||||
|
|
||||||
|
# Insert 200 rows with null_ratio=0.3
|
||||||
|
test_data = self.generate_nullable_data(200, null_ratio=0.3)
|
||||||
|
self.log_data_operation("INSERT", collection_name, len(test_data), "- nullable data null_ratio=0.3")
|
||||||
|
upstream_client.insert(collection_name, test_data)
|
||||||
|
upstream_client.flush(collection_name)
|
||||||
|
|
||||||
|
# Count nulls and not-nulls for nullable_int64 on upstream
|
||||||
|
null_result = upstream_client.query(
|
||||||
|
collection_name=collection_name,
|
||||||
|
filter="nullable_int64 is null",
|
||||||
|
output_fields=["count(*)"],
|
||||||
|
)
|
||||||
|
not_null_result = upstream_client.query(
|
||||||
|
collection_name=collection_name,
|
||||||
|
filter="nullable_int64 is not null",
|
||||||
|
output_fields=["count(*)"],
|
||||||
|
)
|
||||||
|
upstream_null_count = null_result[0]["count(*)"] if null_result else 0
|
||||||
|
upstream_not_null_count = not_null_result[0]["count(*)"] if not_null_result else 0
|
||||||
|
logger.info(f"[UPSTREAM] nullable_int64 null={upstream_null_count}, not_null={upstream_not_null_count}")
|
||||||
|
|
||||||
|
self.log_sync_verification("NULLABLE_FIELDS", collection_name, "null counts match downstream")
|
||||||
|
|
||||||
|
# Wait for sync and verify null/not-null counts match on downstream
|
||||||
|
def check_null_counts():
|
||||||
|
try:
|
||||||
|
d_null = downstream_client.query(
|
||||||
|
collection_name=collection_name,
|
||||||
|
filter="nullable_int64 is null",
|
||||||
|
output_fields=["count(*)"],
|
||||||
|
)
|
||||||
|
d_not_null = downstream_client.query(
|
||||||
|
collection_name=collection_name,
|
||||||
|
filter="nullable_int64 is not null",
|
||||||
|
output_fields=["count(*)"],
|
||||||
|
)
|
||||||
|
d_null_count = d_null[0]["count(*)"] if d_null else 0
|
||||||
|
d_not_null_count = d_not_null[0]["count(*)"] if d_not_null else 0
|
||||||
|
logger.info(
|
||||||
|
f"[SYNC_PROGRESS] downstream nullable_int64 null={d_null_count}/{upstream_null_count}, "
|
||||||
|
f"not_null={d_not_null_count}/{upstream_not_null_count}"
|
||||||
|
)
|
||||||
|
return d_null_count == upstream_null_count and d_not_null_count == upstream_not_null_count
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Nullable sync check failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
sync_success = self.wait_for_sync(
|
||||||
|
check_null_counts, sync_timeout, f"nullable fields sync {collection_name}"
|
||||||
|
)
|
||||||
|
assert sync_success, f"Nullable field counts failed to sync to downstream for {collection_name}"
|
||||||
|
|
||||||
|
duration = time.time() - start_time
|
||||||
|
self.log_test_end("test_nullable_fields_sync", True, duration)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
duration = time.time() - start_time
|
||||||
|
logger.error(f"[ERROR] test_nullable_fields_sync failed: {e}")
|
||||||
|
self.log_test_end("test_nullable_fields_sync", False, duration)
|
||||||
|
raise
|
||||||
|
|
||||||
|
def test_default_values_sync(self, upstream_client, downstream_client, sync_timeout):
|
||||||
|
"""Test that default field values are applied and replicated correctly via CDC."""
|
||||||
|
start_time = time.time()
|
||||||
|
collection_name = self.gen_unique_name("test_default_vals", max_length=50)
|
||||||
|
|
||||||
|
self.log_test_start("test_default_values_sync", "DEFAULT_VALUES", collection_name)
|
||||||
|
self._upstream_client = upstream_client
|
||||||
|
self.resources_to_cleanup.append(("collection", collection_name))
|
||||||
|
|
||||||
|
try:
|
||||||
|
self.cleanup_collection(upstream_client, collection_name)
|
||||||
|
|
||||||
|
# Create default values schema collection
|
||||||
|
self.log_operation("CREATE_COLLECTION", "collection", collection_name, "upstream")
|
||||||
|
upstream_client.create_collection(
|
||||||
|
collection_name=collection_name,
|
||||||
|
schema=self.create_default_values_schema(upstream_client),
|
||||||
|
)
|
||||||
|
|
||||||
|
# Create index and load
|
||||||
|
index_params = upstream_client.prepare_index_params()
|
||||||
|
index_params.add_index(
|
||||||
|
field_name="float_vector",
|
||||||
|
index_type="HNSW",
|
||||||
|
metric_type="L2",
|
||||||
|
params={"M": 8, "efConstruction": 64},
|
||||||
|
)
|
||||||
|
upstream_client.create_index(collection_name, index_params)
|
||||||
|
upstream_client.load_collection(collection_name)
|
||||||
|
|
||||||
|
# Wait for collection creation to sync
|
||||||
|
def check_create():
|
||||||
|
return downstream_client.has_collection(collection_name)
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {collection_name}")
|
||||||
|
|
||||||
|
# Insert 100 rows providing ONLY float_vector — default fields are omitted
|
||||||
|
test_data = [{"float_vector": [random.random() for _ in range(128)]} for _ in range(100)]
|
||||||
|
self.log_data_operation("INSERT", collection_name, len(test_data), "- only float_vector provided")
|
||||||
|
upstream_client.insert(collection_name, test_data)
|
||||||
|
upstream_client.flush(collection_name)
|
||||||
|
|
||||||
|
# Query default_varchar == "default" on upstream — should be 100
|
||||||
|
upstream_result = upstream_client.query(
|
||||||
|
collection_name=collection_name,
|
||||||
|
filter='default_varchar == "default"',
|
||||||
|
output_fields=["count(*)"],
|
||||||
|
)
|
||||||
|
upstream_count = upstream_result[0]["count(*)"] if upstream_result else 0
|
||||||
|
logger.info(f'[UPSTREAM] default_varchar == "default" count: {upstream_count}')
|
||||||
|
assert upstream_count == 100, f"Expected 100 rows with default varchar on upstream, got {upstream_count}"
|
||||||
|
|
||||||
|
self.log_sync_verification(
|
||||||
|
"DEFAULT_VALUES", collection_name, 'default_varchar == "default" count=100 on downstream'
|
||||||
|
)
|
||||||
|
|
||||||
|
# Wait for sync and verify same count on downstream
|
||||||
|
def check_defaults():
|
||||||
|
try:
|
||||||
|
down_result = downstream_client.query(
|
||||||
|
collection_name=collection_name,
|
||||||
|
filter='default_varchar == "default"',
|
||||||
|
output_fields=["count(*)"],
|
||||||
|
)
|
||||||
|
down_count = down_result[0]["count(*)"] if down_result else 0
|
||||||
|
logger.info(f"[SYNC_PROGRESS] downstream default_varchar count: {down_count}/100")
|
||||||
|
return down_count == 100
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Default values sync check failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
sync_success = self.wait_for_sync(check_defaults, sync_timeout, f"default values sync {collection_name}")
|
||||||
|
assert sync_success, f"Default value data failed to sync to downstream for {collection_name}"
|
||||||
|
|
||||||
|
duration = time.time() - start_time
|
||||||
|
self.log_test_end("test_default_values_sync", True, duration)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
duration = time.time() - start_time
|
||||||
|
logger.error(f"[ERROR] test_default_values_sync failed: {e}")
|
||||||
|
self.log_test_end("test_default_values_sync", False, duration)
|
||||||
|
raise
|
||||||
|
|
||||||
|
def test_partition_key_sync(self, upstream_client, downstream_client, sync_timeout):
|
||||||
|
"""Test that partition key schema and data are correctly replicated via CDC."""
|
||||||
|
start_time = time.time()
|
||||||
|
collection_name = self.gen_unique_name("test_part_key", max_length=50)
|
||||||
|
|
||||||
|
self.log_test_start("test_partition_key_sync", "PARTITION_KEY", collection_name)
|
||||||
|
self._upstream_client = upstream_client
|
||||||
|
self.resources_to_cleanup.append(("collection", collection_name))
|
||||||
|
|
||||||
|
try:
|
||||||
|
self.cleanup_collection(upstream_client, collection_name)
|
||||||
|
|
||||||
|
# Create partition key schema (VarChar key)
|
||||||
|
self.log_operation("CREATE_COLLECTION", "collection", collection_name, "upstream")
|
||||||
|
upstream_client.create_collection(
|
||||||
|
collection_name=collection_name,
|
||||||
|
schema=self.create_partition_key_schema(upstream_client, key_type="VarChar"),
|
||||||
|
)
|
||||||
|
|
||||||
|
# Create index and load
|
||||||
|
index_params = upstream_client.prepare_index_params()
|
||||||
|
index_params.add_index(
|
||||||
|
field_name="float_vector",
|
||||||
|
index_type="HNSW",
|
||||||
|
metric_type="L2",
|
||||||
|
params={"M": 8, "efConstruction": 64},
|
||||||
|
)
|
||||||
|
upstream_client.create_index(collection_name, index_params)
|
||||||
|
upstream_client.load_collection(collection_name)
|
||||||
|
|
||||||
|
# Wait for collection creation to sync
|
||||||
|
def check_create():
|
||||||
|
return downstream_client.has_collection(collection_name)
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {collection_name}")
|
||||||
|
|
||||||
|
# Insert 500 rows with random categories as partition key values
|
||||||
|
categories = ["cat_A", "cat_B", "cat_C", "cat_D", "cat_E"]
|
||||||
|
test_data = [
|
||||||
|
{
|
||||||
|
"float_vector": [random.random() for _ in range(128)],
|
||||||
|
"partition_key_field": random.choice(categories),
|
||||||
|
"data_field": f"data_{i}_{random.randint(1000, 9999)}",
|
||||||
|
}
|
||||||
|
for i in range(500)
|
||||||
|
]
|
||||||
|
self.log_data_operation("INSERT", collection_name, len(test_data), "- partition key data")
|
||||||
|
upstream_client.insert(collection_name, test_data)
|
||||||
|
upstream_client.flush(collection_name)
|
||||||
|
|
||||||
|
self.log_sync_verification(
|
||||||
|
"PARTITION_KEY", collection_name, "count=500 and partition_key field on downstream"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Wait for sync and verify count=500 on downstream
|
||||||
|
def check_data():
|
||||||
|
try:
|
||||||
|
result = downstream_client.query(
|
||||||
|
collection_name=collection_name,
|
||||||
|
filter="",
|
||||||
|
output_fields=["count(*)"],
|
||||||
|
)
|
||||||
|
count = result[0]["count(*)"] if result else 0
|
||||||
|
logger.info(f"[SYNC_PROGRESS] downstream count: {count}/500")
|
||||||
|
return count >= 500
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Partition key sync check failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
sync_success = self.wait_for_sync(check_data, sync_timeout, f"partition key data sync {collection_name}")
|
||||||
|
assert sync_success, f"Partition key data failed to sync to downstream for {collection_name}"
|
||||||
|
|
||||||
|
# Verify partition_key_field is present in downstream describe_collection
|
||||||
|
downstream_info = downstream_client.describe_collection(collection_name)
|
||||||
|
downstream_fields = [f["name"] for f in downstream_info.get("fields", [])]
|
||||||
|
logger.info(f"[VERIFY] Downstream fields: {downstream_fields}")
|
||||||
|
assert "partition_key_field" in downstream_fields, (
|
||||||
|
f"partition_key_field not found in downstream collection schema: {downstream_fields}"
|
||||||
|
)
|
||||||
|
|
||||||
|
duration = time.time() - start_time
|
||||||
|
self.log_test_end("test_partition_key_sync", True, duration)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
duration = time.time() - start_time
|
||||||
|
logger.error(f"[ERROR] test_partition_key_sync failed: {e}")
|
||||||
|
self.log_test_end("test_partition_key_sync", False, duration)
|
||||||
|
raise
|
||||||
|
|
||||||
|
def test_clustering_key_sync(self, upstream_client, downstream_client, sync_timeout):
|
||||||
|
"""Test that clustering key schema and data are correctly replicated via CDC."""
|
||||||
|
start_time = time.time()
|
||||||
|
collection_name = self.gen_unique_name("test_cluster_key", max_length=50)
|
||||||
|
|
||||||
|
self.log_test_start("test_clustering_key_sync", "CLUSTERING_KEY", collection_name)
|
||||||
|
self._upstream_client = upstream_client
|
||||||
|
self.resources_to_cleanup.append(("collection", collection_name))
|
||||||
|
|
||||||
|
try:
|
||||||
|
self.cleanup_collection(upstream_client, collection_name)
|
||||||
|
|
||||||
|
# Create clustering key schema
|
||||||
|
self.log_operation("CREATE_COLLECTION", "collection", collection_name, "upstream")
|
||||||
|
upstream_client.create_collection(
|
||||||
|
collection_name=collection_name,
|
||||||
|
schema=self.create_clustering_key_schema(upstream_client),
|
||||||
|
)
|
||||||
|
|
||||||
|
# Create index and load
|
||||||
|
index_params = upstream_client.prepare_index_params()
|
||||||
|
index_params.add_index(
|
||||||
|
field_name="float_vector",
|
||||||
|
index_type="HNSW",
|
||||||
|
metric_type="L2",
|
||||||
|
params={"M": 8, "efConstruction": 64},
|
||||||
|
)
|
||||||
|
upstream_client.create_index(collection_name, index_params)
|
||||||
|
upstream_client.load_collection(collection_name)
|
||||||
|
|
||||||
|
# Wait for collection creation to sync
|
||||||
|
def check_create():
|
||||||
|
return downstream_client.has_collection(collection_name)
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {collection_name}")
|
||||||
|
|
||||||
|
# Insert 300 rows
|
||||||
|
test_data = [
|
||||||
|
{
|
||||||
|
"float_vector": [random.random() for _ in range(128)],
|
||||||
|
"clustering_key_field": random.randint(0, 1000),
|
||||||
|
"data_field": f"data_{i}_{random.randint(1000, 9999)}",
|
||||||
|
}
|
||||||
|
for i in range(300)
|
||||||
|
]
|
||||||
|
self.log_data_operation("INSERT", collection_name, len(test_data), "- clustering key data")
|
||||||
|
upstream_client.insert(collection_name, test_data)
|
||||||
|
upstream_client.flush(collection_name)
|
||||||
|
|
||||||
|
self.log_sync_verification(
|
||||||
|
"CLUSTERING_KEY", collection_name, "count=300 and clustering_key field on downstream"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Wait for sync and verify count=300 on downstream
|
||||||
|
def check_data():
|
||||||
|
try:
|
||||||
|
result = downstream_client.query(
|
||||||
|
collection_name=collection_name,
|
||||||
|
filter="",
|
||||||
|
output_fields=["count(*)"],
|
||||||
|
)
|
||||||
|
count = result[0]["count(*)"] if result else 0
|
||||||
|
logger.info(f"[SYNC_PROGRESS] downstream count: {count}/300")
|
||||||
|
return count >= 300
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Clustering key sync check failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
sync_success = self.wait_for_sync(check_data, sync_timeout, f"clustering key data sync {collection_name}")
|
||||||
|
assert sync_success, f"Clustering key data failed to sync to downstream for {collection_name}"
|
||||||
|
|
||||||
|
# Verify clustering_key_field is present in downstream describe_collection
|
||||||
|
downstream_info = downstream_client.describe_collection(collection_name)
|
||||||
|
downstream_fields = [f["name"] for f in downstream_info.get("fields", [])]
|
||||||
|
logger.info(f"[VERIFY] Downstream fields: {downstream_fields}")
|
||||||
|
assert "clustering_key_field" in downstream_fields, (
|
||||||
|
f"clustering_key_field not found in downstream collection schema: {downstream_fields}"
|
||||||
|
)
|
||||||
|
|
||||||
|
duration = time.time() - start_time
|
||||||
|
self.log_test_end("test_clustering_key_sync", True, duration)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
duration = time.time() - start_time
|
||||||
|
logger.error(f"[ERROR] test_clustering_key_sync failed: {e}")
|
||||||
|
self.log_test_end("test_clustering_key_sync", False, duration)
|
||||||
|
raise
|
||||||
|
|
||||||
|
def test_nullable_with_defaults(self, upstream_client, downstream_client, sync_timeout):
|
||||||
|
"""Test that nullable fields combined with default values sync correctly via CDC.
|
||||||
|
|
||||||
|
Inserts 100 rows in three patterns:
|
||||||
|
i%3==0: explicit values provided for both fields
|
||||||
|
i%3==1: None values (explicit null) provided for both fields
|
||||||
|
i%3==2: fields omitted entirely (uses default / null)
|
||||||
|
"""
|
||||||
|
start_time = time.time()
|
||||||
|
collection_name = self.gen_unique_name("test_null_default", max_length=50)
|
||||||
|
|
||||||
|
self.log_test_start("test_nullable_with_defaults", "NULLABLE_WITH_DEFAULTS", collection_name)
|
||||||
|
self._upstream_client = upstream_client
|
||||||
|
self.resources_to_cleanup.append(("collection", collection_name))
|
||||||
|
|
||||||
|
try:
|
||||||
|
self.cleanup_collection(upstream_client, collection_name)
|
||||||
|
|
||||||
|
# Build custom schema: nullable_with_default (INT64, nullable, default=42)
|
||||||
|
# nullable_no_default (VARCHAR, nullable)
|
||||||
|
schema = upstream_client.create_schema()
|
||||||
|
schema.add_field("id", DataType.INT64, is_primary=True, auto_id=True)
|
||||||
|
schema.add_field("float_vector", DataType.FLOAT_VECTOR, dim=128)
|
||||||
|
schema.add_field(
|
||||||
|
"nullable_with_default",
|
||||||
|
DataType.INT64,
|
||||||
|
nullable=True,
|
||||||
|
default_value=42,
|
||||||
|
)
|
||||||
|
schema.add_field(
|
||||||
|
"nullable_no_default",
|
||||||
|
DataType.VARCHAR,
|
||||||
|
max_length=256,
|
||||||
|
nullable=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
self.log_operation("CREATE_COLLECTION", "collection", collection_name, "upstream")
|
||||||
|
upstream_client.create_collection(
|
||||||
|
collection_name=collection_name,
|
||||||
|
schema=schema,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Create index and load
|
||||||
|
index_params = upstream_client.prepare_index_params()
|
||||||
|
index_params.add_index(
|
||||||
|
field_name="float_vector",
|
||||||
|
index_type="HNSW",
|
||||||
|
metric_type="L2",
|
||||||
|
params={"M": 8, "efConstruction": 64},
|
||||||
|
)
|
||||||
|
upstream_client.create_index(collection_name, index_params)
|
||||||
|
upstream_client.load_collection(collection_name)
|
||||||
|
|
||||||
|
# Wait for collection creation to sync
|
||||||
|
def check_create():
|
||||||
|
return downstream_client.has_collection(collection_name)
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {collection_name}")
|
||||||
|
|
||||||
|
# Insert 100 rows across 3 patterns
|
||||||
|
test_data = []
|
||||||
|
for i in range(100):
|
||||||
|
record = {"float_vector": [random.random() for _ in range(128)]}
|
||||||
|
if i % 3 == 0:
|
||||||
|
# Explicit values
|
||||||
|
record["nullable_with_default"] = random.randint(100, 999)
|
||||||
|
record["nullable_no_default"] = f"explicit_{i}"
|
||||||
|
elif i % 3 == 1:
|
||||||
|
# Explicit None (null)
|
||||||
|
record["nullable_with_default"] = None
|
||||||
|
record["nullable_no_default"] = None
|
||||||
|
# i%3==2: omit both fields — server applies default/null
|
||||||
|
test_data.append(record)
|
||||||
|
|
||||||
|
self.log_data_operation("INSERT", collection_name, len(test_data), "- nullable+default mixed pattern")
|
||||||
|
upstream_client.insert(collection_name, test_data)
|
||||||
|
upstream_client.flush(collection_name)
|
||||||
|
|
||||||
|
self.log_sync_verification("NULLABLE_WITH_DEFAULTS", collection_name, "total count=100 on downstream")
|
||||||
|
|
||||||
|
# Wait for sync and verify total count on downstream
|
||||||
|
def check_data():
|
||||||
|
try:
|
||||||
|
result = downstream_client.query(
|
||||||
|
collection_name=collection_name,
|
||||||
|
filter="",
|
||||||
|
output_fields=["count(*)"],
|
||||||
|
)
|
||||||
|
count = result[0]["count(*)"] if result else 0
|
||||||
|
logger.info(f"[SYNC_PROGRESS] downstream count: {count}/100")
|
||||||
|
return count >= 100
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Nullable+defaults sync check failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
sync_success = self.wait_for_sync(check_data, sync_timeout, f"nullable+defaults sync {collection_name}")
|
||||||
|
assert sync_success, f"Nullable-with-defaults data failed to sync to downstream for {collection_name}"
|
||||||
|
|
||||||
|
duration = time.time() - start_time
|
||||||
|
self.log_test_end("test_nullable_with_defaults", True, duration)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
duration = time.time() - start_time
|
||||||
|
logger.error(f"[ERROR] test_nullable_with_defaults failed: {e}")
|
||||||
|
self.log_test_end("test_nullable_with_defaults", False, duration)
|
||||||
|
raise
|
||||||
|
|
||||||
|
def test_dynamic_with_partition_key(self, upstream_client, downstream_client, sync_timeout):
|
||||||
|
"""Test that dynamic fields combined with a partition key schema sync correctly via CDC."""
|
||||||
|
start_time = time.time()
|
||||||
|
collection_name = self.gen_unique_name("test_dyn_part_key", max_length=50)
|
||||||
|
|
||||||
|
self.log_test_start("test_dynamic_with_partition_key", "DYNAMIC_WITH_PARTITION_KEY", collection_name)
|
||||||
|
self._upstream_client = upstream_client
|
||||||
|
self.resources_to_cleanup.append(("collection", collection_name))
|
||||||
|
|
||||||
|
try:
|
||||||
|
self.cleanup_collection(upstream_client, collection_name)
|
||||||
|
|
||||||
|
# Build schema: enable_dynamic_field=True + VARCHAR partition key
|
||||||
|
schema = upstream_client.create_schema(enable_dynamic_field=True)
|
||||||
|
schema.add_field("id", DataType.INT64, is_primary=True, auto_id=True)
|
||||||
|
schema.add_field("float_vector", DataType.FLOAT_VECTOR, dim=128)
|
||||||
|
schema.add_field(
|
||||||
|
"pk_field",
|
||||||
|
DataType.VARCHAR,
|
||||||
|
max_length=64,
|
||||||
|
is_partition_key=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
self.log_operation("CREATE_COLLECTION", "collection", collection_name, "upstream")
|
||||||
|
upstream_client.create_collection(
|
||||||
|
collection_name=collection_name,
|
||||||
|
schema=schema,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Create index and load
|
||||||
|
index_params = upstream_client.prepare_index_params()
|
||||||
|
index_params.add_index(
|
||||||
|
field_name="float_vector",
|
||||||
|
index_type="HNSW",
|
||||||
|
metric_type="L2",
|
||||||
|
params={"M": 8, "efConstruction": 64},
|
||||||
|
)
|
||||||
|
upstream_client.create_index(collection_name, index_params)
|
||||||
|
upstream_client.load_collection(collection_name)
|
||||||
|
|
||||||
|
# Wait for collection creation to sync
|
||||||
|
def check_create():
|
||||||
|
return downstream_client.has_collection(collection_name)
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {collection_name}")
|
||||||
|
|
||||||
|
# Insert 300 rows with dynamic fields + partition key values
|
||||||
|
partitions = ["region_A", "region_B", "region_C", "region_D"]
|
||||||
|
test_data = [
|
||||||
|
{
|
||||||
|
"float_vector": [random.random() for _ in range(128)],
|
||||||
|
"pk_field": random.choice(partitions),
|
||||||
|
# dynamic fields
|
||||||
|
"dynamic_num": random.randint(1, 10000),
|
||||||
|
"dynamic_tag": f"tag_{i % 10}",
|
||||||
|
"dynamic_score": random.uniform(0.0, 100.0),
|
||||||
|
}
|
||||||
|
for i in range(300)
|
||||||
|
]
|
||||||
|
self.log_data_operation("INSERT", collection_name, len(test_data), "- dynamic+partition_key data")
|
||||||
|
upstream_client.insert(collection_name, test_data)
|
||||||
|
upstream_client.flush(collection_name)
|
||||||
|
|
||||||
|
# Query dynamic_num > 0 on upstream
|
||||||
|
upstream_result = upstream_client.query(
|
||||||
|
collection_name=collection_name,
|
||||||
|
filter="dynamic_num > 0",
|
||||||
|
output_fields=["count(*)"],
|
||||||
|
)
|
||||||
|
upstream_count = upstream_result[0]["count(*)"] if upstream_result else 0
|
||||||
|
logger.info(f"[UPSTREAM] dynamic_num > 0 count: {upstream_count}")
|
||||||
|
|
||||||
|
self.log_sync_verification(
|
||||||
|
"DYNAMIC_WITH_PARTITION_KEY",
|
||||||
|
collection_name,
|
||||||
|
f"dynamic_num > 0 count={upstream_count} on downstream",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Wait for sync and verify the count matches on downstream
|
||||||
|
def check_data():
|
||||||
|
try:
|
||||||
|
down_result = downstream_client.query(
|
||||||
|
collection_name=collection_name,
|
||||||
|
filter="dynamic_num > 0",
|
||||||
|
output_fields=["count(*)"],
|
||||||
|
)
|
||||||
|
down_count = down_result[0]["count(*)"] if down_result else 0
|
||||||
|
logger.info(f"[SYNC_PROGRESS] downstream dynamic_num > 0 count: {down_count}/{upstream_count}")
|
||||||
|
return down_count == upstream_count
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Dynamic+partition_key sync check failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
sync_success = self.wait_for_sync(check_data, sync_timeout, f"dynamic+partition_key sync {collection_name}")
|
||||||
|
assert sync_success, f"Dynamic+partition_key data failed to sync to downstream for {collection_name}"
|
||||||
|
|
||||||
|
duration = time.time() - start_time
|
||||||
|
self.log_test_end("test_dynamic_with_partition_key", True, duration)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
duration = time.time() - start_time
|
||||||
|
logger.error(f"[ERROR] test_dynamic_with_partition_key failed: {e}")
|
||||||
|
self.log_test_end("test_dynamic_with_partition_key", False, duration)
|
||||||
|
raise
|
||||||
@@ -0,0 +1,570 @@
|
|||||||
|
"""
|
||||||
|
CDC sync tests for search and query result verification across vector types.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import random
|
||||||
|
import time
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from pymilvus import AnnSearchRequest, DataType, RRFRanker
|
||||||
|
|
||||||
|
from .base import TestCDCSyncBase, logger
|
||||||
|
|
||||||
|
# fmt: off
|
||||||
|
VECTOR_PARAMS = [
|
||||||
|
("FLOAT_VECTOR", "HNSW", "COSINE", 128),
|
||||||
|
("FLOAT_VECTOR", "IVF_FLAT", "L2", 128),
|
||||||
|
("FLOAT16_VECTOR", "HNSW", "L2", 64),
|
||||||
|
("BFLOAT16_VECTOR", "HNSW", "L2", 64),
|
||||||
|
("INT8_VECTOR", "HNSW", "COSINE", 64),
|
||||||
|
("BINARY_VECTOR", "BIN_FLAT", "HAMMING", 128),
|
||||||
|
("SPARSE_FLOAT_VECTOR", "SPARSE_INVERTED_INDEX", "IP", 0),
|
||||||
|
]
|
||||||
|
# fmt: on
|
||||||
|
|
||||||
|
|
||||||
|
class TestCDCSyncSearchVerification(TestCDCSyncBase):
|
||||||
|
"""Test CDC sync for search and query result verification across vector types."""
|
||||||
|
|
||||||
|
def setup_method(self):
|
||||||
|
"""Setup for each test method."""
|
||||||
|
self.resources_to_cleanup = []
|
||||||
|
|
||||||
|
def teardown_method(self):
|
||||||
|
"""Cleanup after each test method - only cleanup upstream, downstream will sync."""
|
||||||
|
upstream_client = getattr(self, "_upstream_client", None)
|
||||||
|
|
||||||
|
if upstream_client:
|
||||||
|
for resource_type, resource_name in self.resources_to_cleanup:
|
||||||
|
if resource_type == "collection":
|
||||||
|
self.cleanup_collection(upstream_client, resource_name)
|
||||||
|
|
||||||
|
time.sleep(1) # Allow cleanup to sync to downstream
|
||||||
|
|
||||||
|
# -------------------------------------------------------------------------
|
||||||
|
# Internal helper
|
||||||
|
# -------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def _setup_collection(self, client, c_name, vector_type, index_type, metric, dim):
|
||||||
|
"""
|
||||||
|
Create a single-vector-schema collection, insert 500 records,
|
||||||
|
create an index, and load.
|
||||||
|
|
||||||
|
Returns the collection name (same as c_name).
|
||||||
|
"""
|
||||||
|
schema = self.create_single_vector_schema(client, vector_type=vector_type, dim=dim)
|
||||||
|
client.create_collection(collection_name=c_name, schema=schema)
|
||||||
|
|
||||||
|
# Insert 500 records
|
||||||
|
data = self.generate_single_vector_data(500, vector_type=vector_type, dim=dim)
|
||||||
|
client.insert(c_name, data)
|
||||||
|
client.flush(c_name)
|
||||||
|
|
||||||
|
# Build index
|
||||||
|
index_params = client.prepare_index_params()
|
||||||
|
if index_type == "IVF_FLAT":
|
||||||
|
idx_params = {"nlist": 64}
|
||||||
|
elif index_type == "HNSW":
|
||||||
|
idx_params = {"M": 16, "efConstruction": 200}
|
||||||
|
else:
|
||||||
|
idx_params = {}
|
||||||
|
|
||||||
|
index_params.add_index(
|
||||||
|
field_name="vector",
|
||||||
|
index_type=index_type,
|
||||||
|
metric_type=metric,
|
||||||
|
params=idx_params,
|
||||||
|
)
|
||||||
|
client.create_index(c_name, index_params)
|
||||||
|
client.load_collection(c_name)
|
||||||
|
|
||||||
|
return c_name
|
||||||
|
|
||||||
|
# -------------------------------------------------------------------------
|
||||||
|
# Tests
|
||||||
|
# -------------------------------------------------------------------------
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"vector_type,index_type,metric,dim",
|
||||||
|
VECTOR_PARAMS,
|
||||||
|
ids=[p[0] + "_" + p[1] for p in VECTOR_PARAMS],
|
||||||
|
)
|
||||||
|
def test_search_result_consistency(
|
||||||
|
self,
|
||||||
|
upstream_client,
|
||||||
|
downstream_client,
|
||||||
|
sync_timeout,
|
||||||
|
vector_type,
|
||||||
|
index_type,
|
||||||
|
metric,
|
||||||
|
dim,
|
||||||
|
):
|
||||||
|
"""Verify that ANN search results are consistent between upstream and downstream."""
|
||||||
|
start_time = time.time()
|
||||||
|
c_name = self.gen_unique_name(f"test_src_{vector_type[:4].lower()}", max_length=50)
|
||||||
|
|
||||||
|
self.log_test_start(
|
||||||
|
"test_search_result_consistency",
|
||||||
|
f"SEARCH/{vector_type}/{index_type}",
|
||||||
|
c_name,
|
||||||
|
)
|
||||||
|
self._upstream_client = upstream_client
|
||||||
|
self.resources_to_cleanup.append(("collection", c_name))
|
||||||
|
|
||||||
|
try:
|
||||||
|
self.cleanup_collection(upstream_client, c_name)
|
||||||
|
self._setup_collection(upstream_client, c_name, vector_type, index_type, metric, dim)
|
||||||
|
|
||||||
|
# Wait for at least 500 records to appear on downstream
|
||||||
|
def check_sync():
|
||||||
|
try:
|
||||||
|
res = downstream_client.query(
|
||||||
|
collection_name=c_name,
|
||||||
|
filter="",
|
||||||
|
output_fields=["count(*)"],
|
||||||
|
)
|
||||||
|
cnt = res[0]["count(*)"] if res else 0
|
||||||
|
logger.info(f"[SYNC_PROGRESS] downstream count: {cnt}/500")
|
||||||
|
return cnt >= 500
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Sync check failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_sync, sync_timeout, f"data sync 500 records {c_name}"), (
|
||||||
|
f"Downstream did not receive 500 records within {sync_timeout}s"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Build 5 random query vectors
|
||||||
|
dtype = getattr(DataType, vector_type)
|
||||||
|
query_vectors = self._gen_vectors(5, dim if dim > 0 else 1000, dtype)
|
||||||
|
|
||||||
|
avg_overlap, _, _ = self.verify_search_consistency(
|
||||||
|
upstream_client,
|
||||||
|
downstream_client,
|
||||||
|
c_name,
|
||||||
|
query_vectors,
|
||||||
|
anns_field="vector",
|
||||||
|
limit=10,
|
||||||
|
metric_type=metric,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert avg_overlap >= self.SEARCH_OVERLAP_THRESHOLD, (
|
||||||
|
f"Search overlap {avg_overlap:.4f} is below threshold "
|
||||||
|
f"{self.SEARCH_OVERLAP_THRESHOLD} for {vector_type}/{index_type}"
|
||||||
|
)
|
||||||
|
|
||||||
|
finally:
|
||||||
|
self.log_test_end(
|
||||||
|
"test_search_result_consistency",
|
||||||
|
True,
|
||||||
|
time.time() - start_time,
|
||||||
|
)
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"vector_type,index_type,metric,dim",
|
||||||
|
VECTOR_PARAMS,
|
||||||
|
ids=[p[0] + "_" + p[1] for p in VECTOR_PARAMS],
|
||||||
|
)
|
||||||
|
def test_query_data_sampling(
|
||||||
|
self,
|
||||||
|
upstream_client,
|
||||||
|
downstream_client,
|
||||||
|
sync_timeout,
|
||||||
|
vector_type,
|
||||||
|
index_type,
|
||||||
|
metric,
|
||||||
|
dim,
|
||||||
|
):
|
||||||
|
"""Verify scalar field values are identical on both sides via random sampling."""
|
||||||
|
start_time = time.time()
|
||||||
|
c_name = self.gen_unique_name(f"test_qds_{vector_type[:4].lower()}", max_length=50)
|
||||||
|
|
||||||
|
self.log_test_start(
|
||||||
|
"test_query_data_sampling",
|
||||||
|
f"QUERY_SAMPLE/{vector_type}/{index_type}",
|
||||||
|
c_name,
|
||||||
|
)
|
||||||
|
self._upstream_client = upstream_client
|
||||||
|
self.resources_to_cleanup.append(("collection", c_name))
|
||||||
|
|
||||||
|
try:
|
||||||
|
self.cleanup_collection(upstream_client, c_name)
|
||||||
|
self._setup_collection(upstream_client, c_name, vector_type, index_type, metric, dim)
|
||||||
|
|
||||||
|
def check_sync():
|
||||||
|
try:
|
||||||
|
res = downstream_client.query(
|
||||||
|
collection_name=c_name,
|
||||||
|
filter="",
|
||||||
|
output_fields=["count(*)"],
|
||||||
|
)
|
||||||
|
cnt = res[0]["count(*)"] if res else 0
|
||||||
|
logger.info(f"[SYNC_PROGRESS] downstream count: {cnt}/500")
|
||||||
|
return cnt >= 500
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Sync check failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_sync, sync_timeout, f"data sync 500 records {c_name}"), (
|
||||||
|
f"Downstream did not receive 500 records within {sync_timeout}s"
|
||||||
|
)
|
||||||
|
|
||||||
|
output_fields = ["id", "int_field", "varchar_field", "float_field"]
|
||||||
|
match_count, mismatch_count, mismatch_details = self.verify_data_sampling(
|
||||||
|
upstream_client,
|
||||||
|
downstream_client,
|
||||||
|
c_name,
|
||||||
|
sample_ratio=0.2,
|
||||||
|
output_fields=output_fields,
|
||||||
|
)
|
||||||
|
|
||||||
|
logger.info(
|
||||||
|
f"[RESULT] Sampling — match={match_count}, mismatch={mismatch_count}, details={mismatch_details[:3]}"
|
||||||
|
)
|
||||||
|
assert mismatch_count == 0, f"Found {mismatch_count} mismatched records: {mismatch_details[:5]}"
|
||||||
|
|
||||||
|
finally:
|
||||||
|
self.log_test_end(
|
||||||
|
"test_query_data_sampling",
|
||||||
|
True,
|
||||||
|
time.time() - start_time,
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_hybrid_search_consistency(
|
||||||
|
self,
|
||||||
|
upstream_client,
|
||||||
|
downstream_client,
|
||||||
|
sync_timeout,
|
||||||
|
):
|
||||||
|
"""Verify hybrid search (dense + sparse, RRF ranker) results are consistent."""
|
||||||
|
start_time = time.time()
|
||||||
|
c_name = self.gen_unique_name("test_hybrid_srch", max_length=50)
|
||||||
|
|
||||||
|
self.log_test_start("test_hybrid_search_consistency", "HYBRID_SEARCH", c_name)
|
||||||
|
self._upstream_client = upstream_client
|
||||||
|
self.resources_to_cleanup.append(("collection", c_name))
|
||||||
|
|
||||||
|
try:
|
||||||
|
self.cleanup_collection(upstream_client, c_name)
|
||||||
|
|
||||||
|
# Build schema: dense FloatVector(128) + sparse
|
||||||
|
schema = upstream_client.create_schema(enable_dynamic_field=True)
|
||||||
|
schema.add_field("id", DataType.INT64, is_primary=True, auto_id=True)
|
||||||
|
schema.add_field("dense", DataType.FLOAT_VECTOR, dim=128)
|
||||||
|
schema.add_field("sparse", DataType.SPARSE_FLOAT_VECTOR)
|
||||||
|
schema.add_field("int_field", DataType.INT64)
|
||||||
|
schema.add_field("varchar_field", DataType.VARCHAR, max_length=256)
|
||||||
|
|
||||||
|
upstream_client.create_collection(collection_name=c_name, schema=schema)
|
||||||
|
|
||||||
|
# Insert 300 records
|
||||||
|
dense_vecs = self._gen_vectors(300, 128, DataType.FLOAT_VECTOR)
|
||||||
|
sparse_vecs = self._gen_vectors(300, 1000, DataType.SPARSE_FLOAT_VECTOR)
|
||||||
|
data = [
|
||||||
|
{
|
||||||
|
"dense": dense_vecs[i],
|
||||||
|
"sparse": sparse_vecs[i],
|
||||||
|
"int_field": random.randint(0, 1000),
|
||||||
|
"varchar_field": f"hybrid_{i}_{random.randint(1000, 9999)}",
|
||||||
|
}
|
||||||
|
for i in range(300)
|
||||||
|
]
|
||||||
|
upstream_client.insert(c_name, data)
|
||||||
|
upstream_client.flush(c_name)
|
||||||
|
|
||||||
|
# Create indexes
|
||||||
|
index_params = upstream_client.prepare_index_params()
|
||||||
|
index_params.add_index(
|
||||||
|
field_name="dense",
|
||||||
|
index_type="HNSW",
|
||||||
|
metric_type="COSINE",
|
||||||
|
params={"M": 16, "efConstruction": 200},
|
||||||
|
)
|
||||||
|
index_params.add_index(
|
||||||
|
field_name="sparse",
|
||||||
|
index_type="SPARSE_INVERTED_INDEX",
|
||||||
|
metric_type="IP",
|
||||||
|
params={},
|
||||||
|
)
|
||||||
|
upstream_client.create_index(c_name, index_params)
|
||||||
|
upstream_client.load_collection(c_name)
|
||||||
|
|
||||||
|
# Wait for downstream sync
|
||||||
|
def check_sync():
|
||||||
|
try:
|
||||||
|
res = downstream_client.query(
|
||||||
|
collection_name=c_name,
|
||||||
|
filter="",
|
||||||
|
output_fields=["count(*)"],
|
||||||
|
)
|
||||||
|
cnt = res[0]["count(*)"] if res else 0
|
||||||
|
logger.info(f"[SYNC_PROGRESS] downstream count: {cnt}/300")
|
||||||
|
return cnt >= 300
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Sync check failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_sync, sync_timeout, f"hybrid data sync {c_name}"), (
|
||||||
|
f"Downstream did not receive 300 records within {sync_timeout}s"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Build hybrid search requests
|
||||||
|
q_dense = self._gen_vectors(1, 128, DataType.FLOAT_VECTOR)[0]
|
||||||
|
q_sparse = self._gen_vectors(1, 1000, DataType.SPARSE_FLOAT_VECTOR)[0]
|
||||||
|
|
||||||
|
dense_req = AnnSearchRequest(
|
||||||
|
data=[q_dense],
|
||||||
|
anns_field="dense",
|
||||||
|
param={"metric_type": "COSINE", "params": {"ef": 64}},
|
||||||
|
limit=10,
|
||||||
|
)
|
||||||
|
sparse_req = AnnSearchRequest(
|
||||||
|
data=[q_sparse],
|
||||||
|
anns_field="sparse",
|
||||||
|
param={"metric_type": "IP"},
|
||||||
|
limit=10,
|
||||||
|
)
|
||||||
|
|
||||||
|
up_results = upstream_client.hybrid_search(
|
||||||
|
collection_name=c_name,
|
||||||
|
reqs=[dense_req, sparse_req],
|
||||||
|
ranker=RRFRanker(),
|
||||||
|
limit=10,
|
||||||
|
output_fields=["id"],
|
||||||
|
)
|
||||||
|
down_results = downstream_client.hybrid_search(
|
||||||
|
collection_name=c_name,
|
||||||
|
reqs=[dense_req, sparse_req],
|
||||||
|
ranker=RRFRanker(),
|
||||||
|
limit=10,
|
||||||
|
output_fields=["id"],
|
||||||
|
)
|
||||||
|
|
||||||
|
up_pks = set(hit["id"] for hit in up_results[0]) if up_results else set()
|
||||||
|
down_pks = set(hit["id"] for hit in down_results[0]) if down_results else set()
|
||||||
|
union_size = len(up_pks | down_pks)
|
||||||
|
overlap = len(up_pks & down_pks) / union_size if union_size > 0 else 1.0
|
||||||
|
|
||||||
|
logger.info(f"[RESULT] Hybrid search PK overlap={overlap:.4f} (up={len(up_pks)}, down={len(down_pks)})")
|
||||||
|
assert overlap >= self.SEARCH_OVERLAP_THRESHOLD, (
|
||||||
|
f"Hybrid search overlap {overlap:.4f} below threshold {self.SEARCH_OVERLAP_THRESHOLD}"
|
||||||
|
)
|
||||||
|
|
||||||
|
finally:
|
||||||
|
self.log_test_end(
|
||||||
|
"test_hybrid_search_consistency",
|
||||||
|
True,
|
||||||
|
time.time() - start_time,
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_search_iterator_consistency(
|
||||||
|
self,
|
||||||
|
upstream_client,
|
||||||
|
downstream_client,
|
||||||
|
sync_timeout,
|
||||||
|
):
|
||||||
|
"""Verify search iterator returns the same PK set on upstream and downstream."""
|
||||||
|
start_time = time.time()
|
||||||
|
c_name = self.gen_unique_name("test_srch_iter", max_length=50)
|
||||||
|
|
||||||
|
self.log_test_start("test_search_iterator_consistency", "SEARCH_ITERATOR", c_name)
|
||||||
|
self._upstream_client = upstream_client
|
||||||
|
self.resources_to_cleanup.append(("collection", c_name))
|
||||||
|
|
||||||
|
try:
|
||||||
|
self.cleanup_collection(upstream_client, c_name)
|
||||||
|
self._setup_collection(upstream_client, c_name, "FLOAT_VECTOR", "HNSW", "COSINE", 128)
|
||||||
|
|
||||||
|
def check_sync():
|
||||||
|
try:
|
||||||
|
res = downstream_client.query(
|
||||||
|
collection_name=c_name,
|
||||||
|
filter="",
|
||||||
|
output_fields=["count(*)"],
|
||||||
|
)
|
||||||
|
cnt = res[0]["count(*)"] if res else 0
|
||||||
|
logger.info(f"[SYNC_PROGRESS] downstream count: {cnt}/500")
|
||||||
|
return cnt >= 500
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Sync check failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_sync, sync_timeout, f"data sync 500 records {c_name}"), (
|
||||||
|
f"Downstream did not receive 500 records within {sync_timeout}s"
|
||||||
|
)
|
||||||
|
|
||||||
|
query_vec = self._gen_vectors(1, 128, DataType.FLOAT_VECTOR)[0]
|
||||||
|
search_params = {"metric_type": "COSINE", "params": {"ef": 64}}
|
||||||
|
|
||||||
|
def _collect_iterator_pks(client):
|
||||||
|
pks = set()
|
||||||
|
iterator = client.search_iterator(
|
||||||
|
collection_name=c_name,
|
||||||
|
data=[query_vec],
|
||||||
|
anns_field="vector",
|
||||||
|
batch_size=50,
|
||||||
|
limit=200,
|
||||||
|
param=search_params,
|
||||||
|
output_fields=["id"],
|
||||||
|
)
|
||||||
|
while True:
|
||||||
|
batch = iterator.next()
|
||||||
|
if not batch:
|
||||||
|
iterator.close()
|
||||||
|
break
|
||||||
|
for hit in batch:
|
||||||
|
pks.add(hit["id"])
|
||||||
|
return pks
|
||||||
|
|
||||||
|
up_pks = _collect_iterator_pks(upstream_client)
|
||||||
|
down_pks = _collect_iterator_pks(downstream_client)
|
||||||
|
union_size = len(up_pks | down_pks)
|
||||||
|
overlap = len(up_pks & down_pks) / union_size if union_size > 0 else 1.0
|
||||||
|
|
||||||
|
logger.info(f"[RESULT] Search iterator overlap={overlap:.4f} (up={len(up_pks)}, down={len(down_pks)})")
|
||||||
|
assert overlap >= self.SEARCH_OVERLAP_THRESHOLD, (
|
||||||
|
f"Search iterator PK overlap {overlap:.4f} below threshold {self.SEARCH_OVERLAP_THRESHOLD}"
|
||||||
|
)
|
||||||
|
|
||||||
|
finally:
|
||||||
|
self.log_test_end(
|
||||||
|
"test_search_iterator_consistency",
|
||||||
|
True,
|
||||||
|
time.time() - start_time,
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_query_iterator_consistency(
|
||||||
|
self,
|
||||||
|
upstream_client,
|
||||||
|
downstream_client,
|
||||||
|
sync_timeout,
|
||||||
|
):
|
||||||
|
"""Verify that a query iterator retrieves identical PK sets from both sides."""
|
||||||
|
start_time = time.time()
|
||||||
|
c_name = self.gen_unique_name("test_qry_iter", max_length=50)
|
||||||
|
|
||||||
|
self.log_test_start("test_query_iterator_consistency", "QUERY_ITERATOR", c_name)
|
||||||
|
self._upstream_client = upstream_client
|
||||||
|
self.resources_to_cleanup.append(("collection", c_name))
|
||||||
|
|
||||||
|
try:
|
||||||
|
self.cleanup_collection(upstream_client, c_name)
|
||||||
|
self._setup_collection(upstream_client, c_name, "FLOAT_VECTOR", "HNSW", "COSINE", 128)
|
||||||
|
|
||||||
|
def check_sync():
|
||||||
|
try:
|
||||||
|
res = downstream_client.query(
|
||||||
|
collection_name=c_name,
|
||||||
|
filter="",
|
||||||
|
output_fields=["count(*)"],
|
||||||
|
)
|
||||||
|
cnt = res[0]["count(*)"] if res else 0
|
||||||
|
logger.info(f"[SYNC_PROGRESS] downstream count: {cnt}/500")
|
||||||
|
return cnt >= 500
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Sync check failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_sync, sync_timeout, f"data sync 500 records {c_name}"), (
|
||||||
|
f"Downstream did not receive 500 records within {sync_timeout}s"
|
||||||
|
)
|
||||||
|
|
||||||
|
up_count, down_count, match = self.verify_iterator_consistency(
|
||||||
|
upstream_client,
|
||||||
|
downstream_client,
|
||||||
|
c_name,
|
||||||
|
batch_size=100,
|
||||||
|
)
|
||||||
|
|
||||||
|
logger.info(f"[RESULT] Query iterator — upstream={up_count}, downstream={down_count}, match={match}")
|
||||||
|
assert match, f"Query iterator PK sets differ: upstream={up_count}, downstream={down_count}"
|
||||||
|
|
||||||
|
finally:
|
||||||
|
self.log_test_end(
|
||||||
|
"test_query_iterator_consistency",
|
||||||
|
True,
|
||||||
|
time.time() - start_time,
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_search_with_filter_consistency(
|
||||||
|
self,
|
||||||
|
upstream_client,
|
||||||
|
downstream_client,
|
||||||
|
sync_timeout,
|
||||||
|
):
|
||||||
|
"""Verify filtered search produces consistent results honoring the filter predicate."""
|
||||||
|
start_time = time.time()
|
||||||
|
c_name = self.gen_unique_name("test_srch_filter", max_length=50)
|
||||||
|
|
||||||
|
self.log_test_start("test_search_with_filter_consistency", "SEARCH_WITH_FILTER", c_name)
|
||||||
|
self._upstream_client = upstream_client
|
||||||
|
self.resources_to_cleanup.append(("collection", c_name))
|
||||||
|
|
||||||
|
try:
|
||||||
|
self.cleanup_collection(upstream_client, c_name)
|
||||||
|
self._setup_collection(upstream_client, c_name, "FLOAT_VECTOR", "HNSW", "COSINE", 128)
|
||||||
|
|
||||||
|
def check_sync():
|
||||||
|
try:
|
||||||
|
res = downstream_client.query(
|
||||||
|
collection_name=c_name,
|
||||||
|
filter="",
|
||||||
|
output_fields=["count(*)"],
|
||||||
|
)
|
||||||
|
cnt = res[0]["count(*)"] if res else 0
|
||||||
|
logger.info(f"[SYNC_PROGRESS] downstream count: {cnt}/500")
|
||||||
|
return cnt >= 500
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Sync check failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_sync, sync_timeout, f"data sync 500 records {c_name}"), (
|
||||||
|
f"Downstream did not receive 500 records within {sync_timeout}s"
|
||||||
|
)
|
||||||
|
|
||||||
|
filter_expr = "int_field > 500"
|
||||||
|
query_vec = self._gen_vectors(1, 128, DataType.FLOAT_VECTOR)[0]
|
||||||
|
search_params = {"metric_type": "COSINE"}
|
||||||
|
|
||||||
|
up_results = upstream_client.search(
|
||||||
|
collection_name=c_name,
|
||||||
|
data=[query_vec],
|
||||||
|
anns_field="vector",
|
||||||
|
search_params=search_params,
|
||||||
|
filter=filter_expr,
|
||||||
|
limit=10,
|
||||||
|
output_fields=["id", "int_field"],
|
||||||
|
)
|
||||||
|
down_results = downstream_client.search(
|
||||||
|
collection_name=c_name,
|
||||||
|
data=[query_vec],
|
||||||
|
anns_field="vector",
|
||||||
|
search_params=search_params,
|
||||||
|
filter=filter_expr,
|
||||||
|
limit=10,
|
||||||
|
output_fields=["id", "int_field"],
|
||||||
|
)
|
||||||
|
|
||||||
|
# Verify filter is honoured on both sides
|
||||||
|
for hit in up_results[0] if up_results else []:
|
||||||
|
assert hit["int_field"] > 500, f"Filter violated on upstream: int_field={hit['int_field']}"
|
||||||
|
for hit in down_results[0] if down_results else []:
|
||||||
|
assert hit["int_field"] > 500, f"Filter violated on downstream: int_field={hit['int_field']}"
|
||||||
|
|
||||||
|
# Verify PK overlap
|
||||||
|
up_pks = set(hit["id"] for hit in up_results[0]) if up_results else set()
|
||||||
|
down_pks = set(hit["id"] for hit in down_results[0]) if down_results else set()
|
||||||
|
union_size = len(up_pks | down_pks)
|
||||||
|
overlap = len(up_pks & down_pks) / union_size if union_size > 0 else 1.0
|
||||||
|
|
||||||
|
logger.info(f"[RESULT] Filtered search overlap={overlap:.4f} (up={len(up_pks)}, down={len(down_pks)})")
|
||||||
|
assert overlap >= self.SEARCH_OVERLAP_THRESHOLD, (
|
||||||
|
f"Filtered search overlap {overlap:.4f} below threshold {self.SEARCH_OVERLAP_THRESHOLD}"
|
||||||
|
)
|
||||||
|
|
||||||
|
finally:
|
||||||
|
self.log_test_end(
|
||||||
|
"test_search_with_filter_consistency",
|
||||||
|
True,
|
||||||
|
time.time() - start_time,
|
||||||
|
)
|
||||||
@@ -3,8 +3,12 @@ CDC topology setup and configuration test cases.
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
import time
|
import time
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
from common.common_type import CaseLabel
|
from common.common_type import CaseLabel
|
||||||
|
|
||||||
|
from cdc.conftest import CDC_UPDATE_REPLICATE_TIMEOUT_SECONDS, apply_replicate_configuration
|
||||||
|
|
||||||
from .base import TestCDCSyncBase
|
from .base import TestCDCSyncBase
|
||||||
|
|
||||||
|
|
||||||
@@ -41,10 +45,7 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
|||||||
{
|
{
|
||||||
"cluster_id": source_cluster_id,
|
"cluster_id": source_cluster_id,
|
||||||
"connection_param": {"uri": upstream_uri, "token": upstream_token},
|
"connection_param": {"uri": upstream_uri, "token": upstream_token},
|
||||||
"pchannels": [
|
"pchannels": [f"{source_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
|
||||||
f"{source_cluster_id}-rootcoord-dml_{i}"
|
|
||||||
for i in range(pchannel_num)
|
|
||||||
],
|
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cluster_id": target_cluster_id,
|
"cluster_id": target_cluster_id,
|
||||||
@@ -52,10 +53,7 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
|||||||
"uri": downstream_uri,
|
"uri": downstream_uri,
|
||||||
"token": downstream_token,
|
"token": downstream_token,
|
||||||
},
|
},
|
||||||
"pchannels": [
|
"pchannels": [f"{target_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
|
||||||
f"{target_cluster_id}-rootcoord-dml_{i}"
|
|
||||||
for i in range(pchannel_num)
|
|
||||||
],
|
|
||||||
},
|
},
|
||||||
],
|
],
|
||||||
"cross_cluster_topology": [
|
"cross_cluster_topology": [
|
||||||
@@ -67,8 +65,7 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
|||||||
}
|
}
|
||||||
|
|
||||||
# Test normal topology setup
|
# Test normal topology setup
|
||||||
upstream_client.update_replicate_configuration(**config)
|
apply_replicate_configuration([(upstream_client, config), (downstream_client, config)])
|
||||||
downstream_client.update_replicate_configuration(**config)
|
|
||||||
|
|
||||||
# Wait for configuration to take effect
|
# Wait for configuration to take effect
|
||||||
time.sleep(3)
|
time.sleep(3)
|
||||||
@@ -87,9 +84,7 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
|||||||
def check_sync():
|
def check_sync():
|
||||||
return downstream_client.has_collection(test_collection_name)
|
return downstream_client.has_collection(test_collection_name)
|
||||||
|
|
||||||
assert self.wait_for_sync(
|
assert self.wait_for_sync(check_sync, 30, f"collection {test_collection_name} sync")
|
||||||
check_sync, 30, f"collection {test_collection_name} sync"
|
|
||||||
)
|
|
||||||
|
|
||||||
# Cleanup
|
# Cleanup
|
||||||
self.cleanup_collection(upstream_client, test_collection_name)
|
self.cleanup_collection(upstream_client, test_collection_name)
|
||||||
@@ -113,10 +108,7 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
|||||||
{
|
{
|
||||||
"cluster_id": source_cluster_id,
|
"cluster_id": source_cluster_id,
|
||||||
"connection_param": {"uri": upstream_uri, "token": upstream_token},
|
"connection_param": {"uri": upstream_uri, "token": upstream_token},
|
||||||
"pchannels": [
|
"pchannels": [f"{source_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
|
||||||
f"{source_cluster_id}-rootcoord-dml_{i}"
|
|
||||||
for i in range(pchannel_num)
|
|
||||||
],
|
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cluster_id": target_cluster_id,
|
"cluster_id": target_cluster_id,
|
||||||
@@ -124,10 +116,7 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
|||||||
"uri": downstream_uri,
|
"uri": downstream_uri,
|
||||||
"token": downstream_token,
|
"token": downstream_token,
|
||||||
},
|
},
|
||||||
"pchannels": [
|
"pchannels": [f"{target_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
|
||||||
f"{target_cluster_id}-rootcoord-dml_{i}"
|
|
||||||
for i in range(pchannel_num)
|
|
||||||
],
|
|
||||||
},
|
},
|
||||||
],
|
],
|
||||||
"cross_cluster_topology": [
|
"cross_cluster_topology": [
|
||||||
@@ -138,8 +127,7 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
|||||||
],
|
],
|
||||||
}
|
}
|
||||||
|
|
||||||
upstream_client.update_replicate_configuration(**original_config)
|
apply_replicate_configuration([(upstream_client, original_config), (downstream_client, original_config)])
|
||||||
downstream_client.update_replicate_configuration(**original_config)
|
|
||||||
time.sleep(3)
|
time.sleep(3)
|
||||||
|
|
||||||
# Now switch the direction (target -> source)
|
# Now switch the direction (target -> source)
|
||||||
@@ -148,10 +136,7 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
|||||||
{
|
{
|
||||||
"cluster_id": source_cluster_id,
|
"cluster_id": source_cluster_id,
|
||||||
"connection_param": {"uri": upstream_uri, "token": upstream_token},
|
"connection_param": {"uri": upstream_uri, "token": upstream_token},
|
||||||
"pchannels": [
|
"pchannels": [f"{source_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
|
||||||
f"{source_cluster_id}-rootcoord-dml_{i}"
|
|
||||||
for i in range(pchannel_num)
|
|
||||||
],
|
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cluster_id": target_cluster_id,
|
"cluster_id": target_cluster_id,
|
||||||
@@ -159,10 +144,7 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
|||||||
"uri": downstream_uri,
|
"uri": downstream_uri,
|
||||||
"token": downstream_token,
|
"token": downstream_token,
|
||||||
},
|
},
|
||||||
"pchannels": [
|
"pchannels": [f"{target_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
|
||||||
f"{target_cluster_id}-rootcoord-dml_{i}"
|
|
||||||
for i in range(pchannel_num)
|
|
||||||
],
|
|
||||||
},
|
},
|
||||||
],
|
],
|
||||||
"cross_cluster_topology": [
|
"cross_cluster_topology": [
|
||||||
@@ -174,8 +156,7 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
|||||||
}
|
}
|
||||||
|
|
||||||
# Apply switched configuration
|
# Apply switched configuration
|
||||||
upstream_client.update_replicate_configuration(**switched_config)
|
apply_replicate_configuration([(upstream_client, switched_config), (downstream_client, switched_config)])
|
||||||
downstream_client.update_replicate_configuration(**switched_config)
|
|
||||||
time.sleep(3)
|
time.sleep(3)
|
||||||
|
|
||||||
# Test the switched topology by creating collection on downstream (now source)
|
# Test the switched topology by creating collection on downstream (now source)
|
||||||
@@ -191,9 +172,7 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
|||||||
def check_switched_sync():
|
def check_switched_sync():
|
||||||
return upstream_client.has_collection(test_collection_name)
|
return upstream_client.has_collection(test_collection_name)
|
||||||
|
|
||||||
assert self.wait_for_sync(
|
assert self.wait_for_sync(check_switched_sync, 30, f"switched collection {test_collection_name} sync")
|
||||||
check_switched_sync, 30, f"switched collection {test_collection_name} sync"
|
|
||||||
)
|
|
||||||
|
|
||||||
# Cleanup
|
# Cleanup
|
||||||
self.cleanup_collection(downstream_client, test_collection_name)
|
self.cleanup_collection(downstream_client, test_collection_name)
|
||||||
@@ -229,7 +208,9 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
|||||||
}
|
}
|
||||||
|
|
||||||
with pytest.raises(Exception):
|
with pytest.raises(Exception):
|
||||||
upstream_client.update_replicate_configuration(**invalid_config_1)
|
upstream_client.update_replicate_configuration(
|
||||||
|
timeout=CDC_UPDATE_REPLICATE_TIMEOUT_SECONDS, **invalid_config_1
|
||||||
|
)
|
||||||
|
|
||||||
# Test case 4.2: Circular dependency (A -> B, B -> A)
|
# Test case 4.2: Circular dependency (A -> B, B -> A)
|
||||||
invalid_config_2 = {
|
invalid_config_2 = {
|
||||||
@@ -262,7 +243,9 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
|||||||
|
|
||||||
# This may or may not fail depending on implementation, but test it
|
# This may or may not fail depending on implementation, but test it
|
||||||
with pytest.raises(Exception):
|
with pytest.raises(Exception):
|
||||||
upstream_client.update_replicate_configuration(**invalid_config_2)
|
upstream_client.update_replicate_configuration(
|
||||||
|
timeout=CDC_UPDATE_REPLICATE_TIMEOUT_SECONDS, **invalid_config_2
|
||||||
|
)
|
||||||
|
|
||||||
# Test case 4.3: Invalid connection parameters
|
# Test case 4.3: Invalid connection parameters
|
||||||
invalid_config_3 = {
|
invalid_config_3 = {
|
||||||
@@ -292,7 +275,9 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
|||||||
],
|
],
|
||||||
}
|
}
|
||||||
with pytest.raises(Exception):
|
with pytest.raises(Exception):
|
||||||
upstream_client.update_replicate_configuration(**invalid_config_3)
|
upstream_client.update_replicate_configuration(
|
||||||
|
timeout=CDC_UPDATE_REPLICATE_TIMEOUT_SECONDS, **invalid_config_3
|
||||||
|
)
|
||||||
|
|
||||||
# Test case 4.4: Empty cluster list but non-empty topology
|
# Test case 4.4: Empty cluster list but non-empty topology
|
||||||
invalid_config_4 = {
|
invalid_config_4 = {
|
||||||
@@ -305,7 +290,9 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
|||||||
],
|
],
|
||||||
}
|
}
|
||||||
with pytest.raises(Exception):
|
with pytest.raises(Exception):
|
||||||
upstream_client.update_replicate_configuration(**invalid_config_4)
|
upstream_client.update_replicate_configuration(
|
||||||
|
timeout=CDC_UPDATE_REPLICATE_TIMEOUT_SECONDS, **invalid_config_4
|
||||||
|
)
|
||||||
|
|
||||||
# Test case 4.5: Invalid pchannel format
|
# Test case 4.5: Invalid pchannel format
|
||||||
invalid_config_5 = {
|
invalid_config_5 = {
|
||||||
@@ -319,7 +306,9 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
|||||||
"cross_cluster_topology": [],
|
"cross_cluster_topology": [],
|
||||||
}
|
}
|
||||||
with pytest.raises(Exception):
|
with pytest.raises(Exception):
|
||||||
upstream_client.update_replicate_configuration(**invalid_config_5)
|
upstream_client.update_replicate_configuration(
|
||||||
|
timeout=CDC_UPDATE_REPLICATE_TIMEOUT_SECONDS, **invalid_config_5
|
||||||
|
)
|
||||||
|
|
||||||
@pytest.mark.order(-1)
|
@pytest.mark.order(-1)
|
||||||
def test_clear_configuration_disconnect(
|
def test_clear_configuration_disconnect(
|
||||||
@@ -341,10 +330,7 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
|||||||
{
|
{
|
||||||
"cluster_id": source_cluster_id,
|
"cluster_id": source_cluster_id,
|
||||||
"connection_param": {"uri": upstream_uri, "token": upstream_token},
|
"connection_param": {"uri": upstream_uri, "token": upstream_token},
|
||||||
"pchannels": [
|
"pchannels": [f"{source_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
|
||||||
f"{source_cluster_id}-rootcoord-dml_{i}"
|
|
||||||
for i in range(pchannel_num)
|
|
||||||
],
|
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"cluster_id": target_cluster_id,
|
"cluster_id": target_cluster_id,
|
||||||
@@ -352,10 +338,7 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
|||||||
"uri": downstream_uri,
|
"uri": downstream_uri,
|
||||||
"token": downstream_token,
|
"token": downstream_token,
|
||||||
},
|
},
|
||||||
"pchannels": [
|
"pchannels": [f"{target_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
|
||||||
f"{target_cluster_id}-rootcoord-dml_{i}"
|
|
||||||
for i in range(pchannel_num)
|
|
||||||
],
|
|
||||||
},
|
},
|
||||||
],
|
],
|
||||||
"cross_cluster_topology": [
|
"cross_cluster_topology": [
|
||||||
@@ -366,8 +349,7 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
|||||||
],
|
],
|
||||||
}
|
}
|
||||||
|
|
||||||
upstream_client.update_replicate_configuration(**config)
|
apply_replicate_configuration([(upstream_client, config), (downstream_client, config)])
|
||||||
downstream_client.update_replicate_configuration(**config)
|
|
||||||
time.sleep(3)
|
time.sleep(3)
|
||||||
|
|
||||||
# Verify topology is working
|
# Verify topology is working
|
||||||
@@ -380,9 +362,7 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
|||||||
def check_initial_sync():
|
def check_initial_sync():
|
||||||
return downstream_client.has_collection(test_collection_name)
|
return downstream_client.has_collection(test_collection_name)
|
||||||
|
|
||||||
assert self.wait_for_sync(
|
assert self.wait_for_sync(check_initial_sync, 30, f"initial collection {test_collection_name} sync")
|
||||||
check_initial_sync, 30, f"initial collection {test_collection_name} sync"
|
|
||||||
)
|
|
||||||
|
|
||||||
# Now clear the configuration (empty topology)
|
# Now clear the configuration (empty topology)
|
||||||
empty_upstream_config = {
|
empty_upstream_config = {
|
||||||
@@ -390,10 +370,7 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
|||||||
{
|
{
|
||||||
"cluster_id": source_cluster_id,
|
"cluster_id": source_cluster_id,
|
||||||
"connection_param": {"uri": upstream_uri, "token": upstream_token},
|
"connection_param": {"uri": upstream_uri, "token": upstream_token},
|
||||||
"pchannels": [
|
"pchannels": [f"{source_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
|
||||||
f"{source_cluster_id}-rootcoord-dml_{i}"
|
|
||||||
for i in range(pchannel_num)
|
|
||||||
],
|
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"cross_cluster_topology": [],
|
"cross_cluster_topology": [],
|
||||||
@@ -406,17 +383,15 @@ class TestCDCTopologySetup(TestCDCSyncBase):
|
|||||||
"uri": downstream_uri,
|
"uri": downstream_uri,
|
||||||
"token": downstream_token,
|
"token": downstream_token,
|
||||||
},
|
},
|
||||||
"pchannels": [
|
"pchannels": [f"{target_cluster_id}-rootcoord-dml_{i}" for i in range(pchannel_num)],
|
||||||
f"{target_cluster_id}-rootcoord-dml_{i}"
|
|
||||||
for i in range(pchannel_num)
|
|
||||||
],
|
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"cross_cluster_topology": [],
|
"cross_cluster_topology": [],
|
||||||
}
|
}
|
||||||
# Apply empty configuration to disconnect CDC
|
# Apply empty configuration to disconnect CDC
|
||||||
upstream_client.update_replicate_configuration(**empty_upstream_config)
|
apply_replicate_configuration(
|
||||||
downstream_client.update_replicate_configuration(**empty_downstream_config)
|
[(upstream_client, empty_upstream_config), (downstream_client, empty_downstream_config)]
|
||||||
|
)
|
||||||
time.sleep(3)
|
time.sleep(3)
|
||||||
|
|
||||||
# Test that CDC is disconnected - create new collection and verify it doesn't sync
|
# Test that CDC is disconnected - create new collection and verify it doesn't sync
|
||||||
|
|||||||
@@ -0,0 +1,793 @@
|
|||||||
|
"""
|
||||||
|
CDC sync tests for topology switchover and failover scenarios.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import random
|
||||||
|
import subprocess
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
|
||||||
|
from .base import TestCDCSyncBase, logger
|
||||||
|
|
||||||
|
|
||||||
|
class TestCDCSyncSwitchover(TestCDCSyncBase):
|
||||||
|
"""Test CDC sync behaviour during and after topology switchover / failover."""
|
||||||
|
|
||||||
|
def setup_method(self):
|
||||||
|
"""Setup for each test method."""
|
||||||
|
self.resources_to_cleanup = []
|
||||||
|
|
||||||
|
def teardown_method(self):
|
||||||
|
"""Cleanup after each test method - only cleanup upstream, downstream will sync."""
|
||||||
|
upstream_client = getattr(self, "_upstream_client", None)
|
||||||
|
|
||||||
|
if upstream_client:
|
||||||
|
for resource_type, resource_name in self.resources_to_cleanup:
|
||||||
|
if resource_type == "collection":
|
||||||
|
self.cleanup_collection(upstream_client, resource_name)
|
||||||
|
|
||||||
|
time.sleep(1) # Allow cleanup to sync to downstream
|
||||||
|
|
||||||
|
# -------------------------------------------------------------------------
|
||||||
|
# Tests
|
||||||
|
# -------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def test_switchover_basic(
|
||||||
|
self,
|
||||||
|
upstream_client,
|
||||||
|
downstream_client,
|
||||||
|
sync_timeout,
|
||||||
|
switchover_helper,
|
||||||
|
source_cluster_id,
|
||||||
|
target_cluster_id,
|
||||||
|
):
|
||||||
|
"""
|
||||||
|
Basic switchover: insert 200 records, verify sync, switchover, insert 200 more
|
||||||
|
on the new source, verify count == 400 on both sides, sample verify, switch back.
|
||||||
|
"""
|
||||||
|
start_time = time.time()
|
||||||
|
c_name = self.gen_unique_name("test_sw_basic", max_length=50)
|
||||||
|
|
||||||
|
self.log_test_start("test_switchover_basic", "SWITCHOVER_BASIC", c_name)
|
||||||
|
self._upstream_client = upstream_client
|
||||||
|
self.resources_to_cleanup.append(("collection", c_name))
|
||||||
|
|
||||||
|
try:
|
||||||
|
self.cleanup_collection(upstream_client, c_name)
|
||||||
|
|
||||||
|
schema = self.create_manual_id_schema(upstream_client)
|
||||||
|
upstream_client.create_collection(collection_name=c_name, schema=schema)
|
||||||
|
|
||||||
|
index_params = upstream_client.prepare_index_params()
|
||||||
|
index_params.add_index(
|
||||||
|
field_name="vector",
|
||||||
|
index_type="HNSW",
|
||||||
|
metric_type="L2",
|
||||||
|
params={"M": 16, "efConstruction": 200},
|
||||||
|
)
|
||||||
|
upstream_client.create_index(c_name, index_params)
|
||||||
|
upstream_client.load_collection(c_name)
|
||||||
|
|
||||||
|
# Wait for collection to appear on downstream
|
||||||
|
def check_create():
|
||||||
|
return downstream_client.has_collection(c_name)
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {c_name}")
|
||||||
|
|
||||||
|
# Insert 200 records on upstream (original source)
|
||||||
|
batch1 = self.generate_test_data_with_id(200, start_id=0)
|
||||||
|
upstream_client.insert(c_name, batch1)
|
||||||
|
upstream_client.flush(c_name)
|
||||||
|
|
||||||
|
def check_200():
|
||||||
|
try:
|
||||||
|
res = downstream_client.query(
|
||||||
|
collection_name=c_name,
|
||||||
|
filter="",
|
||||||
|
output_fields=["count(*)"],
|
||||||
|
)
|
||||||
|
cnt = res[0]["count(*)"] if res else 0
|
||||||
|
logger.info(f"[SYNC_PROGRESS] downstream count: {cnt}/200")
|
||||||
|
return cnt >= 200
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Sync check failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_200, sync_timeout, f"initial 200-record sync {c_name}")
|
||||||
|
|
||||||
|
# Switchover: target becomes new source
|
||||||
|
logger.info("[SWITCHOVER] Initiating basic switchover...")
|
||||||
|
switchover_helper(target_cluster_id, source_cluster_id)
|
||||||
|
|
||||||
|
# Insert 200 more records on the new source (previously downstream)
|
||||||
|
batch2 = self.generate_test_data_with_id(200, start_id=200)
|
||||||
|
downstream_client.insert(c_name, batch2)
|
||||||
|
downstream_client.flush(c_name)
|
||||||
|
|
||||||
|
def check_400_upstream():
|
||||||
|
try:
|
||||||
|
res = upstream_client.query(
|
||||||
|
collection_name=c_name,
|
||||||
|
filter="",
|
||||||
|
output_fields=["count(*)"],
|
||||||
|
)
|
||||||
|
cnt = res[0]["count(*)"] if res else 0
|
||||||
|
logger.info(f"[SYNC_PROGRESS] upstream count after switchover: {cnt}/400")
|
||||||
|
return cnt >= 400
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Sync check failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
def check_400_downstream():
|
||||||
|
try:
|
||||||
|
res = downstream_client.query(
|
||||||
|
collection_name=c_name,
|
||||||
|
filter="",
|
||||||
|
output_fields=["count(*)"],
|
||||||
|
)
|
||||||
|
cnt = res[0]["count(*)"] if res else 0
|
||||||
|
logger.info(f"[SYNC_PROGRESS] downstream count after switchover: {cnt}/400")
|
||||||
|
return cnt >= 400
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Sync check failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_400_upstream, sync_timeout, f"400-record sync upstream {c_name}")
|
||||||
|
assert self.wait_for_sync(check_400_downstream, sync_timeout, f"400-record sync downstream {c_name}")
|
||||||
|
|
||||||
|
# Sample verify
|
||||||
|
match_count, mismatch_count, details = self.verify_data_sampling(
|
||||||
|
downstream_client, # new source
|
||||||
|
upstream_client, # new target
|
||||||
|
c_name,
|
||||||
|
sample_ratio=0.2,
|
||||||
|
output_fields=["id", "vector"],
|
||||||
|
)
|
||||||
|
assert mismatch_count == 0, f"Data mismatch after basic switchover: {details[:5]}"
|
||||||
|
|
||||||
|
finally:
|
||||||
|
# Restore original topology
|
||||||
|
logger.info("[SWITCHOVER] Restoring original topology after test_switchover_basic...")
|
||||||
|
switchover_helper(source_cluster_id, target_cluster_id)
|
||||||
|
self.log_test_end("test_switchover_basic", True, time.time() - start_time)
|
||||||
|
|
||||||
|
def test_switchover_during_writes(
|
||||||
|
self,
|
||||||
|
upstream_client,
|
||||||
|
downstream_client,
|
||||||
|
sync_timeout,
|
||||||
|
switchover_helper,
|
||||||
|
source_cluster_id,
|
||||||
|
target_cluster_id,
|
||||||
|
):
|
||||||
|
"""
|
||||||
|
Switchover during concurrent writes: background thread inserts 10 batches of 100
|
||||||
|
with 2 s between each; switchover fires at t=12 s; join thread, flush, verify
|
||||||
|
both sides have the same count >= total_inserted.
|
||||||
|
"""
|
||||||
|
start_time = time.time()
|
||||||
|
c_name = self.gen_unique_name("test_sw_writes", max_length=50)
|
||||||
|
|
||||||
|
self.log_test_start("test_switchover_during_writes", "SWITCHOVER_DURING_WRITES", c_name)
|
||||||
|
self._upstream_client = upstream_client
|
||||||
|
self.resources_to_cleanup.append(("collection", c_name))
|
||||||
|
|
||||||
|
try:
|
||||||
|
self.cleanup_collection(upstream_client, c_name)
|
||||||
|
|
||||||
|
schema = self.create_manual_id_schema(upstream_client)
|
||||||
|
upstream_client.create_collection(collection_name=c_name, schema=schema)
|
||||||
|
|
||||||
|
index_params = upstream_client.prepare_index_params()
|
||||||
|
index_params.add_index(
|
||||||
|
field_name="vector",
|
||||||
|
index_type="HNSW",
|
||||||
|
metric_type="L2",
|
||||||
|
params={"M": 16, "efConstruction": 200},
|
||||||
|
)
|
||||||
|
upstream_client.create_index(c_name, index_params)
|
||||||
|
upstream_client.load_collection(c_name)
|
||||||
|
|
||||||
|
def check_create():
|
||||||
|
return downstream_client.has_collection(c_name)
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {c_name}")
|
||||||
|
|
||||||
|
total_inserted = 0
|
||||||
|
lock = threading.Lock()
|
||||||
|
write_error = []
|
||||||
|
|
||||||
|
def background_insert():
|
||||||
|
nonlocal total_inserted
|
||||||
|
for batch_idx in range(10):
|
||||||
|
try:
|
||||||
|
start_id = batch_idx * 100
|
||||||
|
data = self.generate_test_data_with_id(100, start_id=start_id)
|
||||||
|
# Cap pymilvus's retry loop at 10 (default 75). A batch
|
||||||
|
# that lands on the old primary after it flips to
|
||||||
|
# replica returns STREAMING_CODE_REPLICATE_VIOLATION;
|
||||||
|
# the default retry burns ~213 s per failed batch and
|
||||||
|
# makes the thread runtime unbounded when the role flip
|
||||||
|
# takes longer than one batch's cadence. 10 retries
|
||||||
|
# with the decorator's 3 s backoff cap bound each
|
||||||
|
# failure to ~16 s.
|
||||||
|
upstream_client.insert(c_name, data, retry_times=10)
|
||||||
|
with lock:
|
||||||
|
total_inserted += 100
|
||||||
|
logger.info(f"[BACKGROUND] Inserted batch {batch_idx + 1}/10 (total: {total_inserted})")
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"[BACKGROUND] Insert failed on batch {batch_idx}: {e}")
|
||||||
|
write_error.append(e)
|
||||||
|
time.sleep(2)
|
||||||
|
|
||||||
|
insert_thread = threading.Thread(target=background_insert, daemon=True)
|
||||||
|
insert_thread.start()
|
||||||
|
|
||||||
|
# Switchover after 12 s (mid-way through background inserts)
|
||||||
|
time.sleep(12)
|
||||||
|
logger.info("[SWITCHOVER] Initiating switchover during writes...")
|
||||||
|
switchover_helper(target_cluster_id, source_cluster_id)
|
||||||
|
|
||||||
|
# With retry_times=10 each failed batch burns ~16 s (vs ~213 s with
|
||||||
|
# the pymilvus default of 75). Worst case: all 10 batches fail →
|
||||||
|
# ~10 * (16 + 2) = 180 s. 180 s gives ~6x margin.
|
||||||
|
insert_thread.join(timeout=180)
|
||||||
|
assert not insert_thread.is_alive(), "Background insert thread did not finish in time"
|
||||||
|
|
||||||
|
# Transient STREAMING_CODE_REPLICATE_VIOLATION failures are EXPECTED
|
||||||
|
# during the role flip: writes that arrive on the old primary after
|
||||||
|
# it becomes a replica are rejected by design. The invariant that
|
||||||
|
# matters is that every batch which SUCCEEDED ends up replicated
|
||||||
|
# consistently (enforced by the up_cnt == down_cnt check below).
|
||||||
|
if write_error:
|
||||||
|
logger.warning(
|
||||||
|
f"Background inserts had {len(write_error)} transient failure(s) during switchover: {write_error}"
|
||||||
|
)
|
||||||
|
|
||||||
|
# After switchover, downstream is the new primary. Flushing the
|
||||||
|
# old primary (upstream) hits the replica-role rate limiter
|
||||||
|
# (rate=0.1/s) and exhausts pymilvus's 75 retries. Flush the
|
||||||
|
# current primary instead.
|
||||||
|
downstream_client.flush(c_name)
|
||||||
|
|
||||||
|
expected = total_inserted
|
||||||
|
logger.info(f"[INFO] Total inserted: {expected}")
|
||||||
|
|
||||||
|
def check_both(client, label):
|
||||||
|
def _check():
|
||||||
|
try:
|
||||||
|
res = client.query(
|
||||||
|
collection_name=c_name,
|
||||||
|
filter="",
|
||||||
|
output_fields=["count(*)"],
|
||||||
|
)
|
||||||
|
cnt = res[0]["count(*)"] if res else 0
|
||||||
|
logger.info(f"[SYNC_PROGRESS] {label} count: {cnt}/{expected}")
|
||||||
|
return cnt >= expected
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Sync check failed ({label}): {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
return _check
|
||||||
|
|
||||||
|
assert self.wait_for_sync(
|
||||||
|
check_both(upstream_client, "upstream"),
|
||||||
|
sync_timeout,
|
||||||
|
f"upstream count>={expected} after switchover-during-writes",
|
||||||
|
)
|
||||||
|
assert self.wait_for_sync(
|
||||||
|
check_both(downstream_client, "downstream"),
|
||||||
|
sync_timeout,
|
||||||
|
f"downstream count>={expected} after switchover-during-writes",
|
||||||
|
)
|
||||||
|
|
||||||
|
up_res = upstream_client.query(collection_name=c_name, filter="", output_fields=["count(*)"])
|
||||||
|
down_res = downstream_client.query(collection_name=c_name, filter="", output_fields=["count(*)"])
|
||||||
|
up_cnt = up_res[0]["count(*)"] if up_res else 0
|
||||||
|
down_cnt = down_res[0]["count(*)"] if down_res else 0
|
||||||
|
assert up_cnt == down_cnt, (
|
||||||
|
f"Count mismatch after switchover-during-writes: upstream={up_cnt}, downstream={down_cnt}"
|
||||||
|
)
|
||||||
|
|
||||||
|
finally:
|
||||||
|
logger.info("[SWITCHOVER] Restoring original topology after test_switchover_during_writes...")
|
||||||
|
switchover_helper(source_cluster_id, target_cluster_id)
|
||||||
|
self.log_test_end("test_switchover_during_writes", True, time.time() - start_time)
|
||||||
|
|
||||||
|
def test_switchover_with_all_data_types(
|
||||||
|
self,
|
||||||
|
upstream_client,
|
||||||
|
downstream_client,
|
||||||
|
sync_timeout,
|
||||||
|
switchover_helper,
|
||||||
|
source_cluster_id,
|
||||||
|
target_cluster_id,
|
||||||
|
):
|
||||||
|
"""
|
||||||
|
Switchover with comprehensive data types: insert 100 records, verify sync,
|
||||||
|
switchover, verify scalar field sampling, switch back.
|
||||||
|
"""
|
||||||
|
start_time = time.time()
|
||||||
|
c_name = self.gen_unique_name("test_sw_dtypes", max_length=50)
|
||||||
|
|
||||||
|
self.log_test_start(
|
||||||
|
"test_switchover_with_all_data_types",
|
||||||
|
"SWITCHOVER_ALL_DTYPES",
|
||||||
|
c_name,
|
||||||
|
)
|
||||||
|
self._upstream_client = upstream_client
|
||||||
|
self.resources_to_cleanup.append(("collection", c_name))
|
||||||
|
|
||||||
|
try:
|
||||||
|
self.cleanup_collection(upstream_client, c_name)
|
||||||
|
|
||||||
|
schema = self.create_comprehensive_schema(upstream_client)
|
||||||
|
upstream_client.create_collection(collection_name=c_name, schema=schema)
|
||||||
|
|
||||||
|
# Index every vector field; load_collection fails otherwise.
|
||||||
|
index_params = upstream_client.prepare_index_params()
|
||||||
|
index_params.add_index(field_name="float_vector", index_type="AUTOINDEX", metric_type="L2")
|
||||||
|
index_params.add_index(field_name="float16_vector", index_type="AUTOINDEX", metric_type="L2")
|
||||||
|
index_params.add_index(field_name="binary_vector", index_type="BIN_FLAT", metric_type="HAMMING")
|
||||||
|
index_params.add_index(field_name="sparse_vector", index_type="SPARSE_INVERTED_INDEX", metric_type="IP")
|
||||||
|
upstream_client.create_index(c_name, index_params)
|
||||||
|
upstream_client.load_collection(c_name)
|
||||||
|
|
||||||
|
def check_create():
|
||||||
|
return downstream_client.has_collection(c_name)
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {c_name}")
|
||||||
|
|
||||||
|
data = self.generate_comprehensive_test_data(100)
|
||||||
|
upstream_client.insert(c_name, data)
|
||||||
|
upstream_client.flush(c_name)
|
||||||
|
|
||||||
|
def check_100():
|
||||||
|
try:
|
||||||
|
res = downstream_client.query(
|
||||||
|
collection_name=c_name,
|
||||||
|
filter="",
|
||||||
|
output_fields=["count(*)"],
|
||||||
|
)
|
||||||
|
cnt = res[0]["count(*)"] if res else 0
|
||||||
|
logger.info(f"[SYNC_PROGRESS] downstream count: {cnt}/100")
|
||||||
|
return cnt >= 100
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Sync check failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_100, sync_timeout, f"100-record sync {c_name}")
|
||||||
|
|
||||||
|
# Switchover
|
||||||
|
logger.info("[SWITCHOVER] Initiating switchover with all data types...")
|
||||||
|
switchover_helper(target_cluster_id, source_cluster_id)
|
||||||
|
|
||||||
|
# After switchover, verify scalar field sampling (new source is downstream)
|
||||||
|
scalar_fields = [
|
||||||
|
"bool_field",
|
||||||
|
"int8_field",
|
||||||
|
"int16_field",
|
||||||
|
"int32_field",
|
||||||
|
"int64_field",
|
||||||
|
"float_field",
|
||||||
|
"double_field",
|
||||||
|
"varchar_field",
|
||||||
|
]
|
||||||
|
match_count, mismatch_count, details = self.verify_data_sampling(
|
||||||
|
downstream_client, # new source
|
||||||
|
upstream_client, # new target
|
||||||
|
c_name,
|
||||||
|
sample_ratio=0.3,
|
||||||
|
output_fields=scalar_fields,
|
||||||
|
)
|
||||||
|
logger.info(f"[RESULT] All-dtypes sampling — match={match_count}, mismatch={mismatch_count}")
|
||||||
|
assert mismatch_count == 0, f"Data mismatch after switchover with all types: {details[:5]}"
|
||||||
|
|
||||||
|
finally:
|
||||||
|
logger.info("[SWITCHOVER] Restoring original topology after test_switchover_with_all_data_types...")
|
||||||
|
switchover_helper(source_cluster_id, target_cluster_id)
|
||||||
|
self.log_test_end(
|
||||||
|
"test_switchover_with_all_data_types",
|
||||||
|
True,
|
||||||
|
time.time() - start_time,
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_switchover_with_loaded_collection(
|
||||||
|
self,
|
||||||
|
upstream_client,
|
||||||
|
downstream_client,
|
||||||
|
sync_timeout,
|
||||||
|
switchover_helper,
|
||||||
|
source_cluster_id,
|
||||||
|
target_cluster_id,
|
||||||
|
):
|
||||||
|
"""
|
||||||
|
Switchover with a loaded collection: verify search works on downstream before
|
||||||
|
switchover; after switchover verify search works on both sides.
|
||||||
|
"""
|
||||||
|
start_time = time.time()
|
||||||
|
c_name = self.gen_unique_name("test_sw_loaded", max_length=50)
|
||||||
|
|
||||||
|
self.log_test_start(
|
||||||
|
"test_switchover_with_loaded_collection",
|
||||||
|
"SWITCHOVER_LOADED",
|
||||||
|
c_name,
|
||||||
|
)
|
||||||
|
self._upstream_client = upstream_client
|
||||||
|
self.resources_to_cleanup.append(("collection", c_name))
|
||||||
|
|
||||||
|
try:
|
||||||
|
self.cleanup_collection(upstream_client, c_name)
|
||||||
|
|
||||||
|
schema = self.create_default_schema(upstream_client)
|
||||||
|
upstream_client.create_collection(collection_name=c_name, schema=schema)
|
||||||
|
|
||||||
|
index_params = upstream_client.prepare_index_params()
|
||||||
|
index_params.add_index(
|
||||||
|
field_name="vector",
|
||||||
|
index_type="HNSW",
|
||||||
|
metric_type="L2",
|
||||||
|
params={"M": 16, "efConstruction": 200},
|
||||||
|
)
|
||||||
|
upstream_client.create_index(c_name, index_params)
|
||||||
|
upstream_client.load_collection(c_name)
|
||||||
|
|
||||||
|
def check_create():
|
||||||
|
return downstream_client.has_collection(c_name)
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {c_name}")
|
||||||
|
|
||||||
|
data = self.generate_test_data(200)
|
||||||
|
upstream_client.insert(c_name, data)
|
||||||
|
upstream_client.flush(c_name)
|
||||||
|
|
||||||
|
def check_200():
|
||||||
|
try:
|
||||||
|
res = downstream_client.query(
|
||||||
|
collection_name=c_name,
|
||||||
|
filter="",
|
||||||
|
output_fields=["count(*)"],
|
||||||
|
)
|
||||||
|
cnt = res[0]["count(*)"] if res else 0
|
||||||
|
logger.info(f"[SYNC_PROGRESS] downstream count: {cnt}/200")
|
||||||
|
return cnt >= 200
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Sync check failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_200, sync_timeout, f"200-record sync {c_name}")
|
||||||
|
|
||||||
|
# Verify search works on downstream before switchover
|
||||||
|
q_vec = [random.random() for _ in range(128)]
|
||||||
|
pre_down_results = downstream_client.search(
|
||||||
|
collection_name=c_name,
|
||||||
|
data=[q_vec],
|
||||||
|
anns_field="vector",
|
||||||
|
search_params={"metric_type": "L2"},
|
||||||
|
limit=5,
|
||||||
|
output_fields=["id"],
|
||||||
|
)
|
||||||
|
assert pre_down_results and len(pre_down_results[0]) > 0, (
|
||||||
|
"Search on downstream returned no results before switchover"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Switchover
|
||||||
|
logger.info("[SWITCHOVER] Initiating switchover with loaded collection...")
|
||||||
|
switchover_helper(target_cluster_id, source_cluster_id)
|
||||||
|
|
||||||
|
# Verify search still works on both sides after switchover
|
||||||
|
for client, label in [(upstream_client, "upstream"), (downstream_client, "downstream")]:
|
||||||
|
results = client.search(
|
||||||
|
collection_name=c_name,
|
||||||
|
data=[q_vec],
|
||||||
|
anns_field="vector",
|
||||||
|
search_params={"metric_type": "L2"},
|
||||||
|
limit=5,
|
||||||
|
output_fields=["id"],
|
||||||
|
)
|
||||||
|
assert results and len(results[0]) > 0, f"Search returned no results on {label} after switchover"
|
||||||
|
logger.info(f"[VERIFY] Search on {label} after switchover returned {len(results[0])} results — OK")
|
||||||
|
|
||||||
|
finally:
|
||||||
|
logger.info("[SWITCHOVER] Restoring original topology after test_switchover_with_loaded_collection...")
|
||||||
|
switchover_helper(source_cluster_id, target_cluster_id)
|
||||||
|
self.log_test_end(
|
||||||
|
"test_switchover_with_loaded_collection",
|
||||||
|
True,
|
||||||
|
time.time() - start_time,
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_switchover_with_index(
|
||||||
|
self,
|
||||||
|
upstream_client,
|
||||||
|
downstream_client,
|
||||||
|
sync_timeout,
|
||||||
|
switchover_helper,
|
||||||
|
source_cluster_id,
|
||||||
|
target_cluster_id,
|
||||||
|
):
|
||||||
|
"""
|
||||||
|
Switchover after index creation: wait for index to sync, switchover, verify
|
||||||
|
list_indexes returns the same index on both sides.
|
||||||
|
"""
|
||||||
|
start_time = time.time()
|
||||||
|
c_name = self.gen_unique_name("test_sw_index", max_length=50)
|
||||||
|
|
||||||
|
self.log_test_start("test_switchover_with_index", "SWITCHOVER_INDEX", c_name)
|
||||||
|
self._upstream_client = upstream_client
|
||||||
|
self.resources_to_cleanup.append(("collection", c_name))
|
||||||
|
|
||||||
|
try:
|
||||||
|
self.cleanup_collection(upstream_client, c_name)
|
||||||
|
|
||||||
|
schema = self.create_default_schema(upstream_client)
|
||||||
|
upstream_client.create_collection(collection_name=c_name, schema=schema)
|
||||||
|
|
||||||
|
# Insert data first
|
||||||
|
data = self.generate_test_data(200)
|
||||||
|
upstream_client.insert(c_name, data)
|
||||||
|
upstream_client.flush(c_name)
|
||||||
|
|
||||||
|
# Create HNSW index
|
||||||
|
index_params = upstream_client.prepare_index_params()
|
||||||
|
index_params.add_index(
|
||||||
|
field_name="vector",
|
||||||
|
index_type="HNSW",
|
||||||
|
metric_type="COSINE",
|
||||||
|
params={"M": 16, "efConstruction": 200},
|
||||||
|
)
|
||||||
|
upstream_client.create_index(c_name, index_params)
|
||||||
|
|
||||||
|
# Wait for index to sync to downstream
|
||||||
|
def check_index_sync():
|
||||||
|
try:
|
||||||
|
indexes = downstream_client.list_indexes(c_name)
|
||||||
|
result = len(indexes) > 0
|
||||||
|
logger.info(f"[SYNC_PROGRESS] downstream indexes: {indexes} (has_index={result})")
|
||||||
|
return result
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Index sync check failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_index_sync, sync_timeout, f"index sync {c_name}"), (
|
||||||
|
f"Index did not sync to downstream within {sync_timeout}s"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Switchover
|
||||||
|
logger.info("[SWITCHOVER] Initiating switchover with index...")
|
||||||
|
switchover_helper(target_cluster_id, source_cluster_id)
|
||||||
|
|
||||||
|
# Verify list_indexes on both sides
|
||||||
|
up_indexes = upstream_client.list_indexes(c_name)
|
||||||
|
down_indexes = downstream_client.list_indexes(c_name)
|
||||||
|
|
||||||
|
logger.info(f"[VERIFY] After switchover — upstream indexes={up_indexes}, downstream indexes={down_indexes}")
|
||||||
|
assert len(up_indexes) > 0, "No indexes on upstream after switchover"
|
||||||
|
assert len(down_indexes) > 0, "No indexes on downstream after switchover"
|
||||||
|
assert set(up_indexes) == set(down_indexes), (
|
||||||
|
f"Index lists differ after switchover: up={up_indexes}, down={down_indexes}"
|
||||||
|
)
|
||||||
|
|
||||||
|
finally:
|
||||||
|
logger.info("[SWITCHOVER] Restoring original topology after test_switchover_with_index...")
|
||||||
|
switchover_helper(source_cluster_id, target_cluster_id)
|
||||||
|
self.log_test_end("test_switchover_with_index", True, time.time() - start_time)
|
||||||
|
|
||||||
|
def test_rapid_switchover_stress(
|
||||||
|
self,
|
||||||
|
upstream_client,
|
||||||
|
downstream_client,
|
||||||
|
sync_timeout,
|
||||||
|
switchover_helper,
|
||||||
|
source_cluster_id,
|
||||||
|
target_cluster_id,
|
||||||
|
):
|
||||||
|
"""
|
||||||
|
Rapid-switchover stress: loop 5 times — write 50 records to current source,
|
||||||
|
switchover. After loop, wait 30 s and verify counts match. Restore original topology.
|
||||||
|
"""
|
||||||
|
start_time = time.time()
|
||||||
|
c_name = self.gen_unique_name("test_sw_stress", max_length=50)
|
||||||
|
|
||||||
|
self.log_test_start("test_rapid_switchover_stress", "RAPID_SWITCHOVER_STRESS", c_name)
|
||||||
|
self._upstream_client = upstream_client
|
||||||
|
self.resources_to_cleanup.append(("collection", c_name))
|
||||||
|
|
||||||
|
try:
|
||||||
|
self.cleanup_collection(upstream_client, c_name)
|
||||||
|
|
||||||
|
schema = self.create_manual_id_schema(upstream_client)
|
||||||
|
upstream_client.create_collection(collection_name=c_name, schema=schema)
|
||||||
|
|
||||||
|
index_params = upstream_client.prepare_index_params()
|
||||||
|
index_params.add_index(
|
||||||
|
field_name="vector",
|
||||||
|
index_type="HNSW",
|
||||||
|
metric_type="COSINE",
|
||||||
|
params={"M": 16, "efConstruction": 200},
|
||||||
|
)
|
||||||
|
upstream_client.create_index(c_name, index_params)
|
||||||
|
upstream_client.load_collection(c_name)
|
||||||
|
|
||||||
|
def check_create():
|
||||||
|
return downstream_client.has_collection(c_name)
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {c_name}")
|
||||||
|
|
||||||
|
# Track topology state
|
||||||
|
current_source_id = source_cluster_id
|
||||||
|
current_target_id = target_cluster_id
|
||||||
|
current_source_client = upstream_client
|
||||||
|
total_inserted = 0
|
||||||
|
|
||||||
|
for iteration in range(5):
|
||||||
|
# Write 50 records to current source
|
||||||
|
start_id = iteration * 50
|
||||||
|
data = self.generate_test_data_with_id(50, start_id=start_id)
|
||||||
|
current_source_client.insert(c_name, data)
|
||||||
|
current_source_client.flush(c_name)
|
||||||
|
total_inserted += 50
|
||||||
|
|
||||||
|
logger.info(
|
||||||
|
f"[STRESS] Iteration {iteration + 1}/5: inserted 50 records "
|
||||||
|
f"(total={total_inserted}), switching over..."
|
||||||
|
)
|
||||||
|
|
||||||
|
# Switchover
|
||||||
|
switchover_helper(current_target_id, current_source_id)
|
||||||
|
|
||||||
|
# Swap topology
|
||||||
|
current_source_id, current_target_id = current_target_id, current_source_id
|
||||||
|
current_source_client = downstream_client if current_source_id == target_cluster_id else upstream_client
|
||||||
|
|
||||||
|
logger.info(
|
||||||
|
f"[STRESS] All 5 switchovers done. Waiting 30s for final sync (total_inserted={total_inserted})..."
|
||||||
|
)
|
||||||
|
time.sleep(30)
|
||||||
|
|
||||||
|
# Verify counts match on both sides
|
||||||
|
up_res = upstream_client.query(collection_name=c_name, filter="", output_fields=["count(*)"])
|
||||||
|
down_res = downstream_client.query(collection_name=c_name, filter="", output_fields=["count(*)"])
|
||||||
|
up_cnt = up_res[0]["count(*)"] if up_res else 0
|
||||||
|
down_cnt = down_res[0]["count(*)"] if down_res else 0
|
||||||
|
|
||||||
|
logger.info(f"[VERIFY] After stress — upstream={up_cnt}, downstream={down_cnt}, expected>={total_inserted}")
|
||||||
|
assert up_cnt >= total_inserted, f"Upstream count {up_cnt} < expected {total_inserted} after stress"
|
||||||
|
assert down_cnt >= total_inserted, f"Downstream count {down_cnt} < expected {total_inserted} after stress"
|
||||||
|
assert up_cnt == down_cnt, f"Count mismatch after stress: upstream={up_cnt}, downstream={down_cnt}"
|
||||||
|
|
||||||
|
finally:
|
||||||
|
# Restore to original topology
|
||||||
|
logger.info("[SWITCHOVER] Restoring original topology after test_rapid_switchover_stress...")
|
||||||
|
switchover_helper(source_cluster_id, target_cluster_id)
|
||||||
|
self.log_test_end("test_rapid_switchover_stress", True, time.time() - start_time)
|
||||||
|
|
||||||
|
def test_failover_source_down(
|
||||||
|
self,
|
||||||
|
upstream_client,
|
||||||
|
downstream_client,
|
||||||
|
sync_timeout,
|
||||||
|
switchover_helper,
|
||||||
|
source_cluster_id,
|
||||||
|
target_cluster_id,
|
||||||
|
milvus_ns,
|
||||||
|
):
|
||||||
|
"""
|
||||||
|
Failover when source goes down: insert 500 records, verify sync, kill source pods,
|
||||||
|
verify target count >= 500, wait for source to recover and verify count >= 500.
|
||||||
|
"""
|
||||||
|
start_time = time.time()
|
||||||
|
c_name = self.gen_unique_name("test_failover_src", max_length=50)
|
||||||
|
|
||||||
|
self.log_test_start("test_failover_source_down", "FAILOVER_SOURCE_DOWN", c_name)
|
||||||
|
self._upstream_client = upstream_client
|
||||||
|
self.resources_to_cleanup.append(("collection", c_name))
|
||||||
|
|
||||||
|
try:
|
||||||
|
self.cleanup_collection(upstream_client, c_name)
|
||||||
|
|
||||||
|
schema = self.create_manual_id_schema(upstream_client)
|
||||||
|
upstream_client.create_collection(collection_name=c_name, schema=schema)
|
||||||
|
|
||||||
|
index_params = upstream_client.prepare_index_params()
|
||||||
|
index_params.add_index(
|
||||||
|
field_name="vector",
|
||||||
|
index_type="HNSW",
|
||||||
|
metric_type="L2",
|
||||||
|
params={"M": 16, "efConstruction": 200},
|
||||||
|
)
|
||||||
|
upstream_client.create_index(c_name, index_params)
|
||||||
|
upstream_client.load_collection(c_name)
|
||||||
|
|
||||||
|
def check_create():
|
||||||
|
return downstream_client.has_collection(c_name)
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_create, sync_timeout, f"create collection {c_name}")
|
||||||
|
|
||||||
|
# Insert 500 records
|
||||||
|
data = self.generate_test_data_with_id(500, start_id=0)
|
||||||
|
upstream_client.insert(c_name, data)
|
||||||
|
upstream_client.flush(c_name)
|
||||||
|
|
||||||
|
def check_500_downstream():
|
||||||
|
try:
|
||||||
|
res = downstream_client.query(
|
||||||
|
collection_name=c_name,
|
||||||
|
filter="",
|
||||||
|
output_fields=["count(*)"],
|
||||||
|
)
|
||||||
|
cnt = res[0]["count(*)"] if res else 0
|
||||||
|
logger.info(f"[SYNC_PROGRESS] downstream count: {cnt}/500")
|
||||||
|
return cnt >= 500
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Sync check failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_500_downstream, sync_timeout, f"500-record sync {c_name}"), (
|
||||||
|
f"Downstream did not receive 500 records within {sync_timeout}s"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Kill source pods forcefully
|
||||||
|
logger.info(f"[FAILOVER] Killing source pods (instance={source_cluster_id}, ns={milvus_ns})...")
|
||||||
|
kill_cmd = [
|
||||||
|
"kubectl",
|
||||||
|
"delete",
|
||||||
|
"pods",
|
||||||
|
"-l",
|
||||||
|
f"app.kubernetes.io/instance={source_cluster_id}",
|
||||||
|
"-n",
|
||||||
|
milvus_ns,
|
||||||
|
"--grace-period=0",
|
||||||
|
"--force",
|
||||||
|
]
|
||||||
|
result = subprocess.run(kill_cmd, capture_output=True, text=True)
|
||||||
|
logger.info(
|
||||||
|
f"[FAILOVER] kubectl output: stdout={result.stdout!r}, stderr={result.stderr!r}, rc={result.returncode}"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Wait 60 s and verify target still has data
|
||||||
|
logger.info("[FAILOVER] Waiting 60s after pod kill...")
|
||||||
|
time.sleep(60)
|
||||||
|
|
||||||
|
def check_target_intact():
|
||||||
|
try:
|
||||||
|
res = downstream_client.query(
|
||||||
|
collection_name=c_name,
|
||||||
|
filter="",
|
||||||
|
output_fields=["count(*)"],
|
||||||
|
)
|
||||||
|
cnt = res[0]["count(*)"] if res else 0
|
||||||
|
logger.info(f"[FAILOVER] Target count after kill: {cnt}")
|
||||||
|
return cnt >= 500
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Target check failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_target_intact, 30, f"target intact after source kill {c_name}"), (
|
||||||
|
"Target lost data after source pod kill"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Wait 120 s more for source to recover, then verify
|
||||||
|
logger.info("[FAILOVER] Waiting 120s for source recovery...")
|
||||||
|
time.sleep(120)
|
||||||
|
|
||||||
|
def check_source_recovered():
|
||||||
|
try:
|
||||||
|
res = upstream_client.query(
|
||||||
|
collection_name=c_name,
|
||||||
|
filter="",
|
||||||
|
output_fields=["count(*)"],
|
||||||
|
)
|
||||||
|
cnt = res[0]["count(*)"] if res else 0
|
||||||
|
logger.info(f"[FAILOVER] Source count after recovery: {cnt}")
|
||||||
|
return cnt >= 500
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Source recovery check failed: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
assert self.wait_for_sync(check_source_recovered, 60, f"source recovery {c_name}"), (
|
||||||
|
"Source did not recover with expected data count within timeout"
|
||||||
|
)
|
||||||
|
|
||||||
|
finally:
|
||||||
|
self.log_test_end("test_failover_source_down", True, time.time() - start_time)
|
||||||
@@ -24,13 +24,13 @@ max_top_k = 16384
|
|||||||
max_nq = 16384
|
max_nq = 16384
|
||||||
max_partition_num = 1024
|
max_partition_num = 1024
|
||||||
max_role_num = 10
|
max_role_num = 10
|
||||||
default_partition_num = 16 # default num_partitions for partition key feature
|
default_partition_num = 16 # default num_partitions for partition key feature
|
||||||
default_segment_row_limit = 1000
|
default_segment_row_limit = 1000
|
||||||
default_server_segment_row_limit = 1024 * 512
|
default_server_segment_row_limit = 1024 * 512
|
||||||
default_alias = "default"
|
default_alias = "default"
|
||||||
default_user = "root"
|
default_user = "root"
|
||||||
default_password = "Milvus"
|
default_password = "Milvus"
|
||||||
default_primary_field_name = 'pk'
|
default_primary_field_name = "pk"
|
||||||
default_bool_field_name = "bool"
|
default_bool_field_name = "bool"
|
||||||
default_int8_field_name = "int8"
|
default_int8_field_name = "int8"
|
||||||
default_int16_field_name = "int16"
|
default_int16_field_name = "int16"
|
||||||
@@ -63,13 +63,13 @@ default_reranker_field_name = "reranker_field"
|
|||||||
default_new_field_name = "field_new"
|
default_new_field_name = "field_new"
|
||||||
|
|
||||||
all_vector_types = [
|
all_vector_types = [
|
||||||
DataType.FLOAT_VECTOR,
|
DataType.FLOAT_VECTOR,
|
||||||
DataType.FLOAT16_VECTOR,
|
DataType.FLOAT16_VECTOR,
|
||||||
DataType.BFLOAT16_VECTOR,
|
DataType.BFLOAT16_VECTOR,
|
||||||
DataType.SPARSE_FLOAT_VECTOR,
|
DataType.SPARSE_FLOAT_VECTOR,
|
||||||
DataType.INT8_VECTOR,
|
DataType.INT8_VECTOR,
|
||||||
DataType.BINARY_VECTOR,
|
DataType.BINARY_VECTOR,
|
||||||
]
|
]
|
||||||
|
|
||||||
default_metric_for_vector_type = {
|
default_metric_for_vector_type = {
|
||||||
DataType.FLOAT_VECTOR: "COSINE",
|
DataType.FLOAT_VECTOR: "COSINE",
|
||||||
@@ -90,35 +90,51 @@ all_scalar_data_types = [
|
|||||||
DataType.DOUBLE,
|
DataType.DOUBLE,
|
||||||
DataType.VARCHAR,
|
DataType.VARCHAR,
|
||||||
DataType.ARRAY,
|
DataType.ARRAY,
|
||||||
DataType.JSON,
|
DataType.JSON,
|
||||||
DataType.GEOMETRY,
|
DataType.GEOMETRY,
|
||||||
DataType.TIMESTAMPTZ
|
DataType.TIMESTAMPTZ,
|
||||||
]
|
]
|
||||||
|
|
||||||
default_field_name_map = {
|
default_field_name_map = {
|
||||||
DataType.INT8: default_int8_field_name,
|
DataType.INT8: default_int8_field_name,
|
||||||
DataType.INT16: default_int16_field_name,
|
DataType.INT16: default_int16_field_name,
|
||||||
DataType.INT32: default_int32_field_name,
|
DataType.INT32: default_int32_field_name,
|
||||||
DataType.INT64: default_int64_field_name,
|
DataType.INT64: default_int64_field_name,
|
||||||
DataType.BOOL: default_bool_field_name,
|
DataType.BOOL: default_bool_field_name,
|
||||||
DataType.FLOAT: default_float_field_name,
|
DataType.FLOAT: default_float_field_name,
|
||||||
DataType.DOUBLE: default_double_field_name,
|
DataType.DOUBLE: default_double_field_name,
|
||||||
DataType.VARCHAR: default_string_field_name,
|
DataType.VARCHAR: default_string_field_name,
|
||||||
DataType.ARRAY: default_array_field_name,
|
DataType.ARRAY: default_array_field_name,
|
||||||
DataType.JSON: default_json_field_name,
|
DataType.JSON: default_json_field_name,
|
||||||
DataType.FLOAT_VECTOR: default_float_vec_field_name,
|
DataType.FLOAT_VECTOR: default_float_vec_field_name,
|
||||||
DataType.FLOAT16_VECTOR: default_float16_vec_field_name,
|
DataType.FLOAT16_VECTOR: default_float16_vec_field_name,
|
||||||
DataType.BFLOAT16_VECTOR: default_bfloat16_vec_field_name,
|
DataType.BFLOAT16_VECTOR: default_bfloat16_vec_field_name,
|
||||||
DataType.SPARSE_FLOAT_VECTOR: default_sparse_vec_field_name,
|
DataType.SPARSE_FLOAT_VECTOR: default_sparse_vec_field_name,
|
||||||
DataType.INT8_VECTOR: default_int8_vec_field_name,
|
DataType.INT8_VECTOR: default_int8_vec_field_name,
|
||||||
DataType.BINARY_VECTOR: default_binary_vec_field_name,
|
DataType.BINARY_VECTOR: default_binary_vec_field_name,
|
||||||
}
|
}
|
||||||
|
|
||||||
append_vector_type = [DataType.FLOAT16_VECTOR, DataType.BFLOAT16_VECTOR, DataType.SPARSE_FLOAT_VECTOR, DataType.INT8_VECTOR]
|
append_vector_type = [
|
||||||
all_dense_vector_types = [DataType.FLOAT_VECTOR, DataType.FLOAT16_VECTOR, DataType.BFLOAT16_VECTOR, DataType.INT8_VECTOR]
|
DataType.FLOAT16_VECTOR,
|
||||||
all_float_vector_dtypes = [DataType.FLOAT_VECTOR, DataType.FLOAT16_VECTOR, DataType.BFLOAT16_VECTOR, DataType.SPARSE_FLOAT_VECTOR, DataType.INT8_VECTOR]
|
DataType.BFLOAT16_VECTOR,
|
||||||
|
DataType.SPARSE_FLOAT_VECTOR,
|
||||||
|
DataType.INT8_VECTOR,
|
||||||
|
]
|
||||||
|
all_dense_vector_types = [
|
||||||
|
DataType.FLOAT_VECTOR,
|
||||||
|
DataType.FLOAT16_VECTOR,
|
||||||
|
DataType.BFLOAT16_VECTOR,
|
||||||
|
DataType.INT8_VECTOR,
|
||||||
|
]
|
||||||
|
all_float_vector_dtypes = [
|
||||||
|
DataType.FLOAT_VECTOR,
|
||||||
|
DataType.FLOAT16_VECTOR,
|
||||||
|
DataType.BFLOAT16_VECTOR,
|
||||||
|
DataType.SPARSE_FLOAT_VECTOR,
|
||||||
|
DataType.INT8_VECTOR,
|
||||||
|
]
|
||||||
default_partition_name = "_default"
|
default_partition_name = "_default"
|
||||||
default_resource_group_name = '__default_resource_group'
|
default_resource_group_name = "__default_resource_group"
|
||||||
default_resource_group_capacity = 1000000
|
default_resource_group_capacity = 1000000
|
||||||
default_tag = "1970_01_01"
|
default_tag = "1970_01_01"
|
||||||
row_count = "row_count"
|
row_count = "row_count"
|
||||||
@@ -179,43 +195,37 @@ rows_all_data_type_file_path = "/tmp/rows_all_data_type"
|
|||||||
|
|
||||||
"""" List of parameters used to pass """
|
"""" List of parameters used to pass """
|
||||||
invalid_resource_names = [
|
invalid_resource_names = [
|
||||||
None, # None
|
None, # None
|
||||||
" ", # space
|
" ", # space
|
||||||
"", # empty
|
"", # empty
|
||||||
"12name", # start with number
|
"12name", # start with number
|
||||||
"n12 ame", # contain space
|
"n12 ame", # contain space
|
||||||
"n-ame", # contain hyphen
|
"n-ame", # contain hyphen
|
||||||
"nam(e)", # contain special character
|
"nam(e)", # contain special character
|
||||||
"name中文", # contain Chinese character
|
"name中文", # contain Chinese character
|
||||||
"name%$#", # contain special character
|
"name%$#", # contain special character
|
||||||
"".join("a" for i in range(max_name_length + 1))] # exceed max length
|
"".join("a" for i in range(max_name_length + 1)),
|
||||||
|
] # exceed max length
|
||||||
|
|
||||||
valid_resource_names = [
|
valid_resource_names = [
|
||||||
"name", # valid name
|
"name", # valid name
|
||||||
"_name", # start with underline
|
"_name", # start with underline
|
||||||
"_12name", # start with underline and contains number
|
"_12name", # start with underline and contains number
|
||||||
"n12ame_", # end with letter and contains number and underline
|
"n12ame_", # end with letter and contains number and underline
|
||||||
"nam_e", # contains underline
|
"nam_e", # contains underline
|
||||||
"".join("a" for i in range(max_name_length))] # max length
|
"".join("a" for i in range(max_name_length)),
|
||||||
|
] # max length
|
||||||
|
|
||||||
invalid_dims = [min_dim-1, 32.1, -32, "vii", "十六", max_dim+1]
|
invalid_dims = [min_dim - 1, 32.1, -32, "vii", "十六", max_dim + 1]
|
||||||
|
|
||||||
get_not_string = [
|
get_not_string = [[], {}, None, (1,), 1, 1.0, [1, "2", 3]]
|
||||||
[],
|
|
||||||
{},
|
|
||||||
None,
|
|
||||||
(1,),
|
|
||||||
1,
|
|
||||||
1.0,
|
|
||||||
[1, "2", 3]
|
|
||||||
]
|
|
||||||
|
|
||||||
get_invalid_vectors = [
|
get_invalid_vectors = [
|
||||||
"1*2",
|
"1*2",
|
||||||
[1],
|
[1],
|
||||||
[1, 2],
|
[1, 2],
|
||||||
[" "],
|
[" "],
|
||||||
['a'],
|
["a"],
|
||||||
[None],
|
[None],
|
||||||
None,
|
None,
|
||||||
(1, 2),
|
(1, 2),
|
||||||
@@ -225,7 +235,7 @@ get_invalid_vectors = [
|
|||||||
"String",
|
"String",
|
||||||
" siede ",
|
" siede ",
|
||||||
"中文",
|
"中文",
|
||||||
"a".join("a" for i in range(256))
|
"a".join("a" for i in range(256)),
|
||||||
]
|
]
|
||||||
|
|
||||||
get_invalid_ints = [
|
get_invalid_ints = [
|
||||||
@@ -239,7 +249,7 @@ get_invalid_ints = [
|
|||||||
"String",
|
"String",
|
||||||
"=c",
|
"=c",
|
||||||
"中文",
|
"中文",
|
||||||
"a".join("a" for i in range(256))
|
"a".join("a" for i in range(256)),
|
||||||
]
|
]
|
||||||
|
|
||||||
get_invalid_dict = [
|
get_invalid_dict = [
|
||||||
@@ -254,7 +264,7 @@ get_invalid_dict = [
|
|||||||
{1: 1},
|
{1: 1},
|
||||||
{"中文": 1},
|
{"中文": 1},
|
||||||
{"%$#": ["a"]},
|
{"%$#": ["a"]},
|
||||||
{"a".join("a" for i in range(256)): "a"}
|
{"a".join("a" for i in range(256)): "a"},
|
||||||
]
|
]
|
||||||
|
|
||||||
get_invalid_metric_type = [
|
get_invalid_metric_type = [
|
||||||
@@ -269,55 +279,105 @@ get_invalid_metric_type = [
|
|||||||
"(mn)",
|
"(mn)",
|
||||||
"中文",
|
"中文",
|
||||||
"%$#",
|
"%$#",
|
||||||
"".join("a" for i in range(max_name_length + 1))]
|
"".join("a" for i in range(max_name_length + 1)),
|
||||||
|
|
||||||
get_dict_without_host_port = [
|
|
||||||
{"host": "host"},
|
|
||||||
{"": ""}
|
|
||||||
]
|
]
|
||||||
|
|
||||||
get_wrong_format_dict = [
|
get_dict_without_host_port = [{"host": "host"}, {"": ""}]
|
||||||
{"host": "string_host", "port": {}},
|
|
||||||
{"host": 0, "port": 19520}
|
get_wrong_format_dict = [{"host": "string_host", "port": {}}, {"host": 0, "port": 19520}]
|
||||||
]
|
|
||||||
|
|
||||||
get_all_kind_data_distribution = [
|
get_all_kind_data_distribution = [
|
||||||
1, np.float64(1.0), np.double(1.0), 9707199254740993.0, 9707199254740992,
|
1,
|
||||||
'1', '123', '321', '213', True, False, None, [1, 2], [1.0, 2], {}, {"a": 1},
|
np.float64(1.0),
|
||||||
{'a': 1.0}, {'a': 9707199254740993.0}, {'a': 9707199254740992}, {'a': '1'}, {'a': '123'},
|
np.double(1.0),
|
||||||
{'a': '321'}, {'a': '213'}, {'a': True}, {'a': [1, 2, 3]}, {'a': [1.0, 2, '1']}, {'a': [1.0, 2]},
|
9707199254740993.0,
|
||||||
{'a': None}, {'a': {'b': 1}}, {'a': {'b': 1.0}}, {'a': [{'b': 1}, 2.0, np.double(3.0), '4', True, [1, 3.0], None]}
|
9707199254740992,
|
||||||
|
"1",
|
||||||
|
"123",
|
||||||
|
"321",
|
||||||
|
"213",
|
||||||
|
True,
|
||||||
|
False,
|
||||||
|
None,
|
||||||
|
[1, 2],
|
||||||
|
[1.0, 2],
|
||||||
|
{},
|
||||||
|
{"a": 1},
|
||||||
|
{"a": 1.0},
|
||||||
|
{"a": 9707199254740993.0},
|
||||||
|
{"a": 9707199254740992},
|
||||||
|
{"a": "1"},
|
||||||
|
{"a": "123"},
|
||||||
|
{"a": "321"},
|
||||||
|
{"a": "213"},
|
||||||
|
{"a": True},
|
||||||
|
{"a": [1, 2, 3]},
|
||||||
|
{"a": [1.0, 2, "1"]},
|
||||||
|
{"a": [1.0, 2]},
|
||||||
|
{"a": None},
|
||||||
|
{"a": {"b": 1}},
|
||||||
|
{"a": {"b": 1.0}},
|
||||||
|
{"a": [{"b": 1}, 2.0, np.double(3.0), "4", True, [1, 3.0], None]},
|
||||||
]
|
]
|
||||||
|
|
||||||
""" Specially defined list """
|
""" Specially defined list """
|
||||||
L0_index_types = ["IVF_SQ8", "HNSW", "DISKANN"]
|
L0_index_types = ["IVF_SQ8", "HNSW", "DISKANN"]
|
||||||
all_index_types = ["FLAT", "IVF_FLAT", "IVF_SQ8", "IVF_PQ",
|
all_index_types = [
|
||||||
"IVF_RABITQ",
|
"FLAT",
|
||||||
"HNSW", "SCANN", "DISKANN",
|
"IVF_FLAT",
|
||||||
"BIN_FLAT", "BIN_IVF_FLAT",
|
"IVF_SQ8",
|
||||||
"SPARSE_INVERTED_INDEX", "SPARSE_WAND",
|
"IVF_PQ",
|
||||||
"GPU_IVF_FLAT", "GPU_IVF_PQ"]
|
"IVF_RABITQ",
|
||||||
|
"HNSW",
|
||||||
|
"SCANN",
|
||||||
|
"DISKANN",
|
||||||
|
"BIN_FLAT",
|
||||||
|
"BIN_IVF_FLAT",
|
||||||
|
"SPARSE_INVERTED_INDEX",
|
||||||
|
"SPARSE_WAND",
|
||||||
|
"GPU_IVF_FLAT",
|
||||||
|
"GPU_IVF_PQ",
|
||||||
|
]
|
||||||
|
|
||||||
all_dense_float_index_types = ["FLAT", "IVF_FLAT", "IVF_SQ8", "IVF_PQ",
|
all_dense_float_index_types = ["FLAT", "IVF_FLAT", "IVF_SQ8", "IVF_PQ", "IVF_RABITQ", "HNSW", "SCANN", "DISKANN"]
|
||||||
"IVF_RABITQ", "HNSW", "SCANN", "DISKANN"]
|
|
||||||
|
|
||||||
inverted_index_algo = ['TAAT_NAIVE', 'DAAT_WAND', 'DAAT_MAXSCORE']
|
inverted_index_algo = ["TAAT_NAIVE", "DAAT_WAND", "DAAT_MAXSCORE"]
|
||||||
|
|
||||||
int8_vector_index = ["HNSW"]
|
int8_vector_index = ["HNSW"]
|
||||||
|
|
||||||
default_all_indexes_params = [{}, {"nlist": 128}, {"nlist": 128}, {"nlist": 128, "m": 16, "nbits": 8},
|
default_all_indexes_params = [
|
||||||
{"nlist": 128, "refine": 'true', "refine_type": "SQ8"},
|
{},
|
||||||
{"M": 32, "efConstruction": 360}, {"nlist": 128}, {},
|
{"nlist": 128},
|
||||||
{}, {"nlist": 64},
|
{"nlist": 128},
|
||||||
{}, {"drop_ratio_build": 0.2},
|
{"nlist": 128, "m": 16, "nbits": 8},
|
||||||
{"nlist": 64}, {"nlist": 64, "m": 16, "nbits": 8}]
|
{"nlist": 128, "refine": "true", "refine_type": "SQ8"},
|
||||||
|
{"M": 32, "efConstruction": 360},
|
||||||
|
{"nlist": 128},
|
||||||
|
{},
|
||||||
|
{},
|
||||||
|
{"nlist": 64},
|
||||||
|
{},
|
||||||
|
{"drop_ratio_build": 0.2},
|
||||||
|
{"nlist": 64},
|
||||||
|
{"nlist": 64, "m": 16, "nbits": 8},
|
||||||
|
]
|
||||||
|
|
||||||
default_all_search_params_params = [{}, {"nprobe": 32}, {"nprobe": 32}, {"nprobe": 32},
|
default_all_search_params_params = [
|
||||||
{"nprobe": 8, "rbq_bits_query": 8, "refine_k": 10.0},
|
{},
|
||||||
{"ef": 100}, {"nprobe": 32, "reorder_k": 100}, {"search_list": 30},
|
{"nprobe": 32},
|
||||||
{}, {"nprobe": 32},
|
{"nprobe": 32},
|
||||||
{"drop_ratio_search": "0.2"}, {"drop_ratio_search": "0.2"},
|
{"nprobe": 32},
|
||||||
{}, {}]
|
{"nprobe": 8, "rbq_bits_query": 8, "refine_k": 10.0},
|
||||||
|
{"ef": 100},
|
||||||
|
{"nprobe": 32, "reorder_k": 100},
|
||||||
|
{"search_list": 30},
|
||||||
|
{},
|
||||||
|
{"nprobe": 32},
|
||||||
|
{"drop_ratio_search": "0.2"},
|
||||||
|
{"drop_ratio_search": "0.2"},
|
||||||
|
{},
|
||||||
|
{},
|
||||||
|
]
|
||||||
|
|
||||||
Handler_type = ["GRPC", "HTTP"]
|
Handler_type = ["GRPC", "HTTP"]
|
||||||
binary_supported_index_types = ["BIN_FLAT", "BIN_IVF_FLAT"]
|
binary_supported_index_types = ["BIN_FLAT", "BIN_IVF_FLAT"]
|
||||||
@@ -336,10 +396,20 @@ numeric_supported_index_types = ["STL_SORT", "INVERTED", "AUTOINDEX", ""]
|
|||||||
|
|
||||||
default_flat_index = {"index_type": "FLAT", "params": {}, "metric_type": default_L0_metric}
|
default_flat_index = {"index_type": "FLAT", "params": {}, "metric_type": default_L0_metric}
|
||||||
default_bin_flat_index = {"index_type": "BIN_FLAT", "params": {}, "metric_type": "JACCARD"}
|
default_bin_flat_index = {"index_type": "BIN_FLAT", "params": {}, "metric_type": "JACCARD"}
|
||||||
default_sparse_inverted_index = {"index_type": "SPARSE_INVERTED_INDEX", "metric_type": "IP",
|
default_sparse_inverted_index = {
|
||||||
"params": {"drop_ratio_build": 0.2}}
|
"index_type": "SPARSE_INVERTED_INDEX",
|
||||||
default_text_sparse_inverted_index = {"index_type": "SPARSE_INVERTED_INDEX", "metric_type": "BM25",
|
"metric_type": "IP",
|
||||||
"params": {"drop_ratio_build": 0.2, "bm25_k1": 1.5, "bm25_b": 0.75,}}
|
"params": {"drop_ratio_build": 0.2},
|
||||||
|
}
|
||||||
|
default_text_sparse_inverted_index = {
|
||||||
|
"index_type": "SPARSE_INVERTED_INDEX",
|
||||||
|
"metric_type": "BM25",
|
||||||
|
"params": {
|
||||||
|
"drop_ratio_build": 0.2,
|
||||||
|
"bm25_k1": 1.5,
|
||||||
|
"bm25_b": 0.75,
|
||||||
|
},
|
||||||
|
}
|
||||||
default_search_params = {"params": {"nlist": 128}}
|
default_search_params = {"params": {"nlist": 128}}
|
||||||
default_search_ip_params = {"metric_type": "IP", "params": {"nlist": 128}}
|
default_search_ip_params = {"metric_type": "IP", "params": {"nlist": 128}}
|
||||||
default_search_binary_params = {"metric_type": "JACCARD", "params": {"nprobe": 32}}
|
default_search_binary_params = {"metric_type": "JACCARD", "params": {"nprobe": 32}}
|
||||||
@@ -349,46 +419,108 @@ default_diskann_index = {"index_type": "DISKANN", "metric_type": default_L0_metr
|
|||||||
default_diskann_search_params = {"params": {"search_list": 30}}
|
default_diskann_search_params = {"params": {"search_list": 30}}
|
||||||
default_sparse_search_params = {"metric_type": "IP", "params": {"drop_ratio_search": "0.2"}}
|
default_sparse_search_params = {"metric_type": "IP", "params": {"drop_ratio_search": "0.2"}}
|
||||||
default_text_sparse_search_params = {"metric_type": "BM25", "params": {}}
|
default_text_sparse_search_params = {"metric_type": "BM25", "params": {}}
|
||||||
built_in_privilege_groups = ["CollectionReadWrite", "CollectionReadOnly", "CollectionAdmin",
|
built_in_privilege_groups = [
|
||||||
"DatabaseReadWrite", "DatabaseReadOnly", "DatabaseAdmin",
|
"CollectionReadWrite",
|
||||||
"ClusterReadWrite", "ClusterReadOnly", "ClusterAdmin"]
|
"CollectionReadOnly",
|
||||||
privilege_group_privilege_dict = {"Query": False, "Search": False, "GetLoadState": False,
|
"CollectionAdmin",
|
||||||
"GetLoadingProgress": False, "HasPartition": False, "ShowPartitions": False,
|
"DatabaseReadWrite",
|
||||||
"ShowCollections": False, "ListAliases": False, "ListDatabases": False,
|
"DatabaseReadOnly",
|
||||||
"DescribeDatabase": False, "DescribeAlias": False, "GetStatistics": False,
|
"DatabaseAdmin",
|
||||||
"CreateIndex": False, "DropIndex": False, "CreatePartition": False,
|
"ClusterReadWrite",
|
||||||
"DropPartition": False, "Load": False, "Release": False,
|
"ClusterReadOnly",
|
||||||
"Insert": False, "Delete": False, "Upsert": False,
|
"ClusterAdmin",
|
||||||
"Import": False, "Flush": False, "Compaction": False,
|
]
|
||||||
"LoadBalance": False, "RenameCollection": False, "CreateAlias": False,
|
privilege_group_privilege_dict = {
|
||||||
"DropAlias": False, "CreateCollection": False, "DropCollection": False,
|
"Query": False,
|
||||||
"CreateOwnership": False, "DropOwnership": False, "SelectOwnership": False,
|
"Search": False,
|
||||||
"ManageOwnership": False, "UpdateUser": False, "SelectUser": False,
|
"GetLoadState": False,
|
||||||
"CreateResourceGroup": False, "DropResourceGroup": False,
|
"GetLoadingProgress": False,
|
||||||
"UpdateResourceGroups": False,
|
"HasPartition": False,
|
||||||
"DescribeResourceGroup": False, "ListResourceGroups": False, "TransferNode": False,
|
"ShowPartitions": False,
|
||||||
"TransferReplica": False, "CreateDatabase": False, "DropDatabase": False,
|
"ShowCollections": False,
|
||||||
"AlterDatabase": False, "FlushAll": False, "ListPrivilegeGroups": False,
|
"ListAliases": False,
|
||||||
"CreatePrivilegeGroup": False, "DropPrivilegeGroup": False,
|
"ListDatabases": False,
|
||||||
"OperatePrivilegeGroup": False}
|
"DescribeDatabase": False,
|
||||||
all_expr_fields = [default_int8_field_name, default_int16_field_name,
|
"DescribeAlias": False,
|
||||||
default_int32_field_name, default_int64_field_name,
|
"GetStatistics": False,
|
||||||
default_float_field_name, default_double_field_name,
|
"CreateIndex": False,
|
||||||
default_string_field_name, default_bool_field_name,
|
"DropIndex": False,
|
||||||
default_int8_array_field_name, default_int16_array_field_name,
|
"CreatePartition": False,
|
||||||
default_int32_array_field_name, default_int64_array_field_name,
|
"DropPartition": False,
|
||||||
default_bool_array_field_name, default_float_array_field_name,
|
"Load": False,
|
||||||
default_double_array_field_name, default_string_array_field_name]
|
"Release": False,
|
||||||
|
"Insert": False,
|
||||||
|
"Delete": False,
|
||||||
|
"Upsert": False,
|
||||||
|
"Import": False,
|
||||||
|
"Flush": False,
|
||||||
|
"Compaction": False,
|
||||||
|
"LoadBalance": False,
|
||||||
|
"RenameCollection": False,
|
||||||
|
"CreateAlias": False,
|
||||||
|
"DropAlias": False,
|
||||||
|
"CreateCollection": False,
|
||||||
|
"DropCollection": False,
|
||||||
|
"CreateOwnership": False,
|
||||||
|
"DropOwnership": False,
|
||||||
|
"SelectOwnership": False,
|
||||||
|
"ManageOwnership": False,
|
||||||
|
"UpdateUser": False,
|
||||||
|
"SelectUser": False,
|
||||||
|
"CreateResourceGroup": False,
|
||||||
|
"DropResourceGroup": False,
|
||||||
|
"UpdateResourceGroups": False,
|
||||||
|
"DescribeResourceGroup": False,
|
||||||
|
"ListResourceGroups": False,
|
||||||
|
"TransferNode": False,
|
||||||
|
"TransferReplica": False,
|
||||||
|
"CreateDatabase": False,
|
||||||
|
"DropDatabase": False,
|
||||||
|
"AlterDatabase": False,
|
||||||
|
"FlushAll": False,
|
||||||
|
"ListPrivilegeGroups": False,
|
||||||
|
"CreatePrivilegeGroup": False,
|
||||||
|
"DropPrivilegeGroup": False,
|
||||||
|
"OperatePrivilegeGroup": False,
|
||||||
|
}
|
||||||
|
all_expr_fields = [
|
||||||
|
default_int8_field_name,
|
||||||
|
default_int16_field_name,
|
||||||
|
default_int32_field_name,
|
||||||
|
default_int64_field_name,
|
||||||
|
default_float_field_name,
|
||||||
|
default_double_field_name,
|
||||||
|
default_string_field_name,
|
||||||
|
default_bool_field_name,
|
||||||
|
default_int8_array_field_name,
|
||||||
|
default_int16_array_field_name,
|
||||||
|
default_int32_array_field_name,
|
||||||
|
default_int64_array_field_name,
|
||||||
|
default_bool_array_field_name,
|
||||||
|
default_float_array_field_name,
|
||||||
|
default_double_array_field_name,
|
||||||
|
default_string_array_field_name,
|
||||||
|
]
|
||||||
|
|
||||||
|
not_supported_json_cast_types = [
|
||||||
|
DataType.INT8.name,
|
||||||
|
DataType.INT16.name,
|
||||||
|
DataType.INT32.name,
|
||||||
|
DataType.INT64.name,
|
||||||
|
DataType.FLOAT.name,
|
||||||
|
DataType.ARRAY.name,
|
||||||
|
DataType.FLOAT_VECTOR.name,
|
||||||
|
DataType.FLOAT16_VECTOR.name,
|
||||||
|
DataType.BFLOAT16_VECTOR.name,
|
||||||
|
DataType.BINARY_VECTOR.name,
|
||||||
|
DataType.SPARSE_FLOAT_VECTOR.name,
|
||||||
|
DataType.INT8_VECTOR.name,
|
||||||
|
]
|
||||||
|
|
||||||
not_supported_json_cast_types = [DataType.INT8.name, DataType.INT16.name, DataType.INT32.name,
|
|
||||||
DataType.INT64.name, DataType.FLOAT.name,
|
|
||||||
DataType.ARRAY.name, DataType.FLOAT_VECTOR.name,
|
|
||||||
DataType.FLOAT16_VECTOR.name, DataType.BFLOAT16_VECTOR.name,
|
|
||||||
DataType.BINARY_VECTOR.name,
|
|
||||||
DataType.SPARSE_FLOAT_VECTOR.name, DataType.INT8_VECTOR.name]
|
|
||||||
|
|
||||||
class CheckTasks:
|
class CheckTasks:
|
||||||
""" The name of the method used to check the result """
|
"""The name of the method used to check the result"""
|
||||||
|
|
||||||
check_nothing = "check_nothing"
|
check_nothing = "check_nothing"
|
||||||
err_res = "error_response"
|
err_res = "error_response"
|
||||||
ccr = "check_connection_result"
|
ccr = "check_connection_result"
|
||||||
@@ -452,6 +584,7 @@ class CaseLabel:
|
|||||||
GPU:
|
GPU:
|
||||||
For GPU supported cases
|
For GPU supported cases
|
||||||
"""
|
"""
|
||||||
|
|
||||||
L0 = "L0"
|
L0 = "L0"
|
||||||
L1 = "L1"
|
L1 = "L1"
|
||||||
L2 = "L2"
|
L2 = "L2"
|
||||||
|
|||||||
@@ -1,16 +1,3 @@
|
|||||||
[project]
|
|
||||||
name = "milvus-tests"
|
|
||||||
version = "0.0.0"
|
|
||||||
description = "Milvus integration/e2e test suite tooling (ruff config host)"
|
|
||||||
requires-python = ">=3.10"
|
|
||||||
dependencies = []
|
|
||||||
|
|
||||||
[dependency-groups]
|
|
||||||
dev = [
|
|
||||||
"ruff>=0.6.0",
|
|
||||||
]
|
|
||||||
|
|
||||||
[tool.ruff]
|
|
||||||
line-length = 120
|
line-length = 120
|
||||||
target-version = "py310"
|
target-version = "py310"
|
||||||
extend-exclude = [
|
extend-exclude = [
|
||||||
@@ -20,7 +7,7 @@ extend-exclude = [
|
|||||||
"*.ipynb_checkpoints",
|
"*.ipynb_checkpoints",
|
||||||
]
|
]
|
||||||
|
|
||||||
[tool.ruff.lint]
|
[lint]
|
||||||
select = [
|
select = [
|
||||||
"E", # pycodestyle errors
|
"E", # pycodestyle errors
|
||||||
"F", # pyflakes
|
"F", # pyflakes
|
||||||
@@ -33,13 +20,13 @@ ignore = [
|
|||||||
"E741", # ambiguous variable names (l, I, O) common in test code
|
"E741", # ambiguous variable names (l, I, O) common in test code
|
||||||
]
|
]
|
||||||
|
|
||||||
[tool.ruff.lint.per-file-ignores]
|
[lint.per-file-ignores]
|
||||||
"__init__.py" = ["F401", "F403"]
|
"__init__.py" = ["F401", "F403"]
|
||||||
"conftest.py" = ["F401", "F811"]
|
"conftest.py" = ["F401", "F811"]
|
||||||
"**/testcases/**/*.py" = ["E402"]
|
"**/testcases/**/*.py" = ["E402"]
|
||||||
"**/chaos/**/*.py" = ["E402"]
|
"**/chaos/**/*.py" = ["E402"]
|
||||||
|
|
||||||
[tool.ruff.format]
|
[format]
|
||||||
quote-style = "double"
|
quote-style = "double"
|
||||||
indent-style = "space"
|
indent-style = "space"
|
||||||
line-ending = "auto"
|
line-ending = "auto"
|
||||||
Generated
-43
@@ -1,43 +0,0 @@
|
|||||||
version = 1
|
|
||||||
revision = 3
|
|
||||||
requires-python = ">=3.10"
|
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "milvus-tests"
|
|
||||||
version = "0.0.0"
|
|
||||||
source = { virtual = "." }
|
|
||||||
|
|
||||||
[package.dev-dependencies]
|
|
||||||
dev = [
|
|
||||||
{ name = "ruff" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[package.metadata]
|
|
||||||
|
|
||||||
[package.metadata.requires-dev]
|
|
||||||
dev = [{ name = "ruff", specifier = ">=0.6.0" }]
|
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "ruff"
|
|
||||||
version = "0.15.11"
|
|
||||||
source = { registry = "https://pypi.org/simple" }
|
|
||||||
sdist = { url = "https://files.pythonhosted.org/packages/e4/8d/192f3d7103816158dfd5ea50d098ef2aec19194e6cbccd4b3485bdb2eb2d/ruff-0.15.11.tar.gz", hash = "sha256:f092b21708bf0e7437ce9ada249dfe688ff9a0954fc94abab05dcea7dcd29c33", size = 4637264, upload-time = "2026-04-16T18:46:26.58Z" }
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://files.pythonhosted.org/packages/02/1e/6aca3427f751295ab011828e15e9bf452200ac74484f1db4be0197b8170b/ruff-0.15.11-py3-none-linux_armv6l.whl", hash = "sha256:e927cfff503135c558eb581a0c9792264aae9507904eb27809cdcff2f2c847b7", size = 10607943, upload-time = "2026-04-16T18:46:05.967Z" },
|
|
||||||
{ url = "https://files.pythonhosted.org/packages/e7/26/1341c262e74f36d4e84f3d6f4df0ac68cd53331a66bfc5080daa17c84c0b/ruff-0.15.11-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:7a1b5b2938d8f890b76084d4fa843604d787a912541eae85fd7e233398bbb73e", size = 10988592, upload-time = "2026-04-16T18:46:00.742Z" },
|
|
||||||
{ url = "https://files.pythonhosted.org/packages/03/71/850b1d6ffa9564fbb6740429bad53df1094082fe515c8c1e74b6d8d05f18/ruff-0.15.11-py3-none-macosx_11_0_arm64.whl", hash = "sha256:d4176f3d194afbdaee6e41b9ccb1a2c287dba8700047df474abfbe773825d1cb", size = 10338501, upload-time = "2026-04-16T18:46:03.723Z" },
|
|
||||||
{ url = "https://files.pythonhosted.org/packages/f2/11/cc1284d3e298c45a817a6aadb6c3e1d70b45c9b36d8d9cce3387b495a03a/ruff-0.15.11-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:3b17c886fb88203ced3afe7f14e8d5ae96e9d2f4ccc0ee66aa19f2c2675a27e4", size = 10670693, upload-time = "2026-04-16T18:46:41.941Z" },
|
|
||||||
{ url = "https://files.pythonhosted.org/packages/ce/9e/f8288b034ab72b371513c13f9a41d9ba3effac54e24bfb467b007daee2ca/ruff-0.15.11-py3-none-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:49fafa220220afe7758a487b048de4c8f9f767f37dfefad46b9dd06759d003eb", size = 10416177, upload-time = "2026-04-16T18:46:21.717Z" },
|
|
||||||
{ url = "https://files.pythonhosted.org/packages/85/71/504d79abfd3d92532ba6bbe3d1c19fada03e494332a59e37c7c2dabae427/ruff-0.15.11-py3-none-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:f2ab8427e74a00d93b8bda1307b1e60970d40f304af38bccb218e056c220120d", size = 11221886, upload-time = "2026-04-16T18:46:15.086Z" },
|
|
||||||
{ url = "https://files.pythonhosted.org/packages/43/5a/947e6ab7a5ad603d65b474be15a4cbc6d29832db5d762cd142e4e3a74164/ruff-0.15.11-py3-none-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:195072c0c8e1fc8f940652073df082e37a5d9cb43b4ab1e4d0566ab8977a13b7", size = 12075183, upload-time = "2026-04-16T18:46:07.944Z" },
|
|
||||||
{ url = "https://files.pythonhosted.org/packages/9f/a1/0b7bb6268775fdd3a0818aee8efd8f5b4e231d24dd4d528ced2534023182/ruff-0.15.11-py3-none-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:a3a0996d486af3920dec930a2e7daed4847dfc12649b537a9335585ada163e9e", size = 11516575, upload-time = "2026-04-16T18:46:31.687Z" },
|
|
||||||
{ url = "https://files.pythonhosted.org/packages/30/c3/bb5168fc4d233cc06e95f482770d0f3c87945a0cd9f614b90ea8dc2f2833/ruff-0.15.11-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:1bef2cb556d509259f1fe440bb9cd33c756222cf0a7afe90d15edf0866702431", size = 11306537, upload-time = "2026-04-16T18:46:36.988Z" },
|
|
||||||
{ url = "https://files.pythonhosted.org/packages/e4/92/4cfae6441f3967317946f3b788136eecf093729b94d6561f963ed810c82e/ruff-0.15.11-py3-none-manylinux_2_31_riscv64.whl", hash = "sha256:030d921a836d7d4a12cf6e8d984a88b66094ccb0e0f17ddd55067c331191bf19", size = 11296813, upload-time = "2026-04-16T18:46:24.182Z" },
|
|
||||||
{ url = "https://files.pythonhosted.org/packages/43/26/972784c5dde8313acde8ac71ba8ac65475b85db4a2352a76c9934361f9bc/ruff-0.15.11-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:0e783b599b4577788dbbb66b9addcef87e9a8832f4ce0c19e34bf55543a2f890", size = 10633136, upload-time = "2026-04-16T18:46:39.802Z" },
|
|
||||||
{ url = "https://files.pythonhosted.org/packages/5b/53/3985a4f185020c2f367f2e08a103032e12564829742a1b417980ce1514a0/ruff-0.15.11-py3-none-musllinux_1_2_armv7l.whl", hash = "sha256:ae90592246625ba4a34349d68ec28d4400d75182b71baa196ddb9f82db025ef5", size = 10424701, upload-time = "2026-04-16T18:46:10.381Z" },
|
|
||||||
{ url = "https://files.pythonhosted.org/packages/d3/57/bf0dfb32241b56c83bb663a826133da4bf17f682ba8c096973065f6e6a68/ruff-0.15.11-py3-none-musllinux_1_2_i686.whl", hash = "sha256:1f111d62e3c983ed20e0ca2e800f8d77433a5b1161947df99a5c2a3fb60514f0", size = 10873887, upload-time = "2026-04-16T18:46:29.157Z" },
|
|
||||||
{ url = "https://files.pythonhosted.org/packages/02/05/e48076b2a57dc33ee8c7a957296f97c744ca891a8ffb4ffb1aaa3b3f517d/ruff-0.15.11-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:06f483d6646f59eaffba9ae30956370d3a886625f511a3108994000480621d1c", size = 11404316, upload-time = "2026-04-16T18:46:19.462Z" },
|
|
||||||
{ url = "https://files.pythonhosted.org/packages/88/27/0195d15fe7a897cbcba0904792c4b7c9fdd958456c3a17d2ea6093716a9a/ruff-0.15.11-py3-none-win32.whl", hash = "sha256:476a2aa56b7da0b73a3ee80b6b2f0e19cce544245479adde7baa65466664d5f3", size = 10655535, upload-time = "2026-04-16T18:46:12.47Z" },
|
|
||||||
{ url = "https://files.pythonhosted.org/packages/3a/5e/c927b325bd4c1d3620211a4b96f47864633199feed60fa936025ab27e090/ruff-0.15.11-py3-none-win_amd64.whl", hash = "sha256:8b6756d88d7e234fb0c98c91511aae3cd519d5e3ed271cae31b20f39cb2a12a3", size = 11779692, upload-time = "2026-04-16T18:46:17.268Z" },
|
|
||||||
{ url = "https://files.pythonhosted.org/packages/63/b6/aeadee5443e49baa2facd51131159fd6301cc4ccfc1541e4df7b021c37dd/ruff-0.15.11-py3-none-win_arm64.whl", hash = "sha256:063fed18cc1bbe0ee7393957284a6fe8b588c6a406a285af3ee3f46da2391ee4", size = 11032614, upload-time = "2026-04-16T18:46:34.487Z" },
|
|
||||||
]
|
|
||||||
Reference in New Issue
Block a user