wei liuandGitHub
3ae4cb28f5
issue: #49335, #48830, #49338, #49334, #49292, #49336 ## Summary This PR delivers a stability hardening pack for external table lifecycle and spec handling. ### 1) Alter/Refresh metadata correctness - Guard invalid tuple transitions in alter-collection path. - Persist refreshed external tuple on refresh completion. - Disable unsupported `TruncateCollection` on external collections. ### 2) ExternalSpec safety and contract clarity - Require explicit `cloud_provider`; remove scheme-based inference. - Redact credentials from `DescribeCollection`. - Reserve system field names (including `__virtual_pk__`) from user schema. ### 3) Runtime robustness and edge-case handling - Skip anonymous entries in extfs Layer0 zero-init. - Align DataNode explore-manifest sort + format-filter behavior. - Prevent datanode crash on zero-row parquet. - Surface real sampling error in `RefreshFailed`. - Improve compatibility for vortex schemaless view and large Arrow type variants. ## Why External-table alter/refresh flows had validation and persistence gaps, plus multiple edge-case failures in runtime paths. This PR closes those gaps to make behavior safer, deterministic, and debuggable. ## Test Plan - Unit: - `pkg/util/externalspec/external_spec_test.go` - `internal/rootcoord/create_collection_task_test.go` - `internal/rootcoord/ddl_callbacks_alter_collection_properties_test.go` - `internal/core/unittest/test_external_take.cpp` - E2E / go client: - `tests/go_client/testcases/external_table_test.go` - `tests/go_client/testcases/external_table_iceberg_e2e_test.go` --------- Signed-off-by: wei liu <wei.liu@zilliz.com> Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Tests
E2E Test
Configuration Requirements
Operating System
| Operating System | Version |
|---|---|
| Amazon Linux | 2023 or above |
| Ubuntu | 20.04 or above |
| Mac | 10.14 or above |
Hardware
| Hardware Type | Recommended Configuration |
|---|---|
| CPU | x86_64 architecture Intel CPU Sandy Bridge or above CPU Instruction Set - SSE4_2 - AVX - AVX2 - AVX512 or arm64 Linux/MacOS |
| Memory | 16 GB or more |
Software
| Software Name | Version |
|---|---|
| Docker | 19.05 or above |
| Docker Compose | 1.25.5 or above |
| jq | 1.3 or above |
| kubectl | 1.14 or above |
| helm | 3.0 or above |
| kind | 0.10.0 or above |
Installing Dependencies
Troubleshooting Docker and Docker Compose
- Confirm that Docker Daemon is running：
$ docker info
-
Ensure that Docker is installed. Refer to the official installation instructions for Docker CE/EE.
-
Start the Docker Daemon if it is not already started.
-
To run Docker without
rootprivileges, create a user group labeleddocker, then add a user to the group withsudo usermod -aG docker $USER. Log out and log back into the terminal for the changes to take effect. For more information, see the official Docker documentation for Managing Docker as a Non-Root User.
- Check the version of Docker-Compose
$ docker compose version
docker compose version 1.25.5, build 8a1c60f6
docker-py version: 4.1.0
CPython version: 3.7.5
OpenSSL version: OpenSSL 1.1.1f 31 Mar 2020
- To install Docker-Compose, see Install Docker Compose
Install jq
Install kubectl
Install helm
- Refer to https://helm.sh/docs/intro/install/
Install kind
Run E2E Tests
$ cd tests/scripts
$ ./e2e-k8s.sh
Getting help
You can get help with the following command:
$ ./e2e-k8s.sh --help
Python Code Quality (ruff via uv)
Ruff is configured at tests/pyproject.toml and covers all Python code under tests/
(python_client/, restful_client/, restful_client_v2/, benchmark/, scripts/).
uv is only used to host the lint/format toolchain; each
sub-directory continues to manage its runtime dependencies via its own requirements.txt.
$ cd tests/
$ uv sync # install ruff into a local .venv
$ uv run ruff check . # lint
$ uv run ruff check . --fix # lint with auto-fix
$ uv run ruff format . # format in place
$ uv run ruff format --check . # format check only (CI-friendly)
Rules enabled: E, F, W, I, UP. Target Python version: 3.10.
Lint only PR-changed files (Python Lint CI parity)
The GitHub Actions workflow .github/workflows/python-lint.yaml (job
"Python Lint (tests/) / Ruff (changed files only)") runs ruff check
and ruff format --check against the changed tests/**/*.py set on
every PR. Running uv run ruff check . over the whole tree is too coarse
because the historical contents of many files predate the lint config and
will fail unrelated rules.
To reproduce the CI step locally, use the Makefile shipped in this directory:
$ cd tests/
$ make ci # ruff check + format --check on PR-changed *.py (CI equivalent)
$ make lint-fix # ruff check --fix on PR-changed *.py
$ make format # ruff format on PR-changed *.py
$ make help # show all targets and the detected BASE_REF
BASE_REF is auto-detected from the current branch's open PR via gh pr view:
the PR URL is parsed to discover the base <owner>/<repo> and matched against
your local git remotes, producing e.g. upstream/master, upstream/2.x, or
origin/main for direct clones.
If no open PR exists for the branch, you must set BASE_REF explicitly
(no guessing — a wrong base diffs against unrelated commits):
$ make ci BASE_REF=upstream/master
Requires uv (provides uvx) and gh authenticated against GitHub
(gh auth status). The ruff version is pinned in the Makefile via
RUFF_VERSION to match the workflow.