commit 48d1b4e443dde0b3478a52e8f8b11d0c464c603f Author: Ethan Date: Tue Aug 4 14:34:47 2026 +0800 Initial commit: add NocoBase collection migration tools diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..e43b0f9 --- /dev/null +++ b/.gitignore @@ -0,0 +1 @@ +.DS_Store diff --git a/docs/collection-schema-import-export.md b/docs/collection-schema-import-export.md new file mode 100644 index 0000000..00c1fc1 --- /dev/null +++ b/docs/collection-schema-import-export.md @@ -0,0 +1,405 @@ +# NocoBase Collection 结构导入导出脚本使用说明 + +本文说明如何使用项目中的两个脚本，在 NocoBase 环境之间导出、检查和导入主数据源的 Collection 结构。 + +- 导出脚本：`script/export-collection-schema.sh` +- 导入脚本：`script/import-collection-schema.sh` + +## 1. 适用范围 + +脚本迁移以下内容： + +- 主数据源 `main` 中的 Collection 定义； +- Collection 的普通字段和关系字段； +- Collection 分类及分类成员关系； +- Collection 和字段中可移植的配置属性。 + +脚本不迁移以下内容： + +- Collection 中的业务数据记录； +- 非主数据源中的 Collection； +- 页面、区块和菜单配置； +- 角色、权限和用户授权； +- 工作流、自动化和通知配置； +- 插件配置及环境变量。 + +导入脚本只接受由导出脚本生成的、`profile` 为 `portable` 且 `dataSource` 为 `main` 的导出包。 + +## 2. 运行前准备 + +请从项目根目录执行命令，并确认本机已安装： + +- NocoBase CLI：`nb` +- JSON 工具：`jq` + +检查命令： + +```sh +command -v nb +command -v jq +``` + +同时需要满足以下条件： + +1. 源环境和目标环境已经配置为可用的 NocoBase CLI 环境； +2. 当前 CLI 身份可以读取源环境的数据模型； +3. 正式导入时，当前 CLI 身份可以在目标环境创建或更新 Collection、字段和 Collection 分类； +4. 目标环境应安装导出结构所依赖的字段接口或插件； +5. 关系字段引用的外部 Collection 必须已经存在于目标环境。 + +可分别查看两个脚本的内置帮助： + +```sh +./script/export-collection-schema.sh --help +./script/import-collection-schema.sh --help +``` + +## 3. 推荐操作流程 + +建议始终按以下顺序执行： + +1. 从源环境导出 Collection 结构； +2. 检查导出目录中的 `manifest.json` 和 Collection JSON 文件； +3. 对目标环境执行导入预演，不加 `--apply`； +4. 核对预演输出中的环境、Collection 数量、字段数量、分类和外部关系目标； +5. 确认无误后使用 `--apply` 和 `--confirm-env` 正式导入； +6. 检查脚本最终输出的 `verified: true`。 + +## 4. 导出 Collection 结构 + +### 4.1 命令格式 + +```sh +./script/export-collection-schema.sh \ + --env <源环境名称> \ + --output-dir <导出目录> \ + [--category ] \ + [--no-overwrite] +``` + +### 4.2 参数说明 + +| 参数 | 必填 | 说明 | +| --- | --- | --- | +| `--env` | 是 | NocoBase CLI 中已配置的源环境名称。 | +| `--output-dir` | 是 | 导出包保存目录。目录不存在时会自动创建。 | +| `--category` | 否 | 只导出指定 Collection 分类。必须与 NocoBase 中的分类名称完全一致；不指定时导出全部分类及未分类 Collection。 | +| `--no-overwrite` | 否 | 如果输出目录已经存在则终止，不覆盖原目录。 | +| `-h`、`--help` | 否 | 显示帮助信息。 | + +数据源无需指定。导出脚本固定读取主数据源 `main`，也不接受 `--data-source` 参数。 + +### 4.3 导出全部 Collection + +```sh +./script/export-collection-schema.sh \ + --env master-data-local \ + --output-dir ./exports/all-collection-schema +``` + +未指定 `--category` 时： + +- 脚本读取主数据源中的全部 Collection； +- 已加入分类的 Collection 按分类写入导出目录； +- 未加入任何分类的 Collection 写入伪分类目录 `未分类`； +- 同一个 Collection 属于多个分类时，会在多个分类目录中生成相同的结构文件，以保留全部分类关系。 + +### 4.4 按分类导出 + +```sh +./script/export-collection-schema.sh \ + --env master-data-local \ + --category 参考数据 \ + --output-dir ./exports/reference-schema +``` + +导出未加入分类的 Collection： + +```sh +./script/export-collection-schema.sh \ + --env master-data-local \ + --category 未分类 \ + --output-dir ./exports/unclassified-schema +``` + +如果分类名称不存在，脚本会退出并列出当前可用的分类名称。 + +> **分类关系警告：** 使用 `--category` 时，导出包只记录所选分类，不会记录这些 Collection 在其他分类中的成员关系。正式导入会以导出包为准协调本次涉及 Collection 的分类关系，因此可能将它们从目标环境的其他分类中移除。如果需要完整保留全部分类关系，应不指定 `--category`，导出全部 Collection 和分类关系。 + +### 4.5 输出目录覆盖规则 + +默认情况下，如果输出目录已经存在，脚本会： + +1. 先在同一父目录生成完整的临时导出目录； +2. 将旧输出目录临时改名； +3. 用新目录替换旧目录； +4. 替换成功后删除旧目录； +5. 替换失败时尝试恢复旧目录。 + +如需禁止覆盖，请添加 `--no-overwrite`： + +```sh +./script/export-collection-schema.sh \ + --env master-data-local \ + --output-dir ./exports/reference-schema \ + --no-overwrite +``` + +### 4.6 导出包结构 + +典型目录结构如下： + +```text +exports/reference-schema/ +├── manifest.json +└── main/ + └── 参考数据/ + ├── ref_code_sets.json + ├── ref_code_set_versions.json + └── ref_code_items.json +``` + +`manifest.json` 记录： + +- 源环境名称； +- 固定数据源 `main`； +- 导出分类或 `all`； +- 导出时间； +- 覆盖策略； +- 发现和导出的 Collection 数量； +- 导出文件数量； +- 每个文件对应的分类、Collection 和相对路径。 + +每个 Collection JSON 文件保存经过标准化的可移植结构。导出时会排除部分由 NocoBase 自动生成或与当前实例绑定的内容，例如： + +- `id`、创建时间、更新时间、创建人和更新人等系统字段； +- 文件 Collection 和树形 Collection 的部分内置字段； +- 实例内部使用的 `key`、`collectionName`、`parentKey` 等标识。 + +导出完成后，脚本会在标准输出中返回一份不包含文件明细的 JSON 摘要。 + +## 5. 导入 Collection 结构 + +### 5.1 命令格式 + +```sh +./script/import-collection-schema.sh \ + --env <目标环境名称> \ + --input-dir <导出包目录> \ + [--apply --confirm-env <同一目标环境名称>] +``` + +### 5.2 参数说明 + +| 参数 | 必填 | 说明 | +| --- | --- | --- | +| `--env` | 是 | NocoBase CLI 中已配置的目标环境名称。 | +| `--input-dir` | 是 | 导出脚本生成的目录，其中必须包含 `manifest.json`。 | +| `--apply` | 否 | 正式写入目标环境。不指定时只生成预演计划，不进行任何写入。 | +| `--confirm-env` | 正式导入时必填 | 必须与 `--env` 完全一致，用于降低误操作到错误环境的风险。 | +| `-h`、`--help` | 否 | 显示帮助信息。 | + +### 5.3 第一步：导入预演 + +先执行不带 `--apply` 的命令： + +```sh +./script/import-collection-schema.sh \ + --env mdm-uat \ + --input-dir ./exports/reference-schema +``` + +预演模式只读取和校验本地导出包，不写入目标 NocoBase 环境。输出示例： + +```json +{ + "mode": "plan", + "writes": false, + "targetEnvironment": "mdm-uat", + "sourceEnvironment": "master-data-local", + "profile": "portable", + "dataSource": "main", + "collections": 3, + "scalarFields": 12, + "relationFields": 4, + "categories": [ + "参考数据" + ], + "categoryRelations": 3, + "externalRelationTargets": [] +} +``` + +正式导入前至少核对： + +- `targetEnvironment` 是否为预期目标环境； +- `sourceEnvironment` 是否为预期源环境； +- `collections`、`scalarFields` 和 `relationFields` 数量是否合理； +- `categories` 是否符合预期； +- `unclassifiedCollections` 中是否存在不应取消分类的 Collection； +- `externalRelationTargets` 引用的 Collection 是否已在目标环境存在。 + +注意：预演不会连接目标环境检查外部关系目标是否实际存在，该检查在正式导入、首次写入之前执行。 + +### 5.4 第二步：正式导入 + +确认预演结果后执行： + +```sh +./script/import-collection-schema.sh \ + --env mdm-uat \ + --input-dir ./exports/reference-schema \ + --apply \ + --confirm-env mdm-uat +``` + +`--confirm-env` 的值必须与 `--env` 完全相同。例如，`--env mdm-uat` 必须配合 `--confirm-env mdm-uat`。 + +正式导入前，脚本会先完成以下检查： + +- 导出包结构和全部 JSON 文件有效； +- manifest 声明的数据源为 `main`； +- Collection 名称、字段名称和文件路径合法且无重复冲突； +- 目标环境可访问且当前身份已认证； +- 目标环境不存在重名的 Collection 分类； +- 导出包以外的关系目标已存在于目标环境。 + +检查通过后，脚本按四个阶段执行： + +1. 创建或更新 Collection，并应用非关系字段； +2. 应用 `m2o`、`o2m`、`m2m`、`o2o` 关系字段； +3. 创建缺失分类并协调 Collection 分类成员关系； +4. 从服务器回读 Collection、字段和分类关系并校验。 + +成功时输出类似： + +```json +{ + "mode": "applied", + "writes": true, + "verified": true, + "targetEnvironment": "mdm-uat", + "sourceEnvironment": "master-data-local", + "profile": "portable", + "dataSource": "main", + "collections": 3, + "scalarFields": 12, + "relationFields": 4, + "categoryMembership": "verified" +} +``` + +只有出现 `verified: true`，才表示脚本已完成服务器回读校验。 + +## 6. 导入合并与分类处理规则 + +### 6.1 Collection 和字段 + +- 已存在的 Collection 会被更新；不存在的 Collection 会被创建； +- 导入包中的普通字段和关系字段会被创建或更新； +- 导入使用 `replaceFields: false`，不会删除目标环境中额外存在的字段； +- 脚本不会删除目标环境中额外存在的 Collection； +- 如果目标结构无法包含导出包声明的结构，回读校验会失败并输出差异。 + +### 6.2 Collection 分类 + +- 导出包中存在、目标环境中不存在的分类会被创建； +- 对本次导入涉及的 Collection，分类成员关系会调整为导出包声明的状态； +- 目标分类中不属于本次导入范围的其他 Collection 会被保留； +- `未分类` 是导出使用的伪分类，不会在目标环境创建同名分类； +- 被标记为 `未分类` 的 Collection 会从目标环境现有分类中移除。 + +因此，导入前应重点检查预演结果中的 `categories` 和 `unclassifiedCollections`。对于按单个分类生成的导出包，还应确认是否允许移除本次 Collection 在目标环境中的其他分类关系。 + +## 7. 完整迁移示例 + +以下示例将本地环境中的“参考数据”分类迁移到 UAT 环境。 + +### 7.1 从源环境导出 + +```sh +./script/export-collection-schema.sh \ + --env master-data-local \ + --category 参考数据 \ + --output-dir ./exports/reference-schema \ + --no-overwrite +``` + +### 7.2 查看导出清单 + +```sh +jq . ./exports/reference-schema/manifest.json +``` + +列出导出的 Collection： + +```sh +jq -r '.files[].collection' ./exports/reference-schema/manifest.json | sort -u +``` + +### 7.3 在目标环境执行预演 + +```sh +./script/import-collection-schema.sh \ + --env mdm-uat \ + --input-dir ./exports/reference-schema +``` + +### 7.4 正式导入并回读验证 + +```sh +./script/import-collection-schema.sh \ + --env mdm-uat \ + --input-dir ./exports/reference-schema \ + --apply \ + --confirm-env mdm-uat +``` + +## 8. 常见错误及处理 + +### `nb CLI is required` + +当前终端找不到 `nb`。请安装或配置 NocoBase CLI，并确认 `command -v nb` 能返回可执行文件路径。 + +### `jq is required` + +当前终端找不到 `jq`。请先安装 `jq`，并确认 `command -v jq` 能返回可执行文件路径。 + +### `Unknown category` + +`--category` 指定的名称与源环境中的 Collection 分类不完全一致。请根据错误信息列出的分类名称重新执行。 + +### `Output directory already exists` + +使用了 `--no-overwrite`，但输出目录已经存在。请更换目录、移走旧目录，或确认允许替换后去掉 `--no-overwrite`。 + +### `Invalid manifest.json: portable main-data-source export required` + +输入目录不是当前导出脚本生成的主数据源可移植包，或 `manifest.json` 已被修改。请重新导出，不要手工修改 manifest 中的数据源、文件路径和数量字段。 + +### `External relation target is unavailable` + +某个关系字段引用了未包含在导出包中的 Collection，而该 Collection 在目标环境不存在。请先在目标环境创建或迁移这个依赖 Collection，再重新导入。 + +### `--apply requires --confirm-env to exactly equal --env` + +正式导入缺少环境确认，或者两个环境名称不一致。请检查目标环境后重新传入完全相同的值。 + +### `Import read-back verification failed` + +写入完成后的实际结构未包含导出包声明的结构。脚本会输出 Collection、字段或分类关系的预期值与实际值。请根据差异检查： + +- 目标环境的 NocoBase 版本和插件是否支持对应字段接口； +- 关系字段的目标 Collection 和反向字段配置是否有效； +- 当前身份是否拥有完整的数据模型修改权限； +- 目标环境是否存在同名但不兼容的字段或 Collection 配置。 + +该错误可能发生在部分写入已经完成之后。修正问题后，应重新执行预演，再执行正式导入并确认最终返回 `verified: true`。 + +## 9. 使用注意事项 + +- 正式导入前，建议先对目标环境创建可恢复的备份； +- 不要跳过预演，也不要仅凭进程退出码判断结构完全一致，应检查最终 JSON 中的 `verified`； +- 导出包中的 JSON 文件适合纳入版本管理，以便审阅结构变化； +- 不建议手工修改导出包；如确需修改，应同时保证 `manifest.json`、文件路径、Collection 名称及重复分类文件保持一致； +- Collection 结构迁移完成后，业务数据、ACL、页面和工作流需要使用各自独立的迁移与验证流程。 diff --git a/script/export-collection-schema.sh b/script/export-collection-schema.sh new file mode 100755 index 0000000..3910669 --- /dev/null +++ b/script/export-collection-schema.sh @@ -0,0 +1,295 @@ +#!/bin/sh +set -eu + +usage() { + cat <<'EOF' +Usage: + ./script/export-collection-schema.sh \ + --env \ + --output-dir \ + [--category ] \ + [--no-overwrite] + +Options: + --env Required NocoBase CLI environment name. + --output-dir Required export destination. + --category Exact NocoBase Collection category name. Default: all. + --no-overwrite Refuse to replace an existing output directory. + Default: atomically replace the existing directory. + -h, --help Show this help. + +Portable export of the main data source is always enabled. Output files are JSON. + +Example: + ./script/export-collection-schema.sh \ + --env master-data-local \ + --category 参考数据 \ + --output-dir ./exports/reference-schema +EOF +} + +ENV_NAME= +OUTPUT_DIR= +CATEGORY= +NO_OVERWRITE=0 + +while [ "$#" -gt 0 ]; do + case "$1" in + --env) + ENV_NAME=${2:?--env requires a value} + shift 2 + ;; + --output-dir) + OUTPUT_DIR=${2:?--output-dir requires a value} + shift 2 + ;; + --category) + CATEGORY=${2:?--category requires a value} + shift 2 + ;; + --no-overwrite) + NO_OVERWRITE=1 + shift + ;; + -h|--help) + usage + exit 0 + ;; + *) + echo "Unknown argument: $1" >&2 + usage >&2 + exit 2 + ;; + esac +done + +[ -n "$ENV_NAME" ] || { echo "--env is required" >&2; exit 2; } +[ -n "$OUTPUT_DIR" ] || { echo "--output-dir is required" >&2; exit 2; } +command -v nb >/dev/null 2>&1 || { echo "nb CLI is required" >&2; exit 2; } +command -v jq >/dev/null 2>&1 || { echo "jq is required" >&2; exit 2; } + +SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) +PROJECT_ROOT=$(dirname -- "$SCRIPT_DIR") +OUTPUT_PARENT=$(dirname -- "$OUTPUT_DIR") +OUTPUT_NAME=$(basename -- "$OUTPUT_DIR") +[ "$OUTPUT_NAME" != . ] && [ "$OUTPUT_NAME" != .. ] || { + echo "Refusing unsafe --output-dir: $OUTPUT_DIR" >&2 + exit 2 +} +mkdir -p "$OUTPUT_PARENT" +OUTPUT_PARENT=$(CDPATH= cd -- "$OUTPUT_PARENT" && pwd) +OUTPUT_DIR="$OUTPUT_PARENT/$OUTPUT_NAME" + +case "$OUTPUT_DIR" in + /|"$HOME"|"$PROJECT_ROOT"|"$SCRIPT_DIR") + echo "Refusing unsafe --output-dir: $OUTPUT_DIR" >&2 + exit 2 + ;; +esac + +if [ "$NO_OVERWRITE" -eq 1 ] && [ -e "$OUTPUT_DIR" ]; then + echo "Output directory already exists: $OUTPUT_DIR" >&2 + exit 1 +fi + +run_nb() { + RAW_OUTPUT=$("$@") + JSON_OUTPUT=$(printf '%s\n' "$RAW_OUTPUT" | awk ' + !started && /^[[:space:]]*[\{\[]/ { started = 1 } + started { print } + ') + printf '%s\n' "$JSON_OUTPUT" | jq -e . >/dev/null + printf '%s\n' "$JSON_OUTPUT" +} + +validate_segment() { + case "$1" in + ''|.|..|*/*|*\\*) + echo "Unsafe path segment: $1" >&2 + exit 2 + ;; + esac +} + +list_collections() { + PAGE=1 + ALL='[]' + while :; do + RESPONSE=$(run_nb nb api data-modeling collections list \ + --page "$PAGE" --page-size 500 -e "$ENV_NAME" -y -j) + ROWS=$(printf '%s' "$RESPONSE" | jq -c '.data // []') + ALL=$(jq -cn --argjson current "$ALL" --argjson rows "$ROWS" '$current + $rows') + TOTAL_PAGES=$(printf '%s' "$RESPONSE" | jq -r '.meta.totalPage // 1') + [ "$PAGE" -ge "$TOTAL_PAGES" ] && break + PAGE=$((PAGE + 1)) + done + printf '%s\n' "$ALL" +} + +load_categories() { + run_nb nb api resource list \ + --resource collectionCategories \ + --no-paginate \ + --sort sort \ + --fields name \ + --appends collections \ + -e "$ENV_NAME" -y -j +} + +read_collection() { + COLLECTION_NAME=$1 + run_nb nb api data-modeling collections get \ + --filter-by-tk "$COLLECTION_NAME" \ + --appends fields \ + -e "$ENV_NAME" -y -j +} + +portable_schema() { + jq ' + .data as $collection + | ($collection.fields // []) as $fields + | [$fields[] + | select(.interface == "m2o" or .interface == "o2m" or .interface == "m2m" or .interface == "o2o") + | .foreignKey + | select(. != null)] as $foreignKeys + | ($collection.template // "general") as $template + | ($collection | del(.key, .fields, .verify, .category)) + + { + fields: [ + $fields[] + | . as $field + | select(["id", "createdAt", "createdBy", "updatedAt", "updatedBy", "createdById", "updatedById"] | index($field.name) | not) + | select( + if $template == "file" then + ["title", "filename", "extname", "size", "mimetype", "path", "url", "preview", "storage", "storageId", "meta"] + | index($field.name) | not + elif $template == "tree" then + ["parentId", "parent", "children"] | index($field.name) | not + else true end + ) + | select( + ($foreignKeys | index($field.name) | not) + or ($field.interface == "m2o" or $field.interface == "o2m" or $field.interface == "m2m" or $field.interface == "o2o") + ) + | del(.key, .collectionName, .parentKey, .reverseKey, .possibleTypes) + | walk( + if type == "object" and has("rules") and (.rules | type) == "array" then + .rules |= map(del(.key)) + else . end + ) + ] | sort_by(.name) + } + ' +} + +COLLECTIONS=$(list_collections) +CATEGORIES=$(load_categories) + +KNOWN_CATEGORIES=$(printf '%s' "$CATEGORIES" | jq -r '[.data[]?.name, "未分类"] | unique | join(", ")') +if [ -n "$CATEGORY" ] && ! printf '%s' "$CATEGORIES" | jq -e --arg category "$CATEGORY" ' + any(.data[]?; .name == $category) or $category == "未分类" +' >/dev/null; then + echo "Unknown category: $CATEGORY; available: $KNOWN_CATEGORIES" >&2 + exit 1 +fi + +SELECTION=$(printf '%s' "$COLLECTIONS" | jq -c \ + --argjson categoryData "$CATEGORIES" \ + --arg selected "$CATEGORY" ' + [ + .[] + | . as $collection + | [$categoryData.data[]? + | select(any(.collections[]?; .name == $collection.name)) + | .name] as $assigned + | (if ($assigned | length) == 0 then ["未分类"] else $assigned end) as $categories + | { + collection: $collection, + categories: ($categories | map(select($selected == "" or . == $selected))) + } + | select((.categories | length) > 0) + ] +') + +TEMP_DIR=$(mktemp -d "$OUTPUT_PARENT/.${OUTPUT_NAME}.tmp.XXXXXX") +MANIFEST_ROWS="$TEMP_DIR/.manifest-rows.jsonl" +: > "$MANIFEST_ROWS" +trap 'rm -rf "$TEMP_DIR"' EXIT HUP INT TERM + +INDEX=0 +COUNT=$(printf '%s' "$SELECTION" | jq 'length') +while [ "$INDEX" -lt "$COUNT" ]; do + ITEM=$(printf '%s' "$SELECTION" | jq -c ".[$INDEX]") + COLLECTION_NAME=$(printf '%s' "$ITEM" | jq -r '.collection.name') + validate_segment "$COLLECTION_NAME" + SCHEMA=$(read_collection "$COLLECTION_NAME" | portable_schema) + CATEGORY_COUNT=$(printf '%s' "$ITEM" | jq '.categories | length') + CATEGORY_INDEX=0 + while [ "$CATEGORY_INDEX" -lt "$CATEGORY_COUNT" ]; do + CATEGORY_NAME=$(printf '%s' "$ITEM" | jq -r ".categories[$CATEGORY_INDEX]") + validate_segment "$CATEGORY_NAME" + RELATIVE_FILE="main/$CATEGORY_NAME/$COLLECTION_NAME.json" + TARGET_FILE="$TEMP_DIR/$RELATIVE_FILE" + mkdir -p "$(dirname -- "$TARGET_FILE")" + printf '%s\n' "$SCHEMA" > "$TARGET_FILE" + jq -cn \ + --arg dataSource main \ + --arg category "$CATEGORY_NAME" \ + --arg collection "$COLLECTION_NAME" \ + --arg file "$RELATIVE_FILE" \ + '{dataSource:$dataSource, category:$category, collection:$collection, file:$file}' \ + >> "$MANIFEST_ROWS" + CATEGORY_INDEX=$((CATEGORY_INDEX + 1)) + done + INDEX=$((INDEX + 1)) +done + +FILES=$(jq -s 'sort_by(.file)' "$MANIFEST_ROWS") +DISCOVERED=$(printf '%s' "$COLLECTIONS" | jq 'length') +EXPORTED_FILES=$(printf '%s' "$FILES" | jq 'length') +GENERATED_AT=$(date -u '+%Y-%m-%dT%H:%M:%SZ') +OVERWRITE_POLICY=replace-existing +[ "$NO_OVERWRITE" -eq 1 ] && OVERWRITE_POLICY=refuse-existing + +jq -n \ + --arg environment "$ENV_NAME" \ + --arg dataSource main \ + --arg category "${CATEGORY:-all}" \ + --arg generatedAt "$GENERATED_AT" \ + --arg overwritePolicy "$OVERWRITE_POLICY" \ + --argjson discoveredCollections "$DISCOVERED" \ + --argjson exportedCollections "$COUNT" \ + --argjson exportedFiles "$EXPORTED_FILES" \ + --argjson files "$FILES" \ + '{ + environment:$environment, + dataSource:$dataSource, + category:$category, + profile:"portable", + generatedAt:$generatedAt, + overwritePolicy:$overwritePolicy, + discoveredCollections:$discoveredCollections, + exportedCollections:$exportedCollections, + exportedFiles:$exportedFiles, + files:$files + }' > "$TEMP_DIR/manifest.json" + +rm -f "$MANIFEST_ROWS" + +if [ -e "$OUTPUT_DIR" ]; then + BACKUP_DIR="$OUTPUT_DIR.previous.$$" + mv "$OUTPUT_DIR" "$BACKUP_DIR" + if mv "$TEMP_DIR" "$OUTPUT_DIR"; then + rm -rf "$BACKUP_DIR" + else + mv "$BACKUP_DIR" "$OUTPUT_DIR" + exit 1 + fi +else + mv "$TEMP_DIR" "$OUTPUT_DIR" +fi +trap - EXIT HUP INT TERM + +jq --arg outputDir "$OUTPUT_DIR" ' + del(.files) + {outputDir:$outputDir} +' "$OUTPUT_DIR/manifest.json" diff --git a/script/import-collection-schema.sh b/script/import-collection-schema.sh new file mode 100755 index 0000000..5c48e89 --- /dev/null +++ b/script/import-collection-schema.sh @@ -0,0 +1,596 @@ +#!/bin/sh +set -eu + +usage() { + cat <<'EOF' +Usage: + ./script/import-collection-schema.sh \ + --env \ + --input-dir \ + [--apply --confirm-env ] + +Options: + --env Required target NocoBase CLI environment name. + --input-dir Required directory produced by export-collection-schema.sh. + --apply Apply the validated plan. Default: plan only, no writes. + --confirm-env Required with --apply and must exactly equal --env. + -h, --help Show this help. + +Behavior: + - Accepts only portable schemas for the main data source. + - Validates manifest.json and every referenced schema before any write. + - Applies collection shells and non-relation fields first, then relations. + - Reads every imported collection back and verifies the declared schema. + - Creates missing Collection categories and reconciles category memberships. + - Treats the exported pseudo-category "未分类" as no category membership. + - Does not delete extra fields already present in the target environment. + +Plan only: + ./script/import-collection-schema.sh \ + --env mdm-uat \ + --input-dir ./exports/reference-schema + +Apply: + ./script/import-collection-schema.sh \ + --env mdm-uat \ + --input-dir ./exports/reference-schema \ + --apply \ + --confirm-env mdm-uat +EOF +} + +ENV_NAME= +INPUT_DIR= +APPLY=0 +CONFIRM_ENV= + +while [ "$#" -gt 0 ]; do + case "$1" in + --env) + ENV_NAME=${2:?--env requires a value} + shift 2 + ;; + --input-dir) + INPUT_DIR=${2:?--input-dir requires a value} + shift 2 + ;; + --apply) + APPLY=1 + shift + ;; + --confirm-env) + CONFIRM_ENV=${2:?--confirm-env requires a value} + shift 2 + ;; + -h|--help) + usage + exit 0 + ;; + *) + echo "Unknown argument: $1" >&2 + usage >&2 + exit 2 + ;; + esac +done + +[ -n "$ENV_NAME" ] || { echo "--env is required" >&2; exit 2; } +[ -n "$INPUT_DIR" ] || { echo "--input-dir is required" >&2; exit 2; } +[ -d "$INPUT_DIR" ] || { echo "Input directory does not exist: $INPUT_DIR" >&2; exit 1; } +command -v nb >/dev/null 2>&1 || { echo "nb CLI is required" >&2; exit 2; } +command -v jq >/dev/null 2>&1 || { echo "jq is required" >&2; exit 2; } + +if [ "$APPLY" -eq 0 ] && [ -n "$CONFIRM_ENV" ]; then + echo "--confirm-env is only valid together with --apply" >&2 + exit 2 +fi +if [ "$APPLY" -eq 1 ] && [ "$CONFIRM_ENV" != "$ENV_NAME" ]; then + echo "--apply requires --confirm-env to exactly equal --env ($ENV_NAME)" >&2 + exit 2 +fi + +INPUT_DIR=$(CDPATH= cd -- "$INPUT_DIR" && pwd) +MANIFEST_FILE="$INPUT_DIR/manifest.json" +[ -f "$MANIFEST_FILE" ] && [ ! -L "$MANIFEST_FILE" ] || { + echo "Regular manifest.json not found in: $INPUT_DIR" >&2 + exit 1 +} + +if ! jq -e ' + type == "object" + and .profile == "portable" + and .dataSource == "main" + and (.files | type == "array" and length > 0) + and all(.files[]; + .dataSource == "main" + and (.category | type == "string" and length > 0) + and (.collection | type == "string" and length > 0) + and (.file | type == "string" and length > 0) + and (.file | startswith("/") | not) + and (.file | contains("\\") | not) + and (.file | split("/") | all(. != "" and . != "." and . != "..")) + ) + and (([.files[].file] | unique | length) == (.files | length)) + and ((has("exportedFiles") | not) or .exportedFiles == (.files | length)) +' "$MANIFEST_FILE" >/dev/null; then + echo "Invalid manifest.json: portable main-data-source export required" >&2 + exit 1 +fi + +WORK_DIR=$(mktemp -d "${TMPDIR:-/tmp}/import-collection-schema.XXXXXX") +ROWS_FILE="$WORK_DIR/schema-rows.jsonl" +PLAN_FILE="$WORK_DIR/plan.json" +VERIFY_FILE="$WORK_DIR/verification-errors.jsonl" +: > "$ROWS_FILE" +: > "$VERIFY_FILE" +trap 'rm -rf "$WORK_DIR"' EXIT HUP INT TERM + +FILE_COUNT=$(jq '.files | length' "$MANIFEST_FILE") +INDEX=0 +while [ "$INDEX" -lt "$FILE_COUNT" ]; do + ITEM=$(jq -c ".files[$INDEX]" "$MANIFEST_FILE") + RELATIVE_FILE=$(printf '%s' "$ITEM" | jq -r '.file') + COLLECTION_NAME=$(printf '%s' "$ITEM" | jq -r '.collection') + CATEGORY_NAME=$(printf '%s' "$ITEM" | jq -r '.category') + SCHEMA_FILE="$INPUT_DIR/$RELATIVE_FILE" + + [ -f "$SCHEMA_FILE" ] && [ ! -L "$SCHEMA_FILE" ] || { + echo "Schema file is missing or is not a regular file: $RELATIVE_FILE" >&2 + exit 1 + } + if ! jq -e --arg collection "$COLLECTION_NAME" ' + type == "object" + and .name == $collection + and (.title | type == "string" and length > 0) + and (.template | type == "string" and length > 0) + and (.fields | type == "array") + and all(.fields[]; + type == "object" + and (.name | type == "string" and length > 0) + and (.interface | type == "string" and length > 0) + ) + and (([.fields[].name] | unique | length) == (.fields | length)) + ' "$SCHEMA_FILE" >/dev/null; then + echo "Invalid Collection schema: $RELATIVE_FILE" >&2 + exit 1 + fi + + jq -c \ + --arg collection "$COLLECTION_NAME" \ + --arg category "$CATEGORY_NAME" \ + --arg file "$RELATIVE_FILE" \ + '{collection:$collection, category:$category, file:$file, schema:.}' \ + "$SCHEMA_FILE" >> "$ROWS_FILE" + INDEX=$((INDEX + 1)) +done + +if ! jq -se ' + sort_by(.collection) + | group_by(.collection) + | all(.[]; ([.[].schema] | unique | length) == 1) +' "$ROWS_FILE" >/dev/null; then + echo "One Collection is represented by conflicting schema files" >&2 + exit 1 +fi + +jq -s ' + sort_by(.collection) + | group_by(.collection) + | map({ + collection: .[0].collection, + categories: ([.[].category] | unique | sort), + files: ([.[].file] | unique | sort), + schema: .[0].schema + }) +' "$ROWS_FILE" > "$PLAN_FILE" + +COLLECTION_COUNT=$(jq 'length' "$PLAN_FILE") +if ! jq -e --argjson count "$COLLECTION_COUNT" ' + (has("exportedCollections") | not) or .exportedCollections == $count +' "$MANIFEST_FILE" >/dev/null; then + echo "manifest.json exportedCollections does not match the schemas" >&2 + exit 1 +fi + +RELATION_INTERFACES='["m2o","o2m","m2m","o2o"]' +SCALAR_FIELD_COUNT=$(jq --argjson relations "$RELATION_INTERFACES" ' + [.[] | .schema.fields[] | select((.interface as $i | $relations | index($i)) == null)] | length +' "$PLAN_FILE") +RELATION_FIELD_COUNT=$(jq --argjson relations "$RELATION_INTERFACES" ' + [.[] | .schema.fields[] | select(.interface as $i | $relations | index($i))] | length +' "$PLAN_FILE") +EXTERNAL_TARGETS=$(jq -c --argjson relations "$RELATION_INTERFACES" ' + ([.[].collection] | unique) as $included + | [ + .[] | .schema.fields[] + | select(.interface as $i | $relations | index($i)) + | .target + | select(type == "string" and length > 0) + | . as $target + | select(($included | index($target)) == null) + ] | unique | sort +' "$PLAN_FILE") +CATEGORIES=$(jq -c '[.[].categories[]] | unique | sort' "$PLAN_FILE") +REAL_CATEGORIES=$(printf '%s' "$CATEGORIES" | jq -c 'map(select(. != "未分类"))') +UNCLASSIFIED_COLLECTIONS=$(jq -c ' + [.[] | select(.categories | index("未分类")) | .collection] | unique | sort +' "$PLAN_FILE") +CATEGORY_RELATION_COUNT=$(jq ' + [.[] as $item | $item.categories[] | select(. != "未分类") | [$item.collection, .]] + | unique | length +' "$PLAN_FILE") +SOURCE_ENV=$(jq -r '.environment // "unknown"' "$MANIFEST_FILE") + +PLAN_SUMMARY=$(jq -n \ + --arg targetEnvironment "$ENV_NAME" \ + --arg sourceEnvironment "$SOURCE_ENV" \ + --arg inputDir "$INPUT_DIR" \ + --argjson collections "$COLLECTION_COUNT" \ + --argjson scalarFields "$SCALAR_FIELD_COUNT" \ + --argjson relationFields "$RELATION_FIELD_COUNT" \ + --argjson categories "$CATEGORIES" \ + --argjson categoryRelations "$CATEGORY_RELATION_COUNT" \ + --argjson externalRelationTargets "$EXTERNAL_TARGETS" \ + --argjson unclassifiedCollections "$UNCLASSIFIED_COLLECTIONS" \ + '{ + mode:"plan", + writes:false, + targetEnvironment:$targetEnvironment, + sourceEnvironment:$sourceEnvironment, + inputDir:$inputDir, + profile:"portable", + dataSource:"main", + collections:$collections, + scalarFields:$scalarFields, + relationFields:$relationFields, + categories:$categories, + categoryRelations:$categoryRelations, + categoryMembership:"planned-reconciliation", + unclassifiedCollections:$unclassifiedCollections, + externalRelationTargets:$externalRelationTargets, + phases:[ + "validate", + "apply-collections-and-scalar-fields", + "apply-relation-fields", + "reconcile-category-memberships", + "read-back-and-verify" + ] + }') + +if [ "$APPLY" -eq 0 ]; then + printf '%s\n' "$PLAN_SUMMARY" | jq . + exit 0 +fi + +echo "Validated import plan for $ENV_NAME: $COLLECTION_COUNT Collections, $SCALAR_FIELD_COUNT scalar fields, $RELATION_FIELD_COUNT relation fields" >&2 + +run_nb() { + if ! RAW_OUTPUT=$("$@" 2>&1); then + printf '%s\n' "$RAW_OUTPUT" >&2 + return 1 + fi + JSON_OUTPUT=$(printf '%s\n' "$RAW_OUTPUT" | awk ' + !started && /^[[:space:]]*[\{\[]/ { started = 1 } + started { print } + ') + if ! printf '%s\n' "$JSON_OUTPUT" | jq -e . >/dev/null 2>&1; then + printf '%s\n' "$RAW_OUTPUT" >&2 + echo "NocoBase CLI did not return valid JSON" >&2 + return 1 + fi + printf '%s\n' "$JSON_OUTPUT" +} + +load_target_categories() { + run_nb nb api resource list \ + --resource collectionCategories \ + --no-paginate \ + --sort sort \ + --fields id \ + --fields name \ + --appends collections \ + -e "$ENV_NAME" -y -j +} + +# Reachability and authentication gate before the first write. +run_nb nb api data-modeling collections list \ + --page 1 --page-size 1 -e "$ENV_NAME" -y -j >/dev/null + +TARGET_CATEGORIES=$(load_target_categories) +if ! printf '%s' "$TARGET_CATEGORIES" | jq -e ' + [.data[]?.name] | group_by(.) | all(.[]; length == 1) +' >/dev/null; then + echo "Target environment contains duplicate Collection category names" >&2 + exit 1 +fi + +TARGET_COUNT=$(printf '%s' "$EXTERNAL_TARGETS" | jq 'length') +TARGET_INDEX=0 +while [ "$TARGET_INDEX" -lt "$TARGET_COUNT" ]; do + TARGET_NAME=$(printf '%s' "$EXTERNAL_TARGETS" | jq -r ".[$TARGET_INDEX]") + if ! run_nb nb api data-modeling collections get \ + --filter-by-tk "$TARGET_NAME" -e "$ENV_NAME" -y -j >/dev/null; then + echo "External relation target is unavailable in $ENV_NAME: $TARGET_NAME" >&2 + exit 1 + fi + TARGET_INDEX=$((TARGET_INDEX + 1)) +done + +make_collection_payload() { + PLAN_INDEX=$1 + jq --argjson index "$PLAN_INDEX" --argjson relations "$RELATION_INTERFACES" ' + def compact_field: + . as $field + | ({name:$field.name, interface:$field.interface} + + (if $field | has("title") then {title:$field.title} else {} end) + + (if $field | has("description") then {description:$field.description} else {} end) + + (if $field | has("target") then {target:$field.target} else {} end) + + (if $field | has("validation") then {validation:$field.validation} else {} end) + + (if $field | has("validators") then {validators:$field.validators} else {} end) + + (if $field | has("defaultValue") then {defaultValue:$field.defaultValue} else {} end) + + (if $field | has("enum") then {enum:$field.enum} else {} end) + + (if $field | has("reverseName") then {reverseName:$field.reverseName} else {} end) + + (if $field | has("reverseTitle") then {reverseTitle:$field.reverseTitle} else {} end) + + (if $field | has("reverseInterface") then {reverseInterface:$field.reverseInterface} else {} end) + + (if $field | has("reverseField") then {reverseField:$field.reverseField} else {} end)) + + (((($field.settings // {}) + ($field | del( + .name, .title, .interface, .description, .target, .validation, .validators, + .defaultValue, .enum, .reverseName, .reverseTitle, .reverseInterface, + .reverseField, .settings + ))) as $settings + | if $settings == {} then {} else {settings:$settings} end)); + .[$index].schema as $schema + | ({name:$schema.name, title:$schema.title, template:$schema.template, verify:true, replaceFields:false} + + (if $schema | has("description") then {description:$schema.description} else {} end) + + (if $schema | has("viewName") then {viewName:$schema.viewName} else {} end) + + (if $schema | has("inherits") then {inherits:$schema.inherits} else {} end) + + {fields:[ + $schema.fields[] + | select((.interface as $i | $relations | index($i)) == null) + | compact_field + ]}) + + (((($schema.settings // {}) + ($schema | del( + .name, .title, .description, .template, .viewName, .inherits, + .replaceFields, .verify, .fields, .settings + ))) as $settings + | if $settings == {} then {} else {settings:$settings} end)) + ' "$PLAN_FILE" +} + +make_relation_payload() { + PLAN_INDEX=$1 + FIELD_INDEX=$2 + jq --argjson planIndex "$PLAN_INDEX" --argjson fieldIndex "$FIELD_INDEX" ' + .[$planIndex] as $item + | $item.schema.fields[$fieldIndex] as $field + | ({collectionName:$item.collection, name:$field.name, interface:$field.interface} + + (if $field | has("title") then {title:$field.title} else {} end) + + (if $field | has("description") then {description:$field.description} else {} end) + + (if $field | has("target") then {target:$field.target} else {} end) + + (if $field | has("validation") then {validation:$field.validation} else {} end) + + (if $field | has("validators") then {validators:$field.validators} else {} end) + + (if $field | has("defaultValue") then {defaultValue:$field.defaultValue} else {} end) + + (if $field | has("enum") then {enum:$field.enum} else {} end) + + (if $field | has("reverseName") then {reverseName:$field.reverseName} else {} end) + + (if $field | has("reverseTitle") then {reverseTitle:$field.reverseTitle} else {} end) + + (if $field | has("reverseInterface") then {reverseInterface:$field.reverseInterface} else {} end) + + (if $field | has("reverseField") then {reverseField:$field.reverseField} else {} end)) + + (((($field.settings // {}) + ($field | del( + .collectionName, .name, .title, .interface, .description, .target, + .validation, .validators, .defaultValue, .enum, .reverseName, + .reverseTitle, .reverseInterface, .reverseField, .settings + ))) as $settings + | if $settings == {} then {} else {settings:$settings} end)) + ' "$PLAN_FILE" +} + +echo "Phase 1/4: applying Collections and non-relation fields" >&2 +INDEX=0 +while [ "$INDEX" -lt "$COLLECTION_COUNT" ]; do + COLLECTION_NAME=$(jq -r ".[$INDEX].collection" "$PLAN_FILE") + PAYLOAD_FILE="$WORK_DIR/collection-$INDEX.json" + make_collection_payload "$INDEX" > "$PAYLOAD_FILE" + echo " apply $COLLECTION_NAME" >&2 + run_nb nb api data-modeling collections apply \ + --body-file "$PAYLOAD_FILE" -e "$ENV_NAME" -y -j >/dev/null + INDEX=$((INDEX + 1)) +done + +echo "Phase 2/4: applying relation fields" >&2 +INDEX=0 +while [ "$INDEX" -lt "$COLLECTION_COUNT" ]; do + COLLECTION_NAME=$(jq -r ".[$INDEX].collection" "$PLAN_FILE") + FIELD_COUNT=$(jq ".[$INDEX].schema.fields | length" "$PLAN_FILE") + FIELD_INDEX=0 + while [ "$FIELD_INDEX" -lt "$FIELD_COUNT" ]; do + INTERFACE=$(jq -r ".[$INDEX].schema.fields[$FIELD_INDEX].interface" "$PLAN_FILE") + if printf '%s' "$RELATION_INTERFACES" | jq -e --arg interface "$INTERFACE" 'index($interface) != null' >/dev/null; then + FIELD_NAME=$(jq -r ".[$INDEX].schema.fields[$FIELD_INDEX].name" "$PLAN_FILE") + PAYLOAD_FILE="$WORK_DIR/relation-$INDEX-$FIELD_INDEX.json" + make_relation_payload "$INDEX" "$FIELD_INDEX" > "$PAYLOAD_FILE" + echo " apply $COLLECTION_NAME.$FIELD_NAME" >&2 + run_nb nb api data-modeling fields apply \ + --body-file "$PAYLOAD_FILE" -e "$ENV_NAME" -y -j >/dev/null + fi + FIELD_INDEX=$((FIELD_INDEX + 1)) + done + INDEX=$((INDEX + 1)) +done + +echo "Phase 3/4: reconciling Collection category memberships" >&2 +REAL_CATEGORY_COUNT=$(printf '%s' "$REAL_CATEGORIES" | jq 'length') +CATEGORY_INDEX=0 +while [ "$CATEGORY_INDEX" -lt "$REAL_CATEGORY_COUNT" ]; do + CATEGORY_NAME=$(printf '%s' "$REAL_CATEGORIES" | jq -r ".[$CATEGORY_INDEX]") + if ! printf '%s' "$TARGET_CATEGORIES" | jq -e --arg category "$CATEGORY_NAME" ' + any(.data[]?; .name == $category) + ' >/dev/null; then + CATEGORY_VALUES=$(jq -cn --arg name "$CATEGORY_NAME" '{name:$name}') + echo " create category $CATEGORY_NAME" >&2 + run_nb nb api resource create \ + --resource collectionCategories \ + --values "$CATEGORY_VALUES" \ + -e "$ENV_NAME" -y -j >/dev/null + fi + CATEGORY_INDEX=$((CATEGORY_INDEX + 1)) +done + +TARGET_CATEGORIES=$(load_target_categories) +IMPORTED_COLLECTIONS=$(jq -c '[.[].collection] | unique | sort' "$PLAN_FILE") +TARGET_CATEGORY_COUNT=$(printf '%s' "$TARGET_CATEGORIES" | jq '.data | length') +CATEGORY_INDEX=0 +while [ "$CATEGORY_INDEX" -lt "$TARGET_CATEGORY_COUNT" ]; do + CATEGORY_ID=$(printf '%s' "$TARGET_CATEGORIES" | jq -r ".data[$CATEGORY_INDEX].id") + CATEGORY_NAME=$(printf '%s' "$TARGET_CATEGORIES" | jq -r ".data[$CATEGORY_INDEX].name") + CURRENT_MEMBERS=$(printf '%s' "$TARGET_CATEGORIES" | jq -c \ + ".data[$CATEGORY_INDEX].collections // [] | [.[].name] | unique | sort") + DESIRED_IMPORTED=$(jq -c --arg category "$CATEGORY_NAME" ' + [.[] | select(.categories | index($category)) | .collection] | unique | sort + ' "$PLAN_FILE") + DESIRED_MEMBERS=$(jq -cn \ + --argjson current "$CURRENT_MEMBERS" \ + --argjson imported "$IMPORTED_COLLECTIONS" \ + --argjson desiredImported "$DESIRED_IMPORTED" \ + '(($current - $imported) + $desiredImported) | unique | sort') + + if [ "$CURRENT_MEMBERS" != "$DESIRED_MEMBERS" ]; then + CATEGORY_VALUES=$(printf '%s' "$DESIRED_MEMBERS" | jq -c '{collections:map({name:.})}') + echo " update category $CATEGORY_NAME" >&2 + run_nb nb api resource update \ + --resource collectionCategories \ + --filter-by-tk "$CATEGORY_ID" \ + --values "$CATEGORY_VALUES" \ + --update-association-values collections \ + -e "$ENV_NAME" -y -j >/dev/null + fi + CATEGORY_INDEX=$((CATEGORY_INDEX + 1)) +done + +portable_schema() { + jq ' + .data as $collection + | ($collection.fields // []) as $fields + | [$fields[] + | select(.interface == "m2o" or .interface == "o2m" or .interface == "m2m" or .interface == "o2o") + | .foreignKey + | select(. != null)] as $foreignKeys + | ($collection.template // "general") as $template + | ($collection | del(.key, .fields, .verify, .category)) + + { + fields: [ + $fields[] + | . as $field + | select(["id", "createdAt", "createdBy", "updatedAt", "updatedBy", "createdById", "updatedById"] | index($field.name) | not) + | select( + if $template == "file" then + ["title", "filename", "extname", "size", "mimetype", "path", "url", "preview", "storage", "storageId", "meta"] + | index($field.name) | not + elif $template == "tree" then + ["parentId", "parent", "children"] | index($field.name) | not + else true end + ) + | select( + ($foreignKeys | index($field.name) | not) + or ($field.interface == "m2o" or $field.interface == "o2m" or $field.interface == "m2m" or $field.interface == "o2o") + ) + | del(.key, .collectionName, .parentKey, .reverseKey, .possibleTypes) + | walk( + if type == "object" and has("rules") and (.rules | type) == "array" then + .rules |= map(del(.key)) + else . end + ) + ] | sort_by(.name) + } + ' +} + +echo "Phase 4/4: reading back and verifying" >&2 +INDEX=0 +while [ "$INDEX" -lt "$COLLECTION_COUNT" ]; do + COLLECTION_NAME=$(jq -r ".[$INDEX].collection" "$PLAN_FILE") + EXPECTED=$(jq -c ".[$INDEX].schema" "$PLAN_FILE") + READBACK=$(run_nb nb api data-modeling collections get \ + --filter-by-tk "$COLLECTION_NAME" --appends fields \ + -e "$ENV_NAME" -y -j) + ACTUAL=$(printf '%s' "$READBACK" | portable_schema | jq -c .) + + if ! jq -en --argjson expected "$EXPECTED" --argjson actual "$ACTUAL" ' + $actual | contains($expected | del(.fields)) + ' >/dev/null; then + jq -cn --arg collection "$COLLECTION_NAME" \ + --argjson expected "$EXPECTED" --argjson actual "$ACTUAL" \ + '{collection:$collection, scope:"collection", expected:($expected|del(.fields)), actual:($actual|del(.fields))}' \ + >> "$VERIFY_FILE" + fi + + jq -cn --arg collection "$COLLECTION_NAME" \ + --argjson expected "$EXPECTED" --argjson actual "$ACTUAL" ' + [ + $expected.fields[] as $field + | ($actual.fields | map(select(.name == $field.name)) | first // null) as $found + | select($found == null or (($found | contains($field)) | not)) + | {collection:$collection, scope:"field", field:$field.name, expected:$field, actual:$found} + ][] + ' >> "$VERIFY_FILE" + INDEX=$((INDEX + 1)) +done + +CATEGORY_READBACK=$(load_target_categories) +EXPECTED_CATEGORY_RELATIONS=$(jq -c ' + [ + .[] as $item + | $item.categories[] + | select(. != "未分类") + | {category:., collection:$item.collection} + ] | unique | sort_by(.category, .collection) +' "$PLAN_FILE") +ACTUAL_CATEGORY_RELATIONS=$(printf '%s' "$CATEGORY_READBACK" | jq -c \ + --argjson imported "$IMPORTED_COLLECTIONS" ' + [ + .data[]? as $category + | $category.collections[]? + | select(.name as $name | $imported | index($name)) + | {category:$category.name, collection:.name} + ] | unique | sort_by(.category, .collection) +') +if [ "$EXPECTED_CATEGORY_RELATIONS" != "$ACTUAL_CATEGORY_RELATIONS" ]; then + jq -cn \ + --argjson expected "$EXPECTED_CATEGORY_RELATIONS" \ + --argjson actual "$ACTUAL_CATEGORY_RELATIONS" \ + '{scope:"collection-category-memberships", expected:$expected, actual:$actual}' \ + >> "$VERIFY_FILE" +fi + +if [ -s "$VERIFY_FILE" ]; then + echo "Import read-back verification failed:" >&2 + jq -s . "$VERIFY_FILE" >&2 + exit 1 +fi + +jq -n \ + --arg targetEnvironment "$ENV_NAME" \ + --arg sourceEnvironment "$SOURCE_ENV" \ + --arg inputDir "$INPUT_DIR" \ + --argjson collections "$COLLECTION_COUNT" \ + --argjson scalarFields "$SCALAR_FIELD_COUNT" \ + --argjson relationFields "$RELATION_FIELD_COUNT" \ + --argjson categories "$CATEGORIES" \ + --argjson categoryRelations "$CATEGORY_RELATION_COUNT" \ + --argjson unclassifiedCollections "$UNCLASSIFIED_COLLECTIONS" \ + '{ + mode:"applied", + writes:true, + verified:true, + targetEnvironment:$targetEnvironment, + sourceEnvironment:$sourceEnvironment, + inputDir:$inputDir, + profile:"portable", + dataSource:"main", + collections:$collections, + scalarFields:$scalarFields, + relationFields:$relationFields, + categories:$categories, + categoryRelations:$categoryRelations, + categoryMembership:"verified", + unclassifiedCollections:$unclassifiedCollections + }'