Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
2cad0ef
Add test for MS2(未病DB)メタデータ付与機能
lieunguyen-tma May 14, 2026
afc1057
Add create new application and SWORD Client registration in WEKO3
lieunguyen-tma May 18, 2026
9ca31fe
Update the condition in e2e-test.yml so that WEKO can be tested in Me…
lieunguyen-tma May 18, 2026
cef1799
Add ignore_https_errors for test metadata mibyou db
lieunguyen-tma May 19, 2026
74a7acd
Add config sword_mapping_id for test metadata mibyou db
lieunguyen-tma May 19, 2026
9d95c0b
Update include_admin to True for test metadata mibyou db
lieunguyen-tma May 25, 2026
a493c7b
Remove test={} to test CI for metadata mibyou db
lieunguyen-tma May 25, 2026
08f24e5
Add weko_docker_compose_path configuration for SWORD Client registrat…
lieunguyen-tma Jun 8, 2026
8da1ec3
Update the code to remove indentation.
lieunguyen-tma Jun 8, 2026
65125d5
Add mibyo_db_sword_mapping_id for test metadata mibyou db
lieunguyen-tma Jun 11, 2026
61e9b2b
Revert to sword_mapping_id for test metadata mibyou db
lieunguyen-tma Jun 11, 2026
d82487b
Add item type and JSON-LD mapping setup for Mibyo DB test metadata in CI
lieunguyen-tma Jun 15, 2026
a85edf1
Merge branch 'main' into feature/metadata-mibyou-db
lieunguyen-tma Jun 16, 2026
68e1b5d
Update source code for Mibyo DB test metadata in CI
lieunguyen-tma Jun 16, 2026
c739d80
Update source code for Mibyo DB test metadata in CI
lieunguyen-tma Jun 16, 2026
3de40ec
Update invalid syntax for Mibyo DB test metadata in CI
lieunguyen-tma Jun 16, 2026
9182b78
Update source code to align with UI changes for Mibyo DB test metadata
lieunguyen-tma Jun 16, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
135 changes: 135 additions & 0 deletions .github/patches/sword_mapping_51000.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
{
"PubDate": "datePublished",
"データセットの名称": "rdm:name",
"データセットの名称.タイトル": "rdm:name.value.value",
"データセットの名称.言語": "rdm:name.value.language",
"プロジェクト名.プログラム情報識別子.プログラム情報識別子": "name.@id",
"データID.関連名称.関連名称": "identifier",
"メタデータ登録日.日付": "dateCreated",
"メタデータ登録日.日付タイプ": "$Created",
"メタデータ更新日.日付": "dateModified",
"メタデータ更新日.日付タイプ": "$Updated",
"実験(もしくは解析結果・解析ツールなど)の目的": "ams:purposeOfExperiment",
"実験(もしくは解析結果・解析ツールなど)の目的.値": "ams:purposeOfExperiment.value",
"実験(もしくは解析結果・解析ツールなど)の目的.言語": "ams:purposeOfExperiment.language",
"実験状況など(もしくは解析結果・解析ツールの背景など)の説明": "ams:descriptionOfExperimentalCondition",
"実験状況など(もしくは解析結果・解析ツールの背景など)の説明.値": "ams:descriptionOfExperimentalCondition.value",
"実験状況など(もしくは解析結果・解析ツールの背景など)の説明.言語": "ams:descriptionOfExperimentalCondition.language",
"キーワード": "keywords",
"キーワード.主題": "keywords.value",
"キーワード.言語": "keywords.language",
"データセットの分野": "rdm:field",
"データセットの分野.主題": "rdm:field.value",
"データセットの分野.言語": "rdm:field.language",
"解析対象データ": "ams:analysisType",
"解析対象データ.値": "ams:analysisType.value",
"解析対象データ.言語": "ams:analysisType.language",
"解析対象データ(その他)": "ams:analysisTypeAlt",
"解析対象データ(その他).値": "ams:analysisTypeAlt.value",
"解析対象データ(その他).言語": "ams:analysisTypeAlt.language",
"連絡・許諾の要不要": "rdm:licenseInformation.ams:necessityOfContactAndPermission",
"連絡・許諾の要不要.値": "rdm:licenseInformation.ams:necessityOfContactAndPermission.value",
"連絡・許諾の要不要.言語": "rdm:licenseInformation.ams:necessityOfContactAndPermission.language",
"謝辞に記載の要不要": "rdm:licenseInformation.ams:necessityOfIncludingInAcknowledgments",
"謝辞に記載の要不要.値": "rdm:licenseInformation.ams:necessityOfIncludingInAcknowledgments.value",
"謝辞に記載の要不要.言語": "rdm:licenseInformation.ams:necessityOfIncludingInAcknowledgments.language",
"謝辞に記載する名称(個人名、グループ名等)、または謝辞全文": "rdm:licenseInformation.ams:namesToBeIncludedInTheAcknowledgments",
"謝辞に記載する名称(個人名、グループ名等)、または謝辞全文.値": "rdm:licenseInformation.ams:namesToBeIncludedInTheAcknowledgments.value",
"謝辞に記載する名称(個人名、グループ名等)、または謝辞全文.言語": "rdm:licenseInformation.ams:namesToBeIncludedInTheAcknowledgments.language",
"その他条件、あるいは、特記事項": "rdm:licenseInformation.ams:otherConditionsOrSpecialNotes",
"その他条件、あるいは、特記事項.値": "rdm:licenseInformation.ams:otherConditionsOrSpecialNotes.value",
"その他条件、あるいは、特記事項.言語": "rdm:licenseInformation.ams:otherConditionsOrSpecialNotes.language",
"ライセンス": "license",
"ライセンス.ライセンス": "license.name",
"有償・無償": "rdm:licenseInformation.ams:dataPolicyFree",
"有償・無償.値": "rdm:licenseInformation.ams:dataPolicyFree.value",
"有償・無償.言語": "rdm:licenseInformation.ams:dataPolicyFree.language",
"商用利用の可否": "rdm:licenseInformation.ams:availabilityOfCommercialUse",
"商用利用の可否.値": "rdm:licenseInformation.ams:availabilityOfCommercialUse.value",
"商用利用の可否.言語": "rdm:licenseInformation.ams:availabilityOfCommercialUse.language",
"アクセス権.アクセス権": "rdm:accessRightsInformation",
"(アクセス権が「公開」でない場合)公開予定日.日付": "rdm:dateAvailable",
"(アクセス権が「公開」でない場合)公開予定日.日付タイプ": "$Available",
"リポジトリURL・DOIリンク": "rdm:storedIn",
"リポジトリURL・DOIリンク.関連タイプ": "rdm:storedIn.relationType",
"リポジトリURL・DOIリンク.関連名称.関連名称": "rdm:storedIn.rdm:url",
"リポジトリURL・DOIリンク.関連識別子.関連識別子": "rdm:storedIn.@id",
"その他補足事項": "rdm:storedIn",
"その他補足事項.値": "rdm:storedIn.rdm:description",
"データ作成者": "creator",
"データ作成者.作成者姓": "creator.familyName",
"データ作成者.作成者姓.姓": "creator.familyName.value",
"データ作成者.作成者姓.言語": "creator.familyName.language",
"データ作成者.作成者姓名": "creator.name",
"データ作成者.作成者姓名.姓名": "creator.name.value",
"データ作成者.作成者姓名.言語": "creator.name.language",
"データ作成者.作成者名": "creator.givenName",
"データ作成者.作成者名.名": "creator.givenName.value",
"データ作成者.作成者名.言語": "creator.givenName.language",
"データ作成者.作成者識別子": "creator.identifier",
"データ作成者.作成者識別子.作成者識別子": "creator.identifier.value",
"データ作成者.作成者識別子.作成者識別子Scheme": "creator.identifier.jpcoar:nameIdentifierScheme",
"データ作成者.作成者所属": "creator.affiliation",
"データ作成者.作成者所属.所属機関名": "creator.affiliation.name",
"データ作成者.作成者所属.所属機関名.所属機関名": "creator.affiliation.name.value",
"データ作成者.作成者所属.所属機関名.言語": "creator.affiliation.name.language",
"データ作成者.作成者所属.所属機関識別子.所属機関識別子": "creator.affiliation.identifier.value",
"データ作成者.作成者所属.所属機関識別子.所属機関識別子Scheme": "creator.affiliation.identifier.jpcoar:nameIdentifierScheme",
"データ作成者.作成者所属.所属機関識別子.所属機関識別子URI": "creator.affiliation.identifier.uri",
"データ作成者.作成者別名": "creator.alternateName",
"データ作成者.作成者別名.別名": "creator.alternateName.value",
"データ作成者.作成者別名.言語": "creator.alternateName.language",
"データ管理者": "contributor",
"データ管理者.寄与者名": "contributor.givenName",
"データ管理者.寄与者名.名": "contributor.givenName.value",
"データ管理者.寄与者名.言語": "contributor.givenName.language",
"データ管理者.寄与者姓": "contributor.familyName",
"データ管理者.寄与者姓.姓": "contributor.familyName.value",
"データ管理者.寄与者姓.言語": "contributor.familyName.language",
"データ管理者.寄与者姓名": "contributor.name",
"データ管理者.寄与者姓名.姓名": "contributor.name.value",
"データ管理者.寄与者姓名.言語": "contributor.name.language",
"データ管理者.寄与者所属": "contributor.affiliation",
"データ管理者.寄与者所属.所属機関名": "contributor.affiliation.name",
"データ管理者.寄与者所属.所属機関名.所属機関名": "contributor.affiliation.name.value",
"データ管理者.寄与者所属.所属機関名.言語": "contributor.affiliation.name.language",
"データ管理者.寄与者所属.所属機関識別子": "contributor.affiliation.identifier",
"データ管理者.寄与者所属.所属機関識別子.所属機関識別子": "contributor.affiliation.identifier.value",
"データ管理者.寄与者所属.所属機関識別子.所属機関識別子Scheme": "contributor.affiliation.identifier.jpcoar:nameIdentifierScheme",
"データ管理者.寄与者所属.所属機関識別子.所属機関識別子URI": "contributor.affiliation.identifier.uri",
"データ管理者.寄与者識別子": "contributor.identifier",
"データ管理者.寄与者識別子.寄与者識別子": "contributor.identifier.value",
"データ管理者.寄与者識別子.寄与者識別子Scheme": "contributor.identifier.jpcoar:nameIdentifierScheme",
"データ管理者.寄与者識別子.寄与者識別子URI": "contributor.identifier.uri",
"データ管理者.寄与者別名": "contributor.alternateName",
"データ管理者.寄与者別名.別名": "contributor.alternateName.value",
"データ管理者.寄与者別名.言語": "contributor.alternateName.language",
"取得データの対象種別": "ams:targetTypeOfAcquiredData",
"取得データの対象種別.値": "ams:targetTypeOfAcquiredData.value",
"取得データの対象種別.言語": "ams:targetTypeOfAcquiredData.language",
"(IC有の場合)第三者提供の同意.値": "rdm:AccessRights.ams:consentForProvisionToAThirdParty",
"(IC有の場合)海外提供.値": "rdm:AccessRights.ams:overseasOfferings",
"(IC有の場合)産業利用等.値": "rdm:AccessRights.ams:industrialUse",
"(IC無の場合).値": "rdm:AccessRights.ams:icIsNo",
"(ヒト)匿名加工の有無.値": "rdm:AccessRights.ams:anonymousProcessing",
"備考.値": "rdm:description",
"資源タイプ": "dc:type",
"資源タイプ.資源タイプ": "dc:type.value",
"資源タイプ.資源タイプ識別子": "dc:type.@id",
"ファイル情報": "hasPart",
"ファイル情報.ファイル名": "hasPart.name",
"ファイル情報.サイズ.サイズ": "hasPart.contentSize",
"ファイル情報.バージョン情報": "hasPart.version",
"ファイル情報.フォーマット": "hasPart.encodingFormat",
"ファイル情報.日付": "hasPart.datetime",
"ファイル情報.日付.日付": "hasPart.datetime.date",
"ファイル情報.日付.日付タイプ": "hasPart.datetime.type",
"ファイル情報.アクセス": "hasPart.accessRights",
"ファイル情報.公開日.公開日": "hasPart.datePublished",
"ファイル情報.本文URL": "hasPart.identifier",
"ファイル情報.本文URL.ラベル": "hasPart.@id",
"ファイル情報.本文URL.オブジェクトタイプ": "hasPart.identifier.jpcoar:objectType",
"プロジェクトURL": "ams:projectid",
"プロジェクトURL.関連識別子.関連識別子": "ams:projectid.value",
"プロジェクトURL.関連タイプ": "$isVersionOf"
}
2 changes: 1 addition & 1 deletion .github/scripts/setup_rdm.sh
Original file line number Diff line number Diff line change
Expand Up @@ -337,7 +337,7 @@ EOL
bash "${script_dir}/setup_minio.sh" apply "${PWD}"
fi

if [ "${WEKO_ENABLED:-false}" = "true" ]; then
if [ "${WEKO_ENABLED:-false}" = "true" ] || [ "${SKIP_METADATA:-false}" = "false" ]; then
local script_dir
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
python3 "${script_dir}/weko_setup_cert.py" \
Expand Down
69 changes: 69 additions & 0 deletions .github/scripts/setup_weko.sh
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,31 @@ case "$COMMAND" in
exit 1
fi

# Create AMS itemtype
echo 'create ams itemtype'
pg_container=$(docker compose -f "${compose_file}" ps -q postgresql)
docker cp "${WEKO_ROOT}/scripts/demo/ams/item_type_name.sql" "${pg_container}:/tmp/ams_itemtype_name.sql"
docker cp "${WEKO_ROOT}/scripts/demo/ams/item_type.sql" "${pg_container}:/tmp/ams_itemtype.sql"
docker cp "${WEKO_ROOT}/scripts/demo/ams/item_type_mapping.sql" "${pg_container}:/tmp/ams_itemtype_mapping.sql"
docker cp "${WEKO_ROOT}/scripts/demo/ams/rocrate_mapping.sql" "${pg_container}:/tmp/ams_rocrate_mapping.sql"
docker cp "${WEKO_ROOT}/scripts/demo/ams/facet_search_setting.sql" "${pg_container}:/tmp/ams_facet_search_setting.sql"
docker compose -f "${compose_file}" exec -T postgresql psql -U invenio -d invenio -f /tmp/ams_itemtype_name.sql
docker compose -f "${compose_file}" exec -T postgresql psql -U invenio -d invenio -f /tmp/ams_itemtype.sql
docker compose -f "${compose_file}" exec -T postgresql psql -U invenio -d invenio -f /tmp/ams_itemtype_mapping.sql
docker compose -f "${compose_file}" exec -T postgresql psql -U invenio -d invenio -f /tmp/ams_rocrate_mapping.sql
docker compose -f "${compose_file}" exec -T postgresql psql -U invenio -d invenio -f /tmp/ams_facet_search_setting.sql

echo 'add free textarea to file property'
docker compose -f "${compose_file}" exec -T web invenio shell /code/tools/add_free_textarea.py

echo 'create search settings'
record_count=$(docker compose -f "${compose_file}" exec -T postgresql psql -U invenio -d invenio -t -c 'SELECT count(*) FROM search_management;')
if [ "${record_count}" -lt 1 ]; then
docker cp "${WEKO_ROOT}/scripts/demo/ams/search_management.sql" "${pg_container}:/tmp/ams_search_management.sql"
docker compose -f "${compose_file}" exec -T postgresql psql -U invenio -d invenio -f /tmp/ams_search_management.sql
fi
docker compose -f "${compose_file}" exec -T web invenio shell /code/scripts/demo/ams/update_search_management.py

# Update SWORD mapping for item type 30002
echo "=== Updating SWORD Mapping 30002 ==="
mapping_json=$(cat "${SCRIPT_DIR}/../patches/sword_mapping_30002.json")
Expand All @@ -87,6 +112,32 @@ db.session.commit()
print('Updated mapping 30002')
"

# Insert/Update SWORD mapping for item type 51000
echo "=== Updating SWORD Mapping 51000 ==="
mapping_json=$(cat "${SCRIPT_DIR}/../patches/sword_mapping_51000.json")
docker compose -f "${compose_file}" exec -T web invenio shell -c "
from weko_records.api import JsonldMapping
from weko_records.models import ItemTypeJsonldMapping
from invenio_db import db
import json
mapping = json.loads('''${mapping_json}''')
obj = JsonldMapping.get_mapping_by_id(51000)
if obj is None:
obj = ItemTypeJsonldMapping(
id = 51000,
name = '未病アイテムタイプ',
mapping = mapping,
item_type_id = 51000,
)
db.session.add(obj)
db.session.commit()
print('Created mapping 51000')
else:
obj.mapping = mapping
db.session.commit()
print('Updated mapping 51000')
"

# Grant contributor access to Sample Index
echo "=== Granting Contributor Access to Sample Index ==="
docker compose -f "${compose_file}" exec -T web invenio shell -c '
Expand Down Expand Up @@ -135,6 +186,24 @@ obj = JsonldMapping.get_mapping_by_id(30002)
errs = JsonLdMapper(obj.item_type_id, obj.mapping).validate()
result = {"mapping_id": obj.id, "item_type_id": obj.item_type_id, "name": obj.name, "valid": errs is None, "errors": errs or []}
print(json.dumps({"result": result, "invalid": 0 if errs is None else 1}, ensure_ascii=False))
')
echo "${validation_result}"
invalid_count=$(echo "${validation_result}" | python3 -c "import sys, json; print(json.load(sys.stdin)['invalid'])")
if [[ "${invalid_count}" -gt 0 ]]; then
echo "SWORD mapping validation failed: ${invalid_count} invalid mapping(s)" >&2
exit 1
fi

# Validate SWORD mapping 51000
echo "=== SWORD Mapping Validation (51000) ==="
validation_result=$(docker compose -f "${compose_file}" exec -T web invenio shell -c '
from weko_records.api import JsonldMapping
from weko_search_ui.mapper import JsonLdMapper
import json
obj = JsonldMapping.get_mapping_by_id(51000)
errs = JsonLdMapper(obj.item_type_id, obj.mapping).validate()
result = {"mapping_id": obj.id, "item_type_id": obj.item_type_id, "name": obj.name, "valid": errs is None, "errors": errs or []}
print(json.dumps({"result": result, "invalid": 0 if errs is None else 1}, ensure_ascii=False))
')
echo "${validation_result}"
invalid_count=$(echo "${validation_result}" | python3 -c "import sys, json; print(json.load(sys.stdin)['invalid'])")
Expand Down
23 changes: 12 additions & 11 deletions .github/workflows/e2e-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ jobs:
test-group:
- name: user
display_name: "User Tests"
include_admin: false
include_admin: true
skip_admin: true
skip_metadata: false
skip_autofill: true
Expand Down Expand Up @@ -165,7 +165,7 @@ jobs:
flowable_enabled: false
- name: user-minio
display_name: "User Tests (MinIO)"
include_admin: false
include_admin: true
skip_admin: true
skip_metadata: false
skip_autofill: true
Expand Down Expand Up @@ -276,6 +276,7 @@ jobs:
WEKO_ENABLED: ${{ matrix.test-group.weko_enabled == true && 'true' || 'false' }}
JUPYTERHUB_ENABLED: ${{ matrix.test-group.jupyterhub_enabled == true && 'true' || 'false' }}
FLOWABLE_ENABLED: ${{ matrix.test-group.flowable_enabled == true && 'true' || 'false' }}
SKIP_METADATA: ${{ matrix.test-group.skip_metadata == true && 'true' || 'false' }}
MINIO_ROOT_USER: e2eadmin
MINIO_ROOT_PASSWORD: e2epassword123
MINIO_ALIAS: ci-minio
Expand Down Expand Up @@ -385,7 +386,7 @@ jobs:
fetch-depth: 0

- name: Checkout WEKO repository
if: env.WEKO_ENABLED == 'true'
if: env.WEKO_ENABLED == 'true' || env.SKIP_METADATA == 'false'
uses: actions/checkout@v4
with:
repository: RCOSDP/weko
Expand All @@ -400,7 +401,7 @@ jobs:
path: RDM-flowable-gateway

- name: Prepare WEKO compose
if: env.WEKO_ENABLED == 'true'
if: env.WEKO_ENABLED == 'true' || env.SKIP_METADATA == 'false'
working-directory: e2e-tests
run: |
.github/scripts/setup_weko.sh prepare ../weko
Expand Down Expand Up @@ -945,13 +946,13 @@ jobs:
test_rdm_endpoints

- name: Install WEKO stack
if: env.WEKO_ENABLED == 'true'
if: env.WEKO_ENABLED == 'true' || env.SKIP_METADATA == 'false'
working-directory: e2e-tests
run: |
.github/scripts/setup_weko.sh install ../weko

- name: Configure WEKO connection info
if: env.WEKO_ENABLED == 'true'
if: env.WEKO_ENABLED == 'true' || env.SKIP_METADATA == 'false'
run: |
echo "WEKO_URL=https://192.168.168.167" >> $GITHUB_ENV
echo "WEKO_ADMIN_EMAIL=wekosoftware@nii.ac.jp" >> $GITHUB_ENV # gitleaks:allow
Expand All @@ -965,7 +966,7 @@ jobs:
echo "IGNORE_HTTPS_ERRORS=true" >> $GITHUB_ENV

- name: Verify RDM containers can connect to WEKO via HTTPS
if: env.WEKO_ENABLED == 'true'
if: env.WEKO_ENABLED == 'true' || env.SKIP_METADATA == 'false'
working-directory: RDM-osf.io
run: |
set -xeu
Expand Down Expand Up @@ -1140,7 +1141,7 @@ jobs:
if [ "${JUPYTERHUB_ENABLED}" = "true" ]; then
args+=(--jupyterhub)
fi
if [ "${WEKO_ENABLED}" = "true" ]; then
if [ "${WEKO_ENABLED}" = "true" ] || [ "${SKIP_METADATA}" = "false" ]; then
args+=(--weko)
fi
if [ "${FLOWABLE_ENABLED}" = "true" ]; then
Expand Down Expand Up @@ -1278,13 +1279,13 @@ jobs:
if-no-files-found: ignore

- name: Collect WEKO logs
if: always() && env.WEKO_ENABLED == 'true'
if: always() && (env.WEKO_ENABLED == 'true' || env.SKIP_METADATA == 'false')
working-directory: weko
run: |
docker compose -f docker-compose2.yml logs > ../e2e-tests/weko-docker-logs.txt 2>&1 || true

- name: Upload WEKO logs
if: always() && env.WEKO_ENABLED == 'true'
if: always() && (env.WEKO_ENABLED == 'true' || env.SKIP_METADATA == 'false')
uses: actions/upload-artifact@v4
with:
name: weko-logs-${{ matrix.test-group.name }}
Expand Down Expand Up @@ -1319,7 +1320,7 @@ jobs:
retention-days: 7

- name: Stop WEKO stack
if: always() && env.WEKO_ENABLED == 'true'
if: always() && (env.WEKO_ENABLED == 'true' || env.SKIP_METADATA == 'false')
working-directory: e2e-tests
run: |
.github/scripts/setup_weko.sh down ../weko
Expand Down
Binary file not shown.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Loading