Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 31 additions & 3 deletions config_operator/requirements.txt → .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,36 @@
# See the License for the specific language governing permissions and
# limitations under the License.

GitPython==3.1.2
kubernetes==11.0
kopf==0.26
# Keep the Docker build context small and reproducible. The component
# Dockerfiles copy each package's source wholesale (e.g. COPY common/ common/),
# so anything not excluded here would otherwise leak into the image.

# Version control
.git
.gitignore
.github

# Local virtual environments (the image builds its own via `uv sync`)
.venv
**/.venv

# Python build/runtime artifacts
**/__pycache__
**/*.py[cod]
**/*.egg-info
**/*.egg
**/build
**/dist
.python-version

# Test / tooling caches
**/.pytest_cache
**/.mypy_cache
**/.ruff_cache
.coverage
htmlcov

# Editor / OS noise
.idea
.vscode
.DS_Store
11 changes: 11 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Normalize line endings so Linux container builds are not broken by CRLF
# introduced on Windows checkouts. Shell scripts and Dockerfiles are executed
# inside Linux images and MUST stay LF regardless of core.autocrlf.
*.sh text eol=lf
*.bash text eol=lf
Dockerfile text eol=lf
entrypoint text eol=lf

# uv lockfile is generated/consumed verbatim; keep it LF and untouched by
# end-of-line normalization heuristics.
uv.lock text eol=lf
1 change: 1 addition & 0 deletions .python-version
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
3.11
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Changed
- Properly handle skipping of remote collections when reloading the collections config in the CM.
- SDAP-542: Updated collection manager and granule ingester build to python 3.11.
- SDAP-543: Migrated package and dependency management to a uv workspace with a single root `uv.lock`. This supersedes Poetry (SDAP-511), which had itself replaced the original setuptools (`setup.py`) plus `requirements.txt` install workflow. Component Docker builds now use `uv sync --package <component>`.
### Deprecated
### Removed
### Fixed
Expand Down
21 changes: 8 additions & 13 deletions collection_manager/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,24 +9,21 @@ Manager service will publish a message to RabbitMQ to be picked up by the Granul

## Prerequisites

Python 3.11

Use a conda environment for example:

$ conda create -n cmenv python=3.11
$ conda activate cmenv
Python 3.11 and [uv](https://docs.astral.sh/uv/) (uv provisions and manages the
virtual environment).

## Building the service
From `incubator-sdap-ingester`, run:

$ cd common && python setup.py install
$ cd ../collection_manager python setup.py install

$ uv sync --package sdap_collection_manager

This creates `.venv` with the `sdap_collection_manager` package and its dependencies.


## Running the service
From `incubator-sdap-ingester`, run:

$ python collection_manager/collection_manager/main.py -h
$ uv run --package sdap_collection_manager python collection_manager/collection_manager/main.py -h

### The Collections Configuration File

Expand Down Expand Up @@ -111,9 +108,7 @@ the actual dimensions are referenced by index variables.
## Running the tests
From `incubator-sdap-ingester/`, run:

$ cd common && python setup.py install
$ cd ../collection_manager && python setup.py install
$ pip install pytest && pytest
$ uv run --package sdap_collection_manager pytest collection_manager/tests

## Building the Docker image
From `incubator-sdap-ingester`, run:
Expand Down
33 changes: 24 additions & 9 deletions collection_manager/docker/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,9 @@
# See the License for the specific language governing permissions and
# limitations under the License.

# Build from the repository root:
# docker build -f collection_manager/docker/Dockerfile -t sdap-collection-manager .

FROM python:3.11-slim

ARG KUBECTL_VERSION=v1.29.14
Expand All @@ -22,15 +25,27 @@ RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates
apt-get clean && \
rm -rf /var/lib/apt/lists/*

COPY VERSION.txt /VERSION.txt
COPY common /common
COPY collection_manager/collection_manager /collection_manager/collection_manager
COPY collection_manager/setup.py /collection_manager/setup.py
COPY collection_manager/requirements.txt /collection_manager/requirements.txt
COPY collection_manager/README.md /collection_manager/README.md
COPY collection_manager/docker/entrypoint.sh /entrypoint.sh
COPY --from=ghcr.io/astral-sh/uv:0.11.3 /uv /uvx /bin/

ENV UV_COMPILE_BYTECODE=1 \
UV_LINK_MODE=copy \
UV_PYTHON_DOWNLOADS=0

RUN cd /common && pip install .
RUN cd /collection_manager && pip install .
WORKDIR /app

COPY pyproject.toml uv.lock ./
COPY common/pyproject.toml common/
COPY collection_manager/pyproject.toml collection_manager/
COPY config_operator/pyproject.toml config_operator/
COPY granule_ingester/pyproject.toml granule_ingester/
RUN --mount=type=cache,target=/root/.cache/uv \
uv sync --frozen --no-dev --no-install-workspace --package sdap_collection_manager

COPY common/ common/
COPY collection_manager/ collection_manager/
RUN --mount=type=cache,target=/root/.cache/uv \
uv sync --frozen --no-dev --package sdap_collection_manager

ENV PATH="/app/.venv/bin:$PATH"
COPY collection_manager/docker/entrypoint.sh /entrypoint.sh
ENTRYPOINT ["/bin/bash", "/entrypoint.sh"]
2 changes: 1 addition & 1 deletion collection_manager/docker/entrypoint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
# limitations under the License.


python /collection_manager/collection_manager/main.py \
python /app/collection_manager/collection_manager/main.py \
$([[ ! -z "$COLLECTIONS_PATH" ]] && echo --collections-path=$COLLECTIONS_PATH) \
$([[ ! -z "$RABBITMQ_HOST" ]] && echo --rabbitmq-host=$RABBITMQ_HOST) \
$([[ ! -z "$RABBITMQ_USERNAME" ]] && echo --rabbitmq-username=$RABBITMQ_USERNAME) \
Expand Down
65 changes: 65 additions & 0 deletions collection_manager/pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

[project]
name = "sdap_collection_manager"
version = "1.4.0"
description = "A helper that watches a YAML file stored on the filesystem, and all the directories listed in that file."
authors = [{ name = "SDAP PMC", email = "dev@sdap.apache.org" }]
requires-python = ">=3.11,<3.12"
readme = "README.md"
license = "Apache-2.0"
classifiers=[
"Programming Language :: Python :: 3",
"Operating System :: OS Independent",
"Development Status :: 4 - Beta",
]
dependencies = [
"sdap_ingester_common",

"aio-pika==6.7.1",
"aioboto3==11.3.0",
"aiobotocore==2.6.0",
"aiohttp==3.8.5",
"boto3==1.28.17",
"pysolr==3.10.0",
"pyyaml==6.0.2",
"requests==2.31.0",
"tenacity==8.2.3",
"urllib3==1.26.20",
"watchdog==0.10.2",
"yarl<=1.9.4",
]

[project.urls]
Repository = "https://github.com/apache/incubator-sdap-ingester"

[dependency-groups]
dev = ["pytest>=7.1.2,<8"]

[tool.uv]

[tool.uv.build-backend]
module-name = ["collection_manager"]
module-root = ""

[build-system]
requires = ["uv_build>=0.11.3,<0.12.0"]
build-backend = "uv_build"

[tool.pytest.ini_options]
filterwarnings = [
"ignore::DeprecationWarning",
]
27 changes: 0 additions & 27 deletions collection_manager/requirements.txt

This file was deleted.

49 changes: 0 additions & 49 deletions collection_manager/setup.py

This file was deleted.

49 changes: 31 additions & 18 deletions common/setup.py → common/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -13,24 +13,37 @@
# See the License for the specific language governing permissions and
# limitations under the License.

import re

import setuptools

PACKAGE_NAME = "sdap_ingester_common"

setuptools.setup(
name=PACKAGE_NAME,
author="Apache - SDAP",
author_email="dev@sdap.apache.org",
description="a module of common functions for the sdap ingester components",
url="https://github.com/apache/incubator-sdap-ingester",
packages=setuptools.find_packages(),
classifiers=[
[project]
name = "sdap_ingester_common"
version = "1.4.0"
description = "A module of common functions for the sdap ingester components"
authors = [{ name = "SDAP PMC", email = "dev@sdap.apache.org" }]
requires-python = ">=3.11,<3.12"
license = "Apache-2.0"
classifiers=[
"Programming Language :: Python :: 3",
"Operating System :: OS Independent",
"Development Status :: 4 - Beta",
],
python_requires='>=3.11',
include_package_data=True
)
]
dependencies = []

[project.urls]
Repository = "https://github.com/apache/incubator-sdap-ingester"

[dependency-groups]
dev = ["pytest>=7.1.2,<8"]

[tool.uv]

[tool.uv.build-backend]
module-name = ["common"]
module-root = ""

[build-system]
requires = ["uv_build>=0.11.3,<0.12.0"]
build-backend = "uv_build"

[tool.pytest.ini_options]
filterwarnings = [
"ignore::DeprecationWarning",
]
14 changes: 9 additions & 5 deletions config_operator/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,16 +21,20 @@ To update the configmap from the same directory run:

# Developers

git clone ...
cd config_operator
pip install -e .
pytest -d
Python 3.11 and [uv](https://docs.astral.sh/uv/) (uv provisions and manages the
virtual environment). From `incubator-sdap-ingester`, run:

uv sync --package config_operator
uv run --package config_operator pytest config_operator/tests

# Containerization

## Docker

docker build . -f containers/docker/Dockerfile -t nexusjpl/config-operator:latest
From `incubator-sdap-ingester` (the build needs the repository root as context so
uv can resolve the workspace):

docker build . -f config_operator/containers/docker/Dockerfile -t nexusjpl/config-operator:latest

To publish the docker image on dockerhub do (step necessary for kubernetes deployment):

Expand Down
Loading