Skip to content

Commit 3549052

Browse files
claudeclaude
committed
feat: initial release of working Python wrapper
This release includes: - Functional Python wrapper that calls npx promptfoo@latest - Automatic Node.js detection with helpful error messages - Support for all promptfoo CLI commands and arguments - Proper packaging with pyproject.toml and src/ layout - CI/CD with GitHub Actions (lint, typecheck, test, build) - Release automation with release-please - Cross-platform support (Linux, macOS, Windows) - Python 3.9-3.13 compatibility The previous 0.1.0 version was non-functional. This 0.2.0 release is a complete rewrite with actual working code. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
0 parents  commit 3549052

10 files changed

Lines changed: 667 additions & 0 deletions

File tree

‎.github/workflows/release-please.yml‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
name: release-please
2+
run-name: promptfoo-python release by @${{ github.actor }}
3+
4+
concurrency:
5+
group: release-please-${{ github.ref }}
6+
cancel-in-progress: false
7+
8+
on:
9+
push:
10+
branches:
11+
- main
12+
workflow_dispatch:
13+
14+
jobs:
15+
release-please:
16+
runs-on: ubuntu-latest
17+
permissions:
18+
contents: write
19+
pull-requests: write
20+
outputs:
21+
release_created: ${{ steps.release.outputs.release_created }}
22+
tag_name: ${{ steps.release.outputs.tag_name }}
23+
version: ${{ steps.release.outputs.version }}
24+
steps:
25+
- uses: googleapis/release-please-action@v4
26+
id: release
27+
with:
28+
token: ${{ secrets.GITHUB_TOKEN }}
29+
30+
build:
31+
if: needs.release-please.outputs.release_created == 'true'
32+
runs-on: ubuntu-latest
33+
needs: release-please
34+
permissions:
35+
contents: read
36+
steps:
37+
- uses: actions/checkout@v6
38+
39+
- uses: astral-sh/setup-uv@v7
40+
with:
41+
enable-cache: true
42+
43+
- name: Pin Python version
44+
run: uv python pin 3.12
45+
46+
- name: Install dependencies
47+
run: uv sync --extra dev
48+
49+
- name: Run tests
50+
run: |
51+
# Quick smoke test before publishing
52+
uv run python -c "import promptfoo; print(promptfoo.__version__)"
53+
54+
- name: Build package
55+
run: uv build
56+
57+
- name: Verify package version matches release
58+
run: |
59+
EXPECTED_VERSION="${{ needs.release-please.outputs.version }}"
60+
if ls dist/*-${EXPECTED_VERSION}-*.whl 1> /dev/null 2>&1; then
61+
echo "✓ Package version ${EXPECTED_VERSION} matches release"
62+
else
63+
echo "ERROR: Package version mismatch!"
64+
echo "Expected version: ${EXPECTED_VERSION}"
65+
echo "Built packages:"
66+
ls -la dist/
67+
exit 1
68+
fi
69+
70+
- name: Upload build artifacts
71+
uses: actions/upload-artifact@v6
72+
with:
73+
name: dist
74+
path: dist/
75+
76+
publish-pypi:
77+
if: needs.release-please.outputs.release_created == 'true'
78+
needs: [build, release-please]
79+
runs-on: ubuntu-latest
80+
environment:
81+
name: pypi
82+
url: https://pypi.org/project/promptfoo/
83+
permissions:
84+
contents: read
85+
id-token: write
86+
steps:
87+
- name: Download build artifacts
88+
uses: actions/download-artifact@v7
89+
with:
90+
name: dist
91+
path: dist/
92+
93+
- name: Publish to PyPI
94+
uses: pypa/gh-action-pypi-publish@release/v1
95+
with:
96+
print-hash: true

‎.github/workflows/test.yml‎

Lines changed: 146 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,146 @@
1+
name: Python CI
2+
3+
on:
4+
pull_request:
5+
paths-ignore:
6+
- "**.md"
7+
- "LICENSE"
8+
push:
9+
branches:
10+
- main
11+
workflow_dispatch:
12+
13+
permissions:
14+
contents: read
15+
16+
concurrency:
17+
group: ${{ github.workflow }}-${{ github.ref }}
18+
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
19+
20+
jobs:
21+
lint:
22+
name: Lint and Format
23+
runs-on: ubuntu-latest
24+
timeout-minutes: 10
25+
steps:
26+
- uses: actions/checkout@v6
27+
28+
- uses: astral-sh/setup-uv@v7
29+
with:
30+
enable-cache: true
31+
32+
- name: Pin Python version
33+
run: uv python pin 3.12
34+
35+
- name: Install dependencies
36+
run: uv sync --extra dev
37+
38+
- name: Lint with Ruff
39+
run: uv run ruff check src/
40+
41+
- name: Check formatting
42+
run: uv run ruff format --check src/
43+
44+
type-check:
45+
name: Type Check
46+
runs-on: ubuntu-latest
47+
timeout-minutes: 10
48+
steps:
49+
- uses: actions/checkout@v6
50+
51+
- uses: astral-sh/setup-uv@v7
52+
with:
53+
enable-cache: true
54+
55+
- name: Pin Python version
56+
run: uv python pin 3.12
57+
58+
- name: Install dependencies
59+
run: uv sync --extra dev
60+
61+
- name: Type check with mypy
62+
run: uv run mypy src/promptfoo/
63+
64+
test:
65+
name: Test Python ${{ matrix.python-version }}
66+
runs-on: ${{ matrix.os }}
67+
timeout-minutes: 15
68+
strategy:
69+
matrix:
70+
os: [ubuntu-latest, macos-latest, windows-latest]
71+
python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]
72+
steps:
73+
- uses: actions/checkout@v6
74+
75+
- uses: actions/setup-node@v4
76+
with:
77+
node-version: "20"
78+
79+
- uses: astral-sh/setup-uv@v7
80+
with:
81+
enable-cache: true
82+
83+
- name: Pin Python version
84+
run: uv python pin ${{ matrix.python-version }}
85+
86+
- name: Install package
87+
run: uv pip install -e .
88+
89+
- name: Test CLI can be invoked
90+
run: uv run promptfoo --version
91+
92+
- name: Test Node.js detection
93+
run: uv run python -c "from promptfoo.cli import check_node_installed, check_npx_installed; assert check_node_installed(); assert check_npx_installed()"
94+
95+
build:
96+
name: Build Package
97+
runs-on: ubuntu-latest
98+
timeout-minutes: 10
99+
steps:
100+
- uses: actions/checkout@v6
101+
102+
- uses: astral-sh/setup-uv@v7
103+
with:
104+
enable-cache: true
105+
106+
- name: Pin Python version
107+
run: uv python pin 3.12
108+
109+
- name: Build package
110+
run: uv build
111+
112+
- name: Upload artifacts
113+
uses: actions/upload-artifact@v6
114+
with:
115+
name: dist
116+
path: dist/
117+
118+
ci-success:
119+
name: CI Success
120+
needs: [lint, type-check, test, build]
121+
if: always()
122+
runs-on: ubuntu-latest
123+
steps:
124+
- name: Check if all jobs succeeded
125+
run: |
126+
LINT_RESULT="${{ needs.lint.result }}"
127+
TYPE_CHECK_RESULT="${{ needs.type-check.result }}"
128+
TEST_RESULT="${{ needs.test.result }}"
129+
BUILD_RESULT="${{ needs.build.result }}"
130+
131+
echo "Job results:"
132+
echo " lint: $LINT_RESULT"
133+
echo " type-check: $TYPE_CHECK_RESULT"
134+
echo " test: $TEST_RESULT"
135+
echo " build: $BUILD_RESULT"
136+
137+
if [[ "$LINT_RESULT" == "failure" || "$LINT_RESULT" == "cancelled" ||
138+
"$TYPE_CHECK_RESULT" == "failure" || "$TYPE_CHECK_RESULT" == "cancelled" ||
139+
"$TEST_RESULT" == "failure" || "$TEST_RESULT" == "cancelled" ||
140+
"$BUILD_RESULT" == "failure" || "$BUILD_RESULT" == "cancelled" ]]; then
141+
echo "Some CI checks failed!"
142+
exit 1
143+
else
144+
echo "All CI checks passed!"
145+
exit 0
146+
fi

‎.gitignore‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
# Python
2+
__pycache__/
3+
*.py[cod]
4+
*$py.class
5+
*.so
6+
.Python
7+
build/
8+
develop-eggs/
9+
dist/
10+
downloads/
11+
eggs/
12+
.eggs/
13+
lib/
14+
lib64/
15+
parts/
16+
sdist/
17+
var/
18+
wheels/
19+
*.egg-info/
20+
.installed.cfg
21+
*.egg
22+
MANIFEST
23+
24+
# Virtual environments
25+
venv/
26+
ENV/
27+
env/
28+
.venv
29+
30+
# IDE
31+
.vscode/
32+
.idea/
33+
*.swp
34+
*.swo
35+
*~
36+
.DS_Store
37+
38+
# Testing
39+
.pytest_cache/
40+
.coverage
41+
htmlcov/
42+
.tox/
43+
.mypy_cache/
44+
.ruff_cache/
45+
46+
# Distribution
47+
dist/
48+
build/
49+
*.egg-info/

‎.release-please-manifest.json‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
{
2+
".": "0.2.0"
3+
}

‎CHANGELOG.md‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
# Changelog
2+
3+
All notable changes to this project will be documented in this file.
4+
5+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
6+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7+
8+
## [Unreleased]
9+
10+
## [0.2.0] - 2026-01-05
11+
12+
### Added
13+
14+
- Complete rewrite of the Python wrapper with actual working code
15+
- Automatic detection of Node.js and npx availability
16+
- Helpful error messages when Node.js is not installed
17+
- Pass-through of all arguments to the underlying promptfoo CLI
18+
- Support for environment variable configuration
19+
- Proper exit code handling
20+
- Graceful Ctrl+C handling
21+
22+
### Changed
23+
24+
- Now calls `npx promptfoo@latest` to always use the latest version
25+
- Improved README with clear installation and usage instructions
26+
27+
### Fixed
28+
29+
- Package now actually works (previous version was non-functional)
30+
- Proper module structure with `__init__.py` and `cli.py`
31+
- Entry point correctly points to working code
32+
33+
## [0.1.0] - 2024-08-19
34+
35+
### Added
36+
37+
- Initial (non-functional) placeholder release

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2024-2026 Promptfoo Inc.
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

0 commit comments

Comments
 (0)