Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/nightly_examples.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ on:
model:
description: "Model to run the examples against"
required: false
default: "gpt-4o-mini"
default: "gpt-4.1-mini"

jobs:
examples:
Expand All @@ -43,7 +43,7 @@ jobs:
# example is a subprocess, so xdist gets real parallelism across them.
- name: Run every example
env:
EFFECTFUL_LLM_MODEL: ${{ github.event.inputs.model || 'gpt-4o-mini' }}
EFFECTFUL_LLM_MODEL: ${{ github.event.inputs.model || 'gpt-4.1-mini' }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
run: |
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/test_llm.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ jobs:
fail-fast: false
matrix:
python-version: ["3.13", "3.14"]
model: ["gpt-4o-mini"]
model: ["gpt-4.1-mini"]
mypy: ["mypy", "not mypy"]
name: "test-llm (py ${{ matrix.python-version }}, ${{ matrix.mypy }})"
steps:
Expand Down Expand Up @@ -66,7 +66,7 @@ jobs:

- name: Run the basics examples
env:
EFFECTFUL_LLM_MODEL: gpt-4o-mini
EFFECTFUL_LLM_MODEL: gpt-4.1-mini
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
run: |
Expand Down
2 changes: 1 addition & 1 deletion docs/source/llm_examples/choreographies/library.py
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ async def build_codebase(project_spec, architect, coder, reviewer):
Calling it is all there is to it -- the skill calls inside need a model behind
them, which the module launcher supplies::

python -m effectful.handlers.llm.harness your_choreography.py --model gpt-4o-mini
python -m effectful.handlers.llm.harness your_choreography.py

``multi_agent_choreography.py``, alongside this module, is a complete, runnable
version: agents with tools, a review-and-fix loop, and resumption.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
left off::

python -m effectful.handlers.llm.harness \\
docs/source/llm_examples/choreographies/multi_agent_choreography.py --model gpt-4o-mini
docs/source/llm_examples/choreographies/multi_agent_choreography.py

Use ``--restart`` to forget the recorded steps and build from scratch, and pass
``--persist-db PATH`` to the harness to checkpoint each agent's own
Expand Down
16 changes: 0 additions & 16 deletions docs/source/llm_examples/optimization/ds1000.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,22 +31,6 @@
- the data/skill module split that keeps the held-out test honest: each
problem's ``code_context`` contains its reference solution, and lives in a
module the solver's system prompt only names

Run with::

python -m effectful.handlers.llm.harness \\
docs/source/llm_examples/optimization/ds1000.py \\
--model gpt-5-mini --reasoning-effort low --tool-choice none

Expect a few minutes: ~10 solver calls plus a backward call per training
problem and one accumulation per parameter. Results are stochastic run to run,
and the model matters: the run above flipped the held-out problem FAIL -> PASS
(the direct attempt returned the whole ``OptimizeResult`` where the test wants
``res.x``; training on ``scipy_716`` taught exactly that contract). A weaker
model (gpt-4o-mini) visibly *learns* -- its trained attempts fix the
L-BFGS-B bounds format its direct attempt crashed on -- but tends to keep
failing the held-out problem on some other of its four simultaneous
requirements.
"""

import argparse
Expand Down
2 changes: 1 addition & 1 deletion tests/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
from effectful.handlers.llm.harness.legibility.lexical import _tools_in_scope
from effectful.ops.syntax import ObjectInterpretation, implements

EFFECTFUL_LLM_MODEL = os.environ.get("EFFECTFUL_LLM_MODEL", "gpt-4o-mini")
EFFECTFUL_LLM_MODEL = os.environ.get("EFFECTFUL_LLM_MODEL", "gpt-4.1-mini")

_HAS_LLM_API_KEY = litellm.validate_environment(model=EFFECTFUL_LLM_MODEL)[
"keys_in_environment"
Expand Down
Loading