Ground EMAP’s claims in checked-in artifacts

.gitattributes

@@ -0,0 +1 @@
+*.pdf binary

.github/workflows/ci.yml

@@ -0,0 +1,35 @@
+name: CI
+
+"on":
+  pull_request:
+  push:
+    branches:
+      - main
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  test:
+    name: Python 3.11 tests
+    runs-on: ubuntu-latest
+    timeout-minutes: 20
+
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
+        with:
+          persist-credentials: false
+
+      - name: Set up Python
+        uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
+        with:
+          python-version: "3.11"
+          cache: pip
+
+      - name: Install project and test dependencies
+        run: python -m pip install -e ".[dev]"
+
+      - name: Run tests
+        run: python -m pytest -q

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Noah Ingwers
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -4,42 +4,64 @@
 
 Noah Ingwers, 2025
 
+> **Research status:** Exploratory, non-peer-reviewed work. The findings below summarize
+> 12 recorded GPT-4o-mini runs completed in December 2025; they are not results from a
+> peer-reviewed paper or a full-benchmark evaluation.
+
 ## Overview
 
 EMAP is a framework for studying how resource constraints during evolutionary optimization shape multi-agent LLM architectures. Unlike existing approaches that treat computational cost as a secondary objective, EMAP makes budget constraints the primary evolutionary pressure - forcing architectures to adapt to resource scarcity rather than simply trading off against it.
 
-## Key Findings
-
-We conducted 12 experiments across 4 budget regimes (TIGHT 2K, MEDIUM 5K, LOOSE 10K, UNCONSTRAINED 50K tokens) with 3 random seeds each, evolving multi-agent architectures on HumanEval (164 programming tasks).
-
-### Main Results
-
-| Regime | Budget | Pass@1 | Avg Agents | Avg Edges |
-|--------|--------|--------|------------|-----------|
-| TIGHT | 2,000 tokens | 98.4% ± 2.3% | 3.0 | 2.6 |
-| MEDIUM | 5,000 tokens | 100.0% ± 0.0% | 3.0 | 2.4 |
-| LOOSE | 10,000 tokens | 97.6% ± 2.4% | 3.0 | 1.8 |
-| UNCONSTRAINED | 50,000 tokens | 100.0% ± 0.0% | 2.7 | 2.0 |
-
-### Key Discoveries
-
-1. **Evolution consistently produces 3-agent architectures** regardless of constraint severity, contradicting the hypothesis that tight constraints would favor minimal single-agent systems.
-
-2. **Multiple viable topologies emerge** - evolution discovered diverse architectures achieving equivalent performance:
- - Traditional pipeline (planner -> coder -> reviewer)
- - Test-first pipeline (tester -> reviewer -> planner)
- - Hybrid architecture (generalist -> coder -> architect)
- - Hierarchical 4-agent systems under loose constraints
-
-3. **Budget constraints preserve diversity** - under constrained regimes, different seeds produce different topologies (linear vs cyclic). Without constraints, evolution converges toward simpler linear solutions.
-
-4. **The "Goldilocks Zone"** - MEDIUM budget (5K tokens) achieves perfect performance with lowest token usage, representing optimal constraint severity for HumanEval tasks.
-
-## Computational Cost
-
-Total experimental cost: **3.09M tokens** (~27K API calls) across all 12 experiments.
-Estimated API cost: **<$1.00** using GPT-4o-mini.
-Runtime: 1.5-5.2 hours per experiment depending on budget regime.
+## Recorded Findings
+
+The repository contains 12 completed experiment artifacts: 4 token-budget regimes
+(TIGHT 2K, MEDIUM 5K, LOOSE 10K, and UNCONSTRAINED 50K) with seeds 42, 43, and 44.
+Each recorded configuration used 12 generations and a population of 10. The runner
+is configured to sample 12% of the intended 164-task HumanEval corpus per evaluation.
+Each artifact reports a 41-task final evaluation (25%), consistent with that runner
+configuration but not with a full-benchmark evaluation.
+
+### Recorded Results
+
+| Regime | Budget | Mean final score | Final-pop. agents | Final-pop. edges |
+|--------|--------|------------------|-------------------|------------------|
+| TIGHT | 2,000 tokens | 98.4% ± 2.3% | 3.2 | 2.6 |
+| MEDIUM | 5,000 tokens | 100.0% ± 0.0% | 3.1 | 2.4 |
+| LOOSE | 10,000 tokens | 98.4% ± 2.3% | 3.5 | 2.8 |
+| UNCONSTRAINED | 50,000 tokens | 99.2% ± 1.1% | 2.9 | 2.1 |
+
+Values are means and population standard deviations across three recorded runs.
+Agent and edge counts are final-generation population averages, not the sizes of a
+single selected architecture. These numbers are reproduced from the checked-in
+[summary](experiments/results/focused/summary.json) and
+[per-run JSON artifacts](experiments/results/focused/).
+
+### Descriptive Observations
+
+1. **The recorded final populations averaged roughly three agents.** Mean agent
+   counts ranged from 2.9 to 3.5 across the four regimes.
+2. **The selected genomes used several topologies.** Checked-in artifacts include a
+   planner -> coder -> reviewer pipeline, a tester -> reviewer -> planner topology,
+   a generalist -> coder -> architect topology, and two 4-agent LOOSE-budget systems.
+3. **The MEDIUM runs were the strongest in this small sample.** All three recorded
+   MEDIUM final evaluations passed 41/41 sampled tasks and together used 414,390
+   tokens, the lowest observed total among the four regimes. This does not establish
+   5K tokens as a general optimum.
+4. **Topology diversity is a hypothesis, not a causal result.** The 12 artifacts show
+   different structures across seeds, but three runs per regime on varying task
+   samples cannot establish that budget pressure caused those differences.
+
+## Checked-in Evidence and Cost
+
+- [Aggregate summary](experiments/results/focused/summary.json)
+- [Twelve per-run JSON records](experiments/results/focused/)
+- [Recorded run visualizations](experiments/viz/)
+- [Artifact-grounded research note source](paper/main.tex) and
+  [rendered research note](paper/Scarcity_Breeds_Efficiency__Resource_Constrained_Evolution_of_Multi_Agent_Programming_Architectures.pdf)
+
+The 12 JSON files record **3,093,390 tokens**, **27,360 API calls**, and runtimes of
+approximately **1.7-5.2 hours per run**. They do not separate input from output tokens,
+so a historical dollar cost cannot be independently recalculated from the artifacts.
 
 ## Installation
 
@@ -62,19 +84,31 @@ cp .env.example .env
 ### Run Evolution Experiment
 
 ```bash
-python experiments/run_evolution_experiments.py \
+python experiments/run_single_experiment.py \
     --budget 5000 \
     --seed 42 \
     --generations 12 \
     --population 10
 ```
 
-### Run Tests
+The runner writes to `experiments/results/focused/` and skips a run when a completed
+artifact with the same budget and seed already exists. Preserve the checked-in JSON
+files before attempting a fresh rerun.
+
+### Run Offline Tests
 
 ```bash
-pytest tests/ -v
+python -m pytest tests/ -v
 ```
 
+The test suite uses mock backends and does not require an OpenAI API key.
+
+The code-evaluation harness is **not a security sandbox**. It executes generated
+Python with `exec` in a disposable subprocess and applies static checks,
+restricted builtins/imports, a timeout, and best-effort Unix resource limits.
+Run it only in an unprivileged, isolated environment with no secrets or sensitive
+network/filesystem access.
+
 ### Basic API Usage
 
 ```python
@@ -100,7 +134,7 @@ print(f"Tokens: {result.total_tokens_used}")
 ## Project Structure
 
 ```
-E_M_A_P/
+EMAP/
 ├── src/emap/
 │   ├── genome/
 │   │   ├── representation.py    # MultiAgentGenome, AgentGene
@@ -113,13 +147,14 @@ E_M_A_P/
 │   │   └── executor.py          # LLM backends and message routing
 │   └── benchmarks/
 │       ├── humaneval.py         # HumanEval loader
-│       └── sandbox.py           # Safe code execution
+│       └── sandbox.py           # Best-effort subprocess evaluator
 ├── experiments/
-│   ├── run_evolution_experiments.py
-│   └── results/                 # Experiment JSON outputs
+│   ├── run_single_experiment.py # One budget/seed run
+│   ├── run_all_experiments.py   # Recorded 4 x 3 run matrix
+│   └── results/focused/         # Checked-in JSON artifacts
 ├── paper/
-│   ├── main.tex                 # Research paper
-│   └── references.bib           # Bibliography
+│   ├── main.tex                 # Artifact-grounded research note
+│   └── Scarcity_...pdf          # Rendered research note
 └── tests/
 ```
 
@@ -161,23 +196,63 @@ class MultiAgentGenome:
 - Adjust hyperparameters (temperature, max tokens, message length)
 - Change message format and aggregation strategy
 
+## Reproducibility and Limitations
+
+To run against HumanEval, place `humaneval.jsonl` in `data/`, in
+`src/emap/benchmarks/data/`, or at `~/.cache/emap/humaneval.jsonl`. The loader falls
+back to five simple placeholder tasks when it cannot find that file. The recorded
+artifacts have `final_total: 41`, which is consistent with the configured 25% sample
+of the 164-task HumanEval corpus rather than the placeholder set.
+
+Important limits of the recorded study:
+
+- The `--seed` value controls NumPy-based architecture evolution, but task subsets are
+  selected with Python's unseeded `random` module. The OpenAI calls are also
+  nondeterministic. Exact reruns are therefore not guaranteed, and different runs may
+  have used different 41-task final subsets.
+- The JSON files contain aggregate scores, selected genomes, timing, token totals, and
+  configuration. They do not contain final task IDs, per-task outputs, a dataset hash,
+  a Git commit, or dependency/model version metadata. The GPT-4o-mini attribution is
+  supported by the checked-in runner configuration, not by a model field in each JSON.
+- Project dependencies use lower bounds and no lockfile, so the exact December 2025
+  software environment is not preserved.
+- With three runs per regime and sampled final evaluations, the table supports
+  descriptive comparisons only. It does not establish statistical significance,
+  full-HumanEval performance, causal effects of budget pressure, or generalization to
+  other models and benchmarks.
+- This repository and its research note have not been peer reviewed.
+
 ## Research Questions
 
-**RQ1 (Addressed):** Do architectures evolved under different budget regimes exhibit different structures?
-- Finding: All regimes converge to 3-agent architectures, but topology type (linear vs cyclic) is seed-dependent.
+**RQ1 (Explored descriptively):** Do architectures evolved under different budget regimes exhibit different structures?
 
-**RQ2 (Addressed):** Do constraint-evolved architectures exhibit different coordination strategies?
-- Finding: Evolution discovers multiple viable topologies achieving equivalent performance.
+- Recorded observation: final populations averaged 2.9-3.5 agents, while selected
+  genomes ranged from 2 to 4 agents and included both linear and cyclic topologies.
+  The current sample does not isolate budget from seed, task sampling, or model noise.
 
-**RQ3 & RQ4 (Future Work):** Transfer to abundance and cross-benchmark generalization remain to be tested.
+**RQ2 (Explored descriptively):** Do constraint-evolved architectures exhibit different coordination strategies?
+
+- Recorded observation: selected genomes used sequential, hierarchical, and voting
+  aggregation with structured, freeform, and minimal message formats. Whether the
+  constraints caused those choices remains untested.
+
+**RQ3 & RQ4 (Not evaluated):** Transfer to abundance and cross-benchmark generalization remain future work.
 
 ## Broader Implications
 
-Our findings suggest something counterintuitive: constraints may be features, not bugs. The dominant paradigm in AI development is to maximize resources during training and optimization, then compress or distill afterward. But biological evolution tells a different story - organisms that evolved under scarcity develop fundamentally different adaptations than those subjected to scarcity after the fact. A desert plant has different anatomy than a rainforest plant experiencing drought.
+EMAP investigates a counterintuitive hypothesis: constraints may be useful design
+pressures rather than only deployment limits. Biological adaptation under scarcity is
+the motivating analogy, not evidence for an equivalent mechanism in LLM systems.
 
-We observe an analogous phenomenon in multi-agent architectures. Under budget pressure, evolution discovers cyclic feedback topologies, compressed message formats, and aggressive early-exit strategies. Remove that pressure, and evolution converges toward simpler linear pipelines. The constraint doesn't just filter solutions - it shapes the search process itself, maintaining diversity and preventing premature convergence to local optima.
+The checked-in runs contain cyclic feedback topologies, compressed message formats,
+and simpler linear pipelines. Because there are only three runs per regime and final
+task subsets were not controlled, these observations motivate a larger preregistered
+comparison; they do not demonstrate that budget pressure caused the structures or
+preserved diversity.
 
-This has implications beyond code generation. As LLM costs and environmental impact come under scrutiny, the ability to evolve efficient architectures under realistic constraints becomes increasingly valuable. Rather than designing for abundance and hoping efficiency follows, we can design for scarcity and let evolution discover what's truly necessary.
+If the effect holds under controlled, full-benchmark evaluation, evolving systems
+under realistic resource constraints could be useful beyond code generation. Testing
+that proposition across models, benchmarks, and cost measures remains future work.
 
 ## Future Directions
 
@@ -191,15 +266,22 @@ This has implications beyond code generation. As LLM costs and environmental imp
 ## Citation
 
 ```bibtex
-@article{ingwers2025emap,
-  title={Scarcity Breeds Efficiency: Resource-Constrained Evolution of
-         Multi-Agent Programming Architectures},
+@misc{ingwers2025emap,
+  title={EMAP: An Exploratory Artifact Report on Resource-Constrained
+         Evolution of Multi-Agent Programming Architectures},
   author={Ingwers, Noah},
-  journal={arXiv preprint},
-  year={2025}
+  year={2025},
+  note={Unpublished, non-peer-reviewed research note and software repository}
 }
 ```
 
+## Security
+
+Report suspected vulnerabilities privately as described in
+[SECURITY.md](SECURITY.md). The generated-code evaluator is a best-effort
+experiment harness, not a security sandbox; its documented boundary and
+reporting scope are stated there.
+
 ## License
 
-MIT License
+[MIT License](LICENSE)

SECURITY.md

@@ -0,0 +1,37 @@
+# Security Policy
+
+EMAP is exploratory research software. It has not been independently audited,
+and this policy does not create a bug bounty, service-level agreement, or
+guarantee of payment.
+
+## Supported version
+
+Security fixes target the current `main` branch. Please reproduce an issue
+against the latest commit before reporting it.
+
+## Report privately
+
+Use GitHub's private vulnerability reporting form:
+
+https://github.com/noah-ing/EMAP/security/advisories/new
+
+If the form is unavailable, open a public issue asking for a private reporting
+channel, but do not include exploit details, credentials, sensitive data, or
+working generated-code payloads in that issue.
+
+Include a minimal reproduction, affected commit and component, expected and
+observed behavior, impact, preconditions, and any suggested remediation.
+
+The generated-code evaluator is explicitly a best-effort experiment harness,
+not a security sandbox. Python object-model escapes and platform-specific
+resource-limit gaps are therefore documented limitations. Run it only in an
+unprivileged external sandbox with no secrets or sensitive network/filesystem
+access. Reports remain useful when the implementation violates a documented
+fail-closed behavior, silently falls back to in-process execution, exposes a
+credential, or introduces an unsafe default outside those stated limits.
+
+Do not test against systems, accounts, models, or data you do not own or have
+written permission to assess. Do not exfiltrate data, degrade a service, or
+retain sensitive material beyond what is necessary to demonstrate the issue.
+
+Please allow a reasonable remediation window before public disclosure.