Skip to content

[Feature] Added parallelism to ventis build for speedup #22

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Saaketh0 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[Feature] Added parallelism to ventis build for speedup #22

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .dockerignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,3 +19,4 @@ AWSCLIV2.pkg
docker_container
grpc_stubs
stubs
tests
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,3 +40,4 @@ docker_container/
!.env.example
AWSCLIV2.pkg
.python-version
uv.lock
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ Note: Installation of ventis only needs to be done on the machine where you are

- **Python 3.10+**
- **Docker** — Used to manage agents.
- **Docker Buildx** (optional) — If available, `ventis build` builds all agent/workflow images in a single parallel `docker buildx bake` pass; otherwise it falls back to building them sequentially.

---

Expand Down
213 changes: 155 additions & 58 deletions tests/test_cli.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import json
import os
import sys
import tempfile
Expand Down Expand Up @@ -98,79 +99,175 @@ def test_preflight_does_not_require_ssh_fields(self, require_docker, ensure_grpc


class CliBuildTests(unittest.TestCase):
def test_build_does_not_build_global_controller_image(self):
def _run_build(
self, project_dir, agent_yaml_paths, buildx_available, platform="linux/amd64"
):
"""Run cmd_build against project_dir with docker/subprocess calls mocked.

Returns the list of subprocess.run commands that were invoked.
"""
config_path = project_dir / "config" / "global_controller.yaml"
args = SimpleNamespace(config=str(config_path))
docker_calls = []

def fake_run(cmd, check):
docker_calls.append(cmd)
return SimpleNamespace(returncode=0)

with (
patch(
"ventis.cli._get_package_dir",
return_value=str(project_dir / "package"),
),
patch(
"ventis.cli.glob.glob",
side_effect=[agent_yaml_paths, ["proto/a.proto"]],
),
patch("ventis.stub_generator.generate_stub"),
patch("ventis.stub_generator.generate_docker"),
patch("ventis.stub_generator.generate_workflow_docker"),
patch("ventis.cli.subprocess.run", side_effect=fake_run),
patch("ventis.cli._docker_available", return_value=buildx_available),
patch("ventis.cli._docker_platform", return_value=platform),
):
cwd = os.getcwd()
os.chdir(project_dir)
try:
cli.cmd_build(args)
finally:
os.chdir(cwd)

return docker_calls

def _write_agent_and_workflow_config(self, project_dir):
"""Scaffold a project with one agent + one workflow entry; returns the agent YAML path."""
(project_dir / "config").mkdir()
(project_dir / "agents").mkdir()
(project_dir / "workflows").mkdir()
(project_dir / "docker").mkdir()
(project_dir / "docker" / "global-controller.Dockerfile").write_text(
"FROM scratch\n"
)
(project_dir / "agents" / "example_agent.py").write_text("print('ok')\n")
(project_dir / "workflows" / "example_workflow.py").write_text("print('ok')\n")
agent_yaml = project_dir / "agents" / "example_agent.yaml"
agent_yaml.write_text("agent:\n name: ExampleAgent\n")
config_path = project_dir / "config" / "global_controller.yaml"
config_path.write_text(
yaml.safe_dump(
{
"agents": [
{
"name": "ExampleAgent",
"entrypoint": "agents/example_agent.py",
"provider": "local",
},
{
"name": "Workflow",
"type": "workflow",
"workflow_file": "workflows/example_workflow.py",
"provider": "local",
},
]
}
)
)
return agent_yaml

def test_build_falls_back_to_sequential_docker_build_without_buildx(self):
with tempfile.TemporaryDirectory() as tmpdir:
project_dir = Path(tmpdir)
(project_dir / "config").mkdir()
(project_dir / "agents").mkdir()
(project_dir / "workflows").mkdir()
(project_dir / "docker").mkdir()
(project_dir / "docker" / "global-controller.Dockerfile").write_text(
"FROM scratch\n"
agent_yaml = self._write_agent_and_workflow_config(project_dir)

docker_calls = self._run_build(
project_dir, [str(agent_yaml)], buildx_available=False
)
(project_dir / "agents" / "example_agent.py").write_text("print('ok')\n")
(project_dir / "workflows" / "example_workflow.py").write_text(
"print('ok')\n"

flattened = [" ".join(call) for call in docker_calls]
self.assertFalse(
any("global-controller.Dockerfile" in call for call in flattened)
)
self.assertEqual(
sum(call[:2] == ["docker", "build"] for call in docker_calls), 2
)
self.assertFalse(
any(call[:3] == ["docker", "buildx", "bake"] for call in docker_calls)
)

def test_build_uses_buildx_bake_when_available(self):
with tempfile.TemporaryDirectory() as tmpdir:
project_dir = Path(tmpdir)
agent_yaml = self._write_agent_and_workflow_config(project_dir)

docker_calls = self._run_build(
project_dir, [str(agent_yaml)], buildx_available=True
)
agent_yaml = project_dir / "agents" / "example_agent.yaml"
agent_yaml.write_text("agent:\n name: ExampleAgent\n")

bake_file = project_dir / "docker_container" / "docker-bake.json"
self.assertTrue(bake_file.is_file())
with open(bake_file) as f:
bake_config = json.load(f)

self.assertEqual(
sum(call[:2] == ["docker", "build"] for call in docker_calls), 0
)
bake_calls = [
call for call in docker_calls if call[:3] == ["docker", "buildx", "bake"]
]
self.assertEqual(len(bake_calls), 1)
self.assertIn("--file", bake_calls[0])
self.assertEqual(
os.path.realpath(bake_calls[0][bake_calls[0].index("--file") + 1]),
os.path.realpath(bake_file),
)

targets = bake_config["target"]
self.assertEqual(len(targets), 2)
self.assertEqual(
os.path.realpath(targets["exampleagent"]["context"]),
os.path.realpath(project_dir / "docker_container" / "ExampleAgent"),
)
self.assertTrue(os.path.isabs(targets["exampleagent"]["context"]))
self.assertEqual(targets["exampleagent"]["tags"], ["ventis-exampleagent"])
self.assertEqual(targets["exampleagent"]["platforms"], ["linux/amd64"])
self.assertEqual(targets["exampleagent"]["output"], ["type=docker"])
self.assertEqual(
os.path.realpath(targets["workflow"]["context"]),
os.path.realpath(project_dir / "docker_container" / "Workflow"),
)
self.assertEqual(targets["workflow"]["tags"], ["ventis-workflow"])

def test_build_with_no_agents_builds_nothing(self):
with tempfile.TemporaryDirectory() as tmpdir:
project_dir = Path(tmpdir)
(project_dir / "config").mkdir()
(project_dir / "agents").mkdir()
config_path = project_dir / "config" / "global_controller.yaml"
config_path.write_text(yaml.safe_dump({"agents": []}))

docker_calls = self._run_build(project_dir, [], buildx_available=True)

self.assertFalse(any(call[0] == "docker" for call in docker_calls))

def test_build_skips_agent_without_entrypoint(self):
with tempfile.TemporaryDirectory() as tmpdir:
project_dir = Path(tmpdir)
(project_dir / "config").mkdir()
(project_dir / "agents").mkdir()
config_path = project_dir / "config" / "global_controller.yaml"
config_path.write_text(
yaml.safe_dump(
{
"agents": [
{
"name": "ExampleAgent",
"entrypoint": "agents/example_agent.py",
"provider": "local",
},
{
"name": "Workflow",
"type": "workflow",
"workflow_file": "workflows/example_workflow.py",
"provider": "local",
},
{"name": "NoEntrypointAgent", "provider": "local"},
]
}
)
)

args = SimpleNamespace(config=str(config_path))
docker_calls = []

def fake_run(cmd, check):
docker_calls.append(cmd)
return SimpleNamespace(returncode=0)

with (
patch(
"ventis.cli._get_package_dir",
return_value=str(project_dir / "package"),
),
patch(
"ventis.cli.glob.glob",
side_effect=[[str(agent_yaml)], ["proto/a.proto"]],
),
patch("ventis.stub_generator.generate_stub"),
patch("ventis.stub_generator.generate_docker"),
patch("ventis.stub_generator.generate_workflow_docker"),
patch("ventis.cli.subprocess.run", side_effect=fake_run),
patch.dict(os.environ, {}, clear=False),
):
cwd = os.getcwd()
os.chdir(project_dir)
try:
cli.cmd_build(args)
finally:
os.chdir(cwd)
docker_calls = self._run_build(project_dir, [], buildx_available=True)

flattened = [" ".join(call) for call in docker_calls]
self.assertFalse(
any("global-controller.Dockerfile" in call for call in flattened)
)
self.assertEqual(
sum(call[:2] == ["docker", "build"] for call in docker_calls), 2
)
self.assertFalse(any(call[0] == "docker" for call in docker_calls))


if __name__ == "__main__":
Expand Down
79 changes: 67 additions & 12 deletions ventis/cli.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@

import argparse
import glob
import json
import logging
import os
import shutil
Expand Down Expand Up @@ -60,13 +61,13 @@ def _docker_build_cmd(*args):
return ["docker", "build", "--platform", _docker_platform(), *args]


def _docker_available():
def _docker_available(probe_cmd=("docker", "info")):
if not shutil.which("docker"):
return False

try:
result = subprocess.run(
["docker", "info"],
list(probe_cmd),
capture_output=True,
text=True,
check=False,
Expand All @@ -77,6 +78,32 @@ def _docker_available():
return result.returncode == 0


def _write_bake_file(bake_targets, bake_file_path, platform):
"""Write a docker-buildx-bake JSON file describing all build targets.

Context paths are written absolute: `docker buildx bake` resolves relative
`context` values against the invocation cwd (not the bake file's own
directory), so an absolute path sidesteps that ambiguity entirely.
"""
bake_config = {
"target": {
target["name"]: {
"context": os.path.abspath(target["context"]),
"dockerfile": "Dockerfile",
"tags": [target["image_name"]],
"platforms": [platform],
"output": ["type=docker"],
# type=docker could be changed to tarring it up, which would be
# faster but skipped because that change would alter ventis deploy
}
for target in bake_targets
}
}
Comment on lines +88 to +101

@coderabbitai coderabbitai Bot Jul 27, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🌐 Web query:

docker buildx bake target name allowed characters validation

💡 Result:

Docker Buildx Bake targets are strictly validated to ensure compatibility across various systems and naming conventions [1]. The allowed characters for a target name are defined by the regular expression [a-zA-Z0-9_-]+ [2][1]. This means a target name must consist only of: - Uppercase and lowercase letters (a-z, A-Z) - Digits (0-9) - Underscores (_) - Hyphens (-) If a target name includes characters outside of this set—such as dots (.)—docker buildx bake will return an error stating that only those specific characters are allowed [1][3]. In contexts like Docker Compose, where service names might include characters (like dots) that are valid in the Compose specification but invalid for Bake targets, the tooling performs a sanitization step [3][4]. This often involves replacing restricted characters with underscores (e.g., converting service.name to service_name) to ensure they can be processed as Bake targets [2][4]. When working with matrix strategies that dynamically generate names, you can use the sanitize() function within your HCL configuration to ensure generated names comply with these requirements [5].

Citations:

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf 'Files matching cli.py:\n'
fd -a 'cli\.py$' . || true

printf '\nVentis cli.py outline/snippet:\n'
if [ -f ventis/cli.py ]; then
  wc -l ventis/cli.py
  sed -n '300,360p' ventis/cli.py
  sed -n '70,110p' ventis/cli.py
else
  echo "ventis/cli.py not found"
fi

printf '\nSearch agent_name and bake target usages:\n'
rg -n 'agent_name|target\["name"\]|target_names|bake_target|image_name' ventis -S

Repository: CanyonCodeCoreAI/canyoncodecore

Length of output: 12162

Sanitize bake target names and enforce uniqueness.

target["name"] is built directly from agent_name.lower(), while Buildx target names are limited to [A-Za-z0-9_-]; names with spaces or dots will make the generated bake file fail. Case-only differences also become duplicate dict keys, silently dropping one target from the build command path. Normalize to a safe slug and assert that normalized target names are unique before appending targets.

🧰 Tools
🪛 ast-grep (0.44.1)

[warning] 101-101: File path is request-/variable-derived; validate and normalize to prevent path traversal.
Context: open(bake_file_path, "w")
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').

(open-filename-from-request)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@ventis/cli.py` around lines 88 - 101, The bake target construction around
target["name"] must normalize names into Buildx-safe slugs containing only
letters, digits, underscores, or hyphens, and enforce uniqueness after
normalization. Update the target-generation logic that derives names from
agent_name.lower() to sanitize spaces, dots, and other invalid characters, then
validate normalized names are unique before adding targets so collisions fail
explicitly instead of overwriting dictionary entries.

with open(bake_file_path, "w") as f:

@iidsample iidsample Jul 29, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the point of this bake config and what does it do ? How is it configured.

json.dump(bake_config, f, indent=2)
return bake_file_path


def _require_docker_for_ec2(command_name):
if _docker_available():
return
Expand Down Expand Up @@ -218,8 +245,9 @@ def cmd_build(args):
)

# -------------------------------------------------------------- #
# Step 4: Generate Docker contexts and build images #
# Step 4: Generate Docker contexts #
# -------------------------------------------------------------- #
bake_targets = []
for agent_cfg in agents:
agent_name = agent_cfg["name"]
agent_type = agent_cfg.get("type", "agent")
Expand Down Expand Up @@ -248,12 +276,6 @@ def cmd_build(args):
api_port=agent_cfg.get("api_port", 8080),
)

image_name = f"ventis-{agent_name.lower()}"
logger.info("Building Docker image: %s", image_name)
subprocess.run(
_docker_build_cmd("-t", image_name, docker_context), check=True
)

else:
# Agent container
entrypoint = agent_cfg.get("entrypoint")
Expand Down Expand Up @@ -296,10 +318,43 @@ def cmd_build(args):
stub_files=stub_paths,
)

image_name = f"ventis-{agent_name.lower()}"
logger.info("Building Docker image: %s", image_name)
bake_targets.append(
{
"name": agent_name.lower(),
"context": docker_context,
"image_name": f"ventis-{agent_name.lower()}",
}
)

# -------------------------------------------------------------- #
# Step 5: Build all Docker images #
# -------------------------------------------------------------- #
if not bake_targets:
logger.info("No Docker images to build.")
elif _docker_available() and _docker_available(("docker", "buildx", "version")):
docker_container_dir = os.path.join(project_dir, "docker_container")
os.makedirs(docker_container_dir, exist_ok=True)
bake_file_path = os.path.join(docker_container_dir, "docker-bake.json")
_write_bake_file(bake_targets, bake_file_path, _docker_platform())

target_names = [target["name"] for target in bake_targets]
logger.info(
"Building %d Docker image(s) via `docker buildx bake`.",
len(bake_targets),
)
subprocess.run(
["docker", "buildx", "bake", "--file", bake_file_path, *target_names],
check=True,
)
else:
logger.info(
"docker buildx unavailable; falling back to sequential `docker build`."
)
for target in bake_targets:
logger.info("Building Docker image: %s", target["image_name"])
subprocess.run(
_docker_build_cmd("-t", image_name, docker_context), check=True
_docker_build_cmd("-t", target["image_name"], target["context"]),
check=True,
)
Comment on lines +332 to 358

@coderabbitai coderabbitai Bot Jul 27, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

The two _docker_available probes are collapsed into one condition in both the build logic and the test double. cmd_build distinguishes "Docker present" from "buildx present" only inside a single and, so the else branch cannot tell them apart — and the test mock returns one flat value for both probes, so the divergence is never exercised.

  • ventis/cli.py#L332-L358: split the elif into a separate _docker_available() guard that skips builds with an accurate message, and a nested buildx check that chooses bake vs sequential docker build.
  • tests/test_cli.py#L130-L130: replace the flat return_value with a side_effect that inspects probe_cmd, and add a case covering Docker present with buildx missing.
🧰 Tools
🪛 ast-grep (0.44.1)

[error] 354-357: Use of unsanitized data to create processes
Context: subprocess.run(
_docker_build_cmd("-t", target["image_name"], target["context"]),
check=True,
)
Note: [CWE-78] Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection').

(os-system-unsanitized-data)

[error] 344-347: Command coming from incoming request
Context: subprocess.run(
["docker", "buildx", "bake", "--file", bake_file_path, *target_names],
check=True,
)
Note: [CWE-78] Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection').

(subprocess-from-request)

[error] 354-357: Command coming from incoming request
Context: subprocess.run(
_docker_build_cmd("-t", target["image_name"], target["context"]),
check=True,
)
Note: [CWE-78] Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection').

(subprocess-from-request)

📍 Affects 2 files
  • ventis/cli.py#L332-L358 (this comment)
  • tests/test_cli.py#L130-L130
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@ventis/cli.py` around lines 332 - 358, The Docker availability flow in
cmd_build must distinguish Docker absence from buildx absence: update
ventis/cli.py lines 332-358 to guard Docker availability separately with an
accurate no-Docker message, then nest the buildx probe so it selects bake when
available or sequential docker build otherwise. Update the test double at
tests/test_cli.py lines 130-130 to use a side_effect that inspects probe_cmd,
and add coverage for Docker present with buildx missing.


logger.info("Build complete.")
Expand Down
Loading
Loading