refactor: Align database transactions with Frappe standards to improve performance

[Sanjusha-tridz]

Description

This PR addresses recent audit feedback regarding excessive and redundant frappe.db.commit() calls throughout the HUF codebase.

Previously, explicit commits were used inside @frappe.whitelist() API routes and tool handlers. Because the Frappe framework automatically manages transaction boundaries and commits at the end of every HTTP request, these explicit calls resulted in "two commits for a single transaction." As the platform scales, this double-writing creates significant lock contention, unnecessary database I/O, and degrades API latency.

This PR strictly aligns the codebase with Frappe transaction standards while preserving the intermediate persistence required for real-time UI polling in our Agent streaming and Flow Engine loops.

Key Changes

1. Removed Redundant API Commits (~35 calls)
   • Stripped all frappe.db.commit() calls from @frappe.whitelist() endpoints (e.g., permissions_api, prompt_api, flow_api, mcp_oauth).
   • Transaction management for these routes is now entirely delegated to Frappe's native request lifecycle, guaranteeing true atomicity.
2. Introduced Context-Aware Transaction Helpers
   • Created huf/ai/transaction.py containing commit_if_background() and transaction_checkpoint().
   • Flow Engine & Tools: Migrated explicit commits to commit_if_background(). This ensures that when tools run synchronously via API, they do not break transaction boundaries, but when enqueued in background workers, they safely persist state.
   • Agent Streaming: Migrated streaming persistence to transaction_checkpoint(reason="agent_streaming_progress") to explicitly document intentional partial writes for UI polling.
   • Bug Fix: Initialized frappe.local._realtime_log in the new safe_commit() helper to prevent a known Frappe background-worker AttributeError.
3. Dead Code Removal
   • Deleted unreachable commit logic in agent_integration.py.
4. Automated CI Guardrails
   • Added a check_explicit_commits.py script and wired it to .pre-commit-config.yaml to block developers from adding new raw commits locally.
   • Added .semgrep.yml rules to enforce transaction standards in the CI pipeline.

Impact & Performance

• Reduced Database I/O: API endpoints will execute significantly faster as they no longer pause mid-request to perform redundant disk flushes.
• Improved Concurrency: Reduced row-lock times allow the MariaDB connection pool to handle more concurrent agent runs without hanging.
• True Atomicity: If an API request crashes halfway through, Frappe will now successfully roll back all partial state.

Testing Performed

• Verified Agent Chat streaming UI still correctly updates tool-call cards (Checkpoints working).
• Verified CRUD operations in Permissions and Prompt APIs successfully save without explicit commits.
• Verified Flow Engine graph execution correctly persists node states.
• Verified the pre-commit hook correctly blocks new frappe.db.commit() additions.

[esafwan]

Intake review complete (dossier on file) — disposition: technically sound, mergeable after review of risks R1–R5. The two one-line follow-ups the dossier flagged are implemented as stacked draft #410 (webhook POST-only per R1; consistent permission enforcement per R2). Recommend landing #404 before the overlapping flow/agent PRs (#397/#362/#403) to minimize rebase churn.