DoiT Reports and Insights Pipeline

Small Python pipeline to:

• query recent cloud costs from DoiT Reports API,
• generate optimization recommendations with Ava,
• filter near-duplicates against existing FinOps insights,
• create custom insights via Insights API.

Requirements

• Python 3.10+
• Dependencies from requirements.txt:
  • requests
  • python-dotenv

Install:

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Configuration

Create .env with:

DOIT_API_KEY=...
CUSTOMER_CONTEXT=...                 # read/query context
DOIT_API_KEY_WRITES=...              # write key used for insight creation
CUSTOMER_CONTEXT_WRITES=...          # write context used for insight creation
# Optional:
# MAX_INSIGHTS_WRITE=1

Notes:

• DOIT_API_KEY_WRITES is required for the insight-creation phase.
• CUSTOMER_CONTEXT_WRITES is required for the insight-creation phase.
• Account/identity used for writes needs Insights Manager permission.

Run

Normal run (uses cached files when present):

python3 report.py

Force full refresh:

python3 report.py --force

Limit number of insight writes (default: 1, or MAX_INSIGHTS_WRITE if set):

python3 report.py --max-insights-write 3

Pipeline phases

report.py runs these phases:

1. Query phase: fetches last-month costs from Reports API.
2. Ava phase: generates optimization recommendations from cost data.
3. Insights read phase: fetches existing FinOps insights.
4. Compare/filter phase: removes recommendations similar to existing insights.
5. Insights write phase: creates custom insights from filtered recommendations.

Output files

• cost_report_data.json
• ava_generated_with_significant_value.json
• existing_insights.json
• ava_recommendations_filtered.json
• insights_created_results.json

insights_created_results.json includes:

• created count,
• errors with response details on failures,
• created_insights when at least one insight is successfully created.

APIs used

• Reports query endpoint on https://api.doit.com/analytics/v1/reports/query
• Ava endpoint on https://api.doit.com/ava/v1/askSync
• Insights endpoint on https://api.doit.com/insights/v1/results

Docs:

• DoiT Insights API