cintel-04-rolling-monitoring

Professional Python project for continuous intelligence.

Continuous intelligence systems monitor data streams, detect change, and respond in real time. This course builds those capabilities through working projects.

In the age of generative AI, durable skills are grounded in real work: setting up a professional environment, reading and running code, understanding the logic, and pushing work to a shared repository. Each project follows the structure of professional Python projects. We learn by doing.

This Project

This project introduces rolling monitoring.

The goal is to copy this repository, set up your environment, run the example analysis, and explore how system metrics change over time.

You will run the example pipeline, read the code, and make small modifications to understand how rolling windows help smooth short-term variation and reveal trends in time-series data.

Data

The example pipeline reads time-series system metrics from:

data/system_metrics_timeseries_case.csv

Each row represents one observation at a specific timestamp. The pipeline computes rolling averages for requests, errors, and latency, then saves the monitoring results as an artifact.

Working Files

You'll work with just these areas:

• data/ - it starts with the data
• docs/ - tell the story
• src/cintel/ - where the magic happens
• pyproject.toml - update authorship & links
• zensical.toml - update authorship & links

Instructions

Follow the step-by-step workflow guide to complete:

1. Phase 1. Start & Run
2. Phase 2. Change Authorship
3. Phase 3. Read & Understand
4. Phase 4. Modify
5. Phase 5. Apply

Challenges

Challenges are expected. Sometimes instructions may not quite match your operating system. When issues occur, share screenshots, error messages, and details about what you tried. Working through issues is part of implementing professional projects.

Success

After completing Phase 1. Start & Run, you'll have your own GitHub project, running on your machine, and running the example will print out:

========================
Pipeline executed successfully!
========================

And a new file named project.log will appear in the project folder.

Command Reference

The commands below are used in the workflow guide above. They are provided here for convenience.

Follow the guide for the full instructions.

Show command reference

In a machine terminal (open in your Repos folder)

After you get a copy of this repo in your own GitHub account, open a machine terminal in your Repos folder:

git clone https://github.com/KHenn22/cintel-04-rolling-monitoring

cd cintel-04-rolling-monitoring
code .

In a VS Code terminal

uv self update
uv python pin 3.14
uv sync --extra dev --extra docs --upgrade

uvx pre-commit install
git add -A
uvx pre-commit run --all-files

uv run python -m cintel.rolling_monitor_case

uv run ruff format .
uv run ruff check . --fix
uv run zensical build

git add -A
git commit -m "update"
git push -u origin main

Notes

• Use the UP ARROW and DOWN ARROW in the terminal to scroll through past commands.
• Use CTRL+f to find (and replace) text within a file.

Technical Modification

Rolling Standard Deviation and Anomaly Spike Detection

I Added rolling standard deviation calculations and anomaly spike flags to the monitoring pipeline.

What was added: ∙ Rolling standard deviation for requests, errors, and latency (same window size [3] as the rolling mean) ∙ Boolean spike flags for errors and latency — a spike is flagged when a value exceeds the rolling mean +2 standard deviations ∙ Log output reporting the count of detected spikes

Reason: While rolling means alone show trends, they cannot detect outliers. Adding standard deviation creates a dynamic threshold that adapts to the data, making it possible to automatically flag unusual system behavior without the need to hardcode a fixed limit.

Output columns added: ∙ requests_rolling_std ∙ errors_rolling_std ∙ latency_rolling_std ∙ error_spike_flag ∙ latency_spike_flag

Apply Skills to Complete the Project

Airline Delay Rolling Monitor

A continuous intelligence pipeline that monitors U.S. airline departure delay and cancellation anomalies using rolling statistics. Data Source: Bureau of Transportation Statistics October 2025 Airline Data

What It Does

Reads 605,844 flight records from the BTS October 2025 on-time reporting dataset, aggregates them by airline and day, computes 5-day rolling means and standard deviations, and flags anomalous days where delays or cancellations exceed the rolling average by 1 standard deviation.

Results and Interpretation

87 total delay spikes and 77 cancellation spikes flagged across 14 airlines in October 2025.

Airlines exceeding their own rolling average by 1 standard deviation are flagged as a spike. United, Republic, Delta and American showing 8 spike days out of 31 suggests roughly 1 in 4 days had unusually high delays. This is useful for identifying carriers with inconsistent on-time performance rather than just high average delays.​​​​​​​​​​​​​​​​ However, it should be mentioned that a monthly data set is far too small to determine if an airline has an overall issue with operational efficiency.

Setup

git clone <https://github.com/KHenn22/cintel-04-rolling-monitoring>
cd cintel-04-rolling-monitoring
uv venv .venv
uv pip install -r requirements.txt

Run Script

uv run -m src.cintel.airline_delay_rolling_monitor_hennelly

Output and Documentation

Results are saved to artifacts/airline_delay_rolling_metrics_hennelly.csv See docs/index.md for project documentation