Electricity Theft Detection System

A comprehensive Python-based machine learning system for detecting electricity theft using industrial consumption data.

🎯 Features

• Data Loading: Load CSV datasets with electricity consumption data
• Data Cleaning: Handle missing values and outliers automatically
• Feature Engineering: Calculate load_factor, power_factor, and other derived features
• Categorical Encoding: Automatically encode categorical variables
• Train/Test Split: 80/20 split with stratification for balanced datasets
• Data Visualization: Comprehensive plots for data analysis
• Machine Learning Models: Train and compare multiple ML algorithms
• Model Evaluation: Comprehensive metrics (accuracy, precision, recall, F1, AUC)
• Model Comparison: Automatic ranking and selection of best performing model
• Hyperparameter Optimization: GridSearchCV and RandomizedSearchCV for optimal performance
• Feature Importance Analysis: SHAP values for global model interpretability
• Local Interpretability: LIME for individual prediction explanations
• Advanced Visualizations: SHAP plots, feature importance charts, and optimization results
• REST API: FastAPI-based production-ready API
• Model Deployment: Automatic model saving and loading
• Input Validation: Comprehensive data validation with Pydantic
• Batch Processing: Support for multiple predictions
• Interactive Documentation: Swagger UI and ReDoc
• Modular Design: Easy to extend and customize

📊 Expected Data Format

Your CSV file should contain the following columns:

Column	Description	Type
consumer_id	Unique identifier for each consumer	String/Integer
kWh	Active energy consumption	Numeric
kVARh	Reactive energy consumption	Numeric
billed_amount	Amount billed to consumer	Numeric
paid_amount	Amount paid by consumer	Numeric
power_factor	Power factor (will be calculated if missing)	Numeric
load_factor	Load factor (will be calculated if missing)	Numeric
fraudulent	Target variable (0=normal, 1=fraudulent)	Binary

🚀 Quick Start

1. Install Dependencies

pip install -r requirements.txt

2. Prepare Your Data

Option A: Use Sample Data (Recommended for testing)

python example_data_generator.py

Option B: Use Your Own Data Place your electricity consumption CSV file in the project directory.

3. Run the Complete Pipeline

Option A: Complete Pipeline (Data + ML)

python electricity_theft_detection.py

Option B: ML Training Only

python ml_model_training.py

Option C: Advanced Analysis (Hyperparameter Optimization + Interpretability)

python advanced_ml_analysis.py

Option D: API Server (Production Deployment)

python start_api_server.py

📈 Usage Examples

Complete Pipeline (Data + ML)

from electricity_theft_detection import ElectricityTheftDetector, MLModelTrainer

# Step 1: Data preprocessing
detector = ElectricityTheftDetector()
X_train, X_test, y_train, y_test = detector.run_full_pipeline(
    data_path="your_data.csv",
    test_size=0.2,
    random_state=42
)

# Step 2: ML model training and comparison
ml_trainer = MLModelTrainer(X_train, X_test, y_train, y_test)
results = ml_trainer.run_complete_ml_pipeline()

# Get best model
best_name, best_model, best_results = ml_trainer.get_best_model()
print(f"Best model: {best_name} with F1-score: {best_results['f1_score']:.4f}")

Advanced Analysis with Hyperparameter Optimization

# Initialize ML trainer
ml_trainer = MLModelTrainer(X_train, X_test, y_train, y_test)

# Train models
ml_trainer.initialize_models()
ml_trainer.train_models()

# Optimize hyperparameters for best model
opt_results = ml_trainer.optimize_hyperparameters(
    method='random',  # or 'grid'
    n_iter=50,
    cv_folds=5
)

# SHAP feature importance analysis
shap_results = ml_trainer.analyze_feature_importance_shap()

# LIME local interpretability
lime_results = ml_trainer.analyze_feature_importance_lime()

# Run complete advanced analysis
advanced_results = ml_trainer.run_advanced_analysis()

API Usage

import requests

# Single fraud prediction
data = {
    "kWh": 1200.0,
    "kVARh": 200.0,
    "billed_amount": 8000.0,
    "paid_amount": 7800.0,
    "power_factor": 0.92,
    "load_factor": 0.85
}

response = requests.post("http://localhost:8000/predict", json=data)
result = response.json()

print(f"Status: {result['status']}")
print(f"Probability: {result['fraud_probability']}")
print(f"Risk Factors: {result['risk_factors']}")

Individual Model Training

# Initialize ML trainer
ml_trainer = MLModelTrainer(X_train, X_test, y_train, y_test)

# Train specific models
ml_trainer.initialize_models()
ml_trainer.train_models()

# Compare models
comparison_df = ml_trainer.compare_models()

# Generate visualizations
ml_trainer.plot_confusion_matrices()
ml_trainer.plot_roc_curves()

Step-by-Step Data Preprocessing

# Load data
detector.load_data("your_data.csv")

# Clean data
detector.clean_data()

# Engineer features
detector.engineer_features()

# Encode categorical variables
detector.encode_categorical()

# Split data
X_train, X_test, y_train, y_test = detector.split_data(test_size=0.2)

# Visualize data
detector.visualize_data()

🔧 Feature Engineering

The system automatically creates several derived features:

• Load Factor: kWh / (kWh + kVARh)
• Power Factor: cos(arctan(kVARh/kWh))
• Payment Ratio: paid_amount / billed_amount
• Outstanding Amount: billed_amount - paid_amount
• Cost per kWh: billed_amount / kWh

🤖 Machine Learning Models

The system trains and compares multiple ML algorithms:

1. Logistic Regression: Linear model with balanced class weights
2. Random Forest: Ensemble method with optimized parameters
3. XGBoost: Gradient boosting with advanced features
4. Support Vector Machine: RBF kernel with balanced classes

Hyperparameter Optimization

• GridSearchCV: Exhaustive search over parameter grid
• RandomizedSearchCV: Random sampling for faster optimization
• Cross-validation: Robust performance estimation
• Automatic optimization: Best model parameters identified automatically

Model Interpretability

• SHAP Values: Global feature importance and model explanations
• LIME: Local interpretability for individual predictions
• Feature Importance: Tree-based model feature rankings
• Visual Explanations: Interactive plots and charts

Evaluation Metrics

• Accuracy: Overall correctness
• Precision: True positives / (True positives + False positives)
• Recall: True positives / (True positives + False negatives)
• F1-Score: Harmonic mean of precision and recall
• AUC: Area under the ROC curve
• Confusion Matrix: Detailed classification breakdown

📊 Data Visualization

The system generates comprehensive visualizations:

1. Fraud Distribution: Pie chart showing fraud vs non-fraud cases
2. kWh Distribution: Box plot comparing consumption patterns
3. Billed vs Paid: Scatter plot showing payment patterns
4. Correlation Matrix: Heatmap of feature correlations
5. Model Confusion Matrices: Performance comparison across models
6. ROC Curves: Model discrimination ability
7. Feature Importance: Most influential features (for tree-based models)
8. SHAP Summary Plots: Global feature importance and interactions
9. SHAP Bar Plots: Feature importance rankings
10. LIME Explanations: Individual prediction explanations
11. Hyperparameter Optimization Results: Performance improvement charts

🛠️ Customization

Adding Custom Features

# After loading data, add custom features
detector.data['custom_feature'] = detector.data['kWh'] * detector.data['power_factor']

Custom Cleaning Rules

# Override the clean_data method for custom cleaning
def custom_clean_data(self):
    # Your custom cleaning logic here
    pass

📋 Requirements

• Python 3.7+
• pandas >= 1.5.0
• numpy >= 1.21.0
• scikit-learn >= 1.1.0
• matplotlib >= 3.5.0
• seaborn >= 0.11.0
• xgboost >= 1.6.0
• imbalanced-learn >= 0.9.0
• shap >= 0.41.0
• lime >= 0.2.0.1
• optuna >= 3.0.0
• fastapi >= 0.104.0
• uvicorn >= 0.24.0
• pydantic >= 2.0.0
• python-multipart >= 0.0.6

🔍 Troubleshooting

Common Issues

1. File Not Found Error

   • Ensure your CSV file path is correct
   • Check file permissions

2. Missing Columns Error

   • Verify your CSV has the required columns
   • Check column names for typos

3. Memory Issues

   • For large datasets, consider sampling
   • Use chunking for very large files

Data Quality Tips

• Ensure numeric columns don't contain text
• Check for consistent date formats
• Verify the 'fraudulent' column contains only 0s and 1s

🎯 Next Steps

After running the complete pipeline:

1. Model Selection: The system automatically identifies the best performing model
2. Feature Analysis: Review feature importance plots to understand key indicators
3. Model Validation: Use cross-validation for robust performance estimation
4. Hyperparameter Tuning: Fine-tune the best model for optimal performance
5. Production Deployment: Create a real-time fraud detection system
6. Monitoring: Set up model performance monitoring and retraining pipelines

📞 Support

For issues and questions:

• Check the troubleshooting section above
• Review your data format and quality
• Ensure all dependencies are installed correctly

📄 License

This project is open source and available under the MIT License.