🚀 HeroForge.ai Competitor Analysis System

A comprehensive competitor analysis tool built for HeroForge.ai to analyze Aloa.co and other competitors, with advanced MCP integration, context management, and future SEO capabilities.

🎯 Purpose

This system is designed to:

• Scrape competitor websites (starting with Aloa.co) using Playwright MCP
• Analyze messaging, CTAs, value propositions, and positioning
• Compare competitors with HeroForge.ai's positioning
• Generate copy recommendations and strategic insights
• Track SEO elements for future optimization
• Prevent context overflow and hallucination through smart management

🏗️ Architecture

Claude Code Integration

• MCP Servers: Playwright (primary), Sequential Thinking, Task delegation
• Agents: Scraper, Analyzer, Curator, Orchestrator
• Context Management: Automatic checkpointing at 70% usage
• Activity Logging: Complete audit trail in JSONL format

Project Structure

aloa-scraper/
├── .claude/                     # Claude Code configuration
│   ├── CLAUDE.md                # MCP usage rules & context management
│   ├── AGENT_ACTIVITY_LOG.jsonl # Real-time activity tracking
│   └── agents/                  # Agent definitions
├── src/                         # Source code
│   ├── scrapers/                # Playwright MCP integration
│   ├── analyzers/               # Competitive & SEO analysis
│   └── orchestrator/            # Context & state management
├── data/                        # Data storage
│   ├── input/                   # URLs to scrape
│   ├── raw/                     # Raw scraped content
│   ├── processed/               # Processed data
│   └── analysis/                # Analysis results
├── state/                       # State management
│   ├── progress.json            # Scraping progress
│   ├── current_session.json     # Session state
│   └── checkpoints/             # Context checkpoints
└── scripts/                     # Execution scripts

🚀 Quick Start

1. Prerequisites

# Install Python dependencies
pip install pandas openpyxl

# Ensure Claude Code is running with MCP servers enabled

2. Run Scraper

# Check current progress
python scripts/run_scraper.py

# This will show:
# - URLs to scrape
# - Playwright MCP commands
# - Context usage
# - Progress tracking

3. Execute Scraping (via Claude)

Use the generated Playwright MCP commands:

# Navigate to URL
mcp__puppeteer__puppeteer_navigate(url="https://aloa.co/")

# Extract content
mcp__puppeteer__puppeteer_evaluate(script=extraction_script)

# Capture screenshot
mcp__puppeteer__puppeteer_screenshot(name="aloa_home")

4. Run Analysis

# After scraping, run competitive analysis
python scripts/run_scraper.py  # Will auto-detect completion and run analysis

📊 Features

1. Smart Scraping

• Batched Processing: 5 URLs at a time
• Context-Aware: Monitors token usage
• Auto-Checkpointing: Saves state at 70% context
• Error Recovery: Resumes from last checkpoint

2. Competitive Analysis

• Messaging Comparison: Aloa vs HeroForge
• Gap Identification: Missing elements and opportunities
• Copy Adaptation: Transform competitor copy for HeroForge
• Strategic Recommendations: Positioning and differentiation

3. SEO Analysis (Future)

• Keyword Extraction: Primary and secondary keywords
• Content Structure: H1s, meta descriptions, content length
• Technical SEO: Schema, speed, internal linking
• Recommendations: SEO improvements for HeroForge

4. Context Management

• Real-time Monitoring: Track context usage
• Automatic Checkpointing: Save state before overflow
• Memory Optimization: Offload raw data, keep patterns
• Recovery System: Resume from any checkpoint

🤖 Agent System

Scraper Agent

• Navigates to URLs using Playwright MCP
• Extracts content with JavaScript
• Saves raw data immediately
• Updates progress tracking

Analyzer Agent

• Loads data from files (not memory)
• Uses Sequential Thinking MCP for patterns
• Generates competitive insights
• Creates SEO recommendations

Orchestrator Agent

• Monitors all operations
• Manages context window
• Coordinates agent handoffs
• Prevents hallucination

📈 Outputs

1. Scraped Content

{
  "url": "https://aloa.co/",
  "hero": ["Transform Your Business with AI"],
  "ctas": ["Start Free Trial", "Book Demo"],
  "value_props": ["10x faster", "Enterprise-grade"],
  "all_text": "..."
}

2. Competitive Analysis

{
  "gaps_opportunities": {
    "differentiation_opportunities": [
      "Word-of-mouth growth emphasis",
      "Rapid generation vs custom development"
    ]
  },
  "copy_adaptations": {
    "hero_headlines": ["Generate AI Apps in Days"],
    "ctas": ["Start Generating Now"]
  }
}

3. SEO Audit

{
  "keyword_analysis": {
    "primary_keywords": ["ai", "development"],
    "keyword_gaps": ["rapid generation", "word-of-mouth"]
  },
  "recommendations": {
    "title_tags": ["HeroForge - AI App Generator"],
    "meta_descriptions": ["Generate AI apps in days..."]
  }
}

🛡️ Anti-Hallucination Measures

1. Verification Required: All claims must reference files
2. Activity Logging: Every action logged with timestamp
3. State Checking: Verify progress before claiming completion
4. File Validation: Check output files exist before proceeding

📝 Activity Tracking

Monitor in real-time:

# Watch activity log
tail -f .claude/AGENT_ACTIVITY_LOG.jsonl | jq '.'

# Check progress
cat state/progress.json | jq '.'

# Monitor context usage
cat state/current_session.json | jq '.context_usage'

🔧 Configuration

URLs Configuration

Edit data/input/aloa_urls.json:

{
  "urls": [
    {
      "url": "https://aloa.co/",
      "category": "landing",
      "status": "pending"
    }
  ]
}

MCP Optimization

See .claude/CLAUDE.md for:

• MCP server priorities
• Batching strategies
• Context thresholds
• Recovery procedures

🚦 Monitoring

Real-time Dashboard

# Start monitoring
watch -n 5 'python scripts/monitor.py'

Shows:

• Current agent activity
• Context usage percentage
• URLs completed
• Errors and warnings

🔄 Recovery

From Interruption

# Find latest checkpoint
ls -t state/checkpoints/ | head -1

# Resume from checkpoint
python scripts/run_scraper.py --resume checkpoint_001

📚 Future Enhancements

Phase 1 (Current)

• ✅ Aloa.co scraping
• ✅ Competitive analysis
• ✅ Basic SEO analysis
• ✅ Context management

Phase 2 (Next)

• Multiple competitor support
• Advanced SEO tracking
• Backlink analysis
• Content gap analysis

Phase 3 (Future)

• Frontend dashboard
• API for programmatic access
• Automated monitoring
• AI-powered insights

🤝 For HeroForge.ai

Key Differentiators to Emphasize

1. Speed: "Days not months" positioning
2. Growth: Built-in word-of-mouth features
3. Accessibility: No technical team required
4. Completeness: Full-stack from day one

Recommended Actions

1. Implement top CTAs from analysis
2. Test adapted hero headlines
3. Create comparison pages
4. Build speed-focused case studies

📞 Support

For issues or questions:

1. Check .claude/AGENT_ACTIVITY_LOG.jsonl for errors
2. Review state/progress.json for status
3. Verify checkpoints in state/checkpoints/
4. Run recovery script if needed

---

Built with Claude Code MCP integration for maximum efficiency and reliability 🚀