GITHUB_API_USAGE.md

GitHub API Usage Documentation

Overview

This document provides comprehensive details about how the GitHub Package service utilizes various GitHub APIs to discover and process developer profiles for the Brainy knowledge graph.

API Discovery Methods

1. Bootstrap Discovery (Guaranteed to Work)

Purpose: Provides a reliable starting point for data collection using authenticated user's network.

APIs Used:

• GET /user - Fetch authenticated user details
• GET /users/{username}/followers - Get user's followers
• GET /user/starred - Get repositories starred by user
• GET /users/{username} - Get full user details for discovered profiles

Benefits:

• 100% Reliability: Always starts with authenticated user (guaranteed to exist)
• Network Effect: Leverages social graph for organic discovery
• Quality Signal: Followers and starred repos indicate engaged developers
• Rate Limit Efficient: Uses core API endpoints (5000 req/hour)

Implementation:

private async bootstrapDiscovery(): Promise<void> {
  // 1. Process authenticated user
  // 2. Process user's followers (limit 10)
  // 3. Process owners of starred repositories (limit 5)
}

2. Active Contributors Discovery

Purpose: Find developers actively contributing to open source projects.

APIs Used:

• GET /events - Public events stream (real-time activity)
• GET /users/{username} - Get full details for active users

Benefits:

• Real-Time Data: Captures currently active developers
• Quality Filter: Focus on meaningful contributions (Push, PR, Create, Release events)
• Fresh Profiles: Discovers new and emerging contributors
• High Engagement: Active users more likely to have rich profiles

Event Types Monitored:

• PushEvent - Code commits
• PullRequestEvent - Code reviews and contributions
• CreateEvent - New repositories/branches
• ReleaseEvent - Project releases

3. GraphQL Search Discovery

Purpose: Find high-quality developers using advanced search criteria.

APIs Used:

• GraphQL endpoint with search queries
• Combined filters for repositories, followers, and activity

Methods:

# Find Productive Developers
search(query: "type:user repos:>10 followers:>50", type: USER)

# Find Versatile Developers  
search(query: "type:user repos:>2 followers:>5", type: USER)

# Find Influential Developers
search(query: "type:user followers:>200", type: USER)

Benefits:

• Bulk Discovery: Returns multiple users in single request
• Quality Filters: Pre-filters for active, influential developers
• Efficient: Single request for multiple profiles
• Flexible: Can adjust criteria based on needs

Rate Limits and Optimization

Rate Limit Categories

API Type	Limit	Reset	Our Usage
Core/REST	5,000/hour	Hourly	Bootstrap, User details
GraphQL	5,000/hour	Hourly	Search queries
Search	30/minute	Per minute	GraphQL search

Optimization Strategies

1. Intelligent Caching

   • Cache user data to avoid repeated fetches
   • Store rate limit info to optimize request timing

2. Batch Processing

   • Use GraphQL for bulk user discovery
   • Process users in controlled batches

3. Prioritization

   • Focus on high-quality profiles first
   • Skip inactive or low-value accounts

4. Error Handling

   • Graceful degradation on rate limit
   • Automatic retry with exponential backoff
   • Continue other discovery methods if one fails

Data Processing Pipeline

Step 1: Discovery

Bootstrap → Active Contributors → GraphQL Search

Step 2: User Processing

processUser(user) {
  1. Fetch complete user profile
  2. Fetch user's repositories
  3. Analyze skills and technologies
  4. Detect job-seeking signals
  5. Calculate quality scores
}

Step 3: Repository Analysis

processRepository(repo) {
  1. Extract languages and technologies
  2. Analyze README for skills
  3. Check activity levels
  4. Identify key contributions
}

Step 4: Storage in Brainy

storeInBrainy(data) {
  1. Transform to standardized schema
  2. Create nouns (entities)
  3. Create verbs (relationships)
  4. Handle placeholders for missing data
}

Benefits Summary

For Data Quality

• Multi-source validation: Cross-reference data from multiple APIs
• Fresh data: Real-time event stream ensures current information
• Rich profiles: Multiple data points per developer

For Coverage

• Guaranteed starting point: Bootstrap with authenticated user
• Network expansion: Organic growth through social connections
• Active discovery: Find new contributors as they emerge

For Efficiency

• Rate limit aware: Intelligent request management
• Batch operations: Minimize API calls
• Graceful degradation: Continue operating even with limits

Configuration

Environment Variables

# GitHub Authentication
GITHUB_TOKEN=your_token                    # Personal access token
# OR GitHub App (higher rate limits)
GITHUB_APP_ID=your_app_id
GITHUB_PRIVATE_KEY=your_private_key
GITHUB_INSTALLATION_ID=your_installation_id

# Discovery Configuration
AUTO_START_PROCESSING=true                 # Auto-start discovery
PROCESSING_MODE=continuous                 # continuous or single
RATE_LIMIT_THRESHOLD=50                   # Stop when X requests remain

Discovery Tuning

// In githubService.ts
const DISCOVERY_LIMITS = {
  bootstrapFollowers: 10,      // Followers to process
  bootstrapStarred: 5,         // Starred repo owners
  activeContributors: 15,      // Active users from events
  graphqlBatchSize: 100,       // Users per GraphQL query
}

Monitoring and Debugging

Key Metrics

• Users processed per hour
• API rate limit consumption
• Discovery method success rates
• Error rates by API endpoint

API Endpoints for Monitoring

• GET /health - Service health and initialization status
• GET /api/data-status - Processing statistics
• GET /rate-limits - Current GitHub API limits
• GET /stats - Detailed processing metrics

Error Handling

Common Issues and Solutions

1. Rate Limit Exceeded

   • Automatic pause until reset
   • Switch to different API endpoint
   • Continue with cached data

2. Authentication Failure

   • Verify token/app credentials
   • Check token permissions
   • Fallback to public API endpoints

3. Network Timeouts

   • Retry with exponential backoff
   • Continue with other discovery methods
   • Log for later reprocessing

4. Storage Throttling

   • Batch writes to reduce frequency
   • Implement queue with rate limiting
   • Use write-only mode for efficiency

Future Enhancements

Planned Improvements

1. Organization Discovery - Process entire organizations
2. Topic-Based Search - Find developers by technology topics
3. Contribution Analysis - Deep dive into commit history
4. Team Discovery - Find collaborative developer groups
5. Trending Developer - Identify rising stars

API Endpoints to Explore

• /users/{username}/events - User's public activity
• /repos/{owner}/{repo}/contributors - Repository contributors
• /search/users - Advanced user search
• /users/{username}/following - Who user follows
• Webhooks API - Real-time event processing

Conclusion

The GitHub Package service uses a multi-layered approach to discover and process developer profiles:

1. Reliability First: Bootstrap discovery ensures we always have data to process
2. Quality Over Quantity: Focus on active, engaged developers
3. Efficient API Usage: Maximize value from rate limits
4. Graceful Degradation: Continue operating despite failures
5. Rich Data Collection: Multiple data points per developer

This approach ensures consistent data flow into the Brainy knowledge graph while respecting GitHub's API limits and maintaining high data quality.