CS 149 NCU Profiling System

Automated CUDA kernel profiling and LLM-based optimization system for Stanford CS 149.

Overview

This system profiles CUDA kernels using NVIDIA Nsight Compute (NCU), extracts key performance metrics, and structures them for LLM analysis to generate actionable optimization tips.

How It Works

CUDA Kernel (.cu)
       ↓
   [compile.sh]
       ↓
  Executable (saxpy)
       ↓
[run_ncu_complete.sh] → NCU profiles 6 metric sections
       ↓
  6 CSV Files (speed_of_light, memory, compute, occupancy, scheduler, warp)
       ↓
  [parse_ncu.py] → Reads CSVs, identifies bottlenecks, generates recommendations
       ↓
  ├─→ parsed_metrics.json (structured data)
  └─→ llm_prompt.txt (LLM-ready analysis with kernel code)
       ↓
   LLM (GPT-4/Claude) → Optimization suggestions

Components

1. Sample Kernel (saxpy.cu) - Basic SAXPY kernel for testing
2. Compilation (compile.sh) - CUDA compilation script
3. NCU Profiling (run_ncu_complete.sh) - Automated profiling with 6 metric sections
4. Metric Parser (parse_ncu.py) - Extracts and structures NCU CSV output
5. Output Files - JSON metrics + LLM-ready prompt

Quick Start (3 Commands)

1. Compile Your Kernel

./compile.sh

This compiles saxpy.cu → produces saxpy executable.

2. Profile with NCU

./run_ncu_complete.sh

This runs NCU profiling and generates 6 CSV files in ncu_output/:

• speed_of_light.csv - High-level bottleneck identification
• memory_analysis.csv - Memory subsystem metrics
• compute_analysis.csv - SM utilization metrics
• occupancy.csv - Warp occupancy data
• scheduler_stats.csv - Scheduler efficiency stats
• warp_stats.csv - Thread execution efficiency

3. Parse Metrics for LLM

python3 parse_ncu.py ncu_output saxpy.cu

This generates:

• ncu_output/parsed_metrics.json - Structured metrics
• ncu_output/llm_prompt.txt - Ready-to-use LLM prompt

Done! Feed llm_prompt.txt to GPT-4/Claude for optimization suggestions.

Complete Workflow Example

# Step 1: Compile
./compile.sh

# Step 2: Profile (generates 6 CSVs)
./run_ncu_complete.sh

# Step 3: Parse
python3 parse_ncu.py ncu_output saxpy.cu

# Step 4: View results
cat ncu_output/parsed_metrics.json
cat ncu_output/llm_prompt.txt

Sample Output (SAXPY Kernel)

The system successfully identified:

• Primary Bottleneck: Memory-bound (86.3% DRAM throughput)
• Secondary Issue: Scheduler stalls (91.6% cycles with no eligible warps)
• Compute Utilization: Only 16.4% (starved by memory)
• Cache Performance: L1 hit rate 32.9%, L2 hit rate 33.6%
• Occupancy: Healthy at 85.4%

Recommendation: Focus on memory optimizations (coalescing, shared memory, vectorization)

Key Metrics Captured

Speed of Light (Bottleneck Identification)

• Memory Throughput % (DRAM-bound indicator)
• Compute (SM) Throughput % (compute-bound indicator)
• DRAM Throughput %
• L1/TEX Cache Throughput %
• L2 Cache Throughput %

Memory Analysis

• Memory Busy % (memory unit utilization)
• Max Bandwidth % (peak bandwidth usage)
• L1/TEX Hit Rate %
• L2 Hit Rate %
• Mem Pipes Busy %

Compute Analysis

• SM Busy % (streaming multiprocessor utilization)
• Issue Slots Busy %
• Executed IPC Active (instructions per cycle)
• Issued IPC Active

Occupancy

• Achieved Occupancy %
• Theoretical Occupancy %
• Achieved Active Warps Per SM
• Theoretical Active Warps per SM

Scheduler Statistics

• One or More Eligible % (warps ready to execute)
• No Eligible % (stall indicator)
• Active Warps Per Scheduler
• Eligible Warps Per Scheduler

Warp Statistics

• Avg. Active Threads Per Warp (divergence indicator)
• Avg. Not Predicated Off Threads Per Warp
• Warp Cycles Per Issued Instruction (latency metric)
• Warp Cycles Per Executed Instruction

Interpreting Results

Primary Bottleneck Identification

The parser automatically identifies the primary bottleneck:

• Memory-bound (DRAM >70%)

  • Focus: Memory coalescing, shared memory, texture memory
  • Reduce global memory accesses

• Compute-bound (SM >70%)

  • Focus: Algorithmic improvements, faster math operations
  • Reduce arithmetic intensity

• Occupancy-limited (Achieved < Theoretical * 0.8)

  • Focus: Reduce register usage, adjust block size
  • Optimize resource utilization

• Divergent (Instructions/warp < 28)

  • Focus: Minimize branching, reorganize data
  • Ensure uniform control flow

File Structure

cs149-ncu/
├── README.md                      # This file
├── saxpy.cu                       # Sample CUDA kernel (SAXPY)
├── saxpy                          # Compiled executable
├── compile.sh                     # Compilation script
├── run_ncu_complete.sh            # NCU profiling script (working version)
├── parse_ncu.py                   # Metric parser (extracts from CSVs)
└── ncu_output/                    # Generated profiling data
    ├── speed_of_light.csv         # High-level bottleneck metrics
    ├── memory_analysis.csv        # Memory subsystem metrics
    ├── compute_analysis.csv       # Compute utilization metrics
    ├── occupancy.csv              # Occupancy statistics
    ├── scheduler_stats.csv        # Scheduler efficiency
    ├── warp_stats.csv             # Warp execution stats
    ├── full_report.ncu-rep        # Full binary report
    ├── parsed_metrics.json        # Structured JSON output ✨
    └── llm_prompt.txt             # LLM-ready prompt ✨

Requirements

AWS Instance (Current Setup)

• Instance Type: g6.xlarge
• GPU: NVIDIA L4 (23GB VRAM)
• CUDA Toolkit: 12.8
• NCU Version: 2025.1.1.0
• OS: Ubuntu 22.04 (Deep Learning Base AMI)
• GPU Profiling: Enabled (RmProfilingAdminOnly=0)

Python

• Python 3.8+ (included in AMI)
• No external dependencies required
• Uses only standard library: csv, json, pathlib

Advanced Usage

Profile Your Own Kernel

1. Replace or modify saxpy.cu with your kernel code
2. Update compile.sh if needed (change input file, add flags)
3. Run the 3-step workflow

# Edit your kernel
nano my_kernel.cu

# Compile (edit compile.sh to use your file)
./compile.sh

# Profile
./run_ncu_complete.sh

# Parse
python3 parse_ncu.py ncu_output my_kernel.cu

Profile Multiple Kernels in One Binary

If your program launches multiple kernels, NCU will profile all of them by default. To profile a specific kernel:

ncu --kernel-name "my_kernel_name" --set full ./my_program

Add Custom Sections

The run_ncu_complete.sh currently runs 6 sections. To add more:

ncu --set full --section YourSection --csv \
    --log-file ncu_output/your_section.csv ./saxpy

Available sections: ncu --list-sections

Troubleshooting

Parser Returns Empty Metrics

Issue: parsed_metrics.json shows empty {} for all sections

Solution: The parser automatically skips NCU header lines (like ==PROF==, Max error:). If still empty:

• Verify CSV files exist: ls -lh ncu_output/*.csv
• Check CSV format: head -20 ncu_output/speed_of_light.csv
• Ensure CSV header starts with "ID","Process ID",...

NCU Permission Denied

Issue: ERR_NVGPUCTRPERM - Permission denied

Solution: Enable GPU profiling (already done on this instance):

sudo sh -c 'echo "options nvidia NVreg_RestrictProfilingToAdminUsers=0" > /etc/modprobe.d/nvidia-profiling.conf'
sudo update-initramfs -u
sudo reboot

NCU Runs But No CSVs Generated

Issue: Script completes but ncu_output/ is empty

Solution:

• Check if executable runs: ./saxpy (should print "Max error: 0.000000")
• Verify NCU works: ncu --version
• Try manual run: ncu --set full --section SpeedOfLight --csv ./saxpy

Metric Names Don't Match

Issue: Parser looks for wrong metric names

Solution: NCU uses friendly names like "Memory Throughput" not technical names like dram__throughput.avg.pct_of_peak_sustained_elapsed. The parser has been updated to use friendly names.

Integration with LLM

The system generates llm_prompt.txt which is ready to feed to any LLM:

OpenAI API Example

import openai
with open('ncu_output/llm_prompt.txt', 'r') as f:
    prompt = f.read()
    
response = openai.ChatCompletion.create(
    model="gpt-4",
    messages=[{"role": "user", "content": prompt}]
)
print(response.choices[0].message.content)

Claude API Example

import anthropic
with open('ncu_output/llm_prompt.txt', 'r') as f:
    prompt = f.read()
    
client = anthropic.Anthropic(api_key="your_key")
message = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=2048,
    messages=[{"role": "user", "content": prompt}]
)
print(message.content[0].text)

Cost Estimates (AWS)

• g6.xlarge (NVIDIA L4): ~$0.84/hour on-demand
• g6.2xlarge (NVIDIA L4): ~$1.20/hour on-demand
• Spot instances: 60-70% cheaper (~$0.30/hour)
• Storage (EBS): ~$0.10/GB-month

Estimated cost per student submission: < $0.01 (profiling + parsing takes ~10-30 seconds)

References

• NVIDIA Nsight Compute Documentation
• Nsight Compute Metrics Guide
• CUDA C Programming Guide
• AWS EC2 G6 Instances

System Status

✅ FULLY OPERATIONAL

• Compilation: Working
• NCU Profiling: Working (all 6 sections)
• Parser: Working (extracts all metrics)
• Output Generation: Working (JSON + LLM prompt)
• Tested on: SAXPY kernel (memory-bound, 86.3% DRAM utilization)

Last Updated: Profiled on AWS g6.xlarge with NVIDIA L4 GPU

---

For questions about this system, contact the CS 149 teaching team.