Merge branch 'main' of github.com:isamplesorg/isamplesorg.github.io

Author: jkunze

.github/workflows/quarto-pages.yml

@@ -1,6 +1,9 @@
 name: Render using Quarto and push to GH-pages
 
-on: workflow_dispatch
+on:
+  push:
+    branches: [main]
+  workflow_dispatch:
 
 jobs:
   build-deploy:

.gitignore

@@ -3,6 +3,24 @@ uv.lock
 plantuml-images
 docs
 
+# Generated vocabulary documentation (but keep readme.md files)
+models/generated/vocabularies/*.qmd
+models/generated/vocabularies/*.md
+!models/generated/vocabularies/readme.md
+models/generated/extensions/*.md
+!models/generated/extensions/readme.md
+
+# Large data files
+*.parquet
+
+# Node / Playwright
+node_modules/
+package-lock.json
+tests/playwright-report/
+test-results/
+playwright-report/
+playwright/.cache/
+
 .$*
 
 # Byte-compiled / optimized / DLL files

LAZY_LOADING_IMPLEMENTATION.md

@@ -0,0 +1,298 @@
+# Lazy Loading Implementation
+
+**Date**: 2025-10-31
+**Purpose**: Improve perceived performance of Cesium tutorial page
+
+---
+
+## 🎯 What Was Implemented
+
+### 1. **Chunked Rendering for Geocode Points** (lines 194-229)
+
+**Problem**: Rendering thousands of geocode points in one blocking operation made the page unresponsive.
+
+**Solution**: Render points in batches of 500 with yields to browser event loop.
+
+```javascript
+const CHUNK_SIZE = 500;
+for (let i = 0; i < data.length; i += CHUNK_SIZE) {
+    const chunk = data.slice(i, i + CHUNK_SIZE);
+    // ... add points for chunk
+
+    // Yield to browser between chunks
+    if (i + CHUNK_SIZE < data.length) {
+        await new Promise(resolve => setTimeout(resolve, 0));
+    }
+}
+```
+
+**Benefits**:
+- Page remains interactive during rendering
+- User sees progress (not just a frozen browser)
+- Can cancel/navigate away during load if needed
+
+---
+
+### 2. **Dynamic Progress Indicator** (lines 203-207)
+
+**Problem**: User had no feedback during slow initial load.
+
+**Solution**: Update loading div with real-time progress.
+
+```javascript
+if (loadingDiv) {
+    const pct = Math.round((endIdx / data.length) * 100);
+    loadingDiv.innerHTML = `Rendering geocodes... ${endIdx.toLocaleString()}/${data.length.toLocaleString()} (${pct}%)`;
+}
+```
+
+**User Experience**:
+- "Querying geocodes from parquet..." (during SQL query)
+- "Rendering geocodes... 500/1,234 (41%)" (during rendering)
+- Progress hidden when complete
+
+---
+
+### 3. **Performance Telemetry** (lines 132-244)
+
+**Problem**: No visibility into where time is spent.
+
+**Solution**: Use Performance API to measure each phase.
+
+**Measurements Added**:
+1. **locations-query**: Time to execute SQL query (lines 168-173)
+2. **locations-render**: Time to render all points (lines 230-232)
+3. **locations-total**: Total time from start to finish (lines 239-241)
+
+**Console Output**:
+```javascript
+Query executed in 2847ms - retrieved 1234 locations
+Rendering completed in 412ms
+Total time (query + render): 3259ms
+```
+
+---
+
+### 4. **Query Telemetry for Click Events** (lines 400-406, 462-468, 524-530)
+
+**Problem**: No visibility into per-query performance when user clicks geocode.
+
+**Solution**: Added timing to all three query functions.
+
+**Added to**:
+- `get_samples_1()` - Path 1 (direct event location)
+- `get_samples_2()` - Path 2 (via site location)
+- `get_samples_at_geo_cord_location_via_sample_event()` - Eric's query
+
+**Console Output**:
+```javascript
+Path 1 query executed in 1523ms - retrieved 5 samples
+Path 2 query executed in 892ms - retrieved 0 samples
+Eric's query executed in 1401ms - retrieved 5 samples
+```
+
+---
+
+## 📊 Expected Performance Improvements
+
+### Before Lazy Loading:
+- **Initial load**: Page frozen for 5-10 seconds (no feedback)
+- **User perception**: "Is this working? Did it crash?"
+- **Browser**: Unresponsive during point rendering
+
+### After Lazy Loading:
+- **Query phase**: 2-8 seconds (depends on parquet download)
+  - User sees: "Querying geocodes from parquet..."
+- **Rendering phase**: 400-800ms (chunked, with progress)
+  - User sees: "Rendering geocodes... 500/1,234 (41%)"
+- **Total perceived wait**: Same absolute time, but **feels 3-5x faster** due to feedback
+- **Browser**: Remains responsive (can scroll, type, navigate)
+
+### Click Performance (no change):
+- Path 1, Path 2, Eric's queries: Still 1-2 seconds each (structural limitation)
+- Now visible via console telemetry for optimization planning
+
+---
+
+## 🧪 Testing Instructions
+
+### 1. Open Browser Developer Console
+
+**Chrome/Edge**: F12 or Cmd+Option+I (Mac)
+**Firefox**: F12 or Cmd+Option+K (Mac)
+
+### 2. Load the Page
+
+Navigate to: `http://localhost:5860/tutorials/parquet_cesium.html`
+
+### 3. Observe Initial Load
+
+**Watch for**:
+- Loading indicator updates: "Querying geocodes..." → "Rendering... X/Y (Z%)"
+- Console logs with timing measurements
+- Page remains responsive (try scrolling, clicking buttons)
+
+**Expected Console Output**:
+```
+Query executed in 2847ms - retrieved 1234 locations
+Rendering completed in 412ms
+Total time (query + render): 3259ms
+```
+
+### 4. Test Click Queries
+
+**Steps**:
+1. Click any geocode point on globe
+2. Observe three query results tables render
+3. Check console for query timings
+
+**Expected Console Output**:
+```
+Path 1 query executed in 1523ms - retrieved 5 samples
+Path 2 query executed in 892ms - retrieved 0 samples
+Eric's query executed in 1401ms - retrieved 5 samples
+```
+
+### 5. Test with Known Geocode
+
+**Use search box**:
+- Enter: `geoloc_04d6e816218b1a8798fa90b3d1d43bf4c043a57f` (PKAP with samples)
+- Click search
+- Verify camera flies to location
+- Verify all three tables render
+- Check console timings
+
+---
+
+## 📈 Performance Baseline Data
+
+Once you test locally, we can establish baseline metrics:
+
+**Initial Load Metrics**:
+- Query time: _____ ms
+- Render time: _____ ms
+- Total time: _____ ms
+- Number of geocodes: _____
+
+**Click Query Metrics**:
+- Path 1: _____ ms (_____ samples)
+- Path 2: _____ ms (_____ samples)
+- Eric's query: _____ ms (_____ samples)
+
+These baselines will help evaluate whether Phase 2 optimizations (pre-aggregated parquet) are worth pursuing.
+
+---
+
+## 🔄 Next Steps (from PERFORMANCE_OPTIMIZATION_PLAN.md)
+
+### Phase 1 Complete ✅
+- [x] Chunked rendering with progress
+- [x] Performance telemetry
+- [x] Dynamic loading indicators
+
+### Phase 2 (If Needed) - Structural Optimization
+**Goal**: Reduce initial load from 3-8 seconds → <1 second
+
+**Approach**: Pre-aggregate geocode classification query
+1. Create `oc_geocodes_classified.parquet` (~50KB) via server-side script
+2. Replace expensive CTE query with simple `SELECT * FROM read_parquet(...)`
+3. Automate regeneration in GitHub Actions workflow
+
+**When to pursue**:
+- If query time consistently >5 seconds
+- If users complain about initial load
+- If baseline data shows query is primary bottleneck
+
+### Phase 3 (Only if Desperate) - Deep Optimization
+**Goal**: Reduce click queries from 1-2 seconds → 200-400ms
+
+**Approach**: Denormalized edge indexes (see PERFORMANCE_OPTIMIZATION_PLAN.md)
+
+**When to pursue**:
+- Only if click query performance is unacceptable
+- After Phase 2 is complete
+- If baseline data shows queries are consistently >2 seconds
+
+---
+
+## 🔍 Debugging Tips
+
+### If Progress Indicator Not Visible:
+- Check: Is `loading_1` div hidden by CSS?
+- Check: Browser console for JavaScript errors
+- Verify: `loadingDiv.hidden = false` is executing (add console.log)
+
+### If Console Logs Missing:
+- Verify: Browser console is set to show "Verbose" or "All" messages
+- Check: Performance API available (`typeof performance !== 'undefined'`)
+- Verify: No JavaScript errors blocking execution
+
+### If Page Still Freezes:
+- Reduce CHUNK_SIZE from 500 → 100 (more yields, slower but more responsive)
+- Check: Browser is not in "Performance" mode (some browsers batch setTimeout)
+- Verify: `await new Promise(...)` is actually yielding (test with longer timeout)
+
+---
+
+## 💡 Code Changes Summary
+
+**File Modified**: `tutorials/parquet_cesium.qmd`
+
+**Lines Changed**:
+- 131-248: Enhanced `locations` query with telemetry + chunked rendering (+110 lines)
+- 400-406: Added telemetry to `get_samples_1()` (+6 lines)
+- 462-468: Added telemetry to `get_samples_2()` (+6 lines)
+- 524-530: Added telemetry to `get_samples_at_geo_cord_location_via_sample_event()` (+6 lines)
+
+**Total Impact**: ~130 lines added (mostly comments + logging)
+
+---
+
+## 🎬 User Experience Flow
+
+**Before**:
+1. User loads page
+2. *[5-10 seconds of frozen browser with "Loading..." text]*
+3. Globe appears with all points
+4. User clicks point
+5. *[1-2 seconds wait]*
+6. Tables appear
+
+**After**:
+1. User loads page
+2. Globe appears immediately
+3. "Querying geocodes from parquet..." (2-8 sec)
+4. "Rendering geocodes... 500/1,234 (41%)" (0.4-0.8 sec, visible progress)
+5. All points visible, page interactive
+6. User clicks point
+7. *[1-2 seconds wait]* (console shows timing)
+8. Tables appear
+
+**Key Difference**: User knows what's happening and page remains responsive!
+
+---
+
+## 📝 Additional Notes
+
+- **No data model changes**: All optimizations are UX-level improvements
+- **No breaking changes**: Queries return same results, just with timing info
+- **No maintenance burden**: Once deployed, no ongoing work needed
+- **Fully backwards compatible**: Page works exactly the same, just feels faster
+- **Console logs can be removed**: If too noisy, delete console.log lines (keep timing code for future debugging)
+
+---
+
+## ✅ Success Criteria
+
+**Lazy Loading Implementation Complete When**:
+- ✅ Progress indicator shows during initial load
+- ✅ Page remains interactive during rendering
+- ✅ Console logs show timing measurements
+- ✅ No JavaScript errors in console
+- ✅ All points render correctly (same as before)
+- ✅ Click queries work with timing logs
+
+**Ready for Next Phase When**:
+- Baseline metrics collected (query times, render times)
+- User feedback gathered (is it fast enough?)
+- Decision made: Phase 2 optimization needed? (Y/N)