finally got to it - #3
Conversation
PatidarAnjali
commented
May 2, 2026
PatidarAnjali
commented
- wire express to real routes, db init, optional redis, error handler
- fix scrape queue job + cache bust after writes
- rate-limit-redis v4 uses sendCommand; fallback to memory if redis is down
- scraper normalization + eventbrite fallback when html changes
- readme / .env.example / docker-compose tweaks; lowercase code comments
There was a problem hiding this comment.
Pull request overview
This PR moves the project from a scaffolded/demo state toward a runnable Eventflow API by initializing core infrastructure at startup, wiring Express to the real routes/services, and tightening the scraping/cache/worker flow around PostgreSQL and optional Redis.
Changes:
- Initialize the app with database and optional Redis, mount the real health/events routers, and centralize rate limiting/error handling.
- Rework scraping flow with normalized scraper output, Redis cache invalidation after writes, and an updated Eventbrite scraper/worker setup.
- Refresh operational/config docs and local setup defaults across README, env example, Docker, and package scripts.
Reviewed changes
Copilot reviewed 19 out of 19 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
src/services/scrapingService.js |
Adds cache invalidation after scraper writes. |
src/scrapers/eventbriteScraper.js |
Removes old demo-only Eventbrite scraper file. |
src/scrapers/eventbrite.js |
Updates live Eventbrite scraping logic and fallback behavior. |
src/scrapers/Base/BaseScraper.js |
Adds normalization helpers and generated external IDs. |
src/routes/health.js |
Switches health checks to real DB/Redis-aware status reporting. |
src/routes/events.js |
Removes route-level limiter and updates route comments/docs. |
src/models/Event.js |
Replaces placeholder comment with empty module export. |
src/middleware/rateLimit.js |
Creates Redis-backed or in-memory rate limiter dynamically. |
src/middleware/cache.js |
Skips cache when Redis is unavailable and async-writes cache entries. |
src/jobs/scrapeEvents.js |
Simplifies worker processing and optional recurring scrape registration. |
src/config/redis.js |
Adds optional Redis initialization, availability checks, and cache invalidation. |
src/config/queue.js |
Configures Bull queue Redis connection defaults. |
src/config/database.js |
Adds local-friendly DB defaults and startup connectivity check. |
src/app.js |
Refactors into an app factory with startup initialization and real route mounting. |
scripts/migrate.js |
Cleans migration logging and adds connection context output. |
package.json |
Updates entry point, test script, and Docker commands. |
docker/docker-compose.yml |
Drops Compose version stanza. |
README.md |
Rewrites project/setup/API documentation for current architecture. |
.env.example |
Adds a fuller local configuration template. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| health: '/api/v1/health', | ||
| events: '/api/v1/events', | ||
| eventById: '/api/v1/events/:id', | ||
| eventStats: '/api/v1/events/stats/summary', |
There was a problem hiding this comment.
Fixed in commit e0345bf. Moved GET /stats/summary and POST /scrape to register before GET /:id in src/routes/events.js. Express now matches the literal /stats/summary path correctly instead of routing it to the /:id handler with id = "stats".
| stableExternalId(rawEvent) { | ||
| if (rawEvent.id) { | ||
| return String(rawEvent.id); | ||
| } | ||
| const basis = `${this.name}|${rawEvent.title || ''}|${rawEvent.startDate || ''}|${rawEvent.url || ''}`; | ||
| const hash = crypto.createHash('sha256').update(basis).digest('hex').slice(0, 32); | ||
| return `${this.name}-${hash}`; |
There was a problem hiding this comment.
Fixed in commit 0552404. stableExternalId now always returns ${this.name}-${rawEvent.id} (e.g. eventbrite-12345, meetup-12345) so IDs from different sources never collide on the external_id unique constraint.
| if (!isRedisAvailable()) { | ||
| health.services.redis = 'not_configured'; | ||
| } else { | ||
| try { | ||
| await getRedisClient().ping(); | ||
| health.services.redis = 'healthy'; | ||
| } catch (error) { | ||
| health.services.redis = 'unhealthy'; | ||
| health.status = 'degraded'; |
There was a problem hiding this comment.
Fixed in commit 0552404. Added a configured flag in redis.js (set to true before the connection attempt when Redis is not explicitly disabled) exposed as isRedisConfigured(). Health check now has three distinct states: not_configured (REDIS_ENABLED=false, HTTP 200), unhealthy+503 (was intended but startup connection failed), and healthy+200.
| redis: { | ||
| host: process.env.REDIS_HOST || '127.0.0.1', | ||
| port: parseInt(process.env.REDIS_PORT, 10) || 6379, | ||
| password: process.env.REDIS_PASSWORD || undefined, |
There was a problem hiding this comment.
Fixed in commit 0552404. queue.js now has a getRedisUrl() helper that returns REDIS_URL when set, and otherwise constructs a redis://[auth@]host:port URL from REDIS_HOST/REDIS_PORT/REDIS_PASSWORD — consistent with the URL-based strategy used by the rest of the app.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…ram capture Agent-Logs-Url: https://github.com/PatidarAnjali/Eventflow-API/sessions/ac87f394-a662-46d1-953c-12250e9ad1d1 Co-authored-by: PatidarAnjali <101072121+PatidarAnjali@users.noreply.github.com>
…iled, support REDIS_URL in queue Agent-Logs-Url: https://github.com/PatidarAnjali/Eventflow-API/sessions/90d0f314-5e13-438b-b51b-4bdb1cdbd3d8 Co-authored-by: PatidarAnjali <101072121+PatidarAnjali@users.noreply.github.com>
