finally got to it

.env.example

@@ -0,0 +1,27 @@
+PORT=3000
+NODE_ENV=development
+LOG_LEVEL=info
+
+# PostgreSQL (defaults match docker/docker-compose.yml)
+DB_HOST=127.0.0.1
+DB_PORT=5432
+DB_NAME=events_db
+DB_USER=postgres
+DB_PASSWORD=postgres
+
+# Redis — omit REDIS_ENABLED or set true; set REDIS_ENABLED=false to skip Redis entirely
+REDIS_ENABLED=
+REDIS_URL=
+REDIS_HOST=127.0.0.1
+REDIS_PORT=6379
+
+# POST /api/v1/events/scrape
+API_KEY=dev-secret-change-me
+
+# Rate limits
+RATE_LIMIT_WINDOW_MS=900000
+RATE_LIMIT_MAX_REQUESTS=100
+
+# Worker: optional recurring scrape (ms), minimum 60000; leave unset to disable
+SCRAPE_INTERVAL_MS=
+SCRAPE_JOB_CONCURRENCY=1

README.md

@@ -1,80 +1,99 @@
 # Eventflow API
 
-> **Status:** This project is actively under development.
-
-### Event Aggregation and API Gateway
-
-Eventflow API is a backend service that aggregates event data from multiple public sources, normalizes it into a consistent format, and exposes it through a single RESTAPI.
-This project demonstrates web scraping, data normalization, and API design.
+Backend service that aggregates event data from multiple public sources, normalizes it, and exposes it through a REST API.
 
 ## Features
-- Scrapes events from public event pages
-- Normalizes messy source data into a consistent schema
-- Exposes events through a REST API
-- Source-agnostic scraper architecture
-- Basic error handling
+- Scrapes events from public listing pages (Eventbrite HTML + Meetup via Puppeteer)
+- Normalizes source-specific rows into a shared schema
+- Persists events in PostgreSQL with upsert by `external_id`
+- REST API with optional Redis-backed caching and rate limiting
+- Background scraping worker (Bull + Redis)
 
 ## Tech Stack
-- Node.js
-- Express
-- Axios
-- Cheerio
-- PostgreSQL (planned)
-- Redis (planned)
-
-## Project Structure
+- Node.js, Express
+- PostgreSQL (`pg`)
+- Redis (cache, rate limits, Bull queue)
+- Axios, Cheerio, Puppeteer
+
+## Project layout
 ```
 src/
-├── app.js
-├── server.js
-├── routes/
-│ └── events.js
-├── scrapers/
-│ └── eventbrite.js
-├── db/
-└── utils/
+├── app.js                 # Express app factory & HTTP server
+├── config/                # database, redis, Bull queue
+├── jobs/                  # worker job definitions (scrape queue)
+├── middleware/            # auth, cache, errors, rate limit
+├── routes/                # events, health
+├── scrapers/              # per-source scrapers + BaseScraper
+├── services/              # event + scraping orchestration
+├── utils/
+migrations/                # SQL migrations
+docker/docker-compose.yml  # Postgres + Redis for local dev
 ```
 
-## API Endpoints
-
-### GET `/events`
-Returns a list of normalized events.
-
-**Example response:**
-```json
-[
-  {
-    "title": "Intro to Web Development",
-    "date": "2025-01-20T18:00:00.000Z",
-    "location": "Toronto, ON",
-    "price": "free",
-    "source": "eventbrite"
-  }
-]
-```
+## Setup
 
-## Setup & Run Locally
-```
-git clone https://github.com/your-username/atlas-api-gateway
-cd atlas-api-gateway
-npm install
-node src/server.js
-```
-API runs at: http://localhost:3000
+1. **Dependencies**
+   ```bash
+   npm install
+   ```
+
+2. **Infrastructure** (PostgreSQL + Redis)
+   ```bash
+   npm run docker:up
+   ```
+
+3. **Environment**
+   ```bash
+   cp .env.example .env
+   # Edit secrets (especially API_KEY) for non-local use.
+   ```
 
-## Design Decisions
-- Single API gateway abstracts multiple sources
-- Scrapers are isolated per source
-- Normalization ensures consistent API output
+4. **Database schema**
+   ```bash
+   npm run migrate
+   ```
 
-## Roadmap
-- Add database persistence
-- Add more event sources
-- Implement rate limiting
-- Add caching
+5. **Run API**
+   ```bash
+   npm start
+   ```
+   Server: http://localhost:3000  
+
+6. **Optional: background worker** (scheduled / queued scrapes; requires Redis)
+   ```bash
+   npm run worker
+   ```
+
+## API
+
+| Method | Path | Description |
+|--------|------|-------------|
+| GET | `/` | Service metadata |
+| GET | `/api/v1/health` | DB + Redis status |
+| GET | `/api/v1/events` | List events (`start_date`, `end_date`, `category`, `source`, `is_free`, `limit`, `offset`) |
+| GET | `/api/v1/events/:id` | Single event |
+| GET | `/api/v1/events/stats/summary` | Counts / aggregates |
+| POST | `/api/v1/events/scrape` | Run scrapers (`X-API-Key` header; optional JSON `{ "source": "eventbrite" \| "meetup" }`) |
+
+Example:
+```bash
+curl -s http://localhost:3000/api/v1/events | jq
+```
+
+Trigger scrape (set `API_KEY` in `.env`):
+```bash
+curl -s -X POST http://localhost:3000/api/v1/events/scrape \
+  -H "Content-Type: application/json" \
+  -H "X-API-Key: dev-secret-change-me" \
+  -d '{}'
+```
 
-## What I Learned
-- Web scraping real-world HTML
-- Designing APIs around inconsistent data
-- Structuring extensible backend services
+## Design notes
+- Scrapers are isolated per source; `BaseScraper` normalizes rows and generates stable `external_id` when a source omits ids.
+- Without Redis, the API still runs using in-memory rate limits and skips HTTP caching.
+- Eventbrite markup changes often; if live selectors find no cards, a single placeholder event is returned so the ingestion path can still be tested end-to-end.
 
+## Roadmap ideas
+- Additional sources and richer normalization
+- Structured logging / metrics
+- Contract tests against recorded HTML fixtures

docker/docker-compose.yml

@@ -1,4 +1,3 @@
-version: '3.8'
 services:
   postgres:
     image: postgres:15-alpine

package.json

@@ -2,15 +2,15 @@
   "name": "eventflow-api",
   "version": "1.0.0",
   "description": "",
-  "main": "index.js",
+  "main": "src/app.js",
   "scripts": {
     "start": "node src/app.js",
     "dev": "nodemon src/app.js",
     "worker": "node src/jobs/worker.js",
     "migrate": "node scripts/migrate.js",
-    "test": "jest || echo 'Tests passed'",
-    "docker:up": "docker-compose -f docker/docker-compose.yml up -d",
-    "docker:down": "docker-compose -f docker/docker-compose.yml down",
+    "test": "node --check src/app.js && node --check src/routes/events.js && node --check src/jobs/scrapeEvents.js",
+    "docker:up": "docker compose -f docker/docker-compose.yml up -d",
+    "docker:down": "docker compose -f docker/docker-compose.yml down",
     "docker:logs": "docker-compose -f docker/docker-compose.yml logs -f"
   },
   "keywords": [],

scripts/migrate.js

@@ -3,10 +3,9 @@ const fs = require('fs').promises;
 const path = require('path');
 const { pool} = require('../src/config/database');
 
+// runs every migrations/*.sql in lexicographic order (prefix filenames with 001_, 002_, …)
 async function runMigrations() {
 
-  console.log('DB_USER @ runtime:',process.env.DB_USER);
-
   try {
     console.log('Running DB migrations...');
 
@@ -16,6 +15,14 @@ async function runMigrations() {
 
     for(const file of sqlFiles) {
 
+      // quick sanity log — helps spot wrong db host when migrations hit the wrong cluster
+      console.log({
+        host: process.env.DB_HOST,
+        port: process.env.DB_PORT,
+        user: process.env.DB_USER,
+        db: process.env.DB_NAME,
+      });
+
       console.log(`Running migration: ${file}`);
 
       const filePath = path.join(migrationsDir, file);

src/app.js

@@ -1,61 +1,88 @@
 require('dotenv').config();
+const fs = require('fs');
+const path = require('path');
+
+// winston writes under logs/; ensure folder exists on cold start (works on all platforms)
+try {
+  fs.mkdirSync(path.join(process.cwd(), 'logs'), { recursive: true });
+} catch (_) {
+  /* logs dir exists or cwd not writable */
+}
+
 const express = require('express');
 const helmet = require('helmet');
 
-const app = express();
+const { initializeDatabase } = require('./config/database');
+const { initializeRedis } = require('./config/redis');
+const errorHandler = require('./middleware/errorHandler');
+const createRateLimiter = require('./middleware/rateLimit');
+const eventsRouter = require('./routes/events');
+const healthRouter = require('./routes/health');
+
 const PORT = process.env.PORT || 3000;
 
-app.use(helmet());
-app.use(express.json());
-
-// groot route - shows API info
-app.get('/', (req, res) => {
-  res.json({
-    name: 'Event Scraper API',
-    version: '1.0.0',
-    status: 'running',
-    endpoints: {
-      health: '/api/v1/health',
-      events: '/api/v1/events'
-    }
-  });
-});
-
-// health check
-app.get('/api/v1/health', (req, res) => {
-  res.json({ 
-    status: 'ok', 
-    message: 'Server is running!',
-    timestamp: new Date().toISOString()
+// wires postgres + optional redis, then mounts routes (rate limit is applied only under /api/v1/events)
+async function createApp() {
+  await initializeDatabase();
+  await initializeRedis();
+
+  const app = express();
+  app.use(helmet());
+  app.use(express.json());
+
+  app.get('/', (req, res) => {
+    res.json({
+      name: 'Eventflow API',
+      version: '1.0.0',
+      status: 'running',
+      endpoints: {
+        health: '/api/v1/health',
+        events: '/api/v1/events',
+        eventById: '/api/v1/events/:id',
+        eventStats: '/api/v1/events/stats/summary',
+        scrape: 'POST /api/v1/events/scrape (header: X-API-Key)',
+      },
+    });
   });
-});
-
-// events endpoint
-app.get('/api/v1/events', (req, res) => {
-  res.json({ 
-    success: true, 
-    data: [],
-    message: 'Events API endpoint - ready for implementation!' 
+
+  app.use('/api/v1/health', healthRouter);
+  // limiter is created after redis init so it can pick redis vs memory store
+  app.use('/api/v1/events', createRateLimiter(), eventsRouter);
+
+  app.use((req, res) => {
+    res.status(404).json({
+      error: 'Not Found',
+      message: `Cannot ${req.method} ${req.path}`,
+      availableEndpoints: [
+        'GET /',
+        'GET /api/v1/health',
+        'GET /api/v1/events',
+        'GET /api/v1/events/:id',
+        'GET /api/v1/events/stats/summary',
+        'POST /api/v1/events/scrape',
+      ],
+    });
   });
-});
-
-// 404 handler
-app.use((req, res) => {
-  res.status(404).json({
-    error: 'Not Found',
-    message: `Cannot ${req.method} ${req.path}`,
-    availableEndpoints: [
-      'GET /',
-      'GET /api/v1/health',
-      'GET /api/v1/events'
-    ]
+
+  app.use(errorHandler);
+  return app;
+}
+
+async function main() {
+  const app = await createApp();
+  app.listen(PORT, () => {
+    console.log(`Server running on http://localhost:${PORT}`);
+    console.log(`Health: http://localhost:${PORT}/api/v1/health`);
+    console.log(`Events: http://localhost:${PORT}/api/v1/events`);
   });
-});
+}
 
-app.listen(PORT, () => {
-  console.log(`Server running on http://localhost:${PORT}`);
-  console.log(`Health check: http://localhost:${PORT}/api/v1/health`);
-  console.log(`Events: http://localhost:${PORT}/api/v1/events`);
-});
+// when imported (e.g. tests), only exports createApp — no listen until main runs
+if (require.main === module) {
+  main().catch((err) => {
+    console.error('Failed to start server', err);
+    process.exit(1);
+  });
+}
 
-module.exports = app;
+module.exports = { createApp };