A production-oriented distributed job queue built with Go, PostgreSQL, Redis, and a worker pool.
The project demonstrates how to build a reliable asynchronous job-processing backend with:
- User authentication and session-based authorization
- Per-user job ownership
- Durable job persistence in PostgreSQL
- Redis-backed asynchronous job delivery
- Concurrent workers
- Job lifecycle tracking
- Attempt counting
- Exponential retry with bounded backoff
- Permanent failure after retry exhaustion
- Recovery of stale
processingjobs - Startup recovery of pending jobs
- Graceful worker shutdown
- HTTP API suitable for a web frontend
- Automated Go tests and static analysis
The goal is not simply to move messages from an HTTP endpoint to a worker.
The goal is to separate durability, delivery, and execution so that a temporary failure of one component does not automatically mean losing a job.
- Overview
- Why This Project Exists
- Core Concepts
- Architecture
- System Flow
- Job Lifecycle
- Failure and Retry Model
- Recovery Model
- Authentication and Authorization
- Backend Structure
- Technology Choices
- Database Design
- Redis Queue Design
- Worker Design
- API Reference
- Frontend Integration
- Local Development
- Environment Configuration
- Database Setup
- Running the Backend
- Testing
- Manual API Testing
- Example Job Lifecycle
- Reliability Guarantees
- Important Tradeoffs
- Failure Scenarios
- Scaling
- Security Considerations
- Observability
- Current Scope
- Future Improvements
- Engineering Decisions
- Project Status
- License
This project implements a distributed asynchronous job-processing system.
A client submits a job through the HTTP API:
Client
|
| POST /jobs
v
API Server
|
| persist job
v
PostgreSQL
|
| enqueue job ID
v
Redis
|
| worker consumes job ID
v
Worker
|
| execute
v
PostgreSQL
The important architectural decision is that PostgreSQL is the durable source of truth for job state while Redis is responsible for job delivery between the API and workers.
Redis does not need to contain the complete job.
It only needs to carry the job identifier.
That distinction is fundamental to the reliability model.
A naive background-job implementation often looks like:
HTTP request
|
v
push job into queue
|
v
worker executes job
That approach becomes problematic when failures occur.
For example:
- What happens if the API successfully writes the job to the queue but crashes before responding?
- What happens if Redis temporarily fails?
- What happens if a worker crashes after marking a job as processing?
- What happens if a job execution fails?
- What happens if a worker dies halfway through execution?
- What happens when the entire worker process restarts?
- How do we know how many times a job has been attempted?
- How do we know which user owns a job?
This project addresses those problems by making the database the durable state machine and Redis the asynchronous delivery mechanism.
The system revolves around several concepts.
A job represents work that should be executed asynchronously.
A job contains:
- unique ID
- owning user ID
- type
- payload
- status
- current attempt count
- maximum attempts
- creation timestamp
- update timestamp
The model contains:
ID
UserID
Type
Payload
Status
Attempts
MaxAttempts
CreatedAt
UpdatedAt
The backend currently defaults newly created jobs to:
status = pending
attempts = 0
max_attempts = 3
Redis contains job IDs waiting to be processed.
The queue name is:
job_queue
The queue does not act as the permanent database for jobs.
PostgreSQL stores the actual job record.
A worker:
- waits for a job ID from Redis
- loads the job from PostgreSQL
- increments its attempt count
- marks it as
processing - executes it
- marks it
completedon success - retries it on failure when attempts remain
- marks it
failedwhen retry attempts are exhausted
Multiple workers can run concurrently.
┌──────────────────────┐
│ Frontend │
│ Web Application │
└──────────┬───────────┘
│
│ HTTP
▼
┌──────────────────────┐
│ Go API │
│ │
│ Auth │
│ Job Handlers │
│ Middleware │
│ Services │
└───────┬───────┬──────┘
│ │
durable │ │ delivery
state │ │
▼ ▼
┌────────────┐ ┌────────────┐
│ PostgreSQL │ │ Redis │
│ │ │ │
│ users │ │ job_queue │
│ jobs │ │ │
└────────────┘ └─────┬──────┘
│
│ job IDs
▼
┌────────────────────────┐
│ Worker Pool │
│ │
│ Worker 1 │
│ Worker 2 │
│ Worker 3 │
│ ... │
└───────────┬─────────────┘
│
│ update state
▼
PostgreSQL
The system intentionally separates:
Durability -> PostgreSQL
Delivery -> Redis
Execution -> Workers
This gives each component a focused responsibility.
Responsible for:
- users
- jobs
- ownership
- status
- attempt counters
- timestamps
- durable state
Responsible for:
- distributing pending job IDs
- blocking worker consumption
- decoupling API traffic from worker execution
Responsible for:
- claiming work
- executing jobs
- updating lifecycle state
- retrying failed jobs
- recovering stale work
The user registers or logs in.
The backend authenticates the user and establishes a session.
Subsequent protected job requests use the authenticated session.
The client sends:
POST /jobs
Content-Type: application/json
Cookie: session_id=<session>with:
{
"type": "email",
"payload": {
"recipient": "user@example.com"
}
}The API validates:
- authentication
- job type
- payload
The job is then persisted with:
pending
attempts = 0
max_attempts = 3
The job ID is placed into Redis.
Workers block on Redis waiting for job IDs.
When a job ID arrives:
Redis
|
v
Worker
|
v
PostgreSQL
The worker loads the durable job record.
The worker marks the job as:
processing
and executes the job.
On success:
processing -> completed
On failure:
processing -> pending
if retries remain.
If no retries remain:
processing -> failed
The normal lifecycle is:
┌─────────────┐
│ pending │
└──────┬──────┘
│
│ worker claims
▼
┌─────────────┐
│ processing │
└──────┬──────┘
│
┌─────────┴─────────┐
│ │
success failure
│ │
▼ ▼
┌───────────┐ attempts remain?
│ completed │ / \
└───────────┘ yes no
│ │
▼ ▼
pending failed
│
│ retry
└─────────────► processing
The important states are:
| Status | Meaning |
|---|---|
pending |
Waiting to be processed |
processing |
Currently being executed |
completed |
Successfully executed |
failed |
Permanently failed after retry exhaustion |
The worker uses exponential backoff.
The current retry configuration is:
Base delay: 2 seconds
Maximum delay: 30 seconds
The effective sequence is:
Attempt 1 failure -> 2 seconds
Attempt 2 failure -> 4 seconds
Attempt 3 failure -> permanent failure
The backoff is bounded so the delay cannot grow indefinitely.
Conceptually:
delay = min(baseDelay * 2^(attempt-1), maxDelay)
The retry implementation lives under:
internal/retry/backoff.go
Redis contains:
job ID
rather than:
complete job payload
This is intentional.
The worker always reloads the job from PostgreSQL.
That means the database remains the authoritative representation of the job.
The queue becomes a delivery mechanism rather than a second source of truth.
This reduces synchronization problems between Redis and PostgreSQL.
Retries alone are not enough.
Consider:
Worker
|
| job is processing
|
X
worker crashes
Without recovery, the database could remain stuck at:
processing
forever.
The worker therefore periodically searches for stale processing jobs.
The current recovery model considers a job stale after:
2 minutes
The recovery loop runs periodically and requeues stale eligible jobs.
Conceptually:
processing
|
| no update for 2 minutes
v
recovery scanner
|
v
pending
|
v
Redis
|
v
worker
Only jobs that still have retry capacity are eligible for recovery.
A distributed worker system must also consider jobs that were already pending before a worker process restarted.
The worker provides a mechanism to find pending jobs and requeue them.
This means the database can survive worker-process restarts without requiring the previous Redis delivery event to still be present.
The important principle is:
PostgreSQL remembers what needs to happen.
Redis helps workers discover what needs to happen now.
The API uses user authentication before allowing protected job operations.
A job belongs to a specific user:
jobs.user_id
This allows the application to associate work with its creator.
The authenticated user's ID is obtained from request context and passed into the job service.
The service then creates the job with that owner.
Users contain:
id
email
password_hash
created_at
updated_at
Passwords are not stored as plaintext.
Password hashing uses bcrypt.
The password hash is also excluded from JSON responses through the model's JSON configuration.
Authentication uses a session cookie.
The browser receives a cookie such as:
session_id=<opaque-session-value>
The cookie is configured with:
HttpOnly
SameSite=Lax
For local development this makes browser-based frontend integration straightforward.
The frontend should generally rely on the browser's cookie handling rather than storing authentication credentials in local storage.
The project follows a layered architecture.
cmd/
api/
main.go
worker/
main.go
internal/
auth/
config/
database/
handler/
middleware/
models/
queue/
repository/
retry/
service/
worker/
migrations/
Application entry point for the HTTP API.
Responsibilities include:
- loading environment configuration
- connecting to PostgreSQL
- connecting to Redis
- constructing repositories
- constructing services
- constructing handlers
- starting the HTTP server
Application entry point for worker processes.
Responsibilities include:
- loading configuration
- connecting to PostgreSQL
- connecting to Redis
- constructing the worker
- creating the process cancellation context
- starting the worker pool
The worker process listens for:
SIGINT
SIGTERM
and uses context cancellation for shutdown.
Domain models shared between layers.
Current core models:
User
Job
Database access layer.
Repositories encapsulate SQL operations such as:
Create
GetById
GetByUserID
Claim
UpdateStatus
GetStaleProcessingJobs
GetPendingJobs
This keeps SQL concerns out of handlers and workers.
Business logic layer.
The job service is responsible for constructing new jobs and coordinating persistence with queue delivery.
The authentication service is responsible for:
- registration
- email normalization
- password hashing
- duplicate detection
- login
- credential validation
Redis queue abstraction.
The queue exposes operations such as:
Enqueue
Dequeue
The current Redis implementation uses a Redis list and blocking pop semantics.
Retry policy.
Currently this contains:
backoff.go
The backoff policy is intentionally isolated from worker execution logic.
Worker execution and lifecycle management.
Responsibilities:
- consume queue
- load jobs
- claim/transition jobs
- increment attempts
- execute jobs
- retry failures
- mark permanent failures
- recover stale jobs
- requeue pending jobs
- coordinate multiple worker goroutines
Go was chosen for:
- lightweight concurrency
- goroutines
- channels/context support
- low operational overhead
- excellent HTTP support
- strong standard library
- straightforward deployment
The worker pool maps naturally to Go's concurrency model.
PostgreSQL is used as the durable system of record.
Advantages:
- transactions
- strong consistency
- durable storage
- relational ownership model
- indexing/query capabilities
- JSONB payload support
The database is responsible for remembering the state of every job.
Redis is used as the delivery layer.
Advantages:
- very fast queue operations
- simple list-based queue implementation
- blocking consumption
- easy horizontal scaling
- mature Go client
Redis is not used as the authoritative job database.
CREATE TABLE users (
id UUID PRIMARY KEY,
email TEXT NOT NULL UNIQUE,
password_hash TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);CREATE TABLE jobs (
id UUID PRIMARY KEY,
user_id UUID NOT NULL,
type VARCHAR(100) NOT NULL,
payload JSONB NOT NULL,
status VARCHAR(20) NOT NULL DEFAULT 'pending',
attempts INT NOT NULL DEFAULT 0,
max_attempts INT NOT NULL DEFAULT 3,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);The job table deliberately stores the payload as JSONB.
This allows the system to support arbitrary structured job payloads rather than forcing every job type into a fixed schema.
The Redis queue is:
job_queue
Jobs are inserted using:
RPUSH job_queue <job-id>
Workers consume using a blocking pop.
The current implementation uses a five-second dequeue timeout so workers periodically regain control of their context instead of waiting indefinitely inside Redis.
The worker process supports multiple concurrent workers.
The current worker command starts:
3 workers
The conceptual structure is:
Worker process
|
+-- Worker 1
|
+-- Worker 2
|
+-- Worker 3
Each worker independently consumes jobs.
This allows multiple jobs to execute concurrently.
GET /healthResponse:
{
"status": "ok"
}POST /auth/register
Content-Type: application/jsonRequest:
{
"email": "user@example.com",
"password": "password123"
}Successful response:
201 CreatedExample:
{
"id": "uuid",
"email": "user@example.com"
}POST /auth/login
Content-Type: application/jsonRequest:
{
"email": "user@example.com",
"password": "password123"
}Successful response:
200 OK
Set-Cookie: session_id=<session>Example body:
{
"id": "uuid",
"email": "user@example.com"
}POST /auth/logoutThe server invalidates the session and clears the session cookie.
Successful response:
204 No ContentPOST /jobs
Content-Type: application/json
Cookie: session_id=<session>Request:
{
"type": "email",
"payload": {
"recipient": "user@example.com"
}
}Successful response:
201 CreatedExample:
{
"id": "uuid",
"user_id": "uuid",
"type": "email",
"payload": "eyJyZWNpcGllbnQiOiJ1c2VyQGV4YW1wbGUuY29tIn0=",
"status": "pending",
"attempts": 0,
"max_attempts": 3,
"created_at": "2026-08-16T17:34:31.520382+05:30",
"updated_at": "2026-08-16T17:34:31.520382+05:30"
}The request payload is JSON.
Internally the model represents it as []byte.
Go's JSON encoder therefore serializes the response payload as base64.
For example:
{
"message": "hello worker"
}may appear in a response as:
eyJtZXNzYWdlIjogImhlbGxvIHdvcmtlciJ9
A frontend consuming the API should account for this representation.
GET /jobs
Cookie: session_id=<session>The endpoint returns jobs associated with the authenticated user.
Example:
[
{
"id": "uuid",
"user_id": "uuid",
"type": "email",
"payload": "...",
"status": "completed",
"attempts": 1,
"max_attempts": 3,
"created_at": "...",
"updated_at": "..."
}
]The API validates authentication and job input.
Examples include:
401 UnauthorizedExample:
authentication required
or:
invalid session
400 Bad Requestjob type is required
Whitespace-only types are also rejected.
400 Bad Requestpayload is required
400 Bad Requestinvalid request body
The frontend should treat the backend as a session-authenticated HTTP API.
A browser-based client should send credentials with requests where necessary.
For example:
fetch("http://localhost:8080/jobs", {
method: "GET",
credentials: "include"
});For job creation:
fetch("http://localhost:8080/jobs", {
method: "POST",
credentials: "include",
headers: {
"Content-Type": "application/json"
},
body: JSON.stringify({
type: "email",
payload: {
recipient: "user@example.com"
}
})
});The frontend should model job states as:
pending
processing
completed
failed
A useful UI representation is:
Pending -> waiting
Processing -> actively executing
Completed -> successful
Failed -> permanently failed
The frontend should not assume that a job will immediately transition from pending to completed.
Job execution is asynchronous.
Install:
- Go
- PostgreSQL
- Redis
Verify:
go version
psql --version
redis-cli --versionCreate a local .env file.
Example:
PORT=8080
DATABASE_URL=postgres://postgres:postgres@localhost:5432/jobqueue?sslmode=disable
REDIS_URL=redis://localhost:6379
SESSION_SECRET=change-meDo not commit real secrets.
The repository includes an example environment file for local configuration.
Create the database:
createdb jobqueueThen apply the migrations in order:
psql "$DATABASE_URL" -f migrations/001_create_jobs.sql
psql "$DATABASE_URL" -f migrations/002_create_users.sqlThe migrations currently create:
users
jobs
Make sure Redis is running:
redis-serverVerify:
redis-cli pingExpected:
PONG
From the repository root:
go run ./cmd/apiThe API listens on:
http://localhost:8080
Health check:
curl http://localhost:8080/healthExpected:
{"status":"ok"}In another terminal:
go run ./cmd/workerThe worker connects to:
PostgreSQL
Redis
and starts its worker pool.
The architecture supports running multiple worker processes.
For example:
go run ./cmd/workerin multiple terminals.
Each process consumes from the same Redis queue and accesses the same PostgreSQL database.
The queue therefore acts as a shared coordination mechanism.
Run all tests:
go test ./...Run static analysis:
go vet ./...Check formatting:
gofmt -w $(find . -name '*.go' -not -path './vendor/*')Check whitespace/errors in the diff:
git diff --checkRecommended pre-commit validation:
gofmt -w $(find . -name '*.go' -not -path './vendor/*')
go test ./...
go vet ./...
git diff --checkcurl -i -X POST http://localhost:8080/auth/register \
-H "Content-Type: application/json" \
-d '{
"email":"test@example.com",
"password":"password123"
}'Save the session cookie:
curl -i -c cookies.txt -X POST http://localhost:8080/auth/login \
-H "Content-Type: application/json" \
-d '{
"email":"test@example.com",
"password":"password123"
}'curl -i -b cookies.txt -X POST http://localhost:8080/jobs \
-H "Content-Type: application/json" \
-d '{
"type":"success-test",
"payload":{
"message":"hello worker"
}
}'curl -s -b cookies.txt http://localhost:8080/jobs | jqWithout the session:
curl -i -X POST http://localhost:8080/jobs \
-H "Content-Type: application/json" \
-d '{
"type":"auth-check",
"payload":{
"test":true
}
}'Expected:
401 Unauthorized
Missing type:
curl -i -b cookies.txt -X POST http://localhost:8080/jobs \
-H "Content-Type: application/json" \
-d '{
"type":"",
"payload":{
"test":true
}
}'Expected:
400 Bad Request
job type is required
Missing payload:
curl -i -b cookies.txt -X POST http://localhost:8080/jobs \
-H "Content-Type: application/json" \
-d '{
"type":"payload-check"
}'Expected:
400 Bad Request
payload is required
A successful job looks approximately like:
POST /jobs
|
v
pending
|
v
processing
|
v
completed
The API initially returns:
{
"status": "pending",
"attempts": 0,
"max_attempts": 3
}After successful execution:
{
"status": "completed",
"attempts": 1,
"max_attempts": 3
}A job with a simulated failure can be represented using:
{
"type": "fail",
"payload": {
"should": "retry"
}
}The worker intentionally fails this job.
The lifecycle becomes:
pending
|
v
processing
|
X
failure
|
v
pending
|
| 2s
v
processing
|
X
failure
|
v
pending
|
| 4s
v
processing
|
X
failure
|
v
failed
After three attempts:
{
"status": "failed",
"attempts": 3,
"max_attempts": 3
}The job is no longer retried.
The current design provides several important guarantees.
A job is persisted in PostgreSQL before normal worker processing.
This means job state survives a worker process restart.
Every worker execution increments the attempt count.
This prevents an endlessly failing job from being retried forever.
The maximum attempt count is currently:
3
A permanently failing job eventually becomes:
failed
Retries do not happen immediately.
The current delay sequence reduces pressure on the worker and infrastructure:
2s
4s
8s
...
30s maximum
A worker that dies while processing a job can leave a database record in:
processing
The recovery mechanism detects stale processing jobs and requeues eligible work.
Jobs that remain pending can be discovered from PostgreSQL and requeued.
This means the database can reconstruct work that still needs delivery.
Jobs are associated with the authenticated user's ID.
A user's job list is scoped to that user.
No distributed system design is free.
This project intentionally chooses simplicity in several places.
Use:
PostgreSQL -> durable state
Redis -> queue
instead of storing everything in Redis.
The job lifecycle survives Redis delivery failures and worker restarts more cleanly.
Every job involves multiple systems.
The API must coordinate:
PostgreSQL
Redis
which increases operational complexity.
PostgreSQL can implement queues using techniques such as:
SELECT ... FOR UPDATE SKIP LOCKED
That would remove Redis from the architecture.
However, Redis provides a simpler dedicated delivery mechanism and is well suited to high-frequency queue operations.
The tradeoff is another infrastructure dependency.
The current queue uses a Redis list.
Advantages:
- extremely simple
- easy to understand
- easy to debug
- native blocking consumption
- minimal code
A more advanced production system could use:
- Redis Streams
- Kafka
- NATS JetStream
- RabbitMQ
- SQS
- another durable messaging system
The current implementation deliberately avoids introducing those systems before the application actually needs them.
This prevents the queue from becoming a second database.
If the payload were stored independently in Redis, the system would need to reason about synchronization between:
PostgreSQL payload
Redis payload
Using only IDs means:
PostgreSQL = source of truth
Redis = delivery mechanism
This is much easier to reason about.
The architecture should be understood as at-least-once oriented, not exactly-once processing.
A job may potentially execute more than once in failure scenarios.
For example:
worker executes external side effect
|
X
worker crashes before recording completion
|
v
job is recovered
|
v
job executes again
Therefore job handlers should ideally be idempotent.
For example, instead of blindly charging a payment twice:
charge(user, amount)
a real system should use an idempotency key:
charge(payment_id, amount)
and make the downstream operation idempotent.
If the job has already been committed to PostgreSQL but Redis insertion fails, the database contains a pending job.
A recovery/reconciliation mechanism can requeue pending jobs.
This is one reason PostgreSQL is treated as the durable state store.
The job can remain:
processing
until the stale-job recovery mechanism detects it.
If attempts remain, it can return to:
pending
and be requeued.
The job is retried up to:
max_attempts
After that:
failed
This prevents poison jobs from consuming worker capacity indefinitely.
Workers cannot consume new jobs and API requests cannot enqueue jobs.
PostgreSQL still contains the durable job state.
Once Redis is restored, pending jobs can be reintroduced into the queue.
Workers cannot safely load or update durable job state.
Processing should therefore fail rather than pretending the job was successfully completed.
This preserves the database as the source of truth.
The architecture is horizontally scalable at the worker layer.
For example:
Redis
|
┌─────────┼─────────┐
│ │ │
Worker Worker Worker
Node 1 Node 2 Node 3
All workers consume from the same queue.
They all access the same PostgreSQL database.
The application can therefore increase processing capacity by adding worker processes.
The current worker process starts:
3 concurrent workers
This is intentionally configurable at the process/code level rather than creating a separate process for every job.
Goroutines provide cheap concurrency within each worker process.
A deployment can combine:
multiple processes
+
multiple goroutines per process
to increase throughput.
At larger scale, additional concerns become important:
- PostgreSQL connection pool sizing
- Redis connection limits
- queue partitioning
- job priority
- rate limiting
- worker autoscaling
- backpressure
- dead-letter queues
- metrics
- distributed tracing
- structured logging
- idempotency
- graceful deployment
- schema migration tooling
- multi-region behavior
These are intentionally outside the current frozen backend scope.
Passwords are hashed using bcrypt.
Plaintext passwords are never intentionally persisted.
The session cookie uses:
HttpOnly
SameSite=Lax
This reduces exposure to client-side JavaScript and provides a baseline CSRF mitigation posture.
Production deployments should additionally configure:
Secure
when serving over HTTPS.
Secrets must not be committed to Git.
Use:
.env
locally and environment variables in deployed environments.
Never commit:
SESSION_SECRET
database credentials
Redis credentials
production credentials
session cookies
Job creation validates:
authenticated user
job type
payload
Malformed JSON is rejected.
Whitespace-only job types are rejected.
The current backend uses application logging for important worker events.
Examples include:
worker started
job picked
job processing
job completed
job failed
job retrying
job permanently failed
worker shutdown
This makes local debugging and lifecycle verification straightforward.
A production deployment would ideally add:
structured JSON logs
metrics
distributed tracing
The backend currently provides the core functionality required for a distributed job queue:
- registration
- login
- logout
- password hashing
- session-based authentication
- authenticated job access
- create job
- list user's jobs
- job ownership
- input validation
- Redis-backed queue
- enqueue
- blocking dequeue
- concurrent workers
- job execution
- attempt tracking
- retry
- exponential backoff
- permanent failure
- stale-job recovery
- pending-job requeue
- graceful shutdown
- Go formatting
- unit tests
go vetgit diff --check
This project should not be interpreted as a fully managed enterprise queue platform.
It intentionally does not currently provide all of the following:
- priority queues
- scheduled jobs
- delayed delivery as a first-class queue feature
- cron jobs
- dead-letter queues
- job cancellation
- job progress reporting
- distributed tracing
- Prometheus metrics
- OpenTelemetry
- automatic worker autoscaling
- Kafka-style partitioning
- multi-region replication
- exactly-once execution
- workflow/DAG orchestration
Those are possible future layers.
The current system focuses on getting the fundamental job lifecycle correct first.
The biggest architectural improvement would be addressing the dual-write boundary:
PostgreSQL
+
Redis
The current create flow conceptually does:
INSERT job
|
v
ENQUEUE job
If the database succeeds but Redis fails, the system needs recovery/reconciliation.
A more advanced design could introduce an outbox pattern:
HTTP request
|
v
PostgreSQL transaction
|
+---- jobs
|
+---- outbox event
|
v
queue publisher
|
v
Redis
This would provide a stronger guarantee around database-to-queue delivery.
Instead of only marking jobs as:
failed
a production system could move permanently failed jobs into a dead-letter queue.
This would allow:
- inspection
- replay
- manual recovery
- operational debugging
Add an endpoint such as:
POST /jobs/{id}/cancelwith safe lifecycle rules such as:
pending -> cancelled
processing -> cancellation requested
completed -> immutable
failed -> immutable
A dedicated endpoint would improve frontend integration:
GET /jobs/{id}This would allow the frontend to retrieve a single job without loading the entire user's job list.
The current job listing is conceptually:
GET /jobsA larger system should support:
?page=1
&limit=50
or cursor-based pagination.
Cursor pagination would generally be preferable at very large scale.
The current worker uses job type logic such as:
fail
for simulated failures.
A real application could use a registry:
email -> EmailHandler
webhook -> WebhookHandler
thumbnail -> ThumbnailHandler
notification -> NotificationHandler
This would separate job dispatch from worker infrastructure.
Useful metrics include:
jobs_created_total
jobs_completed_total
jobs_failed_total
jobs_retried_total
job_processing_duration
queue_depth
worker_utilization
stale_jobs_recovered_total
A production deployment could propagate a trace ID:
Frontend
|
v
API
|
v
PostgreSQL / Redis
|
v
Worker
This would make debugging asynchronous failures significantly easier.
The project follows a few central principles.
If the system needs to remember something important about a job, it belongs in PostgreSQL.
Redis should answer:
Which work should a worker look at?
PostgreSQL should answer:
What is the actual state of this job?
Workers are not trusted to always finish.
Therefore:
processing
must never be treated as permanent.
Stale-job recovery exists because worker crashes are expected in distributed systems.
A temporary network/database/external-service failure should not immediately destroy a job.
Retries provide resilience.
Infinite retries create poison jobs.
Therefore every job has:
max_attempts
Immediate retries can amplify an outage.
Exponential backoff spreads retry attempts over time.
Distributed systems can experience crashes between side effects and state updates.
Therefore real job handlers should be designed for idempotency.
The project was intentionally built incrementally.
The development sequence focused on:
basic job creation
↓
persistence
↓
Redis queue
↓
worker execution
↓
attempt tracking
↓
retry
↓
backoff
↓
authentication
↓
ownership
↓
lifecycle hardening
↓
stale-job recovery
↓
integration testing
↓
backend freeze
This approach makes each reliability feature understandable instead of introducing an unnecessarily complex distributed system from the beginning.
The backend freeze means the current architecture is treated as the stable integration contract for the frontend.
The frontend can build around:
Authentication
|
+-- register
+-- login
+-- logout
|
v
Authenticated session
|
+-- create jobs
+-- list jobs
|
v
Job lifecycle
|
+-- pending
+-- processing
+-- completed
+-- failed
Future backend changes should therefore be treated as deliberate API/architecture changes rather than incidental refactors.
A basic deployment can look like:
Internet
|
v
Reverse Proxy
|
v
┌─────────────┐
│ API Nodes │
└──────┬──────┘
│
┌──────┴──────┐
│ │
v v
PostgreSQL Redis
^ ^
│ │
└──────┬──────┘
│
Worker Nodes
For a real production environment, PostgreSQL and Redis should themselves be deployed with appropriate:
- persistence
- backups
- monitoring
- access controls
- TLS
- failover strategy
Before deploying:
- Set a strong
SESSION_SECRET - Use HTTPS
- Configure secure cookies
- Use production PostgreSQL credentials
- Use authenticated Redis
- Apply database migrations
- Verify
/health - Start at least one worker
- Verify successful job execution
- Verify retry behavior
- Verify permanent failure behavior
- Verify worker restart recovery
- Configure log collection
- Configure database backups
- Configure Redis persistence/HA as appropriate
- Never commit
.env - Never commit session cookie files
The shortest local-development path is:
# 1. Configure environment
cp .env.example .env
# 2. Start PostgreSQL and Redis
# 3. Create database
createdb jobqueue
# 4. Apply migrations
psql "$DATABASE_URL" -f migrations/001_create_jobs.sql
psql "$DATABASE_URL" -f migrations/002_create_users.sql
# 5. Start API
go run ./cmd/api
# 6. In another terminal, start worker
go run ./cmd/worker
# 7. Verify
curl http://localhost:8080/healthThen authenticate and create a job:
curl -i -c cookies.txt -X POST http://localhost:8080/auth/login \
-H "Content-Type: application/json" \
-d '{
"email":"test@example.com",
"password":"password123"
}'curl -i -b cookies.txt -X POST http://localhost:8080/jobs \
-H "Content-Type: application/json" \
-d '{
"type":"success-test",
"payload":{
"message":"hello worker"
}
}'Then:
curl -s -b cookies.txt http://localhost:8080/jobs | jqThe backend should be considered healthy when:
go test ./...passes,
go vet ./...passes,
and:
git diff --checkreturns no errors.
The lifecycle should also be manually verified with:
authentication
↓
job creation
↓
pending
↓
processing
↓
completed
and:
pending
↓
processing
↓
failure
↓
retry
↓
processing
↓
failure
↓
failed
Frozen for frontend integration.
The backend currently provides the core distributed job queue lifecycle, authenticated job ownership, retry handling, worker execution, and recovery mechanisms required by the application.
The frontend is implemented as the presentation and integration layer over the backend API.
The frontend should consume the backend contract rather than duplicate job-processing logic.
The most important idea in this project is simple:
PostgreSQL
durable truth
▲
│
│ state
│
API ────────► Job Service
│ │
│ │
│ ▼
│ Redis
│ delivery
│ │
│ ▼
│ Worker Pool
│ │
│ │ execute
│ ▼
└──────────► PostgreSQL
PostgreSQL remembers.
Redis delivers.
Workers execute.
Retries absorb transient failures.
Backoff prevents retry storms.
Attempt limits prevent poison jobs.
Stale-job recovery handles crashed workers.
Authentication establishes ownership.
And the frontend observes the resulting job lifecycle through the HTTP API.
That separation is the core architectural decision behind the project.