Spooled Backend

Job queue + worker coordination service built with Rust

Live Demo (SpriteForge) • Documentation • Website

Spooled is a high-performance, multi-tenant job queue system designed for reliability, observability, and horizontal scalability.

✨ Features

• High Performance: Built on Rust + Tokio + PostgreSQL with Redis caching (~28x faster auth)
• Optimized gRPC: HTTP/2 keepalive, TCP optimizations, and connection pooling for ~3x faster throughput
• Multi-Tenant: PostgreSQL Row-Level Security (RLS) for data isolation
• Observable: Prometheus metrics, Grafana dashboards, optional OpenTelemetry export (--features otel)
• Reliable: At-least-once processing with leases + retries (use idempotency keys for exactly-once effects)
• Real-Time: WebSocket + SSE for live job/queue updates
• Secure: Bcrypt API keys with Redis caching, JWT auth, HMAC webhook verification
• Scalable: Stateless API nodes (Kubernetes-friendly) + DB-backed locking (FOR UPDATE SKIP LOCKED)
• Scheduling: Cron-based recurring jobs with timezone support
• Workflows: Job dependencies with DAG execution
• Dual Protocol: REST API (:8080) + real gRPC (GRPC_PORT, default :50051) with streaming support
• Tier-Based Limits: Automatic enforcement across all endpoints (HTTP, gRPC, workflows, schedules)
• Dead Letter Queue: Automatic retry and purge operations for failed jobs
• Webhooks: Outgoing webhook delivery with automatic retries and status tracking
• Billing: Stripe integration for subscriptions and usage tracking

🐳 Quick Start with Docker

Pull and Run

# Pull the multi-arch image (supports amd64 and arm64)
docker pull ghcr.io/spooled-cloud/spooled-backend:latest

# Run with Docker Compose
curl -O https://raw.githubusercontent.com/spooled-cloud/spooled-backend/main/docker-compose.prod.yml
curl -O https://raw.githubusercontent.com/spooled-cloud/spooled-backend/main/.env.example
cp .env.example .env

# Generate secure secrets
export JWT_SECRET=$(openssl rand -base64 32)
export POSTGRES_PASSWORD=$(openssl rand -base64 16)
sed -i "s/your-jwt-secret-minimum-32-characters-long/$JWT_SECRET/" .env
sed -i "s/your_secure_password/$POSTGRES_PASSWORD/g" .env

# Start services
docker compose -f docker-compose.prod.yml up -d

# Verify
curl http://localhost:8080/health

Environment Variables

Plan Limits via Environment Variables (Self-Hosted)

Spooled ships with sensible built-in plan defaults (Free/Starter/Pro/Enterprise), but you can override every plan limit via env vars.

Limits resolution order (lowest → highest precedence):

• Built-in defaults
• SPOOLED_PLAN_LIMITS_JSON (global per-tier JSON map)
• SPOOLED_PLAN_<TIER>_LIMITS_JSON (tier-specific JSON)
• SPOOLED_PLAN_<TIER>_<FIELD> (tier-specific individual fields)
• Organization custom_limits (DB, per-org override)

JSON overrides

• SPOOLED_PLAN_LIMITS_JSON: JSON object mapping tier → overrides (same keys as organizations.custom_limits)
• SPOOLED_PLAN_FREE_LIMITS_JSON, SPOOLED_PLAN_STARTER_LIMITS_JSON, SPOOLED_PLAN_PRO_LIMITS_JSON, SPOOLED_PLAN_ENTERPRISE_LIMITS_JSON

Example:

{
  "free": { "max_jobs_per_day": 5000, "max_payload_size_bytes": 131072 },
  "starter": { "max_active_jobs": 2000 },
  "enterprise": { "max_jobs_per_day": null }
}

Notes:

• For optional limits (like max_jobs_per_day), null means unlimited.

Per-field overrides

You can override individual fields per tier with env vars:

• Limits (support unlimited / none / null / -1):
  • SPOOLED_PLAN_<TIER>_MAX_JOBS_PER_DAY
  • SPOOLED_PLAN_<TIER>_MAX_ACTIVE_JOBS
  • SPOOLED_PLAN_<TIER>_MAX_QUEUES
  • SPOOLED_PLAN_<TIER>_MAX_WORKERS
  • SPOOLED_PLAN_<TIER>_MAX_API_KEYS
  • SPOOLED_PLAN_<TIER>_MAX_SCHEDULES
  • SPOOLED_PLAN_<TIER>_MAX_WORKFLOWS
  • SPOOLED_PLAN_<TIER>_MAX_WEBHOOKS
• Sizes / rates / retention:
  • SPOOLED_PLAN_<TIER>_MAX_PAYLOAD_SIZE_BYTES
  • SPOOLED_PLAN_<TIER>_RATE_LIMIT_RPS
  • SPOOLED_PLAN_<TIER>_RATE_LIMIT_BURST
  • SPOOLED_PLAN_<TIER>_JOB_RETENTION_DAYS
  • SPOOLED_PLAN_<TIER>_HISTORY_RETENTION_DAYS

Where <TIER> is one of: FREE, STARTER, PRO, ENTERPRISE.

🔧 Local Development

Prerequisites

• Rust 1.85+
• Docker & Docker Compose
• PostgreSQL 16+ (or use Docker)
• Redis 7+ (optional, for pub/sub)

Setup

# Clone repository
git clone https://github.com/spooled-cloud/spooled-backend.git
cd spooled-backend

# Start dependencies
docker compose up -d postgres redis

# Run migrations and start server
cargo run

# Run tests
cargo test

API Endpoints

Core Job Management

Dead Letter Queue (DLQ)

Organizations

Schedules & Workflows

Webhooks

Real-Time

Authentication

Billing

Admin API (requires X-Admin-Key header)

Quick Examples

Create Your First Job

# 1. Create an organization (returns initial API key - save it!)
RESPONSE=$(curl -s -X POST http://localhost:8080/api/v1/organizations \
  -H "Content-Type: application/json" \
  -d '{"name": "My Company", "slug": "my-company"}')
echo "$RESPONSE"
# Save the api_key from the response - it's only shown once!
API_KEY=$(echo "$RESPONSE" | jq -r '.api_key')

# 2. Create a job using the API key
curl -X POST http://localhost:8080/api/v1/jobs \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $API_KEY" \
  -d '{
    "queue_name": "emails",
    "payload": {"to": "[email protected]", "subject": "Hello!"},
    "priority": 0,
    "max_retries": 3
  }'

Create a Cron Schedule (Recurring Jobs)

# Run daily sales report every day at 9 AM
curl -X POST http://localhost:8080/api/v1/schedules \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "daily-sales-report",
    "cron_expression": "0 0 9 * * *",
    "timezone": "America/New_York",
    "queue_name": "reports",
    "payload_template": {"report_type": "daily_sales"}
  }'

Create a Workflow (Job Dependencies)

# User onboarding: create account → send email → setup defaults
curl -X POST http://localhost:8080/api/v1/workflows \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "user-onboarding",
    "jobs": [
      {
        "name": "create-account",
        "queue_name": "users",
        "payload": {"email": "[email protected]"}
      },
      {
        "name": "send-welcome",
        "queue_name": "emails",
        "depends_on": ["create-account"],
        "payload": {"template": "welcome"}
      },
      {
        "name": "setup-defaults",
        "queue_name": "users",
        "depends_on": ["create-account"],
        "payload": {"settings": {}}
      }
    ]
  }'

Configure Outgoing Webhooks (Notifications)

# Get notified in Slack when jobs fail
curl -X POST http://localhost:8080/api/v1/outgoing-webhooks \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Slack Alerts",
    "url": "https://hooks.slack.com/services/YOUR/WEBHOOK/URL",
    "events": ["job.failed", "queue.paused"],
    "secret": "your-hmac-secret"
  }'

🔌 gRPC API

Spooled provides a real gRPC API using HTTP/2 + Protobuf for high-performance worker communication.

Endpoints

• Spooled Cloud (TLS): grpc.spooled.cloud:443
• Self-hosted / local: localhost:50051 (or whatever GRPC_PORT is set to)

gRPC TLS (Cloudflare Tunnel)

When using Cloudflare Tunnel with HTTPS origin, gRPC TLS is required because HTTP/2 needs TLS at the origin.

The production docker-compose includes:

• TLS enabled by default (GRPC_TLS_ENABLED=true)
• Self-signed certificates in ./certs/ (10-year validity)
• Performance Optimized: HTTP/2 keepalives, TCP_NODELAY, and tuned connection windows

Cloudflare Tunnel Configuration:

• Service Type: HTTPS
• URL: backend:50051
• HTTP2 Connection: ON
• No TLS Verify: ON (required for self-signed certs)

Note: Cloudflare Tunnel requires HTTPS for HTTP/2 (gRPC). You cannot use plaintext HTTP with gRPC through Cloudflare.

To disable TLS for local development (without Cloudflare):

GRPC_TLS_ENABLED=false cargo run

Proto Definition

The service definitions are in proto/spooled.proto:

service QueueService {
  rpc Enqueue(EnqueueRequest) returns (EnqueueResponse);
  rpc Dequeue(DequeueRequest) returns (DequeueResponse);
  rpc Complete(CompleteRequest) returns (CompleteResponse);
  rpc Fail(FailRequest) returns (FailResponse);
  rpc RenewLease(RenewLeaseRequest) returns (RenewLeaseResponse);
  rpc GetJob(GetJobRequest) returns (GetJobResponse);
  rpc GetQueueStats(GetQueueStatsRequest) returns (GetQueueStatsResponse);
  
  // Server-side streaming for continuous job delivery
  rpc StreamJobs(StreamJobsRequest) returns (stream Job);
  
  // Bidirectional streaming for real-time job processing
  rpc ProcessJobs(stream ProcessRequest) returns (stream ProcessResponse);
}

service WorkerService {
  rpc Register(RegisterWorkerRequest) returns (RegisterWorkerResponse);
  rpc Heartbeat(HeartbeatRequest) returns (HeartbeatResponse);
  rpc Deregister(DeregisterRequest) returns (DeregisterResponse);
}

gRPC Features

• ⚡ ~28x faster than HTTP (with Redis cache: ~50ms vs 1400ms per operation)
• 🛡️ Automatic plan limit enforcement on enqueue operations
• 📦 Batch operations for higher throughput
• 🔄 Streaming support for real-time job processing
• 🔐 Secure authentication via API key metadata (x-api-key header)

Note: The default gRPC port is 50051. If this port is in use (e.g., by Multipass on macOS), set GRPC_PORT=50052 or another available port. See gRPC Server Guide for details.

gRPC Quick Start

# Test with grpcurl (install: brew install grpcurl)
# List services (reflection enabled)
grpcurl -plaintext localhost:50051 list

# Enqueue a job
grpcurl -plaintext \
  -H "x-api-key: sp_live_your_key" \
  -d '{
    "queue_name": "emails",
    "payload": {"to": "[email protected]"},
    "priority": 0,
    "max_retries": 3
  }' \
  localhost:50051 spooled.v1.QueueService/Enqueue

# Dequeue jobs
grpcurl -plaintext \
  -H "x-api-key: sp_live_your_key" \
  -d '{"queue_name": "emails", "worker_id": "worker-1", "batch_size": 10}' \
  localhost:50051 spooled.v1.QueueService/Dequeue

# Stream jobs (server streaming)
grpcurl -plaintext \
  -H "x-api-key: sp_live_your_key" \
  -d '{"queue_name": "emails", "worker_id": "worker-1", "lease_duration_secs": 300}' \
  localhost:50051 spooled.v1.QueueService/StreamJobs

gRPC Features

When to Use gRPC vs REST

💎 Plan Limits & Tiers

Spooled enforces tier-based limits automatically across all endpoints to prevent abuse and enable fair multi-tenancy.

Available Tiers

Limit Enforcement

Limits are automatically enforced on:

• ✅ HTTP API: POST /jobs, POST /jobs/bulk
• ✅ gRPC API: Enqueue operation
• ✅ Workflows: Counts all jobs in the workflow
• ✅ Schedules: When triggered (manual or automatic)
• ✅ DLQ Retry: When retrying jobs from dead letter queue
• ✅ Workers: Registration and concurrent operations
• ✅ Queues: Creation and configuration
• ✅ Webhooks: Creation and updates

Limit Exceeded Response

When a limit is exceeded, the API returns 403 Forbidden:

{
  "error": "limit_exceeded",
  "message": "active jobs limit reached (10/10). Upgrade to starter for higher limits.",
  "resource": "active_jobs",
  "current": 10,
  "limit": 10,
  "plan": "free",
  "upgrade_to": "starter"
}

For gRPC, the status code is RESOURCE_EXHAUSTED with a descriptive message.

Performance Characteristics

HTTP API (with Redis caching enabled):

• First request (cache miss): ~100ms (includes bcrypt)
• Subsequent requests (cache hit): ~50ms (Redis lookup + DB operation)
• ~28x faster with cache compared to bcrypt-only

gRPC API:

• Batch operations: ~50ms per batch
• Streaming: Real-time job delivery with minimal latency
• Recommended for high-throughput workers

Custom Limits

Enterprise customers can request custom limits via custom_limits in the database:

UPDATE organizations 
SET custom_limits = '{"max_active_jobs": 10000, "max_jobs_per_day": 1000000}'::jsonb
WHERE id = 'org-id';

🚀 Production Deployment

Docker Compose (Recommended for Single Server)

# Download production compose file
curl -O https://raw.githubusercontent.com/spooled-cloud/spooled-backend/main/docker-compose.prod.yml

# Configure environment
cat > .env << EOF
POSTGRES_PASSWORD=$(openssl rand -base64 16)
JWT_SECRET=$(openssl rand -base64 32)
RUST_ENV=production
JSON_LOGS=true
EOF

# Deploy
docker compose -f docker-compose.prod.yml up -d

# Enable monitoring (optional)
docker compose -f docker-compose.prod.yml --profile monitoring up -d

Kubernetes

# Create namespace and secrets
kubectl create namespace spooled
kubectl create secret generic spooled-secrets \
  --namespace spooled \
  --from-literal=database-url='postgres://user:pass@postgres:5432/spooled' \
  --from-literal=jwt-secret="$(openssl rand -base64 32)"

# Deploy with Kustomize
kubectl apply -k k8s/overlays/production

# Or with Helm (coming soon)
# helm install spooled ./charts/spooled -n spooled

ARM64 / Raspberry Pi / AWS Graviton

Images are automatically built for both amd64 and arm64:

# Explicit platform selection
docker pull --platform linux/arm64 ghcr.io/spooled-cloud/spooled-backend:latest

📊 Monitoring

Prometheus Metrics

curl -H "Authorization: Bearer $METRICS_TOKEN" http://localhost:9090/metrics

# Key metrics:
# spooled_jobs_pending       - Jobs waiting
# spooled_jobs_processing    - Jobs in progress
# spooled_job_duration_seconds - Processing time histogram
# spooled_workers_healthy    - Healthy worker count

Grafana Dashboard

Access at http://localhost:3000 (admin/admin) when using --profile monitoring.

Pre-configured dashboards:

• Spooled Overview: Job throughput, queue depth, latency
• Worker Status: Health, capacity, distribution

Distributed Tracing (Jaeger)

# Build with OpenTelemetry support
cargo build --features otel

# Run with tracing
OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4317 ./target/release/spooled-backend

🔒 Security

• Authentication: API keys (bcrypt hashed) or JWT tokens
• Multi-tenancy: PostgreSQL Row-Level Security (RLS)
• Rate Limiting: Per-key limits with Redis (fails closed when configured)
• Webhooks: HMAC-SHA256 signature verification
• Input Validation: All inputs sanitized and size-limited
• SSRF Protection: Webhook URLs validated in production

📚 Documentation

Getting Started

• Quick Start Guide — Get running in 5 minutes
• Getting Started (Laravel users) — Familiar concepts for Laravel developers
• Real-world examples — 5 beginner-friendly examples you can copy/paste

Core Concepts

• Jobs & Queues — Job lifecycle, creation, and processing
• Workers — Building production workers
• Retries & DLQ — Retry configuration and dead letter queue
• Webhooks — Incoming and outgoing webhooks

Reference

• API Usage Guide — Complete REST API reference
• gRPC Server Guide — High-performance gRPC API
• SDKs — Node.js, Python, Go, PHP SDKs
• OpenAPI Spec — OpenAPI 3.1 specification

Operations

• Architecture — System design and data flow
• Deployment Guide — Docker, Kubernetes, production checklist
• Operations Guide — Monitoring, maintenance, troubleshooting

🏗️ Architecture

┌─────────────────────────────────────────────────────────────┐
│                     SPOOLED BACKEND                         │
├─────────────────────────────────────────────────────────────┤
│  REST API (Axum)  │  gRPC (Tonic)  │  WebSocket/SSE         │
├─────────────────────────────────────────────────────────────┤
│              Queue Manager (FOR UPDATE SKIP LOCKED)         │
│              Worker Coordination & Heartbeat                │
│              Scheduler (Cron, Dependencies, Retries)        │
├─────────────────────────────────────────────────────────────┤
│  PostgreSQL 16+   │   Redis 7+     │   Prometheus           │
│  (+ PgBouncer)    │   (Pub/Sub)    │   (Metrics)            │
└─────────────────────────────────────────────────────────────┘

🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing)
3. Commit changes (git commit -m 'Add amazing feature')
4. Push to branch (git push origin feature/amazing)
5. Open a Pull Request

📄 License

Apache License 2.0 - see LICENSE for details.

Built with ❤️ in Rust