S3 Encryption Gateway

The Problem

Countless applications write data to S3-compatible storage — database backups, log archives, ML training data, CI/CD artifacts — but most of them don't encrypt that data client-side.

The real threat isn't a rogue storage provider. Most people reasonably trust their cloud provider and their server-side encryption (SSE). The much more common and practical risk is a misconfigured IAM policy, overly broad bucket policy, accidentally public ACL, or compromised access key. Any mistake at the IAM or policy layer directly exposes your plaintext data — because without client-side encryption, whoever can reach the bucket can read everything in it.

By adding a cryptographic layer at the gateway, a configuration mistake in your cloud account no longer immediately translates into a data breach. An attacker who gains unauthorized S3 access — through a policy misconfiguration, a leaked key, or any other account-level compromise — only retrieves ciphertext. They would also need to compromise the gateway — which in a typical deployment never leaves your private network.

This is defense-in-depth for object storage: your cloud account's access controls remain your first line of defense; client-side encryption is the second — and it holds even when the first fails.

Beyond misconfiguration risk, there are valid reasons to want an independent crypto layer: regulated environments that require customer-managed keys, multi-tenant shared infrastructure, or simply a preference for not relying solely on provider controls.

Modifying every application to implement client-side encryption isn't realistic. Different tools use different S3 SDKs, different languages, and different upload strategies. Some are closed-source. Some are operators you don't control.

The result: sensitive data sits on object storage, protected only by IAM policies and SSE keys the provider controls — one misconfiguration away from full exposure.

The Solution

The S3 Encryption Gateway is a transparent HTTP proxy that sits between your applications and any S3-compatible storage backend. It encrypts data on the way in and decrypts it on the way out — without changing a single line of application code.

┌─────────────┐          ┌──────────────────────┐          ┌─────────────────┐
│  S3 Client  │──S3 API──▶  Encryption Gateway │──S3 API──▶  S3 Backend    │
│  (any app) ◀──────────│  encrypt/decrypt    ◀──────────│  (AWS, MinIO,   │
└─────────────┘  plain   └──────────────────────┘  cipher  │   Hetzner, ...) │
                 text                               text   └─────────────────┘

Transparent: Point your S3 endpoint URL at the gateway — that's it. No application changes required.

Who Needs This?

If your compliance team, CISO, or data protection officer asks "Are our S3 objects encrypted client-side?" — and the honest answer is "not all of them" — this gateway fixes that in one place, for all applications at once.

Born from Production

We built this gateway because we needed it ourselves. We run cloud platforms for customers and use CloudNativePG as our PostgreSQL operator on Kubernetes. CNPG handles automated backups, WAL archiving, and point-in-time recovery — but it writes those database dumps to S3 unencrypted.

Full database backups in plaintext on object storage wasn't acceptable. But modifying every application that writes to S3 wasn't practical either — the problem wasn't limited to CNPG.

So we built a transparent proxy that solves the problem once, for every application, without touching a single line of application code. The S3 Encryption Gateway is running in production, protecting data across multiple environments and storage backends.

Features

Object Encryption

All objects are encrypted before being sent to the backend and decrypted on retrieval. Encryption is transparent — any S3 client works without modification.

Recommended: Envelope encryption with a locally-held AES-256 or RSA key, or an external KMS (Cosmian KMIP). A random per-object Data Encryption Key (DEK) is wrapped with the Key Encryption Key (KEK) at encrypt time and unwrapped at decrypt time — no key derivation on the hot path. Envelope encryption is 50–76× faster than PBKDF2 600k for single-object uploads and over 70× faster for range reads. See the Encryption Modes Guide for full benchmark tables and a mode comparison.

Password-derived (legacy, simpler deployment): Derives per-object keys from a gateway password via PBKDF2 or argon2id. Requires no key infrastructure — just a single ENCRYPTION_PASSWORD environment variable — but runs key derivation on every request. See the Encryption Modes Guide for throughput numbers and a migration guide if you are switching from password-derived to envelope encryption.

• AES-256-GCM (default) or ChaCha20-Poly1305: Authenticated encryption with per-object keys
• Chunked streaming: Large files are encrypted in chunks with per-chunk IVs, enabling efficient range requests
• Range requests: Fetches only the encrypted chunks covering the requested plaintext byte range
• FIPS-compliant profile: Build with -tags=fips to restrict to AES-256-GCM + HKDF-SHA256 (FIPS-140 approved). Under envelope encryption, DEK wrapping uses AES-256-GCM (AES KEK) or RSA-OAEP/SHA-256 (RSA KEK) — both FIPS-approved. The argon2id KDF is rejected at startup under -tags=fips.

Encrypted Multipart Uploads

Large objects uploaded via the S3 multipart API are encrypted end-to-end. Each upload gets its own key; each chunk gets a deterministic, collision-free IV derived via HKDF-SHA256.

• Per-upload DEK: Fresh 32-byte AES-256-GCM key generated at CreateMultipartUpload
• DEK wrapping: Via the configured KeyManager (Cosmian KMIP, HSM, or built-in password-based PasswordKeyManager)
• Per-chunk IV: HKDF-Expand(SHA-256, dek, salt=sha256(uploadId), info=ivPrefix‖BE32(part)‖BE32(chunk)) — deterministic and collision-free
• AEAD manifest: Encrypted companion object at <key>.mpu-manifest; main object metadata carries a pointer
• Ranged GET across part boundaries: Precise byte-range fetch via EncRangeForPlaintextRange; only the chunks covering the requested plaintext range are fetched and decrypted
• Tamper detection: AES-GCM tag failure on any chunk returns 500 and emits an mpu.tamper_detected audit event; first-chunk tamper returns 500 before any body is written
• State store: Valkey (or any Redis-protocol-compatible store); 7-day TTL; one hash per active upload
• FIPS: AES-256-GCM + HKDF-SHA256 — both FIPS-140 approved (works under -tags=fips)
• Opt-in per bucket via policy; requires Valkey for in-flight state

See ADR 0009 for the full design rationale.

Enabling encrypted multipart uploads

Encrypted multipart uploads require a Valkey (or Redis-protocol-compatible) instance for in-flight state storage. Enable per bucket via policy and configure the state store in the gateway config:

# config.yaml
multipart_state:
  valkey:
    addr: "valkey.internal:6379"
    password_env: "VALKEY_PASSWORD"  # env var name (not the literal password)
    tls:
      enabled: true
      ca_file: "/etc/gateway/valkey-ca.pem"
      cert_file: "/etc/gateway/valkey-client.pem"
      key_file:  "/etc/gateway/valkey-client-key.pem"
    ttl_seconds: 604800  # 7 days — refreshed on every UploadPart
    pool_size: 16

# policy/my-bucket.yaml
id: my-encrypted-bucket
buckets:
  - "my-important-bucket"
encrypt_multipart_uploads: true

All Valkey settings are also available as environment variables (VALKEY_ADDR, VALKEY_TLS_ENABLED, VALKEY_TLS_CA_FILE, VALKEY_TTL_SECONDS, etc.).

encrypt_multipart_uploads defaults to true as of v0.8. Buckets without a matching policy, or policies that omit the field, will use the encrypted multipart upload path automatically. Set encrypt_multipart_uploads: false explicitly in the policy to opt a specific bucket out.

Fail-closed guarantees

The gateway refuses to silently degrade security under any of these conditions:

The dedicated escape hatch for deployments that cannot run Valkey at all:

server:
  disable_multipart_uploads: true  # env: SERVER_DISABLE_MULTIPART_UPLOADS

This enforces a 5 GiB single-PUT ceiling but guarantees all data is encrypted and requires no state infrastructure.

UploadPart memory cap (server.max_part_buffer)

Each UploadPart request is buffered into a seekable in-memory buffer so the AWS SDK V2 SigV4 signer can re-read the body for payload hashing and retries. The default cap is 64 MiB — parts larger than this are refused with HTTP 413 before any backend write occurs:

server:
  max_part_buffer: 67108864  # 64 MiB (default); env: SERVER_MAX_PART_BUFFER

Raise this value if your workload uploads parts larger than 64 MiB. The cap applies to both encrypted and plaintext multipart upload paths. Server.MaxLegacyCopySourceBytes (default 256 MiB, set via server.max_legacy_copy_source_bytes / SERVER_MAX_LEGACY_COPY_SOURCE_BYTES) separately bounds the allocation incurred when copying legacy (non-chunked) encrypted objects with CopyObject or UploadPartCopy.

Deploying Valkey with the Helm chart

The Helm chart bundles an optional Valkey subchart which is off by default:

# values.yaml
valkey:
  enabled: true
  architecture: standalone  # or "cluster" for HA
  auth:
    enabled: false           # enable + mount a secret for production

When valkey.enabled=true, the deployment template auto-wires VALKEY_ADDR to the subchart's <release>-valkey:6379 service. You can also point at an external Valkey cluster via the config.multipartState.valkey values stanza — the two paths are mutually exclusive.

Cost note for Wasabi / Glacier / S3 IA users: The Valkey state store exists precisely to avoid writing state objects to S3 — which on backends with minimum-storage-duration policies (Wasabi: 90 days on Pay-Go; Glacier / Standard-IA / One Zone-IA: 30–180 days) would incur significant phantom-storage charges. Your data objects still land on whichever backend you choose; only the ephemeral per-upload state lives in Valkey.

Envelope Encryption (Recommended)

Envelope encryption removes key derivation from the per-request hot path: a random per-object Data Encryption Key (DEK) is wrapped with a Key Encryption Key (KEK). The KEK is loaded once at startup. This is 50–76× faster than PBKDF2 600k and is the recommended path for all production deployments. See docs/ENCRYPTION_MODES.md for performance benchmarks.

Migrating from password-only? Set encryption.password to your existing password and enable key_manager. The gateway reads the password for objects encrypted before the switch and uses the KEK for all new objects — no data migration required. To re-encrypt existing objects, use the GET-through-gateway → PUT-through-gateway pattern with any standard S3 client. See docs/MIGRATION.md for details.

Three variants are supported:

Self-contained AES KEK (simplest, no external dependencies)

Generate a 32-byte base64 KEK:

openssl rand -base64 32
# example output: pmW3QqWUWCvjYpcsW1ypkUMPuzdF2w5LfR3ligYtK/o=

Option A — YAML config (uses key_source to read the key from an environment variable):

encryption:
  password: "fallback-password-123456"     # for legacy objects encrypted with password
  key_manager:
    enabled: true
    provider: self_contained
    self_contained:
      type: aes
      aes:
        keys:
          - version: 1
            key_source: "env:S3EG_AES_KEK"   # name of the env var holding the base64 key

export S3EG_AES_KEK="pmW3QqWUWCvjYpcsW1ypkUMPuzdF2w5LfR3ligYtK/o="

Option B — environment variables only (no config file; the SELF_CONTAINED_AES_KEYS format is "version=base64:value"):

export KEY_MANAGER_ENABLED=true
export KEY_MANAGER_PROVIDER=self_contained
export SELF_CONTAINED_TYPE=aes
export SELF_CONTAINED_AES_KEYS="1=base64:pmW3QqWUWCvjYpcsW1ypkUMPuzdF2w5LfR3ligYtK/o="

Self-contained RSA KEK (uses existing PKI)

encryption:
  key_manager:
    enabled: true
    provider: self_contained
    self_contained:
      type: rsa
      rsa:
        private_key_source: "file:/run/secrets/kek_rsa.pem"
        key_version: 1

Cosmian KMIP KMS (external KMS with key rotation)

The gateway supports envelope encryption via Cosmian KMIP with key rotation and dual-read windows.

• Envelope encryption: A unique DEK is generated per object, then wrapped with the KMS master key
• Key rotation: The dual_read_window setting allows reading objects encrypted with previous key versions during rotation
• Fallback support: Objects encrypted with the password (before KMS was enabled) can still be decrypted
• Health checks: KMS health is automatically checked via the /ready endpoint

Quick start

1. Start Cosmian KMS:

docker run -d --rm --name cosmian-kms \
  -p 5696:5696 -p 9998:9998 --entrypoint cosmian_kms \
  ghcr.io/cosmian/kms:5.22.0

2. Create a wrapping key via the Cosmian KMS UI (http://localhost:9998/ui) and note the key ID.

3. Configure the gateway:

encryption:
  password: "fallback-password-123456"
  key_manager:
    enabled: true
    provider: "cosmian"
    dual_read_window: 1
    cosmian:
      endpoint: "http://localhost:9998/kmip/2_1"
      timeout: "10s"
      keys:
        - id: "your-key-id-from-cosmian"
          version: 1

Or via environment variables:

export KEY_MANAGER_ENABLED=true
export KEY_MANAGER_PROVIDER=cosmian
export KEY_MANAGER_DUAL_READ_WINDOW=1
export COSMIAN_KMS_ENDPOINT=http://localhost:9998/kmip/2_1
export COSMIAN_KMS_TIMEOUT=10s
export COSMIAN_KMS_KEYS="your-key-id:1"

Protocol options

JSON/HTTP (recommended, tested in CI):

• Full URL: http://localhost:9998/kmip/2_1 or https://kms.example.com/kmip/2_1
• Base URL also works: http://localhost:9998 (path /kmip/2_1 is auto-appended)
• No client certificates required for HTTP; ca_cert recommended for HTTPS

Binary KMIP (advanced, requires TLS):

• Endpoint format: localhost:5696 or kms.example.com:5696
• Requires ca_cert, client_cert, and client_key
• Not fully tested in CI — use with caution

See docs/KMS_COMPATIBILITY.md for detailed documentation. AWS KMS and Vault Transit adapters are on the roadmap.

Compression

Built-in pre-encryption compression was removed in v1.0. For users who need compression, we recommend external composition:

client → s4 (https://github.com/abyo-software/s4) → s3-encryption-gateway → storage

Rate Limiting

Token-bucket rate limiting protects against abuse.

rate_limit:
  enabled: true
  limit: 100      # requests per window
  window: "60s"

Caching

Optional in-memory response cache reduces backend traffic for frequently read objects.

cache:
  enabled: false
  max_size: 104857600     # 100 MB
  max_items: 1000
  default_ttl: "5m"

Audit Logging

Structured audit events for every S3 operation, with configurable sinks.

audit:
  enabled: false
  max_events: 10000
  sink:
    type: "stdout"      # stdout, file, or http
    file_path: ""
    endpoint: ""
    batch_size: 100
    flush_interval: "5s"

Multipart-specific audit events: mpu.create, mpu.part, mpu.complete, mpu.abort, mpu.tamper_detected, mpu.valkey_unavailable.

Monitoring & Observability

Health endpoints

• GET /health — general health status
• GET /ready (alias /readyz) — readiness probe with per-dependency status:

{
  "status": "ready",
  "checks": {
    "kms":    "ok",
    "valkey": "ok"
  }
}

Returns HTTP 503 with status: "not_ready" if any configured dependency fails its health check.

• GET /live — liveness probe
• GET /metrics — Prometheus metrics

Metrics endpoint routing (in priority order):

1. Dedicated metrics port (metrics.addr: ":9090" / METRICS_ADDR) — recommended for Kubernetes; unauthenticated plain HTTP, restrict via NetworkPolicy
2. Admin port fallback — when metrics.addr is empty and admin is enabled, /metrics is served on the admin port (requires bearer auth)
3. S3 port fallback — when both metrics.addr is empty and admin is disabled, /metrics is served on the S3 data-plane port (legacy, no auth)

Prometheus metrics

• HTTP: request counts, durations, bytes transferred
• S3 operations: operation counts, durations, error rates
• Encryption: encryption/decryption counts, durations, throughput
• System: active connections, goroutines, memory usage

Key metric names: http_requests_total, encryption_operations_total, active_connections, goroutines_total, memory_alloc_bytes.

Seven metrics track the multipart-encryption hot path:

TLS

The gateway can terminate TLS directly.

tls:
  enabled: true
  cert_file: /path/to/cert.pem
  key_file: /path/to/key.pem

All responses also include security headers: X-Frame-Options, X-Content-Type-Options, Strict-Transport-Security, Content-Security-Policy, and others.

Admin API

Bearer-authenticated admin endpoints on a separate port:

Quick Start

Prerequisites

• Docker (recommended), or
• Go 1.25+ for local builds

Docker — Envelope encryption (Recommended)

Generate an AES-256 KEK once and keep it secret — this is the key that protects all your encrypted objects:

openssl rand -base64 32
# example output: pmW3QqWUWCvjYpcsW1ypkUMPuzdF2w5LfR3ligYtK/o=

docker run -p 8080:8080 \
  -e BACKEND_ENDPOINT="https://s3.amazonaws.com" \
  -e BACKEND_REGION="us-east-1" \
  -e BACKEND_ACCESS_KEY="your-key" \
  -e BACKEND_SECRET_KEY="your-secret" \
  -e ENCRYPTION_PASSWORD="your-existing-password" \
  -e KEY_MANAGER_ENABLED=true \
  -e KEY_MANAGER_PROVIDER=self_contained \
  -e SELF_CONTAINED_TYPE=aes \
  -e SELF_CONTAINED_AES_KEYS="1=base64:pmW3QqWUWCvjYpcsW1ypkUMPuzdF2w5LfR3ligYtK/o=" \
  -e GW_ACCESS_KEY_1="gateway-access-key" \
  -e GW_SECRET_KEY_1="gateway-secret-key" \
  cloud37io/s3-encryption-gateway:0.10.0-rc3

Replace the example KEK. The value shown in SELF_CONTAINED_AES_KEYS is a documentation placeholder — generate your own with openssl rand -base64 32 and keep it in a secrets manager. Anyone with this key can decrypt your objects.

ENCRYPTION_PASSWORD is the fallback for objects encrypted before you enabled KEY_MANAGER. If you have no existing objects, set it to any strong random value. If you are migrating from password-only mode, set it to your existing encryption password — existing objects will continue to decrypt transparently.

This runs envelope encryption — per-object DEKs wrapped with a local AES-256 KEK. No key derivation on the hot path. See Envelope Encryption above and benchmark results.

Docker — Password-only (simpler deployment, slower)

docker run -p 8080:8080 \
  -e BACKEND_ENDPOINT="https://s3.amazonaws.com" \
  -e BACKEND_REGION="us-east-1" \
  -e BACKEND_ACCESS_KEY="your-key" \
  -e BACKEND_SECRET_KEY="your-secret" \
  -e ENCRYPTION_PASSWORD="your-password" \
  -e GW_ACCESS_KEY_1="gateway-access-key" \
  -e GW_SECRET_KEY_1="gateway-secret-key" \
  cloud37io/s3-encryption-gateway:0.10.0-rc3

Runs PBKDF2-SHA256 600k on every request. No key infrastructure needed — just a single password — but throughput is 50× lower than envelope encryption.

Authentication is required. As of v0.8, every request must include valid AWS Signature V4 or V2 credentials matching an entry in auth.credentials. Unauthenticated requests will receive AccessDenied.

Point any S3 client at the gateway instead of directly at S3:

# Before: direct to S3 (unencrypted)
aws s3 cp backup.sql s3://my-bucket/ --endpoint-url https://s3.amazonaws.com

# After: through the gateway (encrypted)
aws s3 cp backup.sql s3://my-bucket/ --endpoint-url http://localhost:8080

Kubernetes (Helm)

Envelope encryption (recommended):

# Generate a KEK — replace the placeholder below with this output
openssl rand -base64 32
# example output: pmW3QqWUWCvjYpcsW1ypkUMPuzdF2w5LfR3ligYtK/o=

kubectl create secret generic s3-encryption-gateway-secrets \
  --from-literal=backend-access-key=YOUR_KEY \
  --from-literal=backend-secret-key=YOUR_SECRET \
  --from-literal=encryption-password=YOUR_EXISTING_PASSWORD \
  --from-literal=gateway-access-key=YOUR_GATEWAY_ACCESS_KEY \
  --from-literal=gateway-secret-key=YOUR_GATEWAY_SECRET_KEY \
  --from-literal=master-key=pmW3QqWUWCvjYpcsW1ypkUMPuzdF2w5LfR3ligYtK/o=

# values.yaml
config:
  encryption:
    keyManager:
      enabled:
        value: "true"
      provider:
        value: self_contained
      selfContained:
        type:
          value: aes
        aes:
          activeVersion:
            value: "1"
          keys:
            - version: 1
              secretKeyRef:
                name: s3-encryption-gateway-secrets
                key: master-key

Password-only (simpler, slower):

kubectl create secret generic s3-encryption-gateway-secrets \
  --from-literal=backend-access-key=YOUR_KEY \
  --from-literal=backend-secret-key=YOUR_SECRET \
  --from-literal=encryption-password=YOUR_PASSWORD \
  --from-literal=gateway-access-key=YOUR_GATEWAY_ACCESS_KEY \
  --from-literal=gateway-secret-key=YOUR_GATEWAY_SECRET_KEY

helm install s3-encryption-gateway ./helm/s3-encryption-gateway

See the Helm chart documentation for detailed deployment options.

Docker — Local testing with RustFS

RustFS is a lightweight, S3-compatible object store ideal for local development. It is actively maintained (unlike MinIO, which is archived for self-hosted use):

docker network create s3gw-net

# Start RustFS
docker run -d --name rustfs --network s3gw-net \
  -p 9000:9000 -p 9001:9001 \
  -e RUSTFS_ACCESS_KEY=minioadmin \
  -e RUSTFS_SECRET_KEY=minioadmin \
  rustfs/rustfs:latest

# Start the gateway pointing at RustFS
docker run -p 8080:8080 --network s3gw-net \
  -e BACKEND_ENDPOINT="http://rustfs:9000" \
  -e BACKEND_REGION="us-east-1" \
  -e BACKEND_ACCESS_KEY="minioadmin" \
  -e BACKEND_SECRET_KEY="minioadmin" \
  -e BACKEND_USE_PATH_STYLE="true" \
  -e ENCRYPTION_PASSWORD="dev-password" \
  -e KEY_MANAGER_ENABLED=true \
  -e KEY_MANAGER_PROVIDER=self_contained \
  -e SELF_CONTAINED_TYPE=aes \
  -e SELF_CONTAINED_AES_KEYS="1=base64:pmW3QqWUWCvjYpcsW1ypkUMPuzdF2w5LfR3ligYtK/o=" \
  -e GW_ACCESS_KEY_1="gw-access-key" \
  -e GW_SECRET_KEY_1="gw-secret-key" \
  cloud37io/s3-encryption-gateway:0.10.0-rc3

Docker Compose

For local development and testing with a bundled MinIO backend:

cp docker/docker-compose.example.yml docker-compose.yml
cp docker/docker-compose.env.example .env
nano .env  # Edit with your configuration
docker-compose up -d

Access MinIO Console at http://localhost:9001. Gateway API at http://localhost:8080.

Building from Source

make build
# or
go build -o bin/s3-encryption-gateway ./cmd/server
./bin/s3-encryption-gateway

Configuration

Create a config.yaml file (see config.yaml.example for reference):

listen_addr: ":8080"
log_level: "info"

auth:
  credentials:
    - access_key: "YOUR_GATEWAY_ACCESS_KEY"
      secret_key: "YOUR_GATEWAY_SECRET_KEY"
      # proxied_bucket: "optional-bucket-filter"

backend:
  endpoint: "https://s3.amazonaws.com"
  region: "us-east-1"
  access_key: "YOUR_ACCESS_KEY"
  secret_key: "YOUR_SECRET_KEY"
  provider: "aws"
  use_path_style: false  # set true for some S3-compatible providers

encryption:
  password: "fallback-for-legacy-objects"    # only needed when migrating from password-only
  preferred_algorithm: "AES256-GCM"          # or "ChaCha20-Poly1305"
  supported_algorithms:
    - "AES256-GCM"
    - "ChaCha20-Poly1305"

  # --- Envelope encryption (Recommended) ---
  # Remove key_manager entirely to use password-only PBKDF2 derivation instead.
  key_manager:
    enabled: true
    provider: self_contained                  # or "cosmian"
    self_contained:
      type: aes
      aes:
        keys:
          - version: 1
            key_source: "env:S3EG_AES_KEK"    # name of the env var holding the base64 key

  # --- Password-only KDF (legacy, slower) ---
  # Only active when key_manager.enabled is false or key_manager is absent.
  # kdf:
  #   algorithm: "pbkdf2-sha256"  # or "argon2id"
  #   argon2id:
  #     time: 2
  #     memory: 19456
  #     threads: 1

# multipart_state:
#   valkey:
#     addr: "valkey.internal:6379"
#     tls:
#       enabled: true
#     # encryption_password_env: "VALKEY_ENCRYPTION_PASSWORD"
#     # encrypt_state: true  # default: true

# NOTE: built-in compression was removed in v1.0 (V1.0-MAINT-2).
# Use external composition: client → s4 → s3-encryption-gateway → storage

rate_limit:
  enabled: false
  limit: 100
  window: "60s"

cache:
  enabled: false
  max_size: 104857600     # 100MB
  max_items: 1000
  default_ttl: "5m"

audit:
  enabled: false
  max_events: 10000
  sink:
    type: "stdout"      # stdout, file, or http
    file_path: ""
    endpoint: ""
    batch_size: 100
    flush_interval: "5s"

Environment variables are also supported for every setting:

export LISTEN_ADDR=":8080"
export BACKEND_ENDPOINT="https://s3.amazonaws.com"
export BACKEND_REGION="us-east-1"
export BACKEND_ACCESS_KEY="your-access-key"
export BACKEND_SECRET_KEY="your-secret-key"
export BACKEND_USE_PATH_STYLE=false
export ENCRYPTION_PASSWORD="fallback-for-legacy-objects"
export ENCRYPTION_PREFERRED_ALGORITHM="AES256-GCM"
export ENCRYPTION_SUPPORTED_ALGORITHMS="AES256-GCM,ChaCha20-Poly1305"

# --- Envelope encryption (Recommended) ---
export KEY_MANAGER_ENABLED=true
export KEY_MANAGER_PROVIDER=self_contained   # or "cosmian"
export SELF_CONTAINED_TYPE=aes
export SELF_CONTAINED_AES_KEYS="1=base64:pmW3QqWUWCvjYpcsW1ypkUMPuzdF2w5LfR3ligYtK/o="

# --- Password-only KDF (legacy, omit if using envelope encryption) ---
# export ENCRYPTION_KDF_ALGORITHM="pbkdf2-sha256"  # or "argon2id"
# export ENCRYPTION_KDF_ARGON2ID_TIME="2"
# export ENCRYPTION_KDF_ARGON2ID_MEMORY="19456"
# export ENCRYPTION_KDF_ARGON2ID_THREADS="1"
# export VALKEY_ENCRYPTION_PASSWORD_ENV="VALKEY_ENCRYPTION_PASSWORD"
# export VALKEY_ENCRYPT_STATE="true"
export COMPRESSION_ENABLED=false
export RATE_LIMIT_ENABLED=false
export CACHE_ENABLED=false
export AUDIT_ENABLED=false
export TLS_ENABLED=false

Use Cases

Database Backup Encryption

CloudNativePG, Zalando Postgres Operator, and similar database operators write backups directly to S3. Point the backup endpoint at the gateway:

# CloudNativePG Cluster example
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
spec:
  backup:
    barmanObjectStore:
      endpointURL: "http://s3-encryption-gateway:8080"
      destinationPath: "s3://database-backups/"
      s3Credentials:
        accessKeyId:
          name: s3-creds
          key: ACCESS_KEY
        secretAccessKey:
          name: s3-creds
          key: SECRET_KEY

Kubernetes Backup Encryption

Velero and similar backup tools can route through the gateway by configuring the S3 endpoint:

# Velero BackupStorageLocation example
apiVersion: velero.io/v1
kind: BackupStorageLocation
spec:
  provider: aws
  objectStorage:
    bucket: velero-backups
  config:
    s3Url: "http://s3-encryption-gateway:8080"
    region: us-east-1

Log Data Protection

Log aggregators like Loki store potentially PII-containing log data on S3. Route through the gateway to encrypt at rest:

# Loki S3 storage config example
storage_config:
  aws:
    s3: s3://access-key:secret-key@s3-encryption-gateway:8080/loki-logs
    s3forcepathstyle: true

Compliance & Data Sovereignty

The gateway helps satisfy encryption requirements across multiple compliance frameworks:

• ISO 27001 (A.10.1.1) — Cryptographic controls for data protection
• BSI C5 / IT-Grundschutz — Client-side encryption with customer-managed keys
• GDPR Art. 32 — Technical measures for data protection (encryption at rest)
• PCI DSS Req. 3 — Protect stored cardholder data

Architecture

flowchart LR
  C["S3 Client<br/>(awscli, SDKs)"] --> |S3 API| G[Encryption Gateway]
  subgraph G["Encryption Gateway"]
    D["Middleware<br/>(logging, recovery, security, rate limit)"]
    E["Encryption Engine<br/>AES-256-GCM default<br/>ChaCha20-Poly1305"]
    K["Key Manager<br/>(AES KEK / RSA KEK / Cosmian KMIP)"]
    CMP["Compression<br/>(optional)"]
    D --> E
    K --> |wrap / unwrap DEK| E
    CMP -.-> |pre/post| E
  end
  G --> |S3 API| B[("S3 Backend<br/>AWS, MinIO, Wasabi, Hetzner")]

Data Flow (PUT/GET)

sequenceDiagram
  participant Client
  participant Gateway
  participant S3
  Client->>Gateway: PUT /bucket/key plaintext
  Gateway->>Gateway: generate per-object DEK, wrap with KEK
  Gateway->>Gateway: encrypt AES-GCM or ChaCha20-Poly1305
  Gateway->>S3: PUT /bucket/key ciphertext + metadata
  Note over Gateway,S3: metadata stores wrapped DEK, iv, alg, original size
  Client->>Gateway: GET /bucket/key
  alt Range request
    Gateway->>S3: GET optimized encrypted byte range chunked
  else Full object
    Gateway->>S3: GET object
  end
  Gateway->>Gateway: decrypt stream
  Gateway-->>Client: plaintext

Compatible Backends

The gateway works with any S3-compatible storage service. Tested and compatible backends:

Using a backend not listed here? Open an issue to let us know about your experience.

Security Considerations

• Key Encryption Key (KEK): Store securely using a secrets manager (Kubernetes Secrets, HashiCorp Vault, etc.). For self-contained AES KEK, generate with openssl rand -base64 32 and inject via environment variable or mounted secret file. Never commit KEKs to version control.
• Encryption password (fallback / legacy only): Used only for pre-existing objects when migrating to envelope encryption. If running password-only mode, treat as the primary secret.
• Backend credentials: Use IAM roles, service accounts, or secure credential storage
• Network security: Deploy behind TLS termination or enable built-in TLS
• Access control: Restrict gateway access using network policies, firewalls, or API gateways
• Rate limiting: Enable in production to prevent abuse

Roadmap

v1.0

• AWS KMS adapter — native envelope encryption with AWS-managed keys
• HashiCorp Vault Transit — key management via Vault's Transit secrets engine

Shipped in v0.8 (current release)

• Gateway-managed credential store (V1.0-AUTH-1) — every request validated via SigV4/V2 against a gateway-local credential store; use_client_credentials passthrough removed; credentials configured via auth.credentials in config or env vars
• Full security audit hardening — 23 defence-in-depth findings resolved: context.Context propagation through the encryption engine, length-prefixed AAD canonicalization, keyResolver oracle removal, PBKDF2 default raised to 600 000 iterations, decompression bomb protection, presigned URL expiry cap (7 days), rate limiter map cap, admin token zeroization on shutdown, TLS config for audit HTTP sink, policy reload wired into config watcher, and more
• Dedicated metrics port — /metrics is no longer served on the public S3 port. A dedicated unauthenticated listener can be configured via metrics.addr / METRICS_ADDR; when not set, metrics fall back to the admin port (if enabled) or the S3 port (legacy). The Helm chart wires this end-to-end via metrics.port including Service, ServiceMonitor, PodMonitor, and NetworkPolicy (V1.0-SEC-L01)
• XML injection fix — generateListObjectsXML now uses proper encoding/xml marshaling
• Docker healthcheck — replaced wget healthcheck with a native Go binary; FIPS variant included

Shipped in v0.7

• Audit tool (s3eg-cli) — read-only backend-envelope inspection: inspect, verify-key, list-algorithm (V1.0-CLI-2). Replaces the removed s3eg-migrate (now deprecated). For re-encryption use GET-through-gateway → PUT-through-gateway; see docs/MIGRATION.md.
• Configurable PBKDF2 iterations + per-object KDF metadata (V1.0-SEC-H03) — iteration count recorded in object metadata; mixed-iteration deployments decrypt correctly
• Large MPU streaming fixes — ReadTimeout set to 0 (same as WriteTimeout) to prevent timeout kills on multi-hundred-MiB downloads; active write-deadline refresh during long streams; network errors distinguished from tamper on streaming (#135)
• Constant-time credential comparison — timing-safe comparison for all credential checks
• V1.0-SEC-2 / V1.0-SEC-4 / V1.0-SEC-27 / V1.0-SEC-29 — additional hardening fixes

Shipped in v0.6

• Encrypted multipart uploads — per-upload DEK, HKDF-derived per-chunk IVs, AEAD manifest, ranged GET across part boundaries, end-to-end tamper detection, Valkey state store, Helm subchart wiring (ADR 0009)
• Pluggable KeyManager interface (ADR 0004)
• FIPS-compliant crypto profile (ADR 0005, -tags=fips)
• Multipart copy (ADR 0006)
• Admin API with key-rotation state machine (ADR 0007)
• Object Lock / Retention / Legal Hold pass-through (ADR 0008)

Future

• Azure Key Vault and GCP Cloud KMS adapters
• Per-bucket encryption policies
• S3 Encryption Gateway Kubernetes Operator
• Multi-arch images with SBOM and SLSA provenance

See docs/ROADMAP.md for the complete roadmap.

Performance

Per-provider performance baselines and regression gates live in docs/PERFORMANCE.md. The nightly performance-baseline workflow re-runs 19 micro-benchmarks plus per-provider soak tests (MinIO, Garage, RustFS, SeaweedFS) and fails the job on > 15 % throughput regressions or p99 latency growth.

Test Coverage

The project enforces a ≥ 75% statement coverage gate on every PR and push to make coverage-gate. Nightly mutation testing (Gremlins) runs on the critical non-crypto packages. See docs/COVERAGE.md for the exclusion policy, regeneration guide, and mutation testing scope.

Contributing

We welcome contributions! Please see docs/DEVELOPMENT_GUIDE.md for development setup and guidelines.

Areas Where We'd Love Help

• Additional KMS adapters — AWS KMS, Vault Transit, Azure Key Vault, GCP Cloud KMS
• Backend testing — testing with more S3-compatible storage providers
• Interop matrix for encrypted multipart uploads — verify AWS CLI, boto3, aws-sdk-go-v2, minio-go all round-trip correctly at 1 MiB / 8 MiB / 100 MiB / 500 MiB payload sizes against real backends
• Zero-copy streaming encrypt/decrypt — currently UploadPart buffers one part at a time via io.ReadAll (V0.6-PERF-1 follow-up)
• Documentation — guides, tutorials, and integration examples
• Performance benchmarks — throughput and latency measurements across providers

Getting Started

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Run tests and linter (make test && make lint)
5. Submit a pull request

License

MIT License — see LICENSE file for details.

Support

• Issues: GitHub Issues
• Documentation: docs/ directory