1

Image Gateway

chrislusf edited this page 2026-10-05 04:50:46 -07:00

Table of Contents

• On-demand image gateway
• Request interface
• Running the gateway
• Options
• imgproxy configuration
• Access and caching
• CDN notes

On-demand image gateway

weed image is an optional public image endpoint. Original images remain in SeaweedFS. On a cache miss, a separate imgproxy service resizes and encodes the image; results enter a bounded in-process cache. Subsequent requests recheck anonymous source access before reusing results. Cache eviction and process restarts do not affect stored objects. No S3 objects, Filer entries, or application file records are created for processed images.

flowchart LR
  Browser --> CDN
  CDN --> Gateway[weed image]
  Gateway -->|Check source access and revision on every request| S3[SeaweedFS S3]
  Gateway -->|Cache miss| imgproxy
  imgproxy -->|Anonymous source read| S3
  Gateway --> Cache[Bounded memory cache]

The gateway runs on a separate port and does not change existing weed s3, Filer, or Volume services.

Request interface

/image.png?x-oss-process=image/resize,w_640/quality,Q_85/format,webp
/image.png?x-oss-process=image/resize,w_240,h_240,m_lfit,limit_1/format,webp
/image.png?x-oss-process=image/format,jpg/quality,Q_85

This implements a limited subset of OSS image processing parameters, rather than full Aliyun OSS compatibility:

• resize accepts w, h, m_lfit, and limit_1. It preserves aspect ratio and never enlarges the source. One dimension determines the other; two dimensions specify a bounding box. Any unspecified axis uses maxDimension as its bound, including format-only requests; neither output axis can exceed it.
• quality,Q_1 through quality,Q_100 specify absolute quality, defaulting to 85. Aliyun's relative quality q has no equivalent here and returns 400.
• format accepts jpg, jpeg, png, and webp, defaulting to WebP.
• Repeated parameters, unknown operations, cropping, watermarks, animation processing, automatic format negotiation, and other OSS operations are unsupported. PNG output is lossless; quality mainly affects JPEG/WebP. Animated sources use imgproxy's default first-frame behavior.
• A single versionId may select a specific source version. Other query parameters are rejected.
• GET, HEAD, Range, and conditional requests apply to the returned representation. Processed images have independent SHA-256 ETags and byte lengths; validators do not depend on the source modification date. Without a processing parameter the gateway reads the original image, including single byte ranges.
• Original GET and HEAD responses reject unsupported or malformed media types, including HTML and all image/*+xml subtypes. Non-XML image types and octet-stream objects retain their original Content-Type, including parameters. This prevents uploaded executable documents from being served under the gateway's origin. A 304 may omit Content-Type; an explicit unsupported type is still rejected so conditional responses cannot reclassify previously cached bytes.

Running the gateway

Start a separate imgproxy service, then run:

weed image \
  -source=http://s3:8333/public-bucket \
  -imgproxy=http://imgproxy:8080 \
  -ip.bind=0.0.0.0 -port=8334 \
  -cacheCapacityMB=64 -concurrency=8 \
  -maxSourceMB=25 -maxResultMB=10 -maxDimension=4096 -timeout=15s

Options

Option	Default	Description
-source	(required)	Fixed anonymous S3 HTTP(S) source URL, optionally with a bucket path
-imgproxy	(required)	Separate imgproxy HTTP(S) service URL
-ip.bind	127.0.0.1	Listen address
-port	8334	HTTP port
-concurrency	8	Maximum concurrent requests and encoding jobs; excess requests return 429
-maxDimension	4096	Maximum output dimension
-cacheCapacityMB	64	Processed image memory cache in MiB; 0 disables caching
-maxSourceMB	25	Maximum source image size in MiB
-maxResultMB	10	Maximum processed image size in MiB
-timeout	15s	Separate timeout budgets for source metadata, shared encoding, and client writes

source fixes the source HTTP(S) URL, optionally including a bucket path or a bucket domain pointing to S3. With http://s3:8333/public-bucket, a client request for /a/b.png reads http://s3:8333/public-bucket/a/b.png. Both backend URLs must omit credentials, queries, and fragments. The imgproxy URL must be a root URL without a path prefix. Dot segments and backslashes, including repeatedly escaped forms, are rejected to prevent backend path normalization from escaping a fixed bucket prefix. The gateway and imgproxy must reach the same source.

imgproxy configuration

Configure imgproxy limits and isolate the encoder with container or process resource limits. Suggested imgproxy settings:

IMGPROXY_WORKERS=2
IMGPROXY_REQUESTS_QUEUE_SIZE=8
IMGPROXY_MAX_SRC_FILE_SIZE=26214400
IMGPROXY_MAX_SRC_RESOLUTION=25
IMGPROXY_MAX_RESULT_DIMENSION=4096
IMGPROXY_MAX_ANIMATION_FRAMES=1
IMGPROXY_MAX_REDIRECTS=0
IMGPROXY_ALLOWED_PROCESSING_OPTIONS=rs,q,f
IMGPROXY_ALLOWED_SOURCES=http://s3:8333/public-bucket/

These variables apply to imgproxy 3.x. In imgproxy 4.x, some source security limits move to separate source configuration; configure equivalent restrictions according to that version's documentation. File size limits do not replace pixel and memory limits. The gateway does not decode images and cannot enforce encoder-side limits.

Set hexadecimal IMGPROXY_KEY and IMGPROXY_SALT in both services to sign backend requests. The gateway supports one key/salt pair with full SHA-256 signatures; imgproxy must use the default 32-byte signature length. Unsigned requests are only appropriate on an isolated trusted network. Keep signing material out of command lines, public configuration, and logs.

Access and caching

The endpoint only accepts anonymous public reads. It rejects client Authorization, S3 signature parameters, and x-amz-* request headers. Browser cookies are ignored and never forwarded to backends. The gateway does not create administrative credentials or signed source URLs for private objects. Use the existing S3 endpoint for private images.

Every derived request performs an anonymous source HEAD, including cache hits, HEAD, and 304 requests. The encoder uses anonymous GET, so the source must apply the same public-read policy to HEAD and GET. A source 403 or 404 rejects derived requests even if cached bytes remain. Detected source changes return 409.

Concurrent misses for the same result share one encoding job. A cancelled waiter does not cancel work needed by other waiters. Source metadata checks, shared encoding, and client writes each receive a full timeout budget. Errors are never cached. The cache is limited by bytes and 1024 entries, defaults to 64 MiB, and can be disabled with -cacheCapacityMB=0. Each instance has its own cache, empty after restart.

CDN notes

Responses default to Cache-Control: no-cache, allowing downstream storage with mandatory revalidation so source access is checked on every request. A CDN must preserve x-oss-process and versionId and include the complete query in its cache key. If you explicitly set a CDN TTL for permanently public immutable objects, CDN hits bypass the gateway. Revocation or deletion then requires a CDN purge, otherwise access changes only take effect after TTL expiry. Errors use no-store.

Configure HTTPS, CORS, and external rate limits in the existing reverse proxy or CDN. Route only public image GET/HEAD requests to this gateway. Uploads, listings, signed reads, and other S3 operations continue using the existing S3 endpoint.

Introduction

• Quick Start with weed mini
• Simplest S3 Bucket and User Setup
• Components
• Blob Store Architecture
• Getting Started
• Production Setup
• A typical step‐by‐step example
• Benchmarks
• FAQ
• Applications

API

• Master Server API
• Volume Server API
• Filer Server API
• Client Libraries
• SeaweedFS Java Client

Configuration

• Replication
• Store file with a Time To Live
• Failover Master Server
• Erasure coding for warm storage
• EC Bitrot Detection
• Server Startup via Systemd
• Environment Variables

Filer

• Filer Setup
• Directories and Files
• File Operations Quick Reference
• Data Structure for Large Files
• Filer Data Encryption
• Filer Commands and Operations
• Filer JWT Use
• TUS Resumable Uploads

Filer Stores

• Filer Cassandra Setup
• Filer Redis Setup
• Super Large Directories
• Path-Specific Filer Store
• Choosing a Filer Store
• Customize Filer Store

Management

• Admin UI
• Admin UI OIDC
• Worker
• Plugin Worker Scheduling

Advanced Filer Configurations

• Migrate to Filer Store
• Add New Filer Store
• Filer Store Replication
• Filer Active Active cross cluster continuous synchronization
• Filer as a Key-Large-Value Store
• Path Specific Configuration
• Filer Change Data Capture
• Filer Operation Serialization

FUSE Mount

• Mount on Windows
• FIO benchmark
• fstab and systemd mount
• POSIX Compliance
• Distributed POSIX Locks
• P2P reading in weed mount
• Mount over the Internet

WebDAV

SFTP Server

Cloud Drive

• Cloud Drive Benefits
• Cloud Drive Architecture
• Configure Remote Storage
• Azure Blob Storage Authentication
• Mount Remote Storage
• Cache Remote Storage
• Cloud Drive Quick Setup
• Gateway to Remote Object Storage

AWS S3 API

• Amazon S3 API
• Supported APIs vs Minio
• S3 Lifecycle
  • S3 Lifecycle Recipes
  • S3 Lifecycle Operator Guide
  • S3 Lifecycle Monitoring
  • S3 Lifecycle Troubleshooting
  • S3 Lifecycle Architecture
• S3 Lifecycle vs Volume TTL
• S3 Conditional Operations
• S3 CORS
• S3 Object Lock and Retention
• S3 Object Versioning
• S3 RenameObject
• S3 API Benchmark
• S3 API FAQ
• S3 Bucket Quota
• S3 Rate Limiting
• S3 API Audit log
• S3 Nginx Proxy
• Docker Compose for S3
• Image Gateway

S3 Table Bucket

• S3 Table Bucket
• S3 Table Bucket Commands
• S3 Tables Security
• SeaweedFS Iceberg Catalog
• Iceberg REST Catalog API
• Iceberg Table Maintenance
• SeaweedFS Lance Catalog
• Lance Maintenance Worker

Iceberg Integrations

• Spark Iceberg Integration
• Trino Iceberg Integration
• Dremio Iceberg Integration
• DuckDB Iceberg Integration
• Doris Iceberg Integration
• RisingWave Iceberg Integration
• Lakekeeper Iceberg Integration

Lance Integrations

• LanceDB Integration
• Spark Lance Integration
• DuckDB Lance Integration

S3 Authentication & IAM

• S3 Configuration - Start Here
• S3 Credentials (-s3.config)
• OIDC Integration (-s3.iam.config)
• Kubernetes ServiceAccount Authentication (IRSA-style)
• S3 Policy Variables
• S3 Policy Conditions
• S3 Bucket Policies
• Amazon IAM API
• AWS IAM CLI
• weed shell - Shell IAM Commands

Server-Side Encryption

• Server-Side Encryption
• Server-Side Encryption SSE-KMS
• Server-Side Encryption SSE-C

S3 Client Tools

• AWS CLI with SeaweedFS
• s3cmd with SeaweedFS
• rclone with SeaweedFS
• restic with SeaweedFS
• nodejs with Seaweed S3

Machine Learning

• TensorFlow with SeaweedFS
• Distributing AI Model Files for Multi-GPU Loading

HDFS

• Hadoop Compatible File System
• run Spark on SeaweedFS
• run HBase on SeaweedFS
• Run Trino on SeaweedFS
• Hadoop Benchmark
• HDFS via S3 connector

Replication and Backup

• Async Replication to another Filer [Deprecated]
• Async Backup
• Async Filer Metadata Backup
• Async Replication to Cloud [Deprecated]
• Kubernetes Backups and Recovery with K8up

Metadata Change Events

• Filer Metadata Events
• Filer Notification Webhook

Messaging

• Structured Data Lake with SMQ and SQL
• Seaweed Message Queue
• SQL Queries on Message Queue
• SQL Quick Reference
• PostgreSQL-compatible Server weed db
• Pub-Sub to SMQ to SQL
• Kafka to Kafka Gateway to SMQ to SQL

Use Cases

• Use Cases
• Actual Users

Operations

• System Metrics
• weed shell
• Data Backup
• Deployment to Kubernetes and Minikube
• Helm Chart Recipes
• Deployment with seaweed-up
  • seaweed-up Command Reference
  • Cluster Plan Workflow
  • Cluster Plan Inventory Reference
  • Cluster Plan Day 2 Operations

Rust Volume Server

• Rust Volume Server

Advanced

• Large File Handling
• Optimization
• Optimization for Many Small Buckets
• Volume Management
• Tiered Storage
• Cloud Tier
• Cloud Monitoring
• Load Command Line Options from a file
• SRV Service Discovery
• Volume Files Structure

Security

• Security Overview
• Security Configuration
• Cryptography and FIPS Compliance
• Run Blob Storage on Public Internet

Misc Use Case Examples

• UrBackup with SeaweedFS
• Docker Image Registry with SeaweedFS
• SeaweedFS in Docker Swarm
• Words from SeaweedFS Users
• Independent Benchmarks
• Hardware