diff --git a/Image-Gateway.md b/Image-Gateway.md new file mode 100644 index 0000000..1de509a --- /dev/null +++ b/Image-Gateway.md @@ -0,0 +1,160 @@ +# On-demand image gateway + +`weed image` is an optional public image endpoint. Original images remain in +SeaweedFS. On a cache miss, a separate imgproxy service resizes and encodes the +image; results enter a bounded in-process cache. Subsequent requests recheck +anonymous source access before reusing results. Cache eviction and process +restarts do not affect stored objects. No S3 objects, Filer entries, or +application file records are created for processed images. + +```mermaid +flowchart LR + Browser --> CDN + CDN --> Gateway[weed image] + Gateway -->|Check source access and revision on every request| S3[SeaweedFS S3] + Gateway -->|Cache miss| imgproxy + imgproxy -->|Anonymous source read| S3 + Gateway --> Cache[Bounded memory cache] +``` + +The gateway runs on a separate port and does not change existing `weed s3`, +Filer, or Volume services. + +## Request interface + +```text +/image.png?x-oss-process=image/resize,w_640/quality,Q_85/format,webp +/image.png?x-oss-process=image/resize,w_240,h_240,m_lfit,limit_1/format,webp +/image.png?x-oss-process=image/format,jpg/quality,Q_85 +``` + +This implements a limited subset of OSS image processing parameters, rather +than full Aliyun OSS compatibility: + +- `resize` accepts `w`, `h`, `m_lfit`, and `limit_1`. It preserves aspect ratio + and never enlarges the source. One dimension determines the other; two + dimensions specify a bounding box. Any unspecified axis uses `maxDimension` + as its bound, including format-only requests; neither output axis can exceed it. +- `quality,Q_1` through `quality,Q_100` specify absolute quality, defaulting to + 85. Aliyun's relative quality `q` has no equivalent here and returns 400. +- `format` accepts `jpg`, `jpeg`, `png`, and `webp`, defaulting to WebP. +- Repeated parameters, unknown operations, cropping, watermarks, animation + processing, automatic format negotiation, and other OSS operations are + unsupported. PNG output is lossless; quality mainly affects JPEG/WebP. + Animated sources use imgproxy's default first-frame behavior. +- A single `versionId` may select a specific source version. Other query + parameters are rejected. +- GET, HEAD, Range, and conditional requests apply to the returned + representation. Processed images have independent SHA-256 ETags and byte + lengths; validators do not depend on the source modification date. Without a + processing parameter the gateway reads the original image, including single + byte ranges. +- Original GET and HEAD responses reject unsupported or malformed media types, + including HTML and all `image/*+xml` subtypes. Non-XML image types and + octet-stream objects retain their original `Content-Type`, including + parameters. This prevents uploaded executable documents from being served + under the gateway's origin. A 304 may omit `Content-Type`; an explicit + unsupported type is still rejected so conditional responses cannot + reclassify previously cached bytes. + +## Running the gateway + +Start a separate imgproxy service, then run: + +```sh +weed image \ + -source=http://s3:8333/public-bucket \ + -imgproxy=http://imgproxy:8080 \ + -ip.bind=0.0.0.0 -port=8334 \ + -cacheCapacityMB=64 -concurrency=8 \ + -maxSourceMB=25 -maxResultMB=10 -maxDimension=4096 -timeout=15s +``` + +## Options + +| Option | Default | Description | +|--------|---------|-------------| +| `-source` | (required) | Fixed anonymous S3 HTTP(S) source URL, optionally with a bucket path | +| `-imgproxy` | (required) | Separate imgproxy HTTP(S) service URL | +| `-ip.bind` | 127.0.0.1 | Listen address | +| `-port` | 8334 | HTTP port | +| `-concurrency` | 8 | Maximum concurrent requests and encoding jobs; excess requests return 429 | +| `-maxDimension` | 4096 | Maximum output dimension | +| `-cacheCapacityMB` | 64 | Processed image memory cache in MiB; 0 disables caching | +| `-maxSourceMB` | 25 | Maximum source image size in MiB | +| `-maxResultMB` | 10 | Maximum processed image size in MiB | +| `-timeout` | 15s | Separate timeout budgets for source metadata, shared encoding, and client writes | + +`source` fixes the source HTTP(S) URL, optionally including a bucket path or a +bucket domain pointing to S3. With `http://s3:8333/public-bucket`, a client +request for `/a/b.png` reads `http://s3:8333/public-bucket/a/b.png`. Both +backend URLs must omit credentials, queries, and fragments. The imgproxy URL +must be a root URL without a path prefix. Dot segments and backslashes, +including repeatedly escaped forms, are rejected to prevent backend path +normalization from escaping a fixed bucket prefix. The gateway and imgproxy +must reach the same source. + +## imgproxy configuration + +Configure imgproxy limits and isolate the encoder with container or process +resource limits. Suggested imgproxy settings: + +```text +IMGPROXY_WORKERS=2 +IMGPROXY_REQUESTS_QUEUE_SIZE=8 +IMGPROXY_MAX_SRC_FILE_SIZE=26214400 +IMGPROXY_MAX_SRC_RESOLUTION=25 +IMGPROXY_MAX_RESULT_DIMENSION=4096 +IMGPROXY_MAX_ANIMATION_FRAMES=1 +IMGPROXY_MAX_REDIRECTS=0 +IMGPROXY_ALLOWED_PROCESSING_OPTIONS=rs,q,f +IMGPROXY_ALLOWED_SOURCES=http://s3:8333/public-bucket/ +``` + +These variables apply to imgproxy 3.x. In imgproxy 4.x, some source security +limits move to separate source configuration; configure equivalent +restrictions according to that version's documentation. File size limits do +not replace pixel and memory limits. The gateway does not decode images and +cannot enforce encoder-side limits. + +Set hexadecimal `IMGPROXY_KEY` and `IMGPROXY_SALT` in both services to sign +backend requests. The gateway supports one key/salt pair with full SHA-256 +signatures; imgproxy must use the default 32-byte signature length. Unsigned +requests are only appropriate on an isolated trusted network. Keep signing +material out of command lines, public configuration, and logs. + +## Access and caching + +The endpoint only accepts anonymous public reads. It rejects client +Authorization, S3 signature parameters, and `x-amz-*` request headers. Browser +cookies are ignored and never forwarded to backends. The gateway does not +create administrative credentials or signed source URLs for private objects. +Use the existing S3 endpoint for private images. + +Every derived request performs an anonymous source HEAD, including cache hits, +HEAD, and 304 requests. The encoder uses anonymous GET, so the source must +apply the same public-read policy to HEAD and GET. A source 403 or 404 rejects +derived requests even if cached bytes remain. Detected source changes return +409. + +Concurrent misses for the same result share one encoding job. A cancelled +waiter does not cancel work needed by other waiters. Source metadata checks, +shared encoding, and client writes each receive a full timeout budget. Errors +are never cached. The cache is limited by bytes and 1024 entries, defaults to +64 MiB, and can be disabled with `-cacheCapacityMB=0`. Each instance has its +own cache, empty after restart. + +## CDN notes + +Responses default to `Cache-Control: no-cache`, allowing downstream storage +with mandatory revalidation so source access is checked on every request. A +CDN must preserve `x-oss-process` and `versionId` and include the complete +query in its cache key. If you explicitly set a CDN TTL for permanently public +immutable objects, CDN hits bypass the gateway. Revocation or deletion then +requires a CDN purge, otherwise access changes only take effect after TTL +expiry. Errors use `no-store`. + +Configure HTTPS, CORS, and external rate limits in the existing reverse proxy +or CDN. Route only public image GET/HEAD requests to this gateway. Uploads, +listings, signed reads, and other S3 operations continue using the existing +S3 endpoint. diff --git a/_Sidebar.md b/_Sidebar.md index 0a22789..c964735 100644 --- a/_Sidebar.md +++ b/_Sidebar.md @@ -105,6 +105,7 @@ * [[S3 API Audit log]] * [[S3 Nginx Proxy]] * [[Docker Compose for S3]] +* [[Image Gateway]] ### S3 Table Bucket * [[S3 Table Bucket]]