The files leave the volume the database sits on
Every attachment — a document scan, a fuel receipt, a workshop invoice, a
photo of a part's box — has lived inside pb_data, in a directory beside the
SQLite file. One volume held both, so neither could be sized, backed up or
moved without the other. PocketBase can keep those bytes in an S3 bucket
instead, and now it is told to.
Nothing on the way to a client changes, because an attachment was never a
storage URL to begin with: it is fetched from GET /api/{records}/{id}/file,
which re-checks car access and asks PocketBase for the bytes as the service
account. PocketBase streams from the bucket through that same endpoint rather
than redirecting to it, so the web app, the phone and the plugin cannot tell
which side of the switch they are on.
The bootstrap that already creates the collections now writes PocketBase's
files-storage settings too, from PB_S3_*, on every boot and only when they
differ from what is already there — then asks PocketBase to prove it can reach
the bucket, and says so in the log either way. Two asymmetries are deliberate.
A read of the settings masks the stored secret, so a rotation of the secret
alone is invisible from here and needs another PB_S3_* to move with it. And it
never turns S3 back off: files already written to a bucket are reachable only
while PocketBase still points at it, so dropping the configuration would strand
them rather than undo anything.
Each deployment shape is one compose file with an .env example of the same
name, not a base plus an overlay to remember — six of each per folder, for
Docker and Docker-AIO alike: the plain one, .seaweedfs, .s3, and the three prod
twins. The SeaweedFS files run master, volume, filer and gateway as one process
and a one-shot init container beside it, because PocketBase never issues a
CreateBucket and SeaweedFS will not conjure one on first upload. The credentials
do double duty there — the gateway's only identity is also what PocketBase
authenticates with. In the all-in-one that gateway is a second container rather
than a fourth process under supervisord: keeping the object store inside the
image, on the volume the files are being moved off, would have defeated the
point and would have meant rebuilding.
Files uploaded before the switch are not carried across; PocketBase copies
nothing, and both READMEs say so where an operator will read it.
The TLS overlay and its Caddyfile go. The section they served stays, without
them: nothing in the stack terminates TLS any more, so it now names the four
variables to set in front of whichever proxy already does — TRUST_FORWARDED_PROTO
being the one that decides whether a charger is believed about how it arrived.
Unexercised: this was written on a machine without Docker, so the pinned
SeaweedFS image, the bucket-create and the settings write have not been run
against a live stack. The Go side builds, vets and tests clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
tajniak81
co-authored by
Claude Opus 5
Claude Opus 5
parent
181f55a849
commit
9a2a4ab72e
@@ -379,6 +379,22 @@ Copy `.env.example` to `.env` and fill in. Summary:
|
||||
| `OCPP_PUBLIC_URL` | — | canonical `ws(s)://` base to point chargers at |
|
||||
| `PB_BOOTSTRAP` | `true` | run the on-boot schema create/reconcile (leave on across upgrades) |
|
||||
| `DRIVERVAULT_SUPERADMIN_EMAIL` / `_PASSWORD` / `_NAME` | — / — / `Administrator` | first `superadmin`, created on boot when absent |
|
||||
| `PB_S3_ENABLED` | `false` | keep PocketBase's record files in an S3 bucket instead of on its own volume |
|
||||
| `PB_S3_BUCKET` | `drivervault` | the bucket; it must already exist |
|
||||
| `PB_S3_ENDPOINT` | — | e.g. `http://seaweedfs:8333`. No default: in-stack and external gateways are different addresses |
|
||||
| `PB_S3_REGION` | `us-east-1` | SeaweedFS ignores it, PocketBase insists on one |
|
||||
| `PB_S3_ACCESS_KEY` / `PB_S3_SECRET` | — | S3 credentials |
|
||||
| `PB_S3_FORCE_PATH_STYLE` | `true` | path-style bucket addressing; `false` for AWS S3 proper |
|
||||
|
||||
The `PB_S3_*` block is applied by the same on-boot bootstrap that creates the
|
||||
collections, and only when **all** of bucket, endpoint and credentials are set —
|
||||
a half-filled config logs a warning and leaves uploads on the local volume. It
|
||||
writes PocketBase's *Files storage* settings and nothing else: backups stay
|
||||
where they are, and it never turns S3 back *off*, since files already in a bucket
|
||||
are reachable only while PocketBase still points at it. Attachments are served
|
||||
through this server either way ([`internal/api/attachments.go`](internal/api/attachments.go)),
|
||||
so no client can tell the difference. See [`../Docker`](../Docker) for the compose
|
||||
files that set these.
|
||||
|
||||
`PB_URL`, `PB_ADMIN_EMAIL`, `PB_ADMIN_PASSWORD`, `PORT` and `CORS_ORIGINS` are
|
||||
still honoured for older deployments; the modern names win when both are set.
|
||||
|
||||
@@ -51,12 +51,31 @@ func main() {
|
||||
// bad service account, must not stop the panel from coming up so a superadmin
|
||||
// can log in and fix the connection.
|
||||
if cfg.Bootstrap && cfg.AdminConfigured() {
|
||||
// Record files go to S3 only when the whole bucket is described. Asking
|
||||
// for it and leaving half of it blank is a misconfiguration worth saying
|
||||
// out loud, not a reason to point PocketBase at nowhere.
|
||||
var s3 *bootstrap.S3Options
|
||||
if cfg.StorageConfigured() {
|
||||
s3 = &bootstrap.S3Options{
|
||||
Enabled: true,
|
||||
Bucket: cfg.S3Bucket,
|
||||
Region: cfg.S3Region,
|
||||
Endpoint: cfg.S3Endpoint,
|
||||
AccessKey: cfg.S3AccessKey,
|
||||
Secret: cfg.S3Secret,
|
||||
ForcePathStyle: cfg.S3ForcePathStyle,
|
||||
}
|
||||
} else if cfg.S3Enabled {
|
||||
log.Println("WARNING: PB_S3_ENABLED is set but the bucket, endpoint or credentials are incomplete — file storage stays on the local volume")
|
||||
}
|
||||
|
||||
bootCtx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||
if err := bootstrap.Run(bootCtx, client, bootstrap.Options{
|
||||
UsersCollection: cfg.UsersCollection,
|
||||
SuperAdminEmail: cfg.SuperAdminEmail,
|
||||
SuperAdminPassword: cfg.SuperAdminPassword,
|
||||
SuperAdminName: cfg.SuperAdminName,
|
||||
S3: s3,
|
||||
}); err != nil {
|
||||
log.Printf("WARNING: bootstrap failed: %v", err)
|
||||
} else {
|
||||
|
||||
@@ -29,6 +29,24 @@ type Options struct {
|
||||
SuperAdminEmail string
|
||||
SuperAdminPassword string
|
||||
SuperAdminName string
|
||||
|
||||
// S3, when non-nil and Enabled, points PocketBase's record-file storage at
|
||||
// a bucket. Nil is the normal case — a stack started without one of the
|
||||
// SeaweedFS compose overlays keeps its uploads on the pb_data volume.
|
||||
S3 *S3Options
|
||||
}
|
||||
|
||||
// S3Options describes the bucket PocketBase should keep record files in. It
|
||||
// mirrors PocketBase's own settings block one field at a time, so there is
|
||||
// nothing to translate at the wire.
|
||||
type S3Options struct {
|
||||
Enabled bool
|
||||
Bucket string
|
||||
Region string
|
||||
Endpoint string
|
||||
AccessKey string
|
||||
Secret string
|
||||
ForcePathStyle bool
|
||||
}
|
||||
|
||||
// fieldDef is a schema field normalized to a single shape; it is rendered into
|
||||
@@ -190,6 +208,10 @@ func Run(ctx context.Context, client *pb.Client, opts Options) error {
|
||||
if err := ensureSuperAdmin(ctx, client, opts); err != nil {
|
||||
return fmt.Errorf("super-admin: %w", err)
|
||||
}
|
||||
|
||||
if err := ensureFileStorage(ctx, client, opts.S3); err != nil {
|
||||
return fmt.Errorf("file storage: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -433,6 +455,102 @@ func ensureSuperAdmin(ctx context.Context, client *pb.Client, opts Options) erro
|
||||
return nil
|
||||
}
|
||||
|
||||
// --- file storage ----------------------------------------------------------
|
||||
|
||||
// storageSettings is the slice of PocketBase's settings this step owns. The
|
||||
// json names are PocketBase's own (core.S3Config), so the PATCH body is just
|
||||
// this type marshalled back out.
|
||||
type storageSettings struct {
|
||||
Enabled bool `json:"enabled"`
|
||||
Bucket string `json:"bucket"`
|
||||
Region string `json:"region"`
|
||||
Endpoint string `json:"endpoint"`
|
||||
AccessKey string `json:"accessKey"`
|
||||
Secret string `json:"secret,omitempty"`
|
||||
ForcePathStyle bool `json:"forcePathStyle"`
|
||||
}
|
||||
|
||||
// sameExceptSecret compares everything a read of the settings can be trusted
|
||||
// on. PocketBase masks the stored secret, so a rotation of the secret alone is
|
||||
// invisible from here — changing any other PB_S3_* value forces the write, and
|
||||
// so does editing it in the admin UI.
|
||||
func (s storageSettings) sameExceptSecret(other storageSettings) bool {
|
||||
s.Secret, other.Secret = "", ""
|
||||
return s == other
|
||||
}
|
||||
|
||||
// ensureFileStorage points PocketBase's record-file storage at the configured
|
||||
// bucket, and does nothing at all when no bucket was asked for.
|
||||
//
|
||||
// It never turns S3 *off*: files already written to a bucket are only reachable
|
||||
// while PocketBase is still pointed at it, so dropping the overlay leaves the
|
||||
// setting where it is rather than stranding every existing attachment. Moving
|
||||
// back to local storage is a deliberate act in the admin UI.
|
||||
func ensureFileStorage(ctx context.Context, client *pb.Client, opts *S3Options) error {
|
||||
if opts == nil || !opts.Enabled {
|
||||
return nil // not requested
|
||||
}
|
||||
want := storageSettings{
|
||||
Enabled: true,
|
||||
Bucket: opts.Bucket,
|
||||
Region: opts.Region,
|
||||
Endpoint: opts.Endpoint,
|
||||
AccessKey: opts.AccessKey,
|
||||
Secret: opts.Secret,
|
||||
ForcePathStyle: opts.ForcePathStyle,
|
||||
}
|
||||
|
||||
raw, status, err := client.Raw(ctx, http.MethodGet, "/api/settings", nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if status < 200 || status >= 300 {
|
||||
return fmt.Errorf("read settings: status %d: %s", status, raw)
|
||||
}
|
||||
var current struct {
|
||||
S3 storageSettings `json:"s3"`
|
||||
}
|
||||
if err := json.Unmarshal(raw, ¤t); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if current.S3.sameExceptSecret(want) {
|
||||
log.Printf("bootstrap: • file storage already on S3 (%s)", want.Bucket)
|
||||
} else {
|
||||
// Only the s3 block is sent: everything else in the settings — mail,
|
||||
// backups, rate limits — belongs to whoever set it.
|
||||
raw, status, err = client.Raw(ctx, http.MethodPatch, "/api/settings",
|
||||
map[string]any{"s3": want})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if status < 200 || status >= 300 {
|
||||
return fmt.Errorf("apply settings: status %d: %s", status, raw)
|
||||
}
|
||||
log.Printf("bootstrap: ✓ file storage → S3 (%s at %s)", want.Bucket, want.Endpoint)
|
||||
}
|
||||
|
||||
testFileStorage(ctx, client)
|
||||
return nil
|
||||
}
|
||||
|
||||
// testFileStorage asks PocketBase to prove it can actually reach the bucket,
|
||||
// and only says so in the log. A failure here means uploads will fail, but the
|
||||
// server still has to come up — the endpoint is fixable from the panel, and a
|
||||
// stack that refuses to boot cannot be fixed from anywhere.
|
||||
func testFileStorage(ctx context.Context, client *pb.Client) {
|
||||
raw, status, err := client.Raw(ctx, http.MethodPost, "/api/settings/test/s3",
|
||||
map[string]any{"filesystem": "storage"})
|
||||
switch {
|
||||
case err != nil:
|
||||
log.Printf("bootstrap: WARNING: S3 storage test failed: %v", err)
|
||||
case status < 200 || status >= 300:
|
||||
log.Printf("bootstrap: WARNING: S3 storage unreachable (status %d): %s", status, raw)
|
||||
default:
|
||||
log.Printf("bootstrap: ✓ S3 storage reachable")
|
||||
}
|
||||
}
|
||||
|
||||
// --- helpers ---------------------------------------------------------------
|
||||
|
||||
func asBool(v any) bool {
|
||||
|
||||
@@ -94,3 +94,39 @@ func TestSchemaConsistency(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestSameExceptSecret covers the decision ensureFileStorage makes on every
|
||||
// boot: write the settings, or leave them alone. The secret is excluded because
|
||||
// PocketBase masks it on read — comparing it would make every boot a write.
|
||||
func TestSameExceptSecret(t *testing.T) {
|
||||
want := storageSettings{
|
||||
Enabled: true,
|
||||
Bucket: "drivervault",
|
||||
Region: "us-east-1",
|
||||
Endpoint: "http://seaweedfs:8333",
|
||||
AccessKey: "key",
|
||||
Secret: "secret",
|
||||
ForcePathStyle: true,
|
||||
}
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
current storageSettings
|
||||
same bool
|
||||
}{
|
||||
{"identical", want, true},
|
||||
{"masked secret", func() storageSettings { s := want; s.Secret = ""; return s }(), true},
|
||||
{"rotated secret only", func() storageSettings { s := want; s.Secret = "other"; return s }(), true},
|
||||
{"changed endpoint", func() storageSettings { s := want; s.Endpoint = "http://elsewhere:8333"; return s }(), false},
|
||||
{"changed bucket", func() storageSettings { s := want; s.Bucket = "other"; return s }(), false},
|
||||
{"changed access key", func() storageSettings { s := want; s.AccessKey = "other"; return s }(), false},
|
||||
{"still disabled", func() storageSettings { s := want; s.Enabled = false; return s }(), false},
|
||||
{"path style off", func() storageSettings { s := want; s.ForcePathStyle = false; return s }(), false},
|
||||
{"untouched settings", storageSettings{}, false},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
if got := tc.current.sameExceptSecret(want); got != tc.same {
|
||||
t.Errorf("%s: sameExceptSecret = %v, want %v", tc.name, got, tc.same)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -51,6 +51,26 @@ type Config struct {
|
||||
SuperAdminEmail string
|
||||
SuperAdminPassword string
|
||||
SuperAdminName string
|
||||
|
||||
// PocketBase file storage. With S3Enabled set, bootstrap points PocketBase's
|
||||
// "Files storage" at this bucket instead of the pb_data volume; left unset,
|
||||
// uploads stay on disk exactly as they always have. Only *record files* move
|
||||
// — backups are deliberately not touched.
|
||||
//
|
||||
// Nothing here reaches a container unless one of the SeaweedFS compose
|
||||
// overlays is layered on, so an existing stack is unaffected by an upgrade.
|
||||
// S3Endpoint has no default: an in-stack SeaweedFS and one outside it are
|
||||
// different addresses, and guessing either would be worse than not starting.
|
||||
S3Enabled bool
|
||||
S3Bucket string
|
||||
S3Region string
|
||||
S3Endpoint string
|
||||
S3AccessKey string
|
||||
S3Secret string
|
||||
// S3ForcePathStyle keeps bucket names in the path rather than the hostname.
|
||||
// True by default because that is what a self-hosted gateway serves —
|
||||
// virtual-host style would need a DNS entry per bucket.
|
||||
S3ForcePathStyle bool
|
||||
}
|
||||
|
||||
// EnvFile is the .env path (relative to the working directory) that Load reads
|
||||
@@ -66,6 +86,15 @@ func (c Config) AdminConfigured() bool {
|
||||
return c.PocketBaseAdminEmail != "" && c.PocketBaseAdminPassword != ""
|
||||
}
|
||||
|
||||
// StorageConfigured reports whether S3 file storage has been fully specified.
|
||||
// Anything less than all of it counts as "not asked for": a half-filled .env
|
||||
// leaves uploads on the local volume rather than pointing PocketBase at a
|
||||
// bucket it has no way to reach.
|
||||
func (c Config) StorageConfigured() bool {
|
||||
return c.S3Enabled && c.S3Bucket != "" && c.S3Endpoint != "" &&
|
||||
c.S3AccessKey != "" && c.S3Secret != ""
|
||||
}
|
||||
|
||||
// Load reads configuration from environment variables, applying sensible
|
||||
// defaults. A .env file, if present in the working directory, is loaded first.
|
||||
func Load() Config {
|
||||
@@ -86,6 +115,13 @@ func Load() Config {
|
||||
SuperAdminEmail: firstEnv("DRIVERVAULT_SUPERADMIN_EMAIL", "SUPERADMIN_EMAIL"),
|
||||
SuperAdminPassword: firstEnv("DRIVERVAULT_SUPERADMIN_PASSWORD", "SUPERADMIN_PASSWORD"),
|
||||
SuperAdminName: getenv("DRIVERVAULT_SUPERADMIN_NAME", "Administrator"),
|
||||
S3Enabled: boolEnv("PB_S3_ENABLED", false),
|
||||
S3Bucket: getenv("PB_S3_BUCKET", "drivervault"),
|
||||
S3Region: getenv("PB_S3_REGION", "us-east-1"),
|
||||
S3Endpoint: strings.TrimRight(getenv("PB_S3_ENDPOINT", ""), "/"),
|
||||
S3AccessKey: getenv("PB_S3_ACCESS_KEY", ""),
|
||||
S3Secret: getenv("PB_S3_SECRET", ""),
|
||||
S3ForcePathStyle: boolEnv("PB_S3_FORCE_PATH_STYLE", true),
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user