mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-09-08 15:41:15 +02:00
505049a4de2f2411b81910de6a7345df3552e00b
100
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Chris Lu
|
505049a4de |
volume: skip directory fsync on Windows, report a failed makeupDiff (#10572)
* volume: skip directory fsync on Windows * ci: run the windows jobs for the whole vacuum path Both windows jobs start the same weed mini cluster, so both exercise the volume server's vacuum path, but only one of them watched a single file in it. Cover the compact, reconcile and load files in both. * volume: report a failed makeupDiff instead of discarding it The cleanup removes assigned to the same err the makeupDiff failure was held in, so an aborted compaction returned nil once both removes succeeded. The master then recorded the vacuum as committed and the volume reloaded against the discarded generation. * volume: correct the fsyncDir comments after the windows skip Both comments described the old shape, where windows fell through to a sync whose error was swallowed. * volume: keep the makeupDiff failure ahead of its cleanup errors A failed remove of .cpd/.cpx outranked the failure that abandoned the compaction, so the caller saw the cleanup error instead of the cause. Log it and return the original, matching the Rust do_commit_compact. A leftover temp file is rolled back by reconcile on the next start. |
||
|
Chris Lu
|
d448e9db7b |
iceberg: withhold the S3 endpoint from credential-vending clients (#10570)
* iceberg: withhold the S3 endpoint from credential-vending clients A client that sends X-Iceberg-Access-Delegation: vended-credentials builds its storage credential out of the LoadTable config and drops the one it was configured with. We vend no credentials, so the endpoint we advertised left DuckDB signing nothing: every metadata and data file came back 403, and its attempt to refresh the empty credential 404ed on stage-created tables. Answer those clients with no config at all so they keep their own credentials. Clients that do not ask for delegation still get the endpoint. * iceberg: mark load responses as varying on the delegation header The FileIO config in a table or view load response now depends on whether the client asked for vended credentials, so a cache between us and the client must key on that header rather than on the URL alone. * test: cover the DuckDB vended-credentials access pattern Runs weed mini with -s3.externalUrl, which is what makes the catalog advertise an endpoint at all, and checks both halves: a plain LoadTable still gets the endpoint, while one asking for vended credentials never gets an endpoint without the credentials to sign with. The DuckDB round trip creates a table from a query and reads it back, which is the flow that failed with 403 on every data file. |
||
|
Chris Lu
|
474a0713b0 |
s3: honor the version history of a directory marker (#10571)
* s3: stop listing a directory marker whose latest version is a delete marker A directory marker is stored as the filer directory itself, so deleting the key "dir/" writes its delete marker into dir/.versions while the directory keeps its mime and stays a key object. Every listing kept reporting the key. Consult that history before treating the entry as a key, and demote it in memory when it is delete-marked so live children still hold the prefix. Also skip the container's own .versions entry while listing inside it: the suffix match read it as the history of a nested object named "", which surfaces as a phantom dir/dir key as soon as a live directory version exists. * s3: a directory marker with version history is not also the latest null version The directory entry behind the key "dir/" is that key's null version, but list-object-versions reported it with IsLatest hardcoded true. After a delete the key came back twice, once as the delete marker and once as a null version, both claiming to be latest. Read the pointer under the directory instead. * s3: resolve directory markers through their version history on GET and HEAD GET and HEAD short-circuit any trailing-slash key straight to the filer directory, so a directory marker kept answering 200 after its delete marker was written. Resolve the key from dir/.versions first when the bucket is versioned: a delete-marked current version answers 404 with x-amz-delete-marker, a named delete-marker version answers 405, and a key with no history keeps today's directory-probe behavior untouched. * s3: re-creating a directory marker retires its delete marker PutObject on a trailing-slash key never looked at the bucket's versioning state, so re-creating a marker after a delete left the latest-version pointer on the delete marker and the key stayed invisible to every versioned read. Point the key back at the directory entry, which is its null version, and drop the null version .versions may still hold — the same two steps a suspended write already takes, now shared. * s3: fail a directory-marker request whose version history cannot be read Every lookup of dir/.versions treated any error as "no history", so a filer hiccup served the directory entry for a key whose current version may be a delete marker, reported a null version as latest over one, and let a PUT report success without retiring the delete marker it was meant to retire. Only a confirmed absence takes the no-history path now. * s3: cancel the directory-marker probe stream instead of abandoning it The probe answers off the first entry and returns, leaving the ListEntries stream open for the life of the parent context. Give it a context of its own. |
||
|
Chris Lu
|
d01ed36118 |
test: cover delete-on-close on the windows mount (#10561)
* test: cover delete-on-close on the windows mount Windows software creates temporaries with FILE_FLAG_DELETE_ON_CLOSE and never deletes them explicitly. The conformance suite showed a file outliving its last handle — an aborted test left its file behind and every later test hit a name collision — but nothing in this suite asks for the flag, because os offers no way to. Skips where the flag is unavailable rather than passing quietly. * test: fail delete-on-close on a real error instead of skipping Skipping on any error meant a refused flag looked the same as a platform that cannot ask for it, so the test could pass by never running. It now skips only on that one sentinel and reports everything else. Also stops printing a nil error when the file is still there after its last handle closed, and checks the closes it was discarding. |
||
|
Chris Lu
|
312cfe5ae1 |
Fix volume.merge corrupting every needle it copies (#10565)
* Give volume.merge the needle size the target actually indexes by needleBlobFromNeedle returned the size Append reports, which is Size(n.DataSize) - payload bytes only. The .dat header, the needle map and WriteNeedleBlobRequest.Size all use n.Size, which additionally covers the flags, name, mime and lastModified fields. Every needle volume.merge copied therefore landed with a too-small size. The target indexed it at that length, so every later read failed the header check in ReadBytes with a size mismatch, and on v3 the fresh AppendAtNs stamp landed NeedleHeaderSize+DataSize+NeedleChecksumSize into the blob - exactly on the flags byte - overwriting flags, name size, mime size and the first mime bytes with the top of a timestamp. Needles came back with flags 0x18, no name, no mime and a phantom TTL parsed from two arbitrary timestamp bytes; the ones that decoded as expired 404 and vacuum would drop them. Since merge rebuilds every replica from the merged copy, no clean replica survives. Return n.Size, which Append fills in as it serializes, matching what the normal write path stores via nm.Put. * Reject needle blobs whose size disagrees with their own header WriteNeedleBlob trusts the caller's size for two destructive things: it is what goes into the needle map, and it is where the v3 AppendAtNs stamp is written inside the caller's buffer. A caller passing the payload-only DataSize convention corrupts both, and nothing surfaces until the needle is read back - by which point every replica may already have been rebuilt from it. Parse the blob's own header and refuse the write when the two disagree. Mirrored in the Rust volume server. |
||
|
Chris Lu
|
b1fecf3b44 |
mount: mark windows files archived and ignore a zero timestamp (#10559)
* mount: mark windows files archived and ignore a zero timestamp Windows synthesises NORMAL when a file reports no attributes at all, which is not the same as ARCHIVE and is what create_fileattr_test checks. Utimens also wrote a zero timestamp through. Windows sends zero for a field it is not setting, and storing it put 1970 in the atime overlay, which then overrode the entry's real time — so a file created a moment ago reported an access time of 1970 whenever the caller asked through an open handle. Reading the path instead went down a different route and looked right, which is why a probe of a fresh file showed nothing wrong. * mount: match the file type by its mask, and only treat the epoch as unset S_IFDIR is part of the multi-bit type field rather than a flag, so masking against it alone also matched a symlink, which shares the bit. A regular file is now identified by the type mask. Rejecting every timestamp at or below zero also rejected a date genuinely before 1970. Only the epoch itself is what Windows sends for a field it is not setting, so that is all that is refused. create_fileattr goes back on the known-failures list: the archive fix works and the test simply moves on to ask for READONLY too, which needs Chflags. Taking it off was premature. * mount: drop the time overlays when an inode is released atimeMap and dirMtimeMap are keyed by inode and were only ever trimmed by a random eviction at capacity. Inodes are derived from the path, so a delete and recreate hands the same number to a different file, which then reported the previous file's access time — a file created a moment ago answering with a time from long before it existed. Cleared when Forget actually releases the inode, not on every decrement: a partial forget still has users. Forget now reports that so callers holding state keyed by the inode know when to drop it. * ci: keep getfileinfo listed while its access time is unexplained Two causes have been fixed and neither closed it, so the honest state is listed-with-a-reason rather than removed in hope. * mount: drop timestamp overlays while the inode table is locked Forget released the inode under the table's lock but cleaned up the atime and dir-mtime overlays after returning from it. Inode numbers are derived from the path, so a lookup arriving in that window is handed the same number back and can store a time that the cleanup then deletes. Run the cleanup at the release point instead, as a callback under the lock. The directory-cache purge stays deferred until after the unlock, where it has to be. Claude-Session: https://claude.ai/code/session_01EgY2QA3iiPtiu6ww3P2EBn |
||
|
Chris Lu
|
50464702d2 |
ci: key the Rust cargo cache on the toolchain that built it (#10568)
Key the Rust cargo cache on the toolchain that built it
The cache key was rust-<Cargo.lock hash> with a bare rust- restore prefix, so
one seaweed-volume/target survived across runner images. cargo tracks its own
inputs but not the runner's C toolchain, so build-script output for C
dependencies is reused even when the system libc underneath it changed.
That is how the Rust jobs got wedged: the cached aws-lc-sys objects reference
__isoc23_sscanf and __isoc23_strtol, symbols glibc only grew in 2.38, while the
jobs link on ubuntu-22.04 with glibc 2.35. Every job died at
rust-lld: error: undefined symbol: __isoc23_sscanf
>>> referenced by bcm.c in archive libaws_lc_sys-*.rlib
with nothing in the tree to explain it, and no amount of re-running helped
because the poisoned entry was hit every time.
Fold the glibc and rustc versions into the key so a toolchain change misses the
cache and rebuilds instead of producing an unlinkable target/.
|
||
|
Chris Lu
|
b452a5e41b |
s3: honor a bucket owner recorded as an identity (#10567)
* s3: resolve a bucket owner recorded as an identity The admin UI and weed shell record a bucket's owner as an identity name in s3-identity-id and never write the account id the S3 API stores alongside it, so such a bucket looked unowned: its ACL owner fell back to the default admin account, and under the default BucketOwnerEnforced ownership every object uploaded to it was stamped with that account instead of the bucket owner. Resolve the identity to its account when no account id is recorded, in the one place both the bucket metadata and the bucket config derive the owner from. * s3: drop the recorded account when the bucket owner is reassigned Changing the owner of a bucket created through the S3 API left its old account id behind, and that outranks the identity when the owner is resolved, so the new owner never took effect for object ownership or the bucket ACL. |
||
|
Chris Lu
|
edaee0e426 |
ci: run each conformance test on its own (#10564)
* ci: run each conformance test on its own Run as one batch with --no-abort, a test that fails part way leaves its files behind and the next one fails creating them, so the report showed a cascade of failures that were really one. The whole rdwr and flush group passes when run alone, and was only ever collateral. Each test now gets its own directory and its own invocation, which costs a process start per test and makes the list mean what it says. * ci: clear a case directory before reusing it -Force creates the directory but leaves anything already in it, so a leftover from an interrupted run would defeat the isolation this exists to provide. * ci: list the one real failure isolation exposed With each test on its own, 31 of 32 pass. The exception is rdwr_mmap_test, which compares mapped bytes against what was written and finds them different — a genuine data mismatch that only appeared once the test could run to completion instead of tripping over a previous one's leftovers. |
||
|
Chris Lu
|
f46b2a1925 |
Stop the filer test helpers from pinning gigabytes of log buffers (#10560)
* log buffer: wake the interval loop on shutdown instead of sleeping through it loopInterval parked in time.Sleep(flushInterval) and only re-checked IsStopping when it woke, so a buffer shut down early kept both loop goroutines - and the PreviousBufferCount+1 slabs of BufferSize they reach - alive for up to a full interval afterwards. Select on shutdownCh against a ticker instead, and give the loops a WaitGroup so a test can observe that they exit. * test: release the filers the server tests build Every helper here left its filer's meta log buffer running, so each test pinned PreviousBufferCount+1 buffers of BufferSize for the rest of the run: ~3.5GB of live heap across the package, which overruns the address space on linux/386 and kills the 32-bit job with an out-of-memory throw. Thread the test through the helpers so the buffer is shut down on cleanup, and shut the subscribe harness's filer down outright - its deletion loop keeps the whole filer reachable otherwise. That harness quiesces its flush path first, since Filer.Shutdown closes the store a flush still in flight would write through. |
||
|
Chris Lu
|
5a5cd15054 |
mount: report . and .. from windows directories (#10556)
* mount: report . and .. from windows directories WinFsp strips the dot entries for the root itself and expects every other directory to report them, the way a real NTFS enumeration does: its dirctl test asserts a subdirectory's first two entries are "." and ".." and that a hundred files enumerate as 102 entries. Dropping them unconditionally is what fails querydir_test. The Go test that guarded the old behaviour went with it: os.File.Readdir filters dot entries itself, so it could never have observed either way. * mount: give the windows dot entries their directory type The readdir fills an attribute block only for real children, so "." and ".." arrived with a zeroed one and were reported with mode 0. Windows refuses to enumerate a directory whose first entry is not marked as a directory, which is the assertion querydir_test fails on with STATUS_OBJECT_NAME_NOT_FOUND. They now carry the type the readdir already knew. The explorer walk also names any unexpected entry rather than only counting, so a dot entry leaking through reads differently from a missing file. |
||
|
Chris Lu
|
89ce6e175d |
ci: run WinFsp's conformance suite against the windows mount (#10555)
* ci: run WinFsp's conformance suite against the windows mount
The FUSE mount is held to pjdfstest with an empty known-failures list;
the Windows mount had 24 hand-written tests. winfsp-tests is what WinFsp
uses to check a filesystem behaves like NTFS, and --fuse-external points
it at ours instead of the bundled memfs, so it is the same bar in the
same shape: anything failing that is not listed is a regression.
It reaches oplocks, security descriptors, POSIX unlink-and-rename and
directory-buffer resumption — the places a Windows filesystem actually
breaks, and none of which the current suite touches.
known_failures.txt starts with the four groups that cannot pass by
construction. The first run will show what else needs listing.
* ci: make the conformance runner fail loudly instead of running empty
The first run reported "0 excluded entries" and then died with
STATUS_DLL_NOT_FOUND, so it never tested anything while looking like a
normal failing run.
winfsp-tests links against winfsp-x64.dll, which the installer puts
somewhere the loader does not search, so the WinFsp bin directory goes on
PATH. A missing or empty known-failures list is now an error rather than
a silent run with nothing excluded, which would read as a clean sweep
with no known failures. ${env:ProgramFiles(x86)} needs the braces, and a
mount point without a trailing separator makes Join-Path build a path
relative to the drive's current directory rather than its root.
* ci: read winfsp-tests failures from its report, and list the real ones
The first run exited zero with 30 of 50 tests reporting KO, and the job
went green: --no-abort keeps the suite going past a failure and the exit
code stops reflecting them, so trusting it meant the check could not fail.
The report is now parsed for KO lines and each one named in the error.
known_failures.txt is populated from that run rather than guessed. The
groups are real gaps, not suite quirks: cached and overlapped IO fails as
a block, delete-while-open has no pending state, Windows file attributes
and creation time are not round-tripped, and directory enumeration does
not resume from a marker.
* ci: stop excluding the extended attribute tests
Forwarding landed, so the group runs instead of being taken on trust —
which is the only coverage it has had.
|
||
|
Chris Lu
|
b8cba2982c |
mount: tell windows about changes made elsewhere (#10553)
* mount: tell windows about changes made elsewhere Nothing invalidates a Windows client's cache from this side, so a file created or removed by another mount, the S3 gateway or the filer API stayed invisible in Explorer until the user refreshed by hand. The mount already receives those events; they just had nowhere to go. WFS gains a listener for every applied metadata event, and on Windows that turns into the WinFsp notification for the path. A rename reports both ends, since the destination's own event may never arrive when it falls outside this mount. * mount: report a removed directory as a directory Entry is nil once a path is vacated, so asking it whether the thing that went away was a directory always answered no and every removal was reported as a file. Windows watches the two through different filters, so a folder removed elsewhere never refreshed. The invalidation now carries what used to be there, which the event already knew and simply was not passing on. * mount: report a rename destination once The event stream already carries a second invalidation describing the new path, so reporting RenamedTo here sent the destination twice — and always as a create, so a moved directory arrived as a create followed by a mkdir. |
||
|
Chris Lu
|
a0e278f86f |
mount: forward extended attributes on windows (#10554)
weed/mount implements all four xattr operations and the filer stores the values, but the Windows adapter overrode none of them, so cgofuse's defaults answered every call with 'not implemented'. WinFsp advertises extended attribute support either way, because cgofuse registers the callbacks unconditionally, so applications were told the volume has them and then refused on every use. Attributes written from Linux were invisible from Windows. Untested in CI: exercising Windows extended attributes needs the native NtSetEaFile path rather than anything in os or PowerShell. |
||
|
Chris Lu
|
e377149d39 |
mount: support mounting on Windows through WinFsp (#10536)
* mount: add the WinFsp filesystem adapter WinFsp speaks a path-based FUSE dialect; weed/mount implements the inode-based raw protocol the Linux kernel uses. This translates between them so Windows runs the same filesystem code as everywhere else rather than a second implementation: paths resolve to inodes one Lookup at a time, and the raw operations run unchanged underneath. Errno translation is spelled out rather than passed through. Go numbers Windows errnos as offsets from APPLICATION_ERROR, so the raw value would mean something unrelated by the time WinFsp read it. Hard links return ENOSYS since WinFsp has none, and byte-range locks stay with its kernel driver rather than the mount's lock table. Not reachable from the mount command yet. * mount: build the winfsp errno table with explicit precedence Platforms alias errnos differently: freebsd has no ENODATA and linux makes ENOATTR the same value as it. A map literal with colliding constant keys does not compile, so build the table and let the first entry win, keeping the general codes their own meaning. * mount: wire the winfsp adapter into the mount command RunMount was one function doing filer setup, mount-point preparation and serving. The setup is the same everywhere, so it moves to mount_common.go and each platform keeps only what differs. Windows differs mostly in the mount point: WinFsp wants a drive letter or a path that does not exist yet, so none of the unix preparation applies, and a bad one is worth rejecting up front because WinFsp reports failure as a bare false. Adds -windows.caseInsensitive for software that expects Windows naming rules. * ci: mount on windows and exercise it Builds weed.exe, installs WinFsp, starts a cluster, mounts S: and runs a test suite against it: round trips at several sizes, offset writes, rename, delete, nested directories, concurrent writers, and a directory wide enough to stand in for the case that prompted this. Nothing else here can run the Windows mount, so without this the adapter is only known to compile. * ci: build the windows mount without cgo The runner has MinGW, so cgo is on by default and cgofuse compiles its cgo variant, which needs WinFsp's headers. The nocgo variant loads the DLL at run time and is what the released weed.exe uses. * mount: make the winfsp path splitting portable and test it resolve and resolveParent had the splitting inline in a windows-tagged file, so the cases that matter most there — both separators, empty and dot components, the root having no parent to create in — could not be tested on any runner that builds this. * test: check the windows mount persists across a remount Reading a file back through the same live mount proves nothing about durability; the answer can come from the mount's own caches. Write the fixtures, confirm the filer serves them with the mount out of the path, then re-read after a teardown and remount. * test: cover the windows mount operations that had none Truncate, append, chtimes and the hard-link refusal were implemented but never exercised, and the errno table was only unit-tested for mapping, never end to end. Adds names that have to survive the UTF-16 boundary, rename over an existing target and across directories, and concurrent handles on one file rather than one file each. * ci: dial the filer over ipv4 and run the persistence phases localhost resolves to ::1 first on windows and the cluster binds ipv4 only, so the mount's grpc dial was refused while the http readiness probe passed by falling back to ipv4. * ci: pin the cluster to loopback and probe ports by connecting weed mini advertises the runner's LAN address and binds filer grpc there, so the mount's dial to 127.0.0.1:18888 was refused while http answered. The readiness probe also passed with nothing on 18888: Test-NetConnection reported success for a port that then refused a connection, so it now opens a socket instead. * ci: report listening ports before mounting The readiness probe connects to the filer grpc port and the mount is then refused on it, which cannot both be true; print the actual state. * ci: run the cluster, mount and tests in one step The runner tears down a step's process tree when its shell exits, so the cluster started in an earlier step was already gone: the readiness probe passed against a live filer, the step ended, and the mount then found nothing listening. A diagnostic step reported no weed.exe at all. Everything that needs those processes alive now shares a step. * mount: key windows file io on the handle, not the path Read and Write walked the path on every call to fill in a NodeId the raw filesystem never reads: both look the file up by handle. Under eight writers creating files in one directory the walk transiently missed and the write failed with ENOENT before reaching the filesystem at all. Same for flush, fsync and the release calls. O_EXCL now fails on an existing name instead of taking it over, and Symlink is refused: the entry is easy to create but WinFsp only follows it once the reparse point is wired up, so it read back as an empty file. * mount: translate cgofuse open flags for windows cgofuse reports MSVC's numbering and the raw filesystem tests Go's, so only the access mode and O_TRUNC lined up: O_EXCL arrived as O_APPEND and O_CREAT as nothing at all. Also report which handle a failed write was using, to tell a handle that was never issued from one released while still in use. * mount: report which step of a windows create failed A concurrent create fails with ENOENT and the path walk, the parent lookup and the create itself are indistinguishable from the caller. * ci: send weed logs to stderr on windows glog writes to its own files by default, so the mount's own error output never reached the redirected log. Its flags are global and have to come before the subcommand. * mount: resolve known paths from the inode table on windows Every create walked the parent chain with a filer lookup per component. With eight writers creating files in one directory that is hundreds of concurrent lookups of the same parent, and lookupEntry reports an authoritative ENOENT when the directory is cached, the entry is not in the cache and the inode table has no record — a window a concurrent refresh can open for a directory that plainly exists. A path the mount already tracks now resolves straight out of that table. * test: sync the windows persistence fixtures before closing The mount is killed rather than unmounted, so anything still queued for flush is legitimately lost and the test was measuring crash durability while calling it persistence. A 9MB file lost four chunks that way. * mount: keep the lookup refresh on the target path Resolving a tracked path straight from the inode table skipped Lookup, which is also what refreshes the entry: a truncate then read back the pre-truncate size. Only the parent chain takes the shortcut now, which is where the concurrent creates were racing anyway. * mount: log every windows resolve failure Open suppressed ENOENT and Getattr logged nothing, which hid the two callbacks that can report a missing file during a create. * mount: drop dot entries from windows directory listings readdir reports "." and ".." for the kernel, but Windows enumerates a directory without them and displays whatever it is handed, so a folder of 200 files listed 202. Go's ReadDir filters them, which is why only the PowerShell walk caught it. * mount: flush queued writes when windows mount is interrupted The signal handler exits the process the moment its hooks return, so the WaitForAsyncFlush after Serve never ran on ctrl-c and queued writes were dropped. * mount: let windows mount over an empty directory WinFsp turns a directory mount point into a reparse point, which NTFS allows on an empty directory and refuses on a populated one. The check rejected every existing directory, so the ordinary habit of creating the mount point first failed with a message saying it should not exist. CI now mounts over a pre-created directory and writes through it. * ci: run the windows mount check on any pull request It is the only thing that exercises the Windows mount, so restricting it to pull requests based on master skipped it for stacked ones. Replaces the branch name that was pushed to trigger it. * mount: do not log a missing windows entry as an error Windows probes for entries that do not exist as a matter of course, so ENOENT from getattr and open is an answer rather than a fault and would have filled the log. * mount: take the fast path for parent chains in every windows resolve Narrowing it to resolveParent left Getattr and Open re-walking the parent with a filer lookup per component, and those are what Windows calls before a create: eight writers in one directory still raced a meta cache refresh there. Only the final component needs the Lookup refresh. The pass that suggested otherwise came from a run five times slower than the failing ones, where the race had no room to appear. * mount: drop the windows path resolution shortcut Resolving from the inode table skipped the Lookup that refreshes an entry, and a truncate then read back its old size. Applying it only to the parent chain kept truncate correct but left concurrent creates failing, and applying it to the final component too inverted that. The two cannot both be satisfied this way, so this returns to looking up every component and leaves the concurrent create failure open. * mount: fall back to the open handle when a deferred entry is evicted A create that defers the filer write leaves the entry only in the local cache. Creating many files at once pushes the directory past the hot threshold and evicts it, taking that placeholder with it, so a lookup went to the filer, found nothing, and reported a file that plainly exists as missing. The handle still holding the unflushed entry is authoritative for it. Caught by concurrent creates over a Windows mount, which resolves a path on every call rather than relying on a kernel dentry cache. * mount: let cgofuse resolve to the version the module graph requires rclone already depends on cgofuse at a newer commit than the v1.6.0 pin, so readonly builds refused the go.mod until it matched what MVS picks. The interface and flag values the adapter uses are unchanged there. * mount: wait for a pending async flush before looking up on the filer Open, unlink and rename already wait, but a plain lookup went straight to the filer and read pre-close metadata: truncate a file, close it, and a path probe during the flush window reported the old size. The kernel attr cache hides this on linux; a front end that resolves paths on every operation hit it directly. * mount: reject a umask wider than the file mode it becomes ParseUint allowed 64 bits and the result is narrowed to os.FileMode, which is 32, so an out-of-range umask truncated silently instead of being reported as unparseable. * mount: address review findings on the windows mount WaitForAsyncFlush closed its channel unconditionally and shutdown reaches it from both the interrupt hook and the path that resumes after serving, so a ctrl-c could panic on a second close. The deferred-entry fallback read an open handle's entry without its lock, which is what the other two readers of that field take so FromPbEntry does not walk the chunk slice mid-append. The async-flush wait also sat ahead of the meta cache, making every stat of a recently closed file queue behind uploads; it belongs just before the filer is consulted. Windows entries were persisted as uid 0: the raw filesystem stores InHeader's owner and the adapter left it zero. They now carry the identity the mount was started with. The errno table used Linux numbering while cgofuse decodes MSVC's, so ENAMETOOLONG arrived as EDEADLK and five others were likewise wrong; a windows test pins each value to cgofuse's own constant. Also: break the filer handshake loop on success rather than always running ten rounds, accept a drive letter written S:\\, report a missing WinFsp instead of panicking, keep commas out of the volume label, and drop -windows.caseInsensitive, which told WinFsp the mount folds case while lookups stayed exact. * mount: return windows lookup references so the inode table stays bounded Every operation that hands back an EntryOut grants a reference the Linux kernel returns with FORGET. WinFsp has no FORGET, so the adapter took one per path component per call, plus one per child of every readdirplus, and never gave any back: inodeToPath grew for the life of the mount. Walking the 200k-file directory this exists for stranded 200k references. The adapter now plays the part the kernel plays. Each resolution releases what it took, and an open handle keeps the reference for its inode until Release, counted because the raw filesystem reuses one handle for repeated opens. Holding it is not optional: completeAsyncFlush skips the metadata flush when the saved path no longer maps to the inode, so releasing early would lose a close's metadata. Also stops persisting the display owner. -o uid=-1 makes WinFsp report the calling user whatever we say, but the value handed to the raw filesystem is written to the filer, and 4294967295 is what every other client would read. -windows.uid and -windows.gid set what is recorded. * mount: fix windows behaviours the reference implementations guard against WinFsp has no ro option — it discards the flag and leaves the volume writable — so -readOnly accepted writes and deletes. The refusal now happens in the operations themselves. Windows sends times around its own 1601 epoch, which arrive as a large negative second count; casting them through stored a year-1601 timestamp that every other client then read. Those are now left alone. rclone carries the same guard. Chown returned ENOSYS, and WinFsp passes a chown failure straight out of SetSecurity, so Explorer's Security tab and icacls failed for edits that were not about ownership. It now accepts and discards. Only create and mkdir presented a caller; the rest sent uid 0, which hasAccess treats as root, so deletes and renames skipped the permission check that creates got. Every operation presents the same identity now. A drive letter written S:\ reached WinFsp unnormalised, which recognises a drive only as exactly two characters and then failed as a directory path. A test also pins the open flag translation, since swapping O_EXCL and O_TRUNC would turn 'fail if it exists' into 'truncate it'. * mount: answer windows getattr and truncate from the open handle WinFsp keeps the path a handle was opened with and never updates it when the file is renamed, so resolving the path again fails on a handle that is still perfectly valid — the ordinary write-temp-then-rename save pattern. The handle already knows its inode, which also removes a full path walk from two operations WinFsp calls constantly. Readlink on the root now refuses. WinFsp probes there to decide whether the volume has symlinks and enables them unless it fails, and with them on it resolves a path a component at a time, each one reaching us as its own walk — all for a feature Symlink already refuses. * mount: require the windows mount directory not to exist WinFsp creates the directory itself with FILE_CREATE and removes it when the filesystem goes away, so an existing one — empty or not — fails with "mount point in use". Allowing an empty directory was wrong, and the CI check that appeared to prove otherwise was the vacuous one: listing a plain directory succeeds whether or not anything is mounted on it, so the step passed while the mount had failed and the writes went to local disk. That check now waits for the reparse point, which is what caught this. * mount: apply review comments on the windows mount -windows.uid and -windows.gid reached the adapter but not the filesystem parameters, which is what carries the owner written to the filer, so the flags changed nothing. Readdir re-resolved the path while Getattr and Truncate answer from the handle; a directory renamed during an enumeration then failed on the stale path WinFsp still holds. Utimens now honours UTIME_OMIT instead of writing whatever came with it. * mount: tag the unix-only lock tests away from windows The production lock files were tagged when the package was made to build on windows, but the tests that exercise them were not, so anything that compiles tests for windows still failed on syscall.F_WRLCK. * ci: vet the mount tests for each target too Only compiling the non-test build let an untagged test keep a per-OS syscall constant without anything noticing. |
||
|
Chris Lu
|
4992ac1ca9 |
mount: keep the xattr flag constants off freebsd (#10552)
* mount: keep the xattr flag constants off freebsd x/sys/unix has no XATTR_CREATE or XATTR_REPLACE there, and weedfs_xattr.go is already tagged away from freebsd for that reason. Putting them in a !windows file dragged them back in, so master stopped building for freebsd. * ci: cross-compile freebsd and darwin too The windows-only check missed a freebsd break in the very file it was added to guard, because nothing else on a pull request compiles them. |
||
|
Chris Lu
|
c191b2fe01 |
iceberg: let clients select their table bucket as the catalog warehouse (#10549)
* iceberg: accept bare bucket names and ARNs as the catalog warehouse Only s3://<bucket>/ was recognized. A warehouse spelled as a bare table bucket name or as the s3tables bucket ARN -- the two forms users reach for first, the latter being what AWS S3 Tables itself takes -- was silently dropped, so every call landed on the default "warehouse" bucket and failed with "table bucket warehouse not found". * iceberg: report a missing table bucket as 404, not 500 Pointing a client at a table bucket that does not exist -- which every client with no warehouse set does, since the default bucket "warehouse" rarely exists -- returned InternalServerError with a message naming a bucket the client never asked for. Answer 404 and say how to select one. * admin: show the warehouse in the PyIceberg example The example connected without one, so it always resolved to the default table bucket and every client that copied it failed on the first call. * test: pin bearer auth against a table bucket that exists The subtest called the catalog with no warehouse and accepted 500 as proof that auth had passed, since the default bucket does not exist. A missing table bucket now answers 404, which the test read as an auth failure. Give it a real table bucket so only 200 passes. * test: assert the missing-bucket guidance reaches the client The status and error type were checked but not the message, which is the part of the mapping that tells a user how to select a table bucket. * test: encode the warehouse query value The ARN case pasted raw colons and slashes into the query string. Go's parser tolerates them, so the test passed without modelling how a client actually sends the request. |
||
|
Chris Lu
|
63a180ef75 |
telemetry: sync the server module with the client_golang bump (#10551)
The server module has its own go.mod and replaces the root module from ../.., so bumping prometheus/client_golang in the root leaves this one pinned below what the replacement needs and 'go build' refuses to run. |
||
|
Chris Lu
|
88fd2d1be8 |
telemetry: stack volume servers per cluster, drop the total disk usage chart (#10550)
* telemetry: stack volume servers per cluster over time The fleet-wide server count says how many volume servers reported, but not who they belong to. Carry per-cluster counts in /api/cluster-sizes and draw them the same way as cluster sizes, sharing one cluster ranking so a cluster keeps its colour across both stacks. * telemetry: drop the total disk usage chart from the dashboard The stacked cluster sizes chart right below it has the same fleet total as its stack height, plus the per-cluster breakdown. /api/metrics still serves the aggregate for anyone graphing it elsewhere. |
||
|
Chris Lu
|
cc2775d9f2 |
s3: register an identity's inline account instead of collapsing it into admin (#10548)
* s3: register an identity's inline account instead of collapsing it into admin Credential stores persist an account inline on the identity and never emit a top-level accounts list, so every user created through the IAM API or the admin UI with an email hit the "non exist account ID" branch and was given the shared admin account. Distinct users then presented the same owner id, so ownership checks could not tell them apart and each passed for the others' buckets. Treat an id missing from the account map as undeclared rather than invalid: register it, keeping an email another account already claimed. Both load paths now resolve the account through one helper. * s3: refresh an undeclared account from the identity that carries it The merge path starts from the live account cache, so an identity upserted with the same account id but a new email or display name kept the cached copy: the new address never reached the email index and the replaced one still resolved. Changing a user's email through the admin UI takes exactly that path. An account registered from an inline block is only described by the identity carrying it, so refresh it and move its email claim. Accounts from a top-level list and the predefined defaults are marked declared and stay authoritative. * s3: let an account reclaim an email once its holder moves away Two identities can carry the same email, and the second to load leaves the lookup with the first. Returning early when the incoming metadata matches the cached account meant the loser never re-ran the claim, so an address freed by the holder's update resolved to nobody until the loser itself changed. Re-index on the unchanged path, which is a no-op while another account still holds the address. |
||
|
Chris Lu
|
a9de90ae29 |
test: wait for every queued flush before deleting the log files (#10546)
TestSubscribeLoop_FlushProvenGapSkipsToRetained deleted the log files once the eviction watermark's own flush had landed, while the windows sealed after it were still queued. Those flushes then wrote their files back, and the subscriber served them from disk instead of taking the gap-skip path, so the windows whose files really were gone came out missing. Wait through the last sealed window instead. |
||
|
Chris Lu
|
4f692bf9c3 |
mount: build the package on windows (#10535)
* mount: drop the unused go-fuse fs package dependency WFS embedded fs.Inode but never used any of its methods, and the only other reference was RENAME_EXCHANGE, a constant sitting next to three literals. Removing both drops fs and five internal packages from the mount build graph. * mount: build the package on windows Windows has no fcntl lock types, no O_ACCMODE and no x/sys/unix, so a handful of constants kept weed/mount pinned to unix even though the code using them is portable in-memory logic. Route them through per-OS shims and give setBlksize a windows no-op. The POSIX lock table now compiles on windows but stays unreachable: WinFsp resolves byte-range locks in its own kernel driver, so nothing will feed it there. go.mod points at a go-fuse branch commit and needs repinning to a release tag once that lands. * ci: cross-compile for windows Nothing caught the unix-only constants creeping into weed/mount until a release build failed. * mount: let readdir feed a sink instead of the kernel buffer doReadDirectory wrote directly into fuse.DirEntryList, which is the kernel's wire format. A front end that is not the kernel would have to pack entries only to parse them straight back out. Route it through DirEntrySink instead. ReadDir and ReadDirPlus pass the reply buffer, so nothing changes for the FUSE server. * mount: pin go-fuse v2.9.4 for the windows build |
||
|
Chris Lu
|
529ffa5c86 |
mount: drop the unused go-fuse fs package dependency (#10534)
WFS embedded fs.Inode but never used any of its methods, and the only other reference was RENAME_EXCHANGE, a constant sitting next to three literals. Removing both drops fs and five internal packages from the mount build graph. |
||
|
Chris Lu
|
e696c2585e |
s3 remote: honor s3.support_tagging in UpdateFileMetadata (#10532)
* s3 remote: honor s3.support_tagging in UpdateFileMetadata The write path already skips tagging when the remote is configured without tagging support, but the metadata-update path sent PutObjectTagging or DeleteObjectTagging unconditionally. On remotes that reject tagging requests, any metadata update failed -- including updates with no tags at all, which land on the DeleteObjectTagging branch. * s3 remote: drop the never-read supportTagging field Every maker set it, nothing read it: the tagging decision is made from conf.S3SupportTagging. Keeping a field that looks like the switch but is not invites exactly the inconsistency the previous commit fixed. |
||
|
Chris Lu
|
0d7173a029 |
remote storage: actually delete objects when a directory is removed (#10531)
* remote storage: actually delete objects when a directory is removed On object-store backends RemoveDirectory returned nil without doing anything, so a directory delete synced to the remote as a successful no-op and the objects under that prefix stayed there forever. Nothing surfaced the divergence: the sync logged rmdir, advanced its offset, and the local namespace looked clean. Deleting a bucket-level directory on a filer store that can drop a whole bucket emits no per-child delete events at all, so the single rmdir event was the only chance to clean up the remote. Each backend now lists the prefix and deletes what it finds: S3 in DeleteObjects batches of one listing page, GCS and Azure per object. The prefix always ends with a slash so a sibling like dir2 survives deleting dir, and errors propagate so a failed delete is retried instead of silently skipped. A directory that maps to the bucket root is left alone: wiping every object in the bucket from one namespace event is too destructive, and bucket removal already has its own path. * gcs remote: wrap the per-object delete error The listing error in the same function already wraps, so the delete error should stay inspectable with errors.Is as well. * s3 remote: name the empty-listing test for what it checks The prefix in that test is a normal directory; what is empty is the listing. The bucket-root guard has its own test. * s3 remote: report the scope of a failed delete batch A DeleteObjects response can carry per-key errors for up to a thousand keys. Surfacing only the first hid how much of the batch failed, and surfacing all of them would build an unbounded error string, so report the count with the first failure as the sample. |
||
|
Chris Lu
|
c21d92b70a |
test: wait for async write-budget release after pipeline shutdown (#10530)
Shutdown drops the sealed-chunk map references, but an in-flight uploader goroutine holds the final reference and releases its budget slot only after reacquiring chunksLock. Asserting Used()==0 immediately after Shutdown races those releases on slow runners. Poll with a bounded deadline instead. |
||
|
Chris Lu
|
de00091765 |
test: random needles always carry at least one byte (#10523)
A zero-data needle lands in .dat as a size-0 record, byte-identical to a delete marker, so scans that walk .dat count it as deleted. Once in 1024 writes newRandomNeedle produced one, and the idx-head repair then skipped a row TestRepairIdxHeadTombstones_ReadOnlyVolume expected back. |
||
|
Chris Lu
|
c2701955c7 |
s3: cover three untested STS paths (#10521)
* s3: test the GetCallerIdentity handler The handler had no test, only XML marshalling, so nothing pinned that a caller presenting session credentials is reported as the assumed role rather than the user who minted the session. * s3: drive AssumeRoleWithWebIdentity over HTTP with a real OIDC token Coverage reached the OIDC path either at the IAMManager service layer or through the Authorization: Bearer shortcut. Nothing exercised the public STS entry point an AWS SDK actually calls, which is where parameter parsing, the IAMManager dispatch and the XML response shape live. * s3: test that every STS route emits an audit entry STS responses go out through WriteXMLResponse, which never calls PostLog, so track() is the only thing that logs them. A route registered outside it would mint credentials with no audit trail and nothing would notice. STS has three routing layers, so a new action is easy to attach to the wrong one. * s3: make the STS tests assert what they claim to cover The audit routing test ran against an uninitialized STS service, so every case answered 503 and a non-404 status was the only evidence the request had reached STS at all - the POST-body case could have been served by the dispatcher's IAM branch and still passed. Back it with a real STS service and assert the STS response namespace, which IAM and S3 responses do not carry. The session policy case checked that the policy travelled in the token rather than that it restricted anything; assert the narrowed bucket is allowed and another bucket is refused. Give the forged-token case the same claim set a valid token gets, so it cannot pass for want of a claim, and cover both rejection paths: a key we do not publish, and a key id absent from the JWKS. |
||
|
Chris Lu
|
ef6a706c0e |
master: count only writable volumes as crowded when deciding growth (#10522)
The crowded map keeps volumes that later became unwritable, so their state survives transient writability flips without flapping. But volumes packed to capacity (fs.mergeVolumes) or turned read-only stay above the crowded threshold and get re-marked on every heartbeat, so the raw map size can permanently exceed the writable count. ShouldGrowVolumes then returns true forever, every assign-path grow request passes the gate, and the periodic grow loop fires too, creating volumes without bound -- worse with -volumePreallocate. Count crowded as the intersection with writables instead: growth checks and the layout gauges only see crowded volumes that can still take writes. |
||
|
Chris Lu
|
1ce106e69d |
s3: audit the assumed-role principal and the STS caller (#10519)
* s3: log the requester's principal ARN in the audit entry An STS session authenticates as an opaque session subject, so requester alone gave an operator no way back to the assumed role or the session name. Record the principal ARN next to the identity name and emit it as requester_arn. * s3: record the caller identity in the STS handlers AssumeRole, GetFederationToken and GetCallerIdentity verify the caller themselves and are not wrapped by the auth middleware that records the identity, so every audit entry for minting a session had an empty requester. * s3: resolve the audit principal ARN the way policy evaluation does A JWT-authenticated identity carries no PrincipalArn — the auth layer hands the principal over in a request header — so reading the field directly left requester_arn empty for OIDC callers. buildPrincipalARN is the resolver the policy path already uses: header first, then the identity's own ARN, then a synthesized user ARN for legacy identities that have none. |
||
|
Chris Lu
|
fa432f9a6a |
s3: keep an admin's role session scoped to the role (#10520)
* s3: keep an admin's role session scoped to the role AssumeRole copied the caller's admin standing into the minted session as the is_admin claim, which short-circuits base policy evaluation. An admin assuming a scoped-down role therefore kept full access and the role's attached policies, explicit denies included, were never evaluated. Only a session the caller assumed for itself carries the claim now — a legacy static admin has no IAM policies for such a session to inherit. * s3: name the caller when it assumes a session for itself An identity that carries no principal ARN left the self-assumed session with an empty role name in its assumed-role ARN. callerPrincipalArn synthesizes the canonical user ARN for that case. |
||
|
Chris Lu
|
82c67b5896 |
test: cover listings spanning a run of retracted keys (#10517)
* test: cover listings spanning a run of retracted keys A listing drops entries whose current version is a delete marker. When a run of consecutive entries drops out, the page being filled can come back empty, and an empty page is easily mistaken for the end of the listing — everything after the run then never appears and the caller is told those objects do not exist. Backup repositories produce exactly this shape: a batch of keys under one prefix is retracted while writing continues under the next. Covers a retracted run before live keys and between live keys, walked with page sizes smaller than the run so at least one page is filled entirely from entries that get dropped, plus the version view of the same namespace where every version and every delete marker must still be reported. * test: sweep every page size in both walks and paginate the version listing |
||
|
Chris Lu
|
d8d29c4ede |
s3: carry storage class in the cached listing metadata (#10516)
A listing on a versioned bucket is served from metadata cached on the .versions directory entry so the whole listing is a single scan. The cache carried size, mtime, ETag, owner and the delete-marker flag but not the storage class, so newListEntry found none and fell back to STANDARD. The result was that HEAD and the listings disagreed about the same object: HEAD reported the class the object was stored with, while ListObjectsV2 and ListObjectVersions reported STANDARD for every object. Clients that filter or tier on storage class act on the listing. Caches the class alongside the other listing fields, clears it with them, and copies it in the routed RECOMPUTE_LATEST path so both finalize paths agree. |
||
|
Chris Lu
|
910fa1ff37 |
test: compare ListObjects and ListObjectVersions over the same namespace (#10515)
* test: compare ListObjects and ListObjectVersions over the same namespace The two listings walk the same tree through separate code paths, so a client navigating by versioned listings can see a different namespace than one navigating by plain listings, and concludes keys are missing that are plainly there. Testing each path on its own never catches that; only comparing them does, and nothing compared them. Asserts both report identical current keys and identical common prefixes across a backup-shaped tree: nested prefixes, a prefix naming an object exactly, a key that is simultaneously an object and the parent of other keys, a partial key fragment, and a prefix matching nothing. The version view is reduced to what a plain listing reports — latest versions that are not delete markers — so the comparison is like for like. * test: guard against truncated pages and cover the delete-marker path |
||
|
Chris Lu
|
fce4da5c9c |
test: pin verb parity on lock-arbitration keys through acquire and release (#10514)
* test: pin verb parity on lock-arbitration keys through acquire and release Backup clients arbitrate repository ownership by writing and retracting small keys under a fixed prefix and re-probing them, each probe using a different verb. They trust those verbs to agree; a key reported present by one and absent by another makes the client either spin or declare the repository corrupt, and neither shows up as an error on the storage side because each individual answer is locally correct. The keys are written and immediately deleted by version id, which is the cycle that empties a version container, so parity is asserted on both sides of the delete and across repeated re-acquire cycles where residue accumulates. Reports which verbs disagreed rather than just failing. * test: run the reacquire cycle on every lock key, and drain probe bodies |
||
|
Chris Lu
|
33a974b4c5 |
test: pin that an unusable version id is refused, never resolved (#10513)
* test: pin that an unusable version id is refused, never resolved A version id containing a path separator, or "." / "..", can never name a stored version. Resolving one to the null or latest version instead would let a caller destroy a live version by asking for one that does not exist, on a bucket configured for immutability. The guard exists today and holds on every verb; it had no test. Pins two properties: such a request is refused with a client error rather than a 5xx (a 5xx invites endless retries of something that can never succeed), and the version that does exist survives every refused request. * test: require exactly 400 for an unusable version id |
||
|
Chris Lu
|
2961448a36 |
test: cover delete idempotency on versioned object-locked buckets (#10512)
* test: cover delete idempotency on versioned object-locked buckets Backup clients probe and retract lock keys continuously, so they routinely delete keys and versions that are already gone, and they batch those deletes alongside keys that do exist. S3 makes all of that succeed; returning an error turns ordinary lock arbitration into a job failure. The behaviour is correct today but had no coverage, and it runs through the object-lock retention check, which is the most likely place for a missing object to start being reported as an error. Covers: deleting a key that never existed, deleting a well-formed version id that names nothing (twice, and without disturbing the version that does exist), and a batch whose middle key is missing — every requested key must come back under its own name rather than silently taking another row's slot. * test: verify the deletes actually took effect, not just that they returned |
||
|
Chris Lu
|
f582c8451b |
s3: report a peer that went away as ClientDisconnected, not IncompleteBody (#10511)
* s3: report a peer that went away as ClientDisconnected, not IncompleteBody A streaming PUT whose body ends early is always reported as IncompleteBody (400). That collapses two cases with opposite causes: the peer vanished mid-upload, and the peer sent fewer bytes than it promised while still connected. The first points at the network path, the second at the client, and once merged they cannot be told apart from the logs. Split out ClientDisconnected (499) and select it when the request context shows the peer is gone. The upload itself keeps running on a background context so chunks still finish, which means cancellation races the read error; a missed signal degrades to IncompleteBody exactly as before. * s3: note what request-context cancellation is taken to mean |
||
|
Chris Lu
|
7b8188fc41 |
wdclient: age vid map entries by generation instead of chaining snapshots (#10506)
* wdclient: age vid map entries by generation instead of chaining snapshots The vid map kept its history as a linked list of past snapshots, trimmed in place by storing nil into a node's cache pointer. That cost up to six full copies of the volume-location map, a recursive walk taking a different lock per level, and deletes that had to cascade through every generation. It also had to special-case explicitly-empty entries, or fallback would resurrect locations a newer snapshot had cleared. Keep one map instead, and stamp each entry with the generation it was learned in. resetVidMap bumps the generation and drops entries that were not relearned within the retained window, which is the same retention the chain provided: an entry survives DefaultVidMapCacheSize resets. The first write of a generation replaces an entry rather than merging into it, so a volume that moved answers with where it is now — the property a fresh map per reset used to give for free. Entries are copy-on-write, so locations handed to a caller are no longer shifted underneath it by a concurrent delete. The map is never swapped now, so the client-side lock and its stable / current accessors go away with it. * wdclient: make vid map entries immutable and drop them once emptied Review follow-up. Updating an entry in place left the copy-on-write guarantee resting on callers never holding the entry pointer; install a new entry instead, so the rule is simply that a stored entry never changes. Deleting a volume's last location now drops the entry rather than keeping an empty one, which a client that never resets would otherwise hold for every volume it ever saw deleted. Lookups already treat an empty entry as a miss, so nothing observable changes. * wdclient: let the newest generation decide between regular and EC locations GetLocations checked the regular map first whatever its generation, so a volume that was EC encoded kept answering with the regular copies the previous master knew until they expired — for as long as the retained window, since nothing relearns a copy that no longer exists. The snapshot chain did not have this problem: the newest map was consulted first and only a volume it knew nothing about fell through to older ones. Restore that by comparing generations, with regular copies winning a tie, since a tie means one generation reported both. |
||
|
Chris Lu
|
ae4839e005 |
mount: keep a sealed chunk alive until its own upload finishes (#10504)
Sealing a logic chunk index that already held a sealed chunk dropped the old chunk's only reference and freed its page chunk. That chunk's upload may not have started reading it yet — Execute() returns as soon as the job is handed to a goroutine — so mem.Free could hand a live 2 MiB mem chunk back to the slot pool, the next NewMemChunk would overwrite it, and the in-flight upload shipped whatever bytes were there. Under fio randwrite the volume server rejected those needles with "Content-MD5 did not match md5 of file data" and the FUSE write failed with EIO. Give the sealed chunk a second reference for its upload, dropped only by the upload itself, and let the upload unindex itself only while it still owns the index — the unconditional delete could evict a newer sealed chunk and hide its dirty pages from readers. |
||
|
Chris Lu
|
7fb36025b3 |
wdclient: read the vid map cache link before the live map (#10505)
resetVidMap trims the cache chain by storing nil into a node's cache pointer once it ages past vidMapCacheSize. A lookup that missed in its own map and only then loaded that pointer could find the link already severed, reporting "not found" for a volume that stayed resolvable the whole time. Load the link first, while it is still guaranteed live. A published vidMap's cache pointer only ever goes from its ancestor to nil, so reading it earlier can never yield staler history. |
||
|
Chris Lu
|
3514925581 |
filer: let a nested path rule turn worm off (#10503)
* filer: let a nested path rule turn worm off mergePathConf ORs the booleans, so worm set on a bucket could never be lifted on a directory under it, while every string field is overridden by the more specific rule. Make worm tri-state instead: unset inherits, set wins. readOnly, fsync and disableChunkDeletion keep the OR, so a nested rule still cannot escape a lock the bucket set. Configurations written before this carry an explicit "worm": false on every rule, because they are marshalled with EmitUnpopulated. Reading those back as an override would quietly drop worm from nested paths, so filer.conf is now stamped with a version and the flag is dropped to unset when the version predates it. * filer: copy the worm value out of the matched rule mergePathConf aliased the pointer into the merged result, so a caller that wrote through it would reach into the stored rule. |
||
|
Chris Lu
|
4dc1b70b2f |
test: pin that a .vif replication outranks the superblock (#10499)
* test: pin that a .vif replication outranks the superblock Store.ConfigureVolume rewrites the .vif and never the replica-placement byte in the .dat, so that byte keeps whatever the volume was created with for good. readSuperBlock reads it and then overrides it from the .vif, which is what makes a replication change take effect and survive a remount. Invert that and every replication change silently reverts on the next mount, while the .vif on disk still records what the operator asked for -- a durability setting quietly going back to its old value, with nothing to indicate it. Worth pinning rather than reading off the code, because the field beside it resolves the other way: version takes the superblock over the .vif. Two fields, one function, opposite precedence, each a line to invert wrongly. Covers the empty case too, since a .vif that declares no replication has to leave the superblock standing or a volume whose replication was never configured would be forced to whatever the zero value parses as. * test: drop the unreachable nil check on MaybeLoadVolumeInfo It initialises the returned pointer before the existence check and every return is naked, so it never yields nil. Guarding against it implied a contract the callee does not have. |
||
|
Chris Lu
|
3e74db1609 |
fix(master): bump seaweedfs/raft to v1.2.0 for the snapshot race (#10498)
The master could die with a nil pointer dereference inside raft.(*server).TakeSnapshot. The leader and follower loops ask for a snapshot on every iteration and callers can ask at any time, so several goroutines built one at once, all writing to the one pendingSnapshot field. Whichever finished first saved it and set the field to nil, and the rest either dereferenced nil while attaching peers and state, or stored nil as the current snapshot, leaving the master with no snapshot to send to a lagging peer. v1.2.0 holds a mutex for the length of a snapshot and keeps the one being built in a local. It also stops a snapshot recovery that cannot finish from being reported as done, writes snapshots to one side and renames them into place, loads the snapshot covering the most of the log rather than the last one in filename order, and takes the lock that guards the peer map on both sides, where a snapshot cloning the peers alongside a membership change was a concurrent map iteration and write. |
||
|
Chris Lu
|
63d5140485 |
s3: allow copying an object onto itself in a versioned bucket (#10497)
* s3: allow copying an object onto itself in a versioned bucket The copy writes a new version instead of overwriting in place, which is how an earlier version is restored. Buckets with versioning off or suspended keep rejecting a self-copy that changes nothing. * s3: cover the suspended-versioning self-copy rejection Suspended versioning overwrites the null version in place, so a self-copy that changes nothing stays rejected. Pin that alongside the never-versioned case. |
||
|
Chris Lu
|
e7a678fa72 |
s3: keep the list marker exclusive for versioned objects (#10496)
* s3: keep the list marker exclusive for versioned objects A versioned object lives in a "<key>.versions" directory, so the entry name never matched the marker and start-after/marker returned the marker key itself. * s3: match the list marker against the raw entry name too A backend that echoes the marker it was given returns the ".versions" directory name, which no longer matched once the comparison used the object name alone. Cover both, and unit test each half. |
||
|
Chris Lu
|
ccf5dc34e9 |
test: stop comparing two JWTs minted a second apart (#10495)
TestProxyReadDropsCallerJwtQueryParam mints a read token up front and requires the token the volume server would evaluate to equal it byte for byte. The expiry claim has one-second resolution -- GenJwtForVolumeServer sets it from jwt.NewNumericDate(time.Now().Add(...)) -- so two mints on either side of a tick produce different strings for the same authority and the same file, and the assertion fails for a reason the test is not about. It surfaces on the 32-bit job, where the runner is slow enough that the HEAD subtest (the second one, after a full proxy round trip) lands in a later second than the mint at the top of the test. Confirmed directly: minting the same file id with the same key either side of a boundary yields different tokens. Assert what the test is actually about instead -- that the credential decodes against the read key and authorizes this file id -- which holds whatever second it is minted in, and is a closer statement of the property than string equality. |
||
|
Chris Lu
|
78ed665557 |
webdav: answer PROPFIND child stats from the listing (#10492)
* webdav: answer PROPFIND child stats from the listing golang.org/x/net/webdav discards the FileInfo that Readdir returned and stats every child again, five times over, so a PROPFIND on a directory costs five sequential filer lookups per entry: 15006 lookups and 1.4s for 3000 subdirectories, 24s for 60000. Windows Explorer times out well before that. Hold the entries a listing already fetched for the lifetime of the request and serve those stats from them - 6 lookups and 0.03s for the same 3000 entries. WebDavFile.Stat has to stop dropping its request context for the held entries to be reachable. * webdav: keep the request context on the lookups a listing drives stat, Readdir and Seek reached the filer on context.Background(), so a client that walked away left the listing streaming and the lookups running. Seek also missed the entries the listing had already fetched. Write and cleanup paths keep their own context - a cancelled request must not abandon a flush half done. |
||
|
Chris Lu
|
01937cfad1 |
telemetry: build the over-time charts from confirmed clusters (#10489)
Short-lived clusters report once under a fresh raft topology id and never again, so CI runs and demo stacks each become their own cluster. Held forward for the whole active window they pile up, and the volume server line climbs every day while capacity stays flat. Sum the fleet series over confirmed clusters only, the same set the version and OS charts already use. |
||
|
Chris Lu
|
46ceb253b0 |
telemetry: report anonymous cluster stats by default (#10488)
The reports are what tell us which versions and cluster sizes are actually in use, and almost nobody flips the flag on, so the numbers we have are close to useless. Default it on for master, server and mini, and say in the flag help and the startup log how to turn it off. Nothing new is collected: still an in-memory cluster id that changes on restart, version, os, server counts, volume count and disk bytes, sent once a day by the leader master only. |
||
|
Chris Lu
|
c2183566b6 |
ec.encode: name the shard ids an aborted deletion found (#10486)
Counting by node lost the per-node ShardsInfo, and with it the shard ids the old summary printed -- the message an operator gets when the pre-delete check refuses now says only how many shards each node holds. That is the wrong half. A set holding shards 0-9 and one holding 4-13 are both "10 shards", and only the ids say whether what survived can rebuild the volume, or which node to go looking at. Keep the count and list the ids beside it. |
||
|
Chris Lu
|
a4692005e9 |
ci: harden the fusermount3 repair (#10485)
* ci: move the fusermount3 repair into a composite action Three copies of the same block were already drifting apart, and the target comes from PATH: only ever add setuid root to a root-owned, non-symlink binary under the system bin paths, and say why otherwise. * test: say that the process exited in the wait errors "process exit status 1 before ... accepted connections" is missing its verb. Also mark the SIGTERM return discarded - it fails with os.ErrProcessDone exactly when the select below already handles it. * ci: prefer the distro fusermount3 over escalating a shadow copy The shadowing /usr/local/bin/fusermount3 is not root-owned either, so setting its setuid bit would have handed root to a binary the runner user owns - the repair now symlinks the distro one earlier in PATH and touches nothing, keeping the in-place chmod for a root-owned binary with no distro alternative. A setuid bit only grants root when root owns the file, so accept an existing one only then. * ci: run the FUSE workflows when the shared action changes Their paths filters listed each workflow file but not the composite action all three now call. |
||
|
Chris Lu
|
c4798979d8 |
ec.encode: count shards wherever they landed before deleting the source (#10483)
generateEcShards writes shards beside the source volume, so encoding a volume that lives on a non-default medium puts them on that medium while -diskType still says hdd. The pre-delete check counted only the -diskType bucket, so it saw a complete set as zero shards, called it unrecoverable and aborted -- leaving the volume as both a .dat and a full shard set, which every later reader then disagrees about. Count by node across disks, as waitForEcShardsToRegister in the same file already does. The spread check is unaffected: it locates shards through collectEcShardBitsByNode and only uses diskType to find free slots. |
||
|
Chris Lu
|
167c114dae |
ci: fix FUSE mounts against the new runner image (#10484)
* ci: restore the setuid bit on a shadowed fusermount3 Newer ubuntu-22.04 runner images carry a source-built fusermount3 in /usr/local/bin that shadows the distro one in PATH and is not setuid root. go-fuse looks the helper up through PATH, so every unprivileged mount fails with "mount failed: Operation not permitted". * test: fail a fuse test as soon as its mount process dies A mount that cannot mount at all exits within a second, but the harness still waited out the 30s readiness timeout and then reported "mount point not ready within timeout", leaving the real cause buried in the log tail. Watch the child processes and report their exit instead. * mount: report a failed mount without a goroutine dump A mount failure is an environment problem - no /dev/fuse, fusermount not setuid, stale mount point - and the all-goroutine stack dump Fatalf adds buries the one line that says so. |
||
|
Chris Lu
|
4149346bb7 |
s3: register the advertised ip with the master (#10482)
* s3: register the advertised ip with the master The cluster address came from the bind ip, falling back to the auto-detected interface, so -ip never reached the S3 registration. weed mini -ip=localhost binds the wildcard and ended up registering whatever interface happened to sort first -- on a host with VPN interfaces, an address that stops routing once the tunnel drops. IAM changes are pushed to registered S3 servers over gRPC, so every mutation then blocked the full 10s propagation deadline before logging a failure, and cluster.ps and the admin UI listed a node nothing could reach. Identities still arrived through the /etc/iam metadata subscription, so this cost latency and visibility, not credentials. Add an advertise ip to the gateway option, preferring it over the bind address, and wire the parent -ip through server, filer and mini. * s3: treat any unspecified bind address as a wildcard net.ParseIP + IsUnspecified covers ::, [::] and the expanded IPv6 forms instead of only the 0.0.0.0 literal, so an IPv6 wildcard bind no longer registers an address peers cannot dial. Host names parse as nil and stay addresses in their own right. Apply the same guard to the advertised ip. |
||
|
Chris Lu
|
0002e5cc7f |
s3api: load document-style policies from the advanced IAM config (#10481)
* s3api: load document-style policies from the advanced IAM config The advanced IAM file doubles as the S3 identity config when only -s3.iam.config is given. protojson drops its "document" field, so every policy landed with empty content and warned "skipping invalid policy" on each reload. Worse, if the same file also declares identities the empty content sticks in the policy map and fails the whole runtime policy sync into the IAM manager, so policies created later never reach it. * iam: skip an unparsable policy instead of failing the whole runtime sync One policy the engine cannot parse aborted SyncRuntimePolicies before it touched anything, so every other policy stayed unsynced and the engine kept serving whatever it last held. * s3api: reject a non-role RoleArn in AssumeRole as a bad request arn:aws:iam:::user/name can never resolve to a role, but the handler ran it through the trust-policy check and answered "not authorized to assume role", pointing the caller at a permission problem they do not have. * s3api: build the policy content before touching the entry Deleting "document" up front meant a marshal failure left the policy with neither field, so a later rewrite would emit it with no definition at all. * iam: pin the fail-closed handling of an unparsable policy Say in the comment that dropping it from the desired set deletes it from the engine on purpose, and cover it with a test. * s3api: widen the non-role RoleArn test to canonical ARN shapes The reported ARN omits the account id; a user ARN that carries one, and a non-principal ARN, must be rejected the same way. |
||
|
Chris Lu
|
13176b4edd |
volume: recover .idx rows overwritten by tiered deletes (#10474)
* volume: recover .idx rows overwritten by tiered deletes A delete on a read-only volume backed by a remote tier used to write its tombstone row at .idx offset 0 rather than appending it, so each delete overwrote one more row at the front and lost the Put rows indexing the first needles in .dat. Those needles 404 even though .dat still holds them, and rebuilding .idx with weed fix means stopping the server and pulling the whole .dat back from the tier. The damage has a fingerprint -- .idx opening with a run of offset-0 tombstones, which a healthy .idx never does -- and .idx and .dat grow in lockstep, so the lost rows indexed exactly the first N .dat records. Detect it at load and re-derive them from a header-only walk over the head of .dat, cheap even against a remote tier, appending only the keys the .idx no longer names. * rust volume: mirror the .idx head tombstone recovery Port the Go detection and repair: an .idx opening with a run of offset-0 tombstones lost the Put rows indexing the first needles in .dat, so re-derive them at load from a header-only walk over the head of .dat and append the keys the .idx no longer names. * volume: put recovered .idx rows back in front instead of appending Appending left the offset-0 tombstone run at the head, so every later load re-walked .idx to the tail to notice the volume was already recovered, and the rows for the head of .dat sat past the .dat-tail row -- costing CheckVolumeDataIntegrity its O(1) path and breaking the ascending append order BinarySearchByAppendAtNs assumes. Rewrite .idx as the recovered rows followed by its current contents, through a temp file and a rename. .idx is back in .dat append order, so a later load stops after reading one row. * volume: keep the .idx mode when the repair replaces it The recovery renames a fresh temp file over .idx, so a fixed 0644 (Go) or whatever the umask allows (Rust) would silently widen an index an operator had locked down. Carry the mode off the file being replaced. |
||
|
Chris Lu
|
4b0d09683a |
iceberg: read manifest lists that omit the Avro format version (#10475)
* s3tables: read Iceberg manifest lists that omit the Avro format version The Iceberg spec pins the Avro header metadata of manifest files but says nothing about manifest lists, so writers disagree. Java and PyIceberg record "format-version"; DuckDB writes no header metadata at all. iceberg-go reads a missing entry as v1, so every v2 manifest listed in a DuckDB-written list is rejected with manifest file's 'format-version' metadata indicates version 2, but entry from manifest list indicates version 1 and, because v1 has no "content" field, delete manifests silently decode as data manifests. ReadManifestList derives the version from the record schema the writer embedded - v2 added "content" and the sequence numbers, v3 added "first_row_id" - and splices it into the header before handing the bytes to iceberg-go. Lists that already carry the entry, and input that is not a parseable Avro container, go through untouched. * iceberg: parse DuckDB-written manifest lists in maintenance and data preview Every manifest list read - the four maintenance operations and the admin table data preview - went straight to iceberg-go, so tables written by DuckDB failed detection and all of compact, remove_orphans, rewrite_manifests and expire_snapshots before they touched anything. Route them through s3tables.ReadManifestList, which recovers the format version the writer left out of the Avro header. This also restores the manifest content type on those tables: with the list read as v1 every delete manifest looked like a data manifest, which hid deletes from the compaction guard and made the preview report a table with position deletes as having none. |
||
|
Chris Lu
|
c8cafd8a1a |
telemetry: fix total disk usage over time (#10476)
* telemetry: total disk usage over time counted each cluster on one day GetMetrics aggregated s.instances, which holds only each cluster's most recent report. Every cluster therefore landed in a single date bucket -- the day it last reported on -- so the chart plotted the disk usage of clusters that went silent that day, and piled the whole live fleet onto today. Aggregate the daily histories instead, reusing the day alignment that the per-cluster size series already does. * telemetry: don't pad the charts with days the server has no history for The dashboard asks for 30 days, but daily history only starts when a server first collects it, so the charts opened on a run of zeros and then jumped -- reading as a fleet that appeared overnight. Start the window at the oldest sample on hand when it is younger than the requested range. |
||
|
Chris Lu
|
ac6f3c92ef |
s3api/iceberg: report the reason a table schema was rejected (#10473)
* s3api/iceberg: report the reason a table schema was rejected newTableMetadata swallowed the iceberg-go error and returned nil, so every schema the metadata builder refused came back as a bare 500 "Failed to build table metadata". A v3-only column type is the common case: creating a table with a variant field but no format-version 3 property leaves the client with nothing, while "variant is not supported until v3" sits in the server log. Return the error instead and classify it. Schema, spec and argument failures are the caller's input, so they answer 400 with the underlying reason; the rest stay 500. Paths that build placeholder metadata with no schema keep their existing 500 via newEmptyTableMetadata. * s3api/iceberg: fail LoadTable when placeholder metadata cannot be built buildLoadTableResult dropped a nil from the placeholder path straight into the response. That serializes as "metadata":null under HTTP 200, which no Iceberg client can parse -- a worse outcome than the 500 the nil was meant to signal. Return an error instead and let the five callers answer 500. The nil-return convention goes away with it, so the commit and transaction paths check an error rather than a sentinel. * s3api/iceberg: route rejected schemas through writeManagerError The two helpers added here duplicated work the package already does. writeManagerError is the canonical error-to-response mapper -- it already downgrades client-input failures to 400 and defaults the rest to 500 -- so teach it the iceberg-go schema and spec sentinels instead of standing up a parallel classifier. The placeholder wrapper was a pure alias for newTableMetadata with nil arguments; call that directly. No behavior change beyond the 500 message, which now reads err.Error() like every other manager error rather than carrying its own prefix. |
||
|
Chris Lu
|
9351202ca9 |
volume: scan for on-disk EC shards when staging a decoded volume (#10465)
The staged-new-volume placement skipped a disk holding the vid's EC shards using only the in-memory ecVolumes map, missing a shard present on disk but not mounted. Also scan the candidate disk for <vid>.ecNN files, so the promise holds regardless of mount state. Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu |
||
|
Chris Lu
|
3b3e8af430 |
volume: skip a shard-holding disk when staging a decoded volume (Go+Rust) (#10464)
volume: skip a shard-holding disk when staging a decoded volume ReceiveFile staged-new-volume mode picked any free disk of the target medium. Skip a disk that already holds the vid's EC shards (Go DiskLocation.FindEcVolume / Rust ec_volumes), so a decoded .dat never lands in the same directory as a shard. This lets a caller safely stage onto a shard host that has a spare disk, instead of requiring a host with no shard of the vid at all. Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu |
||
|
Chris Lu
|
9c37e52c9b |
volume: EC decode onto a clean peer via staged-new-volume adopt (Go+Rust) (#10463)
Decoding EC shards back to a normal volume in place reconstructs <vid>.dat
in the shards' own directory, so the vid is momentarily registered as both
an EC and a normal volume in one location — the load/scan path then sees it
as both, risking mount ambiguity and needle loss. VolumeEcShardsToVolume
still supports that in-place path; this adds the primitives to decode onto
a *clean* peer instead:
- ReceiveFile gains a staged-new-volume mode: when the volume does not
exist here and ReceiveFileInfo.disk_type is set, pick a free-slot disk
of that medium and write <base><ext>.copying (not a valid volume name,
so the scanner never half-loads a partial push).
- VolumeEcShardsToVolume gains from_staged: adopt the pushed .dat/.idx/
.vif — rename .copying into place under a .note in-progress marker,
then mount — so <vid> lands on the peer only as a normal volume.
The caller decodes the shards off-box and streams the finished volume to a
peer holding no shard of the vid on the target medium. Go and Rust volume
servers get identical handlers. Proto: ReceiveFileInfo.disk_type (12; 8-11
reserved for versioned-EC), VolumeEcShardsToVolumeRequest.from_staged (3) +
disk_type (4).
Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu
|
||
|
Chris Lu
|
84d3d62697 |
rust volume: mark-readonly notifies the live leader, not the static seed (#10461)
VolumeMarkReadonly mutates raft-replicated master topology, so it must reach the leader. notify_master_volume_readonly targeted the static seed (config.masters.first()), so after any master failover it hit a follower and failed "not current leader". Prefer current_master_url (the live leader the heartbeat tracks), fall back to the seed before the first heartbeat, mirroring store_ec.rs and Go's vs.GetMaster(). Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu |
||
|
Chris Lu
|
62c4333074 |
s3: list the buckets an attached IAM policy grants (#10458)
* s3: list the buckets an attached IAM policy grants ListBuckets served an identity authorized by an attached IAM policy only the buckets it had created itself. A user granted s3:ListBucket on a bucket someone else provisioned could GetObject and ListObjectsV2 against it, but the bucket never showed up in the listing any S3 client uses to build its bucket picker. The owner-index fast path is only valid for an identity whose grants name every bucket it can reach, and the routing check assumed a policy could never be enumerated. Read the names out of the policy instead: statements that allow s3:ListBucket on a concrete bucket ARN become candidates, and the per-bucket permission re-check still decides what is listed. A policy that can reach a bucket it does not name -- a wildcard resource, a policy variable, a NotResource, an STS session policy -- falls back to the full scan, which evaluates the policy per bucket. * s3: share the attached policy name lookup authorizeWithIAM and the ListBuckets enumeration both built an identity's policy names the same way, its own plus the ones from its enabled groups. Pull that into one helper so group eligibility is decided in a single place. * s3: read policy actions the way the IAM authorizer matches them The IAM authorizer matches action names case-insensitively, so a policy granting "S3:LISTBUCKET" or "S3:*" authorizes a list. The ListBuckets classifier read those actions with the local case-sensitive matcher and found no grant, so once the owner index was ready the buckets that policy allows dropped out of the listing. Match the action the looser way in the classifier: case-insensitive, and true for any pattern holding a policy variable. Over-matching only costs a candidate the per-bucket permission check then rejects, while under-matching hides a bucket the caller can read. * s3: infer a multipart grant in any case The classifier matches action patterns case-insensitively but looked the requested action up in a canonical-cased set, so "S3:UPLOADPART" missed the s3:PutObject inference that the authorizer makes. Key the set for lookup in lower case, matching how the IAM authorizer holds it. |
||
|
Chris Lu
|
5536d88fbb |
azure: let the blob endpoint be configured (#10460)
* azure: let the blob endpoint be configured The service url was always derived as <account>.blob.core.windows.net, which leaves out Azure Government, Azure China, and private endpoints. Name the blob service url instead and those accounts become reachable. The url has to be https, since the account key or the bearer token would otherwise travel in the clear. * azure: reject an endpoint that carries no hostname A url like https://:443/ has a host of ":443", so the emptiness check on Host let it through and the request only failed once it reached Azure. The hostname is what has to be there. |
||
|
Chris Lu
|
fee3fcb55a |
mount: report data sizes to df with -df.logical (#10459)
df on a mount shows the space the cluster gives up to the data: every replica of a regular volume, every shard of an ec one. That is the honest answer for capacity planning, but it is not the question a user asks when they want to know how much of their data is stored. Add -df.logical. The master reports the logical sizes alongside the raw ones: one replica per regular volume, the data shards of each ec volume counted once. Free space is divided by the copies the requested replication makes, so used plus available stays the amount of data the mount can still write, and it comes off the cluster-wide usage rather than one collection's, since capacity is cluster-wide too. Statistics through a filer resolves an unset replication to the filer's default rather than the master's, matching where the writes it is sizing for actually land. The flag governs the quota check too, so a mount has one notion of how much it is using. A filer that predates the new fields sends zeros, and the mount keeps reporting the raw sizes. |
||
|
Chris Lu
|
152f1a2096 |
master: count EC volumes in statistics used size (#10457)
Statistics aggregates the volume layouts of a collection, but EC volumes are tracked outside collectionMap, so they were reported as nothing. A mount over a cluster whose volumes have mostly been encoded showed a df used size of a few GiB against terabytes of EC data. Walk the data nodes and add the EC volumes of the requested collection. Every shard copy counts, parity included, the way a regular volume's used size counts every replica, so used size stays the space the cluster actually occupies. File count comes from the volume-wide .ecx and .ecj counts, taking the largest a holder reports rather than summing them: both files travel with the shards on a move, so several nodes can report the same tombstones. |
||
|
Chris Lu
|
3ae4e9c563 |
azure: authenticate with Entra ID instead of a storage account key (#10456)
* azure: authenticate the blob sink with Entra ID Shared account keys have to be distributed and rotated everywhere a sink runs. Leaving account_key empty now falls back to the identity chain, so a workload identity or managed identity carries the authorization instead. * azure: authenticate remote storage with Entra ID The remote storage client demanded an account key and refused to start without one. Fall back to the identity chain when it is absent, and let azure.client_id pin a user-assigned identity. * azure: reject a malformed storage account name The account name is interpolated into the service URL, so a name carrying a "/", "?" or "@" moves the authority elsewhere and an authenticated request follows it. Hold callers to Azure's own naming rule instead. * azure: keep a leftover environment key off the identity path A configured client id asks for Entra ID, but AZURE_STORAGE_ACCESS_KEY still filled in the account key behind it. An old mounted secret would go on authenticating until it rotated, and the failure then blamed the key. * azure: say what the identity path reads from the environment A pinned client id alone is not enough for workload identity: the tenant and the projected token come from the environment, and missing them only surfaces later, when a token is first requested. |
||
|
Chris Lu
|
6b6e6d8547 |
s3: apply filer identity changes despite a static config file (#10392)
* s3: apply filer identity changes despite a static config file A -config file with inline identities disabled the metadata-subscription reload entirely, leaving the best-effort filer->s3 push as the only way s3.configure changes could reach a running gateway. Reload on IAM events regardless: the merge keeps the file's identities protected, and a full credential-manager snapshot now also drops dynamic identities the store no longer has, so revocation works without a restart. * s3: log identity propagation failures as warnings * s3: retry failed IAM reloads and reconcile policies and groups An event-driven reload that fails now hands off to a coalescing retry loop, so a transient filer error cannot strand a revoked credential until the next IAM event. Full-state merges also drop dynamic policies the store no longer has, keeping the static file's, and treat the group snapshot as authoritative even when empty. * s3: serialize IAM configuration loads The SIGHUP file reload, subscription reloads, the retry loop, and the postgres poll run on different goroutines. Without an end-to-end lock a load holding an older store snapshot can commit after a newer one and revert it. Hold reloadMu from snapshot through commit in both load entry points; partial merges from pushed updates stay lock-free and self-heal through the next event-driven reload. * s3: keep static-file groups through full-state reconciliation Group names from the static config file are tracked like identities and policies, and a full snapshot that does not carry them keeps the current definition and its memberships instead of dropping them. * credential: include groups in postgres configuration snapshots Full-state reconciliation treats absent groups as deleted, so a snapshot that never carries them would erase every dynamic group. * s3: revoke static-file groups dropped from the config file A file reload is authoritative for the file's group set while keeping dynamic groups, mirroring how full snapshots are authoritative for dynamic groups while keeping the file's. * credential: fail filer snapshots on unreadable entries A skipped identity or policy file made the load report success with an incomplete snapshot, which reconciliation reads as deletion and the retry loop never sees. Unparseable content is still skipped: it is durable, matches boot behavior, and must not block reloads forever. * s3: ignore groups in static config files Groups are managed through the IAM API and the dynamic store; no deployment defines them in a bootstrap config file. Ignoring them with a warning removes the two-directional group merge: full snapshots are plainly authoritative and file reloads never touch groups. |
||
|
Chris Lu
|
a7f4b88a61 |
s3: require a bucket-policy action to write a bucket policy (#10444)
* s3: require a bucket-policy action to write a bucket policy PutBucketPolicy and DeleteBucketPolicy were gated on ACTION_WRITE, the same action that grants object writes. An explicit Allow in a bucket policy short-circuits IAM entirely -- authRequestWithAuthType sets policyAllows and skips VerifyActionPermission -- so anyone who could write an object could author a policy granting itself, or anonymous, anything on the bucket. That is what separates a bucket policy from the sibling bucket controls also gated on ACTION_WRITE: rewriting cors or lifecycle can destroy data, but only a policy hands out access. Give the two verbs their own actions, mapped to the AWS names that were already defined but unrouted. ACTION_ADMIN would also have closed it, but it resolves to s3:* for IAM identities, forcing a blanket grant on a user holding a precise s3:PutBucketPolicy. Admins are unaffected, since isAdmin short-circuits CanDo, and an operator can delegate with PutBucketPolicy:bucket. The route binding is asserted from the router source: checking the action constants alone still passes when the route says ACTION_WRITE. * s3: also read the action from a direct iam.Auth call in the route test Routes read iam.Auth(cb.Limit(handler, ACTION)), a multi-value pass-through: Limit returns (http.HandlerFunc, Action) and those become Auth's parameters, so the action Auth authorizes on is Limit's second argument and the two cannot disagree -- Auth(Limit(h, X), Y) does not compile. A route that skipped Limit and called Auth with its own action would compile, though, and the test reported that as a missing route rather than as the wrong action. Recognise the two-argument Auth form so it names the action instead. * s3: make the bucket-policy actions grantable through an IAM policy The new actions close the escalation only if an operator can grant them, and they were not reachable: MapToStatementAction had no entry for PutBucketPolicy, so an IAM policy naming s3:PutBucketPolicy was rejected outright with "not a valid action". GetBucketPolicy was unmapped the same way. DeleteBucketPolicy was mapped, but to ACTION_ADMIN -- granting an identity permission to delete a bucket policy handed it full administrative access. Map all three to the actions the router now uses, and add the reverse direction so an identity holding them renders back as a policy statement instead of a bare "s3:". * admin: offer the bucket-policy permissions in the user editor The two new actions are otherwise only grantable by hand-editing identity JSON or by calling the IAM API, so an operator using the UI cannot delegate bucket policy management without granting Admin. Regenerating this file also picks up codegen the repo has not taken yet: the checked-in _templ.go files were produced by templ v0.3.1001 while go.mod pins v0.3.1020, so the generator rewrites the attribute-value calls. That churn is confined to this one file; running `make generate` in weed/admin reproduces it across all 36. |
||
|
Chris Lu
|
c7d0477117 |
volume: widen the gRPC admin gate and stop it drifting (#10443)
* volume: gate the admin RPCs that only shell and workers call checkGrpcAdminAuth covered 19 of the 48 VolumeServer RPCs, so an operator who sets -whiteList expecting it to cover the gRPC surface gets partial coverage. Extend it to ten that mutate state and are only ever called by the shell or a worker: SetState, VolumeCopy, the EC generate/rebuild/copy/unmount/to-volume pair, both tier moves, and VolumeTailReceiver. That is safe because the same callers already reach gated RPCs today -- VolumeMarkReadonly, VacuumVolume*, VolumeEcShardsDelete, VolumeDelete -- so a whitelist deployment already lists those hosts. Nothing here is on a master or peer path, which is what made the earlier fail-closed gate break multi-host clusters. The split is by caller rather than by blast radius: the guard matches a peer IP against the whitelist, and a whitelist holds masters, shell hosts and workers, not every peer volume server. Gating a call one volume server makes to another would break replication, EC and tiering, so those stay open. Two test fakes embedded a nil grpc.ServerStream and only implemented Send; they now implement Context, which the streaming RPCs read to authorize. * volume: fail the build when a gRPC method skips the admin gate The admin gate is an opt-in list in a 48-method service, which is how it drifted down to covering 19 of them: nothing tied adding an RPC to deciding whether it needed the gate. Parse volume_server.proto, walk the AST of every *VolumeServer method, and require each RPC to either call checkGrpcAdminAuth or appear in ungatedVolumeServerRPCs with the reason it stays open. A stale entry naming an RPC that no longer exists fails too, so the list can't quietly stop exempting anything. The exemptions are the cluster-internal calls -- replica sync, EC shard distribution, vacuum reads, backup, tailing -- plus the read-only and liveness RPCs. Closing the cluster-internal ones needs a peer identity rather than an IP whitelist; recording them here makes that a visible decision instead of an omission. The AST walk also corrects the count: a line-window scan credits VacuumVolumeCheck and VolumeServerStatus with a neighbouring function's guard. |
||
|
Chris Lu
|
be81b9d5d7 |
volume: fix EC decode/reconstruct index locality under -dir.idx (#10442)
* volume: fix EC decode/reconstruct index locality under -dir.idx EC->replicated decode failed under -dir.idx and on multi-disk with "volume not found on disk". The reconstruct rebuilds the .dat on the data disk but the on-demand VolumeMount scans only the data directory, matching on .idx/.vif; with the rebuilt .idx off in the index directory it matched the volume's leftover EC .vif and skipped the volume as EC metadata. - Resolve the EC .ecx local-first: prefer the copy co-located with the shards over the shared -dir.idx copy, with a non-empty preference so a 0-byte local stub still yields to a valid sibling (the cross-disk fallback). - Co-locate the rebuilt .idx with the .dat at the end of the reconstruct so the mount finds it; sweep .ecx/.ecj from both the data and index directories on Destroy so a stale copy cannot re-mount as a phantom EC volume. - Add VolumeConsolidateIndex: once the EC shards are deleted, unmount, move the .idx/.sdx from the data disk back to the -dir.idx directory (copy fallback across filesystems), and remount. A no-op without -dir.idx. * volume: tests for EC index locality (local-first .ecx, sweep, consolidate) - NewEcVolume prefers a non-empty local .ecx over the shared index dir, and a 0-byte local stub yields to a non-empty shared copy (the #9212 fallback). - Destroy sweeps .ecx/.ecj from both the data and index directories. - ConsolidateVolumeIndex moves a co-located index back to the -dir.idx dir and keeps the volume mounted; no-op without a separate index dir. - RenameOrCopyFile moves a file and drops the source. * volume: relocate the decoded index in place, without a read gap ConsolidateVolumeIndex previously unmounted the volume, moved the index, and remounted it. Between the EC-shard delete and the remount the volume had neither a normal nor an EC form mounted, so a read landing in that window got a not-found (or was proxied away). Move the index in place instead: RelocateIndexTo takes the data-file write lock, closes the needle map and data backend, moves the .idx (and derived .sdx), then retargets dirIdx and reloads — the same close-swap-load CommitCompact uses. The volume never leaves the mounted set, so a concurrent read blocks briefly on the lock rather than failing. The test now writes a needle before consolidating and reads it back after, proving the in-place reload keeps the volume serving. * volume: address review — maintenance guard, no orphan on copy failure - VolumeConsolidateIndex now rejects the request under maintenance mode, like VolumeConfigure and the other mutating volume RPCs. - RenameOrCopyFile rolls the cross-device copy back when the source cannot be removed, so a failed move never leaves two divergent copies (the loader would keep the data-dir one while the idx-dir orphan goes stale). - RelocateIndexTo logs a failed reopen-after-failed-move instead of swallowing it, since that leaves the volume unusable until the next load. |
||
|
Chris Lu
|
83f754763e |
filer: make the redis connection settings configurable (#10441)
The sentinel stores hardcoded a 30s read timeout and a 1m retry backoff. After a sentinel failover every request that picked a pooled connection to the old master sat there for 30s before the connection was retired, and the pool timeout derived from it (read timeout + 1s) queued the rest behind them. The other redis stores took the go-redis defaults with no way to tune anything. Read the dial, timeout and pool knobs from each redis store section instead, keeping the go-redis default for every key left unset. |
||
|
Chris Lu
|
5cac980b32 |
filer: drop a caller's jwt query param on a proxied read (#10440)
security.GetJwt reads the "jwt" query parameter before the Authorization header, and the proxy forwarded the caller's whole query apart from proxyChunkId. So on a read, where the filer mints a volume token and sets the header itself, a caller-supplied ?jwt= silently outranked it: the volume server validated a credential the caller chose rather than the one the filer attached, and the read failed with a 401 the filer could not explain. Drop it on the read path, where the filer owns the credential. Writes keep theirs -- the proxy forwards a writer's own AssignVolume token either way, so the query parameter is just a second channel for the same credential and stripping it would break a caller that presents it that way. Nothing in the tree passes a jwt by query; maybeAddAuth always sets the header. |
||
|
Chris Lu
|
6824619c16 |
s3: chunk uploads at the filer's maxMB (#10439)
The S3 write path cut fixed 8MB chunks, so an object stored through S3 chunked differently from the same bytes stored through the filer, WebDAV or a mount, and -maxMB had no effect on it. Read maxMB from the filer configuration at startup and use it, falling back to 8MB when the filer reports none. |
||
|
Chris Lu
|
0f718f8509 |
filer: add a placement overlay seam for the write path (#10437)
* filer: add a placement overlay seam for the write path New volumes take their disk type, replication, and data center from the explicit request or the matched filer.conf rule. That leaves no way for a feature to steer a whole collection onto a medium without an operator writing an fs.configure rule by hand. Add a generic PlacementOverlay hook on the filer: a func that maps a collection to a placement override, installed by a factory the way the plugin-worker handlers register. detectStorageOption consults it between the explicit request value and the filer.conf rule, so it overrides the rule but yields to a value the caller asked for. The seam names no feature concepts, so it stays generic; a downstream build registers the overlay it wants (e.g. a storage-class Landing tier). Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu * filer: address review on the placement overlay seam Honor ResolvePlacement's ok flag explicitly rather than relying on empty values falling through the util.Nvl chain, and log at V(4) when the overlay steers a collection. Document that RegisterPlacementOverlay is init-only, so the unsynchronized read in NewFiler cannot race the write. Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu |
||
|
Chris Lu
|
91979ec51e |
filer: fix credential handling on the proxyChunkId path (#10434)
* filer: claim the base fid when minting a volume read token GenJwtForVolumeServer stamped the fid verbatim, but the volume server strips a trailing _N delta suffix before comparing the claim, so a token minted for a batch-assigned fid like 3,01637037d6_1 was checked against 3,01637037d6 and never matched. Reading such a chunk through the filer returned 401 wherever jwt.signing.read.key was configured. Strip the suffix before minting, via a helper shared with the proxyChunkId validation that was already doing the same thing inline. * filer: don't mint a volume write token for an anonymous proxy caller The ?proxyChunkId= branch dispatches and returns before the JWT gate, so whatever credential the proxy attaches is reachable without authentication. It attached a token from maybeGetVolumeReadJwtAuthorizationToken, which fell back to the write signing key when jwt.signing.read.key was unset -- the configuration scaffold/security.toml recommends for a filer, since read JWTs are only supported in a master+volume setup. An anonymous DELETE /?proxyChunkId=<fid> therefore arrived at the volume server holding a write-key token scoped to that fid, and the volume server honored it. Sign read tokens with the read key only. The fallback bought nothing on a read anyway: a volume server enforces read JWTs solely when that same key is set, so when the fallback fired the read was unchecked regardless. Mint only for reads. Writers proxied through the filer carry their own volume JWT from AssignVolume, forwarded with the rest of the caller's headers, so weed mount -filerProxy uploads are unaffected. Moving the dispatch below the JWT gate instead would have broken them, since that token is signed with jwt.signing rather than jwt.filer_signing. On a read with nothing to mint, drop the caller's Authorization rather than relaying it: there it is a filer credential, and forwarding it would hand a volume server a token it never used to see. * filer: keep proxied writes out of the read concurrency semaphore The semaphore is named and documented for reads -- it exists so replication bursts can't open hundreds of connections to one volume server -- but it was applied to every proxied method. A write queued behind sixteen in-flight reads can wait past the 10s default expiry of the AssignVolume token it carries, and the volume server then answers 401. shouldReassignUpload treats a 4xx as final, so the uploader replays the same expired token instead of re-assigning and the write fails up to the caller. This only became reachable once the filer stopped re-minting a fresh token after the wait. |
||
|
Chris Lu
|
4f43153918 |
filer: reject a proxyChunkId that isn't a well-formed fid (#10436)
LookupFileId only requires the fid to contain a single comma, and the value is pasted straight into the volume server URL path, so ?proxyChunkId=3,x/../../status resolves to volume 3 and then addresses an endpoint the caller never named: Go sends the dot segments verbatim, the volume server's mux cleans the path and redirects to /status, and the filer follows the redirect and relays the body. That reaches any handler on the volume server -- status, stats, the UI -- past a filer that operators expect to be the only exposed surface. Parse the fid before the lookup and answer 400 when it doesn't parse. A trailing _N delta suffix from batch assigns is legal, so it is stripped first, but only when it is a non-empty run of digits. The volume server strips at the last "_" unconditionally, which is safe there because its fid came out of a path the mux parsed and so cannot hold a "/"; here the value is raw query input, and an unguarded strip would reduce "3,01637037d6_1/../../status" to a valid fid and wave the traversal through. |
||
|
Chris Lu
|
d867b6e739 |
log_buffer: bound the flush queue in bytes, not in copies (#10433)
The queue holds sixteen sealed windows, which is a memory bound only while a window is BufferSize. An entry larger than that grows its window to fit, and the depth then multiplies straight through: sixteen queued copies of a 100 MB window is 1.6 GB of flush data alone. Account the queued bytes and make producers wait once they pass the ceiling the depth was chosen for. What is charged is the pooled slab rather than the window length, since mem.Allocate rounds up to a size class and the queue holds the whole slab. A window larger than the whole budget still goes through on its own, so an oversized entry is never stuck. Windows are admitted in the order they were sealed. A producer can now park here for seconds, and letting a later window overtake an earlier one would persist them out of order and walk lastFlushedOffset and lastFlushTsNs backwards. A window is copied into its slab under the write lock, before the reservation is taken, so a burst of concurrent oversized writers would each hold a full copy in hand while queueing up -- memory the budget never sees. Large writers wait for queue headroom before they take the lock, which throttles the burst; it does not bound it, since a writer that passes the check still seals unconditionally. Take any room in the queue before the shutdown escape, too: the window is already sealed by then, so dropping it loses records the caller was told were accepted. A shutdown that races a full queue can still drop one -- that predates this change and needs the flush loop's lifetime reworked. Size a grown window to the entry rather than to twice it: the extra room only bought space for a second oversized record in the same window, which doubles the flush copy and the snapshot taken of it. The overflow guard halved its bound for that doubled allocation, so raise it to match what is now allocated and what maxBufferSize documents. |
||
|
Chris Lu
|
2d9227747a |
volume: reject needle blob writes to read-only volumes (#10435)
* volume: reject needle blob writes to read-only volumes WriteNeedleBlob appends the blob to .dat and only then calls nm.Put. On a read-only volume the needle map is a SortedFileNeedleMap whose Put always fails, so the append is never indexed and never rolled back. Nothing upstream stops this: volume.check.disk picks its targets from the master's cached topology, which goes stale the moment a volume server marks a replica read-only itself — a failed data integrity check at load, or an EIO quarantine. Each sync attempt then grows the .dat of a replica that is supposed to be frozen by one unindexed needle, and reports it as "invalid argument", the bare os.ErrInvalid the needle map returns. Check IsReadOnly before touching .dat, same as the upload path does. * volume: say which needle and volume failed to index An index write that fails surfaced as a bare errno with no volume, no needle and no file — "invalid argument" for a read-only needle map, or a plain ENOSPC when .idx lives on its own filesystem via -dir.idx. Both were logged at V(4), so by default the operator saw only the errno the client got back. |
||
|
Chris Lu
|
7b3462be6a |
filer: stop an oversized metadata log flush from wedging the change feed (#10430)
* filer: write the metadata log in pieces a volume server will accept A single oversized metadata event grows the log buffer past the volume server's fileSizeLimitMB, and the flush of that buffer is then rejected forever: the retry loop has no exit, so the blob at the head of the queue blocks every later flush and the metadata feed stalls until restart. Split the flushed buffer into BufferSize pieces, on record boundaries where possible so each piece still decodes on its own, and retry each piece separately so a partial success is not replayed. Log files are already read as a chunk stream, with a whole-file fallback when a chunk does not decode standalone, so a record may cross a piece boundary. * log_buffer: let go of a window array grown for an oversized entry An entry larger than BufferSize grows the window array to 2*size+4, and window arrays cycle through SealBuffer rather than being freed. One such entry therefore leaves every later window carrying its size, and currentSnapshotView allocates a snapshot as wide as the array on each window, so a few KB of metadata keeps paying for it. Drop the array when SealBuffer hands it back. Growth is on demand, so the next oversized entry just reallocates. * iceberg maintenance: store merged data files as chunks, not inline saveFilerFile had no size threshold, so compaction wrote whole merged parquet files -- hundreds of MB -- as Entry.Content. That puts the parquet bytes verbatim in the filer store and sends them through the metadata change log again as one event. Keep manifests and metadata JSON inline, upload anything larger to volume servers in chunks, assigning through the filer so the path's storage rules apply. * filer: follow the file size limit the volume servers report The starting piece size is a constant, so a cluster whose -fileSizeLimitMB is set below it would reject every piece and wedge just the same. The rejection names the limit, so take it from there and re-cut the rest of the flush to fit. Piece the buffer one at a time rather than up front, since the size can change partway through a flush. |
||
|
Chris Lu
|
cba2e5150c |
plugin: fix flaky scheduler lock test (#10432)
plugin: stop the scheduler lock test racing its own background loops TestRunLaneSchedulerIterationLockBehavior constructed the plugin with a cluster-context provider, which makes New start a background scheduler loop per lane. Those loops call runLaneSchedulerIteration on the same lane the test then drives by hand, so a loop could consume the due job — running detection and pushing the next-detection time forward — before the manual call observed the lock. The Default case then saw the lock acquired zero times and failed intermittently. Construct without the provider so no loops start, and set the provider afterward so the manual iteration can still detect. This is the pattern scheduler_status_test.go already uses for the same reason. Reproduced under -race -count=100 -p 4 before, green after. Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu |
||
|
Chris Lu
|
19ce7c0b6f |
consolidate the duplicated transient-error classifiers onto util.IsTransientError (#10429)
* util: match transient error messages case-insensitively, expose the message form The same condition reaches different layers capitalized differently -- a volume server relays its idle timeout as "I/O timeout" inside a JSON string -- and the callers that grew their own substring lists all lower-case first. Also split out IsTransientErrorMessage for the paths that carry only the text, such as the per-file status strings in a batch delete response, and pick up "no route to host" and "network is unreachable" from the gRPC classifier. * filersink: classify transient network errors through util.IsTransientError The local list caught i/o timeout, connection reset, and broken pipe but not connection refused, no such host, unexpected EOF, the syscall errnos, or the gRPC and S3 overload codes. Keep only the bare io.EOF case, which is transient here -- a truncated chunk read -- but a clean stream end elsewhere. * filer deletion: reuse util.IsTransientErrorMessage for the network patterns Six of the sixteen patterns were already covered. Keep the ones specific to this pipeline -- read-only volumes, lookup failures, backpressure -- and note why context cancellation stays retryable here: it decides whether to requeue the deletion, not whether to retry a call. * wdclient: fold the shared classifier into the volume lookup retry check The string tail duplicated the shared list and missed the syscall errnos and net.Error timeouts. Keep "connection" and "timeout", which are broader than the shared classifier on purpose: a volume lookup is a cheap read-only call. |
||
|
Chris Lu
|
c438c5ef94 |
filer.replicate: acknowledge notifications after the sink write, not on receipt (#10427)
* filer.replicate: commit the kafka offset after replicating, not on receipt The partition consumer committed the offset as soon as it handed the message to the channel, so a sink write that failed was logged and the message was already behind the committed offset -- never redelivered, permanently missing from the sink. Commit in onSuccessFn instead, and hold the committed offset behind the oldest offset that failed to replicate so a restart redelivers from there. * filer.replicate: delete the sqs message after replicating, not on receipt ReceiveMessage deleted the message before the replicator had a chance to run, so a failed sink write dropped it for good. Move the delete into onSuccessFn and leave the message in the queue otherwise, letting the visibility timeout redeliver it. |
||
|
Chris Lu
|
652273301e |
filer sync: do not advance the sync offset past a failed event (#10424)
* util: retry transient errors, not just the ones containing "transport" util.Retry only retried when the error string contained "transport", so a plain "read: connection reset by peer" from S3 got zero retries. Classify the error instead: net timeouts, connection resets, and the throttling and overload replies S3 and gRPC return are all worth another attempt, while a cancelled or expired context is not. * filer sync: hold the sync offset behind a failed event A sync job that returned an error was logged and forgotten, and the watermark advanced past it anyway. The offset is the durable resume point, so the event was never replayed: for filer.remote.sync that left the file present locally, absent on the remote, with no RemoteEntry and nothing to retry it. Pin the watermark at the oldest failed event. Later events keep flowing, but the persisted offset stays behind the failure, so a restart replays it. |
||
|
Chris Lu
|
f18ad39142 |
filer: honor the documented TLS options in every redis store (#10425)
The scaffold advertises enable_tls, ca_cert_path, client_cert_path and client_key_path under redis2, redis2_sentinel and redis_cluster2, but only the plain redis2 and redis3 stores ever read them, and under a different name, enable_mtls. Sentinel and cluster setups quietly connected in plaintext. Build the TLS config in one place and use it from all six stores. enable_mtls still works. The CA and the client key pair are optional now, so enable_tls alone verifies against the system roots, and ServerName is left unset so go-redis validates each address the sentinel and cluster clients dial. |
||
|
Chris Lu
|
c392f45705 |
s3: stop listing prefixes whose objects are all delete-marked (#10419)
Deleting the only object under a prefix in a versioned bucket writes a delete marker and keeps the version history, so the filer directory survives with nothing a current-version listing would return. A delimited ListObjects kept reporting that path in CommonPrefixes, because the prefixes come from the directory tree rather than from the keys, while a listing scoped inside the prefix correctly came back empty. Probe a directory before reporting it: one that holds entries but no key the listing returns is neither a CommonPrefix nor a path the trailing-slash probe answers for. Empty directories keep the meaning they have today, and the probe only runs for buckets with versioning configured, the only ones that can reach this state. |
||
|
Chris Lu
|
1e1b2bb2f9 |
iceberg maintenance: record file counters on the snapshots it commits (#10420)
A maintenance snapshot carried only its own labels — merged-files, delete-groups and friends — and no summary counters, so every engine that reads a table's size out of the current snapshot summary reported nothing for it: PyIceberg's inspect.snapshots, Trino's $snapshots and Spark's DESCRIBE all read total-records, total-data-files and total-files-size verbatim, and a table lost them the moment compaction touched it. Accumulate the files each operation adds and removes, and render them the way the spec defines: the added-*/removed-* counters from the files themselves, then the totals carried over from the parent snapshot. Carry a total only when the parent recorded it. Iceberg treats a missing total as zero, which turns a compaction replacing two files with one into a negative total-data-files, or a table with millions of rows into total-records: 0. Leaving the field out lets a reader fall back to the manifests instead of believing a made-up number. Compaction also accounts for the delete files it consumes, so a run that folds every delete into the rewritten data reports them as removed. |
||
|
Chris Lu
|
b4b0346f95 |
iceberg maintenance: resolve table files from the recorded location (#10418)
The worker assumed every file of a table sits under its catalog path, so loadFileByIcebergPath stripped the scheme off a recorded location and joined the remainder onto /buckets/<bucket>/<ns>/<table>. A table the REST catalog placed elsewhere in the bucket — which is what a client gets whenever the catalog path is already occupied — then resolves to a doubled path: lookup /buckets/lake/source/t/lake/source/t-0cd81bca-.../metadata/snap-.avro so the very first manifest list read fails and the job fails again on every scan interval, indefinitely. Resolve absolute references (s3:// URIs and /buckets paths) from the bucket root and keep relative ones under the table's own directory; the bucket-relative form is now the canonical key everywhere references are compared. That directory comes from the metadata location the catalog stores, so reads, writes and deletes all land where the table's other files are instead of splitting it across two trees. References outside the table's bucket are rejected rather than silently misresolved. Rewritten position-delete files now name their data file by absolute URI, the way the table itself names it, instead of a path relative to the table. |
||
|
Chris Lu
|
c194924d13 |
telemetry: per-cluster size over time on the dashboard (#10417)
The dashboard charted one summed disk-usage line, so a step in the total gave no hint which cluster moved. A new panel stacks each cluster's daily size as its own band: the top of the stack is the fleet total, each band is one cluster, and the clusters past the twentieth are summed into an "other" band so the stack still adds up to the total. The series is built from the per-cluster daily histories and served by /api/cluster-sizes. Clusters report roughly once a day at no fixed hour, so a day with no report carries the previous value forward — dropping it to zero would sag the total every day as the clusters that have not reported yet fall out from under it. A cluster that stops reporting past the active window ends at its last sample instead of holding capacity forever. Ranking is by the most recent day, tie-broken on cluster id so the colors do not shuffle between refreshes. Hover and click resolve to the band under the pointer: Chart.js's builtin interaction modes match the nearest line, which on a stack of thin bands is rarely the band being pointed at. Clicking one fills the per-cluster history lookup below it. |
||
|
Chris Lu
|
6e6255b58e |
shell: accept a context in the volume move helpers (#10415)
LiveMoveVolume and the copy, tail, delete, mark, replicate, and configure helpers around it issued every RPC on context.Background(), so a caller had no way to bound or abort a move once it started. They now take a context, which the exported LiveMoveVolume in particular needs: callers outside the shell drive long moves and want to stop them. The deferred restore in copyVolume runs on a detached, bounded context rather than the caller's. Marking the source writable again is cleanup, and cancelling the copy must not skip it and leave the volume readonly — the same guard balance_task.go already applies for the same reason. Shell commands pass context.Background(): their Do signature carries no context, and changing it would touch every command in the package. Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu |
||
|
Chris Lu
|
47b491b53c |
mount: version open file handles by filer log position (#10403)
* filer: stamp a log position on lookup and remote-cache responses Metadata events are logged after their store write and stamped with the filer clock. Reading that clock before serving an entry therefore gives a timestamp with a causal guarantee: every event at or below it is reflected in the returned entry. Clients caching filer state can use it as the entry's version to order the response against subscription events, including events committed before the call but delivered after it. * mount: version open file handles by filer log position A subscription event refreshing an open handle did a second lookup; a transient failure left the handle pinned to its old entry with no retry, since the subscription cursor had already advanced. The deeper problem is ordering: the handle is a cache written by three unordered channels — the async invalidation worker, local mutation acks, and open-time lookups — and overwriting cached state safely requires knowing which write is newer. The filer log timestamp is that order, and it now travels with every value instead of being derived out of band. Events carry it natively; lookup and remote-cache responses carry the log position stamped before the serving read; mutation acks carry it in their returned event; and the local store pairs each read with a version cursor advanced under the same lock as the store write. Each handle records the version its entry reflects, and one rule replaces the per-site reasoning: state at or below the handle's version is old news and must not be installed. The invalidation itself applies the event's own entry — no lookup, so no transient-failure window — except under a cached parent, where the store entry is the ordered merge of the event and anything applied since, and its version outranks the event's. An uncached parent receives no store writes, so a hit there would be a stale leftover masking the event. A vacated path (delete, rename away) keeps the last entry so unlinked-but-open reads still work. Directory builds version the completed directory at the listing snapshot and re-invalidate buffered events at that version, since their mid-build refresh ran against an incomplete store. The tests replay every race this replaces machinery for: rollback of a newer local flush (queued, cached, and read-through), stale leftovers under uncached parents, the build window including abort, handles opened after an event was queued, events landing mid-lookup, and undelivered events at remote-cache time across a filer failover. * filer: serialize the log position fence with mutations, stamp mutation acks The fence stamped before an unlocked entry read could precede state the read returned: a mutation writes storage first and assigns its event timestamp only at notify time, so a lookup racing that window handed the mount an entry newer than its fence, and the event's later delivery looked like fresh news — destroying dirty pages for a change the handle already had. The mutation handlers already hold an exclusive per-path lock across read, write, and notify; the lookup and remote-cache reads now take it shared around the stamp and the read, making the fence exact: everything at or below it is in the entry, nothing above it is. A no-change update returns success without an event, leaving the mount nothing to fence with even though the response confirms current state. Create and update acks now carry a log position stamped under the same lock, and the mount falls back to it whenever the ack has no event. Also regenerate the VT marshalers, which the earlier generation missed: without them a VT round-trip silently zeroed every log position. * java: sync filer.proto * mount: scope store versions to what they vouch for; atomic handle install The store's version cursor claimed too much. Advanced by local mutation acks and directory listing snapshots, it inflated the version of store reads for unrelated paths whose events the subscription still owed, and those events were then fenced out permanently. The cursor now tracks subscription progress only — events arrive in log order, so everything at or below it has been delivered for every path — and a completed listing records its snapshot as a per-directory floor instead of a global claim. Local acks never touch it: they version their own handle directly. Buffered build events advance the cursor at delivery, since their store write may never happen (abort) while their invalidation is already queued; their read-through directory pairs no store read with it, and rename fragments are applied first. Concurrent first opens raced: a slower opener's older lookup could overwrite the newer entry a faster opener had installed, while the monotonic version kept the newer timestamp — an old entry fenced at a new version, immune to every correcting event. Entry and version are now installed as one decision under the handle map lock, and an install that does not outrank the handle's version is dropped. The remote-cache commit also escaped the fence: it wrote storage and notified without the path lock, so a lookup's shared-locked fence and read could land between the two and hand out the cached state under-versioned. The commit now re-reads and writes under the exclusive path lock, and backs off entirely when the entry changed during the download — the concurrent writer supersedes the cached content. * mount: floors gate store applies; installs respect handle users; renames join the fence A directory floor certifies the listing state as of its snapshot, but a delayed event at or below the floor was still applied to the store — rolling the content back to pre-snapshot state while the floor kept claiming the snapshot version, so the correcting events were fenced out of every future read. Events are now gated against the affected directory's floor, each half of a rename independently. Fences are lower bounds: a listing or lookup can include a mutation whose event has not been delivered yet, and that event later passes every gate carrying state the handle already holds. Such a re-delivery now advances the version without destroying dirty pages or reinstalling the entry — invalidating local writes over a no-op was the real damage in every remaining under-fence window, including the unlocked listing snapshot, which no per-path lock can serialize. The concurrent-open install moved from the map lock to the handle lock every reader, writer, and invalidation synchronizes on, and rejects what cannot improve the handle: dirty state (local writes would be lost), unversioned lookup responses (they cannot outrank anything, and two zero-version opens must not overwrite each other), and anything not strictly newer. New handles are still fully initialized before the map exposes them. Renames committed metadata and emitted events with no path lock, so a lookup could read the renamed state under a fence preceding its events. Both rename handlers now hold the source and destination locks, ordered by path, across commit and notification; descendants of a renamed directory are not individually locked and rely on the no-op re-delivery handling above. * mount: per-entry store versions replace the cursor and directory floors The store's aggregate versions — a global subscription cursor and per-directory listing floors — were versions at coarser granularity than the values they described, and every over-claiming bug in this series traced to that gap: an aggregate vouching for state its source never saw. Each store entry now carries the filer log position of the write that produced it — the event that applied it, or the listing snapshot that inserted it, recorded in the store's key-value space under the same lock as the entry write. The store becomes what the handle already is: a last-writer-wins register with one rule, install only what outranks the current claim. The cursor, the floors, their advancement rules, the pairing ordering constraint, and the floor gating all collapse into that rule. Applies are gated per entry, each half of a rename independently; an unversioned local write clears the claim its content no longer proves; version records lingering after a bulk folder wipe cannot fence a recreate, since a claim only blocks while its entry exists. Listing inserts are stamped at build completion, before the buffered replay so newer replayed events override the stamp. Filer side, the fence dance every versioned read must perform is now a single choke point, fencedFindEntry, so a future read RPC gets the lock-serialized stamp by construction rather than by convention. * mount: judge no-op re-deliveries against an immutable base, not the live entry The equal-state skip compared the incoming event to the live handle entry, but local writes mutate the live entry — size, timestamps, chunks — so a delayed event re-delivering the base the handle was opened with no longer matched, and the installer destroyed the dirty pages and rolled the entry back over nothing new. The handle now keeps an immutable snapshot of the filer state it last installed or acknowledged, refreshed at every install and mutation ack (flush acks snapshot the request entry before the id mapping mutates it), and the no-op judgment runs against that base: an event carrying the base brings nothing, whatever the live entry has diverged to since. * mount: tombstones for versioned deletes, absence floors, copy enrollment Four gaps in the per-entry version protocol, all the same shape: a versioned fact with nothing carrying its version. A deletion is a fact about a path with no entry left to hold it — clearing the record let a delayed older event resurrect the deleted path, permanently, since the deletion's own redelivery is dedup-suppressed. Versioned deletes now leave a tombstone record that fences without an entry; renames tombstone their source the same way. Plain records still only block while their entry exists, so records lingering after a bulk folder wipe cannot fence a recreate. A completed listing proves absences as well as presences: a name it omitted was deleted as of the snapshot, and a delayed create below the snapshot re-creates it. The snapshot is kept per directory strictly as an absence fence, consulted only when a path has neither an entry nor a version record — present entries carry their own versions and never touch it, which is what separates this from the over-claiming floor it replaces. A rebuild against a pre-upgrade filer returns no snapshot; stamping now clears the children's records in that case, so a reinserted entry cannot reactivate the stale claim its previous incarnation left behind and reject valid events below it. Server-side copies installed the copied entry without enrolling in the base protocol, so the copy's own event differed from the stale pre-copy base and destroyed writes made to the destination after the copy. The install now refreshes the base and takes its version from the fenced readback. * mount: deletion facts outlive the cache's knowledge of the entry A versioned delete of a path the store held no entry for recorded nothing, so a delayed older event recreated the path — permanently, with the deletion's redelivery dedup-suppressed. The tombstone is now written whenever a versioned event vacates a path: the deletion is a fact about the path, not about what this cache happened to hold. For an absent entry, the listing's absence floor now speaks whatever older record remains: a tombstone at one position does not exhaust what is known about the path when a newer snapshot has confirmed the name still absent, and an event between the two was slipping past both. A committed copy whose readback failed installed a synthesized base with local timestamps; the copy's real event legitimately differs from it, and was read as foreign state — destroying writes made to the destination after the copy. The handle now marks that its own event is en route and adopts that event's state as the base without touching the live entry or the dirty pages; the adoption is one-shot, so a genuinely foreign event still invalidates. * mount: authoritative acks cancel pending event adoption; tombstones scoped and pruned The copy-event adoption flag could outlive its purpose: a flush after the failed readback installs a newer base and advances the version, the copy's own event is then version gated without consuming the flag, and the next genuinely foreign event was silently adopted — base advanced, live entry and dirty pages untouched — leaving the mount to later overwrite that remote change. Every local acknowledgment now installs its base through one helper that also cancels any pending adoption: the ack supersedes the mutation the adoption was waiting for. Tombstones were written for every versioned delete under the mount and survived directory eviction by design, growing LevelDB with historical deletions on delete-heavy mounts. They are now scoped to directories whose cached state the fence actually protects — an uncached parent never serves from the store nor applies the resurrecting insert — and a completed listing prunes the direct-child tombstones its absence floor supersedes, leaving only those above the snapshot. The store gains a key-prefix visitor for the sweep. * mount: acked saves install their value; trailer snapshots; direct-child prune range A version must never advance without its value. saveEntry stamped any open handle with the acknowledgment's version, but a handle opened while the save was in flight holds the pre-mutation entry — stamping it fenced out the events carrying the state it lacked, permanently, with the local apply performing no invalidation and the redelivery deduplicated. The acknowledged entry is now installed together with its version, through the same guarded install the racing-open path uses: under the handle lock, only when it outranks the handle, never over dirty local writes. Empty listings return no in-band snapshot — a snapshot-only response would be read as an entry by older consumers — so directories that end empty gained no absence floor and their tombstones were never pruned. The filer now sends the snapshot in the stream trailer, which older clients ignore, and the client reads it when no in-band snapshot arrived. Empty directories get real floors, their tombstones prune, and their buffered replays gain the snapshot filter instead of the replay-all fallback. Version records now encode the parent directory and name separated by a NUL, making a directory's direct children one contiguous key range: the tombstone prune scans exactly them under the cache lock, instead of walking every descendant record — the whole store, for root. * mount: fix dirty-page loss, uid/gid base, download race, copy adopt, leak; dedup Correctness fixes from the versioned-invalidation review: - A foreign delete/rename-away of a file held open with unflushed local writes destroyed the dirty pages unconditionally. A process may keep writing to an unlinked-but-open file and those writes were already acknowledged; preserve the pages when the handle is dirty. - downloadRemoteEntry stored the handle's base with filer-side uid/gid while every candidate it is later compared against is in local form, so under a non-identity UidGidMapper an unchanged re-delivery looked foreign and force-destroyed dirty pages. Map the base to local. - downloadRemoteEntry wrote the entry/base/version triple under only the handle's shared lock, so two concurrent reads of the same remote-only file could tear it. Serialize the install with a dedicated mutex (invalidation is already excluded by the exclusive handle lock). - A committed server-side copy whose readback failed adopted the FIRST event past the version gate as its base; a foreign write delivered first was silently swallowed. Adopt only an event whose content matches the synthesized base — the copy's own event — and install any other normally. - The deferred-create path relied on AcquireFileHandle installing the passed entry on a pre-existing handle, which the version rework dropped. Restore that install in the compat wrapper; the versioned open path keeps its gated install. Growth and hot-path cost: - Per-entry version records and tombstones leaked when a directory was evicted or read-through without a rebuild. An uncached directory gates its own inserts, so its records fence nothing; clear a directory's child version records when it is wiped for eviction. - FindEntry paid for the version KvGet on every lookup/getattr cache hit and threw it away. FindEntry now reads only the entry; the hot lookupEntry cache-hit path skips the version entirely. Cleanups: - Extract ackVersionTsNs over the shared response interface, replacing the metadata-event-else-log-ts snippet copy-pasted at four ack sites. - Extract acquireRenamePathLocks, replacing the verbatim sorted two-path lock fence in both rename handlers. * mount: no resurrection on foreign delete, version no-event acks, gate downloads, tighten copy adopt Follow-ups to the review patches: - Preserving dirty pages on a foreign delete let the next flush pass the isDeleted guard and CreateEntry, resurrecting the remotely-unlinked name. Mark the handle deleted in the vacate branch: the open fd can still read its buffered writes, but a flush no longer recreates the file. - A no-event acknowledgment (log fence only) synthesized a metadata event with TsNs 0, so the cache stored the entry unversioned and an older subscriber event rolled it back. Stamp the synthesized event with the ack's log position at all four ack sites. - downloadRemoteEntry serialized its install but did not check the version, so an older response arriving last overwrote the entry/base while the monotonic version kept the newer value, fencing corrections out. Install only when the response is at least as new as the handle. - sameEntryContent compared only size and chunks, so a foreign chmod with unchanged content was adopted as the copy's own event. Compare everything except server-assigned timestamps, so a metadata-only foreign change installs instead. * mount: trim comments to the non-obvious why The versioning work accumulated multi-line comment blocks restating what the code says. Keep the constraint a reader cannot derive — why a fence is exact, why a version must not advance without its value, why an uncached parent's records fence nothing — and drop the rest. * mount: distinguish rename from delete, tighten the download and adopt gates - A rename emits a nil old-path invalidation just like an unlink, so the vacate branch marked the handle deleted and later writes through the already-open descriptor were skipped instead of persisted. Carry the delete/rename distinction on the invalidation and mark only an actual delete. - The remote-download install accepted an unversioned response regardless of the handle's version, so during a rolling upgrade a delayed response could install stale content under a newer version. Require the response to be at least as new, with one exception: a handle still lacking local chunks takes the content anyway — it cannot read without it — but does not claim the response's log position. - Copy-event adoption returned without installing, so a foreign touch arriving before the copy's own event lost its timestamps. Content is unchanged either way, so the dirty pages stay valid; a clean handle now takes the entry, while a dirty one keeps its diverged version. * mount: one directory floor instead of a record per child; agree on TTL Review feedback: - Build completion wrote one KV record per direct child inside the cache write lock, so a large directory stalled every other cache operation for O(children) store writes. The directory's listing snapshot already covers every child it saw; make that floor the version for any child without a record of its own, and a child earns a record only when a later event touches it. One map write per build replaces the per-child writes, with the same fencing. - The presence probe read the store directly and so counted a TTL-expired entry as present, judging the path by a record describing content that has logically vanished. It now applies the same expiry the read path does, and an expired path falls back to its directory floor. - Preserve ErrNotFound identity when the commit-time re-read finds the object deleted, so callers still surface a 404. - Assert the rename-away source fence timestamp in the invalidation test. Also record the tombstone ceiling: distinct deleted names in a cached directory accumulate until it is rebuilt or evicted, which prunes everything at or below the new snapshot. * mount: pin the fence's clock domain instead of letting skew decide A log-position fence is stamped by one filer's clock under that filer's in-process lock, so comparing it to an event another filer logged is comparing two unrelated clocks. The two error directions are not equally costly: applying an event the fence already covered is a re-apply the base-equality check absorbs, while skipping one it does not cover leaves the handle holding exactly the state the event was meant to correct, with the subscription cursor already past it — the unhealable staleness this whole PR exists to remove. So refuse to guess. Fences now carry the signature of the filer that stamped them, and a handle records it alongside the position. An event is only fenced out when the filer that logged it is the one that stamped the fence — the logging filer appends its own signature, so its presence identifies the clock domain. Events from any other filer are applied. Positions taken from events keep comparing as before; the subscription already delivers those in order. The invalidation callback takes a struct now: it carries the path, entry, position, delete/rename distinction, and signatures, and was about to need a fifth positional parameter. * mount: follow a foreign rename; key page invalidation on content, not equality - A rename's old-path invalidation now carries the destination, and the handle follows the file there: an open fd tracks the inode, and leaving it on the old path made its next flush recreate that name instead of updating the renamed file. - Dirty pages overlay content, so only a content change invalidates them. Keying that on exact equality meant any timestamp-only event destroyed them, which the copy-adoption marker existed to paper over — a foreign touch could consume the marker and leave the copy's own event to drop the post-copy writes. Comparing content instead makes the marker unnecessary, so it is gone: a metadata-only event keeps the overlay, and a dirty handle keeps its diverged entry unless foreign content supersedes it. - A remote download response that is merely older is now refused even when the handle still lacks chunks; only an unversioned one is taken (and claims no position), since an older response's content predates what the handle reflects. - A refused or unversioned download no longer publishes to the metadata cache, where a zero-position event would clear the entry's version and let an older subscriber event roll the cache back. * mount: page invalidation keys on content alone; unversioned writes claim no position - sameEntryContent compared everything but timestamps, so a foreign chmod, chown, or xattr change counted as a content change and destroyed the dirty-page overlay. It was strict only to serve the copy-adoption marker, which is gone; its one caller now asks the question it actually needs — did the bytes change — so metadata-only events leave the overlay alone. - A rename over an existing file destroys that file, but its open handle was left live and still pointed at the name the renamed source now occupies, so its flush could overwrite it. MovePath already reports the displaced inode; mark that handle deleted. - An acknowledgment was refused whenever its position was numerically lower, even when a different filer stamped the fence it lost to. Two known, differing signatures mean unrelated clocks, so the comparison no longer applies there; unknown signatures still compare as before. - A local write with no log position behind it now records that explicitly instead of deleting its version record. Absence means the directory listing covers the path, which is why the snapshot floor applies; local content the listing never saw must not inherit it, or the events that would correct it are fenced out. * mount: widen the existing lookup functions instead of forking WithVersion twins The versioning work grew a parallel function for every accessor that needed to return a log position — lookupEntryWithVersion beside lookupEntry, maybeLoadEntryWithVersion beside maybeLoadEntry, FindEntryWithVersion beside FindEntry, AcquireFileHandleWithVersion beside AcquireFileHandle, advanceEntryVersion beside advanceEntryVersionTsNs, plus a getPbEntryWithVersion wrapper and an InsertListedEntriesForTest hook. Two names for one operation is two places to keep in step, and the split let callers pick the one that happened to compile. Each pair is now the single original name carrying the position, with callers that do not want it discarding it. filer_pb.GetEntry returns the fence its response already carried rather than a mount-side wrapper re-issuing the lookup, and InsertEntry takes the position its content reflects rather than a test-only twin that inserted without one. The one behavioural knot the merge exposed: AcquireFileHandle had been installing the entry on a pre-existing handle only in its unversioned form, which conflated 'the caller is authoritative' with 'the lookup had no version'. Deferred create is the only caller that means the former, so it now installs explicitly and the map function just acquires. |
||
|
Chris Lu
|
fe0a357624 |
exclusive_locks: clear renew-running flag before dropping isLocked (#10413)
On renewal failure the renew goroutine stored isLocked=false before its deferred renewGoroutineRunning.Store(false) ran. A concurrent RequestLock interleaving there reacquires the lease (sees isLocked=false), sets isLocked=true, then its CompareAndSwap on renewGoroutineRunning fails because the old goroutine's flag is still set — so no replacement renewer starts. The lock is then held locally with nothing renewing it, and silently expires on the master after the lease TTL, admitting a second holder. Clear renewGoroutineRunning before isLocked on the failure path so the reacquire path always starts a fresh renewer. Builds and vets clean; no behavior change on the success path. Claude-Session: https://claude.ai/code/session_01Ks16jnt4S7gdDk8cheQ3xu |
||
|
Chris Lu
|
2e9b944e5c |
test(s3): aim collection force-delete at the master the suite actually runs (#10404)
The copying and tagging tests force-drop each bucket's collection at the master so volume slots are freed deterministically between tests. But the copy-tests CI job runs its master on 9336 and the tagging Makefile on 9338, while the tests default to 9333 — the cleanup dialed a dead port and quietly no-oped. Each test bucket then grows 7 volumes against -volume.max=100, and whenever async deletion lagged the data node ran out of slots and PutObject 500ed with "No writable volumes and no free volumes left". Set MASTER_ENDPOINT where the master port is non-default: the copy-tests workflow step, and the copying/tagging Makefiles (derived from MASTER_PORT). |
||
|
Chris Lu
|
5731f37a2f |
telemetry: validate reports on the collect endpoint (#10401)
* telemetry: validate reports on the collect endpoint /api/collect is anonymous, so reports can't be authenticated, but a real master can't produce a non-UUID topology_id, a version outside N.NN(-enterprise), an unknown GOOS/GOARCH, or absurd counts — reject those to keep casual junk out of the collected data, and cap the request body at 4 KB. * telemetry: integration test fixtures pass collect validation The test's topology id and version were exactly the junk shapes the new validation rejects; use a UUID and a plain version number. |
||
|
Chris Lu
|
8e8b4c4f34 |
telemetry: confirmed-cluster stats (2+ distinct days) (#10402)
telemetry: confirmed-cluster stats Count a cluster as confirmed once it has reported on >=2 distinct UTC days (per-cluster history makes this a length check). Version/OS distributions in /api/stats are computed over confirmed clusters, so a one-shot injected report can't appear in them; falls back to all active clusters while no confirmed ones exist (fresh server). Adds the seaweedfs_telemetry_confirmed_clusters gauge and a dashboard card. |
||
|
Chris Lu
|
de3ad8db12 |
telemetry: per-cluster usage history on the built-in dashboard (#10400)
telemetry: per-cluster usage history Keep one compact sample per cluster per UTC day (disk bytes, volume count, volume servers), retained for -max-age and persisted in the state file. Serve it at /api/history?cluster_id=...&days=90 and add a per-cluster lookup with disk/volume charts to the built-in dashboard. |
||
|
Chris Lu
|
3e9154def2 |
telemetry: persist server state across restarts (#10399)
* telemetry: persist server state across restarts The telemetry server kept the instance map and Prometheus gauges only in process memory, so every deploy or restart reset all collected metrics until clusters re-reported over the next 24h. Snapshot the instance map to a JSON state file (atomic tmp+rename) on a debounced interval and on SIGTERM, and restore it on startup, preserving received_at so the cleanup and active-cluster windows stay correct. Defaults to data/telemetry-state.json, which the deployed systemd unit's WorkingDirectory already provides; -state-file='' disables. * telemetry: keep instances for 90 days by default With state now persisted across restarts, a longer retention default is meaningful; raise -max-age from 30 to 90 days so per-cluster data survives long enough for quarterly views. |
||
|
Chris Lu
|
5457c5b5ed |
ci: fix Deploy Telemetry Server build (nested telemetry/server module) (#10398)
ci: build telemetry server from its own module in deploy workflow telemetry/server has had its own go.mod since #9924, so building ./telemetry/server/main.go from the repo root fails with 'no required module provides package'. Build from within the module instead. |