mirror of
https://github.com/seaweedfs/seaweedfs.git
synced 2026-10-19 20:44:28 +02:00
Chris Lu
6a594da739
* filer: free shared chunks when a folder delete drops the last hard link doBatchDeleteFolderMetaAndData skipped chunk collection for every hard-linked child, relying on maybeDeleteHardLinks to retire the shared inode record. When the delete removes the last link the record is gone and nobody else will ever free those chunks. After the link callback runs, check whether the record survived; if it did not, queue the entry's chunks for deletion. * filer: free orphaned hard-link chunks in the replayed delete sweep The subtree sweep on a replayed directory delete skipped every hard-linked child, so a link that was the inode's last owner leaked its shared chunks. Check the inode record after the store delete; when it is gone, queue the child's chunks like any other swept entry. * filer: mirror remote-object cleanup in the replayed delete sweep Children a lagging peer alone still holds under a replayed directory delete were dropped with plain store deletes: their remote objects leaked and lazy remote fetches could resurrect them. The sweep now records event-stamped tombstones and deletes swept non-directory children through maybeDeleteFromRemoteIfUnchanged, which carries the same stat guard the write-back daemon's processDeleteEvent applies plus a modification-time check — a delayed replay cannot erase an object another writer replaced, even a same-size one. A failed remote delete fails the replay so the event is retried instead of orphaning the object. The directory's own tombstone comes from the event pipeline at the event timestamp and covers the subtree. Metadata-only deletes and cross-cluster events keep the origin's semantics. * filerstore: return real errors from KvGet on MongoDB, ArangoDB, Elastic These stores mapped every lookup failure to ErrKvNotFound, so a database error looked identical to a missing key: hard-link records could appear gone while links still owned their chunks, and path timestamp reads could treat committed state as absent. Return the underlying error and keep ErrKvNotFound for genuinely missing keys. * filer: bind remote-object deletes to the observed object version A guarded stat-then-delete still lets a replacement landing between the two requests go down with the deleted object. Add an optional conditional-delete operation to remote storage clients — S3 If-Match on the live ETag, GCS generation match, Azure IfMatch — and use it everywhere a delete is guarded by a stored stamp: the write-back daemon's delete, rename-source, and pending-rename paths, and the replayed-delete sweep. Backends without it keep the guarded stat-then-delete. * filer: restamp the remote entry after a metadata-only remote update Changing Content-Encoding rewrites the object in place on S3 and advances its LastModified, while the recorded stamp keeps the old mtime. A version-checked delete would then mistake the object for a replacement and keep it forever. Re-stat after UpdateFileMetadata so the saved stamp describes the live object. * filer: mirror the remote-delete ownership checks in the replayed sweep The sweep deleted swept children's remote objects on the listing's snapshot alone. Two gaps: a remote-only entry that took the path since — one a commit record may not cover, such as a lazy fetch materialized from the remote — reads the object as its only copy, and an entry removed while its rename copy was pending leaves the object at the source key orphaned. The per-child delete now runs under the child's entry lock with the commit-time check repeated, keeps a remote-only entry that holds the path, and releases the pending rename's source under the same ownership and stamp conditions the write-back daemon's releasePendingSource applies. * remote.sync: restamp only the object the metadata update touched The live stat after UpdateFileMetadata was saved unconditionally, so a replacement written between the update and the stat stamped the entry with an object it never described — and a later delete would then pass the freshness guard on the replacement's data. The restamp now requires the live object to still match the stored stamp on size and ETag; a different object keeps the old stamp, which the delete guard reads as replaced and keeps. * filer: skip unbudgeted reader-cache test on 32-bit TestReaderCacheUnboundedWithoutBudget proves the reader cache has no implicit memory budget by keeping 96 concurrent 4 MiB downloads in flight (>256 MiB). On linux/386 the weed/filer test binary already runs near the 4 GiB address-space limit, so the burst reliably crashes it with "out of memory" mid-suite. * filer: bail out of a swept child lock instead of waiting on it Replay already holds the event's path locks when the sweep reaches a descendant. A local rename can hold that descendant while waiting on one of the replayed paths, so blocking on the child lock deadlocks both. The sweep now tries the lock and reports ErrLockContended — transient, so the replication retry replays the event once the local write settles. * remote: report the stamp a metadata update produced A metadata write can rewrite the object it touches — S3 changes Content-Encoding through an in-place copy, which can mint a new ETag — so restamping from a fresh stat can reject the very object the update produced, or describe a replacement that landed in between. UpdateFileMetadata now returns the RemoteEntry of the object it wrote (S3 from the copy result, GCS from the update attributes, Azure from a post-update stat); nil means the object was untouched and callers keep the stored stamp. * remote: restamp Azure from the metadata write's own etag
##arangodb
database: https://github.com/arangodb/arangodb go driver: https://github.com/arangodb/go-driver
options:
[arangodb]
enabled=true
db_name="seaweedfs"
servers=["http://localhost:8529"]
#basic auth
user="root"
pass="test"
# tls settings
insecure_skip_verify=true
i test using this dev database:
docker run -p 8529:8529 -e ARANGO_ROOT_PASSWORD=test arangodb/arangodb:3.9.0
database structure
arangodb has a few restrictions which require the use of a few tricks in order to losslessly store the data.
filer store
arangodb does not support []byte, and will store such as a uint64 array. this would be a waste of space. to counteract this, we store the data as a length prefixed uint64 byteset.
filer kv
same as above
filer buckets
s3 buckets are implemented through arangodb collection. this allows us to do very fast bucket deletion by simply deleting the collection
arangodb collection name rules is character set azAZ09_- with a 256 character max. however the first character must be a letter.
s3 bucket name rule is the set azAZ09.- with a 63 characters max.
the rules for collection names is then the following:
- if the bucket name is a valid arangodb collection name, then nothing is done.
- if the bucket name contains a ".", the "." is replaced with "_"
- if the bucket name now begins with a number or "_", the prefix "xN--" is prepended to the collection name
this allows for these collection names to be used.
features i don't personally need but are missing
[ ] provide tls cert to arango [ ] authentication that is not basic auth [ ] synchronise endpoint interval config [ ] automatic creation of custom index [ ] configure default arangodb collection sharding rules [ ] configure default arangodb collection replication rules
complexity
ok, so if https://www.arangodb.com/docs/stable/indexing-index-basics.html#persistent-index is correct
O(1)
- InsertEntry
- UpdateEntry
- FindEntry
- DeleteEntry
- KvPut
- KvGet
- KvDelete
O(log(BUCKET_SIZE))
- DeleteFolderChildren
O(log(DIRECTORY_SIZE))
- ListDirectoryEntries
- ListDirectoryPrefixedEntries