100 Commits
Author SHA1 Message Date
Konstantin LebedevandKonstantin Lebedev 9386a25a4a feat(shell): parallelize volume balance moves (#10737)
* chore: volume.balance parallelization

* chore: volume.balance add ioBytePerSecond

---------

Co-authored-by: Konstantin Lebedev <whitefox@mayflower.work>
2026-08-13 19:11:38 -07:00
d1f503181b [s3] force filer apply s3 expiry metadata (#10469)
* fix: apply S3 Expiry Metadata

* add test Header X-Seaweedfs-Expires-S3

* resolve comments

* test entry lookup by mtime

* filer: skip s3 expiry stamp on versioned entries

The s3 expiry path skips entries carrying a version id, so stamping one
takes away its expiry rather than moving it onto mtime. Files under
.versions/ are written once, so crtime already tracks their needles.

---------

Co-authored-by: Konstantin Lebedev <whitefox@mayflower.work>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-08-05 23:55:20 -07:00
292abfae33 [filer] applyStorageDefaultsToEntry before CreateEntry (#10196)
* applyStorageDefaultsToEntry befor CreateEntry

* Update weed/server/filer_grpc_server.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Update weed/server/filer_grpc_server.go

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* add tests

* fix: tests

* enforce read-only storage rule regardless of explicit TTL, match CreateEntry remote handling

* CreateEntry shares applyStorageDefaultsToEntry

* routed PUT enforces the path rule's max file name length

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Konstantin Lebedev <whitefox@mayflower.work>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-07-02 11:56:49 -07:00
Konstantin Lebedev df833d485f [test] update docker image for s3test (#9811) 2026-06-03 09:45:00 -07:00
Konstantin LebedevandChris Lu 3537312045 [docker] add make test_keycloak_s3 for local develop and debug (#9719)
* add make test_keylock_s3 for local develop and debug

* fix typos

* add condition oidc:azp

* docker: reuse test/s3/iam realm and iam config for keycloak dev compose

Point the keycloak dev compose at the existing test/s3/iam configs instead
of a parallel realm/port/key/role set. Adds one declarative realm import
(seaweedfs-test-realm.json) as the single realm source and drops the
duplicated iam.json/s3.json.

---------

Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-05-28 13:39:32 -07:00
Konstantin Lebedev 7d1b16fbcd fix: ListBucketsHandler for pathStyleDomains (#9510) 2026-05-15 13:12:55 -07:00
Konstantin Lebedev 01b3125815 [shell]: volume balance capacity by min volume density (#8026)
volume balance by min volume density and active volumes
2026-02-19 13:30:59 -08:00
dfac1315ca fix: filer do not support IP whitelist right now #7094 (#7095)
* fix: filer do not support IP whitelist right now #7094

* Apply suggestion from @gemini-code-assist[bot]

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-12-10 14:35:59 -08:00
Konstantin Lebedevandchrislu 4477edbcc4 fix: pass proxied query param (#7477)
* fix: pass proxied query param

* fix: use math/rand/v2

* Shuffle condition

---------

Co-authored-by: chrislu <chris.lu@gmail.com>
2025-11-13 17:25:47 -08:00
Konstantin Lebedevandchrislu 084b377f87 do delete expired entries on s3 list request (#7426)
* do delete expired entries on s3 list request
https://github.com/seaweedfs/seaweedfs/issues/6837

* disable delete expires s3 entry in filer

* pass opt allowDeleteObjectsByTTL to all servers

* delete on get and head

* add lifecycle expiration s3 tests

* fix opt allowDeleteObjectsByTTL for server

* fix test lifecycle expiration

* fix IsExpired

* fix locationPrefix for updateEntriesTTL

* fix s3tests

* resolv  coderabbitai

* GetS3ExpireTime on filer

* go mod

* clear TtlSeconds for volume

* move s3 delete expired entry to filer

* filer delete meta and data

* del unusing func removeExpiredObject

* test s3 put

* test s3 put multipart

* allowDeleteObjectsByTTL by default

* fix pipline tests

* rm dublicate SeaweedFSExpiresS3

* revert expiration tests

* fix updateTTL

* rm log

* resolv comment

* fix delete version object

* fix S3Versioning

* fix delete on FindEntry

* fix delete chunks

* fix sqlite not support concurrent writes/reads

* move deletion out of listing transaction; delete entries and empty folders

* Revert "fix sqlite not support concurrent writes/reads"

This reverts commit 5d5da14e0e.

* clearer handling on recursive empty directory deletion

* handle listing errors

* strut copying

* reuse code to delete empty folders

* use iterative approach with a queue to avoid recursive WithFilerClient calls

* stop a gRPC stream from the client-side callback is to return a specific error, e.g., io.EOF

* still issue UpdateEntry when the flag must be added

* errors join

* join path

* cleaner

* add context, sort directories by depth (deepest first) to avoid redundant checks

* batched operation, refactoring

* prevent deleting bucket

* constant

* reuse code

* more logging

* refactoring

* s3 TTL time

* Safety check

---------

Co-authored-by: chrislu <chris.lu@gmail.com>
2025-11-05 22:05:54 -08:00
Konstantin Lebedev d19eca71eb [master] vaccum fix warn (#7312) 2025-10-28 17:57:51 -07:00
bf58c5a688 fix: Use a mixed of virtual and path styles within a single subdomain (#7357)
* fix: Use a mixed of virtual and path styles within a single subdomain

* address comments

* add tests

---------

Co-authored-by: chrislu <chris.lu@gmail.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2025-10-24 01:45:22 -07:00
Konstantin Lebedev d019848018 fix: pass inflightDownloadDataTimeout to volumeServer (#7206) 2025-09-08 09:40:40 -07:00
Konstantin Lebedevandchrislu fd4154cfed fix: s3 return BadDigest (#6714)
* fix: s3 return BadDigest

* adjust error message checking

---------

Co-authored-by: chrislu <chris.lu@gmail.com>
2025-07-06 14:18:57 -07:00
93007c1842 [volume] refactor and add metrics for flight upload and download data limit condition (#6920)
* refactor concurrentDownloadLimit

* fix loop

* fix cmdServer

* fix: resolve conversation pr 6920

* Changes logging function (#6919)

* updated logging methods for stores

* updated logging methods for stores

* updated logging methods for filer

* updated logging methods for uploader and http_util

* updated logging methods for weed server

---------

Co-authored-by: akosov <a.kosov@kryptonite.ru>

* Improve lock ring (#6921)

* fix flaky lock ring test

* add more tests

* fix: build

* fix: rm import util/version

* fix: serverOptions

* refactoring

---------

Co-authored-by: Aleksey Kosov <rusyak777@list.ru>
Co-authored-by: akosov <a.kosov@kryptonite.ru>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
2025-07-02 18:03:49 -07:00
Konstantin Lebedev 958d88cb85 [shell] volume copy add param noLock (#6871) 2025-06-16 07:39:19 -07:00
Konstantin Lebedev c8b0a04409 golang up version to 1.24 (#6724) 2025-04-23 15:49:28 -07:00
Konstantin Lebedev 8c1cb203fa [s3] fix checksum algorithm CRC64NVMe (#6722) 2025-04-23 07:31:03 -07:00
Konstantin Lebedev aa9121aa9d Revert "[filer] void panic if s3opt.tlsCACertificate is nill" (#6694)
Revert "[filer] void panic if s3opt.tlsCACertificate is nill (#6685)"

This reverts commit 4dca59ed27.
2025-04-07 11:09:40 -07:00
Konstantin Lebedev 4dca59ed27 [filer] void panic if s3opt.tlsCACertificate is nill (#6685)
* fix issues/6684

* add s3.cacert.file option for filer
2025-04-07 09:45:49 -07:00
Konstantin Lebedev 0a4b1909a2 [shell] only apply the balancing for writable volumes (#6346) 2024-12-13 01:10:00 -08:00
Konstantin Lebedev 700b95304b [shell] volume.list show only writable volumes (#6338)
* show only writable volumes

* fix import
2024-12-11 09:06:15 -08:00
Konstantin Lebedev c37281735e volume.list avoid output empty data center and rack and disk info (#6341) 2024-12-11 09:03:16 -08:00
Konstantin Lebedev ff1392f7f4 [shell] use constant for hdd of type (#6337)
use constant for hdd of type
2024-12-10 08:43:59 -08:00
Konstantin Lebedev b65eb2ec45 [security] reload whiteList on http seerver (#6302)
* reload whiteList

* white_list add to scaffold
2024-12-02 10:38:10 -08:00
Konstantin Lebedev e2e97db917 [master] avoid timeout when assigning for main request with filter by DC or rack (#6291)
* avoid timeout when assigning for main request with filter by DC or rack

https://github.com/seaweedfs/seaweedfs/issues/6290

* use constant NoWritableVolumes
2024-11-26 08:33:31 -08:00
Konstantin Lebedev fec88e64eb [master] update LastLeaderChangeTime for hashicorp raft (#6292) 2024-11-26 08:02:45 -08:00
Konstantin Lebedev 8836fa19b6 use ShouldGrowVolumesByDcAndRack (#6280) 2024-11-25 09:30:37 -08:00
Konstantin Lebedev 167b50be88 fix missing register master metric MasterPickForWriteErrorCounter (#6277) 2024-11-25 08:59:11 -08:00
Konstantin Lebedev 7fa4e5c2a1 [master] Revert back automatic volume grow in case of compensate for crowded volumes (#6267)
brought back the Automatic Volume Grow to compensate for crowded volumes
2024-11-21 08:19:19 -08:00
Konstantin Lebedev a143c888e5 [shell] don't require lock when there are no changes for volume.fix.replication (#6266)
* don't require lock when there are no changes

* revert takeAction
2024-11-21 08:17:25 -08:00
Konstantin Lebedev a49d9e020c [master] avoid crowded more writable for auto grow (#6214)
avoid crowded more writable
https://github.com/seaweedfs/seaweedfs/issues/6121
2024-11-21 00:57:42 -08:00
Konstantin Lebedev 254ed8897e [shell] add noLock param for volume.move (#6261) 2024-11-20 08:35:24 -08:00
Konstantin Lebedev 9a5d3e7b31 [shell] add admin noLock for balance (#6209)
add admin noLock for balance
2024-11-05 19:09:22 -08:00
Konstantin Lebedev 5bddf0c085 [shell] volume.balance collect volume servers by dc rack node (#6191)
* chore: balance by rack

* fix: rm check lock

* fix: selected racks

* fix: selected nodes

* fix: containts

* fix: one collectVolumeServersByDcRackNode

* fix: revert lock and add lock

* fix: panic test

* revert noLock
2024-11-03 11:08:45 -08:00
Konstantin LebedevandChris Lu 44b275879b [s3] add {Get,Put,Delete}BucketTagging and PublicAccessBlock Handlers (#6088)
* add {Get,Put,Delete}BucketTagging Handlers

* s3 add skip bucket PublicAccessBlock handlers

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2024-10-04 10:59:14 -07:00
Konstantin Lebedev a3556acd98 [s3] add skip bucket encryption handlers (#6091)
s3 add skip bucket encryption handlers
2024-10-03 08:10:43 -07:00
Konstantin Lebedev bae93e611d [filer] avoid 500 if table doesn't exist (#6075) 2024-09-27 07:15:54 -07:00
Konstantin Lebedev 6d4f5c94ac [volume] remove truncate idx file if size not healthy (#6043) 2024-09-19 01:04:08 -07:00
Konstantin Lebedev 2b13d2c083 [filer] fix return 204 for DELETE on entry not found (#6023)
fix return 204 for DELETE on entry not found
2024-09-16 23:11:03 -07:00
Konstantin Lebedev 34bbaa2cdd [master] process grow request with must grow (#5999)
process grow request with must grow
2024-09-09 23:45:02 -07:00
Konstantin Lebedev 15965f7c54 [shell] fix volume grow in shell (#5992)
* fix volume grow in shell

* revert add Async

* check available volume space

* create a VolumeGrowRequest and remove unnecessary fields
2024-09-09 11:42:56 -07:00
Konstantin Lebedev 16984ef07a [s3] reducing the number of metrics for 403 (#5961)
reducing the number of metrics
2024-09-04 11:06:54 -07:00
Konstantin Lebedev 67a252ee8a [master] refactor func ShouldGrowVolumes (#5884) 2024-09-04 08:16:44 -07:00
Konstantin Lebedev 3e6ca6e706 [webdav] status code 500 if internal error from filer (#5865) 2024-08-06 07:39:52 -07:00
Konstantin Lebedev ef4c555e90 [shell] volume.grow (#5855) 2024-08-05 09:22:17 -07:00
Konstantin Lebedev b2ffcdaab2 [master] do sync grow request only if absolutely necessary (#5821)
* do sync grow request only if absolutely necessary
https://github.com/seaweedfs/seaweedfs/pull/5819

* remove check VolumeGrowStrategy Threshold on PickForWrite

* fix fmt.Errorf
2024-07-30 13:21:35 -07:00
Konstantin Lebedev e1f9336688 [filer] fix panic filer on webdav.maxMB (#5822)
fix panic filer on webdav.maxMB
2024-07-27 13:40:22 -07:00
Konstantin Lebedev 67edf1d014 [master] Do Automatic Volume Grow in background (#5781)
* Do Automatic Volume Grow in backgound

* pass lastGrowCount to master

* fix build

* fix count to uint64
2024-07-16 08:03:40 -07:00
Konstantin Lebedev a53e406c99 [master] refactor HasGrowRequest to atomic bool (#5782)
refactor HasGrowRequest to atomit bool
2024-07-15 10:51:21 -07:00
Konstantin Lebedev 33964fa292 metrics stats of volume layout depends on the data center (#5775)
stats volume layout depends on the data center
2024-07-12 12:32:25 -07:00
Konstantin Lebedev 04f4b10884 fix: avoid timeout if datacenter does not exist in topology (#5772)
* fix: avoid timeout if datacenter does not exist in topology

* fix: error msg

* fix: rm dublicate check

* fix: compare

* revert minor change
2024-07-12 11:19:08 -07:00
Konstantin Lebedev 9c440d472c [s3] revert skip deletion error, since the error file was not found is already skipped on the side of the grpc function (#5760)
* revert skip deletion error, since the error file was not found is already skipped on the side of the grpc function

* fix response error

* fix test_lifecycle_get

* Revert "fix test_lifecycle_get"

This reverts commit 8f991bdcf9.
2024-07-10 08:47:43 -07:00
Konstantin Lebedev f77eee667d add s3test for sql (#5718)
* add s3test for sql

* fix test test_bucket_listv2_delimiter_basic for s3

* fix action s3tests

* regen s3 api xsd

* rm minor s3 test test_bucket_listv2_fetchowner_defaultempty

* add docs

* without xmlns
2024-07-04 11:00:41 -07:00
Konstantin Lebedev 5ffacbb6ea refactor all methods strings to const (#5726) 2024-07-01 01:00:39 -07:00
Konstantin Lebedev 34165013c8 chore: hot fix panic fs.verify (#5696) 2024-06-20 15:24:31 -07:00
Konstantin Lebedev 7988ee0805 [fs.verify] skip failed files if entry not found on filerStore (#5693) 2024-06-19 06:50:55 -07:00
Konstantin Lebedev ece018b1a9 [fs.verify] getmeta data from filer log (#5678)
* initial

* fix: print entry path
2024-06-18 22:56:47 -07:00
Konstantin Lebedev d389c5b27e fix: recreate index include deleted files (#5579)
* fix: recreate index include deleted files
https://github.com/seaweedfs/seaweedfs/issues/5508

* fix: counting the number of files

* fix: log
2024-05-12 11:31:34 -07:00
Konstantin Lebedev 6f9267ebdf fix: skip file size checking for volume with TTL (#5541)
fix: skip file size checking for volume with tll
https://github.com/seaweedfs/seaweedfs/issues/5508
2024-04-28 00:21:57 -07:00
Konstantin Lebedev 3d3ee04eb9 [s3] Put bucket lifecycle configuration (#5510) 2024-04-27 07:39:22 -07:00
Konstantin Lebedev 2f3fee9bb9 filer read chunk retry if status code 499 (#5528) 2024-04-24 06:51:05 -07:00
Konstantin Lebedev 33537ae29f [s3] fix s3 test_multipart_get_part (#5476)
* try fix s3  test_multipart_get_part

* add passed s3 tests

* fix SeaweedFSUploadId

* rm spaces

* convert part request to range

* add passed s3 tests of multipart
2024-04-14 10:41:32 -07:00
Konstantin Lebedev 5189a09de0 [volume] Reduce the number of buffers for uploading one chunk (#5458) 2024-04-11 04:47:21 -07:00
Konstantin Lebedev 3e25ed1b11 [s3] add s3 pass test_multipart_upload_size_too_small (#5475)
* add s3 pass test_multipart_upload_size_too_small

* refactor metric names

* return ErrNoSuchUpload if empty parts

* fix test
2024-04-07 11:52:35 -07:00
Konstantin Lebedev 35cba720a5 [s3] add s3 pass test_multipart_upload (#5474)
add s3 pass test_multipart_upload
2024-04-07 11:51:22 -07:00
Konstantin Lebedev d42a04cceb [s3] fix s3 test_multipart_resend_first_finishes_last (#5471)
* try fix s3 test
https://github.com/seaweedfs/seaweedfs/pull/5466

* add error handler metrics

* refactor

* refactor multipartExt

* delete bad entry parts
2024-04-06 10:56:39 -07:00
Konstantin Lebedev d5d8b8e2ae fix panic at isAllWritable (#5457)
fix panic
https://github.com/seaweedfs/seaweedfs/issues/5456
2024-04-02 09:06:19 -07:00
Konstantin Lebedev f9ec92dd83 [tests] add action s3 tests (#5427) 2024-03-29 07:57:27 -07:00
Konstantin Lebedev 9c1e0f5811 [master] grow volumes if no writable volumes in current dataCenter (#5434)
* grow volumes if no writable volumes in current dataCenter
https://github.com/seaweedfs/seaweedfs/issues/3886

* fix tests with volume grow

* automatic volume grow one volume

* add ErrorChunkAssign metrics
2024-03-29 00:38:27 -07:00
Konstantin Lebedev df40908e57 fix panic 5435 (#5436) 2024-03-28 16:17:59 -07:00
Konstantin Lebedev dc9568fc0d [master] add test for PickForWrite add metrics for volume layout (#5413) 2024-03-22 07:39:11 -07:00
Konstantin Lebedev 25643cfbc6 fix: panic: assignment to entry in nil map on S3Sink.CreateEntry (#5406) 2024-03-21 08:12:31 -07:00
Konstantin Lebedev d821cb3b18 fix: sync without dir /buckets/some/.uploads/hash_hash (#5402) 2024-03-20 12:54:29 -07:00
Konstantin Lebedev 54ee73287d [filer] replace an invalid methods in a metric with an invalid one (#5378)
fix: replace an invalid methods in a metric with an invalid one for filer
2024-03-14 08:19:09 -07:00
Konstantin Lebedev 170b63d6f8 [filer.backup] add param uploader_part_size for S3sink (#5352)
* fix: install cronie

* chore: refactor configure S3Sink

* chore: refactor cinfig

* add filer-backup compose file

* fix: X-Amz-Meta-Mtime and resolve with comments

* fix: attr mtime

* fix: MaxUploadPartst is reduced to the maximum allowable

* fix: env and force set max MaxUploadParts

* fix: env WEED_SINK_S3_UPLOADER_PART_SIZE_MB
2024-03-07 08:35:51 -08:00
Konstantin Lebedev 872b5161e6 [shell] fs log purge (#5344)
* fix: install cronie

* chore: shell delete old logs

* Revert "fix: install cronie"

This reverts commit 920832b04b.

* fix: dayAgo

* fix: Nit: dayAgo => daysAgo
2024-03-05 09:09:08 -08:00
Konstantin Lebedev 7187346cc1 avoid unexpected compact size (#5272)
https://github.com/seaweedfs/seaweedfs/issues/5215
2024-02-24 05:27:35 -08:00
Konstantin Lebedev 6181aa7594 fix: fs verify error counter (#5261) 2024-02-15 01:31:51 -08:00
Konstantin Lebedev 8fb978d6f6 [volume] requests that were canceled by the client are not an internal server problem(500 => 499) (#5304) 2024-02-14 14:18:56 -08:00
Konstantin Lebedev 0f8e76bbd6 fix: clean metric MasterReplicaPlacementMismatch for unregister volume (#5239) 2024-01-25 00:23:24 -08:00
Konstantin Lebedev 0b71fae2e3 fix: s3tests test_get_object_ifnonematch_good (#5237) 2024-01-24 11:05:55 -08:00
Konstantin Lebedev 4e9ea1e628 chore: filer healthz handler check filer store (#5208) 2024-01-17 07:46:37 -08:00
Konstantin Lebedev f9cf13fada [filer] avoid return http ok for not allowed methods (#5209) 2024-01-17 07:17:07 -08:00
Konstantin Lebedev 8d23e36c45 fix: doDeleteFiles deletes files (#5198) 2024-01-12 11:04:29 -08:00
Konstantin Lebedev 1169f94310 Fix filer sync set offset (#5197)
* fix: compose 2mount with sync

* fix: DATA RACE
https://github.com/seaweedfs/seaweedfs/issues/5194
https://github.com/seaweedfs/seaweedfs/issues/5195
2024-01-12 10:57:18 -08:00
Konstantin Lebedev a7fc723ae0 chore: add status code for request_total metrics (#5188) 2024-01-10 10:05:27 -08:00
Konstantin Lebedev b9d32d32e1 chore: filer sync add doDeleteFiles option for create only mode (#5166) 2024-01-06 10:02:16 -08:00
Konstantin Lebedev 3b62b338a5 chore: add maxMB option for webdav (#5165) 2024-01-05 11:10:20 -08:00
Konstantin Lebedev 4f6172f369 Fix missing VersionConfiguration node in get-bucket-versioning response (#5162)
* fix: s3 response for get bucket version
https://github.com/seaweedfs/seaweedfs/issues/5155

* fix: s3 response for get bucket version
https://github.com/seaweedfs/seaweedfs/issues/5155
2024-01-03 12:27:13 -08:00
Konstantin Lebedev 3e43778869 fix: webdav avoid create empty files (#5160) 2024-01-03 09:52:51 -08:00
Konstantin Lebedev b832ddd1ef fix: return etag with md5 in webdav responses (#5158) 2024-01-03 08:50:32 -08:00
Konstantin Lebedev 183352c796 shell meta load add concurrency (#4529)
* fix: increase speed cmd fs meta load

* fix: add wg
2023-12-28 11:54:37 -08:00
Konstantin Lebedev 3c9bcfb864 chore: add dsn for connection to mysql (#5060)
* chore: add dsn for connection to mysql

* add comment

* new comment

* fix: validate dsn and adapt password
2023-12-20 16:20:58 -08:00
Konstantin Lebedev 125ad8fe63 falls back to update only if error contains msg "duplicate entry"
https://github.com/seaweedfs/seaweedfs/issues/5062
2023-12-09 10:18:18 -08:00
Konstantin Lebedev 2b229e98ce fix: add doDeleteFile option for filer backup 2023-11-17 07:37:28 -08:00
Konstantin Lebedev e55fa68848 fix: excludeFileName is empty 2023-11-13 06:23:46 -08:00
Konstantin Lebedev 3c5295a1a6 filer backup add option for exclude file names that match the regexp to sync on filer 2023-11-13 06:23:46 -08:00
Konstantin Lebedev 5e41ab1370 s3 api add not implemented response for PutBucketVersioning 2023-11-13 06:22:23 -08:00
Konstantin Lebedev 7f6152f130 s3 api add default response for GetBucketVersioning 2023-11-13 06:22:23 -08:00