Github-Mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-09-08 15:41:15 +02:00
Code Issues Packages Projects Releases Wiki Activity
9,101 Commits 177 Branches 359 Tags
d4566d4aaa426b33015780c7cc18f887fc07cca4
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Chris Lu d4566d4aaa more solid weed mount (#4089) 
* compare chunks by timestamp

* fix slab clearing error

* fix test compilation

* move oldest chunk to sealed, instead of by fullness

* lock on fh.entryViewCache

* remove verbose logs

* revert slat clearing

* less logs

* less logs

* track write and read by timestamp

* remove useless logic

* add entry lock on file handle release

* use mem chunk only, swap file chunk has problems

* comment out code that maybe used later

* add debug mode to compare data read and write

* more efficient readResolvedChunks with linked list

* small optimization

* fix test compilation

* minor fix on writer

* add SeparateGarbageChunks

* group chunks into sections

* turn off debug mode

* fix tests

* fix tests

* tmp enable swap file chunk

* Revert "tmp enable swap file chunk"

This reverts commit 985137ec47.

* simple refactoring

* simple refactoring

* do not re-use swap file chunk. Sealed chunks should not be re-used.

* comment out debugging facilities

* either mem chunk or swap file chunk is fine now

* remove orderedMutex  as *semaphore.Weighted

not found impactful

* optimize size calculation for changing large files

* optimize performance to avoid going through the long list of chunks

* still problems with swap file chunk

* rename

* tiny optimization

* swap file chunk save only successfully read data

* fix

* enable both mem and swap file chunk

* resolve chunks with range

* rename

* fix chunk interval list

* also change file handle chunk group when adding chunks

* pick in-active chunk with time-decayed counter

* fix compilation

* avoid nil with empty fh.entry

* refactoring

* rename

* rename

* refactor visible intervals to *list.List

* refactor chunkViews to *list.List

* add IntervalList for generic interval list

* change visible interval to use IntervalList in generics

* cahnge chunkViews to *IntervalList[*ChunkView]

* use NewFileChunkSection to create

* rename variables

* refactor

* fix renaming leftover

* renaming

* renaming

* add insert interval

* interval list adds lock

* incrementally add chunks to readers

Fixes:
1. set start and stop offset for the value object
2. clone the value object
3. use pointer instead of copy-by-value when passing to interval.Value
4. use insert interval since adding chunk could be out of order

* fix tests compilation

* fix tests compilation
 2023-01-02 23:20:45 -08:00
Chris Lu dac9c28d05 Revert "refactor: moved to locked entry" (#4035) 
* Revert "refactor: moved to locked entry"

This reverts commit 94bc9afd9d.

* only add LockedEntry, no changes to entryLock

* fix compilation
 2022-12-05 12:32:27 -08:00
Chris Lu f7e4359b59 Temporarily enable cgo 
Temporarily enable cgo following findings from https://github.com/seaweedfs/seaweedfs/discussions/3152 by https://github.com/dyus

Resolve build issue:
```
# github.com/mattn/go-ieproxy
/go/pkg/mod/github.com/mattn/go-ieproxy@v0.0.3/ieproxy.go:36:9: undefined: getConf
/go/pkg/mod/github.com/mattn/go-ieproxy@v0.0.3/ieproxy.go:41:9: undefined: reloadConf
/go/pkg/mod/github.com/mattn/go-ieproxy@v0.0.3/ieproxy.go:48:2: undefined: overrideEnvWithStaticProxy
/go/pkg/mod/github.com/mattn/go-ieproxy@v0.0.3/ieproxy.go:53:13: psc.findProxyForURL undefined (type *ProxyScriptConf has no field or method findProxyForURL, but does have FindProxyForURL)
```
 2022-08-24 02:24:21 -07:00
Chris Lu e88392b50f Fix s3 pagination (#3436) 
* Revert previous changes

* s3: use cursor to track tree traversal

fix https://github.com/seaweedfs/seaweedfs/issues/3166

* special cases for empty prefix and empty directory

* use constants

* address empty folder

* undo local changes

* fix IsTruncated

* adjust counting directories

* fix cases when prefix is a directory

* s3: handle directory object

works for

aws --endpoint-url http://127.0.0.1:8333/ s3api list-objects-v2 --bucket test --prefix "fakedir"
 2022-08-15 00:30:19 -07:00
Chris Lu 9b006250b2 Update pull_request_template.md 2022-08-09 01:25:55 -07:00
Chris Lu 33a5225302 Update README.md 2022-08-08 15:58:36 -07:00
Chris Lu fdd85b34f9 Merge pull request #3399 from kaiwalyajoshi/kjoshi/init-container-support 
feat: Add support for k8s Init-Containers.
 2022-08-03 14:58:02 -07:00
Chris Lu fc8035d672 Merge pull request #3394 from kmlebedev/metricsReplicatedWrite 
Detailed metrics VolumeServerRequestHistogram for writing to disk and replication
 2022-08-01 13:31:01 -07:00
Chris Lu 25b9062152 Merge pull request #3390 from kmlebedev/gentle_vacuum 
vacuum stats of percent util LA
 2022-08-01 09:33:24 -07:00
Chris Lu f0e0073b84 Merge pull request #3391 from seaweedfs/dependabot/go_modules/github.com/aws/aws-sdk-go-1.44.66 
Bump github.com/aws/aws-sdk-go from 1.44.65 to 1.44.66
 2022-08-01 08:54:41 -07:00
Chris Lu 0e441b48a6 Merge pull request #3392 from seaweedfs/dependabot/go_modules/github.com/fclairamb/ftpserverlib-0.19.0 
Bump github.com/fclairamb/ftpserverlib from 0.18.0 to 0.19.0
 2022-08-01 08:51:35 -07:00
Chris Lu ed5b08c284 Merge pull request #3393 from seaweedfs/dependabot/go_modules/google.golang.org/protobuf-1.28.1 
Bump google.golang.org/protobuf from 1.28.0 to 1.28.1
 2022-08-01 08:51:27 -07:00
Chris Lu b59bc607bf Merge pull request #3338 from kmlebedev/issues/3083 
rollback over onPeerUpdate implementation of automatic clean-up of failed servers in favor of synchronous ping
 2022-08-01 08:23:10 -07:00
Chris Lu 69e4725812 Merge pull request #3387 from i404788/master 
Added details on helm deployment
 2022-07-31 17:14:54 -07:00
Chris Lu 0bb3ba17ae Merge pull request #3385 from Woellchen/feature/cluster-status-return-healthy 
Add healthy indicator for raft status
 2022-07-30 11:55:30 -07:00
Chris Lu c15b356b81 Merge pull request #3379 from seaweedfs/dependabot/go_modules/modernc.org/sqlite-1.18.0 
Bump modernc.org/sqlite from 1.17.3 to 1.18.0
 2022-07-29 01:36:14 -07:00
Chris Lu 39c6d53a1b Merge pull request #3383 from seaweedfs/dependabot/go_modules/github.com/ydb-platform/ydb-go-sdk/v3-3.30.0 
Bump github.com/ydb-platform/ydb-go-sdk/v3 from 3.29.5 to 3.30.0
 2022-07-29 01:35:20 -07:00
Chris Lu ef84609010 Merge pull request #3382 from seaweedfs/dependabot/go_modules/github.com/arangodb/go-driver-1.3.3 
Bump github.com/arangodb/go-driver from 1.3.2 to 1.3.3
 2022-07-29 01:35:14 -07:00
Chris Lu b67fe6223b Merge pull request #3381 from seaweedfs/dependabot/go_modules/google.golang.org/api-0.90.0 
Bump google.golang.org/api from 0.87.0 to 0.90.0
 2022-07-29 01:35:08 -07:00
Chris Lu b09dab3477 Merge pull request #3380 from seaweedfs/dependabot/go_modules/github.com/aws/aws-sdk-go-1.44.65 
Bump github.com/aws/aws-sdk-go from 1.44.56 to 1.44.65
 2022-07-29 01:35:01 -07:00
Chris Lu 8c4edf7b40 Added SeaweedMQ_Architecture.png 2022-07-28 23:22:06 -07:00
Chris Lu 3e7e922d16 Merge pull request #3378 from vanhtuan0409/change-put-policy 
PutUserAction should completely replace identity actions
 2022-07-28 19:43:25 -07:00
Chris Lu 32d0413be1 Merge pull request #3374 from famosss/fixcreatebucket 
fix: Buckets are not created and deleted correctly on the filer with …
 2022-07-27 22:37:18 -07:00
Chris Lu b7e4e26778 Merge pull request #3371 from askeipx/issues/3341 
delete disk_size metrics when collection deleted
 2022-07-27 11:25:42 -07:00
Chris Lu 1f592ae933 Merge pull request #3372 from kmlebedev/replicateWithExcludeDirectories 2022-07-27 09:03:23 -07:00
Chris Lu f18bc2d9dd Merge pull request #3367 from kaiwalyajoshi/kjoshi/v3.18-hostpath-fixes 2022-07-26 08:08:08 -07:00
Chris Lu e05641291e Merge pull request #3364 from kmlebedev/fix_login_else 2022-07-26 07:44:03 -07:00
Chris Lu 03ca0ca2c6 Merge pull request #3358 from chrislusf/dependabot/go_modules/cloud.google.com/go/pubsub-1.24.0 2022-07-25 08:28:11 -07:00
Chris Lu 0f19ad9e87 Merge pull request #3357 from chrislusf/dependabot/go_modules/github.com/Shopify/sarama-1.35.0 2022-07-25 08:26:23 -07:00
Chris Lu 498599ed55 Merge pull request #3356 from chrislusf/dependabot/github_actions/docker/build-push-action-3.1.0 2022-07-25 08:24:04 -07:00
Chris Lu becb987672 Merge pull request #3359 from chrislusf/dependabot/go_modules/github.com/ydb-platform/ydb-go-sdk/v3-3.29.5 2022-07-25 08:23:41 -07:00
Chris Lu bfbb9fffa2 Merge pull request #3360 from chrislusf/dependabot/go_modules/cloud.google.com/go/storage-1.24.0 2022-07-25 08:23:23 -07:00
Chris Lu 068b4401fe Merge pull request #3361 from chrislusf/dependabot/go_modules/github.com/linxGnu/grocksdb-1.7.4 2022-07-25 08:23:12 -07:00
Chris Lu 578eb8f39a Merge pull request #3354 from guo-sj/correct_comment 2022-07-24 21:43:48 -07:00
Chris Lu 0716092b39 Merge pull request #3350 from shichanglin5/optimize_masterclient_vidmap 
Solve the problem that `LookupFileId` lookup urls is empty due to lea…
 2022-07-22 02:57:50 -07:00
Chris Lu 8d97add89c Merge pull request #3348 from ningfdx/remote 
optimiz: master ui will render data in order
 2022-07-22 02:43:07 -07:00
Chris Lu 9ec0d1caaa Merge pull request #3337 from guol-fnst/loading_volume 2022-07-20 09:06:06 -07:00
Chris Lu efec31de8d Merge pull request #3340 from vanhtuan0409/put-user-policy 2022-07-20 09:01:14 -07:00
Chris Lu 4820b594a7 Merge pull request #3298 from kmlebedev/evacuateTargetServer 
volume server evacuate to target server
 2022-07-19 13:56:49 -07:00
Chris Lu 7b80605538 Merge pull request #3339 from kmlebedev/issues/3320 2022-07-19 13:53:29 -07:00
Chris Lu 9667588af0 Merge pull request #3318 from kmlebedev/issues/3310 
Use fallback if urls are not found
 2022-07-19 11:38:43 -07:00
Chris Lu f0d560060f Merge pull request #3336 from kmlebedev/raftSrvRemove 
raftServerRemoval typo and logs
 2022-07-19 11:32:17 -07:00
Chris Lu f909265721 Merge pull request #3329 from chrislusf/dependabot/github_actions/actions/dependency-review-action-2.0.4 
Bump actions/dependency-review-action from 2.0.2 to 2.0.4
 2022-07-18 10:36:51 -07:00
Chris Lu ffec365ddf Merge pull request #3330 from chrislusf/dependabot/go_modules/google.golang.org/grpc-1.48.0 
Bump google.golang.org/grpc from 1.47.0 to 1.48.0
 2022-07-18 10:36:43 -07:00
Chris Lu 206715bcd8 Merge pull request #3331 from chrislusf/dependabot/go_modules/github.com/ydb-platform/ydb-go-sdk/v3-3.28.3 
Bump github.com/ydb-platform/ydb-go-sdk/v3 from 3.28.0 to 3.28.3
 2022-07-18 10:36:37 -07:00
Chris Lu 4c0cd753e9 Merge pull request #3332 from chrislusf/dependabot/go_modules/google.golang.org/api-0.87.0 
Bump google.golang.org/api from 0.86.0 to 0.87.0
 2022-07-18 10:36:27 -07:00
Chris Lu 1567ea2f44 Merge pull request #3333 from chrislusf/dependabot/go_modules/github.com/aws/aws-sdk-go-1.44.56 
Bump github.com/aws/aws-sdk-go from 1.44.51 to 1.44.56
 2022-07-18 10:36:18 -07:00
Chris Lu 84ed7ee393 Merge pull request #3334 from chrislusf/dependabot/go_modules/go.mongodb.org/mongo-driver-1.10.0 
Bump go.mongodb.org/mongo-driver from 1.9.1 to 1.10.0
 2022-07-18 10:35:57 -07:00
Chris Lu abeb7f65f8 Merge pull request #3314 from shichanglin5/fix_ETag 
File upload should succeed should return `ETag` instead of `Etag`
 2022-07-14 15:35:41 -07:00
Chris Lu 957c9a2eb2 Merge pull request #3309 from guol-fnst/loading_volume 
optimiz concurrency
 2022-07-13 23:12:24 -07:00
Chris Lu 1a130125de Merge pull request #3308 from guo-sj/check_duplicate_ak 2022-07-13 09:23:37 -07:00
Chris Lu 31f9f528db Merge pull request #3305 from shichanglin5/fix_iam_shell_override 
Check whether there is a duplicate accessKey when modifying iam
 2022-07-12 21:47:39 -07:00
Chris Lu fba5219dab Merge pull request #3303 from guol-fnst/loading_volume 
speeding up loading volumes
 2022-07-12 03:01:24 -07:00
Chris Lu de97ba94c3 Merge pull request #3302 from guo-sj/fix_put_user_policy 2022-07-11 21:15:37 -07:00
Chris Lu cf241c8901 Merge pull request #3301 from chrislusf/dependabot/go_modules/github.com/aws/aws-sdk-go-1.44.51 2022-07-11 10:13:34 -07:00
Chris Lu ebcce363e2 Merge pull request #3300 from chrislusf/dependabot/go_modules/github.com/ydb-platform/ydb-go-sdk/v3-3.28.0 2022-07-11 10:13:09 -07:00
Chris Lu 88dcb17be3 Merge pull request #3299 from chrislusf/dependabot/github_actions/actions/setup-go-3.2.1 2022-07-11 10:12:40 -07:00
Chris Lu 7929a50327 Merge pull request #3296 from guo-sj/fix_put_user_policy 
refine PutUserPolicy
 2022-07-11 01:49:25 -07:00
Chris Lu a77c2232be Merge pull request #3293 from guo-sj/fix_save_inside_filer 2022-07-10 19:19:17 -07:00
Chris Lu b9518e06f5 Merge pull request #3290 from paochiang/fix_dedup_subscribers 2022-07-08 07:40:52 -07:00
Chris Lu 22e05a2942 Merge pull request #3289 from guo-sj/fix_create_policy 2022-07-08 02:46:27 -07:00
Chris Lu aab3ed2347 Merge pull request #3288 from paochiang/fix_spell_error 
fix the spelling error
 2022-07-08 01:48:35 -07:00
Chris Lu d6a5441391 Merge pull request #3286 from duanhongyi/master 
filer tikv support tls
 2022-07-07 23:47:56 -07:00
Chris Lu fa8e33ccb8 Update binaries_dev.yml 2022-07-07 22:11:44 -07:00
Chris Lu bfea7008f3 Merge pull request #3277 from kmlebedev/rabbitry_enable_durable 2022-07-06 00:19:49 -07:00
Chris Lu 05510630fd Merge pull request #3270 from chrislusf/dependabot/go_modules/cloud.google.com/go/pubsub-1.23.1 2022-07-04 12:09:06 -07:00
Chris Lu f69a8127a4 Merge pull request #3271 from chrislusf/dependabot/go_modules/github.com/stretchr/testify-1.8.0 2022-07-04 12:06:46 -07:00
Chris Lu 9ea1d093b0 Merge pull request #3272 from chrislusf/dependabot/go_modules/github.com/aws/aws-sdk-go-1.44.47 2022-07-04 12:06:36 -07:00
Chris Lu c2c9c6d60a Merge pull request #3273 from chrislusf/dependabot/go_modules/modernc.org/sqlite-1.17.3 2022-07-04 12:06:27 -07:00
Chris Lu 00d5be61d3 Merge pull request #3274 from chrislusf/dependabot/go_modules/google.golang.org/api-0.86.0 2022-07-04 12:06:13 -07:00
Chris Lu a122baa427 Merge pull request #3268 from guol-fnst/fix_usage 
add getting bucket quota
 2022-07-04 00:44:43 -07:00
Chris Lu 267d45c313 Merge pull request #3266 from guol-fnst/fix_usage 
fix usage
 2022-07-03 22:11:37 -07:00
Chris Lu c349f81195 Update binaries_dev.yml 2022-07-03 16:16:44 -07:00
Chris Lu 7c32361a7d Added SeaweedMQ_Architecture.png 2022-07-01 21:43:39 -07:00
Chris Lu ddb0f6bc74 Merge pull request #3258 from kmlebedev/issues_3255 
volumeServer.evacuate fix select empty node
 2022-07-01 02:36:30 -07:00
Chris Lu 96c9d16c11 Merge pull request #3263 from kmlebedev/fix_pagination 
fix pagination
 2022-07-01 02:23:45 -07:00
Chris Lu 413cb3d389 Merge pull request #3256 from shichanglin5/fix_keep_connected 
fixed `volume xx not found` caused by missing VolumeLocation events
 2022-06-30 00:10:09 -07:00
Chris Lu 4ae9884a6e Merge pull request #3254 from shichanglin5/fix_subscribe_meta 
[Critical] Avoid synchronizing from scratch due to err
 2022-06-29 12:36:28 -07:00
Chris Lu b98556cff5 Merge pull request #3253 from kmlebedev/constMultipartUploadsFolder 
use const multipart uploads folder
 2022-06-29 12:09:03 -07:00
Chris Lu c6e6e303db Merge pull request #3200 from lapshin-vitaly/bufix/validate-tags-on-copy 
validate tags on copy object and add regex for validating tags
 2022-06-29 01:37:14 -07:00
Chris Lu e22c2c8012 Merge pull request #3188 from kmlebedev/fix_make_dev 
fix ip.bind cmd for docker compose
 2022-06-29 01:36:59 -07:00
Chris Lu a79bf9d92a Merge pull request #3252 from guol-fnst/update_tbk 
remove hdfs, add tikv for full version update
 2022-06-27 22:37:41 -07:00
Chris Lu 8d810f8305 Merge pull request #3251 from shichanglin5/fix_concurrent_write_read 2022-06-27 21:23:48 -07:00
Chris Lu dc6b1a52e8 Merge pull request #3250 from xdadrm/xdadrm-patch-xweo-byte-files-from-webdav 
Fixes WebDAV 0-bytes files
 2022-06-27 11:33:35 -07:00
Chris Lu 4be406a095 Merge pull request #3249 from kmlebedev/issues_2589 
avoid set currentMaster svc.local k8s discovery service domains
 2022-06-27 11:05:07 -07:00
Chris Lu 265b05c8fc Merge pull request #3245 from chrislusf/dependabot/go_modules/github.com/klauspost/reedsolomon-1.10.0 
Bump github.com/klauspost/reedsolomon from 1.9.16 to 1.10.0
 2022-06-27 10:15:23 -07:00
Chris Lu f5273cdd5c Merge pull request #3244 from chrislusf/dependabot/go_modules/github.com/go-redsync/redsync/v4-4.5.1 
Bump github.com/go-redsync/redsync/v4 from 4.5.0 to 4.5.1
 2022-06-27 10:03:32 -07:00
Chris Lu a9389377b5 Merge pull request #3247 from chrislusf/dependabot/go_modules/google.golang.org/api-0.85.0 
Bump google.golang.org/api from 0.83.0 to 0.85.0
 2022-06-27 10:02:19 -07:00
Chris Lu 201023dd15 Merge pull request #3242 from garenchan/ck-dev1 
fix 3238: handle errors for GenerateDirUuid method
 2022-06-27 09:46:46 -07:00
Chris Lu d172d07212 Merge pull request #3246 from chrislusf/dependabot/go_modules/github.com/stretchr/testify-1.7.5 
Bump github.com/stretchr/testify from 1.7.3 to 1.7.5
 2022-06-27 09:44:41 -07:00
Chris Lu 2e3d6104e4 Merge pull request #3243 from chrislusf/dependabot/go_modules/cloud.google.com/go/storage-1.23.0 
Bump cloud.google.com/go/storage from 1.22.1 to 1.23.0
 2022-06-27 09:44:25 -07:00
Chris Lu fdd880b111 Merge pull request #3237 from shichanglin5/remove_query_collection 
Remove the collection query param of s3api and let the collection be …
 2022-06-26 23:02:11 -07:00
Chris Lu 6a2afc3a7e Merge pull request #3235 from blacktear23/list-filer-stores 
Improve filer command help text, add supported filer store list
 2022-06-26 22:20:38 -07:00
Chris Lu cc01f71752 Merge pull request #3233 from blacktear23/filerstore-tikv 
Add TiKV Filerstore back
 2022-06-26 20:12:18 -07:00
Chris Lu dc59ccd110 Merge pull request #3228 from shichanglin5/fix_volumeNotFound 
When the connection with the leader is disconnected, the vidMap shoul…
 2022-06-24 23:06:13 -07:00
Chris Lu 280e33092c Merge pull request #3226 from kmlebedev/advancedtls 
Advancedtls
 2022-06-24 01:34:58 -07:00
Chris Lu 3185154d0e Merge pull request #3218 from famosss/master 
fix: gc create a lot of connections quickly
 2022-06-21 20:33:21 -07:00
Chris Lu d652ce1a36 Merge pull request #3216 from kmlebedev/mk_raft_basedir 
avoid no such raft date directory
 2022-06-21 12:21:30 -07:00
Chris Lu 1846d08a32 Merge pull request #3210 from zeisss/patch-1 
weed mount: Fix link to osxfuse github page
 2022-06-20 14:03:38 -07:00
Chris Lu 0d22877df9 Merge pull request #3209 from chrislusf/dependabot/go_modules/github.com/google/btree-1.1.2 
Bump github.com/google/btree from 1.0.1 to 1.1.2
 2022-06-20 10:40:00 -07:00