Github-Mirror/seaweedfs
1
0 Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-10-07 14:57:48 +02:00
Code Issues Packages Projects Releases Wiki Activity
12,243 Commits 188 Branches 361 Tags
e05994ca7ecbfb7d2e95796242ce3b660a56547a
Commit Graph
51 Commits
Author SHA1 Message Date
chrislusf e05994ca7e ec: add diskType parameter to core EC functions 
Add diskType parameter to:
- ecBalancer struct
- collectEcVolumeServersByDc()
- collectEcNodesForDC()
- collectEcNodes()
- EcBalance()

This allows EC operations to target specific disk types (hdd, ssd, etc.)
instead of being hardcoded to HardDriveType only.

For backward compatibility, all callers currently pass types.HardDriveType
as the default value. Subsequent commits will add -diskType flags to
the individual EC commands.
 2025-12-10 14:24:55 -08:00
Lisandro Pinandchrislu 76e4a51964 Unify the parameter to disable dry-run on weed shell commands to -apply (instead of -force). (#7450) 
* Unify the parameter to disable dry-run on weed shell commands to --apply (instead of --force).

* lint

* refactor

* Execution Order Corrected

* handle deprecated force flag

* fix help messages

* Refactoring]: Using flag.FlagSet.Visit()

* consistent with other commands

* Checks for both flags

* fix toml files

---------

Co-authored-by: chrislu <chris.lu@gmail.com>
 2025-11-09 19:58:38 -08:00
Lisandro Pin 0be020b0fa Nit: unify the default --maxParallelization value for weed shell commands supporting this option (#6788) 2025-05-13 07:59:26 -07:00
Lisandro Pin ba0707af64 Allow configuring the maximum number of concurrent tasks for EC parallelization. (#6376) 
Follow-up to b0210df0.
 2024-12-18 13:26:26 -08:00
Lisandro Pin b0210df081 Begin implementing EC balancing parallelization support. (#6342) 
* Begin implementing EC balancing parallelization support.

Impacts both `ec.encode` and `ec.balance`,

* Nit: improve type naming.

* Make the goroutine workgroup handler for `EcBalance()` a bit smarter/error-proof.

* Nit: unify naming for `ecBalancer` wait group methods with the rest of the module.

* Fix concurrency bug.

* Fix whitespace after Gitlab automerge.

* Delete stray TODO.
 2024-12-12 09:14:44 -08:00
Lisandro Pin 8c82c037b9 Unify the re-balancing logic for ec.encode with ec.balance. (#6339) 
Among others, this enables recent changes related to topology aware
re-balancing at EC encoding time.
 2024-12-10 13:30:13 -08:00
Lisandro Pin 522a25790a Remove average constraints when selecting nodes/racks to balance EC shards into. (#6325) 2024-12-06 09:00:06 -08:00
Lisandro Pin edef485333 Account for replication placement settings when balancing EC shards within the same rack. (#6317) 
* Account for replication placement settings when balancing EC shards within racks.

* Update help contents for `ec.balance`.

* Add a few more representative test cases for `pickEcNodeToBalanceShardsInto()`.
 2024-12-04 10:47:51 -08:00
Lisandro Pin 351efa134d Account for replication placement settings when balancing EC shards across racks. (#6316) 2024-12-04 09:00:55 -08:00
Lisandro Pin b2ba7d7408 Resolve replica placement for EC volumes from master server defaults. (#6303) 2024-12-02 08:44:07 -08:00
Lisandro Pin efdebf712e Refactor ec.balance logic into a weeed/shell/command_ec_common.go… (#6195) 
* Refactor `ec.balance` logic into a `weeed/shell/command_ec_common.go` standalone function.

This is a prerequisite to unify the balance logic for `ec.balance` and `ec.encode'.

* s/Balance()/EcBalance()/g
 2024-11-04 17:56:20 -08:00
chrislu ec30a504ba refactor 2024-09-29 10:38:22 -07:00
chrislu 701abbb9df add IsResourceHeavy() to command interface 2024-09-28 20:23:01 -07:00
Dan 794a5aa67d Add warning with hints to why a EC shard cant be moved (#5790) 2024-07-15 11:17:22 -07:00
Tobias Gurtzick 78dbac7702 fix(ec): volumes created by foreign collection due to bug in ec balance (#4864) 
* fix(ec): ignore 0 byte data files

refers to parts of #4861

Signed-off-by: Tobias Gurtzick <magic@wizardtales.com>

* fix(ec): ignore volumes not from the current collection during balance

fixes #4861

Signed-off-by: Tobias Gurtzick <magic@wizardtales.com>

---------

Signed-off-by: Tobias Gurtzick <magic@wizardtales.com>
 2023-09-25 10:35:43 -07:00
chrislu 645ae8c57b Revert "Revert "Merge branch 'master' of https://github.com/seaweedfs/seaweedfs"" 
This reverts commit 8cb42c39
 2023-09-25 09:35:16 -07:00
chrislu 8cb42c39ad Revert "Merge branch 'master' of https://github.com/seaweedfs/seaweedfs" 
This reverts commit 2e5aa06026, reversing
changes made to 4d414f54a2.
 2023-09-18 16:12:50 -07:00
dependabot[bot]Chris Luchrislu
 		a04bd4d26f Bump github.com/rclone/rclone from 1.63.1 to 1.64.0 (#4850) 
* Bump github.com/rclone/rclone from 1.63.1 to 1.64.0

Bumps [github.com/rclone/rclone](https://github.com/rclone/rclone) from 1.63.1 to 1.64.0.
- [Release notes](https://github.com/rclone/rclone/releases)
- [Changelog](https://github.com/rclone/rclone/blob/master/RELEASE.md)
- [Commits](https://github.com/rclone/rclone/compare/v1.63.1...v1.64.0)

---
updated-dependencies:
- dependency-name: github.com/rclone/rclone
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* API changes

* go mod

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
 2023-09-18 14:43:05 -07:00
Ryan Russell bd2dc6d641 refactor(shell): Decending -> Descending (#3675) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-14 12:06:48 -07:00
chrislu 26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
chrislu 6793bc853c help message when in simulation mode 2022-05-31 14:48:46 -07:00
justin 3551ca2fcf enhancement: replace sort.Slice with slices.SortFunc to reduce reflection 2022-04-18 10:35:43 +08:00
chrislu a2d3f89c7b add lock messages 2021-12-10 13:24:38 -08:00
Chris Lu 119d5908dd shell: do not need to lock to see volume -h 2021-09-13 22:13:34 -07:00
Chris Lu e5fc35ed0c change server address from string to a type 2021-09-12 22:47:52 -07:00
Chris Lu ad70107eb4 ec balance among nodes with HDD 2021-02-16 06:22:50 -08:00
Chris Lu 36f95e50a9 avoid possible nil disk info 2021-02-16 05:13:48 -08:00
Chris Lu f8446b42ab this can compile now!!! 2021-02-16 02:47:02 -08:00
Chris Lu 2bc05cfbc8 fix typo 2020-10-21 22:26:41 -07:00
Chris Lu a595916342 shell: add volumeServer.evacuate command 2020-09-14 23:47:11 -07:00
Chris Lu 73564e6a01 master: add cluster wide lock/unlock operation in weed shell 
fix https://github.com/chrislusf/seaweedfs/issues/1286
 2020-04-23 13:37:31 -07:00
Chris Lu 97ab8a1976 remove ctx if possible 2020-02-25 22:23:59 -08:00
Chris Lu 892e726eb9 avoid reusing context object 
fix https://github.com/chrislusf/seaweedfs/issues/1182
 2020-02-25 21:50:12 -08:00
Chris Lu 66acc2c84f passing in averageShardsPerEcNode instead of expectedTotalEcShards 
addressing https://github.com/chrislusf/seaweedfs/issues/1111
 2019-11-12 15:36:18 -08:00
Chris Lu 40514c5362 add sortEcNodesByFreeslotsDecending and sortEcNodesByFreeslotsAscending 
addressing https://github.com/chrislusf/seaweedfs/issues/1111
 2019-11-12 13:47:36 -08:00
Chris Lu 9c2f3b1b0f fix rack_not_found case 
fix rack_not_found in https://github.com/chrislusf/seaweedfs/issues/1111
 2019-11-10 12:06:41 -08:00
Chris Lu 57092d69ac go fmt 2019-06-10 21:33:32 -07:00
Chris Lu f9d8bd51ad ec shard balancing 2019-06-10 21:32:56 -07:00
Chris Lu 9d9162ca35 ec.balance: collect dc rack info 2019-06-07 00:25:05 -07:00
Chris Lu d344e0a035 fix ec related bugs 2019-06-05 23:20:26 -07:00
Chris Lu ede876cfdb periodic scripts exeuction from leader master 2019-06-05 01:30:24 -07:00
Chris Lu 0e49add2bc adjust help message 2019-06-05 00:11:37 -07:00
Chris Lu 06133ae98f correct count for free ec slots, avoid removing generated shards 2019-06-04 01:49:51 -07:00
Chris Lu 6cea23d091 purge duplicated ec shards 2019-06-04 01:32:36 -07:00
Chris Lu b05456fe07 able to purge extra ec shard copies 2019-06-03 20:25:02 -07:00
Chris Lu 7e80b2b882 fix multiple bugs 2019-06-03 02:26:31 -07:00
Chris Lu 9ce6b949bf adjust help message 2019-06-01 10:37:08 -07:00
Chris Lu d6e2c877fa ec.balance: use specific collection names 2019-06-01 02:00:18 -07:00
Chris Lu ba18314aab ec shard delete also check ec volumes, in addition to volumes 2019-06-01 01:41:22 -07:00
Chris Lu 1d111d6ce8 collection list normal and ec volumes 2019-05-30 09:27:23 -07:00