Commit Graph
25 Commits
Author SHA1 Message Date
8a71327324 fix: report short S3 ReaderAt reads (#10345)
* fix: report short S3 ReaderAt reads

Problem: S3BackendStorageFile.ReadAt returned a short buffer with a nil error, hiding truncated remote data.

Root cause: Every terminal io.EOF was cleared regardless of how many bytes were read.

Fix: Clear io.EOF only when the requested buffer was completely filled.

Validation: go test ./weed/storage/backend/...
Co-authored-by: Codex <noreply@openai.com>

* fix: validate S3 ReaderAt requests

Co-authored-by: Codex <noreply@openai.com>

* s3 ReadAt: simplify negative offset error

* s3 ReadAt: stop reading once the buffer is full

Skips the extra Read that only collected the terminal EOF, and bounds
the loop if the server ignores the Range header and returns more data
than requested, where Read on an empty slice can spin forever.

* s3 ReadAt: cover full reads that arrive with io.EOF

---------

Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Chris Lu <chris.lu@gmail.com>
2026-07-16 02:05:20 -07:00
chalet 804979d68b [Enhancement] support fix for remote files with command fix (#6961) 2025-07-10 06:13:16 -07:00
ludwigxia 5452405a81 cloud tier: support for Alibaba Cloud OSS (#6466) 2025-01-22 00:48:55 -08:00
chrislu e55076c46f cloud tier: add retry when copying data file
fix https://github.com/seaweedfs/seaweedfs/issues/3828
2022-10-12 00:38:32 -07:00
chrislu 39340f7e42 cloud tier: s3 consume all read response body
fix https://github.com/seaweedfs/seaweedfs/issues/3584
2022-09-04 22:23:28 -07:00
chrislu 26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
chrislu 76148ce0f7 use storage_class for backend configuration
fix https://github.com/chrislusf/seaweedfs/issues/3008
2022-05-02 12:20:01 -07:00
chrislu 192983b464 s3 backend support customizing storage class 2022-04-30 17:36:40 -07:00
Chris Lu c857cc7286 cloud tier: remove tagging since not all s3 vendors support this 2021-10-29 12:39:19 -07:00
Chris Lu 81797a059a volume: sync volume file right before compaction
fix https://github.com/chrislusf/seaweedfs/issues/1237
2020-03-19 23:54:52 -07:00
Chris Lu 709f231e23 tiered storage: add s3 endpoint for private s3 implementation
fix https://github.com/chrislusf/seaweedfs/issues/1238
2020-03-19 21:13:56 -07:00
Chris Lu d335f04de6 support env variables to overwrite toml file 2020-01-29 09:09:55 -08:00
Chris Lu 2000284435 rename volume tier info to volume info 2019-12-28 11:21:49 -08:00
Chris Lu 48d28d3eb2 tier: support remote file attributes and remember the file extension 2019-12-25 21:37:24 -08:00
Chris Lu d960b3474a tier storage: support downloading the remote dat files 2019-12-25 09:53:13 -08:00
Chris Lu 356bd1b629 adjust text 2019-12-10 22:09:29 -08:00
Chris Lu 10bd3c6b4b refactoring 2019-12-08 19:44:16 -08:00
Chris Lu 2b8e20f122 remove unused functions 2019-12-08 18:33:25 -08:00
Chris Lu 70648d35ad go fmt 2019-12-02 20:49:58 -08:00
Chris Lu caae543a9f fix test 2019-12-02 15:54:24 -08:00
Chris Lu ec8de250e2 tiered storage: can copy to s3, read from s3
master not aware tiered volume yet, file assigning is not working yet
2019-12-02 15:08:31 -08:00
Chris Lu 0da7b894cc pass backend config from master to volume servers 2019-11-29 01:05:09 -08:00
Chris Lu f60154f330 master load backend storage config from master.toml 2019-11-28 18:33:18 -08:00
Chris Lu 0f9ba84274 s3 2019-11-27 03:09:45 -08:00
Chris Lu 9711a6ffaa WIP 2019-11-18 19:24:37 -08:00