Github-Mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2026-09-08 15:41:15 +02:00
Code Issues Packages Projects Releases Wiki Activity
11,876 Commits 177 Branches 359 Tags
93a09f5da43f8e95676078abe986b71bad4edb7a
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
chrislu 93a09f5da4 explain 2025-09-01 16:47:52 -07:00
chrislu 55cad6dc4a combine parquet results with live logs 2025-09-01 16:26:35 -07:00
chrislu 352fad7172 truncate topic 2025-09-01 16:12:20 -07:00
chrislu e3798c2ec9 sql 2025-09-01 16:12:10 -07:00
chrislu 2fbceca959 fmt 2025-09-01 15:40:09 -07:00
chrislu 55dfb97fc8 parquet file generation remember the sources also 2025-09-01 15:11:06 -07:00
chrislu 2fa8991a52 scan all files 2025-09-01 14:51:23 -07:00
chrislu ada704ad80 parquet file generate stats 2025-09-01 13:55:57 -07:00
chrislu c7598d89f1 remove emoji 2025-09-01 13:55:44 -07:00
chrislu c73ceac79f use parquet statistics for optimization 2025-09-01 13:50:56 -07:00
chrislu 471ba271dc fix reading system fields 2025-09-01 13:28:38 -07:00
chrislu 8498240460 fmt 2025-09-01 13:09:44 -07:00
chrislu 8645f3a264 column name case insensitive, better auto column names 2025-09-01 11:25:04 -07:00
chrislu 32e73811f2 support aggregation functions 2025-09-01 11:15:00 -07:00
chrislu cbf1bfd333 remove emoji 2025-09-01 11:05:23 -07:00
chrislu cf9ad26608 scan topic messages 2025-09-01 10:50:01 -07:00
chrislu 762eeca8d4 Update .gitignore 2025-09-01 08:56:39 -07:00
chrislu ac8e6c8c82 actual column types 2025-09-01 08:56:27 -07:00
chrislu 49c0f74a1f Update describe.go 2025-09-01 08:31:58 -07:00
chrislu 3e54e7356c show tables works 2025-09-01 01:25:54 -07:00
chrislu aa883472a5 show databases works 2025-09-01 01:08:11 -07:00
chrislu 675ec42fad integer conversion 2025-08-31 23:22:20 -07:00
chrislu ec7efe0652 fmt 2025-08-31 23:16:34 -07:00
chrislu 6e35019238 Delete test_utils_test.go 2025-08-31 23:10:54 -07:00
chrislu 0679e5d38b feat: Enhanced SQL CLI Experience 
✅ COMPLETE ENHANCED CLI IMPLEMENTATION:

🚀 **Multiple Execution Modes:**
- Interactive shell with enhanced prompts and context
- Single query execution: --query 'SQL' --output format
- Batch file processing: --file queries.sql --output csv
- Database context switching: --database dbname

📊 **Multi-Format Output:**
- Table format (ASCII) - default for interactive
- JSON format - structured data for programmatic use
- CSV format - spreadsheet-friendly output
- Smart auto-detection based on execution mode

⚙️ **Enhanced Interactive Shell:**
- Database context switching: USE database_name;
- Output format switching: \format table|json|csv
- Command history tracking (basic implementation)
- Enhanced help with WHERE operator examples
- Contextual prompts: seaweedfs:dbname>

🛠️ **Production Features:**
- Comprehensive error handling (JSON + user-friendly)
- Query execution timing and performance metrics
- 30-second timeout protection with graceful handling
- Real MQ integration with hybrid data scanning

📖 **Complete CLI Interface:**
- Full flag support: --server, --interactive, --file, --output, --database, --query
- Auto-detection of execution mode and output format
- Structured help system with practical examples
- Batch processing with multi-query file support

💡 **Advanced WHERE Integration:**
All extended operators (<=, >=, !=, LIKE, IN) fully supported
across all execution modes and output formats.

🎯 **Usage Examples:**
- weed sql --interactive
- weed sql --query 'SHOW DATABASES' --output json
- weed sql --file queries.sql --output csv
- weed sql --database analytics --interactive

Enhanced CLI experience complete - production ready! 🚀
 2025-08-31 23:10:50 -07:00
chrislu 4858f21639 feat: Extended WHERE Operators - Complete Advanced Filtering 
✅ **EXTENDED WHERE OPERATORS IMPLEMENTEDtest ./weed/query/engine/ -v | grep -E PASS
 2025-08-31 23:03:22 -07:00
chrislu db363d025d feat: Time Filter Extraction - Complete Performance Optimization 
✅ FOURTH HIGH PRIORITY TODO COMPLETED!

⏰ **Time Filter Extraction & Push-Down Optimization** (engine.go:198-199)
- Replaced hardcoded StartTimeNs=0, StopTimeNs=0 with intelligent extraction
- Added extractTimeFilters() with recursive WHERE clause analysis
- Smart time column detection (\_timestamp_ns, created_at, timestamp, etc.)
- Comprehensive time value parsing (nanoseconds, ISO dates, datetime formats)
- Operator reversal handling (column op value vs value op column)

🧠 **Intelligent WHERE Clause Processing:**
- AND expressions: Combine time bounds (intersection) ✅
- OR expressions: Skip extraction (safety) ✅
- Parentheses: Recursive unwrapping ✅
- Comparison operators: >, >=, <, <=, = ✅
- Multiple time formats: nanoseconds, RFC3339, date-only, datetime ✅

🚀 **Performance Impact:**
- Push-down filtering to hybrid scanner level
- Reduced data scanning at source (live logs + Parquet files)
- Time-based partition pruning potential
- Significant performance gains for time-series queries

📊 **Comprehensive Testing (21 tests passing):**
- ✅ Time filter extraction (6 test scenarios)
- ✅ Time column recognition (case-insensitive)
- ✅ Time value parsing (5 formats)
- ✅ Full integration with SELECT queries
- ✅ Backward compatibility maintained

💡 **Real-World Query Examples:**
Before: Scans ALL data, filters in memory
  SELECT * FROM events WHERE \_timestamp_ns > 1672531200000000000;

After: Scans ONLY relevant time range at source level
  → StartTimeNs=1672531200000000000, StopTimeNs=0
  → Massive performance improvement for large datasets!

🎯 **Production Ready Features:**
- Multiple time column formats supported
- Graceful fallbacks for invalid dates
- OR clause safety (avoids incorrect optimization)
- Comprehensive error handling

**ALL MEDIUM PRIORITY TODOs NOW READY FOR NEXT PHASEtest ./weed/query/engine/ -v* 🎉
 2025-08-31 22:03:04 -07:00
chrislu 593c1ebef2 fix: Resolve High Priority TODOs - Real MQ Broker Integration 
✅ COMPLETED HIGH PRIORITY TODOs:

🔧 **Real FilerClient Integration** (engine.go:131)
- Implemented GetFilerClient() method in BrokerClient
- Added filerClientImpl with full FilerClient interface compliance
- Added AdjustedUrl() and GetDataCenter() methods
- Real filerClient connection replaces nil fallback

🔧 **Partition Discovery via MQ Broker** (hybrid_message_scanner.go:116)
- Added ListTopicPartitions() method using topic configuration
- Implemented discoverTopicPartitions() in HybridMessageScanner
- Reads actual partition count from BrokerPartitionAssignments
- Generates proper partition ranges based on topic.PartitionCount

📋 **Technical Fixes:**
- Fixed compilation errors with undefined variables
- Proper error handling with filerClientErr variable
- Corrected ConfigureTopicResponse field usage (BrokerPartitionAssignments vs PartitionCount)
- Complete FilerClient interface implementation

🎯 **Impact:**
- SQL engine now connects to real MQ broker infrastructure
- Actual topic partition discovery instead of hardcoded defaults
- Production-ready broker integration with graceful fallbacks
- Maintains backward compatibility with sample data when broker unavailable

✅ All tests passing - High priority TODO resolution complete!

Next: Schema-aware message parsing and time filter optimization.
 2025-08-31 21:12:08 -07:00
chrislu fe41380d51 feat: Phase 2 - Add DDL operations and real MQ broker integration 
Implements comprehensive DDL support for MQ topic management:

New Components:
- Real MQ broker connectivity via BrokerClient
- CREATE TABLE → ConfigureTopic gRPC calls
- DROP TABLE → DeleteTopic operations
- DESCRIBE table → Schema introspection
- SQL type mapping (SQL ↔ MQ schema types)

Enhanced Features:
- Live topic discovery from MQ broker
- Fallback to cached/sample data when broker unavailable
- MySQL-compatible DESCRIBE output
- Schema validation and error handling
- CREATE TABLE with column definitions

Key Infrastructure:
- broker_client.go: gRPC communication with MQ broker
- sql_types.go: Bidirectional SQL/MQ type conversion
- describe.go: Table schema introspection
- Enhanced engine.go: Full DDL routing and execution

Supported SQL Operations:
✅ SHOW DATABASES, SHOW TABLES (live + fallback)
✅ CREATE TABLE table_name (col1 INT, col2 VARCHAR(50), ...)
✅ DROP TABLE table_name
✅ DESCRIBE table_name / SHOW COLUMNS FROM table_name

Known Limitations:
- SQL parser issues with reserved keywords (e.g., 'timestamp')
- Requires running MQ broker for full functionality
- ALTER TABLE not yet implemented
- DeleteTopic method needs broker-side implementation

Architecture Decisions:
- Broker discovery via filer lock mechanism (same as shell commands)
- Graceful fallback when broker unavailable
- ConfigureTopic for CREATE TABLE with 6 default partitions
- Schema versioning ready for ALTER TABLE support

Testing:
- Unit tests updated with filer address parameter
- Integration tests for DDL operations
- Error handling for connection failures

Next Phase: SELECT query execution with Parquet scanning
 2025-08-31 21:01:23 -07:00
chrislu ad86637e0b feat: Phase 1 - Add SQL query engine foundation for MQ topics 
Implements core SQL infrastructure with metadata operations:

New Components:
- SQL parser integration using github.com/xwb1989/sqlparser
- Query engine framework in weed/query/engine/
- Schema catalog mapping MQ topics to SQL tables
- Interactive SQL CLI command 'weed sql'

Supported Operations:
- SHOW DATABASES (lists MQ namespaces)
- SHOW TABLES (lists MQ topics)
- SQL statement parsing and routing
- Error handling and result formatting

Key Design Decisions:
- MQ namespaces ↔ SQL databases
- MQ topics ↔ SQL tables
- Parquet message storage ready for querying
- Backward-compatible schema evolution support

Testing:
- Unit tests for core engine functionality
- Command integration tests
- Parse error handling validation

Assumptions (documented in code):
- All MQ messages stored in Parquet format
- Schema evolution maintains backward compatibility
- MySQL-compatible SQL syntax via sqlparser
- Single-threaded usage per SQL session

Next Phase: DDL operations (CREATE/ALTER/DROP TABLE)
 2025-08-31 20:16:45 -07:00
chrislu 879d512b55 rename 2025-08-30 11:16:26 -07:00
chrislu 6d265cc74b ensure correct next pointer 2025-08-15 10:35:33 -07:00
chrislu 62ed2366f3 skip file with invalid format 2025-08-15 10:35:33 -07:00
chrislu 8eb85415fb correct error 2025-08-15 10:35:33 -07:00
chrislu 9843a10f2b fix stop time 2025-08-15 10:35:33 -07:00
chrislu d5ded63ce3 day-advance fix 2025-08-15 10:35:33 -07:00
chrislu fdfa089754 fix ListAllMyBucketsResult xmlns 
fix https://github.com/seaweedfs/seaweedfs/issues/6676
 2025-08-14 20:38:03 -07:00
chrislu 80db6f4d79 reduce lock scope to improve log buffer performance 2025-08-14 20:38:03 -07:00
chrislu 6bf5a6871c fix presigned signature 2025-08-11 23:57:59 -07:00
chrislu 7889e78f4d use one http client 2025-08-11 16:45:13 -07:00
chrislu a265a07922 fix ttl objects 
fix https://github.com/seaweedfs/seaweedfs/discussions/7107#discussioncomment-14069018
 2025-08-11 02:20:02 -07:00
chrislu dd4880d55a fix for baidu cloud storage 2025-08-06 20:53:05 -07:00
chrislu 2d11e9bc1a Pin helm-gh-pages action to stable version v1.7.0 
- Use stable release instead of @master to avoid potential regressions
- This should help resolve the 'gzip: invalid magic' error
 2025-08-05 08:42:05 -07:00
chrislu 4f38b7c3b9 Revert "Fix helm-gh-pages action configuration" 
This reverts commit 26ef76b7b9.
 2025-08-05 08:41:18 -07:00
chrislu 26ef76b7b9 Fix helm-gh-pages action configuration 
- Add missing commit_username and commit_email parameters
- This should resolve the 'gzip: invalid magic' error in the Helm GitHub Pages action
 2025-08-05 08:39:53 -07:00
chrislu 1cba609bfa fix checking 2025-07-31 23:23:20 -07:00
chrislu cd861646a0 3.96 2025-07-31 22:46:51 -07:00
chrislu 027829f3b3 optionally set the default retention when creating buckets 2025-07-31 22:45:58 -07:00
chrislu f5c53b1bd8 fix reason display 2025-07-30 16:43:14 -07:00
chrislu 124c4281a8 tag parsing decode url encoded 
fix https://github.com/seaweedfs/seaweedfs/issues/7040
 2025-07-28 02:49:43 -07:00
chrislu a4df110e77 address List permission 
fix https://github.com/seaweedfs/seaweedfs/issues/7039
 2025-07-28 02:39:41 -07:00
chrislu 470d450f17 admin server: fix tls setting 
fix https://github.com/seaweedfs/seaweedfs/issues/7038#issuecomment-3124471878
 2025-07-27 13:59:43 -07:00
chrislu 7ab85c3748 return proper default value for locking and versioning 
fix https://github.com/seaweedfs/seaweedfs/issues/6971
fix https://github.com/seaweedfs/seaweedfs/issues/7028
 2025-07-23 22:20:48 -07:00
chrislu 4f72a1778f minor 2025-07-23 21:59:50 -07:00
chrislu dd464cd339 use latest v3.18.4 2025-07-23 02:23:11 -07:00
chrislu 8531326b55 adding admin credential 2025-07-23 02:21:53 -07:00
chrislu d5085cd1f7 newer helm version 
fix https://github.com/seaweedfs/seaweedfs/issues/7029
 2025-07-22 23:58:31 -07:00
chrislu bfe68984d5 fix logging 2025-07-20 20:02:44 -07:00
chrislu 394e42cd51 3.95 2025-07-19 23:57:36 -07:00
chrislu 89706d36dc less logs 2025-07-16 16:30:22 -07:00
chrislu 22465b8a96 unused 2025-07-16 16:30:07 -07:00
chrislu 215c5de579 minor 2025-07-16 09:22:25 -07:00
chrislu 12c9282042 avoid error overwriting 
fix https://github.com/seaweedfs/seaweedfs/issues/6991
 2025-07-16 09:15:50 -07:00
chrislu bb81894078 Update .gitignore 2025-07-16 01:18:23 -07:00
chrislu 64c5dde2f3 support multiple masters 
fix https://github.com/seaweedfs/seaweedfs/issues/6988
 2025-07-15 10:51:07 -07:00
chrislu 44dfa793d5 Collecting volume locations for volumes before EC encoding 
fix https://github.com/seaweedfs/seaweedfs/issues/6963
 2025-07-14 12:17:33 -07:00
chrislu 606d516e34 add integration tests for ec 2025-07-14 12:17:33 -07:00
chrislu a51d993aa9 ensure bucket exists 
related to https://github.com/seaweedfs/seaweedfs/issues/6971
 2025-07-14 09:55:35 -07:00
chrislu 406aaf7c14 increase upload limit via browser 2025-07-14 08:42:15 -07:00
chrislu 24eff93d9a 3.94 2025-07-13 20:31:31 -07:00
chrislu e7dfc3552c admin ui adds object lock permissions 2025-07-13 20:29:25 -07:00
chrislu 49d43003e1 show volume size limit on dashboard 2025-07-11 19:37:09 -07:00
chrislu 4460dc02e4 Delete MULTIPART_COPY_TEST_SUMMARY.md 2025-07-11 18:53:34 -07:00
chrislu 4fcbdc1f61 tweaking dashboard UI 2025-07-11 13:11:39 -07:00
chrislu 3d4a9bdac0 upgrade templ version from v0.3.833 to v0.3.906 
// templ: version: v0.3.833
// templ: version: v0.3.906

fix https://github.com/seaweedfs/seaweedfs/issues/6966#issuecomment-3063449163
 2025-07-11 13:03:04 -07:00
chrislu 14859f0e8c add mq agent options to server.go 2025-07-09 09:02:26 -07:00
chrislu 39b7e44fb5 embed static assets 
fix https://github.com/seaweedfs/seaweedfs/issues/6946
 2025-07-07 12:42:13 -07:00
chrislu 592b6a1e98 less aggressive volume server shutdown on same uuid 
related to https://github.com/seaweedfs/seaweedfs/issues/5439
 2025-07-07 01:22:17 -07:00
chrislu 8493871406 remove sqlite due to GOOS support for freebsd and openbsd 2025-07-06 21:50:08 -07:00
chrislu a7c57caa3f 3.93 2025-07-06 21:06:50 -07:00
chrislu 798f797158 use float for sleep seconds 
fix https://github.com/seaweedfs/seaweedfs/pull/6795
 2025-07-06 14:16:41 -07:00
chrislu 302e62d480 link to volume details 2025-07-04 14:45:13 -07:00
chrislu df30737505 adjust enterprise link 2025-07-04 14:44:38 -07:00
chrislu fc55d2d8e0 link to volume detail page 2025-07-04 13:55:43 -07:00
chrislu d8da465cd3 fix link to volume server; display volume space usage 2025-07-04 13:52:28 -07:00
chrislu e85fbd29a1 refactor 2025-07-04 13:33:58 -07:00
chrislu a5f48de7d6 add vacuum operation 2025-07-04 13:11:43 -07:00
chrislu 0c1d4b2d08 tweak ui 2025-07-04 12:56:28 -07:00
chrislu 84d4ea0995 show volume details 2025-07-04 12:49:34 -07:00
chrislu 2cfe079a1f weed admin: remove system health status 2025-07-02 23:57:36 -07:00
chrislu e89941f390 weed admin: minor 2025-07-02 23:46:26 -07:00
chrislu 645b6d2603 weed admin: disk types card can also show all disk type values, not just a count. 2025-07-02 23:31:17 -07:00
chrislu 5e6d94319b weed admin: add version to volumes page 2025-07-02 23:28:00 -07:00
chrislu ebb16f474c remove status fields 2025-07-02 23:17:27 -07:00
chrislu 40e3eae21f admin ui: filter by collection 2025-07-02 22:48:21 -07:00
chrislu 2268d2f55e add back dynamic columns 2025-07-02 22:06:06 -07:00
chrislu 1cac3e73f9 show counts for rack and disk type 2025-07-02 21:53:06 -07:00
chrislu ac82dd7c07 Delete admin 2025-07-02 19:35:02 -07:00
chrislu 6b706f9ccd rename files 
*_server.go - main server files
*_management.go - business logic
*_data.go - data structures and types
*_middleware.go - middleware logic
 2025-07-02 00:04:46 -07:00
chrislu f47c4aef5a object store users 2025-07-02 00:00:23 -07:00