Without naming a schema, a serialized predicate's column references
are ambiguous: the same field name can refer to different field IDs
across schema-evolution events, and Substrait field-reference IDs
mean nothing without saying which schema they index into.
Make the request shape unambiguous:
- Add ParquetPushdownRequest.SchemaId (Iceberg schema id) to the
proto and to the design's API sketch.
- Document the binding contract in the design: the predicate must
carry only field-id references; the connector binds before
sending, the server checks ids against the schema.
- Validation now rejects (predicate set, schema_id == 0) with
InvalidArgument before any work is done. Deeper "every field id
exists in the snapshot's schema" check needs catalog access and
lands in M3.
- Update / add tests covering the new gate.
The connector-trusted trust mode is the dev-only escape hatch and
must require both -trust=connector-trusted *and*
-dev.connector_trusted_ack. The latter is intentionally awkward to
flip so an operator cannot disable catalog validation by accident.
Add unit tests covering: default ok, port range, missing filer
addresses, connector-trusted without ack rejected, connector-trusted
with ack accepted, unknown trust modes rejected, and that the error
message names the ack flag so the operator knows how to proceed.
Stand up the SeaweedParquetPushdown service on an in-process gRPC
bufconn listener and exercise it from a real gRPC client. Validates
the M0 acceptance bar end-to-end without needing a filer:
- TestPing_ReportsConfiguredVersionAndTrustMode: Ping returns the
daemon's configured version and trust mode.
- TestPing_DefaultsTrustModeToCatalogValidated: zero-Options Service
defaults to catalog-validated.
- TestPushdown_ReturnsUnimplementedWithStats: Pushdown returns
Unimplemented with a PushdownStats attached as a status detail,
with trust_mode populated and server_time_micros non-negative.
This is the M0 acceptance criterion from PARQUET_PUSHDOWN_DEV_PLAN.md.
- TestPushdown_RejectsInvalidRequestBeforeUnimplemented: validation
fires before the Unimplemented path; an empty data_files request
returns InvalidArgument.
A heavier filer + daemon process-level integration test is deferred
to test/parquet_pushdown/ in a later milestone; the dev plan's
filer-startup phrasing covers ground M3 needs anyway, and bufconn is
sufficient to lock in the wire format for M0.
Add weed/parquet_pushdown/daemon, the host process for the
SeaweedParquetPushdown service:
- Config carries bind ip/port, filer addresses, trust mode, and the
ConnectorTrustedAck flag that gates the dev-only connector-trusted
trust mode (default refuses it; explicit ack required).
- filerClient probes the filer with GetFilerConfiguration at startup
so bad endpoints fail fast, mirroring the iam command's pattern.
Bounded by a 60s timeout; later milestones use the same client to
read Parquet data and side-index blobs.
- server.go wires it together: load TLS, probe filer, build the
Service, listen on (host, localhost) gRPC ports, register the
service + reflection on each, install grace.OnInterrupt for
GracefulStop, and Serve.
Run() returns an error so the calling weed pushdown command can
report startup failures instead of glog.Fatalf.
Implement the gRPC service surface of the pushdown daemon:
- Service struct embeds the generated UnimplementedSeaweed
ParquetPushdownServer, exposes Ping (returns version + trust mode)
and Pushdown (returns Unimplemented after request-shape validation).
- request_validation.go enforces hard caps (max data files, columns,
predicate bytes, row-id cap, vector dim, top_k) and content-type
rules (Puffin DV requires blob_length + referenced_data_file;
equality delete requires equality_field_ids; predicate_kind and
predicate must be set together).
- stats.go is a small accumulator that produces a PushdownStats
message stamped with trust_mode and elapsed micros; M0 wires it
into the Unimplemented status's details so the stats wire format
is exercised end to end.
- TrustMode constants encode the design's catalog-validated
(default) and connector-trusted (dev-only) modes.
Unit tests cover the validation matrix (rejects empty/oversize
requests, accepts well-formed vector queries, accepts well-formed
deletion-vector descriptors, rejects DVs missing blob_length /
referenced_data_file).
Chris Lu