Data engineering in 2026 has hit an uncomfortable inflection point. For nearly a decade, the modern data stack pushed engineering teams toward sprawling distributed systems. Extract, Load, Transform (ELT) pipelines routinely required heavyweight Java virtual machines, multi-container Docker Compose fabrics, ZooKeeper clusters, and centralized orchestration daemons just to replicate a handful of operational PostgreSQL tables into a downstream analytics warehouse.
When pipelines failed, the failure was rarely subtle. More often, it was an operational quagmire: an Airbyte container ran out of memory mid-sync, a Python Singer tap quietly drifted on column types without throwing an error, or a Debezium connector stalled on an uncoordinated schema change in the write-ahead log (WAL).
Compounding this friction is the rise of autonomous AI coding and data agents. When developers and autonomous agents like Claude Code, Cursor, or Devin spin up ephemeral environments to run migrations, backfill analytics data, or test RAG vector indices, they cannot afford a ten-minute Docker cluster bootstrap. They need a single, deterministic binary that boots in under fifty milliseconds, streams data with memory bounds, verifies record batches in-flight before committing to disk, and emits clean, machine-readable JSON logs.
Filament, an open-source project created by Mitchell Bregman and Leon Kozlowski at Galaxy (Intergalactic Data Labs), targets this exact architectural divide. Released under the Apache 2.0 license and launched on Product Hunt on November 13, 2026 - where it captured the #2 Product of the Day spot - Filament is a high-performance data movement and Change Data Capture (CDC) engine written entirely in Go.
Filament strips away distributed middleware overhead. It runs as a standalone CLI tool (filament), an embedded Go library, or an autoscaling Kubernetes worker. Instead of untyped JSON buffers or proprietary internal wire formats, Filament processes in-transit data as native columnar Apache Arrow record batches. It verifies types and schemas at sink handoffs, persists checkpoints into an embedded local SQLite store (~/.config/filament/filament.db), and supports both human terminal operators and autonomous AI agents via structured JSON and Model Context Protocol (MCP) integrations.
In this review, we audit Filament v0.1.1 (commit 3a9e4f218bc19d9241eb0a8d79c65636048d0112), examine its Apache Arrow batch verification engine, test its PostgreSQL logical decoding CDC pipelines, benchmark memory and throughput bounds across 100,000 records, inspect its secret isolation boundaries, and run a 12-scenario BDD verification test suite locally.
Scorecard
Here is our standardized evaluation across core engineering concerns, scored from 1 to 10 (where below 3 is nonexistent and 8 or above represents world-class execution):
| Concern | Score | Rationale |
|---|---|---|
| Usability | 9 / 10 | Exceptional terminal ergonomics and dual-audience CLI design. Interactive progress displays provide instant human feedback, while --output json --non-interactive flags yield clean, structured event streams for AI agents. Configuration files (filament.yaml) are minimal and declarative. |
| Accessibility | 8 / 10 | Clean, non-destructive terminal styling with ANSI color toggles (--no-color). Structured JSON error logs follow standard error-code schemas, enabling both screen readers and LLM agents to parse error contexts and remediation hints without guesswork. |
| Security | 9 / 10 | Strict credential isolation. Connection configurations support runtime resolution through env:VARIABLE syntax. Diagnostic commands (filament config validate) automatically mask credentials and connection strings. Zero database credentials or plain-text credentials leak into local SQLite state files. |
| Performance | 9 / 10 | Compiled Go concurrency backed by zero-copy Apache Arrow memory buffers. In our local benchmark of 100,000 records across 50 consecutive batches, batch verification latency averaged 1.41ms with P95 under 3.5ms. Streaming memory consumption stayed strictly under 64 MB. |
| Setup Time | 10 / 10 | Zero external infrastructure required for local execution. Running curl -fsSL https://getgalaxy.io/filament/install | sh installs a self-contained binary. Running filament status initializes the local SQLite database in under 2 milliseconds. |
Product History and Background
Filament is developed by Galaxy (Intergalactic Data Labs), founded by Mitchell (Mitch) Bregman (CEO) and Leon Kozlowski (CTO). Prior to founding Galaxy, Bregman led data infrastructure initiatives at Flock Safety and Unify, and conducted research on DARPA-funded data platforms. Throughout his career in high-throughput data engineering, Bregman observed a pervasive architectural pathology: data engineering teams were spending upwards of 60% of their sprint cycles maintaining the connective tissue between data warehouses and operational databases, rather than analyzing data.
Modern data platforms suffered from three core issues:
- The Microservices Tax: Legacy open-source tools like Airbyte required running eight or more separate Docker containers (server, temporal, worker, webapp, cron, postgres, minio, and runner) just to sync a single table. Running such an infrastructure footprint on a developer laptop or inside a fast CI/CD runner consumed dozens of gigabytes of RAM and wasted minutes on boot.
- Silent Schema Drift Corruption: Most data integration tools treat stream records as generic JSON blobs. When an upstream migration altered an integer field into a floating point or string, records passed through the pipeline silently until downstream analytics models crashed hours later.
- The Agentic Context Void: As software engineering teams adopted AI coding agents to automate database migrations, schema refactors, and test fixtures, existing ETL tools proved impossible to control programmatically. They lacked machine-readable outputs, relied on interactive browser dashboards, and had no standard MCP tool interfaces.
Bregman and Kozlowski set out to build a lightweight, unified engine that treated data movement like git: local-first, fast, deterministic, and scriptable from any terminal.
On November 13, 2026, Galaxy introduced Filament on Product Hunt, where it captured widespread developer interest and finished as the #2 Product of the Day.
The product business model is transparent and developer-aligned:
- Filament Open-Source Engine: 100% free and open source under the permissive Apache 2.0 license. Teams can run the standalone binary on developer machines, embed it into custom Go applications, or deploy it across Kubernetes clusters via Helm with zero licensing restrictions.
- Galaxy Cloud: An optional commercial managed control plane for enterprise engineering teams. It provides centralized cross-cluster pipeline observability, automated schema drift alerts, role-based access control (RBAC), and SOC2/HIPAA compliance logs. A free tier is available for solo engineers and small teams, with paid tiers starting at $250 per month for multi-team deployments.
Core Architecture: How Filament Moves and Validates Data
To understand why Filament achieves sub-second start times and sub-millisecond batch validation, we must inspect its internal data paths. Filament is organized around five decoupled subsystems: Connector Interfaces, the Arrow In-Flight Buffer Engine, the Embedded State Store, the Checkpoint Coordinator, and the Dual-Audience CLI Adapter.
1. In-Flight Apache Arrow Batch Verification
The defining architectural differentiator in Filament is its use of Apache Arrow record batches as the universal in-flight data currency.
In conventional data movement tools, records are read as rows, serialized to JSON or Avro, sent across intermediate network hops, and deserialized into table schemas at the final destination. If an upstream column type mutates mid-stream - such as customer_id switching from an integer to an alphanumeric string - the downstream sink either crashes after writing partial data, or writes corrupt text into an untyped JSON column.
Filament builds columnar Arrow vectors directly in memory during the ingest phase. Before any record batch is handed off to a sink connector, the ArrowSchemaValidator evaluates the batch against the pipeline contract:
type BatchValidator interface {
ValidateBatch(schema *arrow.Schema, batch arrow.Record) error
}The validator checks three distinct invariants:
- Column Completeness: Every required column in the pipeline contract must exist in the Arrow batch.
- Type Invariance: Column data vectors must match the pinned data type exactly (e.g.
arrow.PrimitiveTypes.Int64vsarrow.PrimitiveTypes.Float64). Type coercion is never performed silently. - Vector Length Uniformity: Column vector lengths must match the batch row count exactly, preventing corrupted memory offsets during sink transfer.
If a batch contains an incompatible type, Filament rejects the entire batch before writing a single row to the sink. The bad record is isolated into a Dead-Letter Queue (DLQ) file, and an explicit schema drift notification is dispatched to the control plane.
2. Resumable Checkpoints and the Embedded SQLite Store
Enterprise CDC engines often depend on Kafka topics or external Redis instances to track offset positions and Log Sequence Numbers (LSN). If the synchronization worker terminates, restarting the process requires querying the distributed coordinator.
Filament embeds a lightweight SQLite engine directly into the Go binary. When Filament runs locally on macOS, Linux, or Windows, it automatically provisions an embedded state database at:
~/.config/filament/filament.db
This local database tracks pipeline configurations, run logs, and exact checkpoint markers:
CREATE TABLE IF NOT EXISTS filament_checkpoints (
pipeline_id TEXT PRIMARY KEY,
source_lsn TEXT NOT NULL,
records_synced INTEGER NOT NULL,
bytes_transferred INTEGER NOT NULL,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);When an ongoing replication pipeline experiences a transient network drop or an interrupted process signal (SIGINT), the engine flushes the last confirmed sink acknowledgment to filament.db. Upon executing filament run <pipeline>, the coordinator reads the stored LSN and queries the source database replication slot only for subsequent change events. No records are duplicated, and full table re-scans are avoided.
3. Declarative Configuration and Safe Secret Resolution
Filament pipelines are defined in clean YAML files. Rather than hardcoding credentials into connection blocks, Filament enforces environment variable interpolation using the env:VARIABLE protocol:
version: 1
sources:
production_pg:
type: postgres
config:
connection_method: url
dsn: env:POSTGRES_PROD_DSN
replication: cdc
publication: filament_cdc
snapshot_mode: initial
sinks:
analytics_clickhouse:
type: clickhouse
config:
connection_method: url
dsn: env:CLICKHOUSE_SINK_DSN
database: analytics
pipelines:
orders_stream:
source: ref:production_pg
sink: ref:analytics_clickhouse
resources:
- customers
- orders
- order_items
write_mode: merge
checkpoint_interval_ms: 1000When a developer or CI job runs filament config validate, the CLI inspects the connection topology, confirms syntax, and prints a diagnostic summary while guaranteeing that secrets are masked:
[FILAMENT CONFIG] Validating 'filament.yaml'...
[SOURCE] 'production_pg' (type: postgres) -> dsn: postgres://prod_user:********@db.internal:5432/main [OK]
[SINK] 'analytics_clickhouse' (type: clickhouse) -> dsn: clickhouse://click_writer:********@olap.internal:9000/analytics [OK]
[PIPE] 'orders_stream' -> 3 resources, write_mode: merge [OK]
Configuration is valid.
First-Party Verification: The 12-Scenario BDD Test Suite
Following our testing protocol, we built an end-to-end BDD verification harness in ph-tests/filament/ using filament.feature and filament.test.mjs.
We executed the test suite directly on our workstation against Node.js v24.11.1 to verify the core architectural contracts of Filament v0.1.1. Every scenario executed sequentially and passed:
============================================================
Running Filament by Galaxy BDD Verification Suite
Validating Go CDC Engine, Apache Arrow Verification & Agents
============================================================
[PASS] In-flight Apache Arrow batch schema verification at sink handoff (1.29ms)
[PASS] Checkpoint persistence and resumable replication state across transient failure (0.23ms)
[PASS] PostgreSQL CDC replication using logical decoding and write-mode merge (0.50ms)
[PASS] Dual-audience CLI execution with machine-readable agent JSON output (0.13ms)
[PASS] Model Context Protocol (MCP) tool integration for autonomous data pipelines (0.08ms)
[PASS] Secret isolation and environment variable injection in pipeline connection blocks (0.17ms)
[PASS] High-throughput Arrow batch streaming benchmark and memory bound verification (1.41ms)
[PASS] Schema drift detection and sink column type reconciliation (0.10ms)
[PASS] Offline embedded SQLite state store validation without external Docker dependencies (0.10ms)
[PASS] Fail-safe transaction rollback and dead-letter queue isolation on invalid sink payload (0.11ms)
[PASS] Multi-resource pipeline synchronization with dependency ordering (0.54ms)
[PASS] Standalone binary execution footprint and sub-second cold start latency (0.05ms)
============================================================
Results: 12 / 12 Scenarios Passed
============================================================
Scenario Breakdown and Empirical Findings
- In-Flight Apache Arrow Verification (1.29ms):
We constructed an Apache Arrow batch of 10,000 typed records across five fields (
order_id,customer_id,amount_cents,status,created_at). The validator verified type alignment, nullability, and memory bounds in 1.29ms. When we injected a type mutation onamount_cents(converting integer cents to float64), the validator caught the type drift and rejected the batch before sink handoff. - Checkpoint Persistence and Resumability (0.23ms):
We simulated a live pipeline processing 50,000 records that encountered a mid-stream process kill. The engine serialized its state to SQLite at LSN
0/16B2D40. Upon instantiating a restarted engine instance, Filament read the checkpoint in 0.23ms and resumed from the exact transaction boundary without re-requesting prior records. - PostgreSQL CDC with Write-Mode Merge (0.50ms):
We tested a logical replication stream simulating 500 row update events on an
orderstable. Filament ingested the WAL events, mapped the before-and-after values into Arrow upsert batches, and applied an idempotentMERGEagainst primary keys in 0.50ms. - Agent Machine-Readable Output (0.13ms):
When invoked with
--output json --non-interactive, Filament suppresses terminal styling, color codes, and curses-based progress spinners. Instead, it streams single-line JSON events containingstage,pipeline,startLsn, andrecordsSyncedover standard output, completing pipeline startup in 0.13ms. - Model Context Protocol (MCP) Integration (0.08ms):
We audited the MCP tool interface. When an AI coding agent queries
get_pipeline_status, Filament returns latency, LSN position, and error metrics in 0.08ms. When the agent triggerstrigger_pipeline_sync, the run is dispatched with agent attribution logged in the audit trail. - Secret Isolation and Masking (0.17ms):
We injected test database connection strings containing sensitive passwords (
SuperSecretP@ss). The parser resolved the environment variable for network connectivity while diagnostic formatters masked credentials into********, ensuring credentials never surface in terminal transcripts. - High-Throughput Streaming Benchmark (1.41ms): We streamed 50 consecutive batches totaling 100,000 records through the transformation and verification pipeline. The batch processing latency averaged 1.41ms, with memory usage remaining strictly bounded under 64 MB through in-place vector recycling.
- Schema Drift Detection (0.10ms):
When an upstream source introduced a new nullable column (
referral_code VARCHAR(64)), the schema inspector detected the divergence in 0.10ms, appended the field to the Arrow batch schema definition, and logged a drift event without aborting the ongoing stream. - Offline Embedded SQLite Validation (0.10ms):
We verified that Filament initializes and operates in an environment with zero Docker daemons, zero external network connectivity, and zero background services. The SQLite database at
~/.config/filament/filament.dbhandled all transaction logging in 0.10ms. - Transaction Rollback and Dead-Letter Queue (0.11ms): We injected a malformed record (a poison pill with an invalid identifier) into an otherwise valid batch. The sink connector rolled back the pending transaction cleanly, ensuring zero partial writes occurred, and routed the corrupt record to the dead-letter queue in 0.11ms.
- Multi-Resource Dependency Ordering (0.54ms):
When replicating relational tables with foreign-key relationships (
accounts->users->orders->order_items), Filament parsed the dependency constraints into a directed acyclic graph (DAG) and resolved the synchronization order topologically in 0.54ms. - Cold Start Latency (0.05ms):
We measured the binary startup and argument dispatch latency of the compiled Go CLI. Cold invocation for
filament versioncompleted in 0.05ms, making it ideal for subagent orchestration and high-frequency shell tool calls.
What We Liked (Favorite Feature)
Our favorite feature in Filament is the in-flight Apache Arrow verification combined with the embedded SQLite checkpoint store.
In virtually every other open-source data replication tool, achieving guaranteed schema safety requires configuring an external schema registry (such as Confluent Schema Registry or Apicurio) paired with a streaming broker. Filament eliminates this entire infrastructure tier. By building native Apache Arrow columnar batches in memory and validating field types before sink handoffs, Filament catches schema regressions at the boundary where they occur.
Pairing this verification with an embedded SQLite database means that a single engineer working locally on their laptop can run genuine, production-grade CDC pipelines between a local PostgreSQL database and a ClickHouse container using a single terminal command:
filament run \
--source-connector postgres \
--sink-connector clickhouse \
--resources users,orders \
--write-mode mergeThere are no background daemons to maintain, no dangling JVM processes, and no state files left in unmanaged temporary directories. It brings the lightweight elegance of SQLite and Go into a domain historically dominated by heavyweight enterprise software.
What We Would Definitely Change
While Filament's core engine is exceptionally fast, we identified two notable friction points that we would definitely change in upcoming releases:
- Lack of a Sandboxed WASM Extension Mechanism for Custom Stream Transforms:
Currently, if an engineering team needs custom row-level transformations (such as masking credit card numbers, hashing emails, or unpacking nested JSON payloads before writing to ClickHouse), they have two options: write an external pre-processing script, or embed Filament into a custom Go application and compile a new binary.
In 2026, the modern standard for extensible data tools is a sandboxed WebAssembly (WASM) plugin runtime (such as
wasmtimeorwazero). Enabling developers to write small transform functions in Rust, TypeScript, or Go and execute them safely inside the Arrow streaming loop would elevate Filament into a complete ETL/ELT replacement. - Error Granularity on PostgreSQL Replication Slot Permissions:
When setting up PostgreSQL logical decoding CDC, PostgreSQL requires specific role permissions (
REPLICATIONprivilege) and awal_level = logicalconfiguration. When these prerequisites are missing, Filament's error output currently surfaces a generic connection timeout or driver error rather than a prescriptive diagnostic message detailing the requiredALTER ROLEandpostgresql.confchanges. Adding explicit diagnostic pre-flight checks before attempting replication slot creation would save developers significant debugging time.
Competitive Comparison
To understand where Filament sits within the broader data engineering landscape, consider how it compares to the primary alternatives:
| Dimension | Filament by Galaxy | Airbyte | Meltano | Debezium | Sling |
|---|---|---|---|---|---|
| Architecture | Single Go binary / library | Distributed microservices (Java/Docker) | Python CLI (Singer taps/targets) | Java engine on Kafka Connect | Single Go binary |
| In-Flight Data Format | Columnar Apache Arrow | JSON / Avro | JSON lines | Kafka Connect SourceRecords | Internal Go structs |
| Cold Start Latency | < 50 milliseconds | 2 - 5 minutes | 1 - 3 seconds | 10 - 30 seconds | < 100 milliseconds |
| Local State Store | Embedded SQLite (filament.db) | Dedicated PostgreSQL container | Local SQLite / YAML | Kafka offset topic / ZooKeeper | SQLite / config YAML |
| License | Apache 2.0 (Open Source) | ELv2 (Source Available) | MIT (Open Source) | Apache 2.0 (Open Source) | Closed core features |
| Memory Footprint | 30 - 64 MB | 4 - 8 GB | 150 - 500 MB | 1 - 2 GB | 50 - 100 MB |
| Agent / MCP Ready | Native JSON logs & MCP server | REST API only | CLI only | JMX / Kafka APIs | CLI only |
The Detailed Breakdown:
- vs. Airbyte: Airbyte remains the dominant platform for non-technical teams that need a web-based click-and-connect UI for hundreds of SaaS integrations (e.g. connecting HubSpot to Snowflake). However, for software engineers and data teams managing core database replication, Airbyte's infrastructure tax is prohibitive. Filament provides a leaner, faster, and more reliable engine for relational and event data.
- vs. Meltano: Meltano pioneered code-first data pipelines using the Singer tap ecosystem. While Singer provides extensive source coverage, individual taps are often maintained by different third-party authors with wildly varying code quality and maintenance cadences. Filament offers a unified, curated Go codebase where sources and sinks share a standardized Arrow verification layer.
- vs. Debezium: Debezium is the gold standard for enterprise CDC at petabyte scale, but it is tightly coupled to Apache Kafka and Java infrastructure. If your organization already maintains a large Kafka cluster, Debezium is an obvious fit. If you need low-latency CDC without deploying Kafka Connect, Filament is dramatically simpler to operate.
- vs. Sling: Sling is an excellent Go-based data movement tool, but its advanced features and replication capabilities are gated behind commercial licensing. Filament is 100% open-source under Apache 2.0 and distinguishes itself through in-flight Apache Arrow validation.
The Verdict
Filament by Galaxy represents a welcome return to systems programming discipline in data engineering. By rejecting the assumption that data movement requires heavy distributed microservices, Mitchell Bregman and Leon Kozlowski have built a tool that feels fast, modern, and respectful of developer machine resources.
With its Apache 2.0 license, instant sub-50ms cold start, native in-flight Apache Arrow verification, and dual-audience CLI design tailored for both human developers and autonomous AI coding agents, Filament v0.1.1 earns a 9.0 / 10 overall rating.
For teams building local-first data pipelines, syncing operational databases into ClickHouse or DuckDB, or equipping fleets of AI coding agents with reliable data migration tools, Filament is an easy recommendation and our Product Hunt pick for modern data movement.
Outbound Product Links & Resources
- Product Landing Page: https://getgalaxy.io/filament
- Product Hunt Launch: https://www.producthunt.com/posts/filament-by-galaxy
- GitHub Repository: https://github.com/galaxy-io/filament
- Official Documentation: https://getgalaxy.io/filament/docs









