I build data infrastructure and contribute to the open-source systems it depends on.
Shared foundations
Open-source contributions
Carry the problem across the stack.
Building a data platform means working through the boundaries between engines, formats, and runtimes. I contribute changes at those boundaries: making Arrow implementations interoperate, improving DuckDB's execution and extension interfaces, and reducing PyIceberg's metadata overhead.
Status reviewed . Merged changes and proposals in review are identified separately.
01 / Apache Arrow
Make data travel intact.
A shared format needs implementations that agree.
The interoperability problem
Arrow's IPC format supports custom metadata on individual messages. An application can attach context to a record batch, but that context is only useful if the next runtime reads it and carries it through. Support varied across implementations.
I added reading and writing support in JavaScript, Go, and nanoarrow, and helped carry an existing .NET implementation through rebasing and review. Each change had to fit that project's API and ownership conventions, while preserving interoperability with other runtimes.
In Go, an optional interface preserved compatibility with existing RecordBatch implementations. In nanoarrow, the API supports reading metadata through borrowed views. Cross-language fixtures and round-trip tests made these contracts testable.
Where the work landed
JavaScript
Added reading and writing of IPC message metadata, exposed through RecordBatch, with round-trip tests.
Rebased Christoph Mettler's original implementation and addressed review feedback. Curt Hagenlocher carried the work into the merged PR, retaining contributor credit.
My nanoarrow work also covers array-view appending, decoding dictionary deltas, and writing nested dictionaries. In review, I separated a larger change into independently useful pieces: the array appender, delta decoder, and writer improvements.
The decoder preserves earlier batches' dictionary values when a delta arrives. The writer avoids re-emitting unchanged dictionaries and handles nested dictionary dependencies. Those details determine whether streams remain correct as batches and their lifetimes overlap.
While adding dictionary-encoded output to DuckDB's nanoarrow extension, we encountered an ambiguity: a dictionary of extension values and an extension type backed by a dictionary could serialize identically. Moving the metadata to the wrong level could make a stream unreadable.
Following the community's design discussion, I put together a format proposal and implementations in C++, Go, Rust, and nanoarrow. The proposal gives dictionary values their own metadata slot, with interoperability tests and defined behavior for older readers, which fall back to the storage type.
This work is still in review. The design discussion records the existing consensus, implementation limitations, and follow-up decisions. It has not yet become an adopted format change.
Execution, interoperability, and the interfaces extensions depend on.
From integration to database internals
At Query.Farm, I build extensions that connect DuckDB to other systems. That work depends on the engine's interfaces, memory ownership, and execution model. My core contributions cover those boundaries as well as optimizations that apply to ordinary queries.
Merged / Execution API
Batch work across expensive boundaries.
Calling a window aggregate once per output row amplifies the cost of data marshalling, language bridges, and network calls. I introduced a batched window callback so implementations can amortize that overhead across a chunk of results.
Review expanded the discussion to vectorizing the operation more broadly and adapting DuckDB's internal aggregates. The API work connects extension requirements to the engine's execution model.
When a LEFT JOIN's right-hand keys are unique, each left row can match at most once. I added a single-pass probe that uses uniqueness information already collected during hash-table construction and references left-side columns without slicing them.
The optimization works without declared constraints or a query rewrite. Reported benchmarks ranged from 1.12× to 1.29× faster across the tested column counts. These were single-threaded Apple Silicon runs, using median CPU time across five runs.
Less time decoding manifests before the data scan begins.
Make metadata processing cheaper.
Iceberg uses Avro-encoded manifests to describe data files. For a Python reader, repeatedly decoding integers, lists, and maps can become a substantial cost before it reads the underlying dataset.
In 2023, I contributed a sequence of PyIceberg changes that reduced that overhead. An in-memory decoder and specialized collection reads avoided unnecessary I/O interactions and indirect calls. A subsequent Cython decoder kept the existing decoder object contract and deferred construction of integer-to-integer maps until they were accessed.
Both changes merged upstream. This work targets the metadata path that analytical platforms depend on; the measured result below is specific to the PR's test file.
Across the years
Other open-source contributions
Node.js2020
Reduced unnecessary memory copies and BIO buffer allocations in TLS writes, addressing garbage-collection pressure from large buffers. The change landed through Node's commit workflow.