Bridging Java and Native Execution: Presto Connector Federation with Apache Arrow Flight

Introduction: The Federation Challenge in Modern Query Engines

The evolution of distributed query engines has created a fundamental architectural tension. Over the past decade, the Presto ecosystem has developed an extensive collection of production-grade Java-based connectors, providing federated access to hundreds of heterogeneous data sources. Concurrently, native C++ execution engines built on frameworks like Velox have demonstrated substantial performance improvements, often achieving order-of-magnitude gains for compute-intensive analytical workloads.  

This presents a significant architectural challenge: organizations require both the performance characteristics of native execution for Lakehouse analytics and the comprehensive data source coverage provided by mature Java connectors. The conventional approach of reimplementing all connectors in C++ is neither practical nor beneficial, such an effort would require years of engineering investment while providing minimal performance improvement for federated queries where network I/O and protocol translation dominate execution time.  

The Presto community has addressed this challenge through Connector Federation via Apache Arrow Flight, an architectural pattern that enables native C++ execution engines to consume data from existing Java-based connectors without requiring connector modifications. This approach facilitates hybrid deployment architectures and provides organizations with incremental migration paths that preserve existing investments while enabling selective performance optimization. 

The Core Problem: Two Worlds, One Query Engine

Presto’s Rich Java Connector Ecosystem  

Presto’s strength has always been its connector architecture. With over 50 connectors in the open-source repository alone, Presto can query:  

• Relational databases: PostgreSQL, MySQL, Oracle, SQL Server  
• NoSQL stores: MongoDB, Redis
• Cloud data warehouses: BigQuery, Redshift, Singlestore, etc. 
• Lakehouse formats: Iceberg, Delta Lake, Hudi 
• Specialized sources: Kafka, Elasticsearch, Prometheus 

These connectors represent thousands of engineering hours and deep domain expertise. They handle authentication, connection pooling, predicate pushdown, type mapping, and source-specific optimizations.  

The Rise of Native Execution Engines  

Meanwhile, the Presto Native execution engine built on Velox delivers:  

• 10-20x faster execution for CPU-intensive operations. 
• Vectorized processing with SIMD instructions. 
• Efficient memory management with zero-copy data structures.  
• Lower latency for interactive analytics. 

Native execution is particularly powerful for lakehouse formats (Parquet, ORC) where compute-intensive operations like decompression, decoding, and filtering benefit dramatically from C++ optimization.

The Tension: Performance vs. Ecosystem  

Here’s where it gets interesting. For lakehouse queries, rewriting connectors in C++ makes sense, you’re reading massive Parquet files where decompression and vectorized processing provide huge wins. But for federated queries against remote databases:  

• Network I/O dominates query time  
• The connector’s primary job is protocol translation, not compute  
• Rewriting in C++ provides minimal performance benefit  
• You lose years of battle-tested code and bug fixes

The question becomes: Can we get native execution performance where it matters while preserving the Java connector ecosystem for federation? 

The Solution: Apache Arrow Flight as a Federation Bridge 

What is Apache Arrow Flight?  

Apache Arrow Flight is a high-performance RPC framework designed specifically for transferring columnar data. Some of the benefits of Flight include: 

• Columnar data transfer using Arrow’s in-memory format  
• Zero-copy streaming for efficient data movement  
• Parallel data streams for high throughput  
• Language-agnostic protocol (Java, C++, Python, Rust) 

Flight provides a natural bridge between Java and C++ because both ecosystems have mature Arrow implementations that share the same wire format.

How the Flight Shim Works  

The Presto Flight Shim is a lightweight Java service that exposes Presto connectors as Arrow Flight endpoints. Here’s the conceptual flow:

Key insight: The Java connector doesn’t know it’s being called by a native engine. It just produces Presto Page objects as always. The Flight Shim handles the translation to Arrow format.

No Connector Modifications Required  

This is crucial: existing Java connectors work without any changes. The Flight Shim:  

• Deserializes split metadata from the native engine’s request  
• Invokes the connector’s PageSource using standard Presto SPI  
• Converts Presto Pages to Arrow RecordBatches on-the-fly  
• Streams batches back to the native engine via Flight 

The connector sees a normal Presto query execution; it has no idea a C++ engine is consuming its data.  

Architecture Deep Dive

Components Introduced  

The Flight Shim adds three main components to the Presto ecosystem:  

1. FlightShimServer  

• Standalone Java process hosting the Flight service  
• Manages connector lifecycle and plugin loading 
• Handles SSL/TLS and mTLS authentication  
• Configurable via standard Presto properties 

2. FlightShimProducer  

• Implements Arrow Flight’s FlightProducer interface  
• Receives FlightShimRequest containing: Connector ID, Serialized split metadata, Column handles  
• Creates connector sessions and invokes PageSource  
• Manages back-pressure between Java and native sides 

3. ArrowBatchSource  

• Converts Presto Page objects to Arrow RecordBatch  
• Handles type mapping (Presto types → Arrow types)  
• Supports all common SQL types: primitives, decimals, dates, timestamps, strings  
• Configurable batch size for memory management 

Data Flow: Query to Result

Let’s trace a federated query through the system: A query joining native Iceberg data with federated PostgreSQL data, showing how the Flight Shim bridges Java connectors to native execution. 

Java vs. C++ Engine Responsibility Breakdown

Feature Capabilities, Performance & Trade-offs

Connector Federation Across Java and Native Clusters 

Run federated queries that seamlessly span both native C++ engine tasks and Java connector shims:

SELECT 
  iceberg_table.product_id, 
  SUM(iceberg_table.revenue) as total_revenue, 
  mongo_collection.product_metadata 
FROM native_catalog.iceberg_table 
JOIN flight_federated.mongodb.products mongo_collection 
  ON iceberg_table.product_id = mongo_collection._id 
GROUP BY 1, 3

The Iceberg scan executes natively in C++ with full vectorization, while the MongoDB join federates through Flight to the Java connector.

Running Java Connectors on Native Clusters

Deploy a pure native Presto cluster but retain access to:

• Legacy Oracle databases
• Specialized connectors (e.g., Prometheus, Kafka)
• Internal data sources with only Java-based connector implementations
• Benefit: Eliminates the operational complexity of maintaining parallel Java and native clusters.

Performance Characteristics and Trade-offs

When Flight Federation Shines:  

• Federated queries with small result sets  
• Queries where network I/O dominates  
• Access to sources without native connectors  
• Gradual migration scenarios 

Performance Considerations:  

• Serialization overhead: Converting Pages → Arrow adds minimal overhead 
• Network hop: Flight adds one RPC layer  
• Memory efficiency: Result set is batched into manageable chunks  
• Streaming: Backpressure prevents memory bloat 

Bridge, Not Replacement

Flight federation is a bridge strategy, not the end goal.  

For lakehouse formats (Iceberg, Delta, Hudi):  

• Native C++ connectors provide 10-20x better performance  
• Continue investing in native implementations  
• Flight is not needed for these hot paths 

For federated sources (databases, NoSQL, APIs):  

• Flight federation is often the right long-term solution  
• Network I/O dominates, so native connectors provide minimal benefit  
• Reuse battle-tested Java code 

The Flight Shim lets you optimize where it matters while maintaining compatibility everywhere else.  

Getting Started

Configuration: The Flight Shim is configured via standard Presto properties.

# Flight Shim Server Config
flight-shim.server=localhost
flight-shim.server.port=9443
flight-shim.server-ssl-enabled=true
flight-shim.server-ssl-certificate-file=/path/to/server.crt
flight-shim.server-ssl-key-file=/path/to/server.key

# Optional: mTLS for client authentication
flight-shim.client-ssl-certificate-file=/path/to/client.crt

# Performance Tuning
# Controls the number of rows per Arrow RecordBatch (balances network overhead vs memory)
flight-shim.max-rows-per-batch=10000
# Specifies the thread pool size dedicated to handling parallel gRPC streams
flight-shim.thread-pool-size=16

Connector Setup

Load connectors in the Flight Shim just like you would on a standard Presto server:

# etc/catalog/postgresql.properties
connector.name=postgresql
connection-url=jdbc:postgresql://localhost:5432/mydb
connection-user=user
connection-password=password

Native Worker Configuration

Point the native C++ workers to the Flight Shim for federating catalogs. The properties below correspond to the native worker configuration keys:

# etc/catalog/postgres.properties
connector.name=arrow-flight
protocol-connector.id=postgresql
arrow-flight.server=flight-shim-host
arrow-flight.server.port=9443

# SSL/mTLS Configs (secures connection and enables mutual trust)
arrow-flight.server-ssl-enabled=true
arrow-flight.server-ssl-certificate=/path/to/server.crt
arrow-flight.client-ssl-certificate=/path/to/client.crt
arrow-flight.client-ssl-key=/path/to/client.key
arrow-flight.server.verify=true

Conclusion

Presto Connector Federation with Apache Arrow Flight solves a fundamental challenge in modern query engines: how to embrace native execution performance without abandoning a rich connector ecosystem.  

By using Arrow Flight as a bridge, Presto enables: 

• Seamless federation between Java and native execution  
• Incremental migration paths for organizations  
• Continued innovation in both Java and C++ codebases  
• Practical optimization where it matters most 

This isn’t about choosing between Java and C++; it’s about leveraging the strengths of both. Native execution for compute-intensive lakehouse queries. Java connectors for federated access to diverse data sources. Arrow Flight as the glue that makes it all work together.  

Follow Us and Join Slack Community

• LinkedIn
• GitHub
• YouTube
• X
• Link