Presto Native Engine in 2025

    By: 

    Aditi Pandit, Software Engineer at IBM

    Introduction

    The Presto Native Worker is the latest innovation in the Presto SQL engine. Its optimizations reduce the CPU and memory footprint of Presto production clusters leading to great price performance. We are already seeing the benefits at Meta/Uber/IBM watsonx.data. Hardware accelerations startups like Neuroblade also show great benchmarking results with Presto.

    To get a deep technical overview of the Native engine please see Diving into the Presto Native C++ Query Engine (Presto 2.0). 

    The Native engine is a massive step in the evolution of Presto. We’ve made great progress in the Presto Native Engine development since 2020 and saw serious production use in 2024. See Meta’s PrestoCon2024 talk for the latest in their production use at Presto C++: Towards General Availability at Meta – Sergey Pershin, Meta.

    The Presto community is very motivated to make the Native Engine the default execution engine for Presto. Doing this comes with a lot of challenges. Existing users have to migrate their production workloads from the Java runtime, and the new ones have diverse Lakehouse use-cases to be supported with a high bar on performance expectations. Achieving this while both the Native and Java engines are constantly evolving is complicated (ref Native sidecar). 

    In the spirit of the open source development model for the Native engine, engineers from different companies have come together to work on various aspects of it. The working group meets biweekly to discuss problems, resolve conflicts and celebrate the milestones reached. This blog post provides a summary of the main project themes from the discussions in this forum.

    PrestoSQL Coverage

    To minimize the effort required from end-users to migrate a Presto cluster from the Java to native version, the native engine was developed as a drop-in replacement for a Java worker in the Presto cluster. Our objective is to ensure the Native engine supports all the SQL functions and operators available in the Java engine. We plan to have a single documentation for both Java and Native Presto at https://prestodb.io/docs/current/

    Adding all the types, functions and operators in the Native engine is an enormous task and there is always a tail-end of functionality to close. Because the Presto Native engine has a deep integration with the Velox library, adding these features involves changes in Velox codebase as well. 

    As of Dec 2024, the Native engine supports :

    The types left to implement are meant for advanced applications. 

    • IPADDRESS and IPPREFIX types have been in active development since 2024 and are partially supported. The work should be completed soon. 
    • A team of engineers from Meta, Uber and IBM have started working on Geo-spatial support.
    • P4HYPERLOGLOG, Sketches and advanced quantile algorithms still remain to be developed. 

    This critical set of SQL features has enabled several teams to start production deployments. We welcome users to try the new engine as well. The community is very active and helpful to get new users started.

    Lakehouse features

    The Presto Native Engine is built for Lakehouse architectures. In addition to the Hive connector, we want to support the entire SQL dialect with Iceberg. The Presto Iceberg group is actively and constantly enhancing its feature set.

    While we have some Delta Lake support using symlink tables, there is much lacking in the Hudi and Delta lake integrations in the Native engine. We would be very happy to see strong community contributions in these areas.

    Fuzzer based testing

    To keep the native engine implementation honest to Presto Java, the team follows a fuzzer based testing philosophy. The fuzzer testing framework was initiated at Meta and is maintained in the Velox CI. For each type and SQL the fuzzers are enhanced to test various query patterns with randomly generated data. Boundary conditions, invalid inputs, and spilling get thoroughly tested with this approach. The queries are validated against Presto results for the same. We have caught dozens of bugs with this approach, and the team diligently fixes these bugs at the highest priority.

    Feature lockdown and Native side-car

    As both Presto Java and the Native engine are in active development, keeping both engines consistent is always a catch-up game. “Split-brain” scenarios where the engine behaviors diverge, sometimes deliberately, occur as well. In addition to missing types or functions, configuration settings might not be the same also. 

    To avoid a bad user-experience in such situations, a side-car architecture is being introduced. The side-car exposes checker and validation APIs to the coordinator. Before scheduling tasks on the worker, the coordinator uses these APIs to ensure the worker can execute the fragment correctly and returns meaningful errors if it cannot. More details about the side-car are published in this blog article.

    The side-car project was in active development in 2024 and we plan to start using it in testing and production environments this year.

    Performance features

    The Native engine is driven to maximize its price performance offering. The team aims to have very competitive numbers on standard (notably TPC-DS) benchmarks and customer workloads.

    As production is ramping up on the Native engine, system bottlenecks are discovered, analyzed and fixed. One of the benefits of working collaboratively in an open-source first approach across many companies is that engineers share their experiences and help each other fix issues in each Native Worker working group session.

    In addition to that, the IBM team is aggressively pursuing the TPC-DS benchmark at SF 1K, 10K and 100K scale factors to keep the Native engine competitive at an established industry standard. In 2024, the team focused on serial runs for TPC-DS and saw excellent results. See Embracing Presto Open Source for Breakthrough Performance and Presto C++ TPC-DS Updates and pbench for more technical details.

    In 2025, we aim to make more strides in this direction.

    • From throughput benchmark runs, the group has discovered some contention issues around the Exchange and PartitionedOutput operators. There is an effort to develop a new implementation for these.
    • Meta is building a new lookup-based join operator to offer more physical strategies in the native engine.
    • The TopNRowNumber operator is enhanced for other ranking functions to handle issues in Q67 which is the longest query and an outlier in the TPC-DS benchmark.
    • The Parquet and Iceberg reader were built from the ground up in the Native engine to give the team the most opportunity to build sophisticated optimizations for these fundamental use-cases. There are constant improvements in these areas of the codebase.
    • There are a bunch of other projects which have cross optimizer and Native engine designs. These include CTE materialization and dynamic filters.

    In addition to the core development, a new benchmarking infrastructure called pbench was built and open-sourced last year. This tooling has become very popular and is used by multiple companies including IBM, Meta and Neuroblade to do their testing runs.

    2024 was a great year for us on the performance dimension and we are very confident that 2025 will continue that trend.

    Extensibility and Presto SPI

    Presto’s SPI and modular architecture has been key to its success. Presto is heavily used for federated queries which combine data from multiple systems in a single run. The extensibility features have lent to a wide ecosystem: connectors to different metastores, storage systems, specialized user functions for diverse applications and tool integrations make it very popular.

    In the early development of the Native engine there was an extreme focus on the engine performance and extensibility features weren’t prioritized, but we are now at an inflection point where extensibility has become very important. Modularization and clean composable architectures are a key principle in Velox, so the engine is already built with the framework for the customization features. What remains is to add APIs and usability enhancements to make adding external functions as simple as possible.

    The group has been looking at extensibility from multiple angles. The most important categories are listed below. Various engineers have embarked on these initiatives in 2024, and should complete in 2025.

    User-defined functions

    These features enable Presto clients to write their own custom scalar, aggregate and window functions and plug them into the engine. The pluggability could be via an unfenced mechanism within the Native engine process or in an external fenced manner. While the unfenced method is more performant than the fenced method, it also exposes the Native engine to the vulnerabilities in the user code. To enable unfenced functions, we are exposing a Function SPI and a library for dynamically loading such UDFs in the native engine. For fenced functions, we have a REST client/server based implementation.

    In addition to the existing scalar, aggregate and window functions, there is also activity to add Table functions in Presto.

    Connectors

    At present, the Native engine only supports the Presto Hive and Iceberg connectors out of the box. Velox does expose the equivalent of a Presto ConnectorRecordSetProvider API so teams can write custom connectors. But they have to statically link them into the Native engine and do some manual code changes to start using it, which is cumbersome and limiting. Some engineers are exploring ideas to simplify authoring Presto Native connectors and make them dynamically loadable.

    Arrow flight connector

    Arrow Flight is a modern protocol for high performance data services. It is a framework for building RPC based client-server implementations for data exchange over the network. The Arrow Flight framework makes it simple for data service providers to expose Flight servers for their systems. The framework also provides a spec for Flight clients that are used by applications to retrieve data from the Flight servers.

    The Presto Arrow Flight connector uses a Flight client to retrieve data from a Flight server. The Native engine is being enhanced to support the Arrow flight connector. This opens a very broad ecosystem of data services that the Native engine can query.

    Conclusion

    The project descriptions above give a flavor of what the Native engine developers group have been working on. As we take each feature to production, many subtle things are discovered. The Presto Native Engine Working Group is an active forum to discuss these problems and the engineers quickly diagnose and prototype solutions.

    The list of projects is by no means complete. If you want to explore anything interesting or help the team in any way, we would be very happy for you to join us. Please feel welcome to come to the working group sessions or post questions on the Presto #presto-cpp slack channel.