Inside Velox – Open Source “Universal Engine Block” for Big Data and AI

    Contributors:

    Saurabh Mahawar, Developer Relations Engineer at IBM

    Naveen Kumar Mahadevuni, Software Engineer, IBM

    What is Velox?

    Imagine a world where every car manufacturer had to build a completely new engine from scratch for every type of vehicle they produced, one engine specifically designed for a nimble sports car, another for a heavy-duty pickup truck, and a completely different one for a family SUV. Historically, this is exactly how the tech industry built data management systems, creating dozens of specialized, siloed query engines tailored for specific workloads like interactive analytics, batch processing, or machine learning.

    To solve this massive fragmentation and duplicated engineering effort, Meta developed Velox: a revolutionary, open-source C++ “universal engine block” for the data ecosystem . Rather than constantly reinventing the wheel, developers can now drop Velox’s highly optimized, high-performance execution library into any system. Whether it is powering lightning-fast interactive analytics (like Presto), large-scale batch processing (like Apache Spark), or the machine learning pipelines (like PyTorch), Velox unifies the core mechanics of data processing to deliver state-of-the-art speed, unmatched consistency, and massive engineering efficiency across the entire industry.

    From Meta’s Internal Problem to Open Source

    By the early 2010s, Meta was operating one of the world’s largest data platforms, processing data from billions of users while supporting thousands of engineers running analytics, A/B tests, and machine learning workloads. As data volumes grew, performance became a constant challenge.

    Meta relied heavily on Presto, the distributed SQL engine it created in 2012. Although Presto was highly efficient for a JVM-based system, the JVM itself became a bottleneck at Meta’s scale. Garbage collection, JIT compilation overhead, and limited access to modern CPU features such as SIMD made it increasingly difficult to improve query performance.

    Prestissimo: The Native Execution Experiment

    To overcome these limitations, Meta launched Prestissimo (Presto C++), a project to rewrite only Presto’s execution engine in native C++, while keeping the SQL parser and query planner unchanged.

    During development, the team realized that the core components they were building, columnar memory structures, vectorized execution, high-performance hash tables, and spill-to-disk infrastructure, were not specific to Presto. These same building blocks were needed by Spark, stream processing systems, and ML feature engineering pipelines.

    Rather than implementing the same execution engine separately for every system, Meta extracted these reusable components into a standalone library. That library became Velox, a shared, high-performance execution engine that could power multiple data processing frameworks.

    Why was Velox Needed?

    1. The Fragmentation of Data Processing Engines

        As data workloads evolved from traditional SQL analytics to real-time stream processing and AI pipelines, the industry responded by building specialized execution engines such as Presto, Spark, Trino, Druid, ClickHouse, and DuckDB. While each engine targeted different use cases, they all implemented the same core execution primitives independently.

        Every project built its own:

        • Columnar in-memory data format
        • Expression evaluation engine
        • Hash tables for joins and aggregations
        • Spill-to-disk infrastructure
        • Parquet and ORC readers
        • SQL functions such as SUM(), COUNT(), string, date, and array operations

        This duplicated years of engineering effort across organizations. More importantly, every bug fix, optimization, or hardware improvement had to be implemented separately in every engine, making innovation slow and expensive.

        2. Inconsistent SQL Semantics

          Independent implementations also led to inconsistent behavior across engines. Even simple functions such as substr(), round(), or date_trunc() often produced different results depending on the execution engine. At Meta, engineers found multiple implementations of the same string function, each using different indexing rules and error-handling behavior.

          These semantic differences made migrating workloads difficult and undermined trust in analytical results. The same SQL query could produce different answers across platforms, forcing data teams to spend significant time debugging correctness issues instead of building applications.

          3. The JVM Performance Ceiling

            Most large-scale analytical engines, including Presto, Spark, and Hive, relied on the Java Virtual Machine (JVM). Although the JVM provides excellent portability and developer productivity, it introduces limitations for high-performance analytical execution:

            • Garbage collection causes unpredictable latency during large queries.
            • Java object overhead increases memory consumption and reduces cache efficiency.
            • Limited control over memory management prevents advanced techniques such as memory arenas and efficient spill-to-disk.
            • Access to modern CPU features like SIMD vector instructions (SSE, AVX, AVX2) is significantly more constrained than in native C++.

            As datasets continued to grow, these limitations became a major barrier to improving query performance.

            Architecture: How Velox is Built?

            Before diving into the architecture, it is essential to understand a critical design boundary:

            Velox is an execution backend library, NOT a standalone database server.

            Velox deliberately does not include a SQL parser, query optimizer, distributed RPC scheduler, or cluster manager. It leaves logical planning and scheduling to host engines like Presto or Spark, which pass a serialized physical plan tree (velox::core::PlanNode) down to Velox worker nodes for ultra-fast native C++ execution.

            Calling System (Integration Layer): Layer 1

            • What it does: The host framework (Presto Coordinator or Spark Driver) acts as the brain. It parses your SQL query, optimizes the execution plan, and assigns tasks across the cluster.
            • Handoff: The calling system compiles the query into a Fully Optimized Query Plan (velox::core::PlanNode tree) and hands it down to Velox worker nodes for execution.

            Execution Framework & Operators: Layer 2

            • Task / Pipeline Execution Framework: Spawns non-blocking worker threads (Drivers) that stream data batches through execution pipelines.
            • Operators (exec/): Physical processing units performing table scans (TableScan), filtering (Filter), projections (Project), multi-threaded joins (HashJoin), aggregations (HashAgg), sorting (Sort), and disk spilling (Spiller).
            • Expression Eval (expression/): Compiles SQL math and string expressions into fast execution loops (compileExpressions()) with Common Subexpression Elimination (CSE).
            • Functions (functions/): Built-in library of scalar functions, aggregate functions (facebook::velox::exec::Aggregate), and window functions.
            • Serializers (serializers/): Encodes vector batches (PrestoVectorSerde, UnsafeRowSerializer) for ultra-fast network exchange between cluster nodes.

            Core Data Structures & I/O: Layer 3

            • Vectors (vector/): Columnar memory representation. Data is packed in vertical columns (FlatVector, zero-copy DictionaryVector, deferred LazyVector), enabling CPUs to process hundreds of values per clock cycle using SIMD instructions.
            • Type System (type/): Manages primitive types and complex structures (Arrays, Maps, Structs).
            • Connectors (connectors/): Read/Write data from various data sources (HiveConnector, TpchConnector).
            • I/O — DWIO (dwio/): I/O file readers for Apache Parquet, ORC/DWRF, Meta’s fast columnar format (Nimble), and Text.
            • Filter Pushdown: DWIO readers evaluate filters directly inside storage files, skipping non-matching row groups before downloading data into RAM.

            Memory, Cache & Infrastructure: Layer 4

            • Memory Management (common/memory/): Tracks RAM usage in a strict hierarchy (MemoryPool). If a query approaches memory limits, MemoryReclaimer automatically spills temporary data to disk, preventing Out-Of-Memory (OOM) crashes.
            • File Cache (AsyncDataCache): Asynchronously caches hot metadata and file pages in RAM, avoiding operating system page-cache thrashing.
            • Buffers & Arenas (buffer/): Manages contiguous memory blocks (Buffer, BufferView, StreamArena, MmapArena).
            • Common Utilities (common/): Process logging, metrics (TraceContext), and configuration.

            Storage & Formats: Layer 5

            • Interfacing layer connecting Velox to cloud object stores (AWS S3, Azure ADLS, Google Cloud Storage), distributed filesystems (HDFS, Local Disk), and open table formats (Hive, Apache Iceberg, Delta Lake).

            Experimental Acceleration & Exchange: Layer 6

            • GPU Acceleration: Offloads heavy vector compute to NVIDIA GPUs using Wave (Velox CUDA framework) or RAPIDS cuDF.
            • Breeze: Low-level SIMD vector primitives.
            • UCX Exchange: High-speed RDMA network shuffle plugin for ultra-fast server-to-server data transfers.

            The Journey of a Query Inside Velox

            The journey of a query inside Velox is streamlined, highly vectorized, and occurs entirely on a single host. Because Velox is an execution library rather than a full database, it does not parse SQL or perform global optimization. It sits strictly on the local data plane, executing a physical query plan that has already been fully optimized by a parent engine like Presto or Spark.

            Here is the step-by-step journey of how data moves through Velox:

            1. Query Hand-Off & Translation

            • The parent engine’s coordinator parses the query, optimizes the plan, and hands a specific physical query plan fragment to Velox on the local host.
            • Velox ingests this physical plan and maps it to a Task (the local unit of execution).

            2. Task & Driver Execution

            The task is divided into one or more Pipelines, each consisting of a sequence of operators that can process data continuously. Every pipeline is executed by one or more Drivers, which are Velox’s execution objects scheduled on a shared worker thread pool. If a Driver becomes blocked, for example while waiting for I/O, it yields execution so that other Drivers can continue making progress, keeping CPU resources fully utilized.

            Note: Drivers are execution objects, not operating system threads. They are scheduled on worker threads managed by the host engine.

            3. Pipelined Scan & Expression Evaluation

            Once execution begins, data enters the pipeline through storage connectors and file readers. Velox reads formats such as Parquet, ORC/DWRF, Nimble, and Text using its DWIO layer. During scanning, it applies optimizations such as column pruning and predicate pushdown, ensuring that only the columns and row groups required by the query are read from storage.

            The data is represented internally as columnar vectors and streamed through the execution pipeline in batches. Expression evaluation, including filters, projections, and arithmetic operations are performed over these vector batches, allowing the engine to leverage SIMD instructions and cache-friendly memory access for significantly higher throughput than row-by-row execution.

            4. Memory Arbitration & Spill Handling

            As operators execute, intermediate state such as hash tables, aggregation buffers, and sorting structures are allocated from Velox’s hierarchical MemoryPool system.

            When memory usage approaches configured limits, the MemoryArbitrator coordinates memory reclamation. Spill-capable operators can write intermediate state to local storage, allowing large analytical queries to continue executing even when their working set exceeds available RAM.

            This approach minimizes out-of-memory failures while maintaining predictable query execution.

            5. Result Delivery & Network Shuffle

            After the final operator completes processing, the resulting columnar vectors are prepared for delivery.

            For single-node execution, the vectors are returned directly to the calling engine. For distributed execution, the output may be serialized into engine-specific formats such as Presto Pages or Spark’s UnsafeRow representation before being exchanged with downstream workers or returned to the parent engine.

            From the perspective of Presto or Spark, Velox acts as a highly optimized execution backend that transforms an optimized physical plan into the final query results.

            Major Integrations: Where is Velox Being Used?

            Velox is under active development but is already being integrated into more than a dozen different data systems at Meta and across the broader open-source ecosystem. By providing a unified set of high-performance components, Velox is bridging the gaps between historically siloed workloads.

            Here are some of the most prominent data management systems currently leveraging Velox:

            1. Interactive Analytics: Prestissimo (Presto C++)

            Presto is a distributed query engine that powers massive interactive SQL analytic workloads. Traditionally, Presto worker nodes ran on the Java Virtual Machine (JVM), which was historically prone to operational issues and expensive garbage collection procedures. Through a project called Prestissimo, developers are replacing these traditional Java workers with a C++ process powered entirely by Velox. Prestissimo provides a drop-in replacement that takes a Presto plan fragment from a Java coordinator and hands it directly to Velox for native execution, completely bypassing the JVM on the worker nodes and delivering massive performance improvements.

            2. Batch Processing: Apache Spark (Spruce & Gluten)

            For heavy-duty, large-scale batch processing and ETL (Extract, Transform, Load) workloads, Velox is deeply integrated into Apache Spark. This is achieved through two primary initiatives:

            • Spruce: Meta’s internal integration, which uses Spark’s built-in script transform interface to offload query plan fragments to an external C++ process running Velox.
            • Apache Gluten: An open-source project originally created by Intel. Gluten acts as a bridge, creating a Java Native Interface (JNI) API based on the Apache Arrow data format and Substrait query plans. This decouples Spark’s JVM from the execution engine, allowing Spark to seamlessly delegate compute-intensive tasks to Velox.

            3. Artificial Intelligence & Machine Learning

            One of Velox’s most ambitious capabilities is bridging the gap between traditional data analytics and Artificial Intelligence (AI).

            • TorchArrow: Velox serves as the backend execution engine for TorchArrow, a Python dataframe library for data preprocessing in PyTorch. Because data preprocessing can consume up to 50% of the resources in an ML workload, using Velox standardizes these operations and provides tremendous efficiency wins.
            • F3: Meta’s feature engineering framework, F3, is also unifying its execution engine with Velox. This allows developers to seamlessly extract and generate features for ML algorithms across both offline (batch) and real-time environments.

            Real-World Performance: Velox in Action

            To prove that Velox C++ design translates to real-world speed, Velox has been subjected to rigorous benchmarks across multiple execution environments. The results demonstrate that by replacing traditional Java execution layers or leveraging specialized hardware, Velox delivers staggering performance gains and massive infrastructure cost savings.

            1. The CPU Benchmark: Presto Java vs. Velox C++ (Prestissimo)

            The first major test of Velox’s efficiency was executed at Meta, comparing the traditional Java-based Presto engine against Prestissimo (Presto workers powered by Velox’s C++ engine).

            • Standard TPC-H (3TB Scale): In a warm-cache cluster of 80 nodes, CPU-bound queries saw near-order-of-magnitude improvements. Query 1 (Q1) achieved an 8.4x wall-time speedup (dropping from 42 seconds to just 5 seconds), while Query 6 (Q6) saw a 9x speedup (dropping from 9 seconds to 1 second). Shuffle-heavy queries like Q13 and Q19 also doubled in speed.
            • Shadowing Real Production Traffic: To move past synthetic benchmarks, Meta replayed real production traffic from various interactive analytical tools. The Velox-powered C++ engine delivered an average speedup of 6x to 7x, with a significant portion of queries running over 10x faster than the traditional Java engine.
            • The 3x Server Consolidation Win: Perhaps the most impactful result for infrastructure teams is cluster capacity. In shadowing tests, the Velox-based stack supported identical production workloads with equal or better performance using 3x fewer servers, allowing Meta to consolidate a 60-server Java cluster down to just 20 Velox nodes.

            2. The Distributed Batch Benchmark: Apache Spark + Gluten + Velox on AWS EKS

            To evaluate how Velox scales for heavy-duty batch and ETL processing, Amazon Web Services (AWS) ran a TPC-DS 3TB benchmark on Amazon EKS using Graviton4 nodes (r8gd.12xlarge). This compared native Spark SQL against Spark accelerated by Apache Gluten and Velox.

            • Overall Speedup: The Gluten + Velox stack was 39% faster overall, dropping total suite runtime from 3,650.56 seconds to 2,239.93 seconds (a 1.63x overall speedup). Out of 103 queries, 81 queries saw significant improvements.
            • Deep Dive on Query 93 (4.36x Speedup): In native Spark, Query 93 took 207.9 seconds, which Velox slashed to 47.7 seconds. By replacing Spark’s traditional SortMergeJoin with Velox’s ShuffledHashJoinExecTransformer and eliminating pre-join Sorts:
              • The hot 200-task join stage ran ~12x faster (dropping from 1m 16s to 6.5s).
              • Peak per-task memory fell from 3.3 GB (on-heap) to just 80 MB (off-heap).
              • JVM Garbage Collection time collapsed from 3.6 seconds to a mere 35 milliseconds.
              • Serialized shuffle data footprint was reduced by 24% (compressing from 214.1 GB to 162.2 GB), drastically reducing disk and network pressure.
            • The Whole-Stage Codegen Nuance: The benchmark also revealed interesting structural trade-offs. A small subset of queries regressed (most notably q72, which ran slower). Because Velox does not implement SortMergeJoin, it materialized millions of small columnar batches across downstream operators. In contrast, native Spark’s whole-stage codegen fused the entire stream into a single JVM function without materialization. This highlights that while Velox is incredibly fast, understanding your query patterns and join types is key to avoiding edge-case regressions.

            3. The Hardware Acceleration Benchmark: GPU-Native Velox with NVIDIA cuDF

            By integrating NVIDIA’s cuDF library directly into Velox, query plans from engines like Presto can be translated directly into GPU-accelerated pipelines.

            • Unmatched Single-Node Performance: On a TPC-H Scale Factor 1,000 dataset, a single-node Presto C++ run on an AMD 7965WX CPU took 1,246 seconds. Shifting the entire Presto query plan to the GPU via Velox and cuDF reduced that runtime to 133.8 seconds on an NVIDIA RTX Pro 6000, and to a staggering 99.9 seconds on an NVIDIA GH200 Grace Hopper Superchip, representing an over 12x speedup.
            • Multi-GPU Scaling with NVLink: For distributed query execution, Velox supports a UCX-based Exchange operator that runs the entire execution pipeline on the GPU. On an 8-GPU NVIDIA DGX A100 node, using high-bandwidth NVLink for data exchange yielded a >6x speedup compared to using Presto’s baseline HTTP exchange.

            Whether running on standard CPUs, modern Graviton processors, or state-of-the-art GPUs, Velox’s vectorized C++ architecture delivers on its promises. By democratizing elite runtime optimizations, Velox allows organizations to slash query latencies, eliminate JVM garbage collection overhead, and achieve up to a 3x reduction in hardware footprints.

            Conclusion & The Future of Unified Execution

            Velox represents a fundamental paradigm shift in database infrastructure. By extracting common execution primitives, vectorized data layouts, expression compilers, memory pools, and file readers, into a single, highly optimized C++ library, Velox delivers on three key promises:

            1. Unmatched Efficiency: Slashes query latencies, eliminates JVM garbage collection pauses, and delivers up to a 3x reduction in cluster server counts.
            2. Semantic Consistency: Guarantees that SQL functions produce the exact same results whether executed in Spark SQL, Presto C++, or PyTorch.
            3. Hardware Acceleration Ready: Seamlessly scales from x86 and Graviton4 CPUs to NVIDIA GH200 Grace Hopper GPUs via cuDF.

            Learn More & Get Involved

            Follow Us and Join Slack Community