Presto C++ Unleashed: Dynamically Load Unfenced UDFs, End Rebuilds, and Boost Performance

    By: 

    Saurabh Mahawar, Developer Relations Engineer at IBM

    Dynamic loading in Presto C++ is revolutionizing how developers build and deploy user-defined functions (UDFs). At PrestoCon Day 2025 , Soumya Duriseti explained how Presto C++ now supports dynamic loading of unfenced UDFs, eliminating the need for time-consuming static builds and making it easier than ever to add custom logic without rebuilding the entire binary. This article breaks down the differences between fenced and unfenced UDFs, explains the benefits of dynamic loading, and shows you how to create and deploy custom functions with Presto’s new plugin-based architecture.

    Fenced vs. Unfenced UDFs: A Fundamental Choice 

    Before we explore dynamic loading, it’s essential to understand the two main types of UDFs: fenced and unfenced. They share similarities but differ significantly in execution and implications:

    • Fenced UDFs: 
      • Run in a separate process from the main worker process.
      • Connect via remote processes (e.g., IPC, PCP).
      • Benefit: Enhanced security and peace of mind. Problems in the UDF process won’t affect the main worker, protecting other users’ work.
      • Drawback: Can lead to slower performance and higher cost due to the overhead of maintaining an extra process.

    • Unfenced UDFs: 
      • Run in the same process as the worker.
      • Benefit: Higher performance and lower cost.
      • Drawback:Critical consideration: If there’s buggy code or security issues, the UDF can crash the entire worker process, impacting your work and potentially other users on the same worker. 

    While both options have their merits depending on your priorities for security, performance, and cost, this discussion will focus exclusively on unfenced UDFs.

    The Pain of Static Registration and Why Dynamic Loading is a Game-Changer 

    In the past, registering C++ functions in Presto was a static, arduous process. It involved: 

    • Hardcoding the function registration into Presto’s codebase.
    • Rebuilding the entire Presto binary for every single UDF change. This is a time-consuming process that requires a deep understanding of the Presto codebase and where everything fits. 

    This approach was inefficient and acted as a barrier for developers who simply wanted to integrate a custom function without becoming Presto core experts.

    The solution? The ability to dynamically load custom functions. This new functionality introduces a world of benefits for developers: 

    • Effortless Integration: Easily add cool custom functions like geometry calculations or AI functions to your Presto queries. 
    • Increased Modularity: Promotes better separation between your custom logic and the core engine’s functionality, keeping things clean and organized. 
    • No-More Full Rebuilds: The most significant advantage – you no longer have to rebuild all of Presto just to register a new function. 
    • Enables Plugin Architecture: This capability opens a whole new world of plug-in architecture, allowing for flexible and extensible Presto deployments. 
    • Feature Parity: Brings feature parity with Presto Java, which already offers unfenced UDF capabilities. 
    • Seamless User Experience: Designed for simplicity, allowing users to add functions without extensive knowledge of the underlying mechanisms. 
    • Graceful Failure: Supports graceful failure if there are missing symbols, providing a better debugging experience.

    How Dynamic Loading Works Under the Hood

    The dynamic unfenced UDF functionality in Presto C++ is a core part of the Fusion Next project. For it to work, the sidecar component must be enabled.

    At its core, your custom UDFs are compiled into shared library files. These files have a .dylib extension on Mac OS and a .so extension on Linux. The process is elegantly simple:

    • Create your plugin libraries.
    • Place them in a designated plugin folder on all your Presto workers.
    • When you launch your workers, Presto’s internal dynamic loader automatically scans this directory and loads all detected plugins into the system.

    Once loaded, these custom functions become immediately available for use in your queries. Conversely, removing a plugin file from the directory will make the function disappear.

    Setting Up Your Plugin Directory

    Presto offers flexibility in how you configure your plugin directory. Here are two common scenarios:

    • Custom Plugin Path:
      • You can create your plugin directory in a custom location, for example, /opt/presto-server/plugin (often alongside the etc directory).
      • For this setup, you must explicitly define the custom path by setting the plugin.dir property in each worker’s config.properties file.
    • Default Plugin Path:
      • If you place your plugin directory directly in the root directory of the Presto process (e.g., /plugin at the same level as etc and bin).
      • In this case, you do not need to specify a custom plugin.dir in config.properties, as Presto will automatically look in the root directory for a plugin folder by default.

    Developing and Deploying Your Custom UDF: A Developer’s Guide

    Creating your own unfenced UDF is designed to be straightforward, focusing on the custom logic you want to implement.

    1. Write Your C++ Function:
      • You’ll primarily focus on defining the struct name, return types, and argument types.
      • The function body is entirely dependent on your specific use case.
      • Most of the C++ file is boilerplate code. Documentation is available to guide you on aspects like scalar function registration within the Velox API.
      • Example: Calculating Circle Area
        • A common example is a dynamic_circle_area function that takes a double (data type) radius and returns a double (data type) area (πr²).
        • A crucial step is including the registerExtensions call, which is necessary for the dynamic library to resolve and load the function symbol.

    • Compile Your Function into a Shared Library:
      • Once your C++ file is ready, you’ll compile it into a shared library (e.g., dynamic_function_library.so or dynamic_function_library.dylib).
      • This involves simple compilation commands, adding necessary flags to link the function correctly.

    • Deploy the Shared Library:
      • Drop your compiled shared library files into the designated plugin directory on your Presto workers. This location is either the custom path you configured or the default root directory path.

    • Start Presto:
      • With the libraries in place, start your Presto workers and coordinators. Remember to enable a sidecar worker to be able to see your dynamic functions registered. The dynamic loader will automatically pick up your new function. You can verify this by checking Presto server logs, which will show messages indicating the library

    Verifying Your New UDF

    After deployment, you can quickly confirm your UDF is ready for use:

    • Check Registered Functions: Use the SQL command SHOW FUNCTIONS LIKE ‘dynamic%’ (or a similar pattern matching your function name) to list all dynamically registered functions. You should see your new function, like dynamic_circle_area, listed.
    • Test with a Query: Execute a SELECT statement calling your new function with appropriate arguments, for example: SELECT dynamic_circle_area(21.0). Verify that the output matches your expectations.

    Important Considerations and Future Roadmap

    While dynamic unfenced UDFs offer immense flexibility, there are important considerations:

    • Security Concerns:
      • Unverified Code Execution: Anything containing the registerExtensions function call can technically be loaded. Users must be aware that this introduces a known security risk when using unfenced UDFs.
      • API Compatibility: Inconsistent versioning between your plugin shared libraries and the Presto instance can lead to API mismatches. To mitigate this, it’s strongly recommended that users recompile all their shared libraries every time they update their Presto version.
    • Debugging Challenges: Dynamically introduced functions can be harder to observe and test compared to statically linked ones.

    To address these challenges, the Presto Foundation provides clear documentation, strives for a simple user experience, and offers extensive logging to aid in troubleshooting.

    Future Improvements are actively being planned to enhance the user experience and robustness of dynamic loading:

    • Expanded Function Support: Currently, only scalar functions are supported. The next steps include adding support for aggregate and table-value functions.
    • Enhanced Security: There’s a planned overhaul of the plugin installation process to make it more secure and robust, focusing on validations and security aspects. The architecture for these improvements is currently being finalized.

    Dynamic unfenced UDFs in Presto C++ represent a significant leap forward in empowering developers to extend Presto’s capabilities. By understanding the trade-offs, following best practices, and staying informed about future developments, you can leverage this powerful feature to build more modular, efficient, and customized data processing solutions.

    Follow Presto at Linkedin, Youtube, and Join Slack channel to interact with the community.