Extending Presto at LinkedIn with a Smart Catalog Layer LinkedIn

Extending Presto at LinkedIn with a Smart Catalog Layer LinkedIn

In this talk, Walaa describes how LinkedIn extended its Presto Hive Catalog with a smart logical abstraction layer that is capable of reasoning about logical views with UDFs by using two core components, Coral and Transport UDFs. Coral is a view virtualization library, powered by Apache Calcite, that represents views using their logical query plans. Walaa shows how LinkedIn leverages Coral abstractions to decouple view expression language from the execution engine, and hence execute non-Presto-SQL views inside Presto, and achieve on-the-fly query rewrite for data governance and query optimization.

Prism: Presto Gateway Service at Uber – Hitarth Trivedi, Uber

Prism: Presto Gateway Service at Uber – Hitarth Trivedi, Uber

Prism is a gateway service for all Presto queries at Uber. It addresses Uber specific needs in four main areas – resource management, query gating, monitoring, and security. It is responsible for proxying over three million weekly queries from 6000+ weekly active users across all of Uber. Presto has variable execution times due to high multi-tenancy at Uber. Prism helps in overcoming those challenges using features like query routing, load balancing, query gating, session parameter checks, failover clusters which helps in maintaining a 99.9% availability and reliability SLA for Presto at Uber. Functionality – Query Execution: 1. Async execution API returns data stream 2. Async execution API returns File Descriptor – Routing – Prism can route queries to different clusters based on client sources. Other functionalities: Load Balancing, Query Gating, Failover, Session Properties, Security