Query Execution Optimization for Broadcast Join using Replicated-Reads Strategy – George Wang, Ahana

Query Execution Optimization for Broadcast Join using Replicated-Reads Strategy – George Wang, Ahana

Today presto supports broadcast join by having a worker to fetch data from a small data source to build a hash table and then sending the entire data over the network to all other workers for hash lookup probed by large data source. This can be optimized by a new query execution strategy as source data from small tables is pulled directly by all workers which is known as replicated reads from dimension tables. This feature comes with a nice caching property given that all worker nodes N are now participating in scanning the data from remote sources. The table scan operation for dimension tables is cacheable per all worker nodes. In addition, there will be better resource utilization because the presto scheduler can now reduce the number plan fragment to execute as the same workers run tasks in parallel within a single stage to reduce data shuffles.

Executing Any External Code in Any Language with Presto – A Universal Connector – Ravishankar Nair

Executing Any External Code in Any Language with Presto – A Universal Connector – Ravishankar Nair

Connector based architecture is one of the powerful features in Presto for extensibility. While we have a solid pack of many connectors, the ability to reuse an existing external snippet to fetch data and access through Presto will make it enormously helpful. For example, consider accessing mainframe code through Presto using simple SQL which is quite cumbersome to handle by creating a connector paradigm. Ravishankar explores how he implemented this feature using a protocol server and a protocol connector which eventually helped him to achieve a patent on the concept.