On-Demand Recordings from PrestoCon’s, Webinars, Meetups, and more
Tecton is the leading feature platform for real-time machine learning. Rather than build new SQL engines from scratch, Tecton connects to your existing engine to transform raw data into features for machine learning. This talk will cover Tecton’s new integration with Athena for feature engineering. Derek will demonstrate how Tecton with Athena is the fastest way to build feature pipelines and put new models in production.
Today presto supports broadcast join by having a worker to fetch data from a small data source to build a hash table and then sending the entire data over the network to all other workers for hash lookup probed by large data source. This can be optimized by a new query execution strategy as source data from small tables is pulled directly by all workers which is known as replicated reads from dimension tables. This feature comes with a nice caching property given that all worker nodes N are now participating in scanning the data from remote sources. The table scan operation for dimension tables is cacheable per all worker nodes. In addition, there will be better resource utilization because the presto scheduler can now reduce the number plan fragment to execute as the same workers run tasks in parallel within a single stage to reduce data shuffles.