Apache DataFusion

Mentors: Student: renjj

This project aims to implement comprehensive support for correlated subqueries in Apache DataFusion by applying Hyper's 'Unnesting Arbitrary Queries' framework. With a complete solution that can transform arbitrarily nested correlated subqueries into standard join operations through a systematic approach. The implementation will include enhancing the binding phase to handle multi-level nesting, implementing the core dependent join transformation algorithm, applying operator-specific pushdown rules for all relevant operators, and optimizing redundant operations. This approach provides a general solution that works for all types of subqueries (scalar, EXISTS, IN) and handles complex cases involving aggregations and multiple levels of correlation. By adopting this proven theoretical framework, DataFusion will gain robust SQL compatibility and significantly improved analytical capabilities, enabling users to express complex data relationships efficiently.

Mentors: Student: SeoYoung Lee

Apache DataFusion is an in-memory query execution engine optimized for OLAP workloads using Apache Arrow's columnar format. However, to support queries that exceed available memory, DataFusion must spill intermediate results (e.g., from sort or aggregate operators) to disk. As a continuation of the community effort on external query execution, this project aims to improve the robustness of spilling execution and explore further performance optimizations including compression.