What I see (what problem we are trying to solve)
DataFusion's current join implementations are fairly basic. They are functional enough to run TPCH and TPC-DS, but lack other features such as larger-than-memory processing, ASOF joins, complete subquery support and more.
There seems to be a non trivial desire in the community to improve this.
Some examples of issues / tickets related to enhanced join support / features:
Subqueries (which are implemented as joins)
Join Features
Specialized Joins
Performance
What is blocking significant forward progress
In my mind, the major challenge is that "improving" JOINs can get arbitrarily complicated. There are dozens of academic paper each year on various aspects of join implemnetations, and designing / implementing join capabilities is a substantial engineering effort.
I spent 6 years of my life doing joins at Vertica where they accounted for around 50% of the optimizer's complexity, to give some sense
I don't think the issue is that any particular feature is super complicated to understand, but defining the overall goal, the framework that will accomodate the goal, and then breaking it down into implementable pieces itself I think will require both specialized knowledge and substantial time.
What I suggest
I suggest that people with the relevant skills and time to invest gather together to drive this process worward
- plan out a "join roadmap" (aka prioritize what join features they will push forward)
- Figure out what, if any, new structures are in place
- Start breaking it down into smaller tickets
I can't personally lead such an effort, but I am filing this ticket to try and help connect the relevant people in the community that can.
Some potential people that could help (sorry if I didn't list you)
Related content:
Related blogs (join ordering section in part 2): https://www.influxdata.com/blog/optimizing-sql-dataframes-part-two/
What I see (what problem we are trying to solve)
DataFusion's current join implementations are fairly basic. They are functional enough to run TPCH and TPC-DS, but lack other features such as larger-than-memory processing, ASOF joins, complete subquery support and more.
There seems to be a non trivial desire in the community to improve this.
Some examples of issues / tickets related to enhanced join support / features:
Subqueries (which are implemented as joins)
Join Features
CollectLeftjoins in distributed execution #12454batch_size#14238Specialized Joins
Performance
What is blocking significant forward progress
In my mind, the major challenge is that "improving"
JOINs can get arbitrarily complicated. There are dozens of academic paper each year on various aspects of join implemnetations, and designing / implementing join capabilities is a substantial engineering effort.I spent 6 years of my life doing joins at Vertica where they accounted for around 50% of the optimizer's complexity, to give some sense
I don't think the issue is that any particular feature is super complicated to understand, but defining the overall goal, the framework that will accomodate the goal, and then breaking it down into implementable pieces itself I think will require both specialized knowledge and substantial time.
What I suggest
I suggest that people with the relevant skills and time to invest gather together to drive this process worward
I can't personally lead such an effort, but I am filing this ticket to try and help connect the relevant people in the community that can.
Some potential people that could help (sorry if I didn't list you)
Related content:
Related blogs (join ordering section in part 2): https://www.influxdata.com/blog/optimizing-sql-dataframes-part-two/