DataFusion weekly project plan (Andrew Lamb) - Nov 13, 2023

[alamb]

The idea of this ticket is to log plans I have each week for DataFusion related items, largely for my own personal organizational needs, but also trying to

1. Try some different ways to communicate / coordinate int he community
2. Help provide an interesting summary of what is happening in DataFusion this week

It would be awesome if anyone else who has definite DataFusion plans could try something like this (maybe tickets are the best place 🤔 )

Major items I plan for personally this week:

• Final week of "major" edits to SIGMOD paper (I plan to revise / improve diagrams and some wordig): Write DataFusion paper for (SIGMOD / VLDB / ICDE) #6782 with @ozankabak @Dandandan @yjshen and @JayjeetAtGithub @sunchao and @viirya
• Write up ideas of "growing and sustaining the DataFusion community"
• Improve representation of statistics to support better optimizatons: Introduce a way to represent constrained statistics / bounds on values in Statistics #8078

Projects I am watching / trying to help review and coordinate if needed

• Memory explosion in HashJoin outputs: HashJoinStream memory tracking insufficient #7848 with @korowa
• Disconnecting ListingTable (object store) from streaming usecase, add StreamingTable like API Decouple Streaming Use-Case from File IO Abstractions #7994 with @metegenez and @ozankabak
• ARRAY function hardening: [Epic] General ticket for the concept of the practical implementation of ARRAY #6980 / https://github.com/apache/arrow-datafusion/issues/7988with @jayzhan211 @Veeupup @edmondop and @xudong963 (e.g. Minor: Support nulls in array_replace, avoid a copy #8054)
• Make function packages more modular (make all BuiltInScalarFunctions ScalarUDFs): [EPIC] Unify Function Interface (remove BuiltInScalarFunction) #8045 with @2010YOUY01 (get initial PR merged, help @2010YOUY01 lay the foundation, prepare first function extractions)
• Sort based optimization work started in Combine Equivalence and Ordering equivalence to simplify state #8006 (encapsulating more structs, e.g Encapsulate EquivalenceClass into a struct #8034 and reviews) with @ozankabak and @mustafasrepo
• Review any new ticket / bug reports

[alamb]

Also, help user defined types / LogicalType intoduction with @yukkit : #8143

[alamb]

Wrote up #8152 as part of communty growth

[edmondop]

I was considering this issue #7955 @alamb do you think it's something that makes sense given your plan?

[alamb]

I was considering this issue #7955 @alamb do you think it's something that makes sense given your plan?

I think this is a complicated feature and unfortunately don't have time at the moment to help it through the process. Perhaps other contributors such as @Dandandan or @sunchao / @viirya would have time to help shepherd / work with you on the implementation.

[edmondop]

any simpler, more immediate issues in mind? I tried to search issues tagged with good-first-issue label but many of them had "some history" so maybe there are some of them that come to your mind that are higher priority/more useful

https://github.com/apache/arrow-datafusion/issues?page=2&q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc+label%3A%22good+first+issue%22

[alamb]

any simpler, more immediate issues in mind? I tried to search issues tagged with good-first-issue label but many of them had "some history" so maybe there are some of them that come to your mind that are higher priority/more useful

https://github.com/apache/arrow-datafusion/issues?page=2&q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc+label%3A%22good+first+issue%22

Thanks for asking @edmondop -- I don't know of any that are great candidates at this time (It takes a non trivial amount of time to write them up, and then they get done very quickly :)) -- I will try and write up some more shortly.

I filed some more issues under #6195 that might be interesting to do if you have time and inclination

[alamb]

#8274 for this week