Weekly Plan (Andrew Lamb) March 17, 2025

[alamb]

This is an attempt to organize myself and make what I plan to work on more visible

Weekly High Level Goals

• Make arrow release: Release arrow-rs / parquet minor version 54.3.0 (Mar 2025) arrow-rs#7107
• Complete v1 of tree explain plans [EPIC] Complete SQL EXPLAIN Tree Rendering #14914 ready for wider review with @irenjj
• get tpch data generator screaming fast with @clflushopt: Make it easier to run TPCH queries with datafusion-cli #14608
• Work with @XiangpengHao to get Enable parquet filter pushdown (filter_pushdown) by default #3463 ready for merge

Other projects I think are strategically important

For a list of projects, see

• March 17, 2025: This week(s) in DataFusion #15269

These are the ones I plan to look at / review PRs in order

• Anything performance related
• Change mapping of SQL VARCHAR from Utf8 to Utf8View #15096 (comment)
• Spark functions: feat: Add datafusion-spark crate #15168
• Hardening external sorts: A complete solution for stable and safe sort with spill #14692 with @2010YOUY01
• pushing expressions down into scans: Support Push down expression evaluation in TableProviders #14993

Weekly Stretch / Nice to have goals

• Try to move an example or two from datafusion-examples

Background

I am putting this list on github because:

1. I like how github renders checklists w/ PR titles so it is easy to track (I currently have a local text file...)
2. I thought others might be interested from seeing what I am doing / planning to do
3. It makes me feel better that I don't have time to review all the PRs 😭

The way I am trying to prioritize PRs is in the following order

1. Bug fixes
2. Documentation / UX / API improvements (things that make DataFusion easier/better to work with)
3. Performance improvements
4. New features with wide appeal
5. New functions

Note new features and functions are deliberately at the bottom

[alamb]

REview queue:

DataFusion

• Fix predicate pushdown for custom SchemaAdapters #15263
• fix: core_expressions feature flag broken, move overlay into core functions #15217

Features:

• Support metadata columns (location, size, last_modified) in ListingTableProvider #15181

Arrow

• feat: add rounding logic and scale zero fix parse_decimal to match parse_string_to_decimal_native behavior arrow-rs#7179
• Add hooks to json encoder to override default encoding or add support for unsupported types arrow-rs#7015
• feat: impl display for DataType::List arrow-rs#7051
• Add Parquet Modular encryption support (write) arrow-rs#7111

[alamb]

Update here is we completed the initial version of tree explain plan!

• [EPIC] Complete SQL EXPLAIN Tree Rendering #14914

Also the arrow release is out

Next week's plan:

• Weekly Plan (Andrew Lamb) March 24, 2025 #15393