Release DataFusion `48.0.1`

[blaginin]

Previous release: #15771

Creating this issue to discuss if we want to do a patch release to

• fix Clickbench 25x regression: Datafusion 48 Clickbench Q6 and Q0 regression #16444
• topk limit pushdown fixes / disable?
• fix: column indices in FFI partition evaluator #16480
• Wrong results when pushdown_filters is enabled (starting in 48.0.0) #16588

[blaginin]

posting here @alamb's comment for transparency:

I think we can argue if that was a regression or not (it was a change of a default setting, though we are about to change it again in 49....)

That sounds like a valid point to me. Nothing has seriously broken, and if it's too slow, users can tweak the config.

Not sure if we want to fix the topk pushdown issues? Given that our own test suite is failing, can we sometimes produce incorrect results for users?

[timsaucer]

If we do a patch release I would like to get in this bugfix that I found today: #16480

[alamb]

I think the topK stuff was not included in 48.0.0 so there is no need to backport any fixes

I volunteer to run the release process.

In terms of fixing the regression in clickbench performance (which is due to statistics not being gathered by default for dataframes) with I think we have two options

• backport the change of default value here: Set the default value of datafusion.execution.collect_statistics to true #16447 (will align with what 49.0.0 release)
• revert the change in Make SessionContext::register_parquet obey collect_statistics config #16080 which made the defaults consistent

[alamb]

I think this bug is enough to make me want to do s 48.0.1 release

• Wrong results when pushdown_filters is enabled (starting in 48.0.0) #16588

I will begin work on such a release later today

[alamb]

The process will be:

1. We will create backport PRs for fixes, following the instructions here https://github.com/apache/datafusion/blob/main/dev/release/README.md#how-to-backport-add-changes-to-branch--branch
2. I'll plan to make an Release Candidate from branch-48 in the next day or two

I will handle backporting

• fix: column indices in FFI partition evaluator #16480
• Make SessionContext::register_parquet obey collect_statistics config #16080
• Fix parquet filter_pushdown: respect parquet filter pushdown config in scan #16646

If anyone else has fixes or changes they would like to include in 48.0.1, please create a backport PR (targeting the branch-48 branch)