Intermittent Windows CI failure in "physical_plan::sorts::sort::tests::test_sort_fetch_memory_calculation"

[alamb]

Describe the bug

CI tests begin failing like this on windows

https://github.com/apache/arrow-datafusion/actions/runs/6142807589/job/16665135902

---- physical_plan::sorts::sort::tests::test_sort_fetch_memory_calculation stdout ----
Error: Execution("Spawned Task error: IO error: The system cannot find the file specified. (os error 2)")

To Reproduce

No response

Expected behavior

No response

Additional context

It appears to be intermittent

Here is a failure:
https://github.com/apache/arrow-datafusion/actions/runs/6136441131/job/16651296580

But then it passed on the following run
https://github.com/apache/arrow-datafusion/actions/runs/6137534894/job/16653912130

An then it failed again on the next run
https://github.com/apache/arrow-datafusion/actions/runs/6142807589/job/16665135902

My suspicion is it is related to #7510 from @viirya that started reporting spill file errors

[viirya]

@alamb Yea, I noticed this intermittent test failure locally. But I have no clue why it fails for now. I suspect tempfile/tempdir was dropped too early so the temporary file/dir was deleted before we read it, i.e. https://docs.rs/tempfile/latest/tempfile/#early-drop-pitfall. However the code looks okay as the path: NamedTempFile isn't moved into read_spill:

let result = read_spill(sender, path.path());

So I'm wondering why tempfile was deleted. 🤔

Note that before calling into read_spill_as_stream, we check the file does exist. So the temp file was deleted after proceeding into read_spill_as_stream.

EDIT: proposed a fix at #7534